IRC logs for #baserock for Tuesday, 2016-10-18

*** toscalix has joined #baserock00:51
*** toscalix has quit IRC01:01
*** CTtpollard has quit IRC01:59
*** rdale has quit IRC02:32
*** gtristan has joined #baserock06:39
jjardonpedroalvarez: no problem with ansible when using a gitlab mirror instead our trove: https://gitlab.com/baserock/definitions/builds/519376606:56
*** ctbruce has joined #baserock07:39
*** ctbruce has quit IRC07:40
*** ctbruce has joined #baserock07:47
pedroalvarezjjardon:08:09
pedroalvarez<tiagogomes> HEAD in ansible from git.baserock.org points to refs/heads/master08:09
pedroalvarez<tiagogomes> HEAD in ansible from git://github.com/ansible/ansible points to refs/heads/devel08:09
pedroalvarezthe head seems to be fine in gitlab too08:10
pedroalvarezI still think this is nothing to be fixed in the trove08:11
paulsherwoodpedroalvarez: so you believe that tiagogomes' git gc change will fix it?08:14
* paulsherwood is merging it now08:14
pedroalvarezI'm not sure08:14
pedroalvarezbut the change makes sense08:15
*** locallycompact has joined #baserock08:29
pedroalvarezgah, gitlab -> 50008:40
rjekPerhaps they tried upgrading it08:58
tiagogomesyes, my patch will fix it09:18
tiagogomesThe explanation why other ansible repo can work, is because it won't need to run the garbage collection after the fetch09:21
pedroalvarezwhy does it run the gc for g.b.o repo?09:21
* paulsherwood thinks that gitlab.com is unacceptably slow09:22
tiagogomesI don't know the git internals to know that. But I wonder if gc is being run automatically on gbo.09:23
* paulsherwood has had three attempts to merge this so far... it just sits there saying 'merge in progress'09:25
leeminggitlab.com was being painfully slow yesterday too09:25
leemingalthough it has its merits, their official site is too damn slow :(09:26
jmacsI agree09:27
ChrisPolin+109:27
tiagogomesbear in mind that we are using a rc version ;)09:27
paulsherwoodtiagogomes: rc version of what/09:30
paulsherwood?09:30
paulsherwoodtiagogomes: merged, anyway09:31
leemingI wonder if this is to blame : https://twitter.com/gitlabstatus/status/78830143343529164809:33
tiagogomespaulsherwood, gitlab.com. On Friday there was an alert on gitlab.com saying that it was going to be updated to a new version. That new version ended in "rc"09:34
paulsherwood:/09:34
leemingthey also seem to be aware of delayed build queues since the 14th09:34
jjardonanahuelamo: about the usrmerge, take a look to https://storyboard.baserock.org/#!/story/11 , current WIP branch here: https://gitlab.com/baserock/definitions/commits/jjardon/usr-merge11:01
anahuelamota jjardon11:02
tiagogomesjjardon, about the usr-merge… why are you merging /usr/bin and /usr/sbin?11:04
anahuelamoI can guess is a requirement for using ostree as a tool for update the system11:05
jjardonno, I do not think so; Im doing that because it doesnt cost me nothing and will make the system more deterministic (all is in /usr/bin). Some distros do this already11:06
tiagogomesjjardon, can you give one distro that is doing that? I like the separation between the executables intended to be used by normal users, and administration tools11:09
jjardontiagogomes: Arch11:09
leemingisn't fedora also one of them? I had issues with their atomic stuff11:13
tiagogomesFedora implemented usr-merge but with distinct /usr/bin and /usr/sbin directories11:15
tiagogomesBtw I was planning to try to have the branch build, but I assume guess anahuelamo will11:16
tiagogomes*bah*11:16
tiagogomesI assume anahuelamo will do that work?11:17
anahuelamoI'll try use ostree with baserock, so any work needed for that will be on my list of tasks11:17
jjardontiagogomes: I do not think she would mind some help ;)11:18
anahuelamonot at all! everything is always very welcome11:18
tiagogomes:)11:19
rjekHello.  Say I have a trove that is currently pulling from gbo.  What would I need to do to change it to pulling from upstream directly instead?  I assume a Lorry config change, but how much of one?11:53
rjek(I realise that for some repositories this means it won't be possible to switch back as commit SHAs will be different for things that do not originate from git)11:54
*** locallycompact has quit IRC11:54
SotKI think you just remove the upstream trove bit from lorry-controller.conf11:56
SotKand then add lorries you want in local-config/lorries.git11:57
tiagogomeswho has the powers to do so, can we stop mirroring gitlab sandboxlib from github12:03
paulsherwoodtiagogomes: you mean update its lorry file?12:08
paulsherwoodhas sandboxlib officially moved to gitlab now?12:08
tiagogomesI don't think there are lorry files involved. There is a gitlab baserock/sandboxlib repo that is configured to mirror from github codethinklabs/sandboxlib . I think that the canonical location for the repo should be gitlab/baserock12:11
paulsherwoodwell, the official upstream of sandboxlib has been github so far12:12
tiagogomesI don't know how to interpret "officially" here, as things have been moving in a not officially way12:12
SotKybd moved officially, thats the only thing thats gone github --> gitlab afaik?12:13
tiagogomesyes, I know. But I think gitlab where the ybd and definitions are would be a better home12:13
paulsherwoodi'm not disagreeing... just noting that there'd have to be some process to move it12:13
tiagogomesMaybe there is value in having some repos in both gitlab and github and mirroring from each other, but baserock stuff shouldn't be under the CodethinkLabs umbrella no?12:14
* paulsherwood is ok with moving it, but is not a member of upstream sandboxlib12:14
tiagogomesMoving here is, 1) stop mirroring the repo from github on gitlab 2) add a note on Codethinklabs/sandbox lib about the new location of the repo12:15
SotK3) update the lorry file on g.b.o to point to the new upstream12:16
tiagogomesah I didn't know that sandboxlib was on gbo as well12:18
*** gtristan has quit IRC12:24
leemingas I saw it, github is the upstream master. gitlab was only being used by myself to utilise the ci12:28
leemingis this an issue?12:28
leemingi am unaware if there is any official doctrine for these things12:29
*** locallycompact has joined #baserock12:30
pedroalvareztiagogomes: your fix works12:43
pedroalvarezthanks for that12:43
tiagogomesNow if only the sky had an API that I could use to fix this weather…12:46
pedroalvareznope, there are not API's for real clouds12:46
tiagogomesEven openstack is better then12:58
franredtiagogomes, you can always try to implement Zeus in SkyStack and try to see if he can change the weather13:08
franredtiagogomes, depending on the language you will use, you may implement Jupiter instead :P13:09
tiagogomesImplementing Zeus requires god powers, which I am afraid to not possess :P13:13
pedroalvarezhehe, thge deploy stage at gitlab will also build13:19
pedroalvarezthe artifacts won't be there from the previous stage13:20
pedroalvarezvery annoying if you are testing something very low in the stack13:20
*** mwilliams_ct has joined #baserock13:21
tiagogomesmm, in case of no better solution, then remove the build stage?13:21
mwilliams_ctHey baserockers. I'm currently setting up a trove of my own to mirror from another trove. I don't want it to mirror all repositories, as some is proprietary private stuff that we don't want on this new machine. All those repositories start with "delta/codethink/". Should "ignore": ["delta/codethink/*"] block them?13:22
tiagogomesThe problem will become more aggravated when more contributors come on board.13:23
paulsherwoodpedroalvarez: with private runners, we can retain state/artifacts iiuc13:23
SotKmwilliams_ct: I think so13:24
pedroalvarezpaulsherwood: that would speedup things13:24
mwilliams_ctSotK: OK, I'll give it a shot. jic not, is there an easy way to delete later? It's not a crisis if they do get mirrored by accident (there is password protection) but we will need to eliminate them13:24
pedroalvarezit looks like there are deps missing for deployment: https://gitlab.com/palvarez89/definitions/builds/512638713:26
SotKI expect you can probably just rm the repositories in wherever lorry-controller puts them, but I'm not enough of a gitano expert to know if there is anything else needed13:26
mwilliams_ctSotK: ack, thanks for the help :)13:26
tiagogomeshuh? But doesn't it need the yaml module to build?13:28
* pedroalvarez has no idea13:31
pedroalvarez(not much time to debug this things, just some to give them a try)13:31
pedroalvarezthese*13:31
paulsherwoodSuccessfully built fs pyyaml sandboxlib13:35
paulsherwoodInstalling collected packages: six, fs, pyyaml, sandboxlib, requests13:35
paulsherwoodSuccessfully installed fs-0.5.4 pyyaml-3.12 requests-2.11.1 sandboxlib-0.3.1 six-1.10.013:35
paulsherwoodweird13:35
leemingit is weird. I didn't even know I had updated sandboxlib on pypi13:36
SotKdoes gitlab do something to PYTHONPATH which might be confusing the extensions (which are run with their own PYTHONPATH magic)?13:37
paulsherwoodi doubt it - ybd doesn't fiddle with PYTHONPATH, and it's clearly ok with the install13:38
pedroalvarez(i believe this has been working before)13:39
pedroalvarezbut yeah, SotK could be right13:39
pedroalvarezbut the fact that it has been working before makes me doubt as well13:40
SotKhttps://gitlab.com/baserock/ybd/blob/master/ybd/sandbox.py#L215-219 is the PYTHONPATH thing that lets extensions run properly, but I also doubt it if its been working before13:40
* paulsherwood apologises, then... ybd *does* fiddle with it13:41
paulsherwoodleeming: are you close to fixing https://github.com/CodethinkLabs/sandboxlib/pull/25 ?13:52
leemingno, i am blocking on the issue i was discussing yesterday13:52
paulsherwoodhave you seen tiagogomes comment?13:53
leemingwhich one?13:53
paulsherwood'I believe @leeming is missing adding --unshare-user --gid 0 --uid 0 to the bwrap command line.'13:54
leemingyes, we were discussing this yesterday evening. I am getting an "unknown --uid 0" error when running with ybd13:54
leemingi have just been trying to identify where this comes from13:55
leemingif it is ybd or inside sandboxlib13:55
leemingas I print out the command sandboxlib will be using13:55
leemingand i run it manually, and it does not complain of the same thing13:55
paulsherwoodwhere is your code that implements what tiagogomes is asking?13:56
leeminghttps://github.com/CodethinkLabs/sandboxlib/blob/0972adb5395165d91fefc64a793ee9e687d336e5/sandboxlib/bubblewrap.py#L12813:59
paulsherwoodwhat is stdou: ?13:59
leemingstdout?14:00
leemingone of the run_command args14:00
paulsherwoodshould the extend by '--uid', '0', '--gid', '0' by any chance? (i'm just guessing)14:01
rjek log.info("bubblewrap.run_command({}, stdou:{}, stderr:{}, env:{})"14:01
rjekstdou: should be stdout:?14:01
rjekI know that's just a log line, but it's burning a hole into my retinas14:01
leemingsmall type for debugging but yes radiofree14:01
leemingrjek, **14:01
paulsherwoodleeming: and my question above?14:02
leemingrunning14:02
leemingseems to get past the usual error point, which is strange. so thanks for the suggestion, it seems to have resolved that14:04
* leeming doesn't understand what voodoo python is doing to this list14:04
tiagogomes-bwrap_command.extend(['--unshare-user', '--uid 0', '--gid 0'])14:18
tiagogomes+bwrap_command.extend(['--unshare-user', '--uid', '0', '--gid', '0'])14:18
tiagogomesleeming, do that ^, not tested14:18
leemingtiagogomes, yes I did, it seems to be all working now14:21
leemingthanks14:21
leemingsomeone pointed out to me in another channel that these are exec args, therefore need to be split out into separate members, instead of my naive assumption of shell based args that are just " " joined14:23
locallycompacthow do I install pip for python3.5 on debian14:28
locallycompact*ubuntu14:28
tiagogomesWithout knowing, I would give 'pip3' or 'python-pip3' a shot.14:30
locallycompactthat just installs thing sfor python3.414:31
tiagogomesAlso leeming, you push branches named 'fix-tiago'.14:31
* tiagogomes doesn't need any fix!14:31
SotKlocallycompact: `python3-pip` any good?14:31
locallycompactit's just pointing at python3.414:32
leemingit was not clear what the process was, that was a suggestive fix for a PR14:32
tiagogomesI don't think isn't a 3.0, 3.1, 3.2 version of a library.14:33
tiagogomesSo or you install pip or pip314:33
locallycompactlc@warsong:/usr/local/lib$ pip3 --version14:33
locallycompactpip 1.5.6 from /usr/lib/python3/dist-packages (python 3.4)14:33
locallycompactI wnat that last bit to be python 3.514:33
locallycompactso it installs things in python3.5 dist packages14:33
tiagogomesDo you python 3.5 installed?14:34
locallycompactyes14:34
tiagogomes*have14:34
leemingoh, i thought it was just py2/dist-packages and py3/dist-packages14:34
locallycompactmaybe I'll try just a hacky copy14:35
tiagogomesHave you tried to upgrade pip?14:35
pedroalvareznot sure but maybe `update-alternatives --config python` works in this case?14:35
pedroalvarezor is that a only-debian thing?14:35
leemingmaybe pip is printing out the default py3 in that string, but if you were to include the lib when running py3.5, it would still look in the same place14:35
locallycompactno alternative sfor python14:35
tiagogomesIt is not a debian-ism14:35
*** CTtpollard has joined #baserock14:36
leemingtiagogomes, was this gid the only issue remaining for the sandboxlib pr?14:37
leemingif so, i will squish my latest micro commits14:38
tiagogomesleeming, no. I didn't review the PR yet. And I'd prefer to only review it once you get a successful build with ybd14:38
leemingok14:38
leemingi think you just cursed it14:39
* leeming reads the error14:39
tiagogomesAs you may need to tweak the flags in order to get there (as you had for --uid and --gid)14:39
leemingyou everyone is clearly more tuned into this stuff than I, here is a bit of the log. If anything jumps out? http://paste.baserock.org/acehesixil14:41
leemingI will investivate14:41
leemingah, looks like ybd is the one trying to create devices, then presumingly passing them into the sandbox14:43
tiagogomeslulz, I had asked about how devices nodes were going to work a few days ago…14:43
*** gtristan has joined #baserock14:43
paulsherwoodleeming: i guess this is the crux of the matter :)14:43
leemingwhat my response was tiagogomes ?14:43
paulsherwooddoes bubblerwap allow things to believe they can mknod ?14:44
tiagogomessomething not very distant from dunno14:44
leemingyes, something along the lines that it was outside my knowledge area, and i was just following orders, and people could help, which im pleased you have14:45
leemingwhy aren't devices made inside the sandbox anyway?14:45
leeminge.g. stage2-fhs-dirs's build-command have the mknod command14:46
leemingor what ever os.mknod converts to14:46
tiagogomespaulsherwood, why would you want pseudo-mknod? If the build steps for some chunk requires real devices nodes, pseudo-mknod wouldn't help. If not, we perhaps could get rid of the device nodes.14:47
paulsherwoodtiagogomes: i don't know. ybd just reimplements what morph did in this regard.14:47
* tiagogomes points out that also ostree doesn't support devices nodes on an ostree commit.14:47
* paulsherwood is unclear how device nodes would get created for actual systems, if they aren't in the artifacts14:48
*** gtristan has quit IRC14:48
tiagogomesMorph ran as root, so the os.mknode would work14:48
paulsherwoodso does ybd14:48
SotKI think I did a thing to do it when I did the ostree stuff for the artifact cache14:48
SotKbut I can't remember what that thing was14:48
paulsherwoodlol14:48
leemingwhat about my query?14:49
paulsherwoodyou mean 15:45 < leeming> why aren't devices made inside the sandbox anyway?14:50
leemingyes14:50
paulsherwoodafaik they are14:50
tiagogomesI assume that actual systems would have device nodes automatically via a devtmpfs fs14:50
*** gtristan has joined #baserock14:50
leemingbut you were just saying morph/ybd does it14:50
leemingthen these are passed to the sandbox14:50
tiagogomesBut that needs more research.14:51
leeminginstead of putting in the dev creation inside the definitions14:51
leemingtherefore the sandbox would make the dev14:51
leemingwhat am i missing?14:51
* paulsherwood is confused, and multitasking, sorry14:51
SotKah, my patch made it so that the device nodes are created in the staging area when putting an artifact that would need them there, so that wouldn't help with ostree-based systems I think14:51
tiagogomesleeming, permissions to make the dev if you are not root, is what u are missing.14:52
SotK(as opposed to actually storing the device nodes in the artifact)14:52
tiagogomesAlthough you could bind mount the host devices into the sandbox I guess14:52
leemingthat is how it is trying to do it atm14:52
tiagogomesBut I don't know how much of that affects reproducibility and safety14:53
leemingso you have ot be root to make a device?14:53
leemingeven inside the sandbox?14:53
leemingI thought that was what the --unset-user bit did14:53
paulsherwoodSotK: even so, maybe your patch is what leeming needs here?14:55
*** tardyp has quit IRC14:56
paulsherwoodostree is a separate step/problem14:56
paulsherwoodthis is just about getting artifacts to build without root14:56
leemingwhat is this patch SotK ? I thought you were talking about the ostree issue14:56
SotKpaulsherwood: I think it'd just move the problem from creating artifacts to unpacking them14:58
leemingso how do i proceed?14:59
paulsherwoodSotK: you may be right... but can we see your patch?14:59
* paulsherwood reminds himself that a device node is just a file, so there must be a way to do this15:00
*** tardyp has joined #baserock15:01
* paulsherwood wonders if http://blog.hansenpartnership.com/unprivileged-build-containers/ is any help15:02
tiagogomesI would try to individually bind mount each device node that is required, instead of bind mounting whole /dev15:02
SotKpaulsherwood: it is a patch to morph, idk where the similar code lives in ybd, https://gerrit.baserock.org/#/c/291/15:02
SotK(that series was terribly split up too, so that patch may be missing bits)15:03
paulsherwoodSotK: tvm15:03
paulsherwoodleeming: so i think you were right, currently ybd doesn't create the nodes inside the sandbox. but it should15:11
leemingyes, it is done in snadbox.py:create_devices15:12
leemingbut as tiagogomes explained to me, we still wouldn't be able to make the devices without root15:12
* paulsherwood is not helping, here, today, then15:14
* leeming scratches head15:14
leemingwho would be a good person to track down to ask then?15:15
leemingi dont understand why these have to be root permissions/ normal user can't make them15:15
* paulsherwood is still trying to understand this15:20
paulsherwoodtiagogomes: how does ostree handle nodes, then?15:20
* paulsherwood wonders if we coud just disregard the nodes completely somehow15:21
leemingwell i know some of the build commands access /dev/things15:22
paulsherwoodright, but in the sandbox, does dev/foo need to be a device node?15:22
leemingwell i could mount arbitrary things to /dev if I wanted15:24
leemingif that is what you are asking?15:24
leemingi am not sure what the build instructions are doing/require from /dev15:24
leemingthis is for stage2-fhs-dirs btw15:25
paulsherwoodack15:26
* paulsherwood wonders if dev/foo in a tarball is really a device node15:33
tiagogomespaulsherwood I think there are two distinct aspects re devices nodes. Devices nodes required for building a system, and devices nodes on running systems. ostree is agnostic regarding how the filesystem is built (hence why baserock is a good candidate for building ostree's trees). The devices nodes on running systems may be created automatically by devtempfs.15:34
* leeming reads about fakeroot15:34
tiagogomesdevice nodes is not much more than a normal file with a major and minor number associated. Then the kernel will use those numbers to map to the right driver15:36
leemingwhat are the effects of using host dev?15:36
leemingwell, the dev's that are used15:37
leeming/dev/random,null, ...15:37
leemingso like tiagogomes(?) said earlier, mount each device node separately?15:37
leemingfrom the host15:37
leeming(or am i miss reading/understanding)15:38
paulsherwoodthat could be ok for what the builds need... then there's just the question of what needs to go in the artifacts15:38
* paulsherwood should maybe step back from this, though and let gtristan, tiagogomes and others comment properly15:39
leemingokies15:39
tiagogomes[tiagogomes@tiagogomes-thinkpad bubblewrap]$ bwrap --bind /usr /usr --proc /proc --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --chdir / --bind /dev/urandom /dev/urandom --uid 0 --gid 0 --unshare-user /bin/sh -c 'head -n1 /dev/urandom'15:40
tiagogomeshead: cannot open '/dev/urandom' for reading: Permission denied15:40
leeming"execvp /bin/sh: No such file or directory" , but i've come across this  before :)15:41
leemingyou are using a fedora system right?15:41
*** ctbruce has quit IRC15:49
tiagogomesYes, you will need to use the right symbolic links for your os. Or just use some baseroch chroot15:50
*** mwilliams_ct has left #baserock15:50
tiagogomes bwrap --bind /usr /usr --proc /proc --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --chdir / --dev-bind /dev/urandom /dev/urandom --uid 0 --gid 0 --unshare-user /bin/sh -c 'head -n1 /dev/urandom'15:51
tiagogomes��#�n�_S���{06IMbį״Oڂ�����t#)��Ǎ&�T�x8������Y15:51
tiagogomessuccess ^15:51
pedroalvarezrandom success, but success after all15:51
tiagogomesActually, although `--dev-bind /dev /dev` makes all devices nodes available in the sandbox, `--dev` only makes a reduced sanitized list of devices nodes available:16:03
tiagogomesbwrap --bind /usr /usr --proc /proc --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --chdir / --dev /dev --uid 0 --gid 0 --unshare-user /bin/sh -c 'ls /dev'16:03
tiagogomesconsole  full  null  ptmx  ptsrandomshm  stderr  stdin  stdout  tty  urandom  zero16:03
leemingoh really?16:04
leemingbut better to whitelist the devs we need though?16:05
tiagogomesSo for at list building systems to work without root privileges, the build tool needs to stop parsing the 'devices' field in chunks and creating the respective device nodes, and just mount the sane /dev in the sandbox. The fhs-dirs artifact would cease to have device nodes on it.16:06
tiagogomes*at least16:07
tiagogomesI don't see the point of having a whitelist, if there is already one in bubblewrap.16:08
leemingit was a matter of trustability and only mounting what we need16:09
tiagogomesBut if you want to be extremely suspicious, you could verify that each major/minor number of the devices nodes in the host that will be made available in the sandbox have the expected numbers.16:09
*** fay_ has quit IRC16:09
locallycompactargfh16:27
locallycompacthttps://gitlab.com/baserock/ybd/builds/524453416:27
locallycompactThis is doing my nut16:27
leemingis it a root permissions thing?16:28
locallycompactyes16:28
locallycompactor something16:29
locallycompactffffffs16:29
leemingis it running a command? doesn't the ci tell you the command16:29
locallycompactI don't even know how to get to a state where I can test this16:29
locallycompactIt's fetching the repository16:29
locallycompactIt hasn't got to commands16:30
leemingdoing a git clone?16:30
locallycompactdoing git fetch16:30
leemingof the runner container?16:30
leemingor how ever they run internally16:30
locallycompactI turned that off so I could run it on a local ubuntu runner16:30
locallycompactbut that failed for something16:30
locallycompactnow I will try....16:31
locallycompactsomething esle I have no idea16:31
leemingbe nice if the ci linked ot the .ci-whatsyamajiggy file16:31
locallycompactit hasn't got to that yet16:31
leemingyea, but just in general :)16:32
locallycompactyea16:32
locallycompactOH UBUNTU16:32
locallycompactOH GITLAB16:32
leeminghaha16:32
locallycompactOH DOCKER16:32
locallycompactOH YBD16:32
locallycompactOH PYTHON16:32
leemingshhh behave in these parts16:32
tiagogomess/OH UBUNTU/OH DEBIAN/16:32
locallycompactMake debian haskell jamaica16:33
*** CTtpollard has quit IRC17:30
*** SotK has quit IRC17:33
*** inara` has quit IRC17:33
*** bjdooks has quit IRC17:33
*** gary_perkins has quit IRC17:33
*** SotK has joined #baserock17:33
*** bjdooks has joined #baserock17:33
*** gary_perkins has joined #baserock17:33
*** inara has joined #baserock17:34
*** CTtpollard has joined #baserock18:18
*** locallycompact has quit IRC18:41
*** locallycompact has joined #baserock18:54
*** gtristan has quit IRC19:03
*** tardyp has quit IRC19:45
*** tardyp has joined #baserock19:46
*** jjardon_matrix has quit IRC21:17
*** jjardon_matrix has joined #baserock21:19
*** locallycompact has quit IRC21:22

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!