IRC logs for #baserock for Friday, 2015-03-13

*** zoli__ has joined #baserock06:24
*** petefoth has joined #baserock06:44
*** zoli__ has quit IRC07:12
*** rdale has joined #baserock07:46
petefothThe StoryBoard UI is really not helpful - create a comment, press the 'Comment' button and your comment disappears. Referesh the page a few times and it's still not there. Re-enter the comment, press the 'Comment' button and both your comments are there :(08:00
*** gfinney has joined #baserock08:03
*** a1exhughe5 has joined #baserock08:03
straycatrichard_maw, congrats :)08:14
*** bashrc has joined #baserock09:07
*** jonathanmaw has joined #baserock09:09
pedroalvarezrichard_maw: :D09:13
pedroalvarezrichard_maw: will you patch our systemd now that the fix has been merged?09:13
*** mdunford has joined #baserock09:16
*** gary_perkins has joined #baserock09:17
paulsherwoodrichard_maw: w00t! surprised you didn't show the gbo link, though :-)
*** tiagogomes_ has joined #baserock09:23
richard_mawpedroalvarez: yeah, i'll either do a v219.1 or use a development snapshot for the openstack work. The odds of it happening on other systems are vanishingly small, so updating definitions master to include it is not a high priority, but I'll include it in the openstack branches09:30
tiagogomes_If a Python module is required for running some Python app, should we add it to the build-depends of that app?09:42
KinnisonOnly if it needs the module in order to be built/installed I believe09:42
KinnisonPython is odd about runtime-depends09:42
Kinnisonoften not checking for them in setup.py09:43
KinnisonOTOH if it needs them for running a test suite you intend to enable, then yes include it in the b-deps09:43
tiagogomes_no, the module is only required at build-time09:44
tiagogomes_ops, I mean,  the module is only required at _run-time_09:45
*** ssam2 has joined #baserock09:50
*** ChanServ sets mode: +v ssam209:50
*** franred has joined #baserock09:55
*** CTtpollard has joined #baserock09:56
*** lachlanmackenzie has joined #baserock10:11
*** edcragg has joined #baserock10:15
*** petefotheringham has quit IRC10:35
*** petefotheringham has joined #baserock10:35
radiofreethe chroot needs schroot installing on fedora systems10:36
radiofreeyou end up with a bit of messed up system if you don't have it on when you run manage-baserock10:37
radiofreei'll add it to the wiki10:37
radiofreealso, is there a way to install things somewhere other than /opt ?10:37
jjardonradiofree: yeah, as the wiki said is its needed by Arch as well10:58
radiofreeoh yeah10:59
radiofreeit's the *first thing* mentioned on that page...10:59
radiofree"First, install the following packages from your distribution:"10:59
*** mdunford has quit IRC11:01
*** jonathanmaw has quit IRC11:01
petefothIs there an obvious reason why is taking so long to load? Or is it just me?11:08
mwilliams_ctpetefoth: loading fine for me11:09
petefothmwilliams_ct: OK thanks11:09
*** jonathanmaw has joined #baserock11:17
*** mdunford has joined #baserock11:17
petefothI've created a 'Using Baserock offline' guide I decided a full tutorial taking in the whole 'branch, edit, build, deploy' workflow (all of which *should* work offline with the --no-git-update option) wasn't sensible since only the 'build' part currently works :) All comments, changes, patches welcomed11:27
straycatrichard_maw, thanks for review, that class is basically pointless, will rework11:34
jjardonthanks petefoth11:41
jjardonpetefoth: "deploy built systems across a network" -> AFAIK, you need network connection to deploy to any medium11:42
straycatjjardon, tarball deploy?11:42
petefothjjardon: no, you can deploy to a local VM (I think)11:43
jjardonno sure about tarball deploy, but at least to deploy to raw images you need network11:44
straycatjjardon, what for?11:45
richard_mawstraycat: I also have suggestions for your validation11:45
richard_mawmostly, don't if you don't have a provably complete validation routine11:46
straycatrichard_maw, the validation is very basic11:46
richard_mawinet_pton will do for the IP address stuff11:46
richard_mawI'd rather not validate the hostname than have it too restrictive11:46
richard_mawbut simple stuff like "doesn't contain a space or newline" would be grand11:47
straycatbut anything that's not [a-zA-Z0-9-.]+ is invalid?11:47
jjardonpetefoth: so, I have to rebuild everything first and then I can use morph offline? There is not any command to only download the repos?11:47
straycatsorry a-zA-Z0-9.-]+11:47
straycatsorry [a-zA-Z0-9.-]+11:47
petefothjjardon: not that I found. paulsherwood suggeested the 'modify stage1-bin-utils' trick which worked for me11:52
paulsherwoodjjardon: there may be.11:53
petefothAnd I guess it wouldn't be hard to add one11:53
paulsherwoodi just don't know what it is. i could make a ybd command, if that would help11:53
straycatrichard_maw, i'll use \S if you want11:53
jjardonsorry, but that is a hack, not a propper solution :)11:54
paulsherwoodjjardon: that's fighting talk :-)11:54
* straycat sighs11:54
paulsherwoodstraycat: why?11:54
straycati don't know is all this necessary, really11:54
straycatjust trying to do our best to make something good, and don't need this around11:55
paulsherwoodwhich part? using morph baserock offline is necessary - i've tried to help by describing how i do it. if there's a better way i'm happy to learn it11:55
jjardonshould I add this to the storyboard, in case me or someone have time to implement this?Can anyone confirm a command like that doesnt exist?11:56
petefothjjardon: sorry - I can only confrm that I can't find one :)11:57
radiofreewill this include some functionality to "prepare for offline"? i.e download all the git repos for a system?11:57
paulsherwoodjjardon: not according to `morph help`11:58
straycatrichard_maw, couldnae send a biff eh?11:58
jjardonradiofree: yes11:59
jjardonin jhbuild we have "jhbuild update", that will only update the git repos. ("jhbuild build" will up0date and build everything")11:59
*** zoli__ has joined #baserock12:00
petefothjjardon: that chimes with the conversation with paulsherwood about 'don't use negatives' and 'make *not* updating the default behaviour'12:00
Kinnisonmorph --git-resolve-cache-server=http://nopenopenope/ build system-i-want-to-build-offline12:01
Kinnisonwait for morph ot cache all the gits12:01
Kinnisonsimples *squeak*12:01
paulsherwoodso it can do it, just not in a way that mortals could understand :-)12:01
petefothKinnison: wil that work the first time you try to build?12:01
paulsherwoodpetefoth: try it and find out12:02
paulsherwooddelete al your repos12:02
petefothpaulsherwood: harsh .... but possibly fair12:02
paulsherwoodor rename the repo directory at least12:02
paulsherwoodpetefoth: story of my life :)12:02
radiofreeKinnison: will that download everything  before it starts to build?12:02
petefothpaulsherwood: I can give it a try over the weekend12:02
radiofreeor will it download, build chunck, move onto next, download, build....12:04
ssam2radiofree: it resolves every ref to compute the build graph12:04
ssam2if there's no git-resolve-cache-server then it needs to clone each repo involved in the build to do that12:04
radiofreethat sounds great!12:04
jjardonKinnison: if that work, that wiki page can be vastly simplifyed :)12:05
ssam2if you try this, you'll probably find some other bugs where morph assumes network connectivity and fails later on. That's what happened last time I tried it (Although I fixed the issue I found)12:05
petefothssam2: I bet that's quick ;)12:05
paulsherwoodssam2: i have run morph builds offline, no problem12:05
paulsherwood(following the approach i outlined to petefoth previously)12:05
paulsherwood(and deploys, for that matter, iirc)12:06
petefothssam2: paulsherwood: building and deploying are possible. Currently branch (definitley), edit and checkout (I believe) aren't12:07
paulsherwoodpetefoth: true. my workaround is never to use those commands :-)12:07
*** straycat has left #baserock12:09
* petefoth goes to sort out the networking on his VM, rename some directories, clear his cache then try the Kinnison magic12:10
* Kinnison wonders why noone thought of that before12:14
Kinnisonit was the first thing that sprang to mind when I saw this discussion :)12:15
jjardonthanks Kinnison, it only took one month to know the solution :)12:18
* jjardon will update the ml thread12:18
ssam2actually for the offline thing to work you'll need to remove /src/cache/trees.pickle as well12:18
robtaylormaybe even nice to have an alias of morph --cache-offline? ;)12:19
ssam2once upon a time there was an 'update-gits' command12:19
ssam2i don't know why it got removed12:19
* jjardon doesnt understand why he always found useful the commands that got removed :)12:20
jjardonssam2: what is /src/cache/trees.pickle ?12:20
petefothjjardon: isn't it obvious from the name ;)12:21
ssam2jjardon: read the docstring in, it hopefully explains it12:22
petefoth'fatal: Unable to look up (port 9418) (System error)' sigh!12:26
petefothsorted :)12:29
petefothKinnison magic not working (possibly through my stupidity)
petefothbecause I won't be able to clone the remopte while I'm off line :)12:42
SotKI thought Kinnison's magic was to let you cache all your repos locally without having to run a full build, in order to allow you to work offline?12:44
petefothSotK: that;s why i was stupid to go offline *before* I ran the spells :)12:46
petefothBut now it's failed with
SotKcan you paste the contents of your morph log for that attempt please?12:48
petefothSotK: newbie question - where do I find that?12:49
petefothactually looks like a network problem: ifdown/up then restart and away it goes12:50
persiaI seem to recall --update-gits being removed when it became the default, and not being restored when it stopped being the default (for performance reasons), but my memory could be faulty.12:53
petefothwell it's chuntering away and /cache/gits is filling up :) I'll check it again later, and try kicking off an offline build to run over the weekend12:53
petefothpersia: I believe --update-gits *is* currently the default12:54
SotKpetefoth: if you set up your morph.conf according to the quickstart guide then the log is in /src/morph.log12:54
persiapetefoth: yes, but for performance reasons, it only does that if it can't resolve the ref in the local cache, so you can end up with aging caches if you don't update definitions.12:55
petefothLooks like I didn't do that then:)12:56
ssam2hello all12:57
ssam2I'd like to do some final testing of this afternoon before unleashing it to everyone12:58
ssam2this will involve actually merging one of these patches to master for real:
ssam2if you really don't like any of those, better sign up to gerrit and give a -1 quick :)12:58
ssam2i'm going to lunch now so you have a while!12:59
persia might be an interesting tool for folk wanting to review these.13:01
petefothOH. morph.log is there, and it's quite big13:02
petefothSotK: thanks for the help but it's runnign OK and only falling over occasionally. I'm going to leave it cooking, and check back later13:03
*** CTtpollard has quit IRC13:24
paulsherwoodre the libstdc++, is it me, or tiagogomes_ missing the point?13:33
tiagogomes_paulsherwood, likely the latter :) What's the point13:36
paulsherwoodthe build-depends discussion13:40
tiagogomes_paulsherwood I know the is about the build-depends discussion. But what is the point?13:44
paulsherwoodlibstdc++ does depend on lib-api-headers13:45
tiagogomes_paulsherwood no, strictly speaking libstdc++ depends on stage2-glibc, which by its turn depends on stage2-linux-api-headers (as the headers installed by the stage2-glibc chunk include some headers installed by the stage2-linux-api-headers chunk)13:48
* paulsherwood completely disagrees13:48
* paulsherwood can't discuss now, though13:48
ssam2it's hard to reason about, i don't really understand it.14:21
ssam2if you track runtime deps, and A has a runtime dep on B, and C has a build-dep on A ... then A also has an implicit build-dep on B14:22
ssam2sorry, C  also has an implicit build-dep on B14:22
ssam2for build-deps, i kind of see what paul says -- stage2-glibc doesn't need linux api headers at runtime14:23
ssam2wait, actually it probably does. if linux-api-headers isn't installed in the staging area, a bunch of stuff in /usr/include/ will be missing, right?14:30
ssam2so it's hazy whether to call this 'a runtime dependency between glibc and linux-api-headers' (in that glibc isn't as useful without them), or 'a built time dependency between the thing you are currently building and linux-api-headers' (in that you can't build the thing without linux-api-headers available)14:31
paulsherwoodstag2-glibc, when installed, does not include the headers14:33
ssam2right. so to actually build stuff you need both glibc and linux-api-headers14:38
pedroalvarezwhy we don't disable the recusrive build dependency thing and find out?14:39
ssam2heh, good plan14:39
paulsherwoodpedroalvarez: i'll buy everyone who has ever written a line of code in morph a beer, if it builds without :-)14:40
pedroalvarezI wonder where are we regarding runtime dependencies14:40
paulsherwoodas i said at the time, we've got a long way without them. i'm trying to ensure we've properly nailed build-depends. if we still need runtime-depends after that, i'll be suprised, but happy to support extending the syntax14:41
ssam2i think if we removed recusive build-dependencies we'd be /more/ likely to need to track runtime dependencies as well.14:43
pedroalvarezOTOH, sometimes we are using build-depend wrong, just because they are runtime dependencies of other things like now with linux-api-headers, glibc, etc14:43
paulsherwoodssam2: interesting theory - :-)14:43
tiagogomes_ssam2 yes14:44
paulsherwoodif you're right, then my move to make the build-depends explicit should help the cause of establishing the need for runtime-depends :-)14:45
ssam2first patch merged to master of definitions.git via Gerrit:
pedroalvareznit-pick: it doesn't have Reviewers-by in the Merge commit14:49
ssam2do we care?14:49
pedroalvarezbut i remember flagging this in the past, and I think it was too dificult?14:49
pedroalvarezssam2: no, it is in gerrit14:49
ssam2right. I'm sure it'd be possible to have a Gerrit plugin do this ... but we might have to write it first14:49
pedroalvarezI mean, the reviewers info is in gerrit, so I wouldn't mind14:50
paulsherwoodif we can fix it easily, would be better to do so14:50
ssam2i'll seek out a suitable plugin14:51
pedroalvarezssam2: it was just a nit pick, though. Excelent work :)14:52
jjardonssam2: \o/14:54
ssam2right now Gerrit doesn't create a merge commit at all if it doesn't need to ,actually14:54
ssam2I think this is configurable -- any strong opinions on this?14:55
pedroalvarezthe only reason we were creating merge commits, was to add reviewers info, right?14:55
paulsherwoodi'm not enough of a git expert to know... Kinnison ?14:55
KinnisonYes, we were using merge commits to actually record the review information14:55
ssam2since Gerrit changes can only be one commit anyway, it seems a bit pointless14:55
Kinnisonssam2: I thought they could be a patch series, is that not true?14:56
ssam2i thought so too14:56
KinnisonOh :(14:56
paulsherwooddoes a merge commit make it easier or harder to trace stuff a year later?14:56
Kinnisonpaulsherwood: it makes it easier to trace without the gerrit data14:56
paulsherwoodreally??? petefoth ??14:56
Kinnisonpaulsherwood: but with the gerrit data there's little to no need I imagine14:56
ssam2it's not his fault!14:56
* Kinnison thought we were going to use zuul to do the merge though14:56
ssam2hmm, maybe so14:57
ssam2but not for now, anyway14:57
* paulsherwood recalls petefoth was involved in shepherding stuff for patch series handling in gerrit, ages ago14:57
ssam2oh! hmm14:57
KinnisonIt's possible it got in, and then removed again14:57
Kinnisonfeatures don't always stick around14:57
paulsherwoodsurely other large-scale projects need patch series, not just us (and the original sponsoring organisation)14:58
ssam2you can link related changes under one branch name14:59
ssam2so there is kind of a workaround for the limitation14:59
jjardonssam2: probably you already know, but you can configure the way you merge changes with this:
* pedroalvarez fails to use git-review:
ssam2try git-review from git master, I found it a bit less buggy15:18
pedroalvarezsame error. I guess git-review needs a patch15:21
* paulsherwood giggles at ssam2's email introducing gerrit.baserock.org15:21
*** wschaller has joined #baserock15:24
jjardonyou can add "Reviewer" in the gerrit web ui for a specific change, but not very sure what "Reviewer" means in the gerrit context15:32
rdaleis it possible to build a non-systemd based baserock image with sysv init or similar?15:33
jjardonrdale: I think so, check the minimal system15:33
persiardale: Yes.  The "minimal" system is an example.15:33
rdaleah ok thanks - i couldn't find a chunk with 'init' in its name15:34
richard_mawthat's because it uses busybox init15:35
rdaleso it's done by the busybox-init extension?15:35
* bashrc got gerrit working with emacs, with some limitations15:36
jjardonah, the "reviewers" files is to _request_ a review to specific people15:36
persiabashrc: With ?15:36
bashrcno, magit-gerrit15:37
bashrcpersia: the main limitation is inability to add comments to changes15:37
jjardonssam2: maybe is a good idea to have default reviewers from some of the projects?15:37
persiabashrc: Ah.  magit-gerrit seems to be similar to git-review, whereas gerrit-download is more like gertty15:38
persiaI don't think we need default reviewers.15:38
persiaLooking at all the open ones works well enough, unless/until we end up with so many patches that things are really getting lost.15:40
persiaHistorically, most of the reviewers have been reviewing patches in everything, which culture I think we want to nourish as long as we can, before splitting into definitions team, morph team, etc.15:40
persia is also a nice reporting tool, if you like that sort of thing, rather than the more interactive gertty.15:40
*** straycat has joined #baserock15:40
straycatdoes anyone else get occasional connection refused from gbo?15:41
jjardonssam2: nevermind, I think is actually not possible.  Interested reviewers can watched project and receive email notifications instead15:41
straycat% g pull15:41
straycatssh: connect to host port 22: Connection refused15:41
straycatwas happening for me last night too15:42
Kinnisonconnection-refused is unusual since it implies the sshd wasn't listening15:42
straycatjust got it again15:42
straycatseems intermittent15:42
* pedroalvarez looks at mason to see if mason is having ssh troubles, and realises that is red15:43
pedroalvarezbaserock-import needs morphlib to build15:43
ssam2oh, my fault15:43
straycatpedroalvarez, to run?15:44
*** Nico-D has joined #baserock15:45
pedroalvarezstraycat: maybe just tu run but it checks for it when building?
ssam2i've pushed a fix for that15:46
ssam2I was about to submit a patch for lorries.git via gerrit, then realised it's not there!15:49
ssam2i shall add it15:50
ssam2lorry-controller seems to be eating a lot of cpu on git.baserock.org15:50
ssam2also its state database is 3.9GB15:51
ssam2i don't know if these things are related, but ...15:51
ssam2perhaps we need to think about retiring mirroring history beyond a certain point.15:51
Kinnisonssam2: Ouchie15:52
Kinnisonssam2: I imagine sqlite won't be pleased about a 4G statedb15:52
Kinnisonssam2: sounds like lorry controller needs a mechanism to expire old log data15:52
ssam2it won't actually cause any harm if I just remove the statedb, I think.15:52
ssam2i'm not proposing to do it now. but all we'd lose is logs of old jobs15:53
ssam2i'd like a mechanism to expire data, hopefully someone will have time to look at that15:53
straycatsounds useful15:53
pedroalvarezthe database doesn't have anything useful really..15:53
Kinnisonpresumably it'd be a pretty cheap sqlite command, and adding simple verbs to lorry-controller looks pretty straightforward15:53
Kinnisonfirst-run of it might eat a lot of CPU so perhaps resetting beforehand might make sense :)15:54
Kinnisonpedroalvarez: It's useful since it contains the spread of jobs, but that can be recalculated if you rm the state15:54
*** Krin has quit IRC15:56
persiaIs it possible to archive old data, rather than expiring it?15:58
persiaSo that the DB used by active lorries differs from the DB used when digging through old history?15:58
*** Krin has joined #baserock16:05
*** zoli__ has quit IRC16:07
KinnisonWith enough engineering time, almost anything is possible :)16:18
pedroalvarezhm.. time to add git-review to our devel systems?16:24
Krinwhat is the thing that is not that makes it only an "almost"?16:24
*** gfinney has quit IRC16:26
ssam2nobody seems to have requested to be in the Mergers group on Gerrit yet :(16:27
*** gfinney has joined #baserock16:27
*** ratmice__ has quit IRC16:44
*** ratmice__ has joined #baserock16:44
persiassam2: I thought we were initially populating the group with the current mergers.  Shall we have a review/election process?16:47
* Kinnison votes +1 on "current mergers by default"16:47
ssam2persia: indeed. but we need those people to sign up and tell us their usenames before we can enroll them in the group on Gerrit16:48
ssam2i guess it could be automated based on email address, actually16:50
ssam2but it still needs people to have Gerrit accounts first16:50
pedroalvarezbtw, I didn't have to create an account this time. I assume that is becaue a database in another host?16:51
ssam2you created one already16:52
persiassam2: Hrm.  Maybe the announcement wasn't clear enough?16:54
ssam2perhaps not. but everyone is quite busy, too16:54
jjardonssam2: are you in mergers? I can not see you in,members16:58
ssam2i can see myself there16:59
ssam2i seem to be in every group16:59
jjardonssam2: maybe there is an option so other can see who is in the group16:59
jjardonothers can't*16:59
ssam2yeah, maybe17:00
jjardon(I can not see any other members in any group)17:00
*** a1exhughe5 has quit IRC17:03
* petefoth checks backscroll and recals that our work on gerrit was adding topics17:37
petefothtopics allow more than one changeset to be grouped together17:38
petefothI had bgelieved that a 'changeset' could be more than a single commit, but it was a long time ago and it wasn't me who did the real work :)17:38
ssam2ah, you added topics17:48
ssam2they are a pretty good workaround for the 'one patch per change' limitation.17:48
pedroalvarez--with-suexec-uidmin=500  --with-suexec-gidmin=10017:50
pedroalvarezwhat on earth these things are!17:50
* pedroalvarez shall investigate17:51
*** bashrc has quit IRC17:51
*** Krin has quit IRC17:53
*** gfinney has quit IRC17:56
* petefoth wonders if the reported g.b.o connection refusals and the intermittent failures of his `morph --git-resolve-cache-server=...` jobs17:56
petefothwhis means he's going to have to keep checking his VM and restarting the job until everything is cached17:57
persiaSadly, probably yes.  In the worst days of my use of morph, I would wrap it in while(true), etc, as morph gets pretty fast and efficient once it has actually done everything, so the ninety-first time through the loop isn't very time-consuming.17:58
petefothI have been doing  ifdown/ifup between runs, on the assumption that the problem is with my netwrorking. Looks like thag may not be necessary18:00
petefothThough the error is see is 'fatal: Unable to look up (port 9418) (System error)' which indicates it is my networking not g.b.o being shy18:01
persiaThere are so many potential sources of that failure, it is hard to count.  Your network could be part of it, but yeah, your machine isn't even trying to reach g.b.o yet when that happens.18:02
petefothpersia: thanks. I'll just give my VM lots of attention :)18:03
petefothThough I have an appointment with 'The Second Best Exotic Marigold Hotel' later18:03
ssam2are those errors coming from git ?18:03
ssam2maybe Morph needs to retry 'git clone' a few times if it gets an error, i'd never thought of that18:05
petefothssam2: the errors I see are like this:
ssam2right, so it's `git clone` giving up immediately on error18:05
petefoththat is while running Kinnison's 'morph --git-resolve-cache-server=http://nopenopenope/ build  systems/devel-system-x86_64-generic.morph' spell18:06
ssam2sure. it's expected behaviour, really18:06
ssam2but we could add a simple retry loop to Morph where it runs `git clone`, if this is a common thing18:06
ssam2and i tend to assume bad connectivity is a common thing18:06
*** mdunford has quit IRC18:07
petefothhmm: just failed again. If I rerun the spell without doing ifdown/ifup then it fails with the 'unable to lookup' error, so it's more than just a flaky network connection18:08
* petefoth prebably should whip up a script with the spell and 'ifdown && ifup' in a while loop as persia suggested but is runnign out of time 18:11
*** straycat is now known as sadcat_18:11
*** ssam2 has quit IRC18:16
*** tiagogomes_ has quit IRC18:18
*** gary_perkins has quit IRC18:49
*** jonathanmaw has quit IRC18:52
petefothi've added a task 'restore the 'update-gits' command ' to the 'Improve offline functionality in 'morph' story: it;s a slightly more intuitive UI than the 'morph --git-resolve-cache-server=http://nopenopenope/ build ...' incantation (functional though that incantation may be)18:54
*** lachlanmackenzie has quit IRC18:58
*** edcragg has quit IRC18:58
jjardonpetefoth: +119:00
*** wschaller has quit IRC19:04
petefothOK all the gits seem to have been cloned, so I've killed the build, taken the VM offline and kicked off a --no-git-update build, whcih I guess may be finished by Monday morning.19:14
petefothSee you next week :)19:14
*** zoli__ has joined #baserock19:14
*** pacon has joined #baserock21:10
*** zoli__ has quit IRC21:18
*** Nico-D has left #baserock21:27
*** zoli__ has joined #baserock21:40
*** zoli__ has quit IRC23:24

Generated by 2.14.0 by Marius Gedminas - find it at!