IRC logs for #baserock for Tuesday, 2015-10-20

*** Stanto has quit IRC00:59
*** gtristan has joined #baserock01:04
* paulsherwood believes ybd now culls artifacts by default as well as cleaning up tmpdir01:29
*** gtristan has quit IRC01:30
*** gtristan has joined #baserock01:51
gtristanguys... I cloned git://git.baserock.org/baserock/baserock/lorry.git... I have some python scripts now02:27
gtristanhow do I run the lorry on a single .lorry file to test the import with the command line ?02:27
paulsherwoodgtristan: do you have lorryon your system?02:29
paulsherwood(iirc it's just `lorry lorryfile.lorry`)02:30
gtristanI have a checkout, does that qualify as "on my system" ?02:31
* gtristan has unsatisfied deps... runs around looking for 'cliapp' and tries again02:31
paulsherwoodack02:32
gtristangot it from package manager :)02:32
gtristanhmmm, seems to be doing something02:34
gtristanok02:34
gtristanI think I got it from here :)02:34
gtristanit's doing git svn fetch on a huge repo02:35
gtristanhopefully we can avoid the webkit tarball02:35
* paulsherwood crosses fingers02:35
gtristanI'm trying this: http://paste.baserock.org/pimafimima02:36
gtristanit's hard to tell how they work, but if i recall how svn works, its all just basically paths02:37
gtristanso releases/WebKitGtk/... is where all the releases are found, they should basically act like branches02:37
paulsherwoodshould work i think02:39
pedroalvarezgtristan: check how long it takes and how big is the result02:44
gtristanhmmm02:45
* gtristan enables timestamps in irc client02:46
gtristanalright, can do02:46
pedroalvarez:)02:46
* gtristan expects something around 6GB02:47
pedroalvarezhmm.. IIRC that will be around 18G in g.b.o02:50
pedroalvarezmaybe not that much02:50
* pedroalvarez resurrects mason-x86-3202:51
pedroalvarezI'm happy to prepare the release artifacts if we want to do a release this week02:51
* gtristan notices "I am fork 1" :)02:58
* gtristan is unsure about limiting max jobs03:00
gtristanpaulsherwood, have you tried just 2 instances but letting them just fight for cores/threads without limiting them explicitly ?03:01
* gtristan unsure he understood what instances: 2 does... looks like both instances are trying to build curl at the same time03:04
* gtristan thinks he's just at a part of the build where builds cant be parallel, I've got 90 builds ahead so lets give 'instances' a chance to shine :)03:07
pedroalvarezgtristan: I believe that paulsherwood said that he had that fixed, so that you don't have various instances building the same03:09
pedroalvarezcheck if you are using latest ybd03:09
paulsherwoodgtristan: please use latest tag03:16
paulsherwoodgtristan: max-jobs should be 8  for you03:17
gtristanah03:26
gtristanI see03:26
gtristanlorry:  SVN connection failed somewhere...03:26
gtristan:(03:26
gtristanany idea if it's possible to omit "tags" from the lorry ?03:28
gtristanI think a problem with WebKitGtk is that they work in strange ways, the "tags" in tags/* dont really match up with the branches in releases/WebKitGtk/03:28
gtristananswer is yes03:29
gtristanI can ignore tags/*03:29
* gtristan tries that03:29
gtristanybd app.py has reference to undefined variable in cull()03:36
gtristanpaulsherwood, am I to trash my artifacts completely with this change, now that the treeserver is uncommented ?03:37
gtristanfixing that yields: Culled 0 artifacts in /home/tristan/ybd/artifacts03:40
gtristanso should I mv everything from ~/.cache/ybd -> /home/tristan ?03:40
gtristanok, that seems to have fixed it03:42
gtristanybd master, max-jobs still 403:44
gtristanand one zombie python, the other one building03:44
gtristan1 zombie, and the one that is building I can confirm is limited to 4 gcc's at a time03:45
* gtristan kills and tries again03:46
gtristanah03:46
gtristansorry03:46
gtristan"latest ybd" != "latest tag"03:46
* gtristan still gets max jobs 4 with tag 15.4303:48
gtristanand same situation, 1 zombie and one build limited to 4 jobs03:49
gtristangah, ok so lorry calls logging.debug() all over the place... which is wonderful03:58
gtristanlooking at https://docs.python.org/2/howto/logging.html ... says I should be able to enable it by adding the line: logging.basicConfig (level=logging.DEBUG)03:59
* gtristan would have thought an env var would be nice...03:59
gtristanbut even that seems to not work03:59
gtristanah, --log=DEBUG... gotcha04:00
*** brlogger has joined #baserock04:19
paulsherwoodwhat is 'a zombie'?04:19
gtristanhttp://paste.baserock.org/oyodoresew04:20
gtristanprocess 18451 ^^^^^^^^^^^04:20
gtristanpaulsherwood, unreaped child process04:20
gtristanorphaned04:20
paulsherwoodgtristan: ok i don't know what that is, i'm afraid04:21
paulsherwoodcan you paste your console output?04:21
gtristanhttp://paste.baserock.org/imoyiyeziv <-- ybd console output04:23
gtristanthis is without instances: 2 though, I'm afraid04:23
* gtristan gave up on that for now, need to get it built...04:23
* gtristan can try with explicit max-jobs: 8 and instances: 204:24
paulsherwoodright. ybd forks to do cleanups, for example. maybe that explains the orphan?04:24
gtristanas it will at least pick up where it left off04:24
paulsherwoodthat console output looks ok to me04:24
gtristanpossibly, and it fails to reap the child ?04:24
paulsherwoodi don't know what reaping is. the child processes are supposed to exit04:25
gtristanthat's odd honestly, I would think that from python, you wouldnt have to take care of that04:25
gtristanreaping is basically trapping SIGCHLD and walling waitpid() on it04:25
gtristancleanup work that is supposed to be done from the parent04:25
paulsherwoodwell, i think i'll not worry about it it for now :)04:26
gtristannormally I take it that an orphaned child means that something went wrong, process failed to complete, etc04:26
gtristanbut sure, things seem to basically work anyway04:26
paulsherwood:-)04:27
* gtristan interrupts and tries with max-jobs explicitly set to 8, and 2 instances04:29
gtristanpaulsherwood, do you expect to see "I am fork 2" ever ?04:30
gtristanor fork 1 is the extra instance ?04:30
paulsherwoodyup04:31
* paulsherwood has a plan to change the reporting to 'I am instance' but it's more fiddly than you might think04:31
gtristanpaulsherwood, I think you get more juice from instances: 2 when building ci.morph as you have much more orthogonal dependencies there04:34
gtristanmake sense ?04:34
gtristanthis *cough* again could probably be improved if every package had it's own nicely separate .morph file ;-)04:35
* gtristan ducks incoming traffic04:35
* gtristan suspects something is wrong with the lorry process05:21
gtristanSo05:23
gtristanlorry has been running for an hour and a half05:23
gtristanand this is the output (debug enabled): http://paste.baserock.org/okebuzukof05:23
gtristanit's supposedly doing git svn fetch05:24
gtristanhttp://paste.baserock.org/ekisoroceq <-- lorry related ps axf output05:24
gtristanthe git-svn process is apparently doing something05:24
gtristanaccording to top, it's not dead05:25
gtristannow, what confuses me... is that... I started this process in the lorry checkout directory05:25
gtristanand it created a 'workd' directory05:25
gtristanwhere I would *suspect* it's putting the ultra huge webkit checkout05:25
gtristanbut05:25
gtristandu -hs reveals... that the 'workd' directory, after 1.5 hrs... has accumulated an entire 79MB !05:26
gtristannowhere near the 5GB I would have suspected at all05:27
gtristanAny clues ?05:27
pedroalvarezgtristan: but is it still running?05:33
gtristanpedroalvarez, I just got fed up and finally killed it05:34
gtristanthink I have to investigate deeper whats going on05:34
pedroalvarezgtristan: it might be that it has a lot of history, and that takes a looong time05:35
* gtristan has been cleaning up the rest of the webkit branch in parallel05:35
pedroalvarezI normally go to the git folder being created and use git log to see how it goes05:35
gtristanpedroalvarez, does it download that separately from payload or smth ?05:35
gtristanhmmm05:36
gtristanok lets see05:36
gtristangit log | wc -l says... only a mere 2838 lines of git log05:37
gtristanit's tiny :-/05:37
pedroalvarezin your case that would be: .../workd/WebKitGtk/git05:37
pedroalvarezmaybe git-tmp? I'm not sure05:37
pedroalvarezgtristan: I didn't understand your question :/05:38
gtristanI did that in ./workd/WebKitGtk directly actually05:38
pedroalvarezgit log?05:38
gtristanyeah05:38
pedroalvarezthat is not a git repo I think05:38
gtristanah !05:39
gtristanbecause lorry puts the git in the current directory05:39
gtristanwhich was lorry/05:39
gtristanso that was lorry history05:39
gtristanactually it was 90800 lines of git log05:39
gtristanchecking in the git subdir05:39
gtristanso... looks like you are correct... perhaps it's intensively crunching a lot of history before... well before I dont know05:40
gtristannot sure how the payload of the commits can be separate from the history of it, but I suppose that is the case ?05:40
gtristanlogs for each commit and then diffs after ?05:41
pedroalvarezI don't know :/05:43
pedroalvarezI wonder if there is any git mirror already?05:44
gtristanpedroalvarez, yup there is05:49
gtristanThink we can hack around that one ?05:49
gtristananyway, one problem is lorry is supposed to support svn, so we should be rejecting tarball patches to lorries if there is any VCS05:49
gtristanbut, pedroalvarez ... look at this: http://trac.webkit.org/wiki/WebKitGTK/2.10.x#Howtoaddawebkit-2.10branchtoexistinggit-svnclone05:50
* pedroalvarez opens link05:50
gtristanthere is a git mirror, but it only mirrors trunk, so what I did to test the webkit build, is clone the git mirror, and run those custom git-svn-foo stuff listed on that page05:50
pedroalvarezi see05:51
gtristanSo my thought was hey, they are already basically showing us how to get the branch, there MUST be "the correct" way to do it with the svn type lorry05:52
* gtristan wishes he could kill the lorry process and "resume" it later on, but hopes are not high05:53
* gtristan will have to eat lunch at some point05:53
pedroalvarezI don't think that will work05:54
gtristanno it wont05:54
gtristanlorry patch will probably have to wait for tomorrow, I can run it while I sleep05:54
* gtristan doubts that after lunch, the process will even have time to complete05:54
* gtristan doubts it will complete in less than 5 hours05:54
pedroalvarezI'm happy to give it a try in g.b.o.05:54
pedroalvarezwith fast internet, cpus and ram05:55
gtristanI'm not sure what the bottleneck is05:55
gtristanit's not downloading a lot, and it takes like 5% cpu here05:56
gtristanor less than 2.5 even05:56
gtristanpedroalvarez, http://paste.baserock.org/exelicecap05:57
gtristanshall I gerrit that to start with ?05:57
gtristannote the lacking "tags", the tags are actually pointless anyway05:58
gtristanI tried with "tags" and hit some snag, but that was before I discovered how to enable the python DEBUG logging output05:58
pedroalvarezif you do, please add a -1 note saying that you haven't tested yet how long/big could it be lorrying that05:59
gtristananyway lorry python cases if "tags" in layout so I assume lorry is safe with tagless svn repo05:59
gtristanwell, you want to try it on g.b.o ?05:59
gtristanI would appreciate that, at least it might complete before tomorrow06:00
pedroalvarezyes, I'll start it now ish06:01
pedroalvarezI'll let you know how it goes06:01
gtristanit's certainly doing something06:02
gtristangit log keeps showing new logs06:02
gtristanthanks06:02
* gtristan is [10/229/300] with a fresh webkit build after the latest patch comments addressed06:02
* gtristan suspects people will still complain that I'm making "too many strata"06:03
gtristanfrustratingly slow this is, I solved building WebKit last week, I should be addressing other dependencies to gnome-initial-setup06:04
gtristanpedroalvarez, assuming that the lorry takes ~24 hours to convert from svn (pessimistically), is it at least reasonable to expect < 1hrs updates in the future ?06:07
gtristani.e., it will just pull whatever is new ?06:07
* gtristan goes for lunch06:16
*** gtristan has quit IRC06:17
pedroalvarezit will just piull whatever is new06:19
pedroalvarezI failed to run that lorry, fails for me: http://paste.baserock.org/tuhamoduge06:19
*** paulwaters_ has joined #baserock06:49
*** straycat is now known as lostduck07:01
*** gtristan has joined #baserock07:05
gtristanpedroalvarez, http://paste.baserock.org/bovijexuye07:16
gtristaninterrupted lorry invocation... resumes !07:16
paulsherwoodgtristan: i think 2+8 should still perform faster for (say) gnome system. if there's a bottleneck (only one thing to build), the other instance will wait07:19
gtristanpaulsherwood, yes for sure07:20
gtristanI have noticed a couple of parallelizations07:20
gtristanthey are not so frequent when building gnome (or WebKitGtk target), but they happen07:21
paulsherwoodgtristan: please paste console output for that, i haven't seen anything like that since a couple of tags ago07:27
*** ssam2 has joined #baserock07:29
*** ChanServ sets mode: +v ssam207:29
gtristanpaulsherwood, I generally inspect this by just typing ps axf repeatedly in the shell07:29
gtristanpaulsherwood, I can't say that there is any ybd output to prove this, if that's what you mean07:29
paulsherwoodgtristan: i'm not sure what you're seeing, then. if there's no output from an instance, it's either in mid configure/make/install/artifact-creation, or it's waiting on a timeout/lock-being-freed-on-the-bottleneck-build07:33
paulsherwood(do you have evidence to the contrary?)07:33
gtristanI havent been looking at the output of ybd to make a guess07:33
* gtristan inspects his log07:34
gtristanpaulsherwood, when you say "I haven't seen anything like that since a couple of tags ago", you mean - you havent seen... "Infrequent parallelization in a gnome system build" ?07:34
gtristanI'm saying that I have seen some, but just not many, now that I have the latest "tag"07:35
gtristanit behaves generally well with hardcoded max-jobs to 8 as well07:35
paulsherwoodah, i thought you were meaning you'd seen 2 instances building the same thing07:35
paulsherwoodgtristan: first char of log line is instance number reporting that line07:36
gtristanI have, I think at least07:36
gtristanshouldnt I ?07:36
gtristanonly that its not *very* frequent... my comment from hours ago was just; that I suppose it happens much more when building ci.morph07:37
gtristanoh07:37
gtristansorry, damn I have to read more carefully07:38
* gtristan read building at the same time07:38
paulsherwoodgtristan: no, that shouldn't be possible now. instance 0 (say) claims a lock on building foo. if instance 1 finds itself wanting to build foo,it goes back to the top of the tree... if it finds itself back at foo and foo is still claimed, it waits on the claim or a timeout07:38
gtristanpaulsherwood, yeah, I think I saw that at the beginning of the day when I did not update to latest tag yet, that's certainly fixed07:38
paulsherwoodcool :)07:38
gtristancan I comment in Gerrit on my own patch ?07:40
paulsherwoodi believe so07:41
* gtristan just pushed a new branch, at this point, I can vouch for the correctness of ALL of the patches in that queue, _EXCEPT_ the WebKitGtk addition itself07:41
gtristanoh, I can "Edit the commit message" but I can't seem to add a review07:42
gtristanthat's good enough for now07:42
paulsherwoodgtristan: shall i build it?07:42
gtristanpaulsherwood, the branch wont build because the WebKitGtk does not have a backing lorry07:43
paulsherwoodah.07:43
gtristanI need to test/address the svn WebKitGtk lorry, get that upstream, and then modify the WebKitGtk.morph to point the repo to upstream:WebKitGtk07:43
gtristanfor now, I've addressed comments, for jjardon, I've currently settled for upgrading libxml2 in 2 places07:44
gtristanso libxml2 gets built twice with an unpredictable result I guess, they overwrite eachother in the target07:44
gtristan(that should probably produce some error, maybe)07:44
gtristanand I moved ModemManager -> NetworkManager build, per another comment request07:45
gtristanand I folded enchant/hyphen into the WebKitGtk morph itself07:45
gtristanas per another comment07:45
gtristanso I think for now... just about everything in the queue is ready to go except for WebKitGtk itself07:45
paulsherwoodgtristan: i'm confused... the webkitgtk page says get the code from http://www.webkit.org/building/checkout.html, which mentions git clone git://git.webkit.org/WebKit.git WebKit07:47
gtristanpaulsherwood, that's a mirror07:48
paulsherwoodof what?07:48
gtristanpaulsherwood, look here: http://trac.webkit.org/wiki/WebKitGTK/2.10.x#Howtoaddawebkit-2.10branchtoexistinggit-svnclone07:48
gtristanits a mirror of the real upstream svn07:48
gtristanbut _only trunk_07:48
paulsherwoodah, i see07:48
gtristanso what I did locally actually to test my definitions build, is checkout that git which you mention07:48
gtristanand then fool around with those other tweaks07:49
gtristanyeah07:49
gtristangit log --oneline | wc -l ... tells my I've got up to almost 5000 commits... and counting07:51
* paulsherwood mutters wtf in the general direction of folks who've decided to half-support git07:52
gtristanthey do seem rather disorganized, but it looks like a different crowd of "GTK Port" people who kindof work alongside trunk07:53
gtristantheir branches and release tags are not all back in tags/ branches/ and I guess the code is not in trunk/07:54
*** bashrc has joined #baserock08:02
tiagogomes_mm, baserock doesn't have iptables (userland)08:06
paulsherwoodstrata/connectivity.morph:- name: iptables08:06
paulsherwoodtiagogomes_: i think that's in the qt systems and the genivi08:08
richard_mawalso ppenstack08:08
richard_mawits part of how neutron works08:09
paulsherwoodand weston and xfce08:09
* paulsherwood had failed to scroll down :)08:09
richard_mawthough i think it daft that openstack has libraries shelling out to iptables rather than making a proper library08:09
paulsherwoodinteresting... sounds like an opportunity for a contribution :)08:10
tiagogomes_richard_maw they have a library... that shells out  to iptables08:10
tiagogomes_s/library/private module08:11
richard_mawthats what i meant but failed to unambiguously articulate08:11
paulsherwoodin other news, i'm going to be demonstrating baserock ybd workflows on various combinations for a hands-on with ~50 people on thursday...08:11
paulsherwoodwhich leads me to want to add some stuff about ybd on w.b.o homepage and guides08:12
*** nowster has joined #baserock08:12
* richard_maw is fine with that08:13
paulsherwoodany thoughts/suggestions on how best to handle it?08:13
*** mariaderidder has joined #baserock08:14
* paulsherwood notes we shoulf probably mention CIAT on homepage somewhere, since it is cropping up in conversations too08:14
*** nowster_ has joined #baserock08:15
*** nowster has quit IRC08:18
richard_mawpaulsherwood: I'm not great at documentation, but I think adding the YBD equivalent instructions for all the workflow guides you care about supporting with YBD would be a viable approach.08:27
* tiagogomes_ anticipates confusion08:28
*** franred has joined #baserock08:29
wdutchare trove git-hooks viewable in the webgui?08:30
ssam2what do you mean by trove git-hooks ?08:31
ssam2paulsherwood: i removed all the morph-specific branch and merge stuff from the wiki a while ago, so it may be that we can simply replace 'morph' with $build-tool08:31
ssam2no08:31
ssam2replace 'morph build' with $buildcommand perhaps08:32
ssam2not ideal, but neither is having 2 x docs08:32
tlsayou could do a rosetta page like https://wiki.archlinux.org/index.php/Pacman/Rosetta08:33
lostduckthat's a neat idea08:33
franredtlsa, +!08:34
franred+1 even08:34
wdutchssam2: it's a while since I looked at gitano but I seem to remember there being serverside hooks, I think Kinnison used one for CIAT and I'm trying to find it08:35
richard_mawmight be handy for explaining  how things work compared to other build tools too, so I think it's a good idea, but shorter term paulsherwood needs something up, and the most expedient way to do that would be to s/morph/ybd/08:35
richard_mawwdutch: they aren't in the standard refs namespace, I'll see if I can find an example for you08:35
franredrichard_maw, or s/morph/morph or ybd/08:37
richard_mawwdutch: part of it is that the repository that the global serverside hooks go in isn't viewable from the cgit, since it requires gitano-admin membership to do it08:37
wdutch:/08:38
wdutchmaybe it's with the mason source?08:38
richard_mawhm, there should be one there08:39
wdutchif I can even find that...08:39
richard_mawhm, and the default trove configuration is to not have anyone be gitano-admin directly, leaving trove-admin to do some operations by proxy08:42
richard_mawwdutch: I'll find you that example08:42
richard_mawwdutch: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/trove-setup.git/tree/share/gitano/skel/gitano-admin/global-hooks/post-receive.lua?id=98cf2ab9c80c8efd0df9086353972e7dc73f40b808:44
* wdutch curses weechat for being bad with long urls08:45
wdutchthanks richard_maw08:46
*** jonathanmaw has joined #baserock08:46
*** Stanto has joined #baserock08:48
*** Stanto has quit IRC08:49
*** edcragg has joined #baserock09:09
pedroalvarezhm.. so weston fails to start because http://paste.baserock.org/ekiverifup09:14
pedroalvarezi wonder what will create a /dev/input/event* device09:14
pedroalvareza mouse? a keyboard?09:15
gtristanpedroalvarez, and a power button09:15
gtristanall of those09:15
pedroalvarezaha09:16
pedroalvarezok09:16
*** Stanto has joined #baserock09:16
* gtristan thinks udev is what is creating those in devfs... and then systemd basically plays the role of pausing them and restricting access to them09:16
* gtristan had an xorg experience with that last week09:16
pedroalvarezhehe, this is annoying09:17
pedroalvarezI've been testing the rpi, and failed because maybe it didn't have a mouse/keyboard09:17
gtristanI never really got to the bottom of it, installing the display manager properly magically fixed it09:17
gtristanaha, I see09:18
pedroalvarezalso i had some fun trying to debug the board.. looks like the rpi hangs if I connect the power pin09:19
pedroalvarezprobably because is more power than it needs09:20
gtristanfun09:20
*** petefoth_ has joined #baserock09:24
*** fay_ has quit IRC09:24
*** edcragg has quit IRC09:24
*** rdale has quit IRC09:24
*** tiagogomes_ has quit IRC09:24
*** flatmush has quit IRC09:24
*** jonathanmaw has quit IRC09:24
*** CTtpollard has quit IRC09:24
*** bashrc has quit IRC09:24
*** gary_perkins has quit IRC09:24
*** petefoth has quit IRC09:24
*** petefoth_ is now known as petefoth09:24
*** bashrc has joined #baserock09:24
*** jonathanmaw has joined #baserock09:24
*** tiagogomes_ has joined #baserock09:24
*** CTtpollard has joined #baserock09:24
*** edcragg has joined #baserock09:24
*** flatmush has joined #baserock09:25
*** rdale has joined #baserock09:25
*** fay_ has joined #baserock09:25
*** gary_perkins has joined #baserock09:26
*** locallycompact has joined #baserock09:34
* wdutch doesn't know what to do about the CIAT git hook because he can't hack around with gbo09:43
richard_mawwdutch: what do you need to do?09:44
wdutchwhatever it is that cu10-trove does but sending it ciat1.baserock.org09:45
jonathanmawupdate to the branch baserock/jonathanmaw/k1.0-fixes, the PowerOff button works09:45
richard_mawwdutch: ok, I can add you to gitano-admin, remember great power comes with great responsibility09:46
wdutchbut I don't actually know what to change :(09:47
wdutchfor one thing what is a MASON_ID?09:48
richard_mawwdutch: you should be able to clone ssh://git@git.baserock.org/gitano-admin.git now09:49
richard_mawwdutch: if you revert 0967983042b93c257235cd3b1fffff1605612245 you will have the old hook09:49
wdutchunlimited powah!09:50
* wdutch will look at the hook09:50
gtristanmkay09:52
gtristanso... gnome-initial-setup wants libpwquality09:52
gtristanwhich basically tests password quality09:53
gtristanlibpwquality can't live without cracklib09:53
gtristanis not having cracklib intentional ?09:53
richard_mawnot as far as I'm aware09:53
jjardongtristan: why it would be? :)09:54
gtristanjjardon, because I think we are not using it for login/passwd09:54
gtristanso I thought maybe it's intentional09:54
gtristanI *could* just insert it at the gnome/ level, but generally I think a setup with cracklib is usually integrated lower in the OS09:55
gtristanthats just my impression from a few weeks of looking at setups and reading LFS manuals, though09:55
jjardongtristan: integrate it lower in the stack then09:56
gtristanI think not right away, I suspect that may cause me to lose another day in PAM09:57
jjardonbut that will take you more time probably :)09:57
gtristanalso I am still wary of the idea that fiddling with the lower level package config for gnome effects the rest of the systems09:58
jjardongtristan: I would say have secure pasword is beneficial for all the systems, but I see your point10:00
* gtristan 's impression that every package should be in it's own individual stratum increases with each passing day10:01
jjardongtristan: I have the same feeling when I start to work on this10:09
* gtristan remarks that lorry should probably bail out with a user friendly error when git-svn or mercurial-git are not installed10:09
jjardonlower level stratum seems to more or less work: but it gets more difficult when you go up in the stack10:09
gtristanjjardon, I have some ideas of how it can all come together nicely10:09
gtristanfirst step would be that the morphologies need to understand that they can A.) Have optional dependencies, B.) Be configured with different flavors, C.) Generate different hash keys for the cache based on which flavor a package was compiled for ... and D.) Packages can also be ambivalent as to what dependency provides a required API, in some but not all cases10:11
gtristanjjardon, I think if we can truly embrace the huge complexity of what this is instead of trying to escape it by grouping things statically together, the potential is great10:12
gtristanjjardon, of course if falls apart with the *current* semantics, as it lacks features - once you start duplicating a morph because of a different compilation flavor/set of dependencies, duplication upwards becomes exponential the further up the stack you go10:14
gtristanbut there should not be more than one definition of a given package (and our current approach already does, actually - i.e. libxml2 in GNOME builds twice)10:15
*** edcragg has quit IRC10:19
jjardongtristan: that applies to entires stratum as well, btw (one stratum can depend on different stuff, depending of what you want to build)10:20
gtristanright10:20
gtristanwell, I dont see the difference really10:21
gtristanwhen you say "entries in a stratum", this is just a question of putting all the definitions into one file instead of splitting them up into separate ones10:22
gtristanjjardon, ah, I see you meant to write "entire stratum", and I misread "entires" as "entries" (and interpreted that as entries in a stratum)10:24
gtristanbut really, the stratum serves no real discernable purpose but to generalize these into a group - and in doing so we lose a lot of power and flexibility10:24
gtristanafaics (I'd love to be proven wrong :) )10:25
*** edcragg has joined #baserock10:26
jjardongtristan: agree, but we need to decide if we prefer flexibility or simplicity, I guess (if both can not be achieved)10:31
*** edcragg has quit IRC10:31
jjardonone of the major complains about yocto, for example (and advantage in our side) is that is difficult to understand what are you building taking a look to the yocto recipes. Nobody discussed that yocto is very flexible, though10:32
*** edcragg has joined #baserock10:38
gtristanjjardon, I think with the right approach we can get our tooling to tell us exactly what will be built for a given high level recipe10:39
gtristanhonestly I have a hard time seeing middle ground10:39
gtristantools like buildroot duplicate *everything* into a separate build tree iirc10:40
gtristanso at least you have 100% certainty of what it is you are building10:40
gtristanwith the present situation, taking libxml2 for example... you know using your common sense that libxml2 will only build an extra module if python is present10:41
gtristanand so the library will be installed twice but not differently10:41
gtristanbut we do not *know* that with certainty, that the built library is bit-for-bit exactly the same when built with python present10:41
gtristanthis imo is a level of uncertainty that I personally cannot trust10:42
jjardonagree10:42
*** edcragg has quit IRC10:42
jjardoneverything should be builded only one time, we agree on that; this case is a special case as I can not think in another solution10:43
gtristanwell, there have been others; ICU for instance10:44
jjardongtristan: talking about that: https://gerrit.baserock.org/#/q/status:open+project:baserock/baserock/definitions+branch:master+topic:libepoxy_1_3_1 :)10:44
gtristangrouping these stratum together is prone to this kind of incident10:44
jjardongtristan: beleive that it used to be much more :)10:44
gtristanand then untangling them becomes more difficult too10:44
gtristanbecause, now we have lost some information; we know that 'gnome' depends on 'foundation', but we forget exactly why an individual component in 'gnome' depends on a given component below10:45
gtristanjjardon, so; I think it's obvious at any rate that libxml2 & xslt in the present model do *not* belong in core anyway10:45
ssam2completely agree on the fact that we lose dependency information with the current model, i would like to change it something more flexible10:45
ssam2i'm not sure I understand your comment about python and libxml..10:46
ssam2the cache key of an artifact includes all dependencies10:46
ssam2so a build of libxml with python available will be a different artifact to a build of libxml without python available10:47
jjardongtristan: happy to move them out from core10:47
ssam2that's not a great way of expressing whether or not the python module was built, but it *is* trustworthy10:47
gtristanjjardon, right, the point is I can not efficiently do it because now... other components will be depending on 'core' for libxml2... since we have lost that knowledge base in grouping things, it will take a lot more work to move it out of 'core'10:48
gtristancause every build need validation again10:48
gtristanssam2, that's good to know that the cache key computes it's dependency... in this particular case, libxml2 is installed *twice* under a different morph name10:49
*** edcragg has joined #baserock10:49
gtristanssam2, it's hard to say right now, which of the two will end up in the target image10:49
ssam2yeah10:49
gtristanbut both are certainly built and installed10:49
ssam2originally the plan was to disallow such "overlaps" completely10:49
ssam2but that's never proved practical10:49
gtristanit would be amazing if that level of security was possible10:50
gtristanbut there are many exceptions to catch10:50
ssam2Gentoo does it, I thinj10:50
ssam2*think10:50
gtristanas make install tends to update things in common10:50
gtristanso for instance, ld.so.cache is the first exception10:50
jjardongtristan: exactly; I think I have the record of breaking the ci so believe I though the same as you are saying several times; looking forward to heard about that propossal to fix the current situation :)10:51
gtristanthings which essentially get updated again at system-integration time are normally overlaps10:51
gtristanjjardon, it will need brainstorming, I have a couple of paragraphs but I need to bullet-proof it more10:52
gtristanhave discussed it with pedroalvarez casually over beers, but he was obviously exhausted from the long flight :)10:53
gtristananyway it's good to hear that there is some consensus at least on IRC right now on the things which need fixing - so we can probably move in the same direction in this if a good solution is found & tried :)10:54
gtristaneek, lorried in 12000 webkit commits so far10:57
gtristanonly 388MB used10:57
gtristanand that took.... at least 5hrs10:58
* gtristan is glad that it can resume where it left off, and hopes it will complete overnight10:58
ssam2is that an SVN conversion?10:58
ssam2ouch10:58
gtristanssam2, yup10:59
gtristanI dont even know if it *worked*10:59
gtristanssam2, http://paste.baserock.org/ujivaniten fwiw11:00
gtristanthat should, in theory get us a git mirror of trunk and all the related webkitgtk release branches11:00
ssam2ah... makes sense11:01
ssam2we would probably have to hack it into working on git.baserock.org, as it kills long running jobs normally11:01
ssam2if your conversion succeeds maybe we can just rsync that to the correct place :)11:01
gtristanI am hoping that it would only be one shot11:02
gtristanright11:02
gtristanand every consecutive pull would only pull the commits of the day11:02
*** edcragg has quit IRC11:04
* gtristan pushes lorry patch for libpwquality: https://gerrit.baserock.org/127211:06
gtristanthats a mercurial, tested with lorry :)11:07
* gtristan packs it up and changes locations...11:08
*** gtristan has quit IRC11:11
*** edcragg has joined #baserock11:16
*** gtristan has joined #baserock11:39
gtristanok, frustratingly, I lost all that data after resuming the interrupted lorry failed11:45
gtristanI will run it overnight, but can someone run lorry on this: http://paste.baserock.org/ujivaniten... perhaps on a server instead of a workstation ?11:46
gtristanI have no idea how long it will take, it could literally take an entire week if it continues at the given speed11:46
pedroalvarezjonathanmaw: ta for the fixes :)12:14
pedroalvarezbtw, did you know that weston in the gdp fails to start if you don't have a keyboard or mouse connected?12:16
pedroalvarezWith this error: http://paste.baserock.org/ekiverifup12:17
*** paulwaters_ has quit IRC12:23
CTtpollardI've seen people post similar errors with 1.6 on the genivi-projects / automotive-discussion list pedroalvarez12:24
CTtpollard'I've tried weston-editor on my rpi2 and it will crash when no real12:25
CTtpollardkeyboard is present'12:25
pedroalvarezCTtpollard: good to know. I panicked...12:26
persiaCTtpollard: How does it detect a "real keyboard"?12:33
CTtpollardpersia: I don't know, I was just quoting from a thread12:34
rjektrying to open a hard-coded /dev/input node?12:34
persiaCTtpollard: You can probably work around it with a suitably configured uinput call12:35
persiamod_uinput allows userspace to inject events into the HID event queue, which can be useful for all sorts of things.12:36
*** paulwaters_ has joined #baserock12:48
wdutchrichard_maw: how do I get my changes to ssh://git@git.baserock.org/gitano-admin.git take effect?12:55
richard_mawwdutch: should happen on push13:00
*** ssam2 has quit IRC13:01
wdutchis there are process or can I just push whatever?13:02
richard_mawwdutch: if you want me to look over the commit log, I'll do that, otherwise not really13:02
wdutchprepare for everything to break!13:04
* CTtpollard braces13:04
wdutchI'm going to push then run to the nearest doorframe13:04
lostduckgtristan, if you do move libxml and xslt, beware that the docutils stratum expects to find them in core so docutils will most likely need to be updated as well13:07
lostducki also agree that strata could do with having something like nixos's build-inputs, or else there should be somewhere to make it clear what a chunk's direct dependencies are13:08
lostduck*some way13:08
*** ssam2 has joined #baserock13:16
*** ChanServ sets mode: +v ssam213:16
wdutchurg the new ciat has changed it's IP13:22
*** paulwaters_ has quit IRC13:42
*** paulwaters_ has joined #baserock13:47
*** ssam2 has quit IRC14:10
*** ssam2 has joined #baserock14:24
*** ChanServ sets mode: +v ssam214:24
*** paulwaters_ has quit IRC15:01
wdutchthe ciat hook in gbo is working :)15:26
*** mariaderidder has quit IRC15:30
*** nowster_ has quit IRC15:34
pedroalvarezwdutch: good! :)15:35
wdutchmorph is complaining about "ImportError: No module named fs.tempfs", what do I need to fix this?15:55
ssam2install pyfilesystem15:57
*** franred has quit IRC15:57
wdutchthanks15:58
ssam2there are a bunch of different things on pypi with similar, some of them even install 'fs' packages but without the right things in15:58
ssam2'fs' is the current one I think15:58
ssam2*correct15:58
ssam2or it might be in your distro with a differentname15:58
wdutchfs worked15:59
* richard_maw wasn't a fan of pyfilesystem last time he saw it16:02
richard_mawit assumed that all valid file paths are encodable as unicode16:03
* pedroalvarez wonders where are we running morph that doesn't have all the dependencies16:04
jmacs_richard_maw: (*thinks*) are they not?16:04
richard_mawjmacs_: You could encode some sort of escaping system on top of unicode for them, but you can have paths that don't form valid unicode strings16:06
* richard_maw articulated that badly16:07
richard_mawthis wasn't a theoretical problem either, we had builds fail because of it16:07
rjekThe only byte values that are not valid in a POSIX pathname are NUL and /16:07
rjekISTR libgmp had fruity filenames in its build system?16:08
richard_mawyeah, not sure what caused that16:08
rjekI think the issue was that it was assuming they were valid UTF8 or something?16:09
jmacs_So it had bytes 128-255 in there somewhere?16:10
rjekSomething like that, yeah16:11
* rjek is fuzzy on the details, but it caused all sorts of problems16:11
*** jmacs_ is now known as jmacs16:13
jmacsInvalid UTF8, rather than invalid unicode, makes a bit more sense.16:15
*** ssam2 has quit IRC16:23
lostduckcan anyone think of a chunk we have that uses submodules?16:25
radiofreegstreamer16:25
radiofreelostduck: http://git.baserock.org/cgi-bin/cgit.cgi/delta/gstreamer.git/log/?h=baserock/morph/1.216:25
lostduckradiofree, thanks16:26
*** locallycompact has quit IRC16:26
radiofreeare you writing something to automatically do they "Fix .gitmodules to use baserock upstream" thing? :-o16:26
radiofrees/they/the16:26
lostduckno, i just wanted to quickly check that we can use repo aliases there16:27
radiofreeah16:28
*** jonathanmaw has quit IRC16:35
*** bashrc has quit IRC17:00
* gtristan wonders if we could 'man-in-the-middle' the git submodule functionality, using some mapping of the existing lorries - and not patch or modify the .gitmodules files at all17:05
persiabreaks signatures, but in practice, git doesn't support a provenance horizon through the results of mirroring, so it isn't worse.17:31
*** edcragg has quit IRC18:01
paulsherwoodwdutch: what's going on with ciat, please? it seems to be not actuallty building anything. if it's not used, it shoudl be turned off, or a at least moved to a cheaper box18:27
pedroalvarezpaulsherwood: I believe he has started the migration of it to a cheaper box18:31
*** radiofree has quit IRC18:39
*** radiofree has joined #baserock18:41
paulsherwoodpedroalvarez: yup. but in the meantime why is the live one sitting there inactive?19:15
paulsherwood(i could understand it sitting there running builds, or being turned off)19:16
wdutchpaulsherwood: I haven't turned it off because I thought people were watching it21:45
wdutchI'll happily turn it off whenever21:45
wdutchalthough not right now because I don't have the login details to hand21:46
paulsherwoodwdutch: the mystery is why is it not actually building? i thought it was tracking systemd and linux?21:59
paulsherwood... people are watching it. and wondering why it's doing nothing21:59
wdutchI'll have a look21:59
paulsherwoodtvm21:59
wdutchthe last build was an hour ago22:00
wdutchoh my bad22:00
wdutch10pm yesterday22:00
wdutchtrove notifications are happening, just not to systemd or linux22:02
wdutchthis is expected behavior22:03
paulsherwoodnot for me :)22:18
persiawdutch: Are no commits happening upstream?22:25

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!