IRC logs for #buildstream for Tuesday, 2017-06-13

*** jude has joined #buildstream07:00
*** dabukalam has quit IRC07:40
*** dabukalam has joined #buildstream07:42
*** jonathanmaw has joined #buildstream08:17
*** tristanmaat has joined #buildstream08:18
*** ssam2 has joined #buildstream09:06
*** tristan has joined #buildstream09:50
tristanjonathanmaw, ok so the no dpkg --configure is intentional, I forgot to mention09:52
tristanjonathanmaw, but for your purposes doing it locally on host-arch compatible machine, doing the configures is probably fine anyway09:53
jonathanmawtristan: ok. my problem is that building requires awk, and mawk doesn't install the symlink unless it runs configure09:53
tristanNote that the way I'm doing it for gnome builds is: https://gitlab.com/BuildStream/gnome-modulesets-base/blob/master/elements/base-configure.bst09:53
jonathanmawaha09:54
tristanso you could just use gnome-modulesets-base as a base for your testing, and just rewire base-system.bst to your local debootstrap-ostree creation09:54
tristanand just make your element depend on base.bst09:54
tristanthere are a couple of tricks I do in that repo actually09:54
tristanone of them is the linker-priority.bst09:54
tristanI also do this in the jhbuild2bst generated project.conf:09:55
tristanenvironment:09:55
tristan  LDFLAGS: -L%{libdir}09:55
tristan  LD_LIBRARY_PATH: '%{libdir}'09:55
tristan  PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin09:55
tristan  PKG_CONFIG_PATH: '%{libdir}/pkgconfig'09:55
tristanfor distutils (python) build elements I install into /usr/local, because of setuptools being a bastard09:56
tristanthat's why the PATH thing is there09:56
tristan(you wouldnt need that anyway)09:56
tristanjonathanmaw, then again... for your purposes, hacking your local copy of debootstrap-ostree could just make it configured09:57
tristanjonathanmaw, the multistrap.conf.in does:09:58
tristanignorenativearch=true09:58
tristanWhich is the thing that causes it to avoid doing dpkg --configure even if it happens to be a native arch09:58
tristanmaattristan, any ideas for which current issues might be a good place to start? I'm looking at issue #15 atm10:12
jonathanmawtristan: I get a whole bunch of errors while configuring when running with ignorenativearch=false. I'll use base-configure.10:14
tristanjonathanmaw, sure, we ignore the errors but yeah :)10:21
tristanjonathanmaw, things are sure to go wrong, but after fixing the ultimate #38... I suspect that we might get no errors from that anymore10:22
tristantristanmaat, ummm, that's quite involved, I wouldnt recommend that as a starting point10:23
tristanis there a reason you need that soon ?10:23
tristantristanmaat, I am half hoping we can NOTABUG that one, but maybe it will be necessary, but it involves reinventing how artifacts are stored10:24
tristanmaatNot as far as I know. I'm mostly just looking for something to do that I can sensibly tackle right now10:24
tristantristanmaat, this would be quite useful and is very easy: https://gitlab.com/BuildStream/buildstream/issues/2410:24
tristantristanmaat, basically the gist of that means that, if someone does a typo and writes something like 'rev' instead of 'ref' in a bst element, the engine currently wont tell the user about it10:28
tristanit would need a new API on plugin.py, and the elements and sources would have to use it to validate their configurations; also the loader could use the same code10:30
tristanprobably in _yaml.py we want something that takes a dictionary as input and lists the keys which are valid for a dictionary, something like _yaml.validate_node() would be good10:30
tristanmaatYup, I got that :) I'll see how quickly I can read into this, I haven't done much with YAML parsing before, but it doesn't seem too hard.10:30
tristantristanmaat, ok well, go directly to _yaml.py and also look at plugin.py for the apis we expose to plugin authors (Source and Element plugins derive from Plugin)10:31
tristantristanmaat, because there is already a bunch of helper code there10:31
tristantristanmaat, note that loaded nodes (we refer to the dicts as nodes in our internal lingo)... all have Provenance objects attached10:32
tristantristanmaat, also those contain Provenance info for each member, we use that extensively for raising exceptions10:32
tristanthat's how we deliver the original file column/line info to the user10:33
tristanso there is already a bunch of code that is helpful, most notably you want to look at Plugin.node_get_member()10:33
tristanwhich will fire an exception10:33
tristanmaatOk, thanks, that helps a bunch10:34
gitlab-br-botpush on buildstream@safe-hardlinks (by Tristan Van Berkom): 1 commit (last: _sandboxbwrap.py: Handle EBUSY when trying to remove bubblewrap's device bind mounts) https://gitlab.com/BuildStream/buildstream/commit/91817a5f0130609f5c31330a1e2a7079041b63d210:38
tristanjonathanmaw, can you give safe-hardlinks branch another try ?10:39
jonathanmawtristan: having a go now11:44
tristannice11:45
tristanI'm on the trail of reaping child processes on termination11:46
jonathanmawtristan: I haven't failed while staging any dependencies yet, so I'm hopeful11:46
tristanIt looks like, when forcefully terminating with ^C in non-interactive mode, everything is always reaped11:47
tristanjonathanmaw, you did not before11:47
tristanjonathanmaw, that was not while staging11:47
jonathanmawit's running some configure-commands right now, looks like it's building the kernel :¬|11:48
tristando you have some concurrent builds going on ?11:48
jonathanmawtristan: there were some other tasks, iirc.11:49
tristanjonathanmaw, ummm, can you let it complete a couple of builds ?11:51
tristanjust want to feel a bit safer11:52
jonathanmawtristan: yep11:52
jonathanmawlooks like I'm building systemd, gdb and lsof as well as the kernel, currently11:52
tristanI'm confident in the code... but the change in the nature of the build significantly changes with this11:52
tristanas seen with the leaked device files with bwrap, many systems react differently11:53
tristanjonathanmaw, ok just let me know when one of them completes and it rolls over to the next build11:53
jonathanmawlsof finished, and it's building syslinux, if that helps11:54
tristanthen lets land the crazy hardlinking thing11:54
tristanalright... lets do it11:54
tristanI have been few days with this now and it's working fine here, found an issue on your system11:54
tristanI think we're a go11:54
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 13 commits (last: Added fusepy dependency to setup.py) https://gitlab.com/BuildStream/buildstream/commit/711f9716d88cd8ff31d4c5845faa8393aea23fba11:58
tristanalright, here goes nothing :)11:58
gitlab-br-botbuildstream: Tristan Van Berkom deleted branch safe-hardlinks11:58
gitlab-br-botbuildstream: issue #19 ("Artifacts can get corrupted while running integration commands") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/1912:04
tristannice12:04
tristanOk NB ALL: developer checkouts will require a `pip uninstall buildstream && pip install --user -e .` after pulling recent master12:05
tristanmaatIs there anything that contains valid member names yet? Or should I take that from the documentation?12:15
tristantristanmaat, the documentation is a good starting point...12:17
tristantristanmaat, I think you should start with a low level _yaml.validate_node(node, valid_members)12:17
tristanwhere node is a dict loaded with yaml, and valid_members is a list of strings12:18
tristanand it raises a LoadError12:18
tristanmaatOkay, that makes sense.12:18
tristantristanmaat, for a simple starting point, you can try using it in context.py, which parses the user configuration12:19
tristanwhich is: https://buildstream.gitlab.io/buildstream/config.html#config12:19
tristantristanmaat, so we'll want it first implemented for context.py, project.py and _loader.py (which is a bit more involved)12:20
tristanafter that we can look at adding a Plugin.validate_node() API, or perhaps a simpler API for plugins to use12:21
tristanmaatAre the tests for context.py and loader.py independent?12:22
tristanwell, mostly12:22
tristancontext.py is mostly independent, I think it's also true for _loader.py12:22
tristanmaatOh, I need to worry about python3 on this system. Gotcha12:23
tristantristanmaat, the test cases were developed very early on so most of it is very stand-alone actually, but for the sake of convenience of writing tests, it might be good to change that in the future12:23
tristanWe need >= 3.412:23
tristanmaatOkay, I think I can actually do something now. Thanks again!12:24
tristantristanmaat, you might look at the HACKING file for a bit more guidance... also dont forget to install with the -e option12:24
tristan`pip install --user -e`12:25
tristanthat will ensure that when you run buildstream, it is run directly from your git checkout12:25
tristanotherwise you might find yourself testing things and wondering why your changes are not having any effect :)12:25
tristanmaatYup, already done :)12:26
jonathanmawtristan: I'm getting crashes related to https://gitlab.com/BuildStream/buildstream/commit/d17d99446436334190e3328c4ffdc05ab12892b0#a84d27007c711cee67dd0e3493451db8d78c946a_201_20012:26
jonathanmawitem['element'] is a string, I assume you expected it to be an Element.12:26
tristantristanmaat, I also fully expect that the test cases will have to be modified as a result of your change, i.e. they *probably* contain some invalid yaml12:26
tristanjonathanmaw, oh shit12:26
tristansorry12:26
tristanmaatShould this throw INVALID_YAML or INVALID_DATA, btw?12:27
tristanjonathanmaw, my change was because I was seeing https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugin.py#L73 in messages: "staging {} at {}"12:28
tristanjonathanmaw, that looks much better as an element.name, not a "{}".format(element)12:28
tristanthe latter is more appropriate in exceptions12:28
tristantristanmaat, INVALID_DATA is more appropriate, the INVALID_YAML I added for translating errors from the underlying parser12:29
tristanjonathanmaw, Can you fix it ? And just make sure that element *names* are printed there, not formatted element objects ?12:30
jonathanmawtristan: ok. Also, it looks like https://gitlab.com/BuildStream/buildstream/commit/d39122416192c94fa1deef5053b5e288f8bbf826#b988c7a01766a2dbd7ba8aa301ac7dcf5f75e42f_0_62 snuck in, too12:30
jonathanmawplus I had a problem with missing dirs on bzr elements.12:31
jonathanmawI'll prepare a merge request of misc fixes.12:31
tristan:-/12:31
tristanSo it's calling a non-existing validate_layout(), when it doesnt need a preflight method at all12:31
tristansomehow I missed that12:31
jonathanmawyep12:32
tristanI wonder if I merged the wrong MR ?12:32
jonathanmawI'm sure I removed it, too12:32
tristancrap12:32
tristanAnd that was the time that gitlab decided to have a default enabled "feature" that squashes all commits on an MR :-S12:32
tristanwhich it seems, they have removed that destructive feature altogether now12:32
tristanwow, so I submitted an MR to https://github.com/skorokithakis/python-fuse-sample after finding that his example (which I found helpful) had link/symlink parameters backwards12:37
jonathanmawhrm, I'm failing to repeat the problem with bzr, now. maybe it was incidentally fixed in master .12:37
tristanAnd... my MR was merged *in seconds*12:37
tristanliterally, and again; I commented on one thing there, and he replied *in seconds*, I am a bit worried about this guy12:38
tristanI hope he takes time to eat sometimes :)12:38
* tristan found it quite astonishing to be watching the github page where he submitted the request, and it being merge before his very eyes12:39
ssam2some people set up notification streams that mean they see PRs in real time13:09
tristanyyyyeeaaaaaah13:11
tristanI can see that *someone* thinks I never review code in a timely manner !13:12
tristanhehe13:12
ssam2i think its crazy personally, every PR should be left to ripen for a bit :-)13:13
*** Chris has joined #buildstream13:16
jonathanmawtristan: I think I'm seeing the change to the filesystem at work, now. either that or I've been bitten by it not attempting to assemble the artifact due to it already existing - when assembling base-system-image, it's failing to create /buildstream/sda1 because it's a read-only file system. Time for me to figure out the correct way to make /buildstream so that it's not protected.13:17
tristanjonathanmaw, It's my fault13:24
tristanjonathanmaw, sounds like you found something that I overlooked13:24
jonathanmawtristan: ok. Since you know the most about how the sandboxing works, shall I leave it to you to implement?13:25
jonathanmawor shall I figure out how the sandboxing works and implement it?13:25
tristanjonathanmaw, ok so how to fix this is the question... is it pretty safe to say that with ScriptElement's 'layout' API, anything that is not at the root is going to be read-write ?13:25
tristanjonathanmaw, it will be very easy to fix, my greater concern is API13:26
tristanOh but now... I see13:26
tristanOkay13:26
tristanI think I will have to fix this... I was wondering if I would need this or not13:26
tristanok to answer my own question, yes; any non-root mount will always be read-write13:30
tristanwe dont need to budge on that really13:30
tristanjonathanmaw, Ok so... this might take me an hour... I suppose you are blocking on this ?13:33
tristanjonathanmaw, for *right now*... you should go into the ScriptElement configure_sandbox() method...13:34
tristanjonathanmaw, and you should call sandbox.mark_directory() for each part of the layout needs to be read-write13:34
*** jonathanmaw has quit IRC13:35
tristanjonathanmaw, what I'm going to do in the meantime, is add a keyword argument to Sandbox.mark_directory() ...13:35
tristanoops13:35
tristanWell, I'll finish this what I'm doing first, and then I'll get to that13:36
tristanBut now I think anyway he's not blocking on this for the deb stuff13:36
ssam2good news about the safe-hardlinks branch! that should really improve IO performance right? now that we aren't compressing and decompressing everything all the time13:37
gitlab-br-botbuildstream: merge request (jonathan/fixes->master: Fix some bugs that slipped through) #23 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/2313:38
*** jonathanmaw has joined #buildstream13:39
tristanssam2, right, last week I added a temporary thing to work around it so that the artifact cache was using compression13:47
tristanssam2, the thing was... not only does ostree checkout as hardlinks (which means I was already corrupting the cache but nobody had to know about it... sneaky)...13:47
tristanssam2, but if a file is the same, and only differs in name, e.g. all of those 0644 empty files... then they are also linked together !13:48
tristanso when I ran the dpkg --configure on a checkout like that, I would get all empty files modified (mostly __init__.py files) with some content13:49
tristanhah13:49
ssam2i hit exactly the same issue a few years ago13:49
ssam2that was the final blocker to the project to switch Morph to using OSTree13:49
tristananyway yeah that's fixed and builds will both be faster and consume less disk space13:49
ssam2but we were out of time so Morph never got switched to using OSTree :-(13:49
tristanjonathanmaw, ok so that was assembling an image, so my assumption is that you are not blocking on this for the debian related elements correct ?13:50
tristanjonathanmaw, I'm on it... I need an added feature to the sandbox API, and I need to factor in how this should ideally fit into the ScriptElement API and it's image subclass13:50
tristanssam2, I'm very happy with the copy-on-write hardlinks, though :)13:51
tristanssam2, and it's a great starting point to implement our own fakeroot as a fuse layer, too13:51
jonathanmawtristan: I'm not blocking from it at the moment. base-configure.bst should be run without the rootfs being read-only, so I'm safe.13:52
ssam2tristan, yeah that is a good fix13:53
ssam2actually we had that too, with overlayfs13:53
ssam2the issue was that the system images we produced *also* had all the 0-byte files hardlinked to the same thing13:53
ssam2so the machine would boot, systemd would write a hex string to /etc/machine-id, and every __init__.py file on the system would contain that hex string as well13:54
ssam2which of course caused weird failures in Python programs13:54
tristanssam2, aha, so I did put some thought into this, and had considered filing an ostree bug for it13:54
tristanI think *that* bug is something we still have13:54
tristanI'll file a bug straight away...13:55
tristanNot sure how much people will care about it, but I'll try to make a good case :)13:55
ssam2yeah, I guess the issue is that OSTree is "one way" right now13:56
ssam2your files go in, get deduplicated, but you can't undo that process.13:56
tristanssam2, my opinion is that, even if it's a good optimization, somehow saving the state of *which files* were linked together at commit time, and having at least an option for preserving that at hardlink checkout time is useful13:56
tristanssam2, however that, what you say, is untrue since recently at least13:56
tristanso actually, it's not *necessarily* a problem13:57
tristanbut, my feeling is that in some edge cases, a system itself has an expectation that some files are hardlinks13:57
ssam2ah, if it's possible to preserve info on what files were linked together already, then we're good13:57
tristanssam2, not what I want no, but checking out while forcing files to not be hardlinks is an option13:57
ssam2ah OK13:58
tristanthat is since some time this year13:58
ssam2less of a disaster for us than before then, anyway :-)13:58
tristan--force-copy or such13:58
tristanyeah13:58
tristanBut you know, running dpkg --configure on that sysroot creates a bunch of hardlinks13:58
ssam2but for real systems I think you're right, we'll need OSTree to become a "two-way" store13:58
ssam2and keep track of what used to be a hardlink13:59
tristanSo I *wonder*, I think it's quite plausible for a system to want the same file to be available in more than one place, and then intentionally write to that file13:59
tristanLooks like gitlab's destructive squash commits "feature" re-appeared, but disabled checkmark by default this time14:00
tristanSomeone is watching while I complain in #gitlab :)14:00
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 2 commits (last: x86image: Fix erroneous call to validate_layout) https://gitlab.com/BuildStream/buildstream/commit/1ef4b6878a7694b774b37096ca098338cf81dc3014:01
gitlab-br-botbuildstream: merge request (jonathan/fixes->master: Fix some bugs that slipped through) #23 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/2314:01
jonathanmawta tristan14:02
tristanmaatSo, my "low level" implementation passes its tests. I still have the problem that I can't find valid members for a given key though - the simplest way would be to define a huge dict of key-member pairs, but that doesn't seem very neat.14:04
tristantristanmaat, I would think at least for the low level, a single list-of-keynames is sufficient14:12
tristantristanmaat, for instance, the Context object parses a few sub dictionaries, it can easily have one call to _yaml.node_validate() for each of them right ?14:12
tristanmaatThat's how I have been testing it, it just looks quite messy. I guess there is not much of an option though.14:14
tristantristanmaat, well14:22
tristantristanmaat, what about an API which does both somehow ?14:22
tristanI.e. I can imagine that it will look redundant in code, you have to list out the strings twice, and their expected types once14:23
tristantristanmaat, maybe something which constructs and returns a tuple of values, doing _yaml.node_get() and _yaml.node_validate() in one shot... so that calling code could do something like:14:24
tristanself.foo, self.bar = _yaml.node_get(node, [(foo, str), (bar, int)]) ?14:25
tristanor rather14:25
tristanself.foo, self.bar = _yaml.node_get(node, [("foo", str), ("bar", int)]) ?14:25
tristantristanmaat, not sure if that would be better, it would be less redundant, though14:26
tristanoh yeah, there is also the default value to node_get() to consider, that when left unspecified/None means, that the parsed yaml *must* provide a value14:27
tristanself.foo, self.bar = _yaml.node_get(node, [("foo", str, None), ("bar", int, 10)]) ?14:28
tristanmaatHm. Well, it would look a bit less redundant14:28
tristanfor a required "foo" and a default 10 optionally specified "bar"14:28
tristanRight, but then, it would also be harder to read14:28
tristantristanmaat, I tend to favor readability over saving lines of code to be honest14:29
tristanbut if it can look comprehensive, maybe it's better ?14:29
tristanI have a feeling that _yaml.node_get_multi() could be interesting14:29
tristanjonathanmaw, ssam2, care to weigh in ?14:30
tristanmaatI think it's the better call to just leave it a bit messy. I like this notation, but it would be hard to understand on first glance.14:30
tristanright14:30
tristanI'm a bit ambivalent anyway14:30
tristanI care more about the plugin author facing API, and I have a feeling that those configurations are less complex than what we have to parse in the core14:31
tristanmaatCool, I'll finish implementing this then and look at the API14:32
tristannice :)14:32
tristanssam2, filed https://github.com/ostreedev/ostree/issues/925 in case you're interested14:40
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 5 commits (last: utils.py: Added _kill_proc_tree()) https://gitlab.com/BuildStream/buildstream/commit/d13ef0ac12224647c665171fb65ce60e760b840a14:52
tristandamn process termination, made more robust *again*14:53
tristanThere is still a trailing case14:53
tristanIn *some* cases we hit a 20 second timeout when trying to terminate one of the jobs, quite rare, in that case there is a warning about it and we just exit with an unreaped child; but we do do a kill -9 on everything after that timeout14:54
tristanUnfortunately that can leave a stale SafeHardlinks mount around14:55
*** tristan has quit IRC15:02
*** tristan has joined #buildstream15:15
*** ChanServ sets mode: +o tristan15:15
jonathanmawaha! looks like I've pulled in the necessary dependencies now! It finishes building, and then...16:46
jonathanmaw"error: unable to create '../libpython3-all-dbg_3.5.3-1_amd64.deb': Read-only file system"16:46
jonathanmawnow I can drop into the shell to see what I'm left with16:46
*** jonathanmaw has quit IRC17:04
*** tristanmaat has quit IRC17:17
*** jude has quit IRC17:22
*** ssam2 has quit IRC17:51
*** jude has joined #buildstream18:09
*** jude has quit IRC18:17
*** jude has joined #buildstream18:33
*** pum has joined #buildstream20:00
pum#CHARYBDIS ON FREENODE (amdj)20:02
pum| |  | |  ____| |    |  __ \20:02
pum| |__| | |__  | |    | |__) |20:02
pum|  __  |  __| | |    |  ___/20:02
pum| |  | | |____| |____| |20:02
pum|_|  |_|______|______|_|20:02
pumSOMEONE TOOK OUR WEBSITE AND WE NEED20:02
pumHELP AND SUPPORT TO GET IT BACK UP!!20:02
pumjude tristan Chris dabukalam tiagogomes violeta juergbi mattiasb jjardon[m] albfan[m] waltervargas[m] ironfoot persia gitlab-br-bot brlogger hergertme Laurence20:02
*** pum has left #buildstream20:02
*** hergertme has quit IRC20:13
*** hergertme has joined #buildstream20:14
*** jude has quit IRC20:16
* persia speculates that the website hijack was a result of unsolicited IRC traffic21:01
*** tristan has quit IRC21:08

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!