IRC logs for #buildstream for Wednesday, 2017-07-19

*** jude has joined #buildstream07:00
gitlab-br-botbuildstream: issue #48 ("Incorrect pipeline total element count for `bst track`") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/4807:32
*** tlater has joined #buildstream08:31
gitlab-br-botbuildstream: issue #49 ("BuildStream needs artifact revisions") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/4908:39
gitlab-br-botbuildstream: issue #46 ("Strange artifact pull failures on GitLab CI") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/4608:40
*** tristan has joined #buildstream08:50
*** jonathanmaw has joined #buildstream08:53
*** ChanServ sets mode: +o tristan08:55
*** ssam2 has joined #buildstream08:59
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 1 commit (last: _frontend/main.py: Naming workspace functions with workspace prefix) https://gitlab.com/BuildStream/buildstream/commit/67f9ea78575ab94f99dedb2a1875fdd386e14bd509:00
jonathanmawin the buildstream-tests repo, what's the difference between the base-platform elements and the base-sdk elements?09:16
tristanjonathanmaw, it's incorrect09:16
tristanjonathanmaw, I noted that to sam the other day, I can show you the correct configuration09:16
tristanbut I have to push it somewhere :-/09:17
tristanjonathanmaw, basically I misunderstood what it was back then, we only need to stage the sdk, the platform is a subset of the sdk already09:17
jonathanmawtristan: ok09:17
tristanand we need to stage it in /usr, _AND_ we need to stage a local import element that only does /usr merge at slash09:17
tristanso in my branch to build flatpaks with GNOME modulesets, I'm using something like that, just havent pushed it anywhere relevant yet09:18
tristanSo... my machine is in the middle of a webkit build which is almost at the end of the flatpak runtime build; yelp remains to be built on top of it but that's been known to succeed before09:36
tristanSo anyway everything is building, with the flatpak variants of the GNOME modulesets project in place09:37
tristanThat is one big part of the milestone done, but there remains some tinkering before it can really be used as a flatpak runtime09:38
tristanFrom what I anticipate needs doing; A.) Need to define flatpak specific split-rules, probably in the project variant, with some tweaks in some elements to match the "cleanup-files" and such09:39
tristane.g. https://git.gnome.org/browse/gnome-sdk-images/tree/org.gnome.Sdk.json.in#n37409:39
tristanThat json says that "These files need to go into the SDK but not the Platform"09:39
tristanB.) Need to figure out how to split the locales for the separate Locale extension (still revolves around composing split outputs)09:40
tristanC.) Need to add a compose element to split these09:40
tristanD.) Need an element which basically just spits out some metadata for the outputs09:41
tristanSo, that more-or-less covers it09:42
tristanthere are some other details, but maybe the above can be brute forced enough to make a demo of running a real flatpak against a buildstream-built GNOME SDK09:42
tristanjuergbi, good call on avoiding the push/pull queues if summary fetch fails09:45
tristanjuergbi, anyway... I'm going to need you to fix https://gitlab.com/BuildStream/buildstream/issues/4709:46
tristanthat's pretty crucial09:46
tristanright now every push is failing on gnome7 because of that09:47
juergbiyes, i was planning to do pretty much what you described (but also return checksums for the refs)09:47
juergbionce my summary branch is completely finished09:47
tristanjuergbi, I'm giving you shell access hold on...09:48
tristanwell, I'll let you know... :)09:48
juergbita09:49
juergbitristan: wondering whether i should implement the other push optimization (only send objects as needed) if i anyway have to change the protocol. don't know yet how much effort that will be, though09:49
tristanjuergbi, shell into gnome7 following these steps10:25
tristanfirst ssh -A -p 22100 juergbi@jumpserv.colo.codethink.co.uk10:25
tristanjuergbi, that will get you onto jumpserv, which can login to the 8 GNOME arm servers10:26
tristanjuergbi, then ssh builder@192.168.100.20610:26
juergbita10:27
tristanjuergbi, when you log into builder@gnome7, in the home dir there is a buildstream checkout10:27
tristanjuergbi, so to update it, you need to `sudo pip3 install .` in there, it needs to be a system wide installation for the artifacts user to see it10:28
tristanunless we play weird games of tweaking the artifacts user but screw that10:29
tristanjuergbi, well, regarding the other wondering thing... well...10:29
tristanprobably it cannot be done with the existing messages10:29
tristanas you might hit the same wall with message size10:30
juergbiah, right10:30
juergbithe size field in the header can likely be widened easy enough, though, if it's really just that10:30
juergbibut if the message-based approach doesn't work for this, we might need larger changes10:31
tristanAnd I guess you would have to have the client (after checking if the branch exists with the correct commit sha on the server), send the server all the object names it wants to push10:31
tristanand reply which ones are needed10:31
juergbiyes10:31
tristanfunny thing is this might also add some overhead, in the case of wanting to push lots and lots of small files :)10:32
juergbiindeed10:32
juergbionly check object existence for objects > N kB? hm, would get a bit more complicated10:32
juergbianyway, i think i'll first fix the bug and will consider this later10:33
tristanjuergbi, yeah I wouldnt lose sleep over the other thing10:33
tristanjuergbi, in fact...10:33
tristanjuergbi, you might look into artifact revisioning instead, given the time10:33
tristanthere are higher priority things than this optimization :)10:34
juergbiyes, i was also planning looking into the artifact revision issue this week10:34
tristangreat :)10:34
*** tlater has quit IRC10:55
tristanok so linking webkit has been slowing down my computer for 15 minutes :-/10:58
*** tlater has joined #buildstream10:59
*** locallycompact has joined #buildstream11:03
locallycompactoi oi11:09
tristanlocallycompact, good morning/afternoon :)11:11
gitlab-br-botpush on buildstream@variants-slow-functional-loopy (by Tristan Van Berkom): 14 commits (last: local.py source plugin: Fixed to support symlinks) https://gitlab.com/BuildStream/buildstream/commit/fea302e65836bf66edc0b780f6ddb8fe3ffd837811:11
tristanlocallycompact, so I have a tricky graph problem that I'm sure you can solve, I spent a day on it and had to give up for now11:12
tristanI've just been pushing up some things for reference11:12
tristanlocallycompact, so to start from the beginning, the problem revolves around resolving element variants11:13
tristanan element may have various variants, and an element may explicitly depend on a variant of another element, or it may remain ambivalent of the variant of it's dependency11:14
tristanSome documentation on how it's expected to work is here: https://buildstream.gitlab.io/buildstream/format.html#variant-conditionals11:14
tristanlocallycompact, what makes this a bit more tricky, is that an element may have additional dependencies, depending on it's variant11:15
tristanIf there is no preference of variant, the engine guarantees to choose the first declared variant (of any element which declares variants)11:15
tristanSo of course, the algorithm's goal here is to traverse the graph, where the input is the toplevel target element (and possibly a selected variant, if no variant specified, it's explicitly the first variant of "target")...11:18
tristanAnd chose a configuration for each element11:18
tristaneesh gitlab is being slow11:19
tristanlocallycompact, to get started, you will need some installation of buildstream (https://buildstream.gitlab.io/buildstream/install.html#installing)... it requires ostree (with introspection) and bubblewrap11:20
locallycompactinteresting problem11:20
tristanAnd I have this test data https://gitlab.com/BuildStream/gnome-test-data11:20
tristanthe flatpak-tests branch of that11:20
locallycompactalright11:21
tristanbackground on that; I just pushed that but it's from a continuous conversion process of JHBuild modulesets11:21
tristanthat branch contains some manually setup variants so that I can build a flatpak SDK from the same project11:21
tristanIn there, you can build flatpak with `bst build flatpak/gnome-runtime.bst` ... or build GNOME stuff with other targets, like `bst build core.bst` or such11:22
tristanah gitlab caught up with me11:22
tristanlocallycompact, this is the mess I currently have in place: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L55711:23
tristanwhich is a depth first traversal which takes a hash table of element/variant choices and trucks along11:23
tristanwhat is in place also has a "hack" I added couple days ago, which is just horrible11:24
locallycompactalright, this will take me some time to get my bearings11:24
tristanThe original problem was that, if you traverse the graph depth first and bail out on the first successful variant config, you fail in other branches11:25
locallycompactI've had some thoughts on context-sensitive graphs quite recently: https://gitlab.com/locallycompact/earthquake11:25
tristanlocallycompact, this algorithm however produces the correct result, I believe 100% of the time: https://gitlab.com/BuildStream/buildstream/blob/variants-slow-functional-loopy/buildstream/_loader.py#L55711:25
tristanThe second approach on that branch which "works", starts from the same premise, but instead of bailing out; it returns all non-conflicting configurations up through this recursive thing11:26
tristanand then at the end, the engine choses the first one (ensuring that left side variants are prioritized in the case of ambivalence)11:26
tristanHowever, it's exponentially complex, and takes minutes to resolve for small pipelines11:27
tristanlocallycompact, I've had some thoughts on this of course... one of them is that if we were to be able to pre-sort by dependency, we could use the original linear complexity equation without the risk of incorrectly finding errors11:28
tristanHowever the sorting approach is really difficult to imagine, because different variants can have different dependencies11:29
tristanAnother idea is to first flatten out a list of every possible variant of every element, do a breadth first traversal, and eliminate the impossibilities along the way; not even sure that makes sense11:30
tristanThe whole list would have to be copied for every traversal and a new list returned I suppose11:30
tristanI should note that an interesting property of this resolution thing, is that even in the case where nobody specified a variant of an element (all elements ambiguous), the non-default *can* be chosen11:33
tristanthis is okay, and it happens because that element depends on something differently depending on it's variant, and only it's second variant agrees with the rest of the graph11:34
tristanThis case is covered by `./setup.py test`, as most cases are11:34
tristanexcept for the case of variant disagreement deep in separate branches, which is where I started hitting problems :)11:35
tristanin the above ... (all elements s/ambiguous/ambivalent)... oops11:36
locallycompactWhat is the internal representation of the graph here11:38
tristanSo in the loader11:39
tristanWe have LoadElement11:39
tristanand the Loader itself has a hash of every loaded element in self.elements11:39
tristanfetch an element with self.elements[element_name]11:39
tristanAnd as the elements are not resolved at this point... the LoadElement has a list of dependencies as element.base_deps[]11:40
tristanAnd a list of variants as element.variants[]11:40
tristanlocallycompact, there is a LoadElementConfig() object which is more concrete, currently it's generated and discarded on the fly in the resolve algo11:41
tristanso LoadElementConfig is a potential element with a chosen variant, it will have a list of it's final Dependencies11:42
tristanThe dependencies are not just names but Dependency objects, which include the context of what variant they depend on (if any) and what type (runtime, build or both)11:43
tristanLoadElement() has .apply_element_config() which will do what needs to be done once a LoadElementConfig has been chosen for it11:44
locallycompactthis feels more like a SAT problem than a graph problem11:46
tristanright, well, it's a satisfiability problem which occurs in a graph so11:47
locallycompactso maybe we can transform it into something we can run a DPLL over11:53
locallycompactand then reconstruct the graph, I dno11:54
tristanYeah I'm really unsure, not familiar with DPLL (looking at a wikipedia page...)... I find it tempting to try to flatten out the data in such a way that we have a limited set that we can loop over and reason about, though11:56
tristanFor that, though; it seems logic that every element would require a full graph traversal in order to get context11:57
tristani.e. knowing that distant elements may have a say in which variant to chose11:57
tristanI also feel tempted to think that with the current approach, if we were to have a breadth first traversal, the solution might be closer at hand11:58
tristandescending from the target we make explicit choices going downward, the problem I guess with that is, we cannot really know what choice we can eliminate without descending through the whole graph11:59
tristanso, maybe it's nuts11:59
tristanAlso I know for a fact that, if the graph were sorted, we would not have this problem; but sorting the graph seems impossible12:01
tristannote that once we resolve everything, we sort once to avoid repeated t-sorts at runtime, but even that operation can be expensive without all the caching we've done12:02
locallycompactwhy is the sort expensive?12:03
tristanok actually maybe it's not that bad12:04
tristanright now it's instant12:04
tristanbut it was very expensive before we avoided redundant recursion12:05
* tristan double-checks the comparison12:05
locallycompactit's not really a graph prior to resolution though is it, it's a potential graph12:05
tristanAh12:05
tristanAnd we build a cache of what every element implicitly depends on12:06
tristanavoiding another loop12:06
tristanThe two protections render sorting efficient12:06
locallycompactI don't follow12:06
tristanIt's sorted in such a way that... if element A depends on element B, or anything that B depends on, then B comes before A12:08
tristanSo we create a cache of everything that B depends on recursively, and we build that once12:08
tristanNot for every comparison12:08
tristanIn a potential graph where dependencies can depend on chosen variant, the sort needs to be different for every potential traversal I guess12:09
tristanSo that is ... a lot12:09
locallycompactwhy does that make sense, if A depends on something that B depends on that doesn't necessarily imply that B does not depend on A, why should B some first12:09
locallycompact*come12:10
tristanSo that we dont need topological sorts by dependency later on12:10
tristanit's used for deterministic staging order and such like12:10
tristanlocallycompact, note that sort is a per element dependency sort12:11
tristanNot a huge flat list12:11
tristanthat sort is sorting the deps of a given element12:11
tristanit makes the Element.dependencies() generator function produce reliable results basically12:12
jonathanmawtristan: have you had an opportunity to push the correct configuration for base-sdk elements anywhere?12:18
*** tlater has quit IRC12:22
*** tlater has joined #buildstream12:30
tristanjonathanmaw, yes !12:33
tristanjonathanmaw, so I pushed this repo https://gitlab.com/BuildStream/gnome-test-data12:33
tristanjonathanmaw, flatpak-tests branch has it: https://gitlab.com/BuildStream/gnome-test-data/tree/flatpak-tests12:34
jonathanmawtvm tristan12:34
tristanjonathanmaw, https://gitlab.com/BuildStream/gnome-test-data/blob/flatpak-tests/elements/base.bst ... follow the 'gnome-flatpak-runtime' variant12:35
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 4 commits (last: dpkg-build.py: Add a build element for debian sources) https://gitlab.com/BuildStream/buildstream/commit/80c8bd737fc0e174983bd5caad7abe1461b90f0312:39
gitlab-br-botbuildstream: merge request (jonathan/dpkg-build->master: Jonathan/dpkg build) #37 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/3712:39
tristanjonathanmaw, so that is merged now, should be easier for adding integration tests to buildstream-tests :)12:39
jonathanmaw\o/12:40
tristanjonathanmaw, I didnt look too hard at it this time around as we've already been polishing it off for a while, but it's completely non-invasive to the core which I like12:40
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 3 commits (last: Dockerfile: Include bzr into the image) https://gitlab.com/BuildStream/buildstream/commit/a19b98e95aa0f4402206a417347a8430c375b47812:43
gitlab-br-botbuildstream: merge request (pedro/docker-bzr->master: Add support for bzr sources in Docker) #58 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/5812:43
*** tristan has quit IRC12:46
*** tlater has quit IRC12:53
*** tlater has joined #buildstream12:57
*** tlater has quit IRC13:05
*** xjuan has joined #buildstream13:12
*** tristan has joined #buildstream13:18
*** ChanServ sets mode: +o tristan13:18
*** tristan has quit IRC13:30
*** tlater has joined #buildstream13:54
*** jude has quit IRC14:36
*** tlater has quit IRC14:55
*** tlater has joined #buildstream15:05
tlaterI'm consistently getting a 'target is busy' when trying to unmount /dev/null after base-configure.15:07
tlaterIs there anything that could cause that?15:07
ssam2unmount /dev/null ? how did you mount it?15:08
tlatermount --bind /dev/null $SOMEPATH/dev/null15:08
ssam2on Linux ?15:08
tlater^Sandbox15:08
tlaterYup15:08
ssam2not sure, sorry15:08
*** jude has joined #buildstream15:17
tlaterHm, I guess I'll have to create new /dev/null etc devices. Hopefully I won't need 3 sets of commands for that as well.15:33
ssam2if this is linux, you can just set up a devtmpfs15:34
ssam2i.e. `mount -t devtmpfs none $SOMEPATH/dev`15:34
ssam2i think that's the "one true way" these days15:35
tlaterHow would that port to solaris though? I think `mknod` is a bit more portable15:36
tlaterUnless devtmpfs exists there... Let's try!15:36
tlaterYeah, unfortunately not :/15:38
ssam2does `mount --bind` work on Solaris either ?15:42
ssam2yes to do it portably you probably have to use `mknod`15:42
tlatermount -F lofs works instead15:42
ssam2ah right15:42
tlaterNot sure about AIX15:43
tlaterBut that's an issue for later15:43
ssam2well... not really, if you have to rewrite everything again to support AIX15:43
tlaterThere must be *some* way to mount a directory on AIX15:43
ssam2there's no harm in mknod really, it's just old fashioned15:43
ssam2and requires root15:43
tlaterDo you happen to know how to get solaris to mknod a /dev/null?15:44
tlatermknod null c 1 3 doesn't work :/15:44
ssam2what does `ls -l /dev/null` show in Solaris ?15:44
ssam2i don't know off hand but you should be able to copy from there15:44
tlaterNot sure how to use this...15:45
tlater/dev/null -> ../devices/pseudo/mm@0:null15:45
tlater15:45
ssam2well that's a symlink15:46
tlaterThe linked file doesn't exist15:46
tlaterNeither does its directory15:46
ssam2ls -l /devices/pseudo/mm\@0\:null15:46
ssam2crw-rw-rw-   1 root     sys      143,  2 Jul 19 15:38 /devices/pseudo/mm@0:null15:46
ssam2seems to ...15:46
tlaterAh, nevermind15:46
tlaterYeah15:46
tlaterThanks :)15:47
*** jude_ has joined #buildstream15:49
*** jude has quit IRC15:49
tlaterYou're right, ssam2, probably nothing like this for aix... That means I'll probably have to rethink some of this :/16:00
tlaterAnyway, that's work for tomorrow... Ciao o/16:05
*** tlater has quit IRC16:05
*** tiagogomes has quit IRC16:45
*** locallycompact has quit IRC16:49
ssam2what's the correct user/host/port for pushing artifacts to gnome7 ?16:53
*** tristan has joined #buildstream16:53
ssam2hmm I do see artifacts@gnome7.codethink.co.uk in IRC logs16:53
*** ChanServ sets mode: +o tristan16:53
*** locallycompact has joined #buildstream16:57
tristanssam2, that is correct, but you need a special port16:58
*** locallycompact has quit IRC16:58
tristanssam2, 1000716:58
ssam2aha16:59
ssam2jonathanmaw, ^^16:59
tristanand of course I assume you are authorized16:59
ssam2yeah, i just added jonathanmaw and me to the artifacts account16:59
ssam2it's working now on that port16:59
tristanthat said, pushing is temporarily broken until juergbi fixes https://gitlab.com/BuildStream/buildstream/issues/4716:59
ssam2well, it's working in that I get a "OverflowError: int too big to convert" error :-)17:00
tristanyeah :)17:00
juergbiplanning to have this fixed tomorrow17:00
ssam2jonathanmaw, if this blocks you i could give access to ostree.baserock.org17:00
tristanRight, this is for buildstream-tests17:04
tristanCan we work out a solution where buildstream-tests pulls only data from gitlab to run it's tests ?17:04
tristanMaybe we can have tarballs sitting in separate repos in the BuildStream group for this purpose ?17:05
jonathanmawtristan: hrm, if it's just the tarballs for the packages over the top of the gnome ostree, that might be half-way manageable.17:07
tristanjonathanmaw, that's not what I mean17:09
tristanjonathanmaw, I mean... can we have a tarball that has everything that buildstream-tests needs to run the simplest possible tests for dpkg elements, minus the shit ton of stuff in gnome7 that is certainly unneeded for the smallest possible isolated test17:10
tristanjonathanmaw, so that when the day comes that gnome7 moves somewhere else, or is offline, or whatever it's a completely separate project that we dont depend on it... *out* tests keep working17:10
tristani.e., can we keep it completely unrelated to both GNOME and Baserock please ?17:10
jonathanmawtristan: ok, so a minimal system that provides debhelper, quilt, and build-essential (probably built in debootstrap-ostree), tarred up and stuffed into gitlab.17:11
tristanWe should be able to handle our own integration tests by making sure the data is there, at urls we provide, and ideally on the same service (so for instance; if we migrate the *whole* buildstream group to gitlab.gnome.org, it should be fairly simple to keep it all working)17:11
*** ssam2 has quit IRC17:14
tristanwe could probably theoretically do something weird like: Tar up an ostree repo created with debootstrap-ostree, and commit that tarball to a git repo in the buildstream group, and then have the tests check that out; untar the ostree repo, and use the ostree repo locally in the test; possibly with some sed work on the elements which the tests project uses17:15
tristana bit contorted though17:15
tristanprobably a tarball of the ostree checkout committed into git will be simpler, and just use a tar source17:16
*** jude_ has quit IRC17:28
*** jonathanmaw has quit IRC17:48
*** tristan has quit IRC20:48
*** xjuan has quit IRC21:11

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!