*** jude has joined #buildstream | 07:00 | |
gitlab-br-bot | buildstream: issue #48 ("Incorrect pipeline total element count for `bst track`") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/48 | 07:32 |
---|---|---|
*** tlater has joined #buildstream | 08:31 | |
gitlab-br-bot | buildstream: issue #49 ("BuildStream needs artifact revisions") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/49 | 08:39 |
gitlab-br-bot | buildstream: issue #46 ("Strange artifact pull failures on GitLab CI") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/46 | 08:40 |
*** tristan has joined #buildstream | 08:50 | |
*** jonathanmaw has joined #buildstream | 08:53 | |
*** ChanServ sets mode: +o tristan | 08:55 | |
*** ssam2 has joined #buildstream | 08:59 | |
gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: _frontend/main.py: Naming workspace functions with workspace prefix) https://gitlab.com/BuildStream/buildstream/commit/67f9ea78575ab94f99dedb2a1875fdd386e14bd5 | 09:00 |
jonathanmaw | in the buildstream-tests repo, what's the difference between the base-platform elements and the base-sdk elements? | 09:16 |
tristan | jonathanmaw, it's incorrect | 09:16 |
tristan | jonathanmaw, I noted that to sam the other day, I can show you the correct configuration | 09:16 |
tristan | but I have to push it somewhere :-/ | 09:17 |
tristan | jonathanmaw, basically I misunderstood what it was back then, we only need to stage the sdk, the platform is a subset of the sdk already | 09:17 |
jonathanmaw | tristan: ok | 09:17 |
tristan | and we need to stage it in /usr, _AND_ we need to stage a local import element that only does /usr merge at slash | 09:17 |
tristan | so in my branch to build flatpaks with GNOME modulesets, I'm using something like that, just havent pushed it anywhere relevant yet | 09:18 |
tristan | So... my machine is in the middle of a webkit build which is almost at the end of the flatpak runtime build; yelp remains to be built on top of it but that's been known to succeed before | 09:36 |
tristan | So anyway everything is building, with the flatpak variants of the GNOME modulesets project in place | 09:37 |
tristan | That is one big part of the milestone done, but there remains some tinkering before it can really be used as a flatpak runtime | 09:38 |
tristan | From what I anticipate needs doing; A.) Need to define flatpak specific split-rules, probably in the project variant, with some tweaks in some elements to match the "cleanup-files" and such | 09:39 |
tristan | e.g. https://git.gnome.org/browse/gnome-sdk-images/tree/org.gnome.Sdk.json.in#n374 | 09:39 |
tristan | That json says that "These files need to go into the SDK but not the Platform" | 09:39 |
tristan | B.) Need to figure out how to split the locales for the separate Locale extension (still revolves around composing split outputs) | 09:40 |
tristan | C.) Need to add a compose element to split these | 09:40 |
tristan | D.) Need an element which basically just spits out some metadata for the outputs | 09:41 |
tristan | So, that more-or-less covers it | 09:42 |
tristan | there are some other details, but maybe the above can be brute forced enough to make a demo of running a real flatpak against a buildstream-built GNOME SDK | 09:42 |
tristan | juergbi, good call on avoiding the push/pull queues if summary fetch fails | 09:45 |
tristan | juergbi, anyway... I'm going to need you to fix https://gitlab.com/BuildStream/buildstream/issues/47 | 09:46 |
tristan | that's pretty crucial | 09:46 |
tristan | right now every push is failing on gnome7 because of that | 09:47 |
juergbi | yes, i was planning to do pretty much what you described (but also return checksums for the refs) | 09:47 |
juergbi | once my summary branch is completely finished | 09:47 |
tristan | juergbi, I'm giving you shell access hold on... | 09:48 |
tristan | well, I'll let you know... :) | 09:48 |
juergbi | ta | 09:49 |
juergbi | tristan: wondering whether i should implement the other push optimization (only send objects as needed) if i anyway have to change the protocol. don't know yet how much effort that will be, though | 09:49 |
tristan | juergbi, shell into gnome7 following these steps | 10:25 |
tristan | first ssh -A -p 22100 juergbi@jumpserv.colo.codethink.co.uk | 10:25 |
tristan | juergbi, that will get you onto jumpserv, which can login to the 8 GNOME arm servers | 10:26 |
tristan | juergbi, then ssh builder@192.168.100.206 | 10:26 |
juergbi | ta | 10:27 |
tristan | juergbi, when you log into builder@gnome7, in the home dir there is a buildstream checkout | 10:27 |
tristan | juergbi, so to update it, you need to `sudo pip3 install .` in there, it needs to be a system wide installation for the artifacts user to see it | 10:28 |
tristan | unless we play weird games of tweaking the artifacts user but screw that | 10:29 |
tristan | juergbi, well, regarding the other wondering thing... well... | 10:29 |
tristan | probably it cannot be done with the existing messages | 10:29 |
tristan | as you might hit the same wall with message size | 10:30 |
juergbi | ah, right | 10:30 |
juergbi | the size field in the header can likely be widened easy enough, though, if it's really just that | 10:30 |
juergbi | but if the message-based approach doesn't work for this, we might need larger changes | 10:31 |
tristan | And I guess you would have to have the client (after checking if the branch exists with the correct commit sha on the server), send the server all the object names it wants to push | 10:31 |
tristan | and reply which ones are needed | 10:31 |
juergbi | yes | 10:31 |
tristan | funny thing is this might also add some overhead, in the case of wanting to push lots and lots of small files :) | 10:32 |
juergbi | indeed | 10:32 |
juergbi | only check object existence for objects > N kB? hm, would get a bit more complicated | 10:32 |
juergbi | anyway, i think i'll first fix the bug and will consider this later | 10:33 |
tristan | juergbi, yeah I wouldnt lose sleep over the other thing | 10:33 |
tristan | juergbi, in fact... | 10:33 |
tristan | juergbi, you might look into artifact revisioning instead, given the time | 10:33 |
tristan | there are higher priority things than this optimization :) | 10:34 |
juergbi | yes, i was also planning looking into the artifact revision issue this week | 10:34 |
tristan | great :) | 10:34 |
*** tlater has quit IRC | 10:55 | |
tristan | ok so linking webkit has been slowing down my computer for 15 minutes :-/ | 10:58 |
*** tlater has joined #buildstream | 10:59 | |
*** locallycompact has joined #buildstream | 11:03 | |
locallycompact | oi oi | 11:09 |
tristan | locallycompact, good morning/afternoon :) | 11:11 |
gitlab-br-bot | push on buildstream@variants-slow-functional-loopy (by Tristan Van Berkom): 14 commits (last: local.py source plugin: Fixed to support symlinks) https://gitlab.com/BuildStream/buildstream/commit/fea302e65836bf66edc0b780f6ddb8fe3ffd8378 | 11:11 |
tristan | locallycompact, so I have a tricky graph problem that I'm sure you can solve, I spent a day on it and had to give up for now | 11:12 |
tristan | I've just been pushing up some things for reference | 11:12 |
tristan | locallycompact, so to start from the beginning, the problem revolves around resolving element variants | 11:13 |
tristan | an element may have various variants, and an element may explicitly depend on a variant of another element, or it may remain ambivalent of the variant of it's dependency | 11:14 |
tristan | Some documentation on how it's expected to work is here: https://buildstream.gitlab.io/buildstream/format.html#variant-conditionals | 11:14 |
tristan | locallycompact, what makes this a bit more tricky, is that an element may have additional dependencies, depending on it's variant | 11:15 |
tristan | If there is no preference of variant, the engine guarantees to choose the first declared variant (of any element which declares variants) | 11:15 |
tristan | So of course, the algorithm's goal here is to traverse the graph, where the input is the toplevel target element (and possibly a selected variant, if no variant specified, it's explicitly the first variant of "target")... | 11:18 |
tristan | And chose a configuration for each element | 11:18 |
tristan | eesh gitlab is being slow | 11:19 |
tristan | locallycompact, to get started, you will need some installation of buildstream (https://buildstream.gitlab.io/buildstream/install.html#installing)... it requires ostree (with introspection) and bubblewrap | 11:20 |
locallycompact | interesting problem | 11:20 |
tristan | And I have this test data https://gitlab.com/BuildStream/gnome-test-data | 11:20 |
tristan | the flatpak-tests branch of that | 11:20 |
locallycompact | alright | 11:21 |
tristan | background on that; I just pushed that but it's from a continuous conversion process of JHBuild modulesets | 11:21 |
tristan | that branch contains some manually setup variants so that I can build a flatpak SDK from the same project | 11:21 |
tristan | In there, you can build flatpak with `bst build flatpak/gnome-runtime.bst` ... or build GNOME stuff with other targets, like `bst build core.bst` or such | 11:22 |
tristan | ah gitlab caught up with me | 11:22 |
tristan | locallycompact, this is the mess I currently have in place: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L557 | 11:23 |
tristan | which is a depth first traversal which takes a hash table of element/variant choices and trucks along | 11:23 |
tristan | what is in place also has a "hack" I added couple days ago, which is just horrible | 11:24 |
locallycompact | alright, this will take me some time to get my bearings | 11:24 |
tristan | The original problem was that, if you traverse the graph depth first and bail out on the first successful variant config, you fail in other branches | 11:25 |
locallycompact | I've had some thoughts on context-sensitive graphs quite recently: https://gitlab.com/locallycompact/earthquake | 11:25 |
tristan | locallycompact, this algorithm however produces the correct result, I believe 100% of the time: https://gitlab.com/BuildStream/buildstream/blob/variants-slow-functional-loopy/buildstream/_loader.py#L557 | 11:25 |
tristan | The second approach on that branch which "works", starts from the same premise, but instead of bailing out; it returns all non-conflicting configurations up through this recursive thing | 11:26 |
tristan | and then at the end, the engine choses the first one (ensuring that left side variants are prioritized in the case of ambivalence) | 11:26 |
tristan | However, it's exponentially complex, and takes minutes to resolve for small pipelines | 11:27 |
tristan | locallycompact, I've had some thoughts on this of course... one of them is that if we were to be able to pre-sort by dependency, we could use the original linear complexity equation without the risk of incorrectly finding errors | 11:28 |
tristan | However the sorting approach is really difficult to imagine, because different variants can have different dependencies | 11:29 |
tristan | Another idea is to first flatten out a list of every possible variant of every element, do a breadth first traversal, and eliminate the impossibilities along the way; not even sure that makes sense | 11:30 |
tristan | The whole list would have to be copied for every traversal and a new list returned I suppose | 11:30 |
tristan | I should note that an interesting property of this resolution thing, is that even in the case where nobody specified a variant of an element (all elements ambiguous), the non-default *can* be chosen | 11:33 |
tristan | this is okay, and it happens because that element depends on something differently depending on it's variant, and only it's second variant agrees with the rest of the graph | 11:34 |
tristan | This case is covered by `./setup.py test`, as most cases are | 11:34 |
tristan | except for the case of variant disagreement deep in separate branches, which is where I started hitting problems :) | 11:35 |
tristan | in the above ... (all elements s/ambiguous/ambivalent)... oops | 11:36 |
locallycompact | What is the internal representation of the graph here | 11:38 |
tristan | So in the loader | 11:39 |
tristan | We have LoadElement | 11:39 |
tristan | and the Loader itself has a hash of every loaded element in self.elements | 11:39 |
tristan | fetch an element with self.elements[element_name] | 11:39 |
tristan | And as the elements are not resolved at this point... the LoadElement has a list of dependencies as element.base_deps[] | 11:40 |
tristan | And a list of variants as element.variants[] | 11:40 |
tristan | locallycompact, there is a LoadElementConfig() object which is more concrete, currently it's generated and discarded on the fly in the resolve algo | 11:41 |
tristan | so LoadElementConfig is a potential element with a chosen variant, it will have a list of it's final Dependencies | 11:42 |
tristan | The dependencies are not just names but Dependency objects, which include the context of what variant they depend on (if any) and what type (runtime, build or both) | 11:43 |
tristan | LoadElement() has .apply_element_config() which will do what needs to be done once a LoadElementConfig has been chosen for it | 11:44 |
locallycompact | this feels more like a SAT problem than a graph problem | 11:46 |
tristan | right, well, it's a satisfiability problem which occurs in a graph so | 11:47 |
locallycompact | so maybe we can transform it into something we can run a DPLL over | 11:53 |
locallycompact | and then reconstruct the graph, I dno | 11:54 |
tristan | Yeah I'm really unsure, not familiar with DPLL (looking at a wikipedia page...)... I find it tempting to try to flatten out the data in such a way that we have a limited set that we can loop over and reason about, though | 11:56 |
tristan | For that, though; it seems logic that every element would require a full graph traversal in order to get context | 11:57 |
tristan | i.e. knowing that distant elements may have a say in which variant to chose | 11:57 |
tristan | I also feel tempted to think that with the current approach, if we were to have a breadth first traversal, the solution might be closer at hand | 11:58 |
tristan | descending from the target we make explicit choices going downward, the problem I guess with that is, we cannot really know what choice we can eliminate without descending through the whole graph | 11:59 |
tristan | so, maybe it's nuts | 11:59 |
tristan | Also I know for a fact that, if the graph were sorted, we would not have this problem; but sorting the graph seems impossible | 12:01 |
tristan | note that once we resolve everything, we sort once to avoid repeated t-sorts at runtime, but even that operation can be expensive without all the caching we've done | 12:02 |
locallycompact | why is the sort expensive? | 12:03 |
tristan | ok actually maybe it's not that bad | 12:04 |
tristan | right now it's instant | 12:04 |
tristan | but it was very expensive before we avoided redundant recursion | 12:05 |
* tristan double-checks the comparison | 12:05 | |
locallycompact | it's not really a graph prior to resolution though is it, it's a potential graph | 12:05 |
tristan | Ah | 12:05 |
tristan | And we build a cache of what every element implicitly depends on | 12:06 |
tristan | avoiding another loop | 12:06 |
tristan | The two protections render sorting efficient | 12:06 |
locallycompact | I don't follow | 12:06 |
tristan | It's sorted in such a way that... if element A depends on element B, or anything that B depends on, then B comes before A | 12:08 |
tristan | So we create a cache of everything that B depends on recursively, and we build that once | 12:08 |
tristan | Not for every comparison | 12:08 |
tristan | In a potential graph where dependencies can depend on chosen variant, the sort needs to be different for every potential traversal I guess | 12:09 |
tristan | So that is ... a lot | 12:09 |
locallycompact | why does that make sense, if A depends on something that B depends on that doesn't necessarily imply that B does not depend on A, why should B some first | 12:09 |
locallycompact | *come | 12:10 |
tristan | So that we dont need topological sorts by dependency later on | 12:10 |
tristan | it's used for deterministic staging order and such like | 12:10 |
tristan | locallycompact, note that sort is a per element dependency sort | 12:11 |
tristan | Not a huge flat list | 12:11 |
tristan | that sort is sorting the deps of a given element | 12:11 |
tristan | it makes the Element.dependencies() generator function produce reliable results basically | 12:12 |
jonathanmaw | tristan: have you had an opportunity to push the correct configuration for base-sdk elements anywhere? | 12:18 |
*** tlater has quit IRC | 12:22 | |
*** tlater has joined #buildstream | 12:30 | |
tristan | jonathanmaw, yes ! | 12:33 |
tristan | jonathanmaw, so I pushed this repo https://gitlab.com/BuildStream/gnome-test-data | 12:33 |
tristan | jonathanmaw, flatpak-tests branch has it: https://gitlab.com/BuildStream/gnome-test-data/tree/flatpak-tests | 12:34 |
jonathanmaw | tvm tristan | 12:34 |
tristan | jonathanmaw, https://gitlab.com/BuildStream/gnome-test-data/blob/flatpak-tests/elements/base.bst ... follow the 'gnome-flatpak-runtime' variant | 12:35 |
gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 4 commits (last: dpkg-build.py: Add a build element for debian sources) https://gitlab.com/BuildStream/buildstream/commit/80c8bd737fc0e174983bd5caad7abe1461b90f03 | 12:39 |
gitlab-br-bot | buildstream: merge request (jonathan/dpkg-build->master: Jonathan/dpkg build) #37 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/37 | 12:39 |
tristan | jonathanmaw, so that is merged now, should be easier for adding integration tests to buildstream-tests :) | 12:39 |
jonathanmaw | \o/ | 12:40 |
tristan | jonathanmaw, I didnt look too hard at it this time around as we've already been polishing it off for a while, but it's completely non-invasive to the core which I like | 12:40 |
gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: Dockerfile: Include bzr into the image) https://gitlab.com/BuildStream/buildstream/commit/a19b98e95aa0f4402206a417347a8430c375b478 | 12:43 |
gitlab-br-bot | buildstream: merge request (pedro/docker-bzr->master: Add support for bzr sources in Docker) #58 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/58 | 12:43 |
*** tristan has quit IRC | 12:46 | |
*** tlater has quit IRC | 12:53 | |
*** tlater has joined #buildstream | 12:57 | |
*** tlater has quit IRC | 13:05 | |
*** xjuan has joined #buildstream | 13:12 | |
*** tristan has joined #buildstream | 13:18 | |
*** ChanServ sets mode: +o tristan | 13:18 | |
*** tristan has quit IRC | 13:30 | |
*** tlater has joined #buildstream | 13:54 | |
*** jude has quit IRC | 14:36 | |
*** tlater has quit IRC | 14:55 | |
*** tlater has joined #buildstream | 15:05 | |
tlater | I'm consistently getting a 'target is busy' when trying to unmount /dev/null after base-configure. | 15:07 |
tlater | Is there anything that could cause that? | 15:07 |
ssam2 | unmount /dev/null ? how did you mount it? | 15:08 |
tlater | mount --bind /dev/null $SOMEPATH/dev/null | 15:08 |
ssam2 | on Linux ? | 15:08 |
tlater | ^Sandbox | 15:08 |
tlater | Yup | 15:08 |
ssam2 | not sure, sorry | 15:08 |
*** jude has joined #buildstream | 15:17 | |
tlater | Hm, I guess I'll have to create new /dev/null etc devices. Hopefully I won't need 3 sets of commands for that as well. | 15:33 |
ssam2 | if this is linux, you can just set up a devtmpfs | 15:34 |
ssam2 | i.e. `mount -t devtmpfs none $SOMEPATH/dev` | 15:34 |
ssam2 | i think that's the "one true way" these days | 15:35 |
tlater | How would that port to solaris though? I think `mknod` is a bit more portable | 15:36 |
tlater | Unless devtmpfs exists there... Let's try! | 15:36 |
tlater | Yeah, unfortunately not :/ | 15:38 |
ssam2 | does `mount --bind` work on Solaris either ? | 15:42 |
ssam2 | yes to do it portably you probably have to use `mknod` | 15:42 |
tlater | mount -F lofs works instead | 15:42 |
ssam2 | ah right | 15:42 |
tlater | Not sure about AIX | 15:43 |
tlater | But that's an issue for later | 15:43 |
ssam2 | well... not really, if you have to rewrite everything again to support AIX | 15:43 |
tlater | There must be *some* way to mount a directory on AIX | 15:43 |
ssam2 | there's no harm in mknod really, it's just old fashioned | 15:43 |
ssam2 | and requires root | 15:43 |
tlater | Do you happen to know how to get solaris to mknod a /dev/null? | 15:44 |
tlater | mknod null c 1 3 doesn't work :/ | 15:44 |
ssam2 | what does `ls -l /dev/null` show in Solaris ? | 15:44 |
ssam2 | i don't know off hand but you should be able to copy from there | 15:44 |
tlater | Not sure how to use this... | 15:45 |
tlater | /dev/null -> ../devices/pseudo/mm@0:null | 15:45 |
tlater | 15:45 | |
ssam2 | well that's a symlink | 15:46 |
tlater | The linked file doesn't exist | 15:46 |
tlater | Neither does its directory | 15:46 |
ssam2 | ls -l /devices/pseudo/mm\@0\:null | 15:46 |
ssam2 | crw-rw-rw- 1 root sys 143, 2 Jul 19 15:38 /devices/pseudo/mm@0:null | 15:46 |
ssam2 | seems to ... | 15:46 |
tlater | Ah, nevermind | 15:46 |
tlater | Yeah | 15:46 |
tlater | Thanks :) | 15:47 |
*** jude_ has joined #buildstream | 15:49 | |
*** jude has quit IRC | 15:49 | |
tlater | You're right, ssam2, probably nothing like this for aix... That means I'll probably have to rethink some of this :/ | 16:00 |
tlater | Anyway, that's work for tomorrow... Ciao o/ | 16:05 |
*** tlater has quit IRC | 16:05 | |
*** tiagogomes has quit IRC | 16:45 | |
*** locallycompact has quit IRC | 16:49 | |
ssam2 | what's the correct user/host/port for pushing artifacts to gnome7 ? | 16:53 |
*** tristan has joined #buildstream | 16:53 | |
ssam2 | hmm I do see artifacts@gnome7.codethink.co.uk in IRC logs | 16:53 |
*** ChanServ sets mode: +o tristan | 16:53 | |
*** locallycompact has joined #buildstream | 16:57 | |
tristan | ssam2, that is correct, but you need a special port | 16:58 |
*** locallycompact has quit IRC | 16:58 | |
tristan | ssam2, 10007 | 16:58 |
ssam2 | aha | 16:59 |
ssam2 | jonathanmaw, ^^ | 16:59 |
tristan | and of course I assume you are authorized | 16:59 |
ssam2 | yeah, i just added jonathanmaw and me to the artifacts account | 16:59 |
ssam2 | it's working now on that port | 16:59 |
tristan | that said, pushing is temporarily broken until juergbi fixes https://gitlab.com/BuildStream/buildstream/issues/47 | 16:59 |
ssam2 | well, it's working in that I get a "OverflowError: int too big to convert" error :-) | 17:00 |
tristan | yeah :) | 17:00 |
juergbi | planning to have this fixed tomorrow | 17:00 |
ssam2 | jonathanmaw, if this blocks you i could give access to ostree.baserock.org | 17:00 |
tristan | Right, this is for buildstream-tests | 17:04 |
tristan | Can we work out a solution where buildstream-tests pulls only data from gitlab to run it's tests ? | 17:04 |
tristan | Maybe we can have tarballs sitting in separate repos in the BuildStream group for this purpose ? | 17:05 |
jonathanmaw | tristan: hrm, if it's just the tarballs for the packages over the top of the gnome ostree, that might be half-way manageable. | 17:07 |
tristan | jonathanmaw, that's not what I mean | 17:09 |
tristan | jonathanmaw, I mean... can we have a tarball that has everything that buildstream-tests needs to run the simplest possible tests for dpkg elements, minus the shit ton of stuff in gnome7 that is certainly unneeded for the smallest possible isolated test | 17:10 |
tristan | jonathanmaw, so that when the day comes that gnome7 moves somewhere else, or is offline, or whatever it's a completely separate project that we dont depend on it... *out* tests keep working | 17:10 |
tristan | i.e., can we keep it completely unrelated to both GNOME and Baserock please ? | 17:10 |
jonathanmaw | tristan: ok, so a minimal system that provides debhelper, quilt, and build-essential (probably built in debootstrap-ostree), tarred up and stuffed into gitlab. | 17:11 |
tristan | We should be able to handle our own integration tests by making sure the data is there, at urls we provide, and ideally on the same service (so for instance; if we migrate the *whole* buildstream group to gitlab.gnome.org, it should be fairly simple to keep it all working) | 17:11 |
*** ssam2 has quit IRC | 17:14 | |
tristan | we could probably theoretically do something weird like: Tar up an ostree repo created with debootstrap-ostree, and commit that tarball to a git repo in the buildstream group, and then have the tests check that out; untar the ostree repo, and use the ostree repo locally in the test; possibly with some sed work on the elements which the tests project uses | 17:15 |
tristan | a bit contorted though | 17:15 |
tristan | probably a tarball of the ostree checkout committed into git will be simpler, and just use a tar source | 17:16 |
*** jude_ has quit IRC | 17:28 | |
*** jonathanmaw has quit IRC | 17:48 | |
*** tristan has quit IRC | 20:48 | |
*** xjuan has quit IRC | 21:11 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!