| *** tristan has joined #buildstream | 06:04 | |
| *** ChanServ sets mode: +o tristan | 06:25 | |
| *** jude has joined #buildstream | 07:02 | |
| *** tristan has quit IRC | 07:31 | |
| *** tristan has joined #buildstream | 07:45 | |
| *** jonathanmaw has joined #buildstream | 08:21 | |
| *** ChanServ sets mode: +o tristan | 08:52 | |
| *** ssam2 has joined #buildstream | 09:12 | |
| *** tiagogomes has joined #buildstream | 09:20 | |
| tristan | ... moving here :) | 10:09 |
|---|---|---|
| tristan | Ok, so we we're discussing mostly BuildStream vocabulary | 10:10 |
| persia | tristan: So, yeah taking this out of /query: I'm struggling to understand how "Element", "Source", "Pipeline", etc. map to more traditional concepts of software management. | 10:10 |
| tristan | Right so we should probably work on a list and some definitions | 10:10 |
| tristan | And it *might* make sense to have a section on this in https://buildstream.gitlab.io/buildstream/ | 10:11 |
| tristan | not sure about that, if the docs read well then it should not really be needed | 10:11 |
| tristan | So let me throw some words out first | 10:11 |
| persia | Having something there would certainly make it more accessible. For terms I know (e.g. "series", "package", "component", "distribution", etc.) from other places, it would have been easier to lean with such a page (and one may exist now) | 10:12 |
| tristan | o Project: A directory with a project.conf and some buildstream "Element" declarations (bst files) | 10:12 |
| tristan | o Pipeline: A resolved dependency graph of Elements, one Project may have many possibilities of Pipelines | 10:13 |
| tristan | o Element: An entity which performs some mutation of filesystem data, not all of these are associated with building anything, but in a project, usually a majority of elements build something | 10:14 |
| tristan | o Source: An entity which brings in external data into the Sandbox on behalf of an Element. One Element may own 0 or more Sources | 10:15 |
| tristan | So that is the toplevel of BuildStream entities really, but as we had noticed, there is room for misunderstanding | 10:16 |
| tristan | So, there is a difference between a 'Source' and a 'Source plugin' I guess | 10:16 |
| tristan | Where a Source plugin introduces a "kind" of Source (a type) | 10:16 |
| tristan | which knows how to deal with a specific kind of backend | 10:17 |
| persia | Or, one could say that a "Source" has a type, depending on where/how the data resides. | 10:17 |
| tristan | like git, svn, tar, etc | 10:17 |
| persia | Right. | 10:17 |
| tristan | And we do call it a "kind", for both Elements and Sources | 10:17 |
| persia | And I think there is also the concept of "Repo", which is a collection of Sources with the same name, and same kind (and same location), ideally organised in some sort of history. | 10:18 |
| tristan | So to take this example of an "autotools" kind of Element which uses a "git" kind of Source: https://gitlab.com/BuildStream/buildstream-tests/blob/build-gnome/elements/core/attr.bst | 10:18 |
| persia | "kind" in preference to "type" works for me. | 10:18 |
| tristan | There is no concept of a "Repo" really, that is all managed by the Source itself | 10:19 |
| tristan | persia, see this bug: https://gitlab.com/BuildStream/buildstream/issues/5 | 10:19 |
| tristan | That is a corner case consequence of not having a "Repo" type of concept | 10:19 |
| tristan | But I think that we can solve this particular thing without introducing a new layer | 10:20 |
| tristan | perhaps by using some locking mechanism, written once in the core | 10:20 |
| persia | Yes. I think there is a human concept "Repo", that BuildStream does not model, as everything can sensibly be done with "Source" and "Source kind" | 10:20 |
| tristan | Right so what is that exactly ? | 10:21 |
| tristan | I think it's basically, the 'url' attribute of a Source ? | 10:21 |
| persia | Yes. | 10:21 |
| persia | Or at least, that is how we identify it. | 10:21 |
| persia | In other models of thinking about software management, it is an important concept, but I'm now convinced that it isn't important here. | 10:22 |
| tristan | Right, so to dig into Sources a bit further, Sources can theoretically use different vocabulary, but for Source implementations which live in BuildStream proper, we want to use the same vocab | 10:22 |
| persia | (aside from the odd corner case where one has two Sources from the same Repo, but that should not really happen except in the linux/linux-libc situation) | 10:22 |
| tristan | it's quite rare yes | 10:23 |
| tristan | there are submodules which are a more common occurence | 10:23 |
| tristan | *occurrence | 10:23 |
| tristan | but in any case, this doesnt really get in the way | 10:23 |
| persia | Oh, ugh, right, I try to pretend those don't exist :) | 10:24 |
| persia | Anyway, distraction. | 10:24 |
| tristan | Right so, Source plugin vocabulary, some Source plugins may have additional options (we've yet to learn this, I think the dpkg source plugin has some additional options) | 10:24 |
| persia | So, when one produces an artifact, one does that by processing of an "Element", within a "Pipeline", where the "Pipeline" may only have 1 "Element", or may have many, depending on the dependency graph? | 10:25 |
| tristan | But we basically have: url, ref and track | 10:25 |
| tristan | Yes. | 10:25 |
| tristan | The artifact is the output of an Element | 10:25 |
| tristan | However notably, "stack" elements do not produce output (they produce empty artifacts to be more precise) | 10:26 |
| persia | You should be able to do a debian-archive pull with that. URL would be a host/path, Ref is likely a specific revision, and Track is probably following some series (e.g. jessie). | 10:26 |
| persia | "stack" elements? | 10:27 |
| tristan | So first line: for dpkg source I think we want/need a list of unversioned "package names" in order to control what exactly you want to mirror/stage | 10:27 |
| persia | And, to confirm, an "Element" cannot be executed outside of the context of a "Pipeline", although "Pipelines" can be dynamically constructed based on a request to generate an artifact for a specific "Element"? | 10:28 |
| tristan | Yes. | 10:29 |
| persia | Debian-Archive: Yes, mirror/package identifies the specific thing to get, and there exists such a list of all packages for each series on a mirror. | 10:29 |
| tristan | lets stay off the debian stuff, it's clouding things :) | 10:30 |
| persia | Yes :) | 10:30 |
| * albfan[m] sent a long message: albfan[m]_2017-04-26_10:30:44.txt - https://matrix.org/_matrix/media/v1/download/matrix.org/fuzIfXKcRpyFwnesSRXsvoaP | 10:30 | |
| tristan | I just wanted to say that, there can be justifications that a Source add some extra parameter, for dpkg, without that extra parameter, the only possible expression would be "Please stage every single package of the entire distro here" | 10:30 |
| persia | I certainly hope not, but that is a separate discussion. | 10:32 |
| tristan | albfan[m], I would really like to get to the point of automatically converting the JSON and testing a real flatpak deployment | 10:32 |
| tristan | albfan[m], also, I dont know what matrix UI is like, but you might try sending one liner messages | 10:32 |
| tristan | it looks like your IRC bridge eats them up and then pastes them somewhere :) | 10:32 |
| tristan | persia, Ok so... | 10:33 |
| albfan[m] | tristan: opps! | 10:33 |
| tristan | o ref: This is some Source specific identifier for an exact revision, in the case of git, it's a commit sha | 10:33 |
| tristan | o url: This is the location to obtain the data | 10:33 |
| tristan | o track: This is the symbolic name to update the "ref" with | 10:34 |
| tristan | That is the common vocab for parameters of Sources, which we try to keep the same for all Source implementations in BuildStream | 10:34 |
| albfan[m] | tristan: we can start from simple. Doing a buildstream recipe based on that json manifest from flatpak. With a complete example we can think how to automatically convert from flatpak to buildstream | 10:35 |
| tristan | albfan[m], that would be great if you would like to try it actually ! | 10:35 |
| tristan | albfan[m], so there will be *some* obstacles to get it published into a flatpak that you can actually install | 10:36 |
| *** albfan has joined #buildstream | 10:36 | |
| * albfan takes a look in plain IRC | 10:37 | |
| tristan | :) | 10:37 |
| tristan | so just in case | 10:37 |
| tristan | <tristan> albfan[m], that would be great if you would like to try it actually ! | 10:37 |
| tristan | <tristan> albfan[m], so there will be *some* obstacles to get it published into a flatpak that you can actually install | 10:37 |
| tristan | albfan, so note that https://gitlab.com/BuildStream/buildstream-tests/tree/master master branch currently has an example project which builds a few things | 10:38 |
| tristan | which were manually ported over from flatpak json | 10:38 |
| tristan | Notice gnome-platform.bst and gnome-sdk.bst, these are "import" elements for the actual GNOME flatpak sdk & runtime | 10:39 |
| tristan | that forms the base for building a flatpak for anything which depends on GNOME Runtime/SDK | 10:39 |
| tristan | right now those point to 3.22 | 10:39 |
| tristan | albfan, so I was able to get as far as building them and running them in a shell | 10:40 |
| tristan | However, distributing them to run in a flatpak is a part of the puzzle I still need to try | 10:40 |
| * albfan is taking a look to gedit.bst (again) | 10:41 | |
| albfan | yaml? love it | 10:41 |
| tristan | yep, all yaml :) | 10:41 |
| albfan | https://gitlab.com/BuildStream/buildstream-tests/blob/master/gedit.bst#L10 for deps, right? | 10:41 |
| tristan | persia, that reminds me, even the 'git' source needs some additional parameters, for handling of submodules :) | 10:42 |
| albfan | https://gitlab.com/BuildStream/buildstream-tests/blob/master/gnome-sdk.bst#L17 flatpak runtime is already there, right? | 10:42 |
| albfan | easy as pie | 10:42 |
| tristan | albfan, right so, there is a bit of a problem with the way I've listed the dependencies, as that was just an initial "try" | 10:42 |
| tristan | albfan, technically, the gnome-sdk.bst and gnome-platform.bst should be "build" dependencies | 10:43 |
| albfan | I think some python parser can convert a flatpak.json into yaml bst easily, is that ok (that would be another phase) | 10:43 |
| * paulsher1ood also loves yaml, and hopes it continues to take over the world | 10:44 | |
| tristan | yes, I've looked at all of the possibilities, and we are close to merging the 'bzr' source which is supported by Flatpak, so we should have support for everything | 10:44 |
| albfan | Yes I see them here https://gitlab.com/BuildStream/buildstream-tests/blob/master/gedit.bst#L12 | 10:44 |
| tristan | fwiw, this is the docs on dependencies: https://buildstream.gitlab.io/buildstream/format.html#dependencies | 10:45 |
| albfan | paulsher1ood agreed let's get away for that xml madness | 10:45 |
| tristan | And how they work | 10:45 |
| tristan | albfan, basically what I'm getting at, is that for a Flatpak build, we only want the SDK/Runtime around at build time, but it's not transitive | 10:45 |
| albfan | tristan ok. As I need buildstream for CI is ok to running on a shell. Indeed I just want to be able to run test suite | 10:46 |
| tristan | true, in that case you dont need to care | 10:46 |
| albfan | as it uses installed tests is ok | 10:46 |
| tristan | and you'll need the SDK/Runtime to be there in order to run a shell with everything built | 10:46 |
| albfan | for code coverage is harder because I need to collect gcno and gcda files from make -k check, but I will dig in the docs to access to that | 10:47 |
| albfan | perfect. Second times seems to be better than first tries, wasn't that the phrase? | 10:48 |
| albfan | I have all I need thanks Tristan | 10:48 |
| albfan | waltervargas[m] ^^ | 10:48 |
| tristan | albfan, great, let me know how it plays out :D | 10:49 |
| tristan | albfan, I have a feeling you might want to write an element, or perhaps use the script element kind, to do the actual running instead of using a shell | 10:50 |
| tristan | but by all means try the shell first | 10:50 |
| tristan | i.e. you may want to have your CI element (which doesnt produce output but just runs tests) to be your main target, and then have that run things | 10:51 |
| * persia ponders the value of a CI element capturing logs and storing the logs in an artifact | 10:52 | |
| tristan | persia, actually, I want _all_ artifacts to store logs (and some additional metadata) | 10:56 |
| tristan | thats TODO though for now | 10:56 |
| tristan | Also I think it should be optional to download | 10:56 |
| albfan | https://www.irccloud.com/pastebin/VkijvzY7/ | 10:57 |
| tristan | I think, anyway some logs can get very huge, but it will be nice to say "Here is exactly all the logs of everything in your build" | 10:57 |
| albfan | url for gcovr http://gcovr.com/guide.html#_xml_output | 10:58 |
| persia | Yes. In my imaginary ideal environment, I was thinking of using something like logstash for the logs, but having them in artifacts simplifies the infrastructure, and (assuming enough math is done) provides a provable provenance trail. | 10:59 |
| tristan | in any case regarding logs, metadata and artifacts, I want them stored in the same store, but I dont want additional data to be *in* the artifact but separate | 10:59 |
| persia | +1 | 10:59 |
| tristan | persia, so where were we... oh yeah "stack" elements, ok first off, you can view the existing elements here: https://buildstream.gitlab.io/buildstream/#elements | 11:01 |
| tristan | I have "Build Elements" separate from "General Elements" there so that they are easier to distinguish | 11:01 |
| tristan | they are all Elements, though | 11:01 |
| tristan | persia, so a "stack" is just a convenience for grouping dependencies together, that's all | 11:02 |
| paulsher1ood | +1 for stack, also :) | 11:02 |
| tristan | paulsher1ood, btw, did you try installing ruamel.yaml with apt-get on jessie ? | 11:03 |
| * tristan would update the instructions accordingly if that worked out | 11:03 | |
| paulsher1ood | no, not so far | 11:03 |
| paulsher1ood | i'll try now | 11:03 |
| tristan | on my debian stretch, the package name seems to be "python3-ruamel.yaml" | 11:04 |
| persia | tristan: So a "stack" "Element" only exists as a way of grouping several "Elements" into a logical entity for the purposes of reverse dependencies? | 11:04 |
| persia | Similar to the use of a "metapackage"? | 11:04 |
| * tristan still unclear on "reverse dependencies" | 11:04 | |
| tristan | yes similar to a metapackage | 11:04 |
| persia | In my mind, a "dependency" is when Element A requires Element B in order to execute. | 11:05 |
| persia | Element C may also have such a dependency on Element B. | 11:05 |
| tristan | Yes. | 11:05 |
| persia | Now, if we change Element B, there are no explicit dependencies on Elements A or C, as B is independnet. | 11:05 |
| persia | *BUT* We may need to understand what is affected by the change in Element B, by generating all possible Pipelines in a Project, and following the graphs. | 11:06 |
| persia | The set of things that then depend on Element B (in this example A and C), are the "reverse dependencies" of Element B. | 11:06 |
| tristan | Right, I see | 11:06 |
| tristan | We currently have no means of automatically calculating reverse dependencies | 11:07 |
| tristan | persia, you might file that as an enhancement bug to keep it on the radar | 11:07 |
| persia | One usually only looks at direct reverse dependendencies, but sometimes one wishes to look at the entire transitive set (to more clearly guage the impact of a change). | 11:07 |
| paulsher1ood | tristan: 'sudo apt-get install ruamel.yaml' followed by 'pip3 install --user .' works on jessie | 11:07 |
| persia | tristan: What is the bug? | 11:07 |
| tristan | persia, there is none, I said you might want to file one :) | 11:08 |
| tristan | so that the idea of automatically calculating reverse dependencies of an element in a project is somewhere on the radar :) | 11:08 |
| persia | For the lack of tooling to calculate rdeps? I'll see if I can do that easily. | 11:08 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues | 11:09 |
| tristan | paulsher1ood, great :) | 11:09 |
| tristan | Ok I'll try to clean up the things listed in issue 12 then | 11:10 |
| paulsher1ood | tristan: where can i find instructions for trying out your baserock gnome build? | 11:10 |
| tristan | paulsher1ood, and curiously, now that you've installed, does `man bst` work ? | 11:10 |
| paulsher1ood | tristan: yes, it works | 11:10 |
| tristan | paulsher1ood, https://blogs.gnome.org/tvb/2017/04/10/buildstream-progress-and-booting-images/ ... near the end, search the title "A booting demo" | 11:11 |
| tristan | seems in stretch I dont have a debian package 'ruamel.yaml' | 11:12 |
| tristan | oh, I do | 11:13 |
| tristan | strange that it does not come up for `apt-cache search ruamel` | 11:13 |
| tristan | aha, apt-get says: | 11:13 |
| tristan | Note, selecting 'python3-ruamel.yaml' for regex 'ruamel.yaml' | 11:13 |
| tristan | Note, selecting 'python-ruamel.yaml' for regex 'ruamel.yaml' | 11:13 |
| * paulsher1ood kicks off the 'A booting demo' build on the biggest machine he can find | 11:16 | |
| * paulsher1ood notices there's no count of progress equivalent to ybd [a/b/c] in logs | 11:17 | |
| tristan | paulsher1ood, true, I want to add a line in the status area for overall build time and exactly that | 11:20 |
| tristan | paulsher1ood, I suggest building with `bst --log-file build.log gnome/gnome-system-image.bst` | 11:21 |
| tristan | to capture the entire log | 11:21 |
| paulsher1ood | cool! it's chugging away, anyway :) | 11:21 |
| tristan | without redirecting the output, that way you get interactive _and_ saved log | 11:21 |
| paulsher1ood | i normally just copy and paste | 11:21 |
| tristan | if you have a huge terminal history, that'll work | 11:22 |
| paulsher1ood | we'll see. anyway, for a meaningful comparison vs ybd timings, the second run is what in need (assuming gits have already been cached) | 11:23 |
| tristan | if you want to test that without the fetching part... | 11:25 |
| tristan | then your artifact cache, if you want to remove it, will be in ~/.cache/buildstream/artifacts/ | 11:26 |
| tristan | paulsher1ood, then again... the fact that we parallelize fetches may just be a contributing factor in improved build times :) | 11:26 |
| paulsher1ood | we'll see about that, too :) | 11:27 |
| tristan | of course, the fact that we use an ostree clone to get a base runtime, will offset that number | 11:27 |
| paulsher1ood | i expect will need some fudge factors to convert apples times to orages times here | 11:28 |
| paulsher1ood | unless buildstream just flat out rocks, of course :-) | 11:29 |
| paulsher1ood | hmmm... so no 'total elapsed time so far' either... tut tut :) | 11:30 |
| tristan | sorry, that would be in the status area as I mentioned above, good enhancement to have though | 11:31 |
| tristan | total elapsed time + build N packages out of N | 11:31 |
| tristan | as top line in status area | 11:31 |
| tristan | (maybe even sharing the separator ~~~~~~~~~[time]~~~~~~~~[N/N/N]~~~~~~~~~~~~ | 11:32 |
| tristan | ) | 11:32 |
| tristan | save one line of terminal | 11:32 |
| tristan | paulsher1ood, the total will appear at the end (you have time to wait before that happens) | 11:32 |
| tristan | And I think there is a bug, I think that is the only timer which does not freeze on suspension | 11:33 |
| paulsher1ood | ack. i like your separator idea | 11:33 |
| tristan | So say, if you press ^C once, time stops... for every timer except the global one | 11:33 |
| * tristan pushed a fix for issue 12 and closed fwiw | 11:33 | |
| tristan | docs should be in the process of updating themselves | 11:34 |
| paulsher1ood | thanks for the fix! | 11:34 |
| tristan | thanks for identifying it :D | 11:35 |
| *** jude has quit IRC | 11:38 | |
| tristan | persia, so we got pretty far I think with vocabulary/terminology, do you want to continue and do you think we should turn this into a docs section ? | 11:54 |
| *** jude has joined #buildstream | 12:07 | |
| persia | tristan: Sorry, yeah, I think I understand everything now. | 12:25 |
| persia | If I get uncertain again, I'll ask :) | 12:26 |
| paulsher1ood | tristan: what causes 'maximum build tasks' to be 4 please? | 12:26 |
| tristan | paulsher1ood, there are few ways... the default is in BuildStream's internal default configuration: https://buildstream.gitlab.io/buildstream/buildstream.context.html#module-buildstream.context | 12:29 |
| tristan | values in that configuration can be overridden in a user's config file | 12:29 |
| tristan | Which by default will be searched for as ~/.config/buildstream.conf | 12:30 |
| paulsher1ood | ok. will be interesting | 12:30 |
| tristan | But can be overridden with `bst -c /path/to/config` | 12:30 |
| tristan | Most everything in the user config, can also be overridden directly on the command line | 12:30 |
| * paulsher1ood will not be learning any of your fancy commandline tricks | 12:30 | |
| tristan | which overrides anything else, if specified | 12:30 |
| tristan | in this case `bst --fetchers 8 ...` | 12:31 |
| paulsher1ood | stoppit, now :) | 12:31 |
| tristan | :) | 12:31 |
| tristan | 2 levels of configuration A.) config file overrides defaults B.) command line overrides everything | 12:31 |
| tristan | hope it's not *too* hard to remember :) | 12:31 |
| paulsher1ood | time-since-start is more useful than all these zeroes, imo | 12:33 |
| paulsher1ood | s/zeroes/zeroes and blanks/ | 12:34 |
| tristan | nod, I think zeroes are for the start of timed activities, blanks are for messages which have no relation to timing something | 12:34 |
| tristan | It takes some space, but I think alignment is very important | 12:35 |
| tristan | Not sure it would be intuitive to show time-since-start | 12:35 |
| paulsher1ood | that's how ybd does it, and no-one has complained | 12:36 |
| tristan | When compared to the time it took to complete a matching SUCCESS line for instance | 12:36 |
| paulsher1ood | SUCCESS line should have an Elapsed: xx:xx::xx in it imo | 12:36 |
| tristan | I dont think nobody having complained is a great metric either for a nice UI | 12:36 |
| tristan | Most things are timed, does it not make sense to have one single column reserved for timers ? | 12:37 |
| tristan | Maybe not | 12:37 |
| paulsher1ood | try this: the current info is not useful, time-since-start is | 12:37 |
| tristan | And... then "time since start" ... when the SUCCESS line comes "again time since start ?" I'm more interested in how long it took to complete | 12:38 |
| tristan | and the line length is already usually close to (if not exceeding) 80 chars | 12:38 |
| paulsher1ood | i'm talkgin about time-since-start of whole run. SUCCESS needs to report elapsed of the specific task | 12:40 |
| tristan | Thats what I mean it's unintuitive, START will on average have a time that is > SUCCESS | 12:41 |
| tristan | It could be changed so that the column is _always_ time-since-start, for _everything_, then we would still want elapsed time of specific task | 12:42 |
| paulsher1ood | START and SUCCESS are uninituitive in themselves, to me :) | 12:42 |
| * paulsher1ood would prefer the _always_ time-since-start idea | 12:43 | |
| paulsher1ood | START would go with END or similar, for me | 12:43 |
| *** Chris has quit IRC | 13:18 | |
| *** Chris has joined #buildstream | 13:25 | |
| *** tristan has quit IRC | 13:44 | |
| paulsher1ood | real160m10.573s | 14:32 |
| paulsher1ood | user98m22.756s | 14:32 |
| paulsher1ood | sys22m31.240s | 14:32 |
| ironfoot | bst building gnome? | 14:42 |
| paulsher1ood | yup | 14:42 |
| ironfoot | paulsher1ood: any other result to compare it to? | 14:42 |
| paulsher1ood | aws c4.8x-large | 14:43 |
| paulsher1ood | i need to know which version tristan converted | 14:43 |
| ironfoot | good point | 14:44 |
| ironfoot | paulsher1ood: is it the build-gnome branch in buildstream-tests.git? | 14:48 |
| paulsher1ood | yup | 14:48 |
| ironfoot | i've grep'ed some of the sha1s introduced in latest commits in definitions | 14:50 |
| ironfoot | and they are there, so I my guess is that is converted from latest definitions.git | 14:50 |
| paulsher1ood | oh? | 14:52 |
| * paulsher1ood tries building gnome-system-x86_64 from master definitions with ybd | 14:53 | |
| ironfoot | if my guess turns out to be right, you will have the relevant information already :) | 14:53 |
| paulsher1ood | :) | 14:55 |
| *** albfan has quit IRC | 15:50 | |
| *** jonathanmaw has quit IRC | 16:26 | |
| *** tristan has joined #buildstream | 16:59 | |
| *** ChanServ sets mode: +o tristan | 17:00 | |
| *** jude has quit IRC | 17:23 | |
| *** ssam2 has quit IRC | 18:14 | |
| *** tristan has quit IRC | 20:32 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!