| *** tristan has joined #buildstream | 05:24 | |
| *** tristan has quit IRC | 06:44 | |
| *** tristan has joined #buildstream | 10:29 | |
| paulsherwood | tristan: can we have proper logging of what's going on at the start, please? | 11:42 |
|---|---|---|
| paulsherwood | Loading: 516 | 11:42 |
| paulsherwood | Resolving: 516/516 | 11:42 |
| paulsherwood | Checking: 516/516 | 11:42 |
| paulsherwood | takes quite a long time (10 seconds or more) no timestamping of that step | 11:43 |
| paulsherwood | and i'm assuming it's equivalent to ybd cache-key resolution. that being the case, there are reasons to want to log all of the component cache-keys at this point | 11:44 |
| paulsherwood | also, your overall time clock seems to be excluding this setup step | 12:09 |
| tristan | paulsherwood, Sure we could, I think on my machine for building the baserock full gnome build I'm at about 6 seconds | 13:33 |
| tristan | for what it's worth, cache key calculation is done by the second step, the third step takes longest because it's interrogating the filesystem | 13:33 |
| tristan | asking source plugins if they have a local cache (if a fetch is needed) and asking the artifact cache if the key is locally available | 13:34 |
| tristan | paulsherwood, outputting all the cache keys I'm kind of unsure I like the idea of dumping all of that by default to the console to be honest, I know ybd likes to do that | 13:36 |
| tristan | paulsherwood, on the other hand there is no reason why a CI build script could not do `bst show` in advance to get whatever information it wants into a log file | 13:36 |
| tristan | which is why I'm a bit hesitant to do that huge dump by default | 13:36 |
| paulsherwood | tristan: the cache-keys are part of the audit log for a given build, imo | 13:41 |
| paulsherwood | (and i've seen plenty of use-cases where it's needed to trace back to precisely what artifacts where built, and why) | 13:41 |
| paulsherwood | the common one is 'why did this change rebuild everything?' | 13:42 |
| paulsherwood | another is wanting to look back from an artifact and find what led to it being built | 13:43 |
| paulsherwood | while we're here, any chance you can make buildstream.gitlab.io point to buildstream.gitlab.io/buildstream, rather than 404? | 13:44 |
| tristan | paulsherwood, for the case of printing cache keys, I dont see why say, a build server (or user) cannot just type `bst show --deps all <target>` before running `bst build` so that you _do_ have that information you want stored somewhere | 13:50 |
| tristan | the benefit of separating this is that bst show can be scriptable (I'd like to improve on that actually), and allow for showing more things than just cache key | 13:50 |
| tristan | hard coding that into every launch of bst build is rigid | 13:50 |
| tristan | so say, maybe one day you might want to have a very verbose log, and show the resolved element configurations and all the build environment variables for each element, in advance of launching the build | 13:51 |
| tristan | while a normal user (developer) probably just wants to build | 13:51 |
| tristan | in any case, a normal user can always type `bst show` in a separate terminal on the same project to get that information, even when a long standing build is ongoing | 13:52 |
| tristan | I also know we would like to have a website at buildstream.gitlab.io, I'm just one person, though, I have to prioritize right now on getting builds of the GNOME modulesets working | 13:53 |
| tristan | paulsherwood, a temporary redirect with a static website at buildstream.gitlab.io until we do have a main website could be a good thing, I could make sure that you are in the group and have the necessary permissions if you want to do that | 13:54 |
| paulsherwood | i'll do it if i get time | 13:55 |
| * tristan just added foobarzzz to the group | 13:56 | |
| tristan | fwiw I did report https://github.com/pallets/click/issues/780 and submitted https://github.com/pallets/click/pull/781, which I'm hoping will land in click soon, and will give us a good tab completion UX | 13:58 |
| paulsherwood | tristan: i think you're missing the point a bit on this cache-key stuff. any given log should have cache-keys in it, so we don't accidentally end up reading the wrong log, or wondering which log was for which artifact(s) | 13:58 |
| paulsherwood | ' Target: gnome/gnome-system-image.bst | 13:59 |
| paulsherwood | is useless information, once i have more than a handful of logs | 13:59 |
| tristan | paulsherwood, in that case, every log, at least for anything that was actually built (or attempted to build), already does; but we use a configurable abbrebiated cache key length for this | 13:59 |
| tristan | every log line has the cache key of the related artifact in it | 14:00 |
| tristan | (first 8 chars at least) | 14:00 |
| paulsherwood | not every log line, clearly :) | 14:00 |
| tristan | Some log lines are not related to artifacts, like fetching sources | 14:00 |
| tristan | And when an element has no ref (but only a track attribute), no cache key is available yet | 14:01 |
| paulsherwood | looking at that target line i've just quoted... i a ssume bst knows the key at that point? | 14:01 |
| tristan | That is part of a summary yes, at that point it's there, unless we are running `bst track` | 14:02 |
| paulsherwood | so please state the key in full there? | 14:02 |
| tristan | but then, the cache key should be in the first line of the regular logging mechanism | 14:02 |
| tristan | Sure, can do; I just would _like_ to keep the summary nice and brief | 14:03 |
| paulsherwood | you mean '[--:--:--][0e3e392d][gnome/gnome-system-image.bst ] START Starting build' ? | 14:03 |
| tristan | Right | 14:03 |
| paulsherwood | we had a situation this week where stuff changed *during* a ci pipeline (ybd built one thing, then immediate deploy discovered the change, rebuilt everything) | 14:05 |
| tristan | that sounds like a bug | 14:06 |
| tristan | In any case, a deploy should normally be a build of one pipeline, but you should _never_ end up discovering anything new unless you explicitly run `bst track` | 14:07 |
| paulsherwood | depends on your pov. at the time the build ran, it generated the correct thing. at the time the deploy ran, the input had already changed | 14:07 |
| tristan | in the case of buildstream, that must be an explicit action | 14:07 |
| paulsherwood | maybe. i expect people will manage to misconfigure. having full keys at least made it easy (for me) to track down what had changed | 14:08 |
| paulsherwood | tristan: it's an easy enough flag for config? show-keys: t/f ? | 14:09 |
| tristan | there is going to certainly be some discovery, and it's going to be difficult or impossible to output logs which are exactly what everyone wants; on the other hand it's desirable to not have more than one frontend/logging style | 14:09 |
| tristan | that sounds like a reasonable option, but perhaps instead; a configuration matching the input of `bst show` would be more appropriate/dynamic | 14:10 |
| * paulsherwood notes that the keys would be a few hundred lines, in a log of tens/hundreds of thousands of lines | 14:11 | |
| tristan | so internally, `bst build`, if configured to `show --foo --bar` at build time, would do that action in advance of every build | 14:11 |
| tristan | whether it be only cache keys, or other information about the pipeline, that users might want to see / store at the beginning of a build | 14:12 |
| paulsherwood | yup. i want to see precisely what version of bst ran, for example... not your 0.1 approximation :) | 14:12 |
| paulsherwood | and all of the config | 14:13 |
| tristan | There I think we are getting into the realm of wrapper scripts | 14:13 |
| paulsherwood | really? 0.1 is useless | 14:13 |
| tristan | Because just because buildstream happens to be revisioned by us in a git repo, does not mean that users obtained it by cloning the git repo | 14:13 |
| tristan | at least it will not mean that if it is installed by a package manager | 14:14 |
| tristan | or any other means | 14:14 |
| paulsherwood | when you (or anyone) creates the package, that script can trap the sha, surely | 14:14 |
| tristan | indeed, that could happen | 14:14 |
| tristan | as a part of a `make dist` of sorts | 14:14 |
| paulsherwood | this is a subtle point that i've been arguing about for decades... even expert people such as you keep on getting it wrong :) | 14:15 |
| tristan | I still tend to disagree | 14:15 |
| tristan | I mean, you might want the commit sha encoded into your release data, sure | 14:16 |
| paulsherwood | i know you do, and i'm sorry that you fail to understand it :) | 14:16 |
| tristan | But, if you had made a release, you should have bumped the minor point release, and that is what should be used | 14:16 |
| paulsherwood | every time someone (including you) manually puts some kind of 'version' into something, it's a bug waiting to happen | 14:16 |
| tristan | If you did not make a release, then people are using an unstable/unvetted copy | 14:16 |
| tristan | I see the relevance of encoding it, but I think we should do better by providing properly revisioned release "tarballs" or such | 14:17 |
| paulsherwood | bst --version gives me the wrong thing, is my proof | 14:18 |
| tristan | that is one of the things about tarballs that I have to stand behind, there needs to be a point where what we released is stable and official, and what is git is just development branches, not for production | 14:18 |
| tristan | bst --version reports 0.1 for me, and we're not ready to be stable and making releases | 14:19 |
| paulsherwood | that's fine, as far as it goes. but if your only 'version' in a tarball is something you 'should have bumped', you're broken by design :) | 14:19 |
| tristan | once we hit 1.0 (which may not have any versions in between), then we should be doing that | 14:19 |
| paulsherwood | 'bst --version reports 0.1' for various versions of bst | 14:19 |
| paulsherwood | including latest master, which is not when you set it to 0.1 | 14:20 |
| tristan | they are all 0.1 as far as I can see, so far | 14:20 |
| paulsherwood | which is exactly my point | 14:20 |
| tristan | i.e. they are all pre-release alpha software | 14:20 |
| paulsherwood | it's useless information, tristan | 14:20 |
| paulsherwood | anyway i raised an issue. i'm only going on about it here because i fear you still don't understand my point | 14:21 |
| paulsherwood | (and i've had this argument for decades, as i've said) | 14:21 |
| tristan | paulsherwood, no I do see your point, we can encode the git sha into release metadata for it to end up additionally in the versioning information; I do disagree that major.minor.micro point release versioning is useless, though | 14:29 |
| paulsherwood | i'm not saying point release versioning is useless. but if any versioning can get out of step, or miss a bump accidentall i can't rely on it, can i? | 14:31 |
| tristan | it should be going through a well defined release pipeline, which is something we're not doing yet, either | 14:32 |
| tristan | for instance if you release on ftp.gnome.org, you will not be able to release an already released version, that just wont work | 14:32 |
| tristan | the same will apply if you do the weird pypi wheels thing, too | 14:32 |
| tristan | once upstream releases a version publicly, it becomes available for distro packagers to do their work, etc | 14:33 |
| tristan | there should be checks and balances, just that they are not always git | 14:33 |
| tristan | that said, even in the case that you did all of that and are running a properly released software, does not mean it is not interesting to have the upstream git commit sha (or whatever VCS versioning data in use) encoded into the final versioning information | 14:35 |
| tristan | and, that versioning information becomes especially interesting for those who consume unreleased software | 14:35 |
| paulsherwood | +1 :) | 14:45 |
| paulsherwood | so my build completed in real150m49.805s | 15:59 |
| paulsherwood | (no fetching) | 15:59 |
| paulsherwood | which is sigificantly faster than ybd for the same usecase | 16:00 |
| paulsherwood | does it build all of the same things (for gnome system)? | 16:01 |
| tristan | paulsherwood, yes it does, and it creates a bootable image (so I guess you could compare it to the gnome "cluster") | 16:31 |
| tristan | maybe that part could be optimized, or, is possibly slower than with ybd and root super powers | 16:32 |
| tristan | since the image splicing is intense and not needed when you can mount stuff | 16:32 |
| tristan | actually splitting integration commands into two categories (commands needed for using runtimes vs commands needed for booting systems) will also significantly optimize that build | 16:34 |
| tristan | because it turns out especially the cracklib integration command is time consuming | 16:35 |
| tristan | (and doesnt need to run for the ... probably around 20 or 50 elements it runs for) | 16:35 |
| tristan | then, other things in the build metadata itself could be optimized, for instance the conversion keeps around some hacks that are no longer needed | 16:37 |
| tristan | like generating the pixbuf loader caches for various builds | 16:37 |
| *** tristan has quit IRC | 19:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!