| *** hasebastian has joined #buildstream | 02:32 | |
| *** tristan has quit IRC | 05:58 | |
| WSalmon | crash has https://gitlab.com/celduin/crash/jetbot-system-bst/-/blob/master/elements/application/testing/application-test.bst but the fundimental issue is that while you can cache the "build-root" for use in a shell you can not bring it back in to a new element | 08:21 |
|---|---|---|
| WSalmon | so for python were the "build" is supper cheap then tests work ok but for anything with a big compile time then there is not good solustion altho i hvae been in many conversations about this over the years | 08:22 |
| WSalmon | some work are to build with buildroot on and then use a build shell and runt the test in the shell but as a seperate invocation of bst | 08:23 |
| WSalmon | `bst shell -bt always element.bst -- make test` <- i havent tested that but it should run `make test` in the build directory as it was when the element finished | 08:25 |
| *** tristan has joined #buildstream | 08:35 | |
| *** ChanServ sets mode: +o tristan | 08:35 | |
| juergbi | WSalmon: we have https://gitlab.com/BuildStream/buildstream/-/issues/776 to track this | 08:42 |
| juergbi | it's considered a 2.0 blocker (at least a proposal to figure out whether this requires breaking changes) | 08:42 |
| *** santi has joined #buildstream | 08:46 | |
| *** benschubert has joined #buildstream | 09:31 | |
| *** hasebastian has quit IRC | 09:37 | |
| tristan | benschubert, this is showing promise: https://gitlab.com/BuildStream/buildstream/-/blob/tristan/partial-variables-manual-string-join/src/buildstream/_variables.pyx#L539 | 09:50 |
| benschubert | oh my :'D | 09:52 |
| benschubert | nice | 09:52 |
| benschubert | On the scheduler side, I'm almost happy with the state to have a WIP MR for review+testing | 09:52 |
| tristan | I will still have to rewind to another version before some shuffling to try to get better performance | 09:53 |
| benschubert | main blocker being the CI for bst-plugins-experimental so that I could push a new ostree plugin :) | 09:53 |
| tristan | but just looking at Value.resolve(), that seems to be a net gain according to the profile | 09:53 |
| benschubert | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1988 also if you have time, should be a quick one :) | 09:53 |
| * tristan is really falling behind because of this variables nonsense, quite stressful :'( | 09:54 | |
| tristan | coding_guidelines.rst | 09:54 |
| tristan | benschubert, Do you think we could fix that so that patching it is not effectively a useless effort ? | 09:54 |
| benschubert | tristan: so that patching it is not effectively a useless effort -> not sure I understand? | 09:55 |
| tristan | Many months ago, there was a valuable effort in splitting out the CONTRIBUTING.rst guide into sections | 09:55 |
| tristan | And the result was that everything not directly in the CONTRIBUTING.rst is no longer in a TOC anywhere | 09:55 |
| tristan | you have to "just magically know" where the subpage is and find it somehow with your browser | 09:56 |
| benschubert | oh, what? | 09:56 |
| tristan | So all of that stuff, is as good as non-existant | 09:56 |
| tristan | The release process, etc | 09:56 |
| benschubert | which explains why I couldn't find in in the docs | 09:56 |
| benschubert | do we want this in the BuildStream docs directly? If so, I can add a new "Contribute to BuildStream" part? | 09:56 |
| benschubert | and we could cleanup things as we go? | 09:57 |
| tristan | Hard to say, there is a hack we do I think for CONTRIBUTING.rst and for README.rst | 09:57 |
| tristan | which is that gitlab itself renders it, and they are *also* included in the docs | 09:57 |
| tristan | So maybe we want a minor restructure so that in the docs, the CONTRIBUTING.rst is like an introduction page of the contributing section ? | 09:58 |
| tristan | So it still says something relevant on gitlab.com ? | 09:58 |
| tristan | I don't really care personally if we have a CONTRIBUTING.rst file to satisfy gitlab.com's craving for one at all honestly | 09:59 |
| benschubert | tristan: So: A new 'contributing' section in the docs, which first page is our CONTRIBUTING.rst and then has sub categories with the rest? | 09:59 |
| tristan | benschubert, I would say that CONTRIBUTING.rst is the first ToC entry in the new 'contributing' section in the docs | 10:00 |
| benschubert | yep, ok let me try something | 10:00 |
| tristan | benschubert, that way, we get to keep rendering it in both | 10:00 |
| benschubert | yeah, will try something :) should we go in with this MR already though? | 10:00 |
| tristan | benschubert, does "_can_" do anything in rst ? | 10:03 |
| tristan | I don't think so, that's my only comment though | 10:03 |
| tristan | I'll do it in the MR | 10:03 |
| benschubert | oh I don't know actually good point I'll double hceck :) | 10:05 |
| benschubert | thanks! | 10:05 |
| benschubert | tristan: also, around the messenger: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=82197ded517fd2a37370de58d0c65043cb33084b what do you think about this commit? | 10:06 |
| tristan | I don't know :-S | 10:15 |
| benschubert | what do you dislike? | 10:15 |
| tristan | Well, the fact that there is a root message handler hehe | 10:16 |
| tristan | I'm not seeing where the not-root message handler is being processed | 10:16 |
| benschubert | in every job | 10:16 |
| benschubert | basically everytime you would use `Messenger.set_message_handler` | 10:17 |
| tristan | Every time you launch a thread/job you internally set a job specific message handler ? | 10:17 |
| benschubert | yep https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=82197ded517fd2a37370de58d0c65043cb33084b#d7d6127e94f4700edb5c77e9594ff85138d19dcd_546_541 | 10:18 |
| tristan | And what does that job specific message handler do, it passes the message along to the root message handler ? | 10:18 |
| benschubert | that was done since the start :) | 10:18 |
| benschubert | the job specific message filters out 'LOG' messages, and passes it up yes | 10:18 |
| benschubert | while writing it into a log file | 10:19 |
| tristan | And stamps the task_id ? | 10:19 |
| benschubert | correct | 10:20 |
| benschubert | as was done before, though before it went through the 'queue' | 10:20 |
| benschubert | now it is going through the 'root handler' | 10:20 |
| tristan | benschubert, in terms of naming, I would prefer the opposite I think; e.g. set_job_message_handler() | 10:20 |
| benschubert | Sure, I can do that :) or : `set_thread_local_message_handler` ? | 10:20 |
| tristan | But I still wonder if it doesnt just make more sense to have an internal function which does the job-specific stuff if it's in a job, and unconditionally passes the messages along to the message handler | 10:21 |
| tristan | s/if it doesn't make more sense/if it would be less complicated/ | 10:21 |
| tristan | Also note that your current commits don't document the differences between these two handlers (they just have different names and the doc comments don't explain why) | 10:22 |
| benschubert | right I should document that | 10:22 |
| tristan | Can I see the bit which propagates the message ? | 10:23 |
| benschubert | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=82197ded517fd2a37370de58d0c65043cb33084b#d7d6127e94f4700edb5c77e9594ff85138d19dcd_695_689 | 10:23 |
| benschubert | I can actually use `use_job_handler` instead, which would make later re-work easier | 10:23 |
| tristan | Which file ? | 10:23 |
| benschubert | job.py | 10:24 |
| tristan | Right, I think that whole thing can be much simplified, without having two handlers | 10:25 |
| tristan | Just one handler for the frontend to handle messages | 10:25 |
| tristan | You don't really need these codepaths for "propagation of messages" anymore, if they need not be propagated from a subprocess to the main process | 10:25 |
| benschubert | tristan: we seem to not propagate every LOG message though | 10:26 |
| benschubert | and add information on the element and such | 10:26 |
| tristan | benschubert, The way I see it, before getting to job.py _child_message_handler(), there is a generic entry point to pushing a message | 10:26 |
| benschubert | so it's not such a trivial change | 10:26 |
| benschubert | tristan: correct | 10:26 |
| tristan | Something which consumes a Message() object before pushing it into the complex shebang of message propagation | 10:26 |
| tristan | Now, that is a complex shebang of message propagation *because* of multiprocessing and Queues | 10:27 |
| tristan | Instead of keeping the complex shebang around | 10:27 |
| tristan | Just "if thread_local_job_data_is_around: do_thread_local_job_data_related_stuff(message)" and then "message_handler(message)" | 10:28 |
| tristan | You just need to know if the thread local data has a job reference, and use that to do the local LOG message filtering and task_id stamping | 10:28 |
| tristan | without sending it all the way to another function before getting to the message handler (much less another function which needs to be "registered" with the Messenger) | 10:29 |
| benschubert | tristan: I like this idea better. Will need to jump to a meeting. be back in an hour (or two with lunch :/ ) | 10:29 |
| benschubert | tristan: but basicallly pushing the knowledge of "am I in a job" to the messenger right? | 10:30 |
| tristan | Basically yeah | 10:30 |
| tristan | Less code, I think that code is no longer justified | 10:30 |
| * tristan remembers adding it (before Messenger existed but same difference) | 10:30 | |
| benschubert | I likw the idea. I'll try a change and send you th eocmmit :) | 10:30 |
| tristan | it was there because of Job <--> Main process Queues | 10:31 |
| tristan | WSalmon, before I try to diagnose your sample project's use of `link` element, you can confirm that this is indeed `master` you are working with, and not some tagged version ? | 10:50 |
| *** tristan has quit IRC | 10:56 | |
| *** hasebastian has joined #buildstream | 11:05 | |
| douglaswinship | Does anyone know how to restart / recreate the runners for bst-plugins-external? | 11:07 |
| douglaswinship | From conversation yesterday, I understand that "group runner bastion-debian10-c-2-4gib-ams3-01" is gone. | 11:07 |
| douglaswinship | This has been holding up an MR and at least one other pipeline since Monday evening, and I've got no idea how they work so I don't know how to fix. | 11:07 |
| traveltissues | it needs to use the group runners right? i don't have access to that | 11:13 |
| douglaswinship | From earlier conversation, yes, I think so. It's a shared runner that buildstream also uses. | 11:14 |
| douglaswinship | In other news, "Completely abolish job pickling" may be the best commit title I've seen in a long time. | 11:21 |
| traveltissues | juergbi, do you have access to ci settings? ^^ | 11:21 |
| juergbi | yes | 11:21 |
| juergbi | but afaik the issue is on the runner side, not the ci settings | 11:22 |
| *** tristan has joined #buildstream | 11:22 | |
| *** ChanServ sets mode: +o tristan | 11:22 | |
| juergbi | maybe I could move the still working bastion runner from the buildstream project to the buildstream group | 11:23 |
| juergbi | I've enabled bastion-debian9 for bst-plugins-experimental | 11:25 |
| juergbi | this should unblock CI there | 11:25 |
| traveltissues | ok, i thought that it was a group runner already, seems not | 11:26 |
| traveltissues | thanks | 11:26 |
| traveltissues | douglaswinship: ^ | 11:27 |
| douglaswinship | juergbi: brilliant, thanks! | 11:28 |
| douglaswinship | https://gitlab.com/BuildStream/bst-plugins-experimental/-/pipelines/163620251 <== It's running. :D | 11:29 |
| tristan | douglaswinship, Are you going to follow through with your completions fix ? | 11:36 |
| * tristan thinks we might not have https://gitlab.com/BuildStream/buildstream/-/issues/1358 if that was fixed :-S | 11:37 | |
| douglaswinship | tristan: sorry, I completely forgot about it. I've started using docker images whenver I do local builds, so the issue went away for me personally. | 11:43 |
| douglaswinship | And I didn't fully understand the fix that we came up with either. | 11:43 |
| douglaswinship | But I can take another look at it later today. | 11:45 |
| tristan | douglaswinship, the cli.py tries to get the current API version in the global scope in order to print a pretty default in the click options for `bst init` | 12:10 |
| tristan | douglaswinship, just remove that noise for "2.0" and everyone is happy :) | 12:11 |
| douglaswinship | I remember seing something about that code working fine, for its intended purpose in bst init, and the problem was that it couldn't be run during tab-completion. | 12:13 |
| douglaswinship | I wonder if there's a straightforward way to get it to run as intended, when intended, without being called (and causing the exception), when it's not needed? | 12:14 |
| tristan | douglaswinship, global scope when parsing cli.py the version stuff is not in context | 12:15 |
| tristan | it becomes in context only when executing a command, but not at completion time | 12:15 |
| tristan | douglaswinship, An alternative might be to use the latest in the function body of `bst init` as a default, and set the click function declaration to something like "<LATEST>" or <"CURRENT VERSION"> to indicate in the --help or man pages that the default is the current version | 12:17 |
| tristan | and then special case it inside the function `if version=="<CURRENT VERSION>": call utils.get_api_version()` | 12:17 |
| tristan | douglaswinship, but I doubt we have to be that picky, really, this is just the default min-version to use for a freshly created project.conf, whether it should depend on latest of baseline 2.0 by default is up for debate | 12:18 |
| tristan | douglaswinship, fwiw, I've been poking fun at you for selfishly hoarding your functional bash completions for a week now, I thought you would have noticed :) | 12:21 |
| tristan | douglaswinship, I hope you don't mind hehe | 12:21 |
| douglaswinship | I.... honestly haven't spotted any of that. | 12:22 |
| douglaswinship | So don't know whether to mind or not. | 12:23 |
| douglaswinship | :) | 12:23 |
| tristan | douglaswinship, some of it on this channel while I know your nick was around, some of it in channels where I expect you'd have a bouncer to reply missed messages | 12:23 |
| tristan | but basically just that... completions came up a couple times, and I mentioned the reason why they don't work is because you are hoarding them to yourself | 12:24 |
| douglaswinship | I don't have anything like that set up. If my nick isn't around, then I won't see the messags. | 12:24 |
| douglaswinship | hah! | 12:24 |
| * tristan thought highlighting your nick would be enough for you to notice them the following day, sorry if I misread that ;-) | 12:25 | |
| *** santi has quit IRC | 12:26 | |
| *** santi has joined #buildstream | 12:27 | |
| douglaswinship | oh, if i'm actually in the chat, then yes. I may have missunderstood. | 12:32 |
| douglaswinship | Either way, message received now. | 12:32 |
| tristan | WSalmon, This is what I get: https://bpa.st/WWCA | 12:38 |
| tristan | WSalmon, I did the following: (A) checkout bsp-boardsupport.git, (B) checkout your bst2 branch in there, (C) edit sample/rpi4-features/elements/bsp-boardsupport.bst to set the tracking branch to also point to your branch | 12:39 |
| tristan | (D) cd sample/rpi4-features (E) bst source track bsp-boardsupport.bst (F) bst show deploy/image.bst | 12:40 |
| tristan | (E) is something that (F) otherwise asks me to do of course | 12:40 |
| tristan | If I edit elements/freedesktop-sdk.bst I get a different error :-S | 12:43 |
| tristan | I think I found an issue with the landed juncle, but it's not the same as what WSalmon pointed out | 12:51 |
| tristan | douglaswinship, of course, I could do it myself, but I know you've been sitting on a patch so I sort of thought it would come :) | 12:58 |
| Frazer | hi, im trying to get a valid output from artifactcache in the tests so i can use `bst artifact list-contents` i have built it and tried other ways to try and get it to work, but seem to get `None of the specified artifacts are cached.` does anyone know why they wouldnt be? | 12:58 |
| tristan | Frazer, better to paste your test case somewhere, I presume it is an added pytest ? | 13:00 |
| Frazer | tristan do you mean paste my commands and output? | 13:01 |
| juergbi | abderrahim[m]: fyi: https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/merge_requests/38 | 13:03 |
| *** hasebastian has quit IRC | 13:08 | |
| tristan | Frazer, when you said "in the tests" I suppose you mean the pytest test cases you added in the buildstream tests/ directory somewhere | 13:44 |
| tristan | Frazer, but sure, commands and output could give an indicator :) | 13:44 |
| Frazer | no sorry, just went into the project of artifactcache and ran `bst build` etc, will get it up. in the docs it said in order to use bst artifact you need to be in a project that caches it, which i assumed happened after `bst build` | 13:46 |
| Frazer | tristan https://pastebin.com/wk50H2fE | 13:51 |
| coldtom | Frazer, i suspect you need to pass an element to the command, e.g. `bst artifact list-contents compose-all.bst` | 13:54 |
| Frazer | thanks, think i tried that before but with elements/....bst. sometimes forget that | 13:55 |
| Frazer | is there way to list all of the contents in the elements bst scripts, not just one | 13:56 |
| tristan | Frazer, for artifacts, the `element.bst` target is just a convenience, what `bst artifact *` commands take as input is generally /artifact names/ | 14:01 |
| tristan | which is a path including the projectname/element-name/cache-key | 14:02 |
| tristan | It was intended to support wildcards, not sure if that was implemented | 14:02 |
| tristan | Frazer, I would expect `bst artifact list-contents *` would show you "every content of every artifact", including every different build of every given element in the local artifact cache | 14:03 |
| Frazer | cant seem to get it working that way with a wildcard or .(current directory which it suggested). can work on implementing that since itll prove useful when doing `bst source show` if we go through with it | 14:03 |
| tristan | Ohhhhh | 14:07 |
| tristan | Frazer, you are probably giving the wildcard to bash, not to bst | 14:07 |
| Frazer | its done within `tox -e venv /bin/bash` so you maybe right | 14:08 |
| tristan | I also don't know if wildcarding was ever supported | 14:09 |
| tristan | I said it was the plan, I don't know if it was implemented | 14:09 |
| tristan | Not documented to at least: https://docs.buildstream.build/master/using_commands.html#bst-artifact-list-contents | 14:09 |
| tristan | Frazer, to try it, you might try single quotes | 14:10 |
| Frazer | oh yeah with quotes it works around * | 14:10 |
| tristan | `bst artifact list-contents '*'` or `bst artifact list-contents 'project-name/element.bst/*'` for every artifact produced by that element | 14:10 |
| tristan | note that specifying just an element, will only work if there is an artifact cached for the specific cache key of that element | 14:11 |
| tristan | (so it wont work if you modify the element after building it, but the old artifact matching the old key will still be there) | 14:11 |
| Frazer | ok thanks, will keep that in mind when i start work on `bst source show` and elsewhere | 14:11 |
| tristan | Sources of course are quite different as they don't have cache keys... although with the 'source cache' it might make sense to expose source cache keys :) | 14:13 |
| tristan | juergbi, do sources have cache keys after all ? I wonder if that would make sense | 14:13 |
| tristan | Frazer, it also might make sense to have `bst artifact list`, i.e. list the artifacts which exist in the local cache | 14:14 |
| juergbi | tristan: they do have cache keys right now with some caveats | 14:14 |
| tristan | we don't have such a beast at the moment | 14:14 |
| juergbi | artifact list is actually something I need to bring up on the list | 14:15 |
| juergbi | with remote asset API this would not be possible anymore | 14:15 |
| tristan | juergbi, just wondering if it makes sense to consider exposing source cache keys in the CLI and making a more consistent CLI for artifacts and sources | 14:15 |
| tristan | if so, this might be a suitable blocker too | 14:15 |
| tristan | Ah | 14:15 |
| tristan | No more ? | 14:15 |
| juergbi | right now we internally have artifact list for shell completion of artifacts | 14:16 |
| tristan | Wait, `bst artifact list` would not be possible ? | 14:16 |
| tristan | Yes, lets not lose that :'( | 14:16 |
| juergbi | that's what needs to be discussed | 14:16 |
| juergbi | the remote asset API as remote protocol doesn't block local listing, of course | 14:17 |
| tristan | Right | 14:17 |
| tristan | I wouldn't have expected, thought that surprising | 14:17 |
| juergbi | however, it's not something that is possible in the protocol (wasn't different with the current protocol) | 14:17 |
| tristan | Ah, `bst artifact list` would not be possible remotely ? | 14:17 |
| tristan | And wildcarding in `bst artifact list-content` remotely would not either ? | 14:18 |
| juergbi | yes, this won't work | 14:18 |
| tristan | I mean, I'm more concerned with listing content of artifacts I have locally anyway, but it would be nice to have this power on a dedicated remote too | 14:18 |
| juergbi | and the current plan would be to use casd for the local storage of 'remote assets', so we would have the same limitation for the local case. however, locally, we could extend it, of course | 14:18 |
| tristan | perhaps `bst --remote foo.org/artifacts artifact list-contents 'project/*'` | 14:19 |
| tristan | like that | 14:19 |
| tristan | Hmmm | 14:19 |
| tristan | So remote assets cannot be listed | 14:19 |
| juergbi | yes but besides protocol limitation, that would also be a potential security issue | 14:20 |
| tristan | Really ? I was of course thinking of extending the protocol in the future while remaining backwards compatible... | 14:20 |
| tristan | I think, once you have access to a remote, you have access to it | 14:20 |
| tristan | if you want to hide stuff, then dont give up the keys to that remote | 14:21 |
| juergbi | where you want strict isolation, you would probably not share instances | 14:21 |
| juergbi | however, I think it's a nice attribute that it easily provides very simple although limited isolation | 14:22 |
| juergbi | I don't know whether anyone would rely on this, though (and I suppose listing could always be disabled) | 14:22 |
| juergbi | tristan: what's your use case of artifact listing? | 14:23 |
| tristan | Mostly local | 14:23 |
| tristan | I've built stuff, I want to know what's there, I want to download stuff that I've built, check out various versions of things I've built, etc | 14:23 |
| tristan | I want to know what comes out of the oven and "have" it | 14:24 |
| tristan | For various reasons, I might also want to compare sizes of files for instance | 14:24 |
| tristan | I *definitely* want to be able to sort `bst artifact list` output by date, and by size | 14:25 |
| tristan | If I think of it remotely, then I think: Everything I can do locally, I can do remotely | 14:25 |
| tristan | Instead of my own machine, I'm gonna swap it out for this remote build service | 14:25 |
| tristan | now my artifacts and builds happen on this other machine or cluster | 14:26 |
| tristan | So I basically want everything I would want locally, but remotely | 14:26 |
| tristan | Ideally remoteness is totally transparent to me as a user | 14:26 |
| juergbi | tristan: I'm wondering whether storing manifests (with all cache keys) for completed build sessions could be interesting | 14:27 |
| tristan | Maybe | 14:27 |
| juergbi | that could be stored locally for local sessions independent of whether remote execution was used or not | 14:27 |
| tristan | But then you're creeping into "isolation on shared services" land, I can smell it ;-) | 14:28 |
| juergbi | and even if some artifacts themselves never hit the local cache (as this is optional at least for build-only dependencies) | 14:28 |
| tristan | As a user, I would be surprised, I would say... but gitlab has been contributing to this cache all week, how come I can't see all the artifacts it created for this element and observe their sizes choronologically ? | 14:29 |
| juergbi | not sure what you mean with that. my suggestion would be a purely local log | 14:29 |
| tristan | Or "Hey jim, didn't you build that artifact yesterday ? it's not coming up in my `bst artifact list` output" | 14:29 |
| tristan | Right, I'm projecting | 14:29 |
| juergbi | tbh, I don't want users to work like that | 14:30 |
| tristan | I'm thinking the local log is something you would use internally in the BuildStream cli to achieve separation | 14:30 |
| juergbi | I would want to use versioning on the bst project (source) side | 14:30 |
| tristan | I don't purport to know what users will do, I rather give them everything and enjoy seeing what they end up doing | 14:30 |
| juergbi | and you can always get artifact cache keys from a certain bst project version | 14:30 |
| tristan | Yeah, but that's not convenient | 14:31 |
| juergbi | not sure I agree | 14:31 |
| juergbi | I'd want a change log | 14:31 |
| tristan | Checking this stamped version from last week in order to just observe the artifact output from last week ? | 14:31 |
| juergbi | not just some random builds with dates | 14:31 |
| juergbi | maybe the same element is built from 10 different branches in arbitrary order | 14:31 |
| juergbi | what does the build timestamp help? | 14:31 |
| tristan | Sure, it's the same element though | 14:32 |
| tristan | Well, that's a good point for the timestamp | 14:32 |
| juergbi | why would I care about random builds of a single element? | 14:32 |
| tristan | Well, one would assume that you do | 14:32 |
| tristan | Because you are the one building it, you and your colleagues | 14:32 |
| tristan | In all likeliness, the artifacts from last year are purged | 14:33 |
| juergbi | the LTS branch from last year or even older might still be building | 14:33 |
| tristan | You might be reusing some of them, but if you are looking at a specific one (say a compose or a late stage element ready for deployment), then you pretty much know it's the one you've been building all week | 14:33 |
| tristan | I get it; and agree, BUT - I think we get better context incrementally | 14:34 |
| juergbi | I'd rather be interested in annotating git log output with information about cached artifacts for a particular commit | 14:34 |
| tristan | I.e. we add `bst artifact list` and `bst artifact list-contents` | 14:35 |
| tristan | Then we expose more metadata over time | 14:35 |
| tristan | Feature requests pour in | 14:35 |
| tristan | Perhaps such change-log related data gets added | 14:35 |
| juergbi | so maybe stop it before, nudge it in the right direction ;) | 14:35 |
| tristan | Eventually we get something more and more usable | 14:35 |
| tristan | If we stop it before, we likely never get there | 14:35 |
| tristan | And instead we get this inconvenient experience of having to check out project source data at specific revisions in order to have the data we *just built* at our finger tips | 14:36 |
| juergbi | I would suggest improving the user experience of that but without losing the connection to the git log | 14:37 |
| tristan | But there is no git log | 14:37 |
| tristan | I'm not sure that there ever really should be | 14:37 |
| juergbi | we could cache mappings from bst project trees to cache keys | 14:38 |
| juergbi | which could make it fairly efficient to integrate it with git log (or any other VCS) to get artifact history without explicit checkout | 14:39 |
| tristan | And this would provide context somehow... for a single client, perhaps; a history of builds, could be interesting | 14:39 |
| tristan | Moreso for CI builds | 14:39 |
| juergbi | build manifests could potentially be stored in a remote asset server | 14:39 |
| juergbi | so it could also be available for CI builds | 14:39 |
| juergbi | it's probably far from trivial but from the UX perspective this sounds much more interesting to me | 14:40 |
| tristan | I'm just saying; I don't know if another golden age will come when we get to do that much design up front; it might | 14:40 |
| tristan | And it all sounds pretty nice | 14:40 |
| tristan | I think realistically, if we have those kinds of build logs for given CI topologies (serieses of builds), it would compliment well the low hanging fruits of at least being able to talk to your artifacts, display their file lists, file sizes/artifact sizes and build dates | 14:41 |
| tristan | I think that if we block progress in that direction on better design, we are less likely to get there | 14:43 |
| tristan | It would be nice to look at a "branch" in my artifact cache and observe it's history, though; that'd be awesome | 14:44 |
| douglaswinship | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1989 | 14:44 |
| douglaswinship | ^^ tristan: the hoarding has ended! | 14:44 |
| tristan | I could checkout built revisions of it, possibly rebuilding missing revisions; possibly without even needing to reproduce the original project data | 14:45 |
| tristan | yay ! | 14:45 |
| douglaswinship | Thanks for letting me do it, and not just doing it yourself. | 14:45 |
| Frazer | since the new remote asset api will affect the implementation of `bst artifact list` and needs to be discussed, will `bst source show` be discussed there too since i think that will hvae a similar affect? | 14:47 |
| tristan | douglaswinship, already reviewed :) | 14:48 |
| tristan | Frazer, No | 14:48 |
| Frazer | ok, just checking | 14:49 |
| tristan | Frazer, this is quite unrelated, regardless of whether we use source cache or not, `bst source show` is still interesting/important | 14:49 |
| tristan | Frazer, from what I can tell, the aspects of (A) Addressing sources by cache keys and (B) Listing cached sources using wildcarded cache keys, are affected by this conversation | 14:50 |
| tristan | That's all pretty pie in the sky anyway | 14:50 |
| tristan | possibly closer than I imagine, all depends, let's see how these remote assets things land before thinking of more cool stuff we can potentially do with them :) | 14:51 |
| Frazer | thanks for the info | 14:51 |
| tristan | Wow, that hour flew by | 14:51 |
| tristan | I thought I was going for a midnight walk | 14:51 |
| tristan | hmmm | 14:51 |
| douglaswinship | tristan: good comments. Updated. | 15:00 |
| * tristan hands completions to marge ;-) | 15:01 | |
| douglaswinship | Oooops. We were both editing the MR at the same time. I appear to have unintentionally unassigned marge-bot. | 15:06 |
| douglaswinship | Didn't realise that Gitlab doesn't do edit-locks | 15:06 |
| douglaswinship | (have reassigned it to her) | 15:07 |
| *** benschubert has quit IRC | 16:56 | |
| *** cphang has quit IRC | 17:11 | |
| *** Frazer has quit IRC | 17:11 | |
| *** Frazer has joined #buildstream | 17:38 | |
| *** santi has quit IRC | 17:57 | |
| *** benschubert has joined #buildstream | 19:00 | |
| benschubert | tristan: do you prefer something like: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=ce5b818dc8f71d4d5a58e429338681974d3a8ed5 ? | 19:19 |
| benschubert | My idea of extending it later would be to allow passing a 'job' object in any API that we would need on the messenger to overwrite the 'default' context | 19:19 |
| tristan | benschubert, I was just gonna ask you if you were still around ;-P | 19:19 |
| tristan | at 4am ! | 19:19 |
| tristan | Ben never sleeps ! | 19:19 |
| tristan | hehe | 19:19 |
| benschubert | it's only 20:20 for me ;) | 19:20 |
| tristan | What am I looking at... so many files | 19:20 |
| benschubert | tristan: _messenger.py mainly | 19:20 |
| benschubert | the rest are changes necessary to make this work :) | 19:20 |
| *** xjuan_ has joined #buildstream | 19:21 | |
| tristan | still looks more beefy than I'd have expected | 19:22 |
| tristan | I guess we need the terminator signal handling for log flushing | 19:23 |
| benschubert | we probably don't | 19:23 |
| tristan | And we would want the fd open for the duration of a job, regardless of how many threads get launched over the course of a job | 19:23 |
| tristan | We don't ? | 19:23 |
| benschubert | Well we're in a context manager | 19:24 |
| benschubert | so we're guaranteed that it will be closed when we exit unless we have a really bad error (MemoryError) | 19:24 |
| tristan | Well, if we only handle SIGTERM in the main process, we still want it to join() the threads, flush etc | 19:24 |
| benschubert | or os._exit() | 19:24 |
| benschubert | yep, which we do :) | 19:24 |
| benschubert | (and it seems that it actually doesn't terminate as I would have liked to I need to fix a few things) | 19:24 |
| tristan | But I suppose we we still want to flush on SIGTERM anyway | 19:25 |
| tristan | Because we'll want to pass that fd along to child processes which block and cannot be expected to return to a .join() | 19:25 |
| benschubert | sigterm raises an exception and calls __exit__ on all CMs, so we'd be good | 19:25 |
| benschubert | that's potentially true. Anyways we can try to remove that later on, I'd rather not right now :D | 19:26 |
| tristan | when to we enter record_job() ? | 19:26 |
| tristan | My thoughts are really, we want this state (the log filename, the open log FD, etc) to be persistent across the lifetime of a job | 19:27 |
| benschubert | in job.py() https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=ce5b818dc8f71d4d5a58e429338681974d3a8ed5#d7d6127e94f4700edb5c77e9594ff85138d19dcd_552_545 | 19:27 |
| tristan | and whenever we fire a message, we know if we're in a job related thread, and we just branch and do the right thing | 19:27 |
| benschubert | tristan: that's fine as long as the whole job is done in a separate thread | 19:27 |
| benschubert | and I'd like to remove part of that job later on to be run on the main thread | 19:28 |
| benschubert | as I believe I could make a few things more efficient like this | 19:28 |
| benschubert | So, the default now is, yes we don't need to know, everything works transparently | 19:28 |
| tristan | I think we're saying the same thing | 19:28 |
| benschubert | oh, awesome :) | 19:28 |
| tristan | I.e. the lifetime of a job sees it's resources allocated, and not redundantly opened and closed | 19:29 |
| benschubert | That's correct! | 19:29 |
| tristan | If that takes multiple threads during the lifetime, fine | 19:29 |
| tristan | Which is what makes me wonder about this context manager approach | 19:29 |
| benschubert | and the _JobRecorder is the state for it, so we can pass it around if we want | 19:29 |
| benschubert | I think the context manager should be slightly re-worked later to generate a _JobRecorder but not actually set it for the thread | 19:30 |
| benschubert | and then we can have full explicit control when we want | 19:30 |
| benschubert | and when we enter a thread, we first set it globally | 19:30 |
| benschubert | so the plugins never have to know | 19:30 |
| benschubert | but the core _can_ know | 19:30 |
| benschubert | anyways I need to go and eat. We can probably continue that tomorrow? | 19:30 |
| tristan | benschubert, Can you just launch a benchmark for me ? | 19:31 |
| tristan | I might have something close | 19:31 |
| benschubert | tristan: let me know which branch, I'll run it after food! | 19:31 |
| tristan | hard to say with the shape of this data I'm testing against | 19:31 |
| tristan | I'll want the tip and the previous commit to the tip though (and master) | 19:32 |
| tristan | benschubert, tristan/partial-variables-manual-string-join (7095f4336), and the commit before the tip (69ba55a0d) | 19:33 |
| tristan | and master | 19:33 |
| *** lchlan has joined #buildstream | 19:34 | |
| tristan | I have a feeling the tip might be close performance wise, and I'm curious if the before-tip would be more performant with more usual data | 19:34 |
| tristan | One approach has Value.resolve() looking for resolved values in the dict (which is now double-suck because I need two loops in order to determine string length/width) | 19:36 |
| tristan | The other approach has the values all lined up upon entry, which one would think *should* be more performant | 19:36 |
| tristan | But with valentind's test data, the shape is such that there are many redundant variables to resolve in a single string, e.g. "%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}%{foo}" | 19:37 |
| tristan | Meaning it gets extra loopy in order to pass along one silly resolved "foo" to Value.resolve() | 19:37 |
| tristan | It's a bit wild that the dict approach was doing better and my branch regressed after lining up the values :-S | 19:38 |
| *** jjardon has joined #buildstream | 19:40 | |
| *** ChanServ sets mode: +o jjardon | 19:40 | |
| benschubert | tristan: running! | 19:51 |
| benschubert | For the messenger, withouth a proper review, which one would you rather go for, this version or the previous one? | 19:51 |
| benschubert | I'm getting close to a POC where I think the scheduler would be good enough :) Still have a few experiments I want to do and then, calls for testing and I'll split in multiple MRs | 19:52 |
| benschubert | tristan: here you go: https://gitlab.com/snippets/1994072 | 19:57 |
| *** benschubert_ has joined #buildstream | 19:58 | |
| benschubert_ | tristan: we actually have the coding documentation in the toctree already: https://buildstream.gitlab.io/buildstream/CONTRIBUTING.html#further-information | 20:02 |
| valentind | tristan, There should not be more than one of every variable directly. | 20:14 |
| tristan | valentind, hmmm, okay, I'll take that into account when pondering how these two approaches differ | 20:17 |
| tristan | benschubert_, I can see it's still back a ways off, that same obnoxious .5 seconds, but believe I'm closing in on it :-S | 20:19 |
| valentind | But note that it might use a two variables that expand using the same variables. | 20:19 |
| tristan | valentind, right, when non-recursively collecting variables, you get to loop through a lot of vars to collect inputs, at varying depths | 20:20 |
| tristan | The shape is certainly significantly different than many elements with similar, shallow-in-comparison variable tables | 20:20 |
| tristan | but, the results are consistent | 20:21 |
| tristan | oh, no they're not | 20:22 |
| tristan | well, in a way, anyway | 20:22 |
| tristan | I have a double-hash-lookup now so, with a single it would probably be cheaper than aligning the values | 20:22 |
| tristan | which tells me, there's got to be something wrong with the loop accumulating those values in the first place | 20:23 |
| tristan | although; it's all quite close - .5 seconds off of 6K elements when going from recursive to non-recursive | 20:24 |
| * tristan turns off brain for tonight | 20:24 | |
| *** Trevinho_ has joined #buildstream | 20:25 | |
| *** benschubert has quit IRC | 20:28 | |
| *** benschubert_ is now known as benschubert | 20:28 | |
| *** xjuan_ has quit IRC | 22:44 | |
| *** benschubert has quit IRC | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!