| *** tristan has joined #buildstream | 06:00 | |
| *** ChanServ sets mode: +o tristan | 06:01 | |
| tristan | juergbi, Ummm... I'm going cleanup/churn the scheduler and queues a bit today I think... it's in need, hope it doesnt cause too much grief to your work branch though :-/ | 07:00 |
|---|---|---|
| juergbi | push/pull is actually not that much code, so will hopefully not be a big issue | 07:02 |
| tristan | its been growing for a while | 07:06 |
| tristan | The tipping point is that right now, I wanted to implement `bst build --track` (i.e. optionally additionally track) | 07:07 |
| tristan | And so I want --fetchers and --builders to correspond to queue types, not actual queues (i.e. I want fetch and track to consume the same maximum tasks) | 07:08 |
| tristan | also some refactor and code splitting there will make it easier to reason about showing whats going on in the frontend | 07:09 |
| juergbi | ok | 07:12 |
| juergbi | btw: i added artifact fetching to the fetch queue for now, not the assemble queue suggested on the wiki | 07:12 |
| juergbi | it doesn't make sense to fetch sources if you can fetch the artifact | 07:12 |
| juergbi | however, maybe we want to split this fetch queue into separate artifact and source fetch queues and also share max fetchers | 07:13 |
| juergbi | would probably make things a bit cleaner | 07:14 |
| tristan | juergbi, good point yeah | 07:17 |
| tristan | Actually, I certainly agree I want a separate queue for pulling the artifacts | 07:17 |
| tristan | But only did not want to make it a hard blocker on the initial implementation | 07:18 |
| tristan | juergbi, I'm thinking also it will make sense to have `bst pull` and `bst push` for the sake of running those queues on their own | 07:18 |
| tristan | at least for `bst push` it will certainly make sense to allow this as a separate step, less clear about pull | 07:19 |
| tristan | that said, I'm enjoying this approach of queues, it's making it very interesting to be able to just slap some queues together on a scheduler and do interesting things :) | 07:20 |
| juergbi | yes | 07:21 |
| tristan | juergbi, regarding user configuration... I think it makes sense to have 3 different types right ? | 07:21 |
| tristan | I mean, we have fetchers, builders, and we'll want... I guess pushers | 07:22 |
| juergbi | right now i only push artifacts that have been built in the same session. there is no persistent state of whether an artifact is pushed already (besides simply trying to push every single artifact) | 07:22 |
| tristan | Downloading artifacts, tracking sources and fetching sources, all fall into the "downloading stuff" (fetchers) category | 07:22 |
| juergbi | yes, that's what i added | 07:22 |
| juergbi | these categories make sense for me | 07:22 |
| juergbi | download, cpu/disk, upload | 07:22 |
| tristan | yeah makes sense to me too :) | 07:23 |
| tristan | I hope we wont need any local persistence for that | 07:23 |
| tristan | I.e. if we want to start a `bst push` session, either A.) ostree is smart enough (cause it's actually pulling on the remote side) to make it essentially a noop | 07:23 |
| tristan | Or B.) perhaps using a summary, we can just know in what steps, which cache keys exist in the remote repo | 07:24 |
| tristan | *know in one step | 07:24 |
| tristan | (not sure I understand the summary perfectly, though) | 07:24 |
| tristan | just wild guess that we can do that | 07:25 |
| juergbi | have to take a closer look at the summary, this might help | 07:30 |
| juergbi | fetch the summary at the start and use this to check what needs to be pushed | 07:31 |
| juergbi | might actually also be useful for pulling. that way, we know beforehand what we can expect to be able to pull | 07:31 |
| tristan | yeah indeed | 07:35 |
| tristan | We can avoid waiting for network replies for each artifact | 07:35 |
| tristan | juergbi, in any case, I do know for certain that it's possible for us to know about a ref, without having the object | 07:36 |
| tristan | So there should be some way :) | 07:36 |
| juergbi | yes, it should definitely be possible | 07:36 |
| juergbi | will try to get that implemented before i leave | 07:37 |
| tristan | I know that because I learned the hard way... and ended up doing https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_ostree.py#L178 | 07:37 |
| tristan | :) | 07:37 |
| tristan | Just dont know how to trigger it | 07:37 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Scheduler overhaul.) https://gitlab.com/BuildStream/buildstream/commit/dce8a28277340a76fb10dbf10586b03aaef595b9 | 09:12 |
| tristan | juergbi, fyi, the scheduler refactor is now available (above reported commit) | 09:13 |
| tristan | Things are significantly easier to follow now | 09:13 |
| *** tristan has quit IRC | 11:50 | |
| *** tristan has joined #buildstream | 11:52 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 13 commits (last: source.py: Allow forceful reinterrogation of consistency state) https://gitlab.com/BuildStream/buildstream/commit/137fac4a2824908a026e31758648d728d5bd94bf | 11:52 |
| gitlab-br-bot | buildstream: issue #33 ("bst build could use a --track option") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/33 | 11:53 |
| *** tristan has quit IRC | 12:06 | |
| *** tristan has joined #buildstream | 12:29 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!