| *** jward has quit IRC | 00:16 | |
| *** jjardon has quit IRC | 00:17 | |
| *** lchlan has quit IRC | 00:17 | |
| *** jward has joined #buildstream | 00:17 | |
| *** lchlan has joined #buildstream | 00:17 | |
| *** jjardon has joined #buildstream | 00:18 | |
| *** ChanServ sets mode: +o jjardon | 00:18 | |
| *** jward has quit IRC | 00:18 | |
| *** jward has joined #buildstream | 00:18 | |
| *** jward has quit IRC | 00:19 | |
| *** jward has joined #buildstream | 00:20 | |
| *** tristan_ has quit IRC | 01:43 | |
| *** tristan has joined #buildstream | 02:10 | |
| *** ChanServ sets mode: +o tristan | 02:10 | |
| *** tristan has quit IRC | 02:42 | |
| *** tristan has joined #buildstream | 04:28 | |
| *** ChanServ sets mode: +o tristan | 04:28 | |
| * tristan is curious why we have source cache and element source cache | 04:58 | |
| tristan | juergbi, do you know why we have both ? | 04:59 |
|---|---|---|
| tristan | It looks like when we fetch sources from a remote cache, we first try the element source cache, and we completely disregard the source cache if we succeeded to download from the element source cache | 05:01 |
| juergbi | tristan: the element source cache contains the combined sources of each element together with a buildstream source proto for metadata (not that useful for now but we can extend it with provenance for the sources) | 05:25 |
| juergbi | the other source cache covers individual sources without extra metadata and the idea there is that as far as possible the corresponding Remote Asset URIs will be buildstream-independent | 05:26 |
| juergbi | allowing e.g. a Remote Asset server to import sources from a git server on its own without any BuildStream client having to push it | 05:27 |
| juergbi | the main reason we need the element source cache is because going from the individual sources from the source cache to the combined sources is not always that cheap | 05:28 |
| juergbi | e.g. if `patch` is used, it isn't a simple merging of two sources | 05:28 |
| tristan | Hmmmmmm | 06:02 |
| tristan | So there is a presumption that the result of Source.stage() can be reproduced by a remote service | 06:04 |
| juergbi | tristan: it's not a requirement for all sources, however, the plan is to support this for common ones such as git | 06:06 |
| tristan | Yes I was never completely clear on how that was going to be reliable | 06:06 |
| juergbi | right now we still use buildstream-specific URIs for all sources, so there is no issue in any case. but it would be one of the next steps in this regard | 06:07 |
| tristan | I understand the plan in a vague way, but will there be Source API changes which enforce some things ensuring the results are identical to a described standard, that is reproduced in some other service ? | 06:07 |
| tristan | Maybe it will only be supported by specifically vetted sources ? | 06:08 |
| juergbi | the Source API will definitely at least have to be extended | 06:08 |
| juergbi | yes, I expect this to be opt-in on a per source basis | 06:08 |
| juergbi | the source plugin will have to provide means to generate the standard Remote Asset URI, e.g. | 06:09 |
| tristan | I also wonder, while it may seem interesting to have exotic cases where you configure your source cache and element source cache separately, if we could have a default setup where at least you only have to configure your source cache once | 06:09 |
| tristan | Maybe, more projects are not interested in having a source cache (unless they are suitable to leverage this future remote service), and for now it is better to only configure an element source cache | 06:10 |
| tristan | (not only will the source plugin have to provide a means to generate that standard key/uri thing: It will have to be guaranteed to produce exactly the same results as the remote service, that is the more worrisome part) | 06:11 |
| tristan | For now, without diving too deep into that feature, I | 06:12 |
| tristan | gah | 06:12 |
| tristan | I'm wondering about configuration API surface / UX - there is a lot of different caches and such | 06:12 |
| juergbi | tristan: both caches currently use the same server config | 06:12 |
| juergbi | I'm not planning to enable standard uri for any source unless it's very clearly defined | 06:13 |
| tristan | Ah indeed | 06:13 |
| tristan | "source-caches" is for both, I missed/forgot that | 06:14 |
| juergbi | for git the question is mainly about submodules, I suppose. maybe we should split it up there in a separate request for each repo/submodule. and then it would likely be straight forward | 06:14 |
| juergbi | (haven't looked too far into the details yet of this step) | 06:14 |
| juergbi | we should in any case be very careful about standardizing this | 06:15 |
| tristan | This looks like a pretty far off feature | 06:16 |
| * tristan wonders if we could have the full detailed plan in place so that we don't risk blocking 2.0 on this | 06:16 | |
| juergbi | the main blocker is the source plugin API | 06:17 |
| juergbi | we want/need that to be in place for 2.0, at the very least such that we know it can be extended in a compatible way for this feature | 06:17 |
| tristan | I forget which part of that | 06:17 |
| juergbi | my time on BuildStream is limited, unfortunately | 06:17 |
| tristan | I mean if it is only adding a method to generate the URL, then it need not be added by 2.0 | 06:18 |
| juergbi | yes but we can't really be sure whether that's all that's needed | 06:18 |
| juergbi | likely need a prototype to validate | 06:19 |
| tristan | I am worried that 2.0 keeps sliding off the horizon | 06:19 |
| juergbi | in the further off future also some tracking would work without requiring the buildstream client to have direct access to the sources | 06:19 |
| juergbi | which may also be API relevant | 06:19 |
| juergbi | yes, that's understandable | 06:20 |
| juergbi | we should probably all sync up again on where we stand | 06:20 |
| tristan | So I'm interested in lowering the barrier to that | 06:20 |
| tristan | Yeah | 06:20 |
| juergbi | it may well be that even if that feature would lead to incompatible plugin API changes, we could find a way to support it in a compatible fashion with a slightly less neat API or some flag | 06:21 |
| juergbi | and the latter may be worth it if it unblocks 2.0 | 06:22 |
| tristan | Thats indeed my feeling | 06:22 |
| tristan | For now I think there still remain some important real blockers | 06:22 |
| tristan | So I hope to chop them off one by one and corner the team into a "2.0 or bust" scenario (hehe) | 06:23 |
| tristan | I guess a useful thing will be to get through this remote service config surface refactor (almost done), and the next big ticket proposal for sealing off the sandbox | 06:24 |
| tristan | if I can get over that second hump, it would be an ideal time to regroup | 06:25 |
| nanonyme | tristan, what's the right repo to file PR's to at the moment? | 07:14 |
| tristan | nanonyme, https://github.com/apache/buildstream | 08:16 |
| *** tristan changes topic to "BuildStream 1.6.1 is out ! | https://github.com/apache/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://lists.apache.org/list.html?dev@buildstream.apache.org" | 08:16 | |
| *** santi has joined #buildstream | 09:43 | |
| *** tristan has quit IRC | 12:35 | |
| *** tristan has joined #buildstream | 12:47 | |
| *** ChanServ sets mode: +o tristan | 12:47 | |
| *** santi has quit IRC | 18:36 | |
| *** Mark has quit IRC | 18:53 | |
| *** abderrahim[m] has quit IRC | 18:53 | |
| *** Mark has joined #buildstream | 19:01 | |
| *** TobiasFella has joined #buildstream | 19:47 | |
| nanonyme | tristan, I recently bumped into that eg git track doesn't expand variables. Should it? Could it? | 20:26 |
| *** jjardon has quit IRC | 21:35 | |
| *** lchlan has quit IRC | 21:36 | |
| *** lchlan has joined #buildstream | 21:36 | |
| *** jjardon has joined #buildstream | 21:36 | |
| *** ChanServ sets mode: +o jjardon | 21:36 | |
| *** tristan has quit IRC | 21:49 | |
| *** tristan has joined #buildstream | 21:49 | |
| *** ChanServ sets mode: +o tristan | 21:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!