| *** ltu has quit IRC | 00:17 | |
| *** nexus has quit IRC | 00:17 | |
| *** paulsherwood has quit IRC | 00:18 | |
| *** jmac has quit IRC | 00:18 | |
| *** jmac has joined #buildstream | 00:19 | |
| *** nexus has joined #buildstream | 00:20 | |
| *** paulsherwood has joined #buildstream | 00:21 | |
| gitlab-br-bot | buildstream: merge request (workspace-directory-fix->master: Create workspace directory after checking for potential issues) #281 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/281 | 00:26 |
|---|---|---|
| *** laurenceurhegyi has joined #buildstream | 00:29 | |
| *** Prince781 has joined #buildstream | 00:45 | |
| *** Prince781 has quit IRC | 00:51 | |
| *** tristan has quit IRC | 01:22 | |
| *** Prince781 has joined #buildstream | 01:43 | |
| *** Prince781 has quit IRC | 01:47 | |
| *** Prince781 has joined #buildstream | 02:47 | |
| *** Prince781 has quit IRC | 04:13 | |
| *** Prince781 has joined #buildstream | 04:14 | |
| *** Prince781 has quit IRC | 04:42 | |
| *** Prince781 has joined #buildstream | 06:17 | |
| *** Prince781 has quit IRC | 06:22 | |
| *** Prince781 has joined #buildstream | 06:24 | |
| *** Prince781 has left #buildstream | 07:22 | |
| *** Prince781 has joined #buildstream | 07:25 | |
| *** Prince781 has quit IRC | 07:59 | |
| *** Prince781 has joined #buildstream | 08:00 | |
| *** slaf has quit IRC | 08:12 | |
| *** valentind has joined #buildstream | 08:20 | |
| *** slaf has joined #buildstream | 08:32 | |
| *** Prince781 has quit IRC | 08:48 | |
| valentind | The way cache keys are calculated depends on the machine architecture. But then it becomes difficult to import cross compiled artifacts from a different machine, through junctions. But I wonder why the machine architecture should be in the cache key, given that you need a runtime anyway to run and this runtime will be different depending on your architecture. | 10:21 |
| *** tristan has joined #buildstream | 11:10 | |
| juergbi | valentind: in non-strict mode, the imported artifacts don't affect the (weak) cache key of reverse dependencies, so it's crucial there | 12:18 |
| juergbi | the imported bootstrap could also be multi-arch (e.g., x86-64 and x86-32) | 12:18 |
| juergbi | in which case there may even be an issue in strict mode when build systems use `uname` | 12:18 |
| persia | juergbi: Any suggestions on best practice for importing bootstrap artifacts in such a case? | 12:22 |
| valentind | You still have to tell the compiler what architecture you want to use. | 12:22 |
| juergbi | 'configure' will choose a default based on `uname` | 12:23 |
| juergbi | which means that `uname` influences the build output | 12:23 |
| juergbi | (or at least may) | 12:23 |
| persia | juergbi: Right, but in the situation where I build a payload to put on a new machine to bootstrap, I *cannot* have run configure on the target arch yet. | 12:23 |
| persia | It used to be possible to use buildstream to bootstrap to new architectures. It currently is not. This seems a significant regression unless there is a sensible workaround. | 12:24 |
| persia | (I also cannot have run uname on the target arch yet) | 12:25 |
| juergbi | hm, that always requires an explicit manual (or scripted) import step, right? | 12:25 |
| juergbi | not sure why this wouldn't work anymore | 12:25 |
| valentind | Well, it would be nice to be able to import/export artifacts for junctions using different build architecture. | 12:26 |
| persia | Didn't used to: the old way was that buildstream would build something on arch (a), and then arch(b) could import that same thing. | 12:26 |
| persia | Now, because arch(a) and arch(b) differ, arch(b) cannot consume an artifact produced by arch(a). | 12:26 |
| juergbi | ah, right | 12:26 |
| persia | This is also an annoying waste of build time for things where architecture doesn't matter (e.g. documentation, java). | 12:27 |
| persia | But it is the bootstrap that is currently most painful, as it means that it isn't possible to use current buildstream for a downstream project. | 12:28 |
| juergbi | for java and co. you use a particular build tooling anyway, so i don't think this is a regression | 12:29 |
| juergbi | cross-bootstrap is indeed an issue, though | 12:29 |
| persia | For documentation, yes I use particular tooling, but I don't expect the output to differ based on host architecture. Are you suggesting it should be built for every architecture anyway, just to make sure it doesn't differ? | 12:30 |
| persia | Same for Java. | 12:30 |
| juergbi | right now there is no way for buildstream to know that it will be the same | 12:30 |
| juergbi | that's independent of the change (at least for strict mode) | 12:31 |
| persia | Aha. That makes sense. | 12:33 |
| persia | So I suppose it's just cross-bootstrap that is the problem. | 12:33 |
| juergbi | the plan is that you can tell buildstream what execution environment it should use per project or possibly even per element | 12:36 |
| juergbi | this should then also adapt uname() output accordingly using setarch | 12:36 |
| juergbi | the cache key will then contain that specified execution environment independent of host uname | 12:37 |
| juergbi | and this should work properly across junctions | 12:37 |
| persia | Current cross-bootstrap supports changes not permitted by setarch | 12:37 |
| juergbi | that's not needed | 12:37 |
| juergbi | for the subproject (or elements) that are built on x86-64, execution environemnt will be set to x86-64 | 12:38 |
| juergbi | for the rest, execution environment will be aarch64 | 12:38 |
| juergbi | buildstream will use the right cache key per element | 12:38 |
| juergbi | cache keys will still differ depending on whether e.g., you used x86-64 or power to build the cross-bootstrap | 12:39 |
| juergbi | but no manual import is required | 12:39 |
| juergbi | as long as you specify in a project option (or hard-code it in .bst) what architecture you used for bootstrapping | 12:40 |
| juergbi | and with remote execution, a single bst session could even schedule both parts on appropriate servers (or emulators) | 12:41 |
| persia | Oh, that works. Any chance that we can back out the "add architecture to cache string" change until that lands? Otherwise we're stuck on older buildstream for CI. | 12:43 |
| juergbi | would change cache key again. i'd rather avoid it. especially as it can produce conflicts for GNOME with non-strict builds | 12:44 |
| persia | Well, freedesktop-sdk is currently unable to reuse cache in CI: the last change breaks things in ways that make it unclear how to continue. | 12:45 |
| juergbi | a branch with it reverted for cross CI may be a short time workaround | 12:45 |
| persia | How long is "short term"? | 12:45 |
| juergbi | we should prioritize the story for configuring it, so hopefully not very long | 12:45 |
| persia | what commit do we want to use for the reversion? | 12:46 |
| juergbi | i can quickly prepare a branch and regularly rebase it | 12:47 |
| persia | jjardon[m]: valentind: Does that work for you? | 12:47 |
| juergbi | * [new branch] cache-key-v0 -> cache-key-v0 | 12:52 |
| jjardon[m] | persia juergbi happy to help testing stuff in a branch but master of our ci can only depend in a specific version / SHA of bst | 12:55 |
| juergbi | jjardon[m]: and you don't want to set CI to a bst SHA that is outside master? | 12:59 |
| jjardon[m] | juergbi you cannot rebase then or it will break the build | 12:59 |
| juergbi | jjardon[m]: i don't understand. why does rebase break the build? (i could also use merge instead of rebase, if that helps) | 13:00 |
| persia | rebase would only break the build if the original SHA was garbage-collected. | 13:01 |
| juergbi | ah, i understand now | 13:02 |
| persia | If each rebase is tagged (or similar) in the branch, then everything works smoothly. | 13:02 |
| persia | Or if rebases are carefully coordinated. | 13:02 |
| juergbi | i don't think i want to add tags for something non-permanent like this | 13:02 |
| juergbi | but i can just merge instead | 13:02 |
| persia | Then can we revert master? | 13:02 |
| juergbi | is hopefully not for a long time | 13:02 |
| persia | Merge also preserves hsitory. | 13:02 |
| persia | But I'm curious: does anyone actually use the new cache key formula in production today? | 13:03 |
| persia | Well, freedesktop-sdk does, but it breaks things. Anyone else? | 13:03 |
| juergbi | gnome instructions point to bst master right now, don't they? | 13:03 |
| persia | No idea. | 13:04 |
| persia | But if so, as buildstream is a GNOME project, that's a very strong argument for not reverting. | 13:04 |
| juergbi | they point to buildstream install instructions, which recommend master right now | 13:05 |
| juergbi | (will probably change once we release 1.2) | 13:05 |
| persia | hrm. | 13:12 |
| jjardon[m] | revert is fine from GNOME POV I'd say | 13:27 |
| jjardon[m] | In my opinnion, instructions should point to the 1.0 branch by default | 13:29 |
| juergbi | revert would be up to tristan. i tend not to revert it and fix it properly as soon as possible | 13:32 |
| juergbi | using the cache-key-v0 branch for freedesktop-sdk in the short term | 13:33 |
| juergbi | (and i'll merge instead of rebase) | 13:33 |
| jjardon[m] | juergbi: ok, should we open an issue in best so I can link https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/49 to it? | 13:36 |
| jjardon[m] | In bst* | 13:36 |
| juergbi | yes, that would be good | 13:37 |
| juergbi | configurable execution environment | 13:37 |
| persia | For clarity, for revert, I was thinking the right model would be a new commit undoing the changes in the old commit, rather than anything requiring force-push, so consumers didn't need to care (except regarding cache keys). | 13:45 |
| persia | Whether to do it is separate. | 13:45 |
| jjardon[m] | Of course force push should never be done in master | 13:49 |
| gitlab-br-bot | buildstream: issue #260 ("Not possible to import/export artifacts using different build architecture.") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/260 | 13:56 |
| jjardon[m] | juergbi ^ feel free to reword if needed | 14:00 |
| juergbi | persia: yes, of course. i did revert like that in the new branch but i don't intend to merge that to master, assuming we can properly fix it soon | 14:04 |
| juergbi | jjardon[m]: ta | 14:05 |
| persia | juergbi: My main concern is jjardon[m]'s CI: if that works, then I'm calm. | 14:05 |
| juergbi | yes, i won't force push on that branch, so it should be fine | 14:05 |
| tristan | I'm not up to speed on the details of current discussion; but I dont see how it was ever possible to re-use the cross built runtime without doing any export/import ostree dance | 15:07 |
| tristan | i.e. I would expect the output of a cross build to be exported to a real repo, and that would be used for the self-hosting build in foreign arch to continue the build | 15:07 |
| tristan | This should be better thought out and integrated in the process one day, especially when we have sandboxes which execute under a specified machine arch / host OS | 15:08 |
| juergbi | tristan: the cross-bootstrap can be in the same project as the elements that you build natively (or in a subproject via junction). you then populate the artifact cache using the x86-64 build machine up to the cross bootstrap. when you then build the rest on your target system, it will take the cross bootstrap from the artifact cache as there were no changes in the .bst files | 15:47 |
| juergbi | yes, we need a way to configure the execution environment. to unbreak the freedesktop-sdk cross bootstrap we need at least be able to specify the execution architecture per project | 15:48 |
| persia | tristan: The old way was just to depend on the cross-built artifact, so it ended up populating the sandbox without ever being unpacked/repacked. That optimisation might have been poking more into internals than was strictly appropriate. | 15:53 |
| persia | That said, I think it would be better for buildstream to support depending on artifacts built on other architectures vs. expecting everyone to do some export/import dance (and lost guarantees about the content in doing so). | 15:54 |
| tristan | persia, I'm not saying it's ideal; but I dont have the expectation of things to work without it | 16:23 |
| tristan | not until a solution is really designed for that | 16:23 |
| tristan | as far as I can see, this would completely break if the artifact cache doesnt have a cached artifact for the base system, as you would fallback to trying to cross build the target runtime from x86_64 if the target runtime didnt yet exist in the cache | 16:24 |
| tristan | but a cache is only a cache by current definitions of a cache, as such the cache is allowed to be wiped | 16:24 |
| tristan | but a build should still work in the absence of a cache | 16:24 |
| tristan | This looks like circumventing export/import "worked" purely by coincidence, but not (yet) by well thought out design | 16:29 |
| *** Prince781 has joined #buildstream | 19:29 | |
| *** Prince781 has quit IRC | 20:20 | |
| *** Prince781_ has joined #buildstream | 20:20 | |
| *** Prince781_ is now known as Prince781 | 20:21 | |
| *** tristan has quit IRC | 20:30 | |
| persia | tristan: yeah, I think you're right. I look forward to the right model :) | 21:40 |
| *** valentind has quit IRC | 22:12 | |
| *** Prince781 has quit IRC | 22:32 | |
| *** Prince781 has joined #buildstream | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!