| *** slaf has quit IRC | 00:25 | |
| albfan[m] | tristan, just need to remove the type:build from autotools test! | 04:13 |
|---|---|---|
| albfan[m] | and build meanwhile workspace is open. haha I changed the hello world project based on flatpak runtimes | 04:58 |
| paulsherwood | albfan[m]: can you publish that somewhere? i'd like to try it :-) | 05:01 |
| *** bochecha_ has quit IRC | 05:26 | |
| *** bochecha_ has joined #buildstream | 05:26 | |
| *** bochecha_ has quit IRC | 05:31 | |
| albfan[m] | https://gitlab.gnome.org/albfan/buildstream-flatpak-autools | 06:20 |
| albfan[m] | paulsherwood: just pushed it | 06:20 |
| albfan[m] | I have to talk with tristan to see how we can add more examples (meson, cmake, whatever) and move around on runtimes (I dig with ostree into sdk.gnome.org with still to new for me to get it all) | 06:22 |
| *** tristan has joined #buildstream | 06:41 | |
| paulsherwood | albfan[m]: cool, thanks! | 06:52 |
| albfan[m] | paulsherwood: feedback welcome! (even examples request) | 07:00 |
| albfan[m] | most obvious is that use a local source is not buildstream aligned, PR with a git source would be accepted! | 07:01 |
| paulsherwood | albfan[m]: probably needs a link to builstream install, in case user doesn't have it | 07:08 |
| paulsherwood | perhaps needs a licence in the repo | 07:08 |
| albfan[m] | paulsherwood: See credits on README. But yes all should be extra clear, MR? | 07:09 |
| albfan[m] | Licence, you are right! | 07:09 |
| paulsherwood | albfan[m]: ack | 07:18 |
| *** bochecha_ has joined #buildstream | 07:29 | |
| tristan | just caught the backlog on the handy logger... | 07:44 |
| tristan | albfan[m], so I'd really like to start adding examples, the problem is that I need to establish precedent on an examples section such that examples are consistent (readability wise), that the information on those pages is not too redundant, and that they are reliable (which means: I'd like to literal include the bst fragments from a directory, and at least run `bst build` on them in CI) | 07:47 |
| *** bochecha_ has quit IRC | 07:47 | |
| *** dominic has joined #buildstream | 07:54 | |
| tristan | albfan[m], also note your repo still does the mistake we were doing back then, it imports the runtime twice redundantly | 07:58 |
| gitlab-br-bot | buildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 08:04 |
| *** toscalix has joined #buildstream | 08:07 | |
| tristan | albfan[m], https://gitlab.gnome.org/albfan/buildstream-flatpak-autools/merge_requests/1 | 08:14 |
| tristan | albfan[m], that fixes it, to only stage the SDK, and just setup some symlinks for usr merge | 08:14 |
| *** bethw has joined #buildstream | 08:27 | |
| *** aday has joined #buildstream | 08:36 | |
| *** jonathanmaw has joined #buildstream | 08:45 | |
| albfan[m] | tristan: Merged. I will remove the files/src/src extra dir. If you find time to write down how you want to build those examples (one repo, many repos) and how to document it (I can deal with sphinx syntax) | 08:50 |
| albfan[m] | I have some ideas a platform runtime and a binary tar, etc... | 08:51 |
| tristan | albfan[m], I want it *in buildstream* actually, I have headaches from the days of everyone making their own tutorials on different websites and bitrotting over the years, I dont want to repeat that mistake :) | 08:54 |
| albfan[m] | tristan: cool. maybe an examples under https://gitlab.com/BuildStream/buildstream/tree/master/doc? | 08:55 |
| tristan | albfan[m], for this specific example, I suppose it will use a specific ref from sdk.gnome.org, but for most examples I want to standardize on the alpine image we use for testing, and then change that image atomically for a freedesktop-sdk built base | 08:56 |
| albfan[m] | sure, the idea is to show how to interact with ostree, more than any other runtime | 08:56 |
| tristan | albfan[m], So an example I guess has 3 components; the example rst file... an examples directory with the project demonstrating the example, and a CI integration test ensuring it does what we're saying it does | 08:57 |
| tristan | I guess that the ${name}.bst, should correspond to examples/${name} directory and probably an entry in tests/integration/${name}.py | 08:58 |
| albfan[m] | ok, working on it | 08:59 |
| tristan | in order to display nice yaml fragments in rst, we can use this trick which we repeat a lot: https://gitlab.com/BuildStream/buildstream/blob/master/doc/source/projectconf.rst#L798 | 09:00 |
| tristan | albfan[m], probably yes we want an examples subdirectory in doc/source, so examples would be doc/source/examples/${name}.rst | 09:00 |
| *** tiago has joined #buildstream | 09:02 | |
| tristan | albfan[m], we had a "master plan" but lacked time to really work on it, for now adding an initial cleanly done example is a good start, we may chop it up later to avoid redundancy... i.e. we're thinking of having a "Getting started" section which comes before examples, and shows the simplest possible project, and moves on to building something on a real runtime, getting the user familiar with how to interpret and use variables in build elements, | 09:03 |
| tristan | so that they can make use of the reference material later | 09:03 |
| tristan | albfan[m], so we're hoping to keep this "Getting started" part minimal, add examples of "how to do specific things", and make the examples link to eachother rather than exhaustively redundantly explaining things | 09:03 |
| tristan | but I think this "Build an autotools element on a flatpak SDK" example is a great start | 09:04 |
| tristan | juergbi, valentind: Unfortunately, it seems that the version parsing is really working fine... I am seeing 0.1.7 and no --die-with-parent on debian 8, and 0.20 with --die-with-parent on fedora 27, and 0.21 with --die-with-parent on debian 9, in all cases the function resolves correctly | 09:09 |
| juergbi | tristan: on which system(s) did the issue occur? | 09:10 |
| tristan | different ones | 09:10 |
| tristan | I'm sure I've seen it fail on fedora, and at least one of the debians | 09:10 |
| juergbi | hm, so bwrap is used exactly the same way, no idea what's happening, then | 09:11 |
| tristan | it might not be related to that patch, it's just that I saw it first on one of the pipelines for that MR | 09:12 |
| gitlab-br-bot | buildstream: merge request (valentindavid/update_mirror->master: WIP: update mirror) #440 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/440 | 09:18 |
| gitlab-br-bot | buildstream: merge request (valentindavid/359_cross_junction_elements->master: Allow names for cross junction elements) #454 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/454 | 09:18 |
| albfan[m] | tristan: ack | 09:20 |
| tristan | albfan[m], thanks for looking at this ! | 09:21 |
| *** tristan has quit IRC | 09:33 | |
| albfan[m] | tristan: my pleasure, buildstream deserves it | 09:51 |
| awacheux | Hi! I'm currently working on the SourceTransform plugin feature. To implement it, I'm deriving the Source class to extend its behavior. However, I couldn't figure out a way to access the other Source plugin of a build element from the one of the Source plugin. This is required to stage the previous sources before tracking the SourceTransform. Is there a way to access those other Source plugins from one source plugin? | 09:54 |
| awacheux | Actually, I've implemented a `_is_source_transform()` method and I change the behavior of `Element._track` based on whether the source we are tracking is a `SourceTransform`. I'm not satisfied with this solution as it doesn't look neat to me. | 09:56 |
| toscalix | do we want to organise a bst workshop or similar during GUADEC? | 10:05 |
| toscalix | if so, has there been contacts with the GUADEC content committee already? | 10:05 |
| gitlab-br-bot | buildstream: merge request (juerg/no-remote-summaries->master: Drop use of OSTree summary files and dynamically plan build dependencies) #446 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/446 | 10:29 |
| *** tiago has quit IRC | 10:42 | |
| *** tiago has joined #buildstream | 10:42 | |
| *** tiago has quit IRC | 10:43 | |
| *** tiago has joined #buildstream | 10:43 | |
| *** tiago has quit IRC | 10:48 | |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 10:52 |
| gitlab-br-bot | buildstream: merge request (juerg/no-remote-summaries->master: WIP: Drop use of OSTree summary files and dynamically plan build dependencies) #446 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/446 | 10:57 |
| *** tristan has joined #buildstream | 11:21 | |
| tristan | toscalix, First they will select the talks, the schedule is supposed to be final today actually | 11:23 |
| tristan | toscalix, then, normally they will open up a wiki page for BoFs | 11:23 |
| tristan | once it opens up, we need to reserve a free spot | 11:23 |
| toscalix | ok | 11:24 |
| tristan | awacheux, For SourceTransform, the core certainly needs to provide help here, the semantics for tracking will be different for a SourceTransform as I understand | 11:24 |
| tristan | awacheux, for the check you mention, I think in this case isinstance(plugin, SourceTransform) would be fine | 11:24 |
| tristan | toscalix, actually I sent them an email today because I dont have access to the mailbox I normally use, I will have to sort that out next month I think in Canada | 11:26 |
| tristan | So I hope I didnt miss a window, but I sent them an email anyway saying that if they are waiting for some email confirmation from me, please let me know via this email address | 11:26 |
| *** aday has quit IRC | 11:31 | |
| *** aday has joined #buildstream | 11:32 | |
| gitlab-br-bot | buildstream: merge request (tristan/etag-mr-fixup->master: buildstream/plugins/sources/_downloadablefilesource.py: Store etag along with cache.) #458 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/458 | 11:36 |
| gitlab-br-bot | buildstream: merge request (valentindavid/etag->master: Store etag along with cache.) #441 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/441 | 11:38 |
| juergbi | oh, does anyone know why is Python list += horribly broken? | 11:41 |
| juergbi | it mutates the list and thus does not do the same thing as foo = foo + b | 11:41 |
| juergbi | incredibly unexpected | 11:42 |
| tristan | Whoa, I didnt know that no | 11:42 |
| jmac | Never noticed, I always use .extend() or .append() | 11:44 |
| tristan | I usually use that if I want mutation, and expect that other operations wont mutate indeed | 11:45 |
| tristan | although I am preferring comprehensions | 11:45 |
| tristan | it happened again, on debian 9: https://gitlab.com/BuildStream/buildstream/-/jobs/67522330 | 11:49 |
| tristan | where --die-with-parent is most certainly used | 11:49 |
| tristan | I am thinking of reverting that patch | 11:49 |
| tristan | I will make a branch with it reverted and run CI on it many times | 11:50 |
| *** bochecha_ has joined #buildstream | 11:52 | |
| *** zilbermanka has joined #buildstream | 11:57 | |
| *** zalipka has quit IRC | 11:59 | |
| gitlab-br-bot | buildstream: issue #377 ("Change how `etag` is stored for file download source plugins") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/377 | 12:03 |
| gitlab-br-bot | buildstream: merge request (tristan/etag-mr-fixup->master: buildstream/plugins/sources/_downloadablefilesource.py: Store etag along with cache.) #458 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/458 | 12:03 |
| *** tiago has joined #buildstream | 12:08 | |
| *** bethw has quit IRC | 12:09 | |
| gitlab-br-bot | buildstream: merge request (jennis/nest_workspaces_commands->master: Nest workspace commands in ToC) #456 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/456 | 12:11 |
| gitlab-br-bot | buildstream: merge request (tristan/workspace-soft-reset->master: Add soft reset functionality for workspaces) #459 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/459 | 12:26 |
| gitlab-br-bot | buildstream: merge request (chandan/workspace-soft-reset->master: Add soft reset functionality for workspaces) #457 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/457 | 12:26 |
| *** bethw has joined #buildstream | 12:32 | |
| juergbi | tristan: we originally decided that in non-strict mode we'll still pull strict artifacts if we already have an artifact in the local cache that only matches with the weak key | 12:37 |
| juergbi | however, it seems that this isn't properly working at least for a few months given that the planner filters out locally cached elements and thus they will never even enter the pull queue | 12:37 |
| juergbi | and it's causing problems in my no-remote-summaries branch | 12:38 |
| juergbi | furthermore, without summary files, we could avoid a lot of individual pull jobs if we reverted that decision | 12:38 |
| juergbi | so I'm wondering whether we should change the policy | 12:38 |
| juergbi | or whether to properly fix it | 12:39 |
| juergbi | it would simplify a few things | 12:39 |
| juergbi | (we would still attempt to download strict artifact before weak artifact but only if we don't even have a weak artifact in the local cache) | 12:40 |
| gitlab-br-bot | buildstream: issue #375 ("Provide a way to run configure-commands again for workspaced elements") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/375 | 12:42 |
| gitlab-br-bot | buildstream: merge request (tristan/workspace-soft-reset->master: Add soft reset functionality for workspaces) #459 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/459 | 12:42 |
| jmac | juergbi: I've just done a full fetch and build of freedesktop-sdk/bootstrap. In the whole build there were only two uses of a sandbox which didn't run commands and they were 20 seconds out of a 34 minute build. | 12:53 |
| jmac | Could you point me at the places we're likely to use the CAS-backed sandbox? | 12:53 |
| juergbi | jmac: e.g. compose and import plugins manipulate the sandbox directory outside of commands | 12:55 |
| jmac | Yes, but a compose plugin must run the integration commands of its dependencies | 12:57 |
| juergbi | yes, if there are any | 12:57 |
| tristan | juergbi, you want to *not* pull artifacts by their non-strict keys ? | 12:57 |
| tristan | oh no | 12:58 |
| juergbi | jmac: why are you looking for cases without commands? | 12:58 |
| tristan | you want to stop pulling the strong artifact if we have a weak one ? | 12:58 |
| juergbi | tristan: exactly | 12:58 |
| tristan | juergbi, that sounds less than desirable doesnt it ? | 12:58 |
| tristan | juergbi, dont we want to get the closest possible ? | 12:58 |
| jmac | juergbi: Because we can't use the CAS-backed virtual directory implementation if the sandbox needs to run commands | 12:58 |
| juergbi | tristan: I'm not sure | 12:58 |
| juergbi | tristan: in general, yes, this would be desirable | 12:59 |
| tristan | so, that's why we did that, right ? | 12:59 |
| tristan | I mean, it still makes sense to me | 12:59 |
| jmac | I don't know why import jobs don't show up in my logs, I'll have to check that | 12:59 |
| juergbi | tristan: yes, however, performance of small incremental builds take a big hit | 12:59 |
| juergbi | especially without summary files | 12:59 |
| juergbi | jmac: this is just for testing without FUSE? | 13:00 |
| jmac | juergbi: It's for the phase before we have the FUSE, yes | 13:00 |
| tristan | juergbi, without summary files, wouldnt we: A.) Try to pull the strong key... if that fails.. B.) Try to pull a weak one ? | 13:00 |
| juergbi | jmac: if you have compose in a project without integration commands, that could work | 13:00 |
| jmac | juergbi: Right, but I'd like some evidence that it actually helps... I need to find a project in which that's the case | 13:01 |
| juergbi | tristan: yes, but it means that we'll pull tons of elements in every session even if nothing changed | 13:01 |
| tristan | juergbi, and then of course fallback to doing a local build... but isnt that preferable performance wise than building locally ? | 13:01 |
| juergbi | or rather, attempt to | 13:01 |
| juergbi | tristan: it's not about pulling vs. local build, it's about pulling vs. nothing | 13:02 |
| juergbi | if we already have a weak artifact locally | 13:02 |
| tristan | I think I have to reboot | 13:02 |
| * tristan will be back in some minutes... | 13:02 | |
| *** tristan has quit IRC | 13:03 | |
| jmac | juergbi: I'll take a look around some other example projects and see what I can find | 13:03 |
| juergbi | jmac: not sure whether it makes too much sense to test performance of CAS-backed virtual directories without FUSE | 13:03 |
| juergbi | most time will typically be spent in regular element builds | 13:03 |
| juergbi | for that we'd need to test the combination of CAS-backed virtual directory and FUSE | 13:03 |
| jmac | juergbi: My point is, once we have FUSE we may as well just use the existing file-based directory system backed onto the FUSE | 13:04 |
| juergbi | ah, no, with remote execution at some point we want to avoid the need to have complete artifacts locally | 13:04 |
| juergbi | so we want to be able to manipulate Merkle trees without having the corresponding files present | 13:04 |
| juergbi | well, it might also be possible with FUSE | 13:05 |
| juergbi | depends on some details | 13:06 |
| jmac | I thought all the Merkle tree data would be local in this implementation | 13:06 |
| *** xjuan has joined #buildstream | 13:06 | |
| juergbi | the Merkle trees themselves will most likely always be local | 13:06 |
| juergbi | but we'd like to not require having all referenced files locally | 13:06 |
| jmac | So you are manipulating Merkle trees, just through a filesystem-style interface | 13:07 |
| jmac | Doing in a move/copy/link through the FUSE wouldn't need you to have the files | 13:07 |
| * paulsherwood wishes he had more time to dig into this... it feels like we're making it harder than it needs to be | 13:07 | |
| jmac | paulsherwood: I'm making sure here that it's necessary to do a bit of work. I'd much rather we added less code if possible. But it may be necessary. | 13:08 |
| paulsherwood | jmac: ack | 13:09 |
| paulsherwood | i'm probably wrong.. but in the baserock implementation we only cared about git repos... so 'Merkle trees' didn't need to be thought about because we could rely on git for the magic | 13:10 |
| paulsherwood | i appreciate that here folks are trying to keep working outside git repos, but i do wonder if it's really worth the bother | 13:11 |
| paulsherwood | (and again.. i may be completely confused... but for me... a remote git repo already has all the hashes i would need... i don't need to recover all the files to find them out) | 13:12 |
| paulsherwood | sorry if this is completely irrelevant again | 13:12 |
| *** tristan has joined #buildstream | 13:13 | |
| jmac | paulsherwood: If you're on the BuildStream mailing list, juergbi's post called "Proposal for Remote Execution" may be useful for background | 13:13 |
| tristan | annoying, GNOME Shell has a bug where the battery indicator stops working and gets confused - impossible to tell if my plug is properly connected | 13:14 |
| paulsherwood | i did read it. i didn't understand why we woudl have to copy a tmp dir rather than just move it in place, so assumed it was beyond me | 13:14 |
| tristan | juergbi, ok so I think I am catching on... | 13:14 |
| tristan | juergbi, basically you are saying... we keep trying to download a strong one when we have a usable one available, and depending on network, we spend a variable amount of time finding out that a strong one is "still not available remotely" | 13:15 |
| tristan | juergbi, I have two impressions... first is, why is it taking so long just to find out if there is a ref ? surely this could be done in a single round trip | 13:16 |
| tristan | juergbi, my second impression is... yeah if we want to change that behavior, then I think we will want two policies | 13:16 |
| Nexus | tristan: on https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_frontend/cli.py#L224 Doesn't track_save need to be in the correct order to be assigned correctly? | 13:17 |
| tristan | juergbi, i.e. if you run `bst --no-strict build ...`, maybe you want that because network performance might suck beyond regular expectations... | 13:17 |
| jmac | paulsherwood: I don't know which tmp dir you're referring to there | 13:17 |
| tristan | juergbi, but what if you are working in non-strict mode, and you explicitly call `bst pull` ? I feel like in that case, you would want to try to get the more accurate strong artifacts no ? | 13:17 |
| juergbi | paulsherwood: this is about the sandbox root (sysroot), not related to git/source trees | 13:18 |
| tristan | Nexus, for stylistic purposes, I think I prefer the correct order yes; in python with those decorators technically speaking: not at all | 13:18 |
| tristan | Nexus, the decorator knows what arg name it will assign, if you rename "track_save" to "track_pony", then it will cause a runtime error because a "track_save" argument could not be found | 13:19 |
| juergbi | tristan: yes, of course, the behavior in non-strict mode is anyway clear | 13:20 |
| juergbi | as there are never multiple artifact candidates in that case | 13:20 |
| juergbi | ah, you mean pull in non-strict mode, misread that | 13:20 |
| juergbi | that could make sense | 13:21 |
| juergbi | and that's probably the case where it currently works properly | 13:21 |
| juergbi | for bst build it doesn't currently work properly (and hasn't in months, if ever, afaict) | 13:21 |
| tristan | I'm still a bit baffled why the performance seems to be so bad, though | 13:21 |
| tristan | juergbi, one thing however is certain... we have a semantic problem with "retries" | 13:21 |
| juergbi | maybe the performance is not as much of an issue | 13:23 |
| juergbi | we don't have persistent HTTP connections at the moment but it should still be just a single HTTP request | 13:23 |
| tristan | juergbi, I think I failed to file a bug about that, but basically if we have a retrying task, the job currently has no way to inform the core "Does it make sense to retry this job ?", i.e.... I tried to download a file, and I successfully contacted the server, who said the file doesnt exist... or I successfully downloaded the file, and failed because the checksum was incorrect | 13:23 |
| juergbi | so maybe it's ok | 13:23 |
| tristan | like, if you download a file and assert it's incorrect, please dont go and download it 2 more times, that's silly | 13:23 |
| juergbi | hm, I thought we're handling this properly already | 13:24 |
| juergbi | does not exist => no failure but returns False | 13:24 |
| juergbi | I don't see any retries on non-existing pulls here | 13:24 |
| tristan | Ok that is good | 13:24 |
| tristan | I think there is a case where it's not handled well | 13:24 |
| tristan | will have to find it | 13:24 |
| juergbi | ok, so let's keep the policy for now and let's properly fix it. change it later if necessary | 13:25 |
| juergbi | this does mean we need slightly different plans for bst fetch and bst build, though | 13:25 |
| juergbi | assuming we don't want to fetch sources for cached elements | 13:25 |
| tristan | Right, that is an interesting point | 13:26 |
| juergbi | or use the same plan but use skip_cached=True for bst fetch -d plan | 13:26 |
| tristan | the default for `bst fetch` is `--deps plan` | 13:26 |
| juergbi | in FetchQueue | 13:26 |
| tristan | which means we would also want to skip downloading sources for elements which are not cached, but never need to be locally cached | 13:26 |
| tristan | juergbi, so remind me again the rationale for non-recursive Element._get_full_name(), I think it was because we dont want duplicate elements from the same sub-sub projects appearing | 13:28 |
| juergbi | we'd need some other canonicalization otherwise | 13:29 |
| tristan | juergbi, right now this is causing headache for addressing of junctioned elements; ideally I think we would want symmetry here | 13:29 |
| juergbi | given that we coalesce subprojects based on junction name | 13:29 |
| tristan | i.e. if I want to open a workspace, or track a junctioned element, that is fine `bst track junction.bst:element.bst` | 13:29 |
| tristan | but what if I want to track a sub-sub-project element ? | 13:29 |
| tristan | two different naming schemes would be bad | 13:30 |
| juergbi | hm, the issue being that a sub-junction name can't be found because it's not loaded yet? | 13:31 |
| tristan | Basically yeah | 13:32 |
| juergbi | I don't think there is any other option for addressing than specifying the full path | 13:33 |
| tristan | I guess recursively loading *everything* first before any addressing begins is a kinda-sorta option | 13:34 |
| tristan | but it's heavy duty | 13:34 |
| juergbi | loading all .bst files, that would be a big change | 13:34 |
| tristan | yeah it would, I dont like it | 13:34 |
| tristan | but also I dont like that addressing an element is different from how it appears, I also dont like that sub-sub-sub-project element names are too big, though | 13:35 |
| juergbi | we could change the output such that it's recursive in the case that the junction is indeed specific to a particular subproject and not shared | 13:36 |
| tristan | How is a junction shared ? | 13:37 |
| * tristan scratches head, not sure he gets how that works | 13:37 | |
| juergbi | if you have two junctions with the same name in two different subprojects, they are merged | 13:37 |
| juergbi | buildstream requires that you have a junction with the same name in a common ancestor of these subprojects | 13:37 |
| tristan | Aha ! special feature | 13:38 |
| juergbi | as that decides which junction config to use | 13:38 |
| tristan | We have test coverage for this ? | 13:38 |
| juergbi | yes, should be covered | 13:38 |
| juergbi | I think I mentioned a while ago that we might want to make this more configurable, i.e., also allow multiple junctions of the same name without merging but let's ignore that for now | 13:39 |
| tristan | "BuildStream uses the junction element name as key to determine which junctions to merge. It is recommended that the name of a junction is set to the same as the name of the linked project." | 13:39 |
| tristan | http://buildstream.gitlab.io/buildstream/elements/junction.html#module-elements.junction | 13:39 |
| juergbi | yep | 13:39 |
| tristan | valentind, Hmmmm, ok so I've got the story straight - still not sure what is the best thing to *do* | 13:41 |
| tristan | valentind, the problem is that for sub-sub-project elements, they are encouraged to not be repeated, although contrary to what I thought in the comment, they are not disallowed | 13:42 |
| tristan | i.e. it is possible to have two separate junctions which refer to the same project, even at different versions; but the diamond sibling projects which reffer to a common dependency should use the same junction name to avoid mismatches | 13:42 |
| tristan | and the parent project should provide a junction for that sub-sub-project in order to dictate terms, if I understand correctly (i.e. if two sibling projects disagree about which ref to use for the sub-sub project, the parent should decide) | 13:43 |
| tristan | valentind, so this explains that, there can be multiple paths to qualify the same junction, in fact | 13:44 |
| juergbi | it's enforced even if the refs were identical | 13:44 |
| tristan | Ok, so parent *must* provide one to decide, that is preferable, even | 13:44 |
| tlater | Did we ever show progress updates for fetch operations or is that my imagination? | 13:48 |
| juergbi | depends on the source, iirc | 13:49 |
| tlater | Ah, right, git probably would be a bit hard. | 13:49 |
| tristan | tlater, yeah, I've been wanting to remove them for `ostree` for the longest time | 13:49 |
| tristan | they are A.) obnoxiously a lot of output and B.) completely incorrect anyway (percentage reported by ostree jumps up and down while it discovers more objects it wants to download) | 13:50 |
| tlater | IMO they would still be nice if they were accurate enough | 13:50 |
| tristan | progress bars in the status area would be cool, sstriker asked for that at one point | 13:50 |
| tristan | then it might be worth doing for git as well | 13:51 |
| tristan | and some build systems can even contribute to progress bars, that would be very tricky to implement though (e.g. CMake does, I think meson + ninja does) | 13:51 |
| tlater | A progress-bar API for plugins! :) | 13:52 |
| tristan | tlater, and an implementation in _frontend/status.py yeah | 13:55 |
| valentind | tristan, But the name of the junction not the project is in the path. I do not see how you get a diamond. | 13:56 |
| valentind | tristan, is it possible to give path of junctions to an element that is ambiguous? | 13:56 |
| tristan | valentind, lets pretend that gnome-build-meta depends on two separate projects, and both of those depend on the bootstrap project | 13:56 |
| valentind | Or from an element, is it possible to have several valid paths? | 13:56 |
| tristan | to refer to elements in the bootstrap project, you can address it via the freedesktop "sdk" project, or the other one | 13:57 |
| valentind | tristan, in my opinion if two sub projects depend on a same project, for buildstream, they should be considered as different. | 13:57 |
| tristan | valentind, they are absolutely not, that is very intentional | 13:57 |
| tristan | valentind, and there are good use cases for that to be supported, too; and enforced that they use the same sources | 13:58 |
| valentind | But if they have different revisions or different options? | 13:58 |
| valentind | How can you enforce two subsubprojects are actually the same? | 13:59 |
| tristan | valentind, it is recommended to use the project name as the junction name, and BuildStream is documented to merge them on that junction name | 13:59 |
| tristan | valentind, even if the parent project does *not* directly use that junction, it must define it, in order to ensure the sub-sub project gets the correct ref/configuration | 14:00 |
| tristan | valentind, in fact, this is all in the immediate backlog here :) | 14:00 |
| tristan | and documented here: http://buildstream.gitlab.io/buildstream/elements/junction.html#module-elements.junction | 14:00 |
| tristan | there are cases where you intentionally want a sub-sub project to be separate, but it is usually more desirable to enforce that they are exactly the same, and produce the same artifact | 14:01 |
| valentind | tristan, So is there an error if a conflict is detected? | 14:02 |
| valentind | tristan, The problem here is that for "workspace open", I do not see a way to discover the element unless the top project redefines all the junctions. | 14:03 |
| valentind | tristan, I would have thought that buildstream had enough information to detect which elements are the same and use the same artifacts. | 14:04 |
| valentind | tristan, could we fail, or warn if we detect an subsubproject has no junction at top level? No matter whether there are conflicts or not. | 14:07 |
| valentind | The documentation says you should do this to solve conflicts. It is not user friendly. Asking the user to fix those in the beginning instead of behaving in a surprising way, would be better. | 14:08 |
| gitlab-br-bot | buildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 14:13 |
| tristan | valentind, If you happened to define the same sub-sub project elements with exactly the same configuration, they would have the same artifact cache key, and those artifacts would be reused indeed | 14:15 |
| tristan | valentind, coalescing on junction name is an additional feature to *ensure* that you dont mistakenly have different configurations | 14:16 |
| tristan | still, the fact that those elements are the same still means we need to display them as the same, there is the uniqueness of the artifact, and there is also the uniqueness of how it is displayed in logs and loaded in the pipeline | 14:17 |
| tristan | i.e. we could end up loading two times those elements, which would be undesirable; as such _get_full_name() shows the expected thing | 14:18 |
| tristan | that doesnt solve the addressing part for loading purposes, but it does explain the *why* | 14:19 |
| tristan | juergbi, valentind; if we're going to have to have different ways to address and display cross junction elements, I think the bigger question here is how (and where) do we express this in documentation | 14:22 |
| valentind | I think it makes sense to have the input to also use this _get_full_name() given the way it works. | 14:23 |
| tristan | and if we can avoid it, the only solution I can think of is to recursively load everything | 14:23 |
| tristan | valentind, I agree that that is more desirable | 14:23 |
| tristan | but finding it is tricky | 14:23 |
| tristan | it means that if there are many junctions, all of those junctions need to be tracked and fetched before any addressing of cross junction elements can occur | 14:23 |
| valentind | Well, if you have not redeclared junctions on top level, you cannot open a workspace on subsub projects. | 14:24 |
| valentind | That is fine behavior. | 14:24 |
| tristan | Ah | 14:24 |
| tristan | I see what you mean | 14:24 |
| tristan | I hadn't thought of it that way | 14:24 |
| valentind | I think I should also limit the parsing. | 14:24 |
| valentind | Just so that everything works the same. | 14:24 |
| tristan | it should | 14:24 |
| tristan | indeed | 14:25 |
| valentind | But there should be a plan to warn users when we detect subsub projects not declared on top level. | 14:25 |
| tristan | juergbi, just to be extra clear... *if* you have a junction element of the same name in a parent project, which is used in a sub-project, that junction element overrides the subproject junction, correct ? | 14:25 |
| tristan | juergbi, i.e. *even if* there are no diamonds ? | 14:25 |
| juergbi | yes | 14:26 |
| valentind | tristan, the thing is that the user might not even know if there is a diamond. | 14:26 |
| tristan | valentind, I agree | 14:26 |
| juergbi | essentially the top-most version wins | 14:26 |
| juergbi | if there is a conflict, report an error | 14:26 |
| juergbi | conflict as in, there is no single top-most version | 14:26 |
| tristan | valentind, maybe not a warning | 14:26 |
| tristan | valentind, maybe we need to rather enhance how we display what was loaded in the heading of a master build log | 14:27 |
| tristan | such that it is more clear what is the shape of depending projects | 14:27 |
| valentind | Also you have the case where you update a subproject, and then suddenly there is a diamond that appear. | 14:28 |
| tristan | valentind, the problem I see with a warning is that, it means something is wrong... and if for example, you are only writing a flatpak, and you depend on a runtime, and that runtime uses subprojects, etc... then basically the topmost project should not have to care | 14:29 |
| valentind | Because the new version uses a same subsubproject as another subproject. | 14:29 |
| *** tristan has quit IRC | 15:00 | |
| *** tristan has joined #buildstream | 15:27 | |
| *** noisecell has quit IRC | 15:30 | |
| tristan | so far I've run the reverted bwrap version checks a bunch of times without a failure (there were failures but not of that variety, rather typical, expected rebase and failed to get the git ref failures) | 15:36 |
| tristan | and I've seen the spurious failure occur twice (once for each of two other merge requests, which I merged via "Retry") | 15:37 |
| tristan | I'm running three more pipelines on the revert, I know it's not a "real fix", but would rather revert and get it out of the way for now | 15:38 |
| * tristan will run more pipelines over the night and see tomorrow | 15:38 | |
| tristan | lets run a couple on master too and see if the spurious error wants to rear it's ugly head | 15:38 |
| * jmac pushes his branch again to see if the unexplained errors go away | 15:41 | |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 15:45 |
| tristan | jmac, right, I havent merged the revert... but you might try a rebase against `tristan/revert-bwrap-checks` if you want to try your code against the revert for now | 15:48 |
| jmac | Ah, ok | 15:49 |
| valentind | For freedesktop-sdk, I would like to be able to depend to elements with a sysroot. When the element's artifact is staged as runtime, nothing change, it goes in /. But when it is staged as build dependency, then I want it to go in the specific directory. | 15:53 |
| tristan | valentind, not sure exactly what you're after, but staging artifacts at specific locations is doable with `script` element | 15:54 |
| valentind | tristan, script is problematic | 15:54 |
| tristan | valentind, there is also a base class for more specialized `script` elements | 15:54 |
| valentind | Because we lose the splits. | 15:54 |
| valentind | So to keep splits we have to make several images an rebuild it after. | 15:55 |
| valentind | And also, I might want to depend only on specific subset of elements, then I have to create a script for that subset. | 15:55 |
| tristan | it's late for me (1am), and I have take-out to eat, this sounds more involved | 15:56 |
| valentind | Or one script per element. | 15:56 |
| valentind | So (number of elements)*(number of splits) scripts. | 15:56 |
| tristan | That sounds like what the "filter" element is actually for (subsets of elements) | 15:56 |
| valentind | tristan, we can talk about it tomorrow. | 15:56 |
| valentind | Not urgent. | 15:56 |
| tristan | :) | 15:57 |
| valentind | Bon appétit. | 15:57 |
| tristan | Sure, can also always use the ML, fwiw I'm more interested in exactly "The practical use case and thing you want to achieve" | 15:57 |
| valentind | OK | 15:58 |
| gitlab-br-bot | buildstream: merge request (valentindavid/359_cross_junction_elements->master: Allow names for cross junction elements) #454 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/454 | 16:03 |
| *** noisecell has joined #buildstream | 16:24 | |
| *** bethw has quit IRC | 16:37 | |
| *** aday has quit IRC | 16:41 | |
| *** aday has joined #buildstream | 16:41 | |
| *** aday has quit IRC | 16:45 | |
| *** dominic has quit IRC | 16:45 | |
| *** finn has quit IRC | 16:50 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:56 |
| *** jonathanmaw has quit IRC | 17:15 | |
| *** noisecell has quit IRC | 17:16 | |
| *** aday has joined #buildstream | 17:21 | |
| *** aday has quit IRC | 17:25 | |
| *** toscalix has quit IRC | 17:32 | |
| *** bochecha_ has quit IRC | 17:58 | |
| *** aday has joined #buildstream | 18:51 | |
| *** aday has quit IRC | 18:55 | |
| *** jsgrant has joined #buildstream | 20:00 | |
| *** tristan has quit IRC | 20:02 | |
| *** xjuan has quit IRC | 20:42 | |
| *** xjuan has joined #buildstream | 21:05 | |
| *** xjuan has quit IRC | 22:07 | |
| *** cs_shadow has quit IRC | 23:01 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!