| *** mohan43u has joined #buildstream | 02:50 | |
| *** tristan has quit IRC | 05:08 | |
| *** hasebastian has joined #buildstream | 05:22 | |
| *** tristan has joined #buildstream | 05:53 | |
| *** ChanServ sets mode: +o tristan | 05:53 | |
| tristan | WSalmon, There are no API guarantees on 1.9x whatsoever - I don't know why we ever jumped from 1.91 to 1.93, but I suppose it may have represented a milestone of sorts and people wanted to bump the minor number for some reason | 05:55 |
|---|---|---|
| tristan | juergbi, do we have a tag yet, and if you want me to bump it, what should I do ? do we need a NEWS update with that ? anything else ? | 05:56 |
| tristan | s/bump it/tag it/ | 05:56 |
| juergbi | tristan: we don't have a tag yet. should just need a NEWS update and tag | 05:58 |
| juergbi | we haven't formally announced 1.9x releases | 05:58 |
| juergbi | s/releases/snapshots/ | 05:59 |
| tristan | juergbi, So just put the next number instead of "(unreleased)" in the NEWS, and let the next (post-tagging) NEWS update to add a new "(unreleased)" section ? | 06:08 |
| tristan | 1.93.4 I think | 06:09 |
| juergbi | tristan: yes, and maybe a quick check if we've missed an important change | 06:09 |
| tristan | Is there any reason why we ever did 1.93 ? | 06:09 |
| tristan | Or just marketing nonsense | 06:09 |
| juergbi | someone wanted this, let me check | 06:09 |
| tristan | yeah, that's my guess, someone wanted to say "Yay we now have 1.93 ! and that is > 1.91 !" | 06:10 |
| tristan | well, not sure but it often is the case ;-) | 06:10 |
| juergbi | !1804 is the MR but there was no discussion about it | 06:11 |
| juergbi | also don't see anything in the IRC logs | 06:11 |
| juergbi | would have to ask cs-shadow | 06:11 |
| tristan | juergbi, I think we will want to tag next week again after landing the juncle anyway, not sure if we need to wait on any "important things to land" before the tag | 06:12 |
| tristan | rather, it's less sweat off our back if we consider these tags as cheap | 06:12 |
| juergbi | with "check if we've missed an important change" I meant the NEWS file | 06:12 |
| tristan | Do we incur externalized work ? | 06:12 |
| tristan | Ah I understand what you mean now | 06:12 |
| * tristan hopes that people dont feel obligated to do a pip dev release for every single tag we roll | 06:13 | |
| juergbi | but I think we have the main ones | 06:13 |
| juergbi | well, there probably should be one but ideally, it would be automated | 06:13 |
| juergbi | it doesn't make sense to have pip dev releases but leave them outdated | 06:14 |
| tristan | I didnt put any entry for full paths | 06:14 |
| juergbi | although I suppose if we indeed tag another snapshot next week, we can defer the pip dev release | 06:14 |
| tristan | My thoughts on release dances are that for dev releases, let people do their dances if they want | 06:14 |
| tristan | If people start wanting a fresh pip release they can roll one | 06:15 |
| tristan | When it comes to stable we'll be much more conservative about rolling releases and the ripple effects that incurs | 06:15 |
| juergbi | sure, it's definitely more important for stable releases | 06:15 |
| juergbi | my point was simply that it would be good to be consistent | 06:15 |
| juergbi | if we don't need pip dev releases, we shouldn't have any | 06:16 |
| tristan | yeah if it could be automated that would be nice | 06:16 |
| juergbi | I think cs-shadow has been keeping those up-to-date so far | 06:16 |
| juergbi | don't we already have it automated for bst-plugins-experimental? | 06:16 |
| tristan | But I think it's more important that dev tags be *cheap* | 06:16 |
| juergbi | so should really do it for core as well | 06:16 |
| juergbi | sure, we shouldn't block the tag on anything extra | 06:17 |
| tristan | We should be able to push one in 5 minutes time, everyone be happy, and get on with our lives without incurring any work | 06:17 |
| juergbi | yes | 06:17 |
| tristan | I believe the automation is blocked on how our gitlab is setup, there is a possibility of leaking credentials | 06:18 |
| tristan | because of our "anyone can request dev status" policy | 06:18 |
| tristan | and all that | 06:18 |
| juergbi | tristan: aren't variables hidden from regular devs? | 06:23 |
| juergbi | I wouldn't expect anyone except maintainer/owner to have access | 06:23 |
| juergbi | and variables can be limited to protected branches | 06:24 |
| juergbi | yes, only maintainers and owners, afaict: https://docs.gitlab.com/ee/user/permissions.html | 06:25 |
| tristan | juergbi, I think that I'm able to create a merge request which runs CI on my branch, echoing protected variables into files and such, allowing me to observe these through the artifacts this pipeline creates | 06:38 |
| juergbi | tristan: there is a 'protect variable' flag: "Export variable to pipelines running on protected branches and tags only." | 06:39 |
| juergbi | so they would have to merge it | 06:39 |
| tristan | Hmm | 06:40 |
| tristan | Ok, I think this was the concern before | 06:40 |
| tristan | as I recall, it's possible gitlab changed since then, or that we didn't fully investigate the options | 06:40 |
| tristan | I'm adding some highlights to the NEWS | 06:41 |
| tristan | cross junction plugin origin is a big one | 06:41 |
| tristan | I don't think we've missed any breaking changes | 06:42 |
| tristan | And now DownloadableFileSource is public | 06:47 |
| tristan | juergbi, WSalmon ... NEWS updated, 1.93.4 is now tagged | 06:51 |
| juergbi | ta | 06:52 |
| tristan | juergbi, On the juncle... what do you think about duplicate validation ? | 06:52 |
| tristan | I think we should arguably not have it, as it would induce unnecessary subproject fetches | 06:53 |
| tristan | However there is another tangentially related issue, which is links | 06:53 |
| juergbi | yes, I don't think we want to do anything with listed junctions that are not in the current pipeline | 06:53 |
| tristan | juergbi, I feel that one should be able to have a link to a subproject junction, as this empowers the link element | 06:54 |
| juergbi | duplicates should only be specified for the actual junctions, not links, right? | 06:54 |
| juergbi | don't we already have that? | 06:54 |
| *** hasebastian has quit IRC | 06:54 | |
| tristan | i.e. I may want to refer to a link, in my local project or in a subproject, as a duplicate | 06:54 |
| tristan | The link itself is not the real full subproject path | 06:54 |
| tristan | And resolving the link requires loading it | 06:54 |
| tristan | I.e. the strength of a link is that I can use it as an alias, and possibly have conditional statements in the link resolving what it points to | 06:55 |
| tristan | I should not have to duplicate those conditional statements in project.conf | 06:56 |
| juergbi | what if the link points to different projects of different names depending on a conditional? | 06:56 |
| tristan | exactly | 06:56 |
| juergbi | you wouldn't have to duplicate the conditional statements, though. you could unconditionally list all potential targets | 06:57 |
| juergbi | may still not be ideal but it might not be that bad | 06:57 |
| tristan | juergbi, What I'm saying is that we have these two opposing forces... (A) We don't want to load projects we don't need to, (B) We should be able to use links in the duplicate lists | 06:57 |
| juergbi | I think I would defer link support in duplicate config until we see a use case where it's really useful | 06:58 |
| juergbi | i.e., worth the trouble, not just saving a line or so | 06:58 |
| tristan | So at first, document that only a true full path to a subproject can be used | 06:58 |
| juergbi | yes, unless you already see this blocking real use cases | 06:59 |
| tristan | I see the API in general as flawed without this | 06:59 |
| tristan | A link purports to be usable as an alias, but cannot be used in this case, it's weird | 06:59 |
| tristan | juergbi, whether or not there is a conditional statement, I can see it being practical from a project's API evolutionary standpoint | 07:00 |
| juergbi | it's not ideal, however, the whole duplicate configuration is mainly about special cases | 07:00 |
| tristan | I.e. if I have a public API for downstream projects, and I have a public subproject | 07:00 |
| juergbi | ideally, projects don't even need duplicate config | 07:00 |
| tristan | I may one day want to replace that subproject with something else which provides essentially the same thing | 07:01 |
| tristan | So I would want to have the ability to replace my public subproject junction with a link | 07:01 |
| tristan | If a reverse dependency project is broken by this, then a link cannot really be used here | 07:01 |
| juergbi | do we already have a use case where this can't be solved with internal junction? | 07:01 |
| tristan | I have listed at least one in my mail which discussed use cases | 07:02 |
| tristan | For instance, what if I'm regularly doing CI of images of fdsdk, this is an interesting use case to me | 07:02 |
| tristan | I build the entire fdsdk, and then I stage that on an fdsdk + additional tooling used to create a bootable image | 07:03 |
| tristan | I don't really want to rebuild that tooling just because I want to test my "latest", my deployment tooling doesnt need to be rebuilt at all, I can use last year's fdsdk until such a time my deployment tooling needs to change | 07:04 |
| tristan | This might be possible (with "internal") in the case of fdsdk doing CI on itself, indeed | 07:05 |
| tristan | I'm not sure that it will always be the case with duplicates | 07:05 |
| tristan | I mean with projects | 07:05 |
| tristan | If I'm doing this with GNOME, I could have an internal fdsdk for the deployment tooling and a primary dependency on fdsdk for my payload | 07:06 |
| tristan | that is true | 07:06 |
| juergbi | not sure how to handle this best | 07:06 |
| tristan | To be honest I think that duplicates covers all the grounds; and as such I'm writing it up first; while "internal" is a nice to have feature on top of that, which doesn't necessarily cover all grounds | 07:07 |
| tristan | Also, the 'link' problem is not necessarily unique to 'duplicates' | 07:08 |
| tristan | I.e. if I have a chain of projects which depend on eachother in a stack, and each project depends on an internal project for a compiler, it can make good sense to want to derive this internal junction from the subproject | 07:09 |
| tristan | in which case it makes good sense to provide a link | 07:09 |
| juergbi | tristan: true, I agree that it would make sense to resolve links for internal | 07:22 |
| juergbi | can't we do this lazily, though? i.e., only if that link is actually used | 07:22 |
| tristan | juergbi, that's the problem :-S | 07:24 |
| tristan | You cant know what it is until you resolve it | 07:24 |
| juergbi | so the issue you see is if the target is accessed without the link? | 07:25 |
| tristan | The core runs like this: When a loader is loaded, it checks a global context of loaded loaders | 07:25 |
| tristan | originally just a table of project names, now with some extra stuff marking junction names as internal etc | 07:25 |
| tristan | From that loading perspective, we can know what the full path of the junction being used is, and we can know what the project relative path is for each project up the hierarchy | 07:26 |
| tristan | Then we need to ask those projects if there are any whitelisting (duplicates or internal) | 07:26 |
| tristan | We ask them with knowledge of what the project relative junction paths are | 07:26 |
| tristan | But if they provide any links, those need to be pre-resolved to know if they point to this project relative path | 07:27 |
| tristan | Well wait a sec | 07:28 |
| tristan | "Can we do it lazily", this is a good question | 07:28 |
| tristan | I think the answer is "probably", which is not good enough | 07:29 |
| tristan | I.e. to do so, would be to assume that just because a link exists, it will be used to access a subproject | 07:29 |
| *** santi has joined #buildstream | 08:46 | |
| *** chipb_ has joined #buildstream | 09:00 | |
| *** chipb has quit IRC | 09:00 | |
| tristan | Does python code in cython files behave subtly differently ? | 10:24 |
| tristan | For instance, with regards to instance variables initialized in __init__ to mutable defaults | 10:25 |
| tristan | (like empty lists) | 10:25 |
| WSalmon | tristan, the thing i always get wrong is that i havent recompiled it, but benschubert is our expert | 10:26 |
| WSalmon | IIRC there are a lot of doc's on the subject | 10:26 |
| WSalmon | but I havent looked at them in a while | 10:27 |
| tristan | Nah, it seems that I've got a logic error | 10:29 |
| tristan | Swimming in junction duplicates, not very easy | 10:29 |
| tristan | Ok I see what I'm getting wrong | 10:31 |
| tristan | So now my error message says "Bla bla bla conflicting junctions bla..." and the detail has a line for line with each place a project was referenced, including the full path of it's junction and the provenance responsible for initially referring to that junction | 10:32 |
| tristan | But when a project is not completely duplicated, I suppose it would be helpful to, instead of just mentioning that it was "(duplicated)", also mention the provenance from the project.conf and project-name/junction full path of the project which marked it as a duplicate | 10:33 |
| *** hasebastian has joined #buildstream | 10:46 | |
| tristan | juergbi, thinking back, I'm not sure, maybe lazy link resolution in duplicate whitelisting can make sense | 10:57 |
| tristan | juergbi, We'd have to say that links can be specified in the duplicate whitelist, but they must be the means which is used by said project to lookup the junction/subproject | 10:58 |
| tristan | https://bpa.st/4EZQ <-- how's this error message, when the toplevel duplicates a subproject only once, but junctions it twice ? | 11:02 |
| tristan | juergbi, fwiw I have an initial implementation of "duplicates" on !1901, it doesnt support links and still needs documentation, but testing is fairly thorough | 11:22 |
| * tristan should be able to finish up this weekend | 11:22 | |
| tristan | I'm ordering the branch such that "duplicates" comes after replacing coalescing with "overrides" and adding initial conflict tests | 11:23 |
| tristan | So there is a gap where the conflict detection is done with a simple table and there is no tolerance for duplicate projects | 11:24 |
| tristan | since the changes are rather significant, I think it's worthwhile to keep these separate in the history | 11:24 |
| * tristan now has to solve this insane error from picking of jobs https://gitlab.com/BuildStream/buildstream/-/jobs/589968432 | 11:25 | |
| traveltissues | can someone please take a look https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/120 | 11:33 |
| coldtom | LGTM traveltissues | 11:35 |
| traveltissues | thanks | 11:36 |
| tristan | looks to me that jobpickler.py should not really be in _scheduler/jobs, this is not a Job implementation at all | 11:52 |
| valentind | tristan, Do you know if bst 2 allows cycle in runtime dependencies? If so, will that be fixed in bst 1.6? | 12:09 |
| tristan | valentind, what does that mean ? | 12:10 |
| tristan | circular dependencies cannot be allowed, I don't see how it could be possible | 12:10 |
| tristan | but something else ? | 12:10 |
| valentind | Sometimes the build dependencies are inverted from the runtime dependencies. | 12:10 |
| tristan | Inverted | 12:11 |
| * tristan not following | 12:11 | |
| tristan | Do we have a test case ? | 12:11 |
| valentind | OK, I will open an issue when I have a nice case. | 12:13 |
| tristan | Thanks, I'm sure you must be talking about a deep subtle bug, I'm just having a hard time understanding what it is :) | 12:13 |
| *** tristan has quit IRC | 12:18 | |
| *** hasebastian has quit IRC | 12:26 | |
| *** phildawson_ has joined #buildstream | 14:56 | |
| *** phildawson_ has quit IRC | 16:57 | |
| *** tristan has joined #buildstream | 16:59 | |
| *** ChanServ sets mode: +o tristan | 16:59 | |
| *** santi has quit IRC | 17:28 | |
| *** xjuan has joined #buildstream | 20:27 | |
| *** chipb_ is now known as chipb | 22:50 | |
| *** chipb has joined #buildstream | 22:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!