| *** SweetSense has joined #buildstream | 01:34 | |
| *** SweetSense has quit IRC | 01:37 | |
| *** CandyWendy has joined #buildstream | 06:46 | |
| *** CandyWendy has quit IRC | 06:49 | |
| *** mohan43u has quit IRC | 08:13 | |
| *** mohan43u has joined #buildstream | 08:14 | |
| *** tristan has quit IRC | 09:16 | |
| jjardon | mmm, strange buildstream tests seems to run extremely slow with any new Linux distro ^ | 09:31 |
|---|---|---|
| jjardon | (others fail) | 09:32 |
| *** tristan has joined #buildstream | 09:40 | |
| *** ChanServ sets mode: +o tristan | 09:48 | |
| tristan | juergbi, actually yes, efe5547f1fecc8076d3d769ae5468a1935dac10b adds functionality to refer to a subproject's subproject | 09:48 |
| * tristan was thinking he might try to add better error reporting around the regular usage | 09:48 | |
| abderrahim[m] | jjardon: someone mentioned coverage being broken with python 3.8 | 09:56 |
| * tristan recalls this comes back down to the ability to *compare* junction configurations, which cannot be done practically speaking, without a method of comparing *sources* for equality | 09:56 | |
| abderrahim[m] | I think that's what's happening here | 09:57 |
| * tristan re-reads the thread, I seem to recall we had a few theories around this | 09:57 | |
| jjardon | abderrahim[m]: mmm, how come the python3 job is ok with current master then? | 09:59 |
| tristan | jjardon, coverage is ignored for 3.8 | 09:59 |
| jjardon | also seems is not only slow, several test seems to be failing as well | 10:00 |
| tristan | see tests-python-3.8-buster in .gitlab-ci.yml | 10:00 |
| jjardon | ah, that maybe explain things | 10:00 |
| tristan | coverage is however collected for all other python versions, so I'd think it should still report something accurate enough | 10:00 |
| tristan | assuming we dont have python 3.8 specific codepaths | 10:01 |
| jjardon | problem is that we can not test any modern linux distro atm: https://gitlab.com/BuildStream/buildstream/-/issues/1283 | 10:01 |
| abderrahim[m] | yup, I'm using `tox -e py38-nocover` | 10:02 |
| jjardon | abderrahim[m]: do you know if it's a problem with python 3.8 itself or how we do the coverage? | 10:02 |
| abderrahim[m] | I think it's the package we use for coverage that's not compatible with 3.8 | 10:02 |
| jjardon | abderrahim[m]: what package that would be? tox? | 10:03 |
| * jjardon wants to search for an issue upstream | 10:03 | |
| abderrahim[m] | that'd be either coverage or pytest-cov | 10:03 |
| jjardon | cheers | 10:04 |
| tristan | it's hard to tell whether those jobs are hanging or slow... all three get stuck at around the same 3 tests | 10:04 |
| tristan | but I think there is parallel test processing | 10:04 |
| tristan | So could it be that one of them hangs around there ? (I do recall a long time ago hanging in test_push_pull(), could that have come back ?) | 10:05 |
| * jjardon wonders why covergae is pined to 4.4 version | 10:10 | |
| gitlab-br-bot | jjardon opened MR !1856 (jjardon/update_python_deps->master: Update python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1856 | 10:14 |
| tristan | jjardon, as I recall we had trouble with coverage in conjunction with multiprocessing | 10:20 |
| jjardon | tristan: yeah, that seems to be fixed with the new 5.0 version of coverage | 10:20 |
| jjardon | https://coverage.readthedocs.io/en/coverage-5.0.4/changes.html#changes | 10:21 |
| tristan | That's worth revisiting then :) | 10:21 |
| jjardon | Fixed in version Version 5.0a6 | 10:21 |
| jjardon | tristan: now that you are here, can you take a look to https://gitlab.com/BuildStream/buildstream/-/merge_requests/1853 ? Im a bit worried I had to change a API call to ostree to make the MR not to fail | 10:26 |
| * tristan takes a look | 10:31 | |
| tristan | jjardon, you want to *replace* all previous fedora images with the new fedora 31 ? | 10:32 |
| jjardon | yes | 10:32 |
| tristan | jjardon, would it not make sense to remove only the one which is really end of life ? | 10:32 |
| tristan | are fedora 29 _and_ 30 both end of life ? | 10:32 |
| jjardon | F30 will be very soon | 10:32 |
| jjardon | problem is: when changing the api call; everything works, even the old debian-9 jobs | 10:33 |
| tristan | yeah | 10:33 |
| jjardon | so I'm a bit confused about that | 10:33 |
| * tristan recalls looking at this before | 10:33 | |
| * tristan is also confused about this | 10:33 | |
| tristan | I recall looking at changes in ostree's gtk-doc statements which generate the introspection bindings | 10:34 |
| tristan | I think something around that changed at some point but not very significantly | 10:34 |
| tristan | Maybe the arguments became more strict | 10:35 |
| abderrahim[m] | the new one should be correct | 10:35 |
| tristan | but the underlying C API did not break | 10:35 |
| abderrahim[m] | the missing argument is an out parameter | 10:35 |
| jjardon | yeah, the removed argument is a return parameter | 10:35 |
| tristan | From memory, I concur with abderrahim[m] that the C API did not change and the GIR only became stricter | 10:35 |
| abderrahim[m] | but we should probably bump the dependency | 10:35 |
| abderrahim[m] | does something check the ostree version? | 10:36 |
| tristan | Yes, but only at install time | 10:36 |
| tristan | And it appears to be missing :-S | 10:36 |
| jjardon | https://lazka.github.io/pgi-docs/#OSTree-1.0/classes/Repo.html#OSTree.Repo.remote_gpg_import | 10:36 |
| abderrahim[m] | ostree is now in plugins-experimental IIRC | 10:37 |
| tristan | OK so... | 10:37 |
| jjardon | this is for bst-1 | 10:37 |
| tristan | Ahhhh | 10:37 |
| tristan | Yes I finally clicked on that point, I'm looking at master | 10:37 |
| jjardon | ostree C code didnt changed since 2005 I think, you are rigth | 10:38 |
| jjardon | I've checked that as well | 10:38 |
| tristan | At least for that function yeah it didn't | 10:38 |
| abderrahim[m] | yeah, it's either the annotation or the version of pygobject | 10:39 |
| tristan | ostree has been stable afaict | 10:39 |
| jjardon | ok, so It should be safe to merge https://gitlab.com/BuildStream/buildstream/-/merge_requests/1853 then? | 10:39 |
| jjardon | I can keep F31 if we really want to keep it for another month | 10:40 |
| tristan | you mean F30 | 10:40 |
| jjardon | yeah, sorry | 10:41 |
| abderrahim[m] | jjardon: tristan it was fixed in ostree v2019.2-10-gaa5df899 | 10:41 |
| tristan | Aha ! | 10:41 |
| tristan | I was just going to point out that I wish we had that information | 10:41 |
| tristan | jjardon, I think we should explain this in the commit message | 10:42 |
| tristan | Having "v2019.2-10-gaa5df899" in the commit message might come in handy in the future | 10:42 |
| abderrahim[m] | tristan: btw, once you're done with this I'd like you to take a look at !1835 | 10:42 |
| gitlab-br-bot | MR !1835: WIP: _project.py: resolve options before running the final assertions https://gitlab.com/BuildStream/buildstream/-/merge_requests/1835 | 10:42 |
| jjardon | ok | 10:42 |
| jjardon | abderrahim[m]: was it a fix in the annotation code? | 10:43 |
| abderrahim[m] | yup | 10:43 |
| tristan | abderrahim[m], this allows one to use append/prepend directives and such in the option definitions themselves ? | 10:44 |
| jjardon | ok, thanks | 10:44 |
| * jjardon updates MR | 10:44 | |
| *** phoenix has joined #buildstream | 10:44 | |
| abderrahim[m] | tristan: no, in the project.conf | 10:44 |
| abderrahim[m] | for example in `shell:` | 10:45 |
| abderrahim[m] | `host-files` | 10:45 |
| jjardon | ok to merge https://gitlab.com/BuildStream/buildstream/-/merge_requests/1853 then? | 10:45 |
| abderrahim[m] | jjardon: I guess the question is does this break with fedora 30 | 10:47 |
| gitlab-br-bot | tristanvb approved MR !1853 (jjardon/bst-1-fedora-31->bst-1: bst-1: Use current stable fedora 31 for tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1853 | 10:47 |
| abderrahim[m] | and how badly | 10:47 |
| * jjardon will leave f30 there to check | 10:47 | |
| tristan | jjardon, but please | 10:47 |
| tristan | jjardon, did you update the commit message itself ? | 10:47 |
| jjardon | yup | 10:47 |
| tristan | OK :) | 10:48 |
| jjardon | now I wonder; how come has this been working all this time for F31? XD | 10:48 |
| *** phoenix has quit IRC | 10:48 | |
| abderrahim[m] | jjardon: this only happens when using gpg, so it doesn't happen for freedesktop-sdk | 10:49 |
| tristan | abderrahim[m], I see - so conditional statements within project.conf were not working for 'shell' ? | 10:49 |
| abderrahim[m] | yeah, and instead of adding yet another special-case, I thought this is a better general fix | 10:50 |
| abderrahim[m] | (for both element/source overrides and shell) | 10:50 |
| tristan | Ummm | 10:52 |
| tristan | abderrahim[m], so basically we now resolve directives on the element/source overrides dictionaries while they are still on the main config dictionary, before separating them | 10:53 |
| abderrahim[m] | yup | 10:53 |
| tristan | Yeah that looks right to me :) | 10:54 |
| *** phoenix has joined #buildstream | 10:54 | |
| * tristan comments on the issue | 10:54 | |
| tristan | or MR | 10:54 |
| abderrahim[m] | tristan: feel free to suggest a better commit message | 10:55 |
| tristan | abderrahim[m], and basically this fixes the 'shell' portion of the config because of the timing of _assert_fully_composited() ? | 10:55 |
| abderrahim[m] | exactly | 10:56 |
| tristan | No need, commit message looks fine | 11:00 |
| tristan | Just made a comment as a matter of my own brain exercise ;-) | 11:00 |
| tristan | In any case, our tests are very strong in this area | 11:01 |
| * tristan struggles to figure out what to do with https://mail.gnome.org/archives/buildstream-list/2019-April/msg00021.html | 11:02 | |
| tristan | The code cs-shadow added is great because it allows one to inherit junction configurations from subprojects | 11:02 |
| tristan | But the problem remains that projects in a dependency graph can disagree on the ref of a common junction, and no warning or error is reported | 11:03 |
| tristan | So, the declaration of a junction is an explicit statement that it overrides any subproject's declaration of the same junction, however it is unclear that the toplevel project even knows that a subproject also declares this junction | 11:04 |
| * tristan just talks outloud to see if he's (re)understood the problem correctly, I think I have | 11:05 | |
| cphang | tristan mmm I think there should be at least a warning | 11:06 |
| tristan | cphang, that's what I'm arguing in the referred thread yes | 11:06 |
| tristan | Nobody disagrees, but the work has not been done yet to make that happen | 11:06 |
| cphang | gotcha, thanks :) | 11:08 |
| tristan | juergbi, points to the possibility of making these fatal errors, but allowing the toplevel (or higher level) junction declaration itself whitelist the override | 11:08 |
| cphang | Do you know what the new url for https://docs.buildstream.build/elements/junction.html#nested-junctions is? That currently gives a 404 | 11:10 |
| tristan | cphang, it should be reachable from a few places actually... but it's a part of the documentation of the junction element itself | 11:11 |
| tristan | cphang, add 'master/' between '...buildstream.build/' and 'elements/...' | 11:12 |
| tristan | now our docs are nicely version specific :) | 11:12 |
| cphang | thanks | 11:13 |
| juergbi | tristan: I'm not sure I'm understanding you correctly. BuildStream does not currently allow such disagreements | 11:13 |
| tristan | juergbi, I'm saying that... if the subproject I have depended on for a while... starts to depend on another subproject I also depend on, I will *inadvertently* override the original project's reference to the newly common junction | 11:14 |
| juergbi | or do you mean declaration in top/higher level is not sufficiently explicit? | 11:14 |
| tristan | it's explicit, but it does not necessarily mean that you know what's going on | 11:14 |
| juergbi | ah, right | 11:14 |
| juergbi | the override part is implicit in a way | 11:14 |
| tristan | yeah | 11:14 |
| juergbi | I wanted more control in that area almost since we've added junctions | 11:15 |
| tristan | lets do it :) | 11:15 |
| juergbi | I'm not sure what exactly the best way would be | 11:16 |
| juergbi | I've had some thoughts about it but have to try to remember the details | 11:16 |
| tristan | So maybe the junction declaration itself should list the subproject junctions it overrides explicitly ? | 11:16 |
| tristan | juergbi, this email you wrote might refresh your memory https://mail.gnome.org/archives/buildstream-list/2019-April/msg00020.html :) | 11:17 |
| tristan | or this one https://mail.gnome.org/archives/buildstream-list/2019-April/msg00016.html | 11:17 |
| tristan | anyway that thread entirely was a 20 or 30min read | 11:17 |
| tristan | Actually | 11:18 |
| tristan | juergbi, So now we have a way to make a junction derive from another https://docs.buildstream.build/master/elements/junction.html#targeting-other-junctions | 11:18 |
| tristan | with 'target' | 11:18 |
| cphang | tristan I think that would be ideal. I'd maybe go further and say that users using junctions must specify it in the top-level project, regardless of whether they are overriding or not. | 11:18 |
| tristan | juergbi, And if we had such a list that says "This junction overrides the configuration of the following junctions: [...]" ... then the *same* semantic could be used to (A) Inherit a configuration ... and (B) Explicitly use an inherited configuration to override another junction | 11:19 |
| tristan | cphang, I'd rather not force the toplevel project to have knowledge of how their sub-project artifacts are composed until we have to | 11:20 |
| tristan | Ideally you depend on a project and it gives you something stable, and how that is implemented is none of your business | 11:21 |
| tristan | But when it does become your business, I want a big red flashing light telling you :D | 11:21 |
| juergbi | yes, for all other things building elements via a junction is identical to building them directly in that subproject | 11:22 |
| juergbi | so I would like a way to not do this implicit name-based coalescing of junctions | 11:22 |
| tristan | cphang, I think a key part of the equation here is to remember that "Users using junctions" do not control the projects they depend on (I.e. not all projects are maintained by the same organization) | 11:22 |
| juergbi | however, I'm not sure how to balance the flexibility that provides with trying to catch users that accidentally use different versions of a subproject | 11:26 |
| cphang | that's true, but if I as a maintainer of a top-level project don't know about the implicit dependencies of the subprojects that I'm bringing in, then I consider that to be a bug, rather than a feature. If there is a feature that allows me to override a downstream junction, then that implies that there should be a single source of truth for all the | 11:26 |
| cphang | buildstream projects that top-level project is consuming. | 11:26 |
| cphang | I should say that implicit dependencies in this context is the buildstream project, rather than any elements contained within that project | 11:26 |
| gitlab-br-bot | marge-bot123 merged MR !1853 (jjardon/bst-1-fedora-31->bst-1: bst-1: Add current stable fedora 31 for tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1853 | 11:26 |
| cphang | So I care about the junctions, and only the junctions | 11:27 |
| juergbi | maybe we could handle it similar to git submodules | 11:27 |
| cphang | I was thinking that it was potentially analogous to that juergbi | 11:27 |
| juergbi | require the top-level project to explicitly list 'subjunctions' in a junction file | 11:27 |
| juergbi | and if an unlisted subjunction is detected, error out | 11:27 |
| juergbi | but also allow simply listing it without ref to get the ref from the subproject | 11:28 |
| juergbi | alternatively you can specify a top-level junction name to override it | 11:28 |
| cphang | I think that sounds sensible juergbi | 11:29 |
| * tristan stepped out to buy a sandwich but it was closed... reads backlog... | 11:34 | |
| * tristan is on the fence on this one... I *do* consider it a feature | 11:36 | |
| tristan | I.e., I consider it a feature that (A) I depend on this runtime for a long time... (B) I can update that junction to that runtime without friction... (C) I really don't want to know the implementation details of that runtime/junction that I use | 11:37 |
| tristan | *unless* there is a potential conflict, I really don't want to have to know, I want upgrades to be frictionless | 11:37 |
| * tristan has not had experience with git submodules where a sub-submodule is also an adjacent submodule | 11:38 | |
| tristan | I think that the subjunctions file would be another interesting approach, I might have that as a configurable warning though | 11:47 |
| tristan | if you set it as fatal, then you are required to at least set the junction | 11:48 |
| tristan | But then, what configurations would it have ? | 11:48 |
| tristan | juergbi, cphang... lets say you had a subjunctions file, so for each subjunction you would want to be able to say either (A) Inherit... (B) Redefine the junction here... (C) Override with this other junction... correct ? | 11:49 |
| tristan | And another question is; Does this allow us to drop the coalescing of junctions by junction name, even ? | 11:51 |
| tristan | Without the ability to compare Sources for equality, it seems there can not be any single source of truth for identifying a junction other than it's name | 11:52 |
| cphang | My question is, how does git handle this? | 12:10 |
| cphang | Because submodules could come up with this problem as well? | 12:11 |
| tristan | I'm quite sure that with git, you are not forced to have knowledge of sub-submodules | 12:15 |
| tristan | I could be wrong but I doubt it, perhaps you do have the opportunity to override them, and if so; you would be accessing them by project relative subdirectory (to where they are staged) | 12:15 |
| cphang | Indeed, but if there is a conflict in .gitmodules definitions when you allow recursing of submodules, how does git handle it? | 12:15 |
| tristan | I suppose in our case, the name of the junction would be analogous to the subdirectory of a git project | 12:16 |
| tristan | So with git, you could probably override a sub-submodule to stage an entirely different repository (rather than just a different version), but I'm speculating... | 12:17 |
| tristan | https://git-scm.com/docs/gitmodules | 12:17 |
| * cphang reads | 12:18 | |
| * tristan gives the above a reading | 12:18 | |
| tristan | also :) | 12:18 |
| tristan | Interestingly the .gitmodules file allows specifying the default cloning behavior, while in practice I've never seen that used | 12:20 |
| tristan | mostly people use `git clone --recursive`, or do `git submodules fetch` | 12:21 |
| cphang | Going to get some lunch, but it's the kind of thing it might be quicker just to run a quick experiment on :) | 12:25 |
| jjardon | ah, seems coverage 5.x is not directly compatible for python 3.6 and python 3.7: https://gitlab.com/BuildStream/buildstream/-/jobs/499144958 and https://gitlab.com/BuildStream/buildstream/-/jobs/499144957 | 12:26 |
| * jjardon tries to enable python3.8 and see what happens | 12:26 | |
| tristan | cphang, looking at https://stackoverflow.com/questions/46352657/modify-url-of-a-nested-submodule-in-git and https://stackoverflow.com/questions/39894103/can-i-override-the-url-of-a-nested-git-submodule-without-forking/48268906 ... | 12:27 |
| tristan | I'm not even sure there is a way for a git project to override the URL (or sha1) of a sub-submodule, without forking the direct submodule first | 12:28 |
| *** slaf has quit IRC | 12:45 | |
| gitlab-br-bot | abderrahimk opened (was WIP) MR !1835 (abderrahim/options->master: _project.py: resolve options before running the final assertions) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1835 | 12:58 |
| gitlab-br-bot | marge-bot123 merged MR !1835 (abderrahim/options->master: _project.py: resolve options before running the final assertions) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1835 | 12:59 |
| *** slaf has joined #buildstream | 13:01 | |
| *** tristan has quit IRC | 13:27 | |
| jjardon | mmm, updating to coverage 5.0 makes test fail with python3.8 as well (although only 5 instead 11): https://gitlab.com/BuildStream/buildstream/-/jobs/499176539 | 13:28 |
| *** phoenix has quit IRC | 13:51 | |
| *** tristan has joined #buildstream | 15:13 | |
| *** toscalix has joined #buildstream | 15:37 | |
| *** toscalix has quit IRC | 16:01 | |
| *** toscalix has joined #buildstream | 16:03 | |
| *** LittleTina has joined #buildstream | 18:17 | |
| *** LittleTina has quit IRC | 18:21 | |
| *** phoenix has joined #buildstream | 19:35 | |
| gitlab-br-bot | abderrahimk opened MR !1857 (abderrahim/options-1->bst-1: [bst-1] _project.py: resolve options before running the final assertions) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1857 | 19:38 |
| *** phoenix has quit IRC | 19:39 | |
| *** toscalix has quit IRC | 19:57 | |
| *** phoenix has joined #buildstream | 20:55 | |
| *** phoenix has quit IRC | 23:49 | |
| *** narispo has quit IRC | 23:54 | |
| *** narispo has joined #buildstream | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!