IRC logs for #buildstream for Monday, 2020-05-04

*** mohan43u has quit IRC02:58
*** mohan43u has joined #buildstream03:01
*** tristan has quit IRC03:05
*** mohan43u has quit IRC03:12
*** mohan43u has joined #buildstream03:17
*** mohan43u has quit IRC04:26
*** mohan43u has joined #buildstream04:28
*** mohan43u has joined #buildstream04:33
*** mohan43u has quit IRC04:41
juergbirobjh: you don't get any output after launching it with maximum verbosity (--log-level=trace) and issuing requests?04:50
juergbiyou could also check the buildbox-casd logs in the logs subdirectory. however, you should also see bst-artifact-server logs directly in stderr04:51
juergbithere are no logs without requests from clients, though04:52
*** mohan43u has joined #buildstream05:14
*** mohan43u has joined #buildstream05:14
*** mohan43u has quit IRC05:19
*** mohan43u has joined #buildstream05:19
*** mohan43u has quit IRC05:26
*** mohan43u has joined #buildstream05:26
*** mohan43u has quit IRC05:29
*** mohan43u has joined #buildstream05:29
*** mohan43u has quit IRC05:32
*** mohan43u has joined #buildstream05:32
*** phildawson has joined #buildstream07:10
*** aminbegood has joined #buildstream07:22
*** aminbegood has quit IRC07:24
*** pointswaves has joined #buildstream07:25
*** benschubert has joined #buildstream07:27
*** pointswaves has quit IRC07:30
*** tristan has joined #buildstream07:36
*** ChanServ sets mode: +o tristan07:36
*** jude has joined #buildstream07:38
*** seanborg__ has joined #buildstream07:47
robjhjuergbi, i go to http://192.168.9.15:11001/ in a browser and i get "unable to connect", and the program produces no output. i'll look for the logs (any hints as to where?)07:48
juergbirobjh: there is no web interface. there might not be logs for completely unknown/invalid requests07:49
juergbicasd logs are in the logs subdirectory of the storage directory you specify on the CLI07:49
robjhahh, there are files there07:49
juergbibut you're probably not even reaching casd07:50
juergbihave you tried using it from bst instead of web browser?07:50
juergbi"unable to connect" does seem odd, I'd rather expect an HTTP error07:50
robjhi have. but not recently07:51
juergbithe error message you get in bst might be more useful07:51
robjhi thought it was a rest api, because the instructions talk about using a reverse proxy07:51
juergbiit's using gRPC, which is a HTTP-based API07:52
juergbibut there is no web interface07:52
robjhWARNING Failed to initialize remote http://mcr.robjh.com:11001: Remote initialisation failed: failed to connect to all addresses07:53
robjhsimilar results with both http and https (it is configured for https)07:54
robjhwith a cert from letsencrypt07:54
*** aminbegood has joined #buildstream07:55
tristanHmmm07:55
tristanWe removed `bst build --track` but, in the process we lost something important07:55
tristanThe tests/frontend/buildtrack.py was the only test I can recall which really thoroughly tested the the `--except` logic07:56
tristanwhich we still support in `bst source track --deps all foo.bst --except bar.bst --except baz.bst ...`07:57
tristanAm I missing where that test was migrated to in the process of removing build/track support ? I'm not seeing it in tests/frontend/track.py07:57
*** lachlan has joined #buildstream08:00
robjhive fiddled with it a bit, and it looks like the problems might have been a combination of the server being on my lan and docker-compose run not exposing the port08:04
*** lachlan has quit IRC08:08
juergbitristan: I think you're right. we have to resurrect it from commit 576be1fc19bb58d25309a911589ceb9f08f7812d08:11
tristanshouldnt be too hard08:12
juergbiwe do have some except testing, e.g., test_show_except_simple08:12
tristanAh08:13
juergbiis for bst show but tracking should no longer have separate except handling, iirc08:13
juergbimight not be as thorough as it was, though, not sure08:13
tristanWell, iirc at least the except code itself is the same08:13
tristanYes I think the buildtrack.py test was very thorough, we fixed regressions through that path before08:13
juergbiah, test_show_except actually seems to test fairly complex cases08:13
juergbibuild --track was more complex, though, due to the different except combinations for track and build parts08:14
tristanI came across this in the context of https://gitlab.com/BuildStream/buildstream/-/merge_requests/1893/diffs#note_33569618608:15
*** seanborg__ has quit IRC08:15
robjhalso, i was accidentally giving it an expired cert from 201908:16
tristanjuergbi, I think the show test is not covering for instance, the case of `bst show --deps plan foo.bst --except bar.bst --except baz.bst`08:16
tristanbuild plan selections can be more tricky08:16
tristanbut yeah, it does look like a good test in tests/frontend/show.py08:17
* tristan didn't realize it was there08:17
*** seanborg__ has joined #buildstream08:19
tristanjuergbi, can I bribe you for a review of https://gitlab.com/BuildStream/buildstream/-/merge_requests/1892 and https://gitlab.com/BuildStream/buildstream/-/merge_requests/1894 ?08:19
tristanjuergbi, any particular MR you want a review on ?08:19
* tristan was looking at the MR list trying to buy himself some review points hehe08:20
juergbitristan: regarding except, at least it seems like all lines of Pipeline.except_elements() are covered by tests08:26
juergbireview: I actually don't have an MR open right now but I can try to review your MRs in a bit08:26
tristanThanks :)08:27
juergbifor the pip one I hope benschubert or cs-shadow also find time for a review08:27
tristanyeah08:28
* tristan has been hoping to find cs-shadow for a while now08:28
tristanthe thing is that the pip one is based on the other one I wrote just before it, and touches the same areas of code08:29
tristanso landing the other would help me avoid more conflict resolution anyway08:29
* tristan thinks this week is time to tackle the junction jungle08:30
benschubertjuergbi: !1894 ? I'll look at it now08:31
*** tpollard has joined #buildstream08:41
*** santi has joined #buildstream08:41
tristanbenschubert, note that many of the commits in !1894 are actually from !189208:42
tristanbenschubert, and: https://buildstream.gitlab.io/-/buildstream/-/jobs/536561483/artifacts/public/format_project.html#loading-plugins08:42
tristanThat is the rendered documentation of what !1892 adds, plus a general overhaul of the plugin origin documentation08:42
benschuberttristan: yep, just had a look at 1892 first, it's a really nice change :)08:43
* tristan renamed "External plugins" to "Loading plugins" also... as "External" was an outlier in BuildStream 1, but we expect will be the norm in BuildStream 208:43
*** lachlan has joined #buildstream08:48
robjhat what point does a bst client push objects up to a server?08:51
robjhim not seeing as much traffic between the two as i would expect08:51
tristanSo you can do `return foo, bar` in python to return a tuple, no need to `return (foo, bar)` ?08:52
*** lachlan has quit IRC08:54
tristanseems tests pass this way too08:55
tpollardtristan: yep08:55
tristanlearn something new every day :)08:58
tristanSeems we now have a bit of both in the codebase, and both pylint and black don't seem to care08:59
* tristan doesn't personally care all that much, minor inconsistency08:59
*** aminbegood has quit IRC09:00
benschuberttristan: we can configure pylint to care if wanted :)09:02
benschubertbut yeah in python you can have (a, b) = b, a or such weird but gosh you miss them in other languages stuff :)09:02
benschubertjuergbi, tristan anything else you need my input on? :) Sorry, have been quite busy lately09:03
tristanNo worry09:04
juergbibenschubert: thanks, nothing from my side right now. except feedback on how buildbox-run is working for you :)09:05
tristanbenschubert, We should get to the bottom of the cross junction plugin loading story though09:05
benschubertjuergbi: haven't been able to run it "in production" but I'd like to have a test deployment this month and will definitely test it thoroughly then. I'm just hodling back our updates until we have the buildbox-casd issue fixed, to reduce number of breaking changes09:06
juergbithe timeout issue, I presume. makes sense09:07
benschuberttristan: right, I'll give you an answer to that today (probably late afternoon London time, so you'll have it tomorrow morning). the TLDR is that I'm worried that the variable overriding with your solution would be unintuitive, but I need to re-read the whole thing to ensure I got everything09:07
benschubertjuergbi: correct09:07
juergbihttps://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/merge_requests/169 will likely land very soon09:07
juergbi(in addition to the buildstream side mitigation)09:07
tristanbenschubert, sure, I thought an IRC chat might help us to align09:07
tristanbenschubert, personally I feel like I want "cross junction loading" simplicity, especially to cater to the use case of both depending on elements and using plugins across a junction, with simple syntax (even if the subtle details of how things work need an explanation somewhere)09:08
tristanbenschubert, at the same time I can see that using an element to stage plugins somewhere helps if you want to "just get the data" regardless of project.conf09:09
tristanbut in that view point, I really wonder why we would need a junction *also* (i.e. if you wanted to use an element to stage plugins and then load them, you could completely drop the need of a junction altogether)09:10
*** lachlan has joined #buildstream09:11
benschuberttristan: one of my concerns with this approach is if you end up with a junction using plugins from `pip` and import them, then you _might_ think you have a guarantee that you are using the exact same version when you might not be.09:11
tristanbenschubert, anyway sorry I realize you don't have the bandwidth right now :)09:11
benschuberttristan: no worries, irc might indeed be better :D09:11
tristanI think that if you use a junction, you are indeed subjugating yourself to the whims of the project from whence your plugins come from09:12
benschuberttristan: > why we would need a junction *also*09:12
benschubertSeems my proposal was not 100% clear, You would not _require_ a junction, just allow plugins to be defined as elements and thus track them as such, and you would gain the "accross junction" for free09:12
tristanIn multiple ways09:12
benschubertthat's also true. Do you think that "just" declaring "I want to use this plugin from this junction" we sufficient for all the needs of gnome and fsdk and such? (I don't have most of those problems since we control the environment more tightly)09:13
tristan... to elaborate "in multiple ways", I mean: I could envision use cases where a lower level project provides "local" plugins which are intended for consumption by reverse dependency projects09:13
tristanSo changes to that static plugin means that updating the junction also brings in changes there, but it seems appropriate09:14
benschubertyeah, if you update your junction you could expect this. agreed09:14
tristanRegarding the gnome/fdsdk cases, I do think it would be sufficient for the custom plugins being used there - however, "I want to use this plugin from this junction" I think is not really enough09:16
benschubertright09:16
*** lachlan has quit IRC09:16
tristanWe would need to at least have our upstream plugin packages structured in a way that they have a project.conf and we can consume those09:16
tristanTogether I think that would be enough09:16
tristanIt would mean that we have a paradigm which allows people to distribute plugins as a "both pip and junction" and allow users to choose (whether it's the upstream ones, or other side projects)09:17
benschuberthow do you handle third party pip dependencies? Vendoring? Leave users install them manually (I *think* we should add import-error guards in our plugin and give a nice error in that case)09:17
tristanBack to this territory :)09:17
tristanIn this instance, I totally agree09:18
benschubertand so for the "registration" we would say: `{"origin": "junction", "junction": "my-junction.bst", "elements": ["pip"], "sources": ["docker"]}` ?09:18
tristanIn the original design, `Plugin.preflight()` is the place for plugins to report specific concise messages about system requirements09:19
tristanBut we did not really expect imports from outside the non standard library (those crept in initially with ostree though so it is a reality of course)09:19
tristanAnd handling import errors in Plugin.preflight() is not ideal09:19
benschubertthe problem with that is that it makes importing third party deps harder :) and you might just forget, I think catching a *importError* would solve most of this09:19
benschubertyep09:19
benschubertwe _could_ catch ImportError and register a dummy plugin that has the preflight that returns the error about import? :)09:20
tristanThe loader could do this and we could have a test for this too09:20
tristanThe thing is, once preflight is called, it's too late09:20
tristanPlugins usually import at the toplevel of their module09:20
tristanRight I see what you're suggesting... nah I don't think it's important to delay the error until preflight time09:21
traveltissuescan i get a review please https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/10409:21
tristantraveltissues, This is bazel element isn't it :-S09:22
benschubertdepends, if you have many plugins and they all have an error, getting all errors one by one is not ideal, if we keep the error to preflight, we could make optimizations there later and show all errors at time :)09:22
benschubertbut as a first pass sure, just catching and showing a nice error would be much better already09:22
traveltissuestristan: yes09:22
benschuberttristan: ok, last point about the proposal: would we allow projects to have different configuration for their junctioned plugins than the junction that contains the plugin?09:24
tristantraveltissues, So I don't really have buffer space for this plugin right now, but I don't think it should prevent you from landing it in bst-plugins-experimental09:24
tristanbenschubert, So this is an interesting question, I'm not sure I understand which context you mean09:24
tristanbenschubert, Say... I want to depend on elements from project A, and I want to use plugins that project A also uses or provides, but possibly at a different version in the history of project A ?09:25
benschubertlet's say you use a cmake plugin through a junction. The cmake plugin is configured to install things in /usr. You want ot install things _from your project_ in /usr/local to have a clear separation between both.09:26
benschubert-> Do we allow that?09:26
benschubert-> How would we express this?09:26
benschubertNot different version (that could be solved with two junctions :) )09:26
tristanRight I was going to say that regarding two junctions. so indeed... this is the tricky thing.09:27
tristanbenschubert, I think we may need to offer the choice here09:28
tristanEither observe the origin project.conf when loading the plugin or don't09:28
benschubertI'm not 100% sure about this solution, you might want to keep _some_ defaults :) (especially if we go with trimming most defaults)09:29
tristanbenschubert, orthogonal to this, I think we're treading into territory where plugins have expectations about the projects they are used in09:29
benschubertwould special-casing the loader to load from the junction then copy and merge the project's default?09:29
tristanI.e. there is going to be user stories where the user uses a plugin for the first time, and at loading time the plugin explodes and says "Hey, I don't have any variable to expand named %{sysconfigdir} !"09:30
tristanAnd then the user goes and adds something to their project.conf in order to satisfy the needs of the plugin09:30
benschubertah yes absolutely, not what I meant sorry :)09:30
benschubertWhat I meant is "i might be fine with all the configuration given by my junction but 2 of the parameters, please override those to, but only for my project, not the junction"09:31
benschubertand I don't want to keep the others in sync09:31
benschubert(and was saying that this will become even more prevalent with less defaults set)09:31
tristanbenschubert, I think that specific case is handled by the local project's overrides always having a chance to set things09:32
tristani.e. you can set %{sysconfigdir} or whatever variable in the elements: dictionary of the project which used a cross-junction plugin, and it will still have the second-last say about those element configurations09:32
benschuberttristan: if we load the plugin in the junction's context though, wouldn't that affect the junction?09:32
tristanbefore variables get resolved09:32
benschubertah right09:33
benschubertthe variables and the plugins are handled separately09:33
benschubertah no wait09:33
tristanThere might be a few gritty implementation details in there which indeed mean it's not exactly "just load it from that project", of course09:33
benschubertif that's the case, how do you get the "defaults" from the plugin frm the junction?09:34
benschubertOr do we expect users to have to re-write all of them?09:34
benschubertlike "use the plugin as it is configured in my junction"09:34
tristanActually I expect includes to handle most of this09:34
benschubertor is that somethign that was never requested?09:34
benschubertok09:34
tristanRight well, that's the other option, I was thinking... maybe we want to give the user the choice to observe the junction's project.conf or not when loading the cross-junction plugin09:35
benschubertyeah, that's actually a better phrasing of one of the things I'm worried about, that this might become an expectation :)09:35
tristanIf there is a choice, and you choose to ignore the project.conf configuration from the junction, then you need to provide everything the plugin needs of course09:35
tristanBut in practice, I expect the majority of defaults to be provided by include files in low level projects09:36
tristanSo it could be that both projects junction the same base runtime which defines %{prefix} and all the general system layout09:36
benschubertOk, I think that make sense and I don't think we overlooked anything09:38
benschubertI'll have to think a bit longer about it to ensure every thing is fine and I'll reply to the list / ping you more ok?09:38
tristanAwesome :)09:38
tristanHappy we had this talk :)09:38
tristanbenschubert, you are happy with the 2 MRs I presume ? I will rework the removal of the tarball which sounds great09:39
benschubertI'm happy with those yep! And the removal of the tarball will indeed be good, it's always such a pain to keep those things in sync :D09:39
tristanAh more comments, I'll go ahead and address them all :)09:40
benschubertI'll ping cs-shadow also to ask him if he wants to review09:40
tristanYeah it is, I really wanted a test fixture which builds a package and adds it to PYTHONPATH for the duration of a test09:40
tristanbut that turned out to be asking for too much :-S09:40
benschubertyeah, it's a bit of a pain09:41
benschubertunless we start faffing with venvs, but then we'd be having side effects so hey09:41
benschubertbut yeah, once the comments are addressed I'm happy with it09:41
tristanThanks :)09:42
juergbitristan: create a bst project and build buildstream and the python plugin package inside that. inception ;)09:42
WSalmondo we need to update https://buildstream.build/source_install.html#installing-dependencies for buildbox-run? i have just hit `[00:00:22] FAILURE bootstrap/gnu-config.bst: This platform does not support local builds: buildbox-run not found` is there a MR up already i can test?09:44
*** lachlan has joined #buildstream09:45
juergbiWSalmon: https://buildstream.build/buildbox_install.html is already updated09:45
WSalmonjuergbi, would `BuildStream master also requires buildbox-casd: Install BuildBox `-> read better as `BuildStream master also requires buildbox: Install BuildBox` I skim read that like and only registed the `buildbox-casd` bit which i had09:50
juergbiright, should tweak that09:51
WSalmonthanks09:53
*** lachlan has quit IRC10:07
tristanbenschubert, When cs-shadow and I discussed this, we said we should link to https://www.python.org/dev/peps/pep-0440/ and mention that version constraints are only guaranteed to work with plugin packages which follow that specification in how they declare their versions10:37
tristanBut, this pep does not seem to define the *constraints* themselves :-/10:38
*** lachlan has joined #buildstream10:38
benschuberttristan: ah, fair enough -_-'10:39
tristanI will add that10:40
tristanI did ask in #python and was given a decent link, but unfortunately not from an official upstream URL :-S10:40
* tristan is looking for it in his history now10:40
*** lachlan has quit IRC10:41
tristanbenschubert, here it is: https://python-poetry.org/docs/versions/10:42
tristanbenschubert, Do you think we should link that ?10:42
tristanthe material itself looks good and user friendly10:42
* tristan doesnt know where 'python-poetry.org' is from or what they are associated to, though10:43
benschuberthttps://www.python.org/dev/peps/pep-0508/ :D10:43
benschubertIs that it?10:43
benschubertpoetry is an alternative backend to create python packages10:43
benschubertit is quite well known and reputable10:43
benschubertah shoot, no it doesn't specify the constraints10:44
benschubertoh well, if poetry is the best we can get, let's use that10:44
benschubertbbetter than nothing10:44
tristanYeah agreed, it is actually a nice page10:45
tristanI'll also link 0440 probably in the same sentence10:45
benschubertseems good, thanks a lot10:46
*** seanborg__ has quit IRC11:00
*** seanborg__ has joined #buildstream11:00
*** seanborg__ has quit IRC11:12
*** seanborg__ has joined #buildstream11:13
*** lachlan has joined #buildstream11:23
*** lachlan has quit IRC11:29
*** seanborg__ has quit IRC11:34
*** seanborg__ has joined #buildstream11:41
*** seanborg__ has quit IRC11:44
*** seanborg__ has joined #buildstream11:44
*** seanborg__ has quit IRC11:47
*** seanborg has joined #buildstream11:48
*** tristan has quit IRC12:00
WSalmondose anyone else get a crash if they try to tab complete from `bst ar`?12:15
WSalmonon master12:15
WSalmonhttps://paste.gnome.org/pa2maa2ed12:17
*** cs-shadow has joined #buildstream12:20
*** jude has quit IRC12:21
*** jude has joined #buildstream12:34
coldtomWSalmon i've experienced a crash from attempting to tab complete something or other12:39
WSalmonits especally annoying as it hangs rather than exiting12:40
tpollardWSalmon: does the same thing happen if you try it in a bst project dir12:43
WSalmonyep12:44
tpollardurkj12:44
WSalmoni only made the paste in a diffrent dir as i had finally got bst to go and had lost it off the scroll back12:44
tpollardI just remembered we used to have a tab bug when calling outside of a project dir12:45
*** lachlan has joined #buildstream13:16
*** lachlan has quit IRC13:20
*** lachlan has joined #buildstream13:23
*** lachlan has quit IRC13:26
*** scottclarke has joined #buildstream13:32
*** lachlan has joined #buildstream13:43
*** lachlan has quit IRC13:47
*** scottclarke has quit IRC13:50
*** scottclarke has joined #buildstream13:51
*** lachlan has joined #buildstream14:16
*** lachlan has quit IRC14:20
*** lachlan has joined #buildstream14:37
*** lachlan has quit IRC14:42
*** lachlan has joined #buildstream14:59
*** tristan has joined #buildstream15:02
*** lachlan has quit IRC15:06
*** lachlan has joined #buildstream15:33
*** lachlan has quit IRC15:38
*** lachlan has joined #buildstream16:21
*** seanborg has quit IRC16:32
*** lachlan has quit IRC17:02
*** lachlan has joined #buildstream17:32
*** santi has quit IRC17:33
*** lachlan has quit IRC18:00
*** tpollard has quit IRC18:01
*** pointswaves has joined #buildstream18:01
*** phoenix has joined #buildstream18:14
*** aminbegood has joined #buildstream18:33
*** mohan43u has quit IRC18:40
*** cs-shadow has quit IRC18:49
*** aminbegood has quit IRC19:08
*** phoenix has quit IRC19:18
*** phoenix has joined #buildstream19:20
*** benschubert has quit IRC19:45
*** phoenix has quit IRC19:56
*** phildawson has quit IRC21:02
*** jude has quit IRC22:05
WSalmonwhat dose [23:48:02][--:--:--][        ][    main:core activity                 ] WARNING Failed to initialize remote http://XXX.XXX.XXX.XXX:50051: Remote initialisation failed: failed to connect to all addresses mean?22:50
*** pointswaves has quit IRC22:53
*** pointswaves has joined #buildstream23:05
*** pointswaves has quit IRC23:08

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!