IRC logs for #buildstream for Tuesday, 2020-04-21

gitlab-br-botjuergbi opened MR !1873 (juerg/temporary-failure->master: _exceptions.py: Propagate `temporary` to the base class) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187305:13
*** phildawson has joined #buildstream07:04
*** benschubert has joined #buildstream07:34
*** phildawson_ has joined #buildstream07:36
*** phildawson has quit IRC07:37
*** tristan has joined #buildstream07:49
*** tristan_ has joined #buildstream07:50
*** ChanServ sets mode: +o tristan_07:50
*** tristan_ is now known as tristan07:52
*** tpollard has joined #buildstream08:36
*** santi has joined #buildstream08:36
*** phildawson-ct has joined #buildstream08:37
*** phildawson_ has quit IRC08:38
*** rdale has joined #buildstream08:38
gitlab-br-botmarge-bot123 merged MR !1871 (tristan/fix-doc-builds->master: Documentation build fixes) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187108:53
*** lachlan has joined #buildstream09:07
gitlab-br-botmarge-bot123 merged MR !1869 (tristan/remove-old-version-annotations->master: Remove old version annotations) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/186909:39
gitlab-br-botjuergbi closed MR !1872 (seanborg/documentation-typos->master: Fix typo namspaceing -> namespacing) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187209:41
gitlab-br-botjuergbi reopened MR !1872 (seanborg/documentation-typos->master: Fix typo namspaceing -> namespacing) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187209:41
gitlab-br-botjuergbi closed MR !1872 (seanborg/documentation-typos->master: Fix typo namspaceing -> namespacing) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187209:46
tristanmin-version is a bit challenging... as something similar needs to be done for plugins09:49
tristanI suppose it makes sense to have semantic versioning for plugins too, replacing the old BST_FORMAT_VERSION number, and recommend that plugins be versioned in sync with their release tags ?09:50
gitlab-br-botBenjaminSchubert approved MR !1873 (juerg/temporary-failure->master: _exceptions.py: Propagate `temporary` to the base class) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187309:51
tristanOr does it instead make sense to abolish it entirely, given that projects can reference precise versions of plugins they use ?09:52
tristanI suppose when it comes to separately installed system packages, there needs to be some story there which is anyway going to be out-of-bounds09:52
tristanlike "Install this plugin with version == 1.2.3 using pip install foo..."09:53
douglaswinshipSlightly over my head, I'm afraid.09:54
tristanCurrently projects can require a specific minimal version number of the projects they use: https://docs.buildstream.build/master/format_project.html#pip-plugins09:57
tristanSo when a project requires that the user install a "plugin package" alongside BuildStream, it can assert a minimal version of the plugin to ensure compatibility09:57
tristanIn this scenario, we had assumed that plugins would always be stable (which they of course werent)09:58
douglaswinshipYup, i've always wondered about that. If they require a minimum version, rather than a specific version, doesn't that open the door for variable outputs?09:58
tristanSemantic versioning could allow plugins to specify 1.0, where 1.2 and 1.99 would be compatible, and the plugin could version bump to 2.0 to signify API breakage (without creating an entirely new namespace)09:58
tristandouglaswinship, it does not if people follow the rules, but people don't follow rules09:59
tristanWe kind of opened up pip plugin installations as a means for people to provide *stable* outputs09:59
douglaswinshipNo, they never do :(09:59
tristanstable plugins I mean09:59
tristanBut yeah, so now we will introduce a way to use junctions to load plugins, which will allow plugins to be packaged as BuildStream projects, and open the door for BuildStream to just fetch the plugins at load time10:00
tristanSo anyway, I'm wondering if it's worthwhile to extend this functionality for BuildStream 2.x, or to abandon it altogether and give up hope that people might use it at all10:01
* tristan has a meeting...10:01
*** lachlan has quit IRC10:03
douglaswinshipOh, I opened an issue yesterday, a bug report. #128810:06
gitlab-br-botIssue #1288: Prevent buildstream sandbox folders from appearing in checked out artifacts https://gitlab.com/BuildStream/buildstream/-/issues/128810:06
douglaswinshipIs there anything I should/could be doing to promote it and encourage people to look at it?10:07
douglaswinshipI'm not confident that I'll be able to get any further on my own.10:07
*** cs-shadow has joined #buildstream10:10
*** lachlan has joined #buildstream10:19
*** lachlan has quit IRC10:27
tristanHmmm, that's strange10:32
tristandouglaswinship, my first intuition is that the severity is low if the directories are not actually in the artifacts themselves10:32
tristanbut then, it's still something important, depending on how you use buildstream10:33
tristanmight be easy to fix10:34
*** lachlan has joined #buildstream10:40
tristancs-shadow, do you have an opinion about plugin versioning ?10:41
tristancs-shadow, I'm looking at doing the s/format-version/min-version change now, and ran into the problem of "what to do with plugin version requirements"10:42
cs-shadowtristan: hi, sorry i may be missing some context, but looking at the scrollback, is it about the minimum plugin format version in the project configuration?10:44
tristanYes :)10:44
tristanYou walked in directly after I went through spilling my guts10:45
tristanit was a very emotional scene :'(10:45
tristancs-shadow, I'm currently tossing a coin where on one hand... we replace the silly integer with proper semantic versioning, and on the other hand; we just throw it all away10:46
tristanI pinged you because I think, you are the most intimately involved with the ins and outs of pip plugin source10:47
cs-shadowthere's one issue i see with recommending that plugin versions being the same as their package versions, which is when you have multiple plugins in the same package and the user only needs some of them. The integer version could've allowed some users to selectively upgrade (or not)10:47
cs-shadowthat being said, I don't think this will be a major incovenience10:48
tristanSo in your view, using pip sourced plugins kind of goes hand in hand with the user using a venv, and being "in control" of which plugins are being used, for a (possibly multi-project and junctioned) build pipeline, basically ?10:49
tristanLike, a user has the ability to decide10:50
cs-shadowmy view is that since BuildStream has no opinion on how the pip plugins are installed, it should find whatever is in the environment (whether it be virtual or not)10:51
tristanI guess that semantic versioning buys you nothing in the case that you might need 1.x of one plugin in one project and 2.x of another plugin in another project in the same pipeline10:51
cs-shadoware you expecting plugins to have breaking changes within bst 2.x lifecycle?10:52
tristanMost definitely 100%10:52
tristanOn a very regular basis10:52
persiaBeyond just pip, I'd suggest we shouldn't try to guess how a user has installed plugins  (maybe pip (maybe in a venv), maybe system packages, maybe local install).  It might be nice to leverage per-project plugins, but general environment plugins should just come from the environment.10:52
tristancs-shadow, I don't expect that to happen for core plugins maintained under BuildStream governance, though10:53
tristanSo before we walk down a completely different road than I was expecting or insinuating... let me clarify: I'm wondering if we really do need that versioning at all10:53
tristanMy observation has been so far that, I have not really seen a plugin ever set it's BST_FORMAT_VERSION to anything other than 010:54
tristanThe reason I ask about user control of installed plugins, is mostly because; if the user does have control, they will install exactly the versions which work for their pipeline; and maybe we don't need it there ?10:55
tristanAlso on the flipside, once we have the ability to load plugins over junctions: Versioning of those plugins becomes not useful either (A project is completely explicit about exactly which plugin it loads)10:56
cs-shadowI think we can defer to user to have them install the correct version, the only thing I expect BuildStream to do is to raise sensible errors.10:56
cs-shadowmost breaking changes that I can think of, can also be done by adding a similar flag in the plugin configuration. So, I think we can get rid of it.10:56
*** mohan43u has quit IRC10:57
tristancs-shadow, so in BuildStream core we are already getting rid of format-version in favor of matching that with installed BuildStream semantic version10:57
tristancs-shadow, because of this, I think it would be more uniform (and easier on users) if we did the same for plugins10:57
*** mohan43u has joined #buildstream10:57
tristanRegardless of whether the semantic versioning of major points is actually ever used (plugins might remain 1.x for their entire lifetime without breaking), it's easier to have an error saying "I need plugin foo >= 1.4"10:58
cs-shadowI think the difference is that min-version works for Core because we aren't expecting breaking changes there. Plugins will need to be able to declare max versions as well if they are allowed to break compatilbility10:58
tristanThan to have the user see "I need a plugin with format version 26, please go figure out what plugin version has that"10:59
tristanmin-version also works in a forward thinking way where BuildStream 3 might one day materialize too, at which point min-version could be set to "3.0" and a "Please install BuildStream 3" message could be issued11:00
tristan(but that's not really relevant)11:00
cs-shadowyes, but I'm thinking the case where there's a plugin 2.0 and 3.0, and my project only works with 2.011:00
tristanThe main driver for semantic versioning was that it makes more sense to users to say "This project requires BuildStream >= 2.2"11:00
tristanRight11:01
tristancs-shadow, I could imagine that a plugin might do that if they break their YAML API11:01
cs-shadowyep, that's what I meant when I was asking whether we are expecting plugins to break their API11:01
tristanHonestly, I don't expect it for plugins where the version guards are important11:02
tristanBut it could11:02
tristanI mean, if you are installing your plugins separately and hoping things work; you are probably very diligent about the plugins you use11:03
tristanBut you might still find a reason that you might want to break your plugin package I guess11:03
tristanSo first off, what I'm reading here is: "Yes we still want version guards for plugins"11:04
tristancs-shadow, ^^^ correct ?11:04
* tristan doesn't have a huge opinion or preference here really, just haven't seen people using it and wondered if we could afford dropping it11:05
cs-shadowi think so, my main concern is good error messages. I am less concerned whether the logic lives in BuildStream core or plugins11:05
tristanYou think this could be done on a per plugin basis ?11:05
tristancs-shadow, I guess that it's still better to have the version guard expressed in project.conf: https://docs.buildstream.build/master/format_project.html#pip-plugins ?11:06
cs-shadowone idea I had was that we completely abolish the plugin format version from core, and let plugins read the yaml and "deal with it". For example, plugins can recognize a field and assume that it's format version such and such implicitly11:07
cs-shadow(just thinking out loud)11:07
tristanyeah I get it... I think that it would be more onerous for the user (project author) though11:07
*** lachlan has quit IRC11:08
tristanNow... considering that pip source plugins are the only use case where plugins are not explicitly defined by the project; I wonder if we could simplify things here a little11:08
tristancs-shadow, What if we used semantic versioning *only* for pip plugins, and actually used pip package metadata to implement it ?11:09
tristanIf you are loading a local plugin, or a cross-junction plugin, your project defines already _exactly_ what's getting loaded, so it seems unnecessary to have a version check there11:10
*** CTtpollard has joined #buildstream11:11
cs-shadowI'd be in favor of that. It'll also be clerarer from the error message that `BuildStream expected plugin>2.0, you have 1.4, aborting`. Then user could install plugin > 2.0 and try again, as opposed to figuring out which format version comes from which package version11:11
*** tpollard has quit IRC11:11
*** CTtpollard is now known as tpollard11:12
tristanexactly, that's the main motivation for core as well11:12
tristanthat, and using a different format/word that is hard required (to be able to distinguish between bst-1 and bst-2 projects)11:12
cs-shadowI guess one question would be that how complicated expressions we want to support there, like:11:13
cs-shadow`plugin > 2.0`11:13
cs-shadow`plugin < 3.0`11:13
cs-shadow`plugin != 2.1`11:13
cs-shadow`plugin > 2.0, !=2.1`11:13
cs-shadow...11:13
tristancs-shadow, is there some setuptools-foo we could use to offload this completely ?11:14
persiaI think that the complication of expression support is something one only learns over time.  Python 2.6 introduced a number of new ways to express version incompatibility as a results of lessons learned with attempts to specify stuff for python 3, for example.11:14
tristanOr maybe we want explicit reimplementation in order to *avoid* offloading this comparison to a potentially "moving target" ?11:15
cs-shadowpersia is right, pep 440 (https://www.python.org/dev/peps/pep-0440) define the "new" versioning scheme that project _should_ follow but some don't. So, I'd be in favor of saying that if we the plugin or your requirement don't follow pep 440, bad for you.11:17
cs-shadowIf we say that, then there's a well-used library - distlib (https://pypi.org/project/distlib/) - that's used by things like `pip` to do this for us, and we can support everything that pep 440 supports11:17
tristancs-shadow, I'm in the midst of writing up the min-version change and would have liked to put it behind us this week hehe... that said, shall I make a small writeup of this and post that to the list ?11:19
* tristan suspects he could still get it all done in a week, it's not a *huge* change11:19
cs-shadowtristan: that seems perfect11:20
tristanAlright will do11:20
cs-shadowcheers!11:21
*** lachlan has joined #buildstream11:22
*** lachlan has quit IRC11:26
*** narispo has quit IRC11:30
*** narispo has joined #buildstream11:30
*** tpollard has quit IRC11:45
*** CTtpollard has joined #buildstream11:45
*** CTtpollard is now known as tpollard11:45
*** lachlan has joined #buildstream11:46
valentindcoldtom, There were failures to upload to cache: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/51931864511:51
valentindNot sure if you saw that.11:51
valentindMaybe it was restarted.11:51
valentindFeel free to restart the jobs if you want to test the cache server: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/merge_requests/237811:52
coldtomah, looks like redis fell over again...11:54
cphangvalentind https://gitlab.com/celduin/infrastructure/celduin-infra/-/merge_requests/78 is going to get merged today where it'll get deployed to our staging cluster. Once we've done some testing, we'll look to promote it to public asap11:55
valentindGreat.11:55
tristancs-shadow, sent as https://mail.gnome.org/archives/buildstream-list/2020-April/msg00017.html11:56
*** lachlan has quit IRC11:57
gitlab-br-botjuergbi opened MR !1874 (juerg/validate-path->master: _casbaseddirectory.py: Validate path components) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187412:01
*** tristan has quit IRC12:04
gitlab-br-botmarge-bot123 merged MR !1873 (juerg/temporary-failure->master: _exceptions.py: Propagate `temporary` to the base class) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187312:12
*** narispo has quit IRC12:54
*** narispo has joined #buildstream12:54
*** tristan has joined #buildstream12:55
cs-shadowtristan: +1, thanks13:00
*** hasebastian has joined #buildstream13:08
*** narispo has quit IRC13:11
*** narispo has joined #buildstream13:11
*** tristan has quit IRC13:18
*** lachlan has joined #buildstream13:21
*** lachlan has quit IRC13:31
*** phoenix has joined #buildstream13:37
*** phoenix has quit IRC13:38
*** d0a372eb0 has joined #buildstream13:43
*** d0a372eb0 has left #buildstream13:43
*** lachlan has joined #buildstream13:48
*** lachlan has quit IRC13:56
*** lachlan has joined #buildstream14:22
*** benschubert has quit IRC14:28
*** benschubert has joined #buildstream14:28
*** lachlan has quit IRC14:33
*** tristan has joined #buildstream14:37
*** lachlan has joined #buildstream14:47
*** cs-shadow has quit IRC14:52
*** cs-shadow has joined #buildstream14:52
*** lachlan has quit IRC15:04
*** lachlan has joined #buildstream15:17
*** lachlan has quit IRC15:41
WSalmonI have https://gitlab.com/BuildStream/bst-plugins-experimental/-/commits/willsalmon/git_tag_gitlfs which in CI uses the git_tag plugin which i need to fix up to pass the tests, https://gitlab.com/BuildStream/bst-plugins-experimental/-/jobs/520533065, but when i run locally it dose not find the git_tag plugin https://hastebin.com/duyonicepu.md dose any one have any idea what is up?16:04
*** lachlan has joined #buildstream16:05
WSalmonphildawson-ct, benschubert maybe?16:05
benschubertWSalmon: did you `register_repo_kind` on a kind named "git" ? Also `gitRepo` is not public, so I think you'd need to write a repo for git_tag?16:28
WSalmonyep i can see that there is loads todo but in CI it brakes cos the tests should brake but locally loads of things are failing cos bst-plugins-experimental dosent seem to be installed inside tox or atleas the plugins from it are not there16:31
benschubertif you run tox on bst-plugins-experimental locally, the plugins should be there, how are you running it?16:31
WSalmon(venvbuildstream) [will@wsx280 bst-plugins-experimental]$ tox -e py35-bst-fixed16:33
WSalmonhttps://hastebin.com/ihezakuzux.sql16:33
WSalmonbut i get the same for py37 too and for fixed or master16:34
benschubertI can't access hastebin, can you use github's gists/gnome paste/gitlab snippets?16:34
WSalmonhttps://paste.gnome.org/pfdnyjpj616:35
WSalmonoh wow gnome paste is much better if you just click raw16:36
*** lachlan has quit IRC16:36
WSalmonthat venv has bst-master and the plugins installed in to it but i presume tox will make its own and they wont interfear?16:37
cs-shadowWSalmon: the package name should `bst-plugins-experimental`, not `builstream-external`16:37
cs-shadowin your project.conf16:37
WSalmoni fixed that and it dose the same16:37
WSalmonthere are loads that are failing16:38
cs-shadowcan you share a new snippet with logs after that fix pleae16:38
cs-shadowmaybe also run with `-x`, one failure should be enough to debug it if they're all the same16:38
WSalmonso this is master16:39
WSalmonhttps://paste.gnome.org/pqsbcl4uz16:39
*** jjardon has quit IRC16:40
benschubertWSalmon: with the errors please :)16:40
*** jjardon has joined #buildstream16:43
*** ChanServ sets mode: +o jjardon16:43
*** jjardon has quit IRC16:44
*** jjardon has joined #buildstream16:45
*** ChanServ sets mode: +o jjardon16:45
*** jjardon has quit IRC16:46
*** jjardon has joined #buildstream16:47
*** ChanServ sets mode: +o jjardon16:47
WSalmonhttps://paste.gnome.org/pwtdvbuez16:48
WSalmonbenschubert,16:48
WSalmonthere was way too much for all of them, i can do a diffrent one if its more helpful16:48
coldtomit's installing a too-old version of bst-plugins-experimental by the looks of it16:49
coldtom0.12.0 is old16:49
*** lachlan has joined #buildstream16:50
WSalmonumm i will uninstall it from the venv with tox in it incase it is leaking in to the tox created one16:50
benschubertbst-plugins-experimental==0.12.0 yeah that's super old16:50
WSalmonumm exactly the same16:51
coldtomyeah, tox is installing it16:51
coldtomdoes buildstream depend on that for its test suite?16:52
WSalmonill do a recreate thing16:52
WSalmonbut tox made the py35 one just now16:52
*** lachlan has quit IRC16:55
WSalmonnope17:06
*** tpollard has quit IRC17:13
*** narispo has quit IRC17:28
*** narispo has joined #buildstream17:28
*** lachlan has joined #buildstream17:31
*** hasebastian has quit IRC17:37
*** rdale has quit IRC17:43
*** santi has quit IRC17:56
*** santi has joined #buildstream17:58
*** lachlan has quit IRC18:15
*** santi has quit IRC18:20
*** phildawson-ct has quit IRC18:26
*** narispo has quit IRC18:41
*** narispo has joined #buildstream18:41
*** narispo has quit IRC18:53
*** narispo has joined #buildstream18:53
*** toscalix has joined #buildstream18:54
*** ceibal has joined #buildstream19:20
*** ceibal has quit IRC19:24
juergbivalentind: are there tests for the oci element plugin anywhere?19:25
valentindjuergbi, I do not think I have made tests.19:26
juergbivalentind: it needs to be ported to the (virtual) Directory API. however, doing this blind doesn't sound ideal19:27
valentindI understand.19:27
valentindjuergbi, It could be done using valentindavid/bst2 branch for freedesktop-sdk.19:28
valentindI know it would be better to have tests though.19:28
juergbiright, should at least be able to test that it can build the fd-sdk oci images without throwing an exception19:28
valentindThat would be a good first step.19:29
valentindIf you have it,  then you can ping me and I can do a real test running the images.19:29
juergbiok, sounds like a plan19:29
valentindGood.19:29
*** benschubert has quit IRC19:32
*** ceibal has joined #buildstream19:34
*** ceibal has quit IRC19:37
gitlab-br-botcs-shadow opened MR !1875 (chandan/no-nosoftware->master: CONTRIBUTING: Remove link for obsolete issue policies) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/187520:44
*** toscalix has quit IRC21:31
*** toscalix has joined #buildstream21:34
*** cs-shadow has quit IRC22:11
*** toscalix has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!