IRC logs for #buildstream for Wednesday, 2020-01-08

*** rdale has quit IRC00:22
*** traveltissues has joined #buildstream07:34
*** rdale has joined #buildstream09:37
*** tiagogomes has joined #buildstream09:46
*** tme5 has joined #buildstream09:50
*** jonathanmaw has joined #buildstream10:01
tme5was there a specific reason that the git source was factored out into a base class or was it for prospective future work?10:07
coldtomtme5: it was supposed to be so that the git_tag source could subclass from the base class and reduce maintenance cost10:09
tme5ah ok, so I suppose if git_tag were merged, it could be folded back into git.py10:10
coldtomhowever, the base class isn't public api as of right now, and i was advised to wait for the refactor in https://gitlab.com/BuildStream/bst-plugins-experimental/issues/310:10
tme5ok, I hope none of that will be necessary and we can have just the one git plugin!10:12
tme5thanks for the info10:12
juergbitme5: iirc, there are other use cases for customization of the git plugin. not all of which could be folded into git.py10:25
juergbiin particular, customizing tracking. e.g., github provides an API for releases. gitlab probably as well. this could be very useful but we can't support everything in git.py10:26
tme5so you want to move towards it being public API instead?10:26
juergbithe plan was to make it public API in a separate bst-plugin-git repo or similar, iirc10:27
benschuberttme5: sorry for last time, I forgot to answer you about the state of bst-plugins-experimental. The aim is to split this one into multiple repository, each containing a specific set of tools (e.g. we could have a bst-plugins-python containiing pip source, pip element, etc). The problem is, as for other projects (e.g. ansible), many people want plugins to do many different things. Having those split up lets people that are10:27
benschubertinterested in one plugin/plugin set move at a pace different from official BuildStream release. It also reduces the burden on the main team around maintaininig such plugins10:27
benschubertjuergbi: I recall teh same :)10:27
tme5np benschubert. thanks for the info10:32
*** lachlan has joined #buildstream10:37
*** lachlan has quit IRC10:42
*** lachlan has joined #buildstream10:51
*** lachlan has quit IRC11:02
tme5what do you think about changing the 'git' ref-format default to git-describe from sha1?11:15
tme5(this is assuming tag tracking is merged in; with tag tracking there would probably need to be dual behaviour otherwise, as when using it you'd want git-describe refs by default)11:17
benschubertI would be more or less strongly against it, shas make much more sense IMO, they represent the commit that you want with all it's tree. Git-describe might change if you add a new tag between both etc, whereas the content would still be the same (hence breaking cache hits etc)11:19
*** lachlan has joined #buildstream11:19
tme5how would it break cache hits? the tag part of the git-describe would change but the part after that is the commit sha itself no?11:21
benschubertthen you would only store the last part in the ref? Or have the ref not represent the exact element?11:22
*** lachlan has quit IRC11:23
benschubertand what happens when the tag moves/is deleted? :)11:23
tme5benschubert, I don't think either is necessary but I'm not a git wizard. And what happens if a tag moves, well i'd guess nothing breaks, as the describe contains the commit sha anyway11:24
benschubertshas are nicely immutable and won't change unless it is removed from the repo (which is much harder and less common than moving tags)11:24
tme5yeah, I do agree a track changing a ref when nothing has actually changed is a bit odd11:24
*** lachlan has joined #buildstream11:26
tme5oh i see what you mean about refs changing now11:26
tme5the tag part becomes part of the cache key11:27
benschubertunless we strip it away11:27
benschubertbut in that case we have a weird duality where part of the ref is not useful at all and someone could write anything11:27
benschubertwhich is fine for the git_tag since it relies on tags11:27
benschubertbut I don't think the 'git' plugin should have anything to do with it :)11:28
*** lachlan has quit IRC11:48
tme5benschubert, hmm, well 'git' already has this problem then when using git-describe11:49
tme5(problem of a unnecessary cache key change)11:50
tme5i think it makes sense to address this in the main plugin too11:51
tme5also, experimenting with `git show` tells me that the beginning part of a describe tag is ignored by git anyway11:52
tme5so i don't think it's bad for buildstream to do the same11:52
benschuberttme5: oh good point, sorry I took a shortcut here. I do believe that the 'git' plugin, should not have the default to 'tag', because it is unintuitive when not explictely working with tags, does that make more sense? I am happy with a fix for not changing the ref anymore :)11:53
tme5"should not have the default to 'tag'" to clarify, you mean should not have ref-format default to git-describe?11:55
benschubertfor the git plugin, correct11:55
tme5I disagree that it is unintuitive -- It adds information which is, at the very least, completely ignorable, while removing nothing and having (after said fix) no ill effects. But I'm happy with the alternative, which is for a hypothetical merged git_tag to have varying defaults depending on whether tags are tracked11:58
benschuberttme5: `git describe` will fail with 'fatal: No names found,c annot describe anything' whenever you have a repository without any tags, which is relatively common, so a newcommer would have a hard time understanding why his track of 'master' doesn't work :)12:01
benschubertor if there are tags but none can be seen on the current tree12:01
benschubertwhich is also common-ish12:01
tme5ahhh, well that's no good, looks like sha1 it is then12:02
benschubertWhich is, now that I think about it, also one potential reason why having git_tag in a separate plugin (even if same repo) might be easier :)12:02
bochecha> It adds information which is, at the very least, completely ignorable12:03
bochechait isn't always ignorable12:03
benschubertbochecha: do you have examples? :)12:03
bochechaso, right now on my repo, `git describe --long --tags HEAD` gives me this: v1.0-59-g4dfbafd (I really need to make a new release :P)12:04
bochechaif you ignore everything except the sha part, you get 4dfbafd12:05
gitlab-br-bottlater opened MR !1790 (tlater/fix-monkeypatch-warning->master: tests/artifactcache/config.py: Cast our tmpdir to a string) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/179012:05
bochechagit helpfully shortens the sha1 to make it easier on the human eye12:05
bochechabut it shortens it in a way that the short version is still unique12:05
gitlab-br-botBenjaminSchubert approved MR !1790 (tlater/fix-monkeypatch-warning->master: tests/artifactcache/config.py: Cast our tmpdir to a string) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/179012:05
tme5benschubert, looks like git_tag handles it gracefully when tracking, resorting to sha1 when there are no tags. so might not be a huge issue12:05
bochechalet's say a future commit has a hash that also starts with 4dfbafd12:05
tme5bochecha, the plugins use --abbrev=40 when building the describe ref12:06
bochechaat that point, that short hash won't be enough to uniquely identify the commit12:06
tme5so it has a full length id at the end12:06
benschuberttme5: but then you don't have a consistent output, and I would rather not have that on the 'git' plugin :)12:06
bochechaah, if it has the full length then yes, that's all fine :)12:06
bochechaI did not know it did that12:06
tme5benschubert, personally i dont understand the problem with that, with our hypothetical fix to the cache keys.12:07
tme5if you ask for tag tracking, it makes sense that the ref you get back contains the tag information if available12:07
benschuberttme5: now, depending on the tags in your repo, you might get one, or the other, with the tag one possibly failing when a repo use tags that move :)12:08
benschubertah correct12:08
benschubertif you ask for tag tracking12:08
*** toscalix has joined #buildstream12:09
*** toscalix has quit IRC12:13
*** toscalix has joined #buildstream12:17
*** lachlan has joined #buildstream12:30
*** douglaswinship has joined #buildstream12:31
*** lachlan has quit IRC12:37
*** toscalix has quit IRC12:50
*** lachlan has joined #buildstream12:51
*** lachlan has quit IRC12:55
douglaswinshipHi all. I'm working on a plugin for listing source urls, both original source urls and mirror urls. I could use some advice.14:05
douglaswinshipFor a given element source url (like:"savanah:config.git", the manifest lists the alias (savanah), the translated url (from the aliases defined in project.conf) and any mirror urls (from the mirror-aliases in project.conf).14:05
douglaswinshipThe plan is to write a few more scripts that consume the json output. Those scri14:05
douglaswinshippts will check that every source has a mirror defined, and that the mirror actua14:05
douglaswinshiplly exists.14:06
douglaswinshipMy biggest problem is that I had to use protected members in the script: "_get_a14:06
douglaswinshiplias()" and "_get_project". That doesn't feel like the right way to do it.14:06
douglaswinshipThe plugin is here: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/DouglasWinship/mirror_url_manifest/plugins/elements/url_manifest.py14:06
douglaswinshipAnd a sample output is here: https://gitlab.com/snippets/1928064 (a more realistic output would have dozens of entries).14:06
douglaswinshipCan anyone recommend a better way to do this, without having to use protected me14:06
douglaswinshipmbers of the element class?)14:06
douglaswinshipMaybe there are functions available that I haven't noticed? Or perhpas I ought to take a whole different approach. This is my first try at a buildstream plugin, so any suggestions would be appreciated.14:09
tlater[m]douglaswinship: Why would you want to check mirrors in the build?14:11
tlater[m]Or do I misunderstand what that plugin is intended to do?14:13
douglaswinshipThese are for scripts that will run in the automated CI on gitlab, the tests that we run on new code.14:14
tlater[m]I think it would be neat to do that at a source level14:14
tlater[m]Ah, so just to assert that people are keeping up to your coding standards?14:15
coldtomtlater[m]: the idea is to check whether we have any unused mirrors or if there are sources without mirrors14:15
tlater[m]In that case, I would suggest not doing that as a BuildStream element plugin. It'd make more sense to write this like a linter14:15
coldtomit's convenient to do it inside buildstream because of all the yaml stuff being done for you14:15
tlater[m]Yes, this is definitely more useful as a linter14:15
douglaswinshipin the first stage, yes, to make sure people are using aliases in the urls, which enanables mirroring. And then at a later stage, to check that we've actually created a mirror for it.14:15
tlater[m]douglaswinship: You can import BuildStream without writing a plugin14:16
tlater[m]This is a workflow we've thought about in the past14:16
tlater[m]You can then just use BuildStream's yaml parsing to get the set of elements14:16
tlater[m]And run your assertions on those14:16
tlater[m]The problem with this approach is that you'll be dirtying your build output with files that aren't relevant to the actual output14:17
tlater[m]I.e., these manifests will be permanently in your file tree14:17
tlater[m]You probably don't want that when it is actually deployed.14:17
douglaswinshipwell the manifest element wouldn't be included in any of the actual builds14:18
douglaswinshipor rather, it would be it's own build, that's only built when you want to generate the manifest14:18
douglaswinshipif someone builds platform.bst, or platform-sdk.bst, this manifest wouldn't end up in that artefact14:18
tlater[m]Ah, I see what you mean14:19
tlater[m]Yeah, I still think it makes more sense to just write this as the first BuildStream linter14:19
tlater[m]You can call a number of `buildstream.stream` functions to get the list of sources14:19
tlater[m]And then create a manifest from those14:19
douglaswinshipThat does sound interesting. I'm not at all sure where to start though.14:20
tlater[m]douglaswinship: Let me see if I can sketch something for you14:20
douglaswinshipthanks14:20
*** lachlan has joined #buildstream14:21
tpollarddouglaswinship: for what it's worth, I don't think it's illegal to use those accessor methods on a source subclass14:22
douglaswinshiptpollard: probably not. But i can't get the code accepted as it is, on the freedesktop-sdk project. It's failing pylint because of those.14:23
douglaswinshipAnd i think that they're a big clue that I'm not doing this the best way it could be done14:24
tpollardif they were intended to be private, they should be __14:24
douglaswinshiptpollard: I didn't think python really _does_ private?14:24
tme5it doesn't14:24
tpollardbuildstream should consider all accessors as public, even if _ private14:24
douglaswinshipoh, i didn't spot the underscores in your message tpollard. I understand what you mean14:25
tpollardhttps://docs.buildstream.build/master/hacking/coding_guidelines.html has more info14:25
tme5douglaswinship, seems like the issue is between conventions of buildstream and those of typical python14:25
douglaswinshipbut in any case, a linter sounds better if it's an option that we can actually implement.14:26
douglaswinshipI'd quite like to be able to write code that says "look at every element in this project"14:26
douglaswinshipusing a plugin, i have to say "look at every element in this particular dependency graph"14:27
coldtomtlater[m]: is there documentation anywhere for the entire public API surface of bst ooi? i never knew that you could call anything from buildstream.stream, for example14:28
tme5unrelated question: is ok for a plugin to accept multiple "node types" for a config key? like either string or list of strings?14:32
tlater[m]coldtom: That's not public ;)14:33
tlater[m]tpollard: We don't guarantee API stability for those14:33
tlater[m]I.e., it's fine within BuildStraem14:33
tlater[m]*BuildStream14:33
tlater[m]But the leading _ is specifically "we don't promise anything if you use it in a plugin"14:33
tlater[m]So it's definitely detrimental to douglaswinship's use case14:33
tlater[m]douglaswinship: From some digging, I believe it can be as simple as this: https://hastebin.com/zirirunove.py14:33
tlater[m]The key is the app.initialized method14:33
tlater[m]It's what we use to set up BuildStream in general14:33
tlater[m]Now, the problem is that this is also private14:33
tlater[m]There has been a fair amount of interest in this though, we'd definitely like to make BuildStream projects more generally parseable14:33
tlater[m]So if you want to do something like this, I'd love it if you could send out a ML post about potentially making those surfaces public14:33
tlater[m]So we have a way to write external tools that can analyze BuildStream projects :)14:33
tlater[m]Can you finish my sketch with just that or would you like me to actually build a little POC?14:33
tlater[m]tme5: I don't see any problem with that, per-se. I'm not sure if our accessor methods support that.14:33
tlater[m]But I've definitely written code like that14:33
tlater[m]If our accessor methods don't support that I guess you can just call them twice14:34
tlater[m]We have things like this for our configuration parsing, so I think it's acceptable14:34
tme5ideal. thanks14:37
coldtomi'd definitely like to see that become public api, it'd be great to be able to analyze projects better14:37
tlater[m]coldtom: +114:38
tlater[m]Hence I'm trying to get douglaswinship to do that instead of trying to mess with element plugins :)14:38
benschubertA few things, it might have already been said but aven't noticed it, if so sorry:14:47
benschubertThings that do start with a "_", are NOT part of the API, even when implementing a Source/Element plugin. This is defined in the documentation. Don't expect any stability there and I would recommend against using it.14:47
benschubertIf you need access to more things however, please do open an issue with the rational behind it, and we will probably open the API more/add new accessors for some parts.14:47
benschubertAlso, we don't support calling BuildStream as an API, the only API is the plugins (However, feel free to open a ML post on this, I personnally would like that)14:48
tlater[m]benschubert: Yep, that's basically what we've said14:50
tlater[m]douglaswinship: A more complete POC: https://hastebin.com/puhipubazu.py14:50
tlater[m]I'm not sure it actually works14:50
tlater[m]But it seems to output an empty list for me, at least14:50
tlater[m]Ah, I need to get the element dependencies...14:50
tlater[m]There we go14:52
tlater[m]https://hastebin.com/etowesulub.py14:52
tlater[m]It's actually a lot less painless than I thought14:52
tlater[m]IMO, since you'll anyway end up having to use private API, it would be nice if we could start a discussion on linter things like this, and make the API public in general14:53
tpollardIs there any merit in something similar to that becoming a source subcommand?14:55
tlater[m]I've made a snippet for it: https://gitlab.com/BuildStream/buildstream/snippets/192808814:55
tlater[m]tpollard: Hm, I don't think so at the moment. We have support for real source mirroring,14:57
tlater[m]This is just a convention a project follows to provide a different, less infrastructure heavy mirroring14:57
benschuberttpollard: like 'bst source lint' ? I'm not sure, we would need that on elements too right? It's a tricky question (even though I'd like to have linters made for that and would have having to parse this yaml myself)14:58
*** lachlan has quit IRC14:59
tlater[m]benschubert: I've liked the idea of a `bst source lint` in the past14:59
tlater[m]But I think it'd be more flexible to allow generic access to our API14:59
benschubertdepends how much of the api we make public :)15:00
tlater[m]That's true, I suppose15:00
tlater[m]Still, for this specific case I don't think it makes sense to be opinionated on our end15:00
tlater[m]Even if in general linting would be a nice thing15:00
benschubertbecause I like not having to much public, it's easier for improving performance, even though I end up having to break the plugin APIs way too often fo rthis15:00
benschubertI'm not sure project should be visible15:01
benschubertbut I definitely think we could make other things visible15:01
benschubertlike the mirrors15:01
gitlab-br-botjjardon merged MR !1789 (chandan/no-warn-bwrap->master: setup.py: Remove check for bubblewrap) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/178915:07
*** lachlan has joined #buildstream15:08
*** lachlan has quit IRC15:12
*** lachlan has joined #buildstream15:21
*** narispo has quit IRC15:21
*** narispo has joined #buildstream15:21
tme5how is it possible for a scalar yaml node to be None? Like can it be read as None from a yaml document?15:23
tme5just wondering if I need to sanitise against it somehow15:24
*** lachlan has quit IRC15:24
benschuberttme5: https://yaml.org/type/null.html yep15:24
benschubertyou can  do `if node.is_none()` to check for this15:25
tme5great, ty15:26
benschuberttme5: worst case, if you do `node.get_str()` and it is None, you'll get an exception too15:27
benschubertwhich is handled by BuildStream at any moment too, so worst case you can let buildstream report that it should not be None15:27
cs-shadowtlater[m]: I slightly disagree with `lint`. I think we should do formatters not linters if we do anything in that space. Like `black` but for bst15:48
tlater[m]cs-shadow: Ah, yes, I definitely agree with that15:48
* cs-shadow believes that `go fmt` is a great example of this thing really working15:49
*** nielsdg has left #buildstream15:49
tlater[m]Any "best practice" issues should just be forbidden entirely.15:49
tlater[m]Or raise warnings during builds15:49
tlater[m]Instead of requiring an additional step15:49
tlater[m]I just used "lint" there because we happened to be talking about a case that would require linting, not formatting.15:50
cs-shadowI think if we can't format it automatically, and can't warn about it either, then maybe it should not live as part of the `bst` tooling, I.e. if we want a linter that takes care of more than just formatting issues15:52
*** bochecha_ has joined #buildstream15:58
*** bochecha has quit IRC15:58
*** bochecha_ is now known as bochecha15:58
tlater[m]cs-shadow: Yes, hence I'm suggesting this particular thing should go through the bst API, and that we should make some of it public for cases like this, as sketched in that snippet :)15:58
tlater[m]For reference, this is a project trying to assure that they use the `alias:` option for every source in the project16:00
cs-shadowI'd really like some part of our API to become public, although I'm probably most interested in the stream API. Or if not that, a Python equivalent of the CLI, so that one could avoid shelling into `bst` when writing a tool that uses it16:00
cs-shadowalthough I'd feel very nervous about opening that up now as-is :)16:01
tlater[m]We probably don't want that as a generic warning/error16:01
tlater[m]But it does make sense in that context16:01
tlater[m](Also ensuring that any and all aliases actually function)16:01
tlater[m]cs-shadow: Yeah... Though hopefully douglaswinship can get a thread started on this.16:09
*** lachlan has joined #buildstream16:22
*** delli3 has quit IRC16:24
*** delli3 has joined #buildstream16:25
*** lachlan has quit IRC16:26
*** bochecha has quit IRC16:34
*** lachlan has joined #buildstream16:42
*** narispo has quit IRC16:44
*** narispo has joined #buildstream16:45
*** lachlan has quit IRC16:47
*** lachlan has joined #buildstream16:49
*** lachlan has quit IRC16:53
*** lachlan has joined #buildstream17:01
*** lachlan has quit IRC17:05
gitlab-br-bottraveltissues opened (was WIP) MR !1761 (traveltissues/1216->master: mtime support) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/176117:12
*** traveltissues has quit IRC17:19
tlater[m]Is marge still broken?17:22
tlater[m]Should I just use the button?17:22
gitlab-br-bottlater merged MR !1790 (tlater/fix-monkeypatch-warning->master: tests/artifactcache/config.py: Cast our tmpdir to a string) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/179017:22
* tlater[m] assumes so17:22
*** lachlan has joined #buildstream17:46
*** phildawson_ has joined #buildstream17:47
*** tme5 has quit IRC17:47
*** phildawson-ct has quit IRC17:48
*** lachlan has quit IRC17:49
*** paulsherwood has quit IRC18:07
*** paulsherwood has joined #buildstream18:07
*** adds68 has quit IRC18:07
*** ikerperez has quit IRC18:07
*** bethw has quit IRC18:11
*** robjh has quit IRC18:11
*** jward has quit IRC18:11
*** laurence has quit IRC18:12
*** robjh has joined #buildstream18:15
*** jward has joined #buildstream18:16
*** bethw has joined #buildstream18:16
*** laurence has joined #buildstream18:16
*** lachlan has joined #buildstream18:17
*** lachlan has quit IRC18:21
gitlab-br-bottlater closed issue #1135 (We often use a string instead of a PipelineSelection in Stream) on buildstream https://gitlab.com/BuildStream/buildstream/issues/113518:24
gitlab-br-bottlater merged MR !1787 (tlater/pipeline-enums->master: Make PipelineSelection a proper enum type) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/178718:24
benschubertnice :D18:29
*** tiagogomes has quit IRC18:35
*** douglaswinship has quit IRC18:40
*** paulsherwood has quit IRC18:40
*** WSalmon has quit IRC18:42
*** robjh has quit IRC18:42
*** laurence has quit IRC18:42
*** jward has quit IRC18:42
*** bethw has quit IRC18:42
*** valentind has quit IRC18:42
*** laurence has joined #buildstream18:44
*** robjh has joined #buildstream18:44
*** jward has joined #buildstream18:45
*** WSalmon has joined #buildstream18:46
*** bethw has joined #buildstream18:46
*** valentind has joined #buildstream18:47
*** jonathanmaw has quit IRC18:47
*** phildawson_ has quit IRC18:58
*** lachlan has joined #buildstream19:13
*** lachlan has quit IRC19:18
*** adds68 has joined #buildstream20:06
*** lachlan has joined #buildstream20:10
*** lachlan has quit IRC20:23
*** lachlan has joined #buildstream20:52
*** lachlan has quit IRC20:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!