IRC logs for #buildstream for Tuesday, 2018-12-11

*** xjuan has joined #buildstream00:03
*** lsfranco has quit IRC01:03
*** lsfranco_ has joined #buildstream01:03
*** lsfranco_ is now known as lsfranco01:04
*** ajsalminen has joined #buildstream01:08
*** alatiera has quit IRC01:13
*** xjuan has quit IRC02:47
*** kapil___ has joined #buildstream02:52
*** flatmush_ has joined #buildstream03:07
*** flatmush has quit IRC03:08
*** nimish has quit IRC03:25
*** lsfranco has quit IRC04:08
*** kapil___ has quit IRC05:01
*** shinchiro has joined #buildstream05:10
*** tristan has joined #buildstream06:32
*** youtah has joined #buildstream06:47
*** tristan has quit IRC08:04
*** WSalmon_ has joined #buildstream08:37
*** WSalmon_ has quit IRC08:39
*** tristan has joined #buildstream08:42
*** WSalmon_ has joined #buildstream08:53
*** alatiera has joined #buildstream08:55
*** WSalmon_ has quit IRC08:58
*** ChanServ sets mode: +o tristan08:59
paulsher1oodtristan: i'm worried that even after all this time we may have a different understanding of cache-key compatibility09:11
paulsher1oodmy understanding is that *desired* behaviour is that cache-key-algorithm is versioned (and by that i mean whole number versions, not semver or anything fancy)09:13
paulsher1oodonce we have a version of the algorithm, bst (or any other tool) should be able to calculate it09:14
paulsher1oodif we bump the version (because we realise we've missed an input/factor, say) we add support for the new version into bst (or any other tool)09:14
tristanpaulsher1ood, I think that we have the same understanding, except for the "any other tool" part09:14
*** phildawson has joined #buildstream09:15
paulsher1oodi don't see why bumping the version means that a later version of bst can't support all previous versions of the algorithm09:15
paulsher1oodand hence i don't see why that was not done from day one on bst09:15
paulsher1oodany other tool could mean, for example that some other tool might want to figure out the cache-key for a given set of bst files using some particular version of the algorithm09:16
tristanpaulsher1ood, That was not done from day one because of the efforts to support every artifact format version would have been high and caused a lot more friction during the feature development09:17
tristanpaulsher1ood, Another script/tool should be able to use `bst show` with a specific artifact version set (when we support setting explicit artifact versions), but it could not parse a project and determine it by itself09:18
*** toscalix has joined #buildstream09:18
paulsher1oodtristan: i think think that was simply a mistake. the effort is not that high09:20
paulsher1oodwe're just talking about some kind of hash of some inputs, after all.09:20
tristanpaulsher1ood, Not really no09:23
tristanpaulsher1ood, Not all plugin configurations contribute to a cache key, and plugins need to contribute what goes into the cache key09:23
paulsher1oodit's still just a hash of inputs09:24
paulsher1ooda plugin is either providing some inputs or not09:24
tristanAnd the configuration in the YAML is loaded by the plugin, the plugin is the only one who knows what that configuration means09:25
paulsher1oodwell, 2 hashes, given the strong and weak approachs09:25
tristanAnd which part of that configuration needs to contribute to the cache key or not09:25
gitlab-br-bottristanvb opened MR !1004 (tristan/stop-stripping-whitespace->master: Stop stripping whitespace) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/100409:42
*** jonathanmaw has joined #buildstream09:53
jmacDoes anyone know where we raise issues against the remote execution protocol these days? I'm guessing https://github.com/bazelbuild/remote-apis/issues09:56
juergbijmac: yes, I think that's the right place. that said, I don't see an actual problem in the protocol from the ML thread10:00
juergbi(although it could probably use some clarification in the wording)10:01
jmacOK, let's say my server supports instance names, and I request this: /uploads/6c92172c-8064-4351-93a2-640d5e8761fe/blobs/187d384348c73a2c0246a42fd061167039e551c1fe8c24a51d9538f4536fa72c/1034/uploads/6c92172c-8064-4351-93a2-640d5e8761fe/blobs/187d384348c73a2c0246a42fd06116\10:01
jmac7039e551c1fe8c24a51d9538f4536fa72c/1034/uploads/6c92172c-8064-4351-93a2-640d5e8761fe/blobs/187d384348c73a2c0246a42fd061167039e551c1fe8c24a51d9538f4536fa72c/1034/foo.c10:01
jmacWhat's the instance name?10:01
*** phildawson has quit IRC10:02
*** phildawson has joined #buildstream10:02
*** bilelmoussaoui has joined #buildstream10:04
*** tpollard has joined #buildstream10:05
*** bilelmoussaoui has left #buildstream10:05
valentindtristan, would be nice to get !1000 merged and then backport to put it in the coming 1.2 release.10:12
gitlab-br-botMR !1000: Force updating tags when fetching git repository https://gitlab.com/BuildStream/buildstream/merge_requests/100010:12
juergbijmac: if I were to implement this on the server side, I would use component-level mapping. and would reject resource names that start with a /10:13
jmacI don't know what you mean by component-level mapping10:14
juergbijmac: i.e., the instance name would depend on the server configuration. if the server is configured supporting with an instance with an empty name, your URL (if it didn't start with a /) I'd map to that instance with an empty name10:14
juergbiif there is no such instance configured, I'd check whether the server configuration has an instance whose first component is called 'uploads'10:15
juergbilooking at the buildfarm code, it does not appear to follow the protocol, though, it requires {size} to be the last URL component10:17
juergbiI have to admit that the whole thing is pretty odd and probably worth raising an issue10:17
jmacBut you could legitimately have two instances, one called /uploads/6c92172c-8064-4351-93a2-640d5e8761fe/blobs/187d384348c73a2c0246a42fd061167039e551c1fe8c24a51d9538f4536fa72c/1034/ and another being that duplicated10:18
juergbiyes, you couldn't allow that in the server config10:18
jmacIn which case, you may as well make a simpler rule and say slashes are not allowed in instance names for your particular server10:19
juergbiinstance name followed by 'uploads' must not be a prefix of another instance name10:19
jmacBut then you're applying a patch to a hole in the protocol, IMO10:19
juergbiyou could argue that, I guess10:19
juergbiyou could also argue it's disallowing impossible configs10:20
juergbi(besides 'uploads', also 'blobs' must be disallowed)10:20
juergbiand actions:execute and a few others10:20
juergbiin any case, I agree it's worth upstream discussion10:21
juergbishould at the very least be clarified10:21
*** bilelmoussaoui_ has joined #buildstream10:26
*** bilelmoussaoui_ has left #buildstream10:26
*** lachlan has joined #buildstream10:38
*** nimish has joined #buildstream11:00
*** nimish has quit IRC11:15
*** nimish has joined #buildstream11:15
*** nimish has joined #buildstream11:16
tristanvalentind, !1000 looks fine to me11:19
gitlab-br-botMR !1000: Force updating tags when fetching git repository https://gitlab.com/BuildStream/buildstream/merge_requests/100011:19
tristanvalentind, I don't like how the summary talks about multiple remotes and mirroring, but I don't think the actual patch regresses or changes anything about that11:20
tristanvalentind, I.e. if the behavior of fetch() or track() is impacted by having multiple remotes known for different mirrors, that would seem to be a bug, I think we are very explicit about fetching/tracking from one remote at a time11:21
tristanand if we are not, then the git plugin would appear to be breaking rules11:21
valentindtristan, Interestingly, on the clone, we use "origin".11:22
valentindThen on fetch, we create a new remote for the same url.11:22
tristanSeems like that is buggy indeed11:23
valentindWhen we use mirrors they might have different urls.11:23
tristanI guess the initial clone can only come from one of the mirrors11:23
valentindI think  something also broke when we started to set "alias_override" even when we did not use a mirror.11:25
valentindI remember the code for mirroring in source.py slightly different.11:26
valentindSo there are some more weird thing happening. However the test case I added legitimately requires a different remote. It is just that we seem to use extra remotes when we do not need to.11:30
valentindin _fetch, the "if alias_override:" is a wrong condition. Instead we should look if "origin" has the required url.11:31
tristanvalentind, I think that is a separate issue from your patch, though11:43
valentindtristan, yes.11:44
tristanvalentind, I.e. you patch fixes that the fetch should pull in refs, and by that virtue I think is still good, and doesnt make things worse11:44
tristanSo lets go ahead with your patch + backport as you said11:44
tristanvalentind, it would of course be good to raise that as a separate issue ;-)11:44
valentindOK, I just test changing last_commit, then I merge and backport.11:45
*** alatiera has quit IRC11:47
*** alatiera has joined #buildstream11:52
*** abderrahim has quit IRC11:53
*** abderrahim has joined #buildstream11:54
tristanvalentind, Right that sounds sensible, I was asking the question there because I genuinely was unsure if the distinction is important11:59
tristanI think that changing the behavior of latest_commit() to use HEAD instead of master will be correct, though11:59
valentindtristan, yes, it worked.12:07
valentindtristan, I fixed that and simplified and clarified the commit message12:13
*** nimish has quit IRC12:20
*** nimish has joined #buildstream12:21
tristanvalentind, I guess go ahead and merge and backport then :)12:24
tristancs-shadow, I thought I'd go ahead and "just fix" that issue about stripping whitespace, since it was such a simple thing to do and has been open for months12:25
tristancs-shadow, unfortunately I hit a wall: https://gitlab.com/BuildStream/buildstream/issues/403#note_12404647612:25
*** nimish has quit IRC12:26
*** nimish has joined #buildstream12:26
*** raoul has joined #buildstream12:33
*** alatiera has quit IRC12:42
gitlab-br-botvalentindavid closed issue #812 (Not all git commits fetched.) on buildstream https://gitlab.com/BuildStream/buildstream/issues/81212:44
gitlab-br-botvalentindavid merged MR !1000 (valentindavid/git_force_fetch_tags->master: Force updating tags when fetching git repository) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/100012:44
*** nimish has quit IRC12:51
*** nimish has joined #buildstream12:52
*** phildawson_ has joined #buildstream12:55
*** phildawson has quit IRC12:56
*** lachlan has quit IRC13:03
jonathanmawWSalmon: I've pushed some more changes to !92413:08
gitlab-br-botMR !924: Support invoking buildstream from a workspace outside a project https://gitlab.com/BuildStream/buildstream/merge_requests/92413:08
jonathanmawspecifically: I've made it search for both the project.conf and the .bstproject.yml file; I've extended the tests to cover trying to run commands from subdirs of the workspace; and also the rest of the nitpicks.13:09
cs-shadowtristan: that's a shame, I thought it'd be a small one-liner patch, but apparently not13:09
cs-shadowtristan: FWIW I would mention that for our use-case, we don't really need this at the moment. And, it's not usually very bad when it happens, so it may not be very high priority item13:11
*** tristan has quit IRC13:12
*** lachlan has joined #buildstream13:19
gitlab-br-botadds68 closed issue #816 (Unexpected caching behaviour with bst plugin) on buildstream https://gitlab.com/BuildStream/buildstream/issues/81613:20
*** nimish has quit IRC13:22
*** nimish has joined #buildstream13:23
Nexusjuergbi: could you have a look at https://gitlab.com/BuildStream/buildstream/merge_requests/925 and https://gitlab.com/BuildStream/buildstream/merge_requests/926 for me pls? :)13:33
juergbiadding them to my list13:33
Nexusthanks c:13:33
juergbiNexus: do you think they are ready for review or do you have questions?13:34
juergbithey are marked WIP13:34
Nexusjuergbi: i think they're ready for review, i haven't done the other commands, as i wanted to put that as a seperate MR13:35
Nexusbut i wanted to make sure it matched what you expected13:35
gitlab-br-botknownexus opened (was WIP) MR !925 (issue-638-validate-all-files->master: Added --all functionality to `bst show`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/92513:35
gitlab-br-botknownexus opened (was WIP) MR !926 (issue_640-Build-All->master: Added build all functionality to `bst build`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/92613:35
*** tristan has joined #buildstream13:36
*** nimish has quit IRC13:36
*** ChanServ sets mode: +o tristan13:37
tristancs-shadow, good to hear it's not that high priority13:37
tristancs-shadow, maybe we can consider an only variables-get-stripped approach, but then it might have higher impact breakage on plugins13:38
tristanindeed the suspected behavior of auto stripping unless quoted would have been great13:38
cs-shadowyeah, if I find time for this, I'll look into the spec and ruamel to see if the library is doing the right thing13:40
tristanI did some quick googling but was unable to easily figure it out13:41
tristanthere is a lot of material on "how to quote" for yaml13:41
tristanI didnt try parsing the actual spec, though13:42
cs-shadow:/ otherwise, stripping only variables may make sense but it may also make things more confusing13:42
tristanit has a higher chance of breaking plugins, though13:42
tristanas the default behavior would change to leading/trailing newlines for any string specified with the "|" thing13:43
tristankind of drastic change, sort of feels like forcing ruamel.yaml to strip everything that is not quoted, even if defying the spec, would be better hehe13:43
cs-shadowto be honest, I don't think that a lot of plugins will be relying on whitestpace getting stripped but I may be wrong on that. But, I'll look more into it later in the day to see if we can get the desired behavior of only stripping non-quoted strings13:45
tristancs-shadow, "track" parameters13:46
tristanwe think that people specify them as "track: branch-name", but *if* they specify it as "track: |" and then specify the string below, suddenly the plugin gets newlines13:47
tristanthat is weird13:47
tristanany string parameter really, if the user decides to express it that way, boom13:47
cs-shadowusing "|" shouldn't have a leading newline as per the spec IIUC13:50
cs-shadowas per https://yaml.org/spec/1.2/spec.html#|%20literal%20style//13:51
coldtomtristan: would using ">" rather than "|" not strip the newlines?13:52
tristancs-shadow, so the fact that we usually put an empty line between the | and the string for better readability, causes the leading newline perhaps, and the trailing newline is just not optional ?13:53
cs-shadowyeah the leading newline will be because of the explicit empty line and not if the text immediately follows "|" on the next line13:54
tristancoldtom, perhaps, I'm not familiar enough, but I've seen "|>" used and wondered what it means13:54
tristancs-shadow, I would have expected that the string starts on the first line with indentation13:54
cs-shadowhttps://stackoverflow.com/a/21699210 is a good summary of all the nine (or 63, depending on how you count) different ways of having new lines in yaml13:55
tristancs-shadow, i.e. in the test case I posted on the issue, the empty line is an empty line, not aligned with the beginning of the string13:55
cs-shadowtristan: yes, preserving that empty line is the expected behavior IMHO13:56
tristancs-shadow, interesting, so we *could* say that you can use ">-" to explicitly strip13:57
tristancs-shadow, not sure what you mean or if you agree or disagree... I am saying that that empty line should be ignored by YAML and really be meaningless, *because* it is different from a line with 2 leading spaces followed by a newline13:58
tristanif I wanted to express an empty leading line, I would put two leading spaces on that line13:58
tristanbut this is just opinion at this point :)13:58
cs-shadowtristan: ah! I think I am saying that since the user put that newline explicitly after "|", I would expect it to be preserved. That is if we say that we are following the yaml spec13:59
* cs-shadow needs to go to our standup now :)13:59
gitlab-br-botvalentindavid opened MR !1005 (valentindavid/git_force_fetch_tags-1.2->bst-1.2: [backport 1.2] Force updating tags when fetching git repository) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/100514:00
gitlab-br-botknownexus closed MR !926 (issue_640-Build-All->master: Added build all functionality to `bst build`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/92614:14
gitlab-br-botknownexus reopened MR !926 (issue_640-Build-All->master: Added build all functionality to `bst build`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/92614:15
gitlab-br-botvalentindavid merged MR !1005 (valentindavid/git_force_fetch_tags-1.2->bst-1.2: [backport 1.2] Force updating tags when fetching git repository) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/100514:25
WSalmonmy mr for chosing werethere or not to use a build tree for the bst shell --build https://gitlab.com/BuildStream/buildstream/merge_requests/986 is now ready for review :D14:33
WSalmonthanks Kinnison and tpollard for looking already14:33
jmacNexus: I'm confused by !925 - where's the '--all' flag for bst show?14:35
gitlab-br-botMR !925: Added --all functionality to `bst show` https://gitlab.com/BuildStream/buildstream/merge_requests/92514:35
Nexusjmac: i've updated t, sorry my description was out of date14:38
jmacAha14:38
*** alatiera has joined #buildstream14:50
coldtomdoes anyone know what the default value of %{libexecdir} is?14:58
coldtomah, found it, sorry for the noise14:59
*** toscalix_ has joined #buildstream15:12
Nexusjmac: re: https://gitlab.com/BuildStream/buildstream/merge_requests/925#note_12410115415:22
NexusWon't it only remove strings that match the whole string provided?15:22
Nexusnot the characters individually15:22
Kinnisonlstrip() takes an iterable of characters15:25
Kinnisonso I think jmac is right15:25
jmacTry it: "hello".lstrip("eh")15:26
Kinnison>>> element="elements/elementsong.bst"15:28
Kinnison>>> element.lstrip("elements/")15:28
Kinnison'ong.bst'15:28
Kinnison>>>15:28
*** toscalix_ has joined #buildstream15:28
tpollardthat's caught me a number of times15:30
jmacEsp. since "elements/elementsong.bst".lstrip("elements") appears to do what you want15:35
skullmanblimey, I've gone years with that happening to work as I expected every time15:35
KinnisonGiven Python 3 has some special stuff for managing paths, does it have a path-prefix stripper?15:38
skullmannot that I can see, though there might be something weird in pathlib15:39
jmacos.path.relpath should do it, I think15:40
*** kapil___ has joined #buildstream15:44
Nexuskk, updated that15:45
gitlab-br-botjonathanmaw merged MR !924 (jonathan/workspace-fragment-create->master: Support invoking buildstream from a workspace outside a project) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/92416:01
*** toscalix has quit IRC16:54
*** offus has joined #buildstream16:59
*** tiagogomes has quit IRC17:13
*** tpollard has quit IRC17:23
*** nimish has joined #buildstream17:29
*** jonathanmaw has quit IRC18:00
gitlab-br-botjjardon closed issue #609 (CAS server doesn't seem to automatically remove artifacts so it gets fill up) on buildstream https://gitlab.com/BuildStream/buildstream/issues/60918:03
jjardonvalentind: thanks a lot for all your work there ^18:03
*** tristan has quit IRC18:04
valentindThanks18:06
*** raoul has quit IRC18:19
jjardonvalentind: are you ok if I add the "blocker" label to https://gitlab.com/BuildStream/buildstream/issues/806 ?18:23
valentindjjardon, Sure.18:25
jennisahh, I just ran into this one18:26
jennischanging x86_64 -> x86-64 fixed everything for me... then checking out a commit before !969 breaks with the same error again18:28
gitlab-br-botMR !969: Execution environment reqs https://gitlab.com/BuildStream/buildstream/merge_requests/96918:28
jennislooks like enforcing a '- 'rather than a '_' is intentional18:30
jennisas there are a lot of explicit x86_64 -> x86-64 changes in the diff18:31
*** kapil___ has quit IRC18:33
*** nimish has quit IRC18:39
*** nimish has joined #buildstream18:40
*** johnward has quit IRC18:44
*** jmac has quit IRC18:44
*** benbrown has quit IRC18:44
*** laurence has quit IRC18:44
*** jmac has joined #buildstream18:44
*** benbrown has joined #buildstream18:44
*** paulsher1ood has quit IRC18:44
*** ikerperez has quit IRC18:44
*** adds68 has quit IRC18:45
*** coldtom has quit IRC18:45
*** jennis has quit IRC18:45
*** paulsherwood has joined #buildstream18:45
*** coldtom has joined #buildstream18:47
*** johnward has joined #buildstream18:48
*** laurence has joined #buildstream18:48
*** nimish has quit IRC18:59
*** nimish has joined #buildstream19:00
*** nimish has quit IRC19:10
*** nimish has joined #buildstream19:10
*** lachlan has quit IRC19:12
*** adds68 has joined #buildstream19:12
*** nimish has quit IRC19:25
*** nimish has joined #buildstream19:26
*** nimish has quit IRC19:41
*** nimish has joined #buildstream19:41
cs-shadowjuergbi: Kinnison: hi, do either of you remember why the ".bst" extension check was added as a warning and not a LoadError?19:44
Kinnisoncs-shadow: IIRC it happens outside of the loader too19:44
Kinnisoncs-shadow: Erm19:44
Kinnisoncs-shadow: Or perhaps it was simply that we were worried we'd break existing projets19:45
juergbihttps://mail.gnome.org/archives/buildstream-list/2018-November/msg00067.html19:45
juergbitristan suggested it in the above mail19:45
juergbiyou can make the warning fatal19:45
KinnisonEither way, good luck :-)19:46
* Kinnison heads off to the gym19:46
cs-shadowI see, thanks!19:46
cs-shadowjuergbi: I was asking this mostly because I was planning to implement a check for valid element names, and was wondering whether that should an error or a warning19:47
cs-shadowI was thinking that having invalid characters in the filename should probably be an error, but it seemed a bit odd to have the invalid suffix produce warnings, while the invalid chars check produces errors19:48
cs-shadowjuergbi: do you have any preference on this?19:48
juergbiI'd keep the two consistent19:49
cs-shadowjuergbi: I feel the same, but I don't think that having invalid chars in the name should probably be an error, as it hurts more than just discoverability. Perhaps this is a topic for the ML thread19:51
cs-shadowsorry, that should have been "I _think_ that  it should be an error"19:52
juergbiwell, except for very few exceptions (such as `:`), these names will typically still work19:53
juergbion the same platform, that is19:53
juergbiwe restrict the subset in order to reduce issues e.g. across platforms19:54
juergbiand with different unicode normalization and things like that19:54
juergbiso I don't think it's that bad if it's only a warning19:54
cs-shadowhmm, I was thinking mostly about `:` and that may produce unexpected behavior19:55
juergbihm, actually, can we even detect invalid `:`?19:56
juergbionce we support junctions with `:`, we interpret this specially and won't warn/error about it in a dependency list19:56
juergbi(i.e., the error would then be that it would complain that the specified junction doesn't exist)19:57
juergbiso we could only warn about those unsupported characters that actually still mostly work19:58
cs-shadowfair point! I didn't think about how to catch this once we support the junctions syntax19:59
cs-shadowyeah, I guess a warning makes sense in that case20:00
cs-shadowit's a shame though we won't catch the only invalid character that may really cause issues, but oh well :)20:00
*** phildawson_ has quit IRC20:15
*** phildawson has joined #buildstream20:22
*** xjuan has joined #buildstream20:25
*** nimish has quit IRC20:30
*** phildawson has quit IRC20:34
*** alatiera has quit IRC21:31
*** juergbi has quit IRC21:42
*** juergbi has joined #buildstream21:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!