IRC logs for #buildstream for Friday, 2018-09-14

*** xjuan has quit IRC00:28
*** tristan has joined #buildstream02:25
*** ChanServ sets mode: +o tristan02:25
*** tristan has quit IRC04:53
*** bochecha has joined #buildstream05:59
*** finn has joined #buildstream07:18
*** iker has joined #buildstream07:19
*** tristan has joined #buildstream07:39
*** ChanServ sets mode: +o tristan07:40
qinustyhttps://gitlab.com/BuildStream/website/merge_requests/85/ is looking for contributions, feel free to add any terms which you wish to define in the glossary as commits or comments in the MR.08:01
*** toscalix has joined #buildstream08:11
*** iker has quit IRC08:13
*** phildawson has joined #buildstream08:14
*** iker has joined #buildstream08:14
qinustyIs there a potential for a tag relating to MR's which are ready to be reviewed within buildstream? I realise the WIP system is in place, however I feel that a `Ready For Review` tag would make searching for MR's which need reviewing easier to filter and process. When reviewed and going back to further work, the reviewer can remove the tag which is08:15
qinustymuch easier than editing the title of the MR. Perhaps toscalix can weigh in on this?08:15
coldtomisn't that what the 'Review' tag is for?08:16
*** rdale has joined #buildstream08:22
finnI've configured the autotools example to use remote execution. I can build it. I can pull it. Though I can't checkout08:40
finnError while staging dependencies into a sandbox: 'Artifact hello.bst is configured to use remote execution but has no push remotes. The remote artifact server(s) may not be correctly configured or contactable.'08:40
finnI do have push remotes and surely for a checkout, this shouldn't be an issue?08:41
*** jonathanmaw has joined #buildstream08:43
finnIf I remove my remote configuration, I can checkout08:44
qinustyIs there a review tag coldtom?08:51
gitlab-br-botbuildstream: merge request (richardmaw/builddir-sockets->master: Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type.") #783 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78308:53
coldtomi was thinking of "verify"08:53
jmacCheckout needs to stage dependencies, so it does require remote execution08:54
finnEven after I've run the build and pulled everything into my local cache?08:55
jmacYes. Staging runs integration commands which are no different to build commands.08:56
jmacI realise it isn't ideal08:56
finnI have my remote still setup though08:57
finnSo even so, it shouldn't be an issue08:57
jmacAnd there was nothing in the initialization messages from buildstream indicating a problem with the artifact cache?08:59
jmacDid you see [00:00:00][][] SUCCESS Initializing remote caches09:00
finnls09:04
finnnope09:05
finnI get09:05
finnhttps://pastebin.com/wDpDerfx09:05
jmacOh, interesting, I don't think it's even tried to setup the remote09:05
jmacPerhaps bst doesn't bother if it's just doing a checkout09:06
gitlab-br-botbuildstream: merge request (richardmaw/builddir-sockets->master: Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type.") #783 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78309:06
jmacI'll have to log it as an issue, I'm in the middle of something else at the moment09:07
finnI'll send you the details for the cluster we have set up here too09:11
tiagogomestristan I need some design help here. Recently, a pull_tree method was added to the artifact cache. This method is run from SandboxRemote and adds content the artifact cache. I need to communicate the size of the content added to the main process for dynamically update the artifact cache. But having the sandbox class return that data seems odd to me09:11
tiagogomesAny idea for a nice and clean solution?09:12
*** abderrahim has quit IRC09:14
*** abderrahim has joined #buildstream09:15
tristanHrrrrrm09:15
tristanThis is a bit curious, I think it's better to take a step back and consider the design of SandboxRemote first09:16
tristanIs it, per chance... doing more than what a Sandbox should do ?09:16
tristantiagogomes, let me safely task switch, I'm trying to add a regression test for `bst build --track` which accidentally deletes required artifacts09:17
tiagogomessure, no rush09:17
tristanhttps://gitlab.com/BuildStream/buildstream/blob/master/buildstream/sandbox/_sandboxremote.py#L5909:20
tristanFirst question to ask here is... Is this a different artifact cache ?09:20
tristanIf so... Why ?09:20
tristanA lot of thinking to do around here, I have it on the menu to examine the design with juergbi and jmac, and have a lot of catching up to do09:21
tristaninput from juergbi would be good09:21
tristanon what is currently intended09:21
jmacNope, same cache as the rest of buildstream09:22
*** ikerperez has joined #buildstream09:22
tristanBut honestly, we need to look at the big picture; I have a feeling that our artifact cache is only a part of the local CAS; and we probably intend that the quota applies to anything stored in the CAS, not only the artifacts09:23
*** iker has quit IRC09:23
tristanjmac, interesting, how come we are not using the regular Platform.get_platform() + platform.artifactcache to access it ?09:23
qinustyThis includes the build/ folder tristan? I was seeing this as an issue when looking into the issues with filling up on disk space09:24
tristanSeems dangerous to have multiple instances of the CasCache with different state, accessing the same store no ?09:24
tristanqinusty, It does not include the build/ directory09:24
qinustyNot currently. Which is the problem09:24
tristanqinusty, I believe that the build/ directory is coined for automatic cleanup *all the time* in the technical debt follow up issue related to caching of failed builds09:24
qinustybuild/ is part of the cache directory, and is a large contributor towards storage during a build.09:25
tristanqinusty, I have outlined everything that has to happen to that build/ directory in the related issue09:25
tristanqinusty, search for Blocker issues for milestone 1.4 to find it quickly09:25
gitlab-br-botbuildstream: merge request (issue-642-Invalid_project.conf_seen_as_missing->master: Incorrect error when malformed project.conf) #792 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79209:25
qinustyOkay09:25
tlater[m]qinusty: Also, only *failed* builds remain in build/, so it should generally not explode like the old artifact cache.09:27
tristanDon't mix artifact caches with the build/ directory09:27
tristanWhile it would have been a good idea from the start to offer only one directory as configuration where we would have stored build/ and artifacts/ underneath from the beginning, we did not09:28
tristanAlso, unrelated to this, those are separate, unrelated things; builds dont happen in the artifact cache09:29
jmacIf Platform is available inside a sandbox, I agree platform.artifactcache() would be a better way of obtaining it09:29
tristanjmac, Platform.get_platform() is a class method of Platform09:29
tristani.e. you don't need an instance, it's a singleton09:29
qinustyMy concerns were regarding filling a disk. When you're approaching a full disk. You have 2G headroom, you build in build/ then copy to artifacts/ meaning you need double the size of that current artifact available at the time of copy?09:30
gitlab-br-botbuildstream: merge request (issue-642-Invalid_project.conf_seen_as_missing->master: Incorrect error when malformed project.conf) #792 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79209:30
*** lachlan has joined #buildstream09:31
tristanqinusty, Ignore the fact that they exist on the same partition, and read the issue09:31
tristanqinusty, My point to tlater[m] is not to mix up terminology and concepts and make everyone confused as a result09:32
tristanThe build directory is one thing, the source cache is another thing (and another can of worms), the artifact cache is another thing09:32
tristanThose are all things, not one thing09:32
* tlater[m] apologizes for being ambiguous here09:33
toscalixqinusty: you are trying to solve with labels what should be solved by assigning the MR to a person or several people at the same time.09:37
toscalixIn any case, if you think the label is the best way, I will not stopped. I am not a gate keeper in this project09:38
toscalix.. stop it...09:38
toscalixadding labels makes the usage of the tool more expensive. And the real benefits comes when everybody use them. The principle is... less features used by everybody instead of more used by just a few09:40
qinustyI agree this would work in some cases toscalix, but I feel it is easier for everyone if there are a collection of MRs awaiting reviews and people took some time to review MRs throughout the week. Assigning to a specific person means you need to pick people (who aren't clearly defined currently) and the same people will end up being assigned MR afte09:40
qinustyr MR.09:40
adds68Is there a document anywhere that explains what each label is used for and where they should be applied?09:41
toscalixchecking the overall usage of the high amount of labels we have.... I would claim we are making a mistake, making the usage of the tool more and more expensive and, at some point, we will not be able to pay the price09:41
qinustyWell perhaps we need a clean up of the ones we do not use?09:41
qinustyRather than considering the negative impact of adding new labels09:41
toscalixadds68: I did with the ones originally. Thos who have created the new ones have not followed that process, I am afraid09:42
toscalixadds68: https://gitlab.com/BuildStream/nosoftware/alignment/blob/master/BuildStream_policies.md09:42
toscalixqinusty: erasing labels is tempting... but is a bad practice09:43
toscalixit might have little impact on our project at this time09:43
toscalixbecause we are not doing reports or analising any past data09:44
toscalixat some point we will09:44
toscalixin any case, erasing the labels does not solve the real problem.... we are creating new labels while we are not using as we should the ones we already have09:44
adds68toscalix, why is this in a separate project? If the labels are so important, they should be presented to the user via the readme or contributing page09:45
gitlab-br-botbuildstream: merge request (richardmaw/builddir-sockets->master: Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type.") #783 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/78309:46
toscalixthat is questionable09:46
toscalixyou are assuming that gitlab will be use only by code contributors09:46
toscalixthe structure does not only affect to the buildstream repo09:47
toscalixas I mentioned yesterday, I think policies are part of the governance model. There used to be a time where only devs care about them. Today covernance is part of the decision process to adopt a tool or not09:49
adds68toscalix, the service primarily exists for code contributors, there are other things such as a wiki for people who are not well versed in git09:49
toscalixthe audience for these policies is wider than developers09:49
adds68toscalix, or your website :)09:49
toscalixthe policie was created beforethe website09:49
toscalixand yes, if you check the content structure document and the sitemap, the plan is to include the policies on the website09:50
toscalixsadly there is now a discussion that intends to put part of the policies in the buildstream repo09:50
adds68The policies are buildstreams policies, would that not make sense?09:51
tlater[m]toscalix: Personally, I wasn't aware we had these definitions. I think I agree with adds68 that label definitions should at least *also* be in our readme.09:51
toscalixthe policies are part of the governance model, that transcend the contributors09:51
adds68nobody is going to go to a separate git project to find policies, they will just assume you don't have any09:52
phildawsonperhaps referencing the policies document on the contributing page would help09:52
toscalixadds68: the policies affecting the website, the graphic design, the roadmapping.....09:52
adds68toscalix, nobody cares about those, they come for BuildStream09:53
toscalixall those are part of the governance model of a project.09:53
adds68If they care about the website, they will go there09:53
tristanThere is not IMO much reason to fragment all of the things into separate repos; The HACKING.rst document for instance is accessible via the "Contributing" link in the documentation: http://buildstream.gitlab.io/buildstream/09:53
toscalixadds68: nobody that you know care about those. Look at a wider scope. In order to be successful, wqe need to attract people that are not like ours09:54
tristanThe trick is not to make everything more complex with fragmentation, but to have the website point to the things which concern those who are looking for them only09:54
toscalixlike us09:54
tristanSo that everyone finds their way, and that everything is still accessible in one repo09:54
toscalixthe trick is to have a clear idea about who are we doing this for09:54
toscalixif you do it for ourselves....then you take some decisions09:54
toscalixif we do it for a wider audience who is not aware of bst, then you might take different decisions09:55
tristanWe can make decisions which satisfy all of the use cases, fragmentation makes things more complicated and harder to maintain - the right audience needs a way to find the right materials09:56
tristanThat is unrelated to which git repo those materials are created and stored in09:56
toscalixI think that today, the governance model has a wider audience than those pariticipating in the project, that is a fundamental maturity variable and that we should assume that it will be a matter of inspection by those who need to decide if they will use the tool ... in corporate environments09:56
toscalixtristan: agree, this is wht adds68 proposal makes no sense09:56
toscalixit is not a matter of the repo, it is a matter of how you present it and for who09:57
tristanI think there is confusion in the conversation09:57
tristanI agree with you toscalix on that front certainly09:57
toscalixback to the labels....09:57
toscalixyou want to use  a new one, fine. Just make sure that everybody use it09:58
coldtomi have been using buildstream for 3 months and had no idea there was a written policy, having this document hidden away in a separate repo makes things far more confusing. A link in the README would be sufficient to make people aware it exists09:58
toscalixmake the extra cost worth it. This would mean a change compared to the current behaviour09:58
qinustyI agree with coldtom on the separate repo thing. Information like this should atleast be referenced from within buildstream HACKING or README09:59
tristantlater[m], If I change the required artifacts for required elements, as we previously discussed, would you agree that this statement no longer makes sense ? https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_artifactcache/artifactcache.py#L20310:00
*** robertheywood_ has joined #buildstream10:00
jmacThe current behaviour is that we assign reviewers in a private IRC channel, which is definitely not what we want10:00
*** robjh has quit IRC10:01
toscalixcoldtom: https://mail.gnome.org/archives/buildstream-list/2018-May/msg00033.html10:02
tlater[m]tristan: So, you're changing this such that we only know what the locked keys are when we are actually removing things.10:03
tlater[m]I assume you do this by walking through all elements and collecting their keys just before we start cleaning10:03
adds68toscalix, there is no direct link from IRC to the mailing list FYI :)10:03
tlater[m]Surely we still want to delete both weak and strong keys, for the reasons stated?10:03
tlater[m]We just don't "lock" them anymore.10:04
toscalixadds68: are you suggesting you will work on it?10:04
tristantlater[m], wrote the failing test, fails correctly; now about to write the code10:04
tlater[m]tristan: I think the point of that comment is to say "if we're only deleting one of the keys we're probably doing something wrong"10:04
tlater[m]I think that still holds.10:05
adds68toscalix, yea sure, how do i go about doing that?10:05
tristantlater[m], my intention is to store a list of elements that are required, and while walking the LRU artifacts, delete only the ones which are not required10:05
toscalixqinusty: tlater[m] if you think this policy should be referenced somewhere else.... please take action and include it10:05
tristantlater[m], actually yes you are right, let's leave that in10:05
toscalixadds68: you are asking the wrong person10:05
qinustyadds68 ask an OP, tristan, jjardon, ironfoot10:06
tlater[m]adds68: While you're at it, mind adding this to HACKING.rst? :)10:06
qinustyIf you mean the message at the top in irc.10:06
adds68tlater[m], sure :)10:06
adds68qinusty, yea i guess ironfoot jjardon or tristan can fix that10:07
adds68If we are pointing people to the mailing list for context, then we should make that obvious10:07
*** tristan changes topic to "BuildStream 1.2.0 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://buildstream.gitlab.io/buildstream | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://mail.gnome.org/mailman/listinfo/buildstream-list | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Road"10:08
tristanadds68, :)10:08
adds68ta tristan :)10:08
jmacfinn: In your pastebin, is 'hello.bst' publically available? I'd like to include it in the issue report10:08
finnit's the autotools example10:08
tristanWhenever we deem the website is in the right shape, we'll make that the first link in the topic :)10:09
adds68tristan, yea that is a good idea also10:09
jmacfinn: Where can I find that?10:09
finnbuildstream/doc/examples/autotools10:10
*** lachlan has quit IRC10:10
jmacThanks finn10:10
jonathanmawtiagogomes: how is https://gitlab.com/BuildStream/buildstream/merge_requests/671? Are you currently awaiting review?10:11
*** cs-shadow has joined #buildstream10:11
tlater[m]juergbi: Am I right in assuming that the new CAS server isn't insecure by default anymore? I.e., it implements its own SSL, and doesn't need to be proxied through nginx?10:12
gitlab-br-botbuildstream: issue #653 ("bst checkout fails due to artifact cache error when remote execution is enabled") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/65310:13
tiagogomesjonathanmaw Qinusty reviewed it. Now there's a new pull_tree in master that will cause the cache size to not be accurate for remote execution if the MR is merged as it is10:13
juergbitlater[m]: it supports both. if --server-key is specified, TLS is enabled, otherwise it's insecure10:14
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72610:14
juergbii.e., correct, no proxying is required10:14
jonathanmawtiagogomes: okay, I'll refrain from further review for now10:14
tlater[m]Thanks juergbi - I didn't realize you could omit --server-key10:15
juergbihandy for local testing10:15
juergbi(or if you still want a proxy)10:15
gitlab-br-botbuildstream: merge request (issue-642-Invalid_project.conf_seen_as_missing->master: Incorrect error when malformed project.conf) #792 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79210:17
tiagogomesjonathanmaw, hopefully tristan will have bright ideas about how to have pull_tree to tell the bytes added to the main process10:17
tlater[m]Yep, that makes me wonder if my docker implementation is flawed10:17
* tlater[m] will leave it as-is for now, if someone needs to go ssl-less they can always implement that.10:17
* tristan is annoyed that there is no location, other than Element.__calculate_cache_key(), where we know that "A cache key has been discovered", due to the multi-return nature of Element._update_state()10:21
*** lachlan has joined #buildstream10:24
gitlab-br-botbuildstream: merge request (issue-642-Invalid_project.conf_seen_as_missing->master: Incorrect error when malformed project.conf) #792 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79210:26
*** finn has quit IRC10:27
gitlab-br-botbuildstream: merge request (richardmaw/test-config-fixes->master: Fix tests that attempt to access the home directory) #780 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78010:27
WSalmonpython3 setup.py test --addopts --integration tests/integration/workspace.py::test_workspace_commanddir10:27
WSalmoninvalid command name 'tests/integration/workspace.py::test_workspace_commanddir'10:27
WSalmonskullman et al. ^ should this work or am i being think10:28
gitlab-br-botbuildstream: merge request (richardmaw/test-config-fixes->master: Fix tests that attempt to access the home directory) #780 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78010:28
tiagogomes python3 setup.py test --addopts '--integration tests/integration/workspace.py::test_workspace_commanddir'10:28
WSalmonbrill10:28
WSalmonthanks10:28
*** lachlan has quit IRC10:29
tristantlater[m], So that fixes it \o/ with a nice new regression test... will push a branch soon10:29
tristantlater[m], However there is one problem... I guess we can keep the LOC where we mark the required element at the end of Element._assemble_done(), for the sake of updating the atime; but that is not "correct"10:31
tristantlater[m], That line assumes that that is the only place where we might discover a cache key in mid-build10:31
tristantlater[m], On the other hand, placing the line of code deep in Element.__calculate_cache_key() is also flawed and out of place10:32
tlater[m]Sounds like we need some sort of streamlined way to say "oh, cache keys updated"10:32
tristanHow about we keep that in Element._assemble_done(), but add a FIXME comment explaining how this is problematic ?10:32
skullmantristan: could do with a follow up review on https://gitlab.com/BuildStream/buildstream/merge_requests/788 when you're free10:33
tristantlater[m], that is why I expressed my annoyance above indeed10:33
tlater[m]tristan: I think that's fine for now, since we do actually only update cache keys there to my knowledge10:34
tlater[m]Just important to know if we change that in the future.10:34
tristanskullman, ah good spot about the update.py, I always operated that by having BuildStream installed with "pip install --user -e...", I think it only breaks when not installed ?10:36
gitlab-br-botbuildstream: issue #654 ("Use Platform.artifact_cache() instead of reconstructing CASCache from context") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/65410:36
tristanskullman, Looks good, please merge :)10:37
* skullman queues it10:37
gitlab-br-botbuildstream: merge request (richardmaw/subprocess-PWD->master: Address post-merge review of Ensure PWD is set in process environment) #788 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78810:37
*** jonathanmaw_ has joined #buildstream10:37
*** jonathanmaw has quit IRC10:38
tristantlater[m], I do not believe we only update cache keys there, the strong keys I think can change dynamically when --no-strict and pulling artifacts along the way... they can also change dymaically as a result of `bst build --track-all`, etc10:38
tristantlater[m], Of course, the likelihood of already having a cached artifact in those cases are slim (although with tracking not always, if you've rewinded and then fast forwarded with build + track to test some regression)10:39
tlater[m]tristan: In that case, maybe an issue is more appropriate, so we don't forget about this?10:40
tlater[m]I don't think it's likely enough to block the fix, but having something actually broken in a FIXME isn't nice.10:41
*** lachlan has joined #buildstream10:41
tristantlater[m], Will do10:46
* tristan prepares MR10:47
toscalixcoldtom: tlater[m] adds68 if you guys are interested in following the improvements in the policy or are willing to contribute to it, please track your effort on this ticket: https://gitlab.com/BuildStream/nosoftware/alignment/issues/910:47
toscalixand qinusty:10:47
adds68toscalix, i am adding some changes now10:48
adds68toscalix, tristan do you think we should link to the labels document in the issue template?10:48
tristantlater[m], that is at least, the *last* opportunity where we can discover a cache key10:48
toscalixok, then please track them in the ticket10:48
gitlab-br-botbuildstream: merge request (Qinusty/message-helpers->master: WIP: Continued work on improving BuildStream messaging API) #670 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/67010:49
toscalixsorry, wrong ticket, the right one is https://gitlab.com/BuildStream/nosoftware/alignment/issues/1110:49
tristantoscalix, Ummm, it's a good question - on the one hand I think it's useful to have a description of the available labels, on the other hand I don't like the idea of introducing someone to an entirely different repository if the information is going to be fragmented10:49
toscalixcoldtom: tlater[m] adds68 qinusty10:50
toscalixtristan: the information is not going to be fragmented10:50
tristantoscalix, Ideally, the page where labels are configured, is read-only for someone who doesnt have privileges, and we just point them there ?10:50
toscalixthe idea has always been to have all the policies listed in the website under the community page10:50
toscalixas proposed in the content design document and applied in the diagrams10:51
toscalixit is just not implemented for this release because we would not have time10:51
tristanRight, which will probably be a link to the HACKING.rst (http://buildstream.gitlab.io/buildstream/HACKING.html)10:51
toscalixso this fragmentation problem did not exist...until the hacking.rts proposal came up10:52
tristanThere is another downside though which I worry about10:52
tristanWhich is pointing people away from the issue page while they are filing an issue10:52
toscalixit seems clear to me that the content design was not considered carefully10:52
tristantoscalix, I think rather no; not based on fragmentation (I was rather worried we might point to something in "alighment"), but more because I want the issue tracker to be optimized towards non-contributors10:53
toscalixcheck the management box: https://gitlab.com/BuildStream/nosoftware/alignment/blob/master/content_design/BuildStream_Content_Structure_all.pdf10:53
tristanThe onus is on us to do the triage, not the issue reporter who may have encountered a problem, and is courtious enough to file an issue10:53
toscalixthe policies are considered there, structured in for areas10:54
tristantoscalix, I am aware that we all did not take real time to pay attention to the content design proposals10:54
*** alatiera_ has joined #buildstream10:54
tristantoscalix, but on the subject of linking someone away from the issue template, I disagree10:54
adds68How about moving the "no-software" project to a wiki inside the Buildstream project?10:55
gitlab-br-botbuildstream: merge request (richardmaw/test-config-fixes->master: Fix tests that attempt to access the home directory) #780 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/78010:55
tristanContributors should read the policies about contributing, issue reporters should have as hassle free of a time as possible10:55
toscalixthey were not considered for the website, which means we will need to do it. If there is pressure not to do them.... well, I would advice to first finish the website content we have and then discuss which content we want to do next10:55
toscalixso we can have a serious discussion about the design before we just.... solve the gap. Nothing different from what you are requesting to do with the code10:56
tlater[m]Could I ask someone to review my README update to this patch? It's usage instructions... https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/5310:56
tristanWhen issue reporters happen to also be contributors, they will have read the policies anyway10:56
toscalixI would really prefer in finishing what we have now half way through10:56
toscalixhavinbg people fixing the labels policy when we have not finished the web is.... not ideal10:57
tristanRelated to labels, I was thinking that if we have "domains" clearly defined (as we already use a great deal of labels for in interesting ways), they should be brought inline with the proposed committer policy, if that goes through (https://mail.gnome.org/archives/buildstream-list/2018-September/msg00044.html)11:00
tristanI.e. "Frontend", "Logging", etc, would be nice if they correspond to the subsystems for which certain groups are approved for review/direct commit11:00
toscalixadds68: nosoftware subgroup was to host everything that is not code11:00
tristanBut let's not have that conversation unless we have it first on the mailing list11:00
toscalixthat requires a specific structure on gitlab, different from buildstream repo11:01
adds68toscalix, that is usually what people use the Gitlab wiki for see: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/wikis/home for EG11:01
toscalixadds68: no, the milestones are diferent, the label structure is different....11:02
toscalixthat is not what the gitlab wiki is for11:02
toscalixthe wiki is for contents11:02
toscalixcheck the labels, the milestones... they are different in nosoftware11:02
* adds68 is unsure what you are looking for, but for example Buildstream_polocies seems like a wiki page to me11:02
toscalixwe decided to avoid using the wiki on gitlab because we have the gnome wiki11:03
toscalixat the same time, we want to to keep the policies on git11:03
toscalixso we can control who changes it11:04
toscalixsince they affect the governance of the project11:04
adds68toscalix, where is the link to gnome-wiki?11:05
toscalixsadly we had to move the website to the root tree because of a limitation of gitlab with managing pages from repos inside subgroups11:05
toscalixon the website11:06
toscalixcommunity page11:06
toscalixhttps://wiki.gnome.org/Projects/BuildStream/Monthly-Meeting11:06
toscalixby the way, there is a broken link on the community page11:06
toscalixthat is the only page that we are promoting for now11:07
toscalixthe rest of the contents are being deprecated or listed in the fron wiki page for now11:07
toscalixthe work in that front is being tracked here: https://gitlab.com/BuildStream/website/issues/2011:08
adds68toscalix, so the website has it's own wiki page?11:11
toscalixno, it is the gnome wiki page. That is the only wiki we have11:11
adds68toscalix, ah ok, so you can update the wiki page via Gitlab? Nice!11:13
toscalixgnome wiki ? No we cannot, which is why we are moving socntent that requires some level of control away from the wiki and into the website11:20
toscalixs/socntent/some content11:21
toscalixthe gnome wiki is open to those with an account11:21
toscalixfor some time, it was the only thing we had, so we added there everything11:22
toscalixnow that we have the website, there is content we can move there, keeping the website for stuff where people without git knowledge might/can contribute to, or content that does not require certain level of control11:23
toscalixobviosuly which contents belongs where is questionable11:23
toscalixso far, I think info about meetings belongs there11:24
toscalixthe front wiki page will remain as a directory for now11:24
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts->master: Don't delete required artifacts when tracking is enabled) #793 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79311:24
toscalixsince it is still the number one output on searchs11:24
flatmushI'm trying to get binutils to build from git source, the easiest step seems to be to use a script in the repository to build a tarball, and then build binutils from the tarball.11:25
flatmushIs there an easy way to get buildstream to source a tar file from inside another artifact?11:25
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts-1.2->bst-1.2: Don't delete required artifacts when tracking is enabled) #794 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79411:26
tristandamn, those MRs are not ready :-S11:29
*** robertheywood__ has joined #buildstream11:29
*** lachlan has quit IRC11:29
tristanaha11:29
*** robertheywood_ has quit IRC11:30
tlater[m]flatmush: You'll probably want to have an element named `binutils-tar.bst`. It's just easiest to split that build into two steps.11:31
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts->master: Don't delete required artifacts when tracking is enabled) #793 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79311:31
tlater[m]Alternatively, of course, you can set your own build commands for this11:31
* tlater[m] doesn't think there's an easy way to say "use this element as a source"11:31
coldtomi think you could use a script element and stage the tarball element in a writable directory11:32
coldtomthen recreate the install11:32
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts-1.2->bst-1.2: Don't delete required artifacts when tracking is enabled) #794 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79411:32
gitlab-br-botbuildstream: merge request (chandan/update-project-homepage->master: setup.py: Make website the primary homepage) #795 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79511:32
tlater[m]coldtom: I think in this case pre-pending the "build tarball" commands is a better option, since you can then just use the default plugin commands afterwards.11:33
*** iker has joined #buildstream11:34
coldtomtlater: agreed11:34
tristantlater[m], Can you give https://gitlab.com/BuildStream/buildstream/merge_requests/793 a lookover ?11:37
tlater[m]tristan: Sure!11:37
tristanthanks :)11:37
* tristan should really get a 1.2.1 out asap after this11:38
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Adamjones/contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79611:38
tristanOr people will get corrupted local artifact caches with 1.2.011:38
tristanwhen they reach quota11:38
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79611:38
tristanadds68, You wanna break tradition for the sake of GitLab, cause they use a hard coded name and decided that people should call it CONTRIBUTING ?11:39
tristanadds68, I was very annoyed that GitLab wanted to make this decision for everyone else - but meh; I don't mind11:40
tristanthat's what people want, let's give it to them :)11:40
adds68tristan, it's annoying that you can't change it i agree, but i feel the UI link the Contributing is very valuable for new comers11:40
* tristan thinks that we are on a higher level of cool than the younger projects to keep the traditional HACKING filename, but I digress11:41
adds68tristan, haha11:42
*** lachlan has joined #buildstream11:44
tristanadds68, I brought it up in #gitlab a while ago, when this obnoxious "Contributing" empty box appeared in the project page, being empty when we *clearly* had a HACKING file in place, with the correct name :)11:45
adds68tristan, no way, what was their response?11:45
tristanbut meh, no setting11:45
adds68"it sounds too technical" ?11:45
tristanI suppose they had some debate about the filename, and CONTRIBUTING was more modern and perhaps politically correct11:46
tristanpeople don't want to refer to development by it's proper name "hacking", anymore, those days are sadly gone11:46
tristanadds68, I'm a bit more generally peeved that they decided on a name at all, as with everything they want to hard code us into their garden of how things work11:47
tristaninstead of remaining extensible11:47
adds68tristan, yea i imagine some web developer somewhere didn't want to have to read another variable from your gitlab conf :P11:48
tristanFor instance, I want a way to generate my *own* analytics on their pages, by feeding my data from pipelines and interpreting that data for their chart representations11:48
tristanBut no... they keep coming up with "Codequality" plugin... "It shall work this way and no way else !"11:49
tristanadds68, https://gitlab.com/BuildStream/buildstream/merge_requests/796 points the user to a file in the "alignment" repo, more fragmentation; I think that everything that materializes into what we want people to see, either goes onto the website, or in the docs11:53
adds68tristan, so should i change that link to point to the wesbite?11:53
adds68website*11:54
tristanI guess we have to actually create the content first ?11:54
adds68toscalix, ^ ?11:55
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts->master: Don't delete required artifacts when tracking is enabled) #793 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79312:08
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts-1.2->bst-1.2: Don't delete required artifacts when tracking is enabled) #794 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79412:11
toscalixadds68: if we think that the incorporating a section related with policies in the community page is a priority at this point, then find with me. We can, as second step, move the policies to the website12:16
toscalixonce we finish the current contents12:16
toscalixdid I answered the question?12:17
tristanFrankly, this part will probably land up in CONTRIBUTING.rst anyway, as the plan is to structure it better and make it much more elaborate12:19
tristanfiling issues, committer policy, coding guidelines, are all part of a contributors guide afaics12:19
*** lachlan has quit IRC12:20
tristantlater[m], unfortunately, there is no straight forward way to "only turn an iterable into a list if the iterable is a generator"12:20
tristantlater[m], so I'm living with "elements = list(elements)"12:20
tlater[m]tristan: That's what I expected you to do :)12:20
tristanI don't like it, but options to work around that are limited :-S12:21
tlater[m]It's a bit of a shortcoming of python, with the whole quacking ducks thing.12:21
tristanisinstance(iterable, collections.Sequence) is an option12:21
tristanyeah12:21
tlater[m]tristan: My idea here is just to clearly document what kind of iterables are expected. In this case we expect a generator - if someone gives us a list, it won't be optimized.12:22
tlater[m]But I don't think this is going to be the end of the world for a few hundred elements anyway.12:23
tristantlater[m], it's a one off, but indeed I think that long term we should have clear guidelines on how to handle iterable inputs12:24
tlater[m]Type hints may help - jmac has been pushing for us to use those.12:24
tristantlater[m], it's certainly better to handle on the receiving end, and say "You can call this with any iterable", than the other way around, where you tell the caller to "be careful"12:25
tristantlater[m], done with comments I suppose ?12:26
tlater[m]tristan: They're actually incorporated into the function header12:26
tristantlater[m], as a point of interest; it's interesting how the regression test fails without the fix applied12:26
tristantlater[m], since they are all import elements, the cleanup job just happily cleans everything up as needed, and the build reports SUCCESS !12:27
tristantlater[m], but only the last built artifacts are cached (but show up as "waiting", because they depend on deleted deps)12:27
tlater[m]Hmm12:28
tlater[m]We might want to check if all built elements actually end up "cached" before we report success, then12:28
tristanIn real life, this usually aborts with an "artifact not found" error/bug when staging dependencies for a build12:28
tristantlater[m], Nah, adding more checks is usually not the right answer12:29
tristantlater[m], We now have a regression test for this - but adding runtime checks to the code, because we don't trust it to do what it's supposed to do, adds more complexity and takes away from clarity over time12:30
tristanSuch that we trust less and less that it does what it's supposed to12:30
adds68tristan, so we merge this for now, open a new issue for me to update once the website is updated?12:30
tlater[m]I suppose the only way we can ever run into this outside of the test suite is if a user decides to deliberately remove an artifact during cleanup.12:30
tristanadds68, I'm fine with that12:30
tristanadds68, It will be the CONTRIBUTING.rst I think which gets updated, and the link which goes away12:30
adds68:)12:32
tristantlater[m], I don't think it will be allowed ultimately12:32
tristantlater[m], We already know that we have issues with concurrent instances12:32
tristantlater[m], It's low priority but it's on the radar, we should avoid stop gap fixes and slowly work towards perfection12:33
tristantlater[m], stop gaps are good if we really have serious issues that effect users and we need a fix right away12:33
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts->master: Don't delete required artifacts when tracking is enabled) #793 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79312:34
tristantlater[m], in case that was ambiguous somehow, the expected behavior should be that `bst artifact delete <artifact name>` returns non-zero and says "This artifact is required by an ongoing build" or such12:35
tlater[m]Y12:35
tlater[m]*Yeah that was pretty clear :)12:35
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72612:39
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79612:43
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts->master: Don't delete required artifacts when tracking is enabled) #793 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/79312:44
gitlab-br-botbuildstream: merge request (tristan/fix-required-artifacts-1.2->bst-1.2: Don't delete required artifacts when tracking is enabled) #794 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/79412:44
tristantiagogomes, Your question is a very hard one for me to answer at the moment12:48
tristantiagogomes, and approaching 10pm, I don't think it will be answered this week12:48
tristanbochecha, They have gone a long way towards debugging the confidential issue, and are pointing out they have another issue https://gitlab.com/gitlab-org/gitlab-ce/issues/41973, which is about the inability to invite participants to discuss confidential issues !12:50
bochechatristan: shaving the yak all the way down :)12:50
tristanbochecha, they are asking me to gather information from you because of this... they are unable to reproduce :-S12:51
tristansec12:51
tristanbochecha, they ask if the edit was made through this UI: https://gitlab.com/BuildStream/bst-external/edit/master/bst_external/elements/collect-integration.py ?12:51
tristanbochecha, and if the process involved creating a merge request and such and such12:51
bochechatristan: I browsed to the file: https://gitlab.com/BuildStream/bst-external/blob/master/bst_external/elements/collect-integration.py12:52
tristanbochecha, clearly they take the issue seriously, but eh... what canna do12:52
bochechaclicked "Edit"12:52
bochechawhich indeed lead me to the edit UI you linked to12:52
bochechathe "target branch" was prefilled as "master"12:53
bochechaI clicked "commit changes"12:53
bochechatrying it again, the "target branch" is prefilled as "patch-1", so at least this doesn't seem like it would happen inadvertently like it did12:53
bochechait did not create a merge request, it just committed to master12:53
tristanbochecha, I'll paste your comments back to the issue12:54
tristanbochecha, thanks :)12:59
bochechanp12:59
tristanAnyway, it's not a huge deal to me, just trying to be helpful to them :)12:59
tristanWSalmon, I didnt get a chance to reply your email, but I think that what you are saying is that we should somehow make it clear who is committer in which area (I think the COMMITTERS file does this ?) in policy, so that reviewers are easier to find ?13:06
tristanWhich I think is a large part of the intent ?13:06
tristanBut... anyway I'll write back to the list when I get a chance13:06
*** lachlan has joined #buildstream13:08
*** iker has quit IRC13:08
*** ikerperez has quit IRC13:08
*** iker has joined #buildstream13:08
*** ikerperez has joined #buildstream13:09
*** tristan has quit IRC13:09
*** lachlan has quit IRC13:11
*** alatiera_ has quit IRC13:46
*** alatiera_ has joined #buildstream13:47
*** lachlan has joined #buildstream13:47
*** lachlan has quit IRC13:51
tiagogomeshttps://twitter.com/gnomealex/status/1040558646122430464 :o13:52
bochechatiagogomes: yeah, he announed it in #flatpak earlier and some of us were going bonkers :)13:55
tiagogomesxD13:56
*** lachlan has joined #buildstream13:59
*** lachlan has quit IRC14:02
*** ikerperez has quit IRC14:12
gitlab-br-botbuildstream: merge request (jonathan/pickle-yaml->master: WIP: Add a cache of parsed and provenanced yaml) #787 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78714:17
*** ikerperez has joined #buildstream14:20
*** iker has quit IRC14:21
*** lachlan has joined #buildstream14:38
*** lachlan has quit IRC14:45
*** lachlan has joined #buildstream14:48
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/fix-654->master: sandbox/_sandboxremote.py: Acquire CASCache via Platform) #797 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79714:56
Kinnisonjmac: Thanks for the chat I had with you earlier.  I've submitted !797 to resolve #654 - it passed tests on my machine but I'm still waiting for the pipelines on gitlab.14:58
gitlab-br-botbuildstream: merge request (tpollard/494->master: WIP: Don't pull artifact buildtrees by default) #786 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78614:59
gitlab-br-botbuildstream: merge request (tpollard/494->master: WIP: Don't pull artifact buildtrees by default) #786 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78615:02
* qinusty wanted to use pytest-quickcheck for a test... Ended up raising a ticket https://bitbucket.org/pytest-dev/pytest-quickcheck/issues/12/configuring-different-ranges-per-parameter15:02
Kinnisonqinusty: can you not use the decorator twice, once for each parameter?15:03
qinustyheh15:06
* qinusty just tried this15:06
qinustyYes. Yes you can, HOWEVER it uses the min_num, max_num specified by the first fixture.15:06
qinustyfor BOTH values15:06
Kinnisonoh jeepers15:06
KinnisonSomeone didn't test that very well15:07
*** lachlan has quit IRC15:11
*** robertheywood__ has quit IRC15:12
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79615:16
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79615:20
*** lachlan has joined #buildstream15:23
*** lachlan has quit IRC15:30
qinustyCan someone dockery take a look at this https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/66, Adding a development dependency for some tests I'm working on. I can merge but would appreciate a glance from someone :D15:35
qinustyAnyone got any ideas what's going on here https://hastebin.com/kazafelaje.cpp? I assume the [x-y] is the parameters separated by a hyphen (I'm printing the space separated numbers on the right...)15:41
qinustySOMETIMES, the parameter numbers aren't identical....15:41
gitlab-br-botbuildstream: merge request (tiagogomes/some-cleanups->master: Bunch of cleanups) #798 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79815:42
KinnisonThis is pointing at pytest-quickcheck not being very safe to use15:42
gitlab-br-botbuildstream: merge request (tiagogomes/issue-287-backport->bst-1.2: Backport of !678 (Add validation of configuration variables) to 1.2 branch.) #789 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78915:43
qinusty:D15:44
qinusty:(15:44
* qinusty will just add 4-5 test cases manually and leave it at that.15:46
*** lachlan has joined #buildstream15:52
gitlab-br-botbuildstream: merge request (Qinusty/unit-test-utils->master: Add unit tests for some utils.py functions) #799 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79916:01
gitlab-br-botbuildstream: merge request (Qinusty/unit-test-utils->master: Add unit tests for some utils.py functions) #799 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79916:01
*** lachlan has quit IRC16:02
gitlab-br-botbuildstream: merge request (chandan/automate-pypi-release->master: WIP: .gitlab-ci.yml: Publish to PyPI when new tags are pushed) #731 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/73116:06
gitlab-br-botbuildstream: merge request (tiagogomes/some-cleanups->master: Bunch of cleanups) #798 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/79816:08
gitlab-br-botbuildstream: merge request (tiagogomes/issue-287-backport->bst-1.2: Backport of !678 (Add validation of configuration variables) to 1.2 branch.) #789 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/78916:10
gitlab-br-botbuildstream: merge request (coldtom/strip-rules->master: WIP: Upstream freedesktop-sdk strip rules) #750 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/75016:10
gitlab-br-botbuildstream: merge request (chandan/automate-pypi-release->master: .gitlab-ci.yml: Publish to PyPI when new tags are pushed) #731 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/73116:25
gitlab-br-botbuildstream: merge request (jmac/remote_exec_checkout_fix->master: Remote exec: Remove early warning and check directory is not None) #800 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/80016:33
*** CTtpollard has quit IRC16:34
*** ikerperez has quit IRC16:37
*** lachlan has joined #buildstream16:38
jmachttps://gitlab.com/BuildStream/buildstream/merge_requests/800 is a potential fix for `bst checkout` under remote execution (but see the caveat in the descripton)16:39
*** lachlan has quit IRC16:45
gitlab-br-botbuildstream: merge request (tiagogomes/issue-573->master: Reduce IO overhead caused by artifact cache size calculation) #671 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/67116:47
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79616:47
gitlab-br-botbuildstream: issue #287 ("Special / Magic variables are unprotected") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/28716:49
KinnisonThanks jmac, if noone else gets to it, I'll give it a look over on Monday16:53
*** lachlan has joined #buildstream17:01
*** lachlan has quit IRC17:06
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79617:08
gitlab-br-botbuildstream: issue #655 ("Update CONTRIBUTING.rst to point to website") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/65517:11
gitlab-br-botbuildstream: merge request (adamjones/contribution-guide->master: Update contribution guide) #796 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/79617:14
gitlab-br-botbuildstream: merge request (jonathan/pickle-yaml->master: WIP: Add a cache of parsed and provenanced yaml) #787 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78717:18
*** toscalix has quit IRC17:21
*** lachlan has joined #buildstream17:32
*** jonathanmaw_ has quit IRC17:38
*** lachlan has quit IRC17:40
*** lachlan has joined #buildstream17:51
*** lachlan has quit IRC17:58
*** rdale has quit IRC18:05
*** lachlan has joined #buildstream18:16
*** lachlan has quit IRC18:34
*** lachlan has joined #buildstream19:21
*** cs-shadow has quit IRC19:29
*** lachlan has quit IRC19:39
*** tristan has joined #buildstream20:16
*** ChanServ sets mode: +o tristan20:16
*** xjuan has joined #buildstream20:33
*** alatiera_ has quit IRC20:47
*** alatiera_ has joined #buildstream20:47
*** bochecha has quit IRC21:00
*** bochecha has joined #buildstream21:02
*** bochecha has quit IRC21:36
*** tristan has quit IRC21:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!