IRC logs for #buildstream for Wednesday, 2020-04-29

*** Mia7584 has joined #buildstream00:57
*** Mia7584 has quit IRC00:59
*** Clara1293 has joined #buildstream01:30
*** Clara1293 has quit IRC01:31
*** rdale_ct has quit IRC02:31
*** mohan43u has quit IRC05:35
*** mohan43u has joined #buildstream05:39
*** mohan43u has quit IRC05:45
*** mohan43u has joined #buildstream05:49
*** tristan has quit IRC05:50
*** tristan has joined #buildstream06:03
*** ChanServ sets mode: +o tristan06:03
gitlab-br-bottristanvb approved MR !1888 (juerg/sandbox->master: Switch preferred sandbox from bwrap to buildbox-run) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/188806:10
gitlab-br-botmarge-bot123 closed issue #719 (Support BuildBox for local builds) on buildstream https://gitlab.com/BuildStream/buildstream/-/issues/71907:02
gitlab-br-botmarge-bot123 merged MR !1888 (juerg/sandbox->master: Switch preferred sandbox from bwrap to buildbox-run) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/188807:02
tristangah, something happened in bst-plugins-experimental which ended up using the removed BST_FORMAT_VERSION :-S07:06
tristanah, ok no worries... I thought for a second I would have to play "redirect the reference to the other repository" for two repos now07:09
tristanDo we have a lot less runners in bst-plugins-experimental ? or is there an issue: https://gitlab.com/BuildStream/bst-plugins-experimental/pipelines/ ?07:18
traveltissueslooks like it might be stuck07:21
*** mohan43u has quit IRC07:22
WSalmontristan, AFAIK the runners are shared accross the group so if a lot of ci is happening in one project it can effect another availability of runners in another07:23
tristanRight now we have one running in buildstream and one running in bst-plugins-experimental07:24
tristanSo I think we're a little short handed, or something is stuck07:24
*** jude has joined #buildstream07:25
*** mohan43u has joined #buildstream07:25
tristanHmmm, my other pipeline had no trouble starting in buildstream, bst-plugins-experimental still on standby07:27
juergbitristan: the permanent runners have been disabled because of the docker cleanup issue. however, I think it's better to re-enable them, as we can temporarily run the cleanup manually07:27
juergbidone07:27
tristanAnd tada ! the pipeline runs :)07:28
tristanOk so lets try to merge this BST_MIN_VERSION plugin origins overhaul once this thing finishes07:28
tristanjuergbi, we have tagging to do on bst-plugins-experimental then, are we waiting for anything else for that ?07:29
*** benschubert has joined #buildstream07:29
tristanand as I understand, there is also another metadata you mentioned ?07:29
tristansetup() in setup.py needs a version bump07:30
tristananything else ?07:30
juergbiI'm not aware of any pending MRs that are crucial before a snapshot07:30
juergbitristan: setup.py and doc/source/conf.py07:30
tristanFor BuildStream snapshots, I would like to squeeze in something else07:30
juergbido we also need a new snapshot on the BuildStream side?07:31
tristanI have a branch in the works which addresses #1291, as that is PluginOrigin related (and also enhances errors/warnings to include provenance which has been missing thus far)07:31
tristanjuergbi, actually, I'm not calling for one honestly07:32
tristanI thought you were this week, but maybe that already happened ;-)07:32
juergbi1.93.2 already happened :)07:32
tristanok ok :)07:32
juergbior do the min-format changes in bst-plugins-experimental break with 1.93.2? in that case, we might need another one07:34
*** gitlab-br-bot has quit IRC07:34
tristanOh interesting game of whackamole, rebasing the bst-plugins-experimental branch causes the commit sha to be GC'ed and fails on buildstream side07:35
juergbiyes, that can happen07:36
juergbithe only breakage I see is due to tests/sources/git_tag/lfs/project.conf07:36
juergbidropping the plugin format version from project.conf will break on 1.93.2 I suppose07:36
tristanRight it will07:37
tristanindeed07:37
tristanOh damn07:39
tristanI expect it's pipeline will fail07:39
tristanif only we landed this before that new project.conf snuck in :-S07:39
juergbitristan: I actually see other project.conf files in tests with the same issue07:39
juergbibut your branch doesn't change them07:40
juergbiwondering what's up with those07:40
tristanindeed yeah07:40
tristanmaybe those tests arent being run in CI ?07:40
juergbimaybe, I don't know07:41
* tristan will first update those project.conf's in the branch regardless07:41
juergbitheoretically, we could generate project.conf programmatically and be smart about what version we're running against07:41
juergbibut probably not worth it07:41
tristanAnd I guess we have a situation where only brute force will work07:41
tristanWell yeah, brute force or bending over backwards07:42
tristanOk I've updated the branch (_and_ the tox.ini reference in the BuildStream side branch)07:47
*** seanborg_ has joined #buildstream07:48
tristanI take it that passing tests for https://gitlab.com/BuildStream/buildstream/-/merge_requests/1885 are an indication that the bst-plugins-experimental branch works07:48
*** benschubert has quit IRC07:48
*** dbuch[m] has quit IRC07:48
*** walterve[m][m] has quit IRC07:48
*** jward has quit IRC07:48
*** benschubert has joined #buildstream07:48
*** jward has joined #buildstream07:49
tristanWait a sec, where does bst-plugins-experimental get it's BuildStream from ?07:49
tristanAha, there is "master" and a tag in bst-plugins-experimental07:50
tristanin tox.init07:50
tristanini07:50
WSalmontristan, https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/tox.ini#L2007:51
tristanSo I suppose it would run the tag in regular CI and against master nightly or such07:51
WSalmonhttps://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/tox.ini#L4607:51
WSalmonhttps://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/tox.ini#L507:52
*** walterve[m][m] has joined #buildstream07:52
WSalmonif you just run `tox --intergration`07:52
WSalmonthen it will run master and fixed for every version of python you have installed07:53
juergbitristan: it runs both in regular MR pipelines but master is allowed to fail07:53
WSalmonthe CI has stages that run a verity of fixed, master and versions of python07:53
tristanAh allowed to fail also makes sense07:53
tristanAlright so, once this pipeline passes again on master, I'll brute force bst-plugins-experimental and then roll a tag on bst-plugins-experimental (with updated metadata)07:55
tristanThen we'll probably want another snapshot, because we'll want to update bst-plugins-experimental CI to reference a new tag of BuildStream for all to be fine across the rainbow07:55
*** phildawson has joined #buildstream08:07
*** dbuch[m] has joined #buildstream08:15
tristanAlright, bst-plugins-experimental is tagged and metadata adjusted to 1.93.208:19
tristanAnd !1885 is adjusted to refer to the branch and given to marge08:20
*** tpollard has joined #buildstream08:20
tristanAfter the land, we'll need to adjust bst-plugins-experimental tox.ini to get the new buildstream08:20
*** rdale has joined #buildstream08:22
*** santi has joined #buildstream08:36
*** seanborg_ has quit IRC08:50
*** seanborg_ has joined #buildstream08:50
traveltissuesbst-plugins-experimental@master is intended to be broken atm then?08:52
tristanjuergbi, when you do a buildstream snapshot, do you just push a tag and thats it ?08:52
tristantraveltissues, for another 20min or so yes08:52
traveltissuesok08:52
juergbitristan: first add a new heading to NEWS but then yes08:53
tristanOk08:53
juergbiwe haven't done any official announcements of those snapshots, afair08:53
tristanGah, that means the NEWS has to go through CI08:54
juergbioh, that reminds me, I forgot to mention the sandbox change in NEWS08:54
juergbitristan: there is this: https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd08:55
tristanDid we figure out how to merge a patch with [no-ci] ?08:55
juergbiI haven't tried it out yet, though, forgot it last time08:55
tristanI tried it and was able to have a branch not run CI, but I was not able to merge it08:55
tristanThe CI correctly "doesn't run" and that was it08:55
tristanmaybe something specific with that branch though08:55
juergbiit makes sense that gitlab doesn't simply allow skipping the CI requirement with that flag08:56
juergbihowever, with sufficient privilege it would make sense that it could be overridden in the MR08:56
juergbitheoretically, the setting could be changed temporarily but that feels wrong08:56
tristanyeah right now we're in lock down, even force push from command line is refused08:57
tristantraveltissues, this conversation mostly means: bst-plugins-experimental will remain broken for say... another 40min hehe08:58
tristantime it takes to run yet-another-ci-for-NEWS-update08:58
traveltissuesnp09:00
tristanjuergbi, I missed my news entry too... so what shall I say about sandboxes... "buildbox is now the default sandbox implementation" ?09:08
juergbitristan: maybe "buildbox-run is now the default sandbox backend on all platforms"09:09
tristanhttps://gitlab.com/BuildStream/buildstream/-/merge_requests/189109:12
juergbita09:13
*** tpollard has quit IRC09:19
*** tpollard has joined #buildstream09:19
*** lachlan has joined #buildstream09:20
*** phildawson_ has joined #buildstream09:42
*** phildawson has quit IRC09:42
*** phoenix has joined #buildstream09:48
*** lachlan has quit IRC09:55
*** phoenix has quit IRC09:58
*** CTtpollard has joined #buildstream10:10
*** tpollard has quit IRC10:10
*** coldtom has quit IRC10:14
tristantraveltissues, it should be fixed now10:17
tristanis there anything I need to change in bst-plugins-experimental other than the tox.ini to refer to which tag of BuildStream to use ?10:17
*** coldtom has joined #buildstream10:21
traveltissuesi think that's all tristan10:35
traveltissuesthanks10:35
*** CTtpollard is now known as tpollard10:35
traveltissueshttps://gitlab.com/BuildStream/bst-plugins-experimental/-/commit/d54e71daefc8434fe4967fe2687071f5d6cef12310:36
tristanHmmm, what's up with source consistency, we threw it out in favor if is_cached() ?10:46
* tristan kind of liked that API10:47
tristanis there some ML discussion I can read up about that ?10:47
tristanAlso what replaces it ? How do we determine the difference between "It's locally cached" and "We wouldnt even know what to download until you ran bst track" ?10:48
*** lachlan has joined #buildstream10:57
tristanAlso where are the message APIs on Context() ?10:58
tristanmessenger, got that10:58
tristannever got as far as getting some convenience global message APIs though :-S10:59
*** lachlan has quit IRC11:02
juergbitristan: !173911:31
juergbisometimes we only need to know whether the source is reolved, not whether it's locally cached. and determining the latter can take a bit of tim11:32
juergbi*time11:32
tristanSo is_resolved() is added too...11:33
tristanjuergbi, makes sense, I did like having a single method for this but, I guess there is a real gain to changing this :)11:34
* tristan posts !1892 ... and wonders where our gitlab bot has gone11:35
* tristan thinks the policing of loaded plugins could be more improved11:36
tristanI was adding a plugin to test deprecation warnings, and it's unclear how long it would take me to figure out what methods I'm really required to implement11:37
tristanmaybe it would be good to do some checks at plugin loading time instead of just raising ImplError "whenever the vfunc gets used"11:37
WSalmontristan, can i use the latest tag of bst and bst-plugins-experimental together now?11:38
tristanWSalmon, yes, but if you use the latest tag of bst-plugins-experimental, the tox in bst-plugins-experimental will break11:40
tristani.e. running tox in bst-plugins-experimental would break, but everything else (latest both tags) works together11:40
WSalmonok, im not worried about testing but using them on projects11:41
WSalmonthanks tristan11:41
tristanWSalmon, yeah then you don't need to care11:42
WSalmon:)11:44
tristanhmmmm, broken pipeline in bst-plugins-experimental...11:53
tristanThe FUSE stager child process unexpectedly died with exit code 111:53
tristanlooks unrelated :-S11:54
tristanjuergbi, these new bst-plugins-experimental failures seem related to your buildbox-run switcheroo11:55
tristanmaybe we need to update .gitlab-ci or tox.ini to install buildbox correctly ?11:55
* tristan switches locations11:55
traveltissuesyes, i'm hitting that now11:58
*** tristan has quit IRC11:58
traveltissuesError executing command: std::runtime_error exception thrown at [buildboxcommon_client.cpp:723]11:58
*** lachlan has joined #buildstream11:58
*** lachlan has quit IRC12:02
juergbiah, yes, I had to fix an issue with the debian image12:04
juergbineed to bump the docker image in bst-plugins-experimental to the same as in buildstream master12:04
traveltissuesthanks12:05
traveltissuescan i get a review/merge please: https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/10212:15
traveltissuesWSalmon: i don't have permissions for the merge button12:27
WSalmonme nether12:27
traveltissuesjuergbi?12:27
* juergbi clicks12:28
traveltissuesthanks12:28
juergbishould probably fix those permissions12:28
WSalmonjuergbi, could you add traveltissues and myself to bst-external and bst-plugins-experimental, we have had things merged to both and are conservitave merges12:28
juergbiyes, I see that currently only maintainers have the permission to merge. I think in buildstream we have an explicit list12:29
WSalmoni belive that is the case12:29
WSalmonany one can be a dev and then people are explicitly added after they have had work merged was how it used to work12:30
WSalmoni presume we can extend that to the plugin repos?12:30
WSalmonunless you want to do something more sweaping, that might want a ML?12:30
juergbiyes, I think the policy should be the same. I've added you two to the list12:31
WSalmonta12:32
WSalmonfor both?12:32
juergbiah, no, only experimental. let me check bst-external12:33
WSalmonhas coldtom had stuff merged to bst-plugins-experimental?12:33
WSalmonhe can already merge to bst-external i belive12:34
traveltissuesty12:35
WSalmonoh no he cant, can juergbi add coldtom too12:35
WSalmonhe has done lots of maintenance there12:35
juergbiyes, makes sense12:35
WSalmonthanks juergbi12:35
*** lachlan has joined #buildstream13:02
*** Nevaeh2639 has joined #buildstream13:09
*** Nevaeh2639 has quit IRC13:11
traveltissuesseems that bst-plugins-experimental is falling over sometimes because the cache fills up13:17
traveltissuesdoes the cache get cleared immediately following a job?13:20
WSalmoni dont think so, we could key it off the cache key version maybe13:22
traveltissuesi think it's the same issue for the failure on your branch WSalmon13:22
traveltissuesand @master13:23
traveltissueshttps://gitlab.com/BuildStream/bst-plugins-experimental/-/issues/2713:23
WSalmoni yes i accently pused that after it had been merged, need to work out how to get rid of it13:24
*** phildawson_ has quit IRC13:36
*** phildawson has joined #buildstream13:45
*** phildawson has quit IRC13:48
traveltissuesWSalmon: i'm not sure what you mean by "we could key it off the cache key version maybe"13:49
traveltissuesdo you mean the job id?13:49
juergbitraveltissues: there is a cleanup issue which is why the runners were disabled13:51
juergbimanual cleanup seems to work, though13:51
juergbimaybe I should simply set up a cron job until this is solved properly13:51
*** narispo has quit IRC13:53
WSalmontraveltissues, you can add a key to the gitlab-cache, usually people use the branch or something like that as the key, but for us there is no point keeping cache after a cache key change so i wondered if we could find a way to do it off that but i think it would need to be manual13:53
*** narispo has joined #buildstream13:53
traveltissuesWSalmon: sure, that makes sense13:53
traveltissuesjuergbi: i think the issue is sharing this between jobs13:54
juergbiI don't think it's an issue with the gitlab runner cache, although I'm not completely sure13:54
juergbidocker containers and volumes from old jobs seem to stay around, filling up the disk13:54
juergbiright now it should be fine again but it's filling up quickly13:54
WSalmonoh sorry13:55
WSalmoni got the wrong end of that stick13:55
traveltissuesjuergbi: odd, is that only affecting bst-plugins-experimental? is the disk not shared over the organization?13:56
WSalmonim supprised that we get though that many docker images, i know we have 10 or so but we dont rotate them that often13:56
WSalmontraveltissues, i think this is effecting all the ci, is that right juergbi13:57
juergbitraveltissues: the disk is shared across projects13:57
juergbiWSalmon: afaict, it's about the overlays on top of the prebuilt images. mainly the .tox directory created as part of the job13:57
juergbibst-plugins-extra might be causing bigger .tox directories, so it affects that more13:58
juergbiI don't know whether it only happens for failed jobs or for all jobs. haven't looked into it that closely13:58
traveltissuesi've only seen this on failed jobs13:59
*** lachlan has quit IRC14:01
*** seanborg__ has joined #buildstream14:05
*** seanborg_ has quit IRC14:05
juergbitraveltissues: what I meant was, I don't know whether the docker volumes stay around also for jobs that completed successfully or whether only failed jobs contribute to this issue14:10
traveltissuesi see14:11
*** lachlan has joined #buildstream14:18
*** tristan has joined #buildstream14:27
*** ChanServ sets mode: +o tristan14:29
*** phildawson has joined #buildstream14:29
tristantraveltissues, Does https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/94 fix the crazy buildbox related CI breakage we've been seeing ?14:32
* tristan didn't expect that other factors would break after landing the BST_MIN_VERSION patch in bst-plugins-experimental14:33
tristandidn't see that coming :-/14:33
traveltissuesnot that i'm aware, if you mean the issue related to "<traveltissues> Error executing command: std::runtime_error exception thrown at [buildboxcommon_client.cpp:723]" that was resolved in !10214:33
tristantraveltissues, is it rebased on latest master of bst-plugins-experimental today ?14:35
tristanif so, then it accidentally fixes things14:35
* tristan thinks so because he sees a bunch of CI which happened post breakage14:35
traveltissuesit's rebased14:36
traveltissuesas of 2h ago at least14:36
tristanyeah, so without looking deep enough (it's late here, and I just went to have dinner)... I believe the recent breakage on master is related to having an outdated version of BuildBox in the bst-plugins-experimenal CI14:36
tristanI'm expecting that a recent change of default sandbox to use buildbox by default is what caused this breakage in bst-plugins-experimental CI14:37
tristanbecause I pointed tox.ini to recent master, naively thinking that _only_ my patch would have broken things14:37
traveltissues`824         [00:00:00] FAILURE project-a.bst: Cache too full`14:38
traveltissueson master14:38
traveltissuesso it might just be cache related14:38
traveltissues`error: Writing content object: min-free-space-percent '3%' would be exceeded, at least 4.1 kB requested`14:39
tristanyeah what I'm seeing is: The FUSE stager child process unexpectedly died with exit code 114:39
traveltissueson master?14:39
tristanfor a lot of tests14:39
tristanmaster bst-plugins-experimental14:39
tristanBut your branch is passing14:40
tristanso if you are rebased against master and pointing to latest BuildStream, then I believe you've inadvertently also fixed this issue14:40
tristanpossibly by updating a docker image or such14:40
traveltissuesi'm not seeing that on master: https://storage.googleapis.com/gitlab-gprd-artifacts/62/15/6215215f70b8b4f59a12d61af2e4629469304c721c3bebaafd34ea33393a3e3c/2020_04_29/531656522/578045880/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&GoogleAccessId=gitlab-object-storage-prd@gitlab-production.iam.gserviceaccount.com&Signature=d0iZYBXYdtIp9aQs%2FWHz3wmMUbYriXsB6tR7%2FsptGOqzMmtnrSP7lLiRj9Eu%14:40
traveltissues0AKbrgG87ZJjSD0WOX9PjyTKvs0OVwGZMzf%2FdLMoYSckWgnRdj7kp7VSoLFu%2Bi%0AaFqAQOF7D%2BbKYGsiF%2BUXCRNMUpPXbQEhzw9WtZ%2FpsVxDZZ2iUhzrOLUu9Waf%0A2zGD6DxpgNVmh6HrKFPlvMBY8ox6x6%2BbOEasokksxOmDFCw8M3lQclknZJaG%0AXKAWHyFR%2FosCAj3qAkuc9IDxYQUKBIkNH8LygYsENKA%2B%2BxnE7mEteVobbFoq%0AL8rKt2WvZLF6EkduedGG4SlTeY%2Bu1klGLlCTHw0XVA%3D%3D&Expires=158817181914:40
traveltissuesoops, sorry14:40
traveltissuesyes, that (docker image update) was a change in !10214:41
tristanAh14:42
tristanOk that makes sense, thanks for !102 :)14:42
traveltissuesnp14:42
*** tpollard has quit IRC15:01
*** tpollard has joined #buildstream15:01
*** lachlan has quit IRC15:28
*** lachlan has joined #buildstream15:54
*** seanborg__ has quit IRC15:54
*** lachlan has quit IRC16:05
*** seanborg__ has joined #buildstream16:06
*** seanborg_ has joined #buildstream16:16
*** seanborg__ has quit IRC16:16
*** lachlan has joined #buildstream16:19
*** lachlan has quit IRC16:21
*** lachlan has joined #buildstream16:22
*** lachlan has quit IRC16:30
*** lachlan has joined #buildstream16:33
*** jude has quit IRC16:34
*** lachlan has quit IRC16:40
*** lachlan has joined #buildstream16:42
*** seanborg_ has quit IRC16:49
*** lachlan has quit IRC16:50
*** lachlan has joined #buildstream16:52
*** lachlan has quit IRC16:59
*** tpollard has quit IRC17:11
*** santi has quit IRC17:16
*** lachlan has joined #buildstream17:51
*** lachlan has quit IRC18:03
*** benschubert has quit IRC19:18
*** phoenix has joined #buildstream19:25
*** phildawson has quit IRC19:26
*** phoenix has quit IRC22:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!