IRC logs for #buildstream for Wednesday, 2019-08-21

tlater[m]adds68: Can't say I have x)01:06
tlater[m]Will see if I can help pinpoint the problem tomorrow morning01:07
*** narispo has quit IRC01:40
*** narispo has joined #buildstream01:40
*** lantw44 has quit IRC03:05
*** lantw44 has joined #buildstream03:06
*** narispo has quit IRC03:17
*** narispo has joined #buildstream03:18
*** narispo has quit IRC03:48
*** narispo has joined #buildstream03:48
*** tristan has joined #buildstream05:43
*** tristan_ has joined #buildstream05:48
*** tristan_ has quit IRC06:07
*** tristan has quit IRC06:18
*** slaf has quit IRC06:29
*** slaf has joined #buildstream06:37
*** slaf has joined #buildstream06:58
*** SotK has quit IRC07:08
*** SotK has joined #buildstream07:08
*** tristan has joined #buildstream07:28
*** tristan has quit IRC07:29
*** tristan has joined #buildstream07:29
*** tristan has quit IRC07:36
*** tristan has joined #buildstream07:37
*** ahmed89 has joined #buildstream07:37
*** tristan_ has joined #buildstream08:02
*** tristan has quit IRC08:02
*** rdale has joined #buildstream08:08
*** tristan__ has joined #buildstream08:20
*** tristan_ has quit IRC08:21
*** ahmed89 has quit IRC08:40
*** ahmed89 has joined #buildstream08:40
*** tme5 has joined #buildstream08:44
jennisjuergbi, I'm a bit confused by your comment here: https://gitlab.com/BuildStream/buildstream/merge_requests/1553#note_20618160608:44
jennisArtifact does not have a "_ref", (ArtifactElement does)08:45
jennisWere you suggesting to handle Scope.NONE in ArtifactElement.get_dependency_refs?08:46
jennisnvm, I think I've addressed it08:51
juergbiI meant in Artifact08:53
benschubert_Rendezvous of RPC that terminated with:08:56
benschubertstatus = StatusCode.UNAVAILABLE08:56
benschubertdetails = "Empty update"08:56
benschubertdebug_error_string ="{\"created\":\"@1566377630.392977152\",\"description\":\"Failed to pick subchannel\"}"08:56
benschubertjuerbi: any idea what could be the problem here? (casremote.py, while doing a push). This has happened twice in two days (no idea how to reproduce either...)08:56
juergbihm, 'failed to pick subchannel' doesn't sound familiar08:57
juergbido you have a stack trace?08:57
benschubertsure, let me clean it up and I'll post a snippet08:58
juergbita08:58
benschubertjuergbi: https://gitlab.com/snippets/188766409:01
*** tpollard has joined #buildstream09:03
*** phildawson_ has joined #buildstream09:04
juergbihm, that's the only part that doesn't go via casd (yet, should be fixed). so that part did actually not change with casd support, afaict09:08
juergbibenschubert: 'twice in two days', I assume the previous one was also already after casd merge?09:08
benschubertjuergbi: ah we are not yet on the casd version09:09
juergbiah, general issue09:09
benschubertthis is a BuildStream at commit 57a2baaa3a260f6a6dd50552ec8c0d71e1b8ccf109:09
juergbiif you never saw this before, is it possible there was a gRPC update on either client or server?09:09
juergbisearching the web there are a few places where the 'empty update' issue is mentioned but not with a clear cause/solution, afaict09:10
benschubertboth client and servers are using this exact commit of BuildStream and haven't been redeployed in a few weeks09:11
juergbiok, I don't have an idea off the top of my head09:13
benschubertOk, do you think it's worth opening an issue on BuildStream for that?09:14
juergbiwe should at least be able to improve our error handling09:15
juergbisuch that we cover as much as possible with the existing retry mechanism, which I believe didn't work here09:16
benschubertjuergbi: agreed, I'll open an issue09:16
benschubertalso concerning   https://gitlab.com/BuildStream/buildstream-docker-images/issues/46 would march=x86_64 be good or do we want to support only a subset of those with "recent" (sse2) optimizations?09:17
benschubertjuergbi: what do you think?09:17
juergbiSSE2 is a core part of x86-6409:18
juergbiso I think march=x86_64 makes sense09:18
benschubertok! I'll add a patch for this09:18
juergbiok. I don't expect a big performance difference with AVX, so let's keep it generic09:19
benschubertright09:20
juergbi(although original AVX is probably widespread enough)09:20
*** traveltissues has joined #buildstream09:21
benschubertyeah, not sure it's a huge issue :)09:23
benschubertjuergbi: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/13109:25
*** jonathanmaw has joined #buildstream09:25
juergbibenschubert: checking the man page, it's actually called -march=x86-6409:28
juergbidon't know whether it would also accept x86_6409:28
benschubertousp, let me fix this09:28
*** lachlan has joined #buildstream09:29
*** ahmed89 has quit IRC09:35
*** phil has joined #buildstream09:35
*** phildawson_ has quit IRC09:36
*** ahmed89 has joined #buildstream09:36
*** lachlan has quit IRC09:40
*** tristan__ has quit IRC09:41
benschubertjennis: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/131 you had the bug triggering right? could you try running this new image  locally? :)09:44
*** lachlan has joined #buildstream09:44
jennisbenschubert, sure09:56
jennisbenschubert, works for me :)10:05
tme5is there a pytest.ini?10:05
tlater[m]tme5: It's all in the tox config10:06
tme5or is config done by conftest.py10:06
tme5ah ok10:06
tlater[m]tme5: conftest.py mostly sets up fixtures10:07
benschuberttme5: pytest config is done in setup.cfg10:07
tlater[m]Oh, setup.cfg10:07
* tlater[m] o\10:08
tme5does the datafiles mark get registered anywhere? basically just need to register a mark10:08
tme5i can put it in setup.cfg10:08
*** lachlan has quit IRC10:09
juergbijennis: you've built the docker image locally? that wouldn't verify the fix as the issue is when the build system has AVX 512 support10:09
juergbihowever, I just pulled the CI-built image and it works fine here as well10:09
benschuberttme5: yep setup.cdg :)10:10
benschubert*cfg10:10
jennisah, juergbi, my bad, I was just responding to: <benschubert> jennis: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/131 you had the bug triggering right? could you try running this new image locally? :)10:10
benschubertjuergbi: your laptop is recent though :)10:11
tme5i can't see it there haha10:11
tme5but i will put my mark there, thanks10:11
tlater[m]That particular mark is built-in, I think10:11
tme5ooh ok10:11
tlater[m]So you won't see it anywhere :)10:11
tlater[m]Might be a dependency, but I don't think so.10:12
tme5oh, it's not, at least not latest pytest10:12
tme5i'll do some grepping10:12
tlater[m]Yeah, you're right, it's a dependency: https://pypi.org/project/pytest-datafiles/10:12
tme5requirements/dev-requirements.in:pytest-datafiles >= 2.010:12
tme5aye10:12
tlater[m]So you won't find it registered10:12
tme5alright, thanks all, I'll use setup.cfg10:12
tlater[m]We do have an integration mark10:12
tlater[m]That should be set up somewhere10:13
benschubertJennis but did you pull the image from the gitlab ci registry to test?10:13
juergbibenschubert: no AVX-512 laptop has shipped yet, not counting one extremely rare cannon lake laptop10:14
tlater[m]tme5: Yup, that's in conftest. But the mark doesn't *do* anything so it's not registered10:14
juergbiand I could reproduce the issue before10:14
tlater[m]You'll need to come up with the pattern yourself. The pytest docs are ok on this iirc10:14
*** lachlan has joined #buildstream10:15
tme5ok, ty, I suppose I'll register the other two as well, for good measure?10:15
tlater[m]Which other two? integration/datafiles?10:15
tlater[m]Why?10:15
tme5yes, and remoteexecution -- well, the docs say marks ought to be registered10:16
tme5(though they say unregistered marks raise warnings, so i'm not sure where they have gone..)10:17
jennisbenschubert, where is it? I don't see it here: https://gitlab.com/BuildStream/buildstream-docker-images/container_registry10:17
tlater[m]I'm fairly sure that "registering" just means that pytest will do whatever they need to do when it encounters them10:17
tlater[m]So for datafiles, it's already registered10:17
tlater[m]And for integration it makes no sense.10:17
tme5it says here registration of marks is just so pytest knows you typed what you meant https://docs.pytest.org/en/latest/mark.html10:18
*** ahmed89 has quit IRC10:18
tme5and they show in pytest --markers or something10:18
tlater[m]Could you try that command and see what it lists?10:19
tme5just all built-in marks rn10:20
tlater[m]Right, I suppose you can add the custom marks for good measure then. Nice that they can be documented.10:20
juergbijennis: registry.gitlab.com/buildstream/buildstream-docker-images/buildstream:nightly-bschubert-fix-segfault-7749849710:23
jennisthanks juergbi10:33
*** lachlan has quit IRC10:37
jennisbenschubert, juergbi, that worked for me10:37
gitlab-br-bottraveltissues approved MR !1541 (tmewett/artifact-help->master: Clarify bst artifact subcommand help text) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/154110:39
*** lachlan has joined #buildstream10:42
*** ahmed89 has joined #buildstream10:51
benschubertjuergbi: jennis oh ok, sorry about the misunderstanding, perfect I'll merge it then, thanks!10:53
*** lachlan has quit IRC11:17
benschubertjuergbi: do you have an idea what I could be missing in https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/276872719 ? I'm trying to add buildbox to the nightly image11:30
tpollardI think qinusty had that issue yesterday? ^11:35
juergbibenschubert: oh, it should be libuuid-devel, not uuid-devel11:41
juergbilooks like a bug in the testsuite image (and documentation) as well, the right package probably just happens to be installed there as well11:42
juergbiqinusty: ^^ regarding the buildbox build issue you had11:42
*** tristan__ has joined #buildstream12:00
*** tristan_ has joined #buildstream12:05
*** tristan__ has quit IRC12:06
*** lachlan has joined #buildstream12:07
*** lachlan has quit IRC12:10
*** ahmed89 has quit IRC12:13
benschubertOh thanks!12:25
qinustyOhhhh, thanks juergbi :D12:25
qinustyI think the intention is for remoteapistesting to move to using buildstream:nightly-extra once it's up and running. We currently build our own image which is effectively the same thing12:25
tme5in tests/internals/storage_vdir_import.py, tests have been written in a way which reduces the number of tests parametrisations that are written in the output (10 instead of 100, afaict)12:26
tme5as part of my changes it would be a lot cleaner to have to 10x10 test parametrised normally, would that be ok? Otherwise ten tests at a time would run in the same subprocess12:27
qinustyjuergbi I've raised https://github.com/buildbarn/bb-remote-execution/issues/18 with Buildbarn in relation to the issues seen when trying to use buildstream with buildbarn12:28
juergbita12:33
benschuberttme5: not sure I understand, what do you mean by " which reduces the number of tests parametrisations that are written in the output" ?12:37
tme5benschubert, I am referring to the comment at L27612:37
tme5and the general form of the following test12:38
benschuberttme5: the "test_descend" ? I don't see any parametrization there12:39
tme5oh wait, let me git fetch haha, the comment is before test_fixed_cas_import12:40
benschubertoh I see12:40
benschubertotherwise, do we _really_ need the value for each value? Or could we have like 1,2,5,10? or something like that?12:41
tme5yeah, the random ones could be reduced, the import test has a preset list of 10 example file trees to work with, i guess that could be reduced12:45
benschubertwas there a reason for that many?12:53
benschubertjuergbi: https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/276960738 is this expected or did I forget something in the dependencies?12:54
juergbibenschubert: do you mean the size? it depends on whether it's statically linked, I suppose12:56
benschubertjuergbi: I meant the stderr output (red) about not finding grpc.cmake :)12:56
juergbibenschubert: ah, yes, it's expected12:57
benschubertok! thanks12:57
juergbidue to dual checks in cmake to cover platforms without pkgconfig12:58
juergbi(probably room for improvement but that's an upstream buildbox question)12:58
juergbibenschubert: doesn't work yet, see comment12:59
benschubertjuergbi: thanks yeah, seems I'm removing too much :)13:06
juergbishould probably try to keep testsuite and nightly as close as possible13:07
juergbiideally, we could share some parts of the templates. not sure whether that would be trivial13:07
juergbimight be something for another MR13:07
benschubertyeah, I'm thinking about transforming the definitions to python or add a jinja level :)13:11
*** lachlan has joined #buildstream13:12
*** lachlan has quit IRC13:16
*** lachlan has joined #buildstream13:35
*** lachlan has quit IRC13:42
tlater[m]benschubert: BuildStream!13:44
benschuberttlater[m]: what?13:44
tlater[m]We could convert the definitions into a buildstream project!13:45
tlater[m];(13:45
tlater[m]s/(/)/13:45
benschubertnot 100% sure about that :D at least not now13:46
*** tristan__ has joined #buildstream13:55
juergbiI was thinking that as well. we'd have a potential bootstrapping issue, however, we could build it with bst-1, I suppose13:55
juergbibut then we'd still need to build a bst-1 docker image without buildstream13:56
*** tristan_ has quit IRC13:56
juergbialso, distro packages would still have to be installed via distro package manager, unless we want to build static / distro-independent binaries13:56
tlater[m]juergbi: Yeah, that latter point is the main issue IMO13:57
tlater[m]It would work pretty well for the user facing image (because nobody needs fedora to be part of their docker image when they just want buildstream), but not so well for the distro-testing images.13:57
*** toscalix has joined #buildstream14:02
*** traveltissues has quit IRC14:06
*** benschubert_ has joined #buildstream14:15
*** jonathanmaw has quit IRC14:16
*** benschubert has quit IRC14:16
*** swick has quit IRC14:16
*** lchlan has quit IRC14:16
*** skullone[m] has quit IRC14:16
*** tchaik[m] has quit IRC14:16
*** theawless[m] has quit IRC14:16
*** krichter[m] has quit IRC14:16
*** reuben640[m] has quit IRC14:16
*** jjardon[m] has quit IRC14:16
*** segfault3[m] has quit IRC14:16
*** Trevinho[m] has quit IRC14:16
*** rafaelff[m] has quit IRC14:16
*** asingh_[m] has quit IRC14:16
*** mohan43u has quit IRC14:16
*** johnward has quit IRC14:16
*** hergertme has quit IRC14:16
*** bethw has quit IRC14:16
*** mrmcq2u[m] has quit IRC14:16
*** jswagner has quit IRC14:16
*** benschubert_ is now known as benschubert14:16
*** lchlan has joined #buildstream14:16
*** jonathanmaw has joined #buildstream14:16
*** swick has joined #buildstream14:16
*** skullone[m] has joined #buildstream14:16
*** tchaik[m] has joined #buildstream14:16
*** theawless[m] has joined #buildstream14:16
*** krichter[m] has joined #buildstream14:16
*** reuben640[m] has joined #buildstream14:16
*** jjardon[m] has joined #buildstream14:16
*** segfault3[m] has joined #buildstream14:16
*** Trevinho[m] has joined #buildstream14:16
*** rafaelff[m] has joined #buildstream14:16
*** asingh_[m] has joined #buildstream14:16
*** mohan43u has joined #buildstream14:16
*** johnward has joined #buildstream14:16
*** hergertme has joined #buildstream14:16
*** bethw has joined #buildstream14:16
*** mrmcq2u[m] has joined #buildstream14:16
*** jswagner has joined #buildstream14:19
*** jswagner has joined #buildstream14:20
*** phil has quit IRC14:21
*** lachlan has joined #buildstream14:42
tme5i get pycodestyle warnings locally on files I haven't changed, wondering if there is a config i have to load or something14:52
tlater[m]Yeah, setup.cfg :D14:53
*** lachlan has quit IRC14:54
tme5ahh brilliant haha, but looks like it loads that automatically anyway, and i'm getting E302 which isn't on the list14:55
tme5i see the latest master pipeline failed, but maybe unrelated?14:56
*** phil has joined #buildstream14:59
*** lachlan has joined #buildstream15:03
tlater[m]Unrelated, yeah15:05
tlater[m]Perhaps you accidentally based your branch on something that wasn't quite readyu15:05
tlater[m]*ready15:05
tme5i'm on master, so are the offending lines. well, it can't be an issue as linting CI seemed to pass fine15:06
tpollardhow are you linting?15:08
tme5just `pycodestyle tests` (where i'm working)15:08
tlater[m]Try `tox -e lint` :)15:08
*** lachlan has quit IRC15:12
gitlab-br-botbeckyella16 opened MR !1555 (becky/list_contents_long_option->master: Addition of --long option to list-contents) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/155515:18
*** tristan__ has quit IRC15:23
tme5i'm v happy with this subprocess tests change, it's simplified a lot of the concerned tests15:27
tlater[m]Does anyone know where our CAS server decides what directories to put blobs in?15:35
tlater[m]I'm afraid I may have broken it15:35
tlater[m]I'm getting all blobs dumped in the root directory, but our test suite refuses that because it's supposed to have per-element subdirectories.15:36
tpollardloads it in context I think15:36
tpollardwell that'd hopefully where we determine where to put the blobs, if everything is working....15:37
tlater[m]Ah, I see something that looks like it15:38
tlater[m]Hm, this might be entirely broken with split caches15:38
* tlater[m] digs a little15:38
tlater[m]ta tpollard!15:39
gitlab-br-botcoldtom opened MR !1556 (coldtom/update-plugin-tests->master: Remove test dependency on bst-plugins-experimental) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/155615:39
tlater[m]Eugh, testing the cas server with the test suite is a PITA15:39
tlater[m]No error messages :|15:39
*** lachlan has joined #buildstream15:41
tlater[m]tpollard: Looks to be determined by the cache key in the request15:41
tlater[m]Interesting, I wonder if I can escape the cache directory by giving a cache key like ../../../etc/passwd15:42
*** lachlan has quit IRC15:48
*** phildawson_ has joined #buildstream15:54
tlater[m]juergbi: Is ^ something we care about?15:54
tlater[m]I've tried it, it looks like I can15:55
*** phil has quit IRC15:55
tlater[m]-> Change the artifact key in UpdateArtifact to some random path15:55
adds68Hi all, i've configured a buildstream cache on port 1102, i've set this in my project.conf with the correct push keys15:59
adds68When i start buildstream, it initializes the cahe with "success"15:59
adds68During a build i see "SUCCESS Caching artifact"16:00
adds68however when i view the cas directory on the sever it is empty?16:00
adds68there are no logs on my sever, so i'm so confused16:01
tlater[m]Caching artifact means that it is locally cached16:01
tlater[m]Nothing more, nothing less16:01
tlater[m]You'd need to see messages along the lines of 'SUCCESS Pushed artifact to https://adds68.com'16:01
adds68tlater[m], interesting...16:02
tlater[m]It's a cute little -> instead of 'to', actually16:02
tlater[m]Might make it easier to skim :)16:02
adds68tlater[m], coldtom pointed out to try bst push16:03
adds68however i see16:03
adds68[00:00:02][][] FAILURE Push16:03
adds68No artifact caches available for pushing artifacts16:03
tlater[m]adds68: This is what it looks like for me currently (message slightly altered): https://hastebin.com/iqaqireciz.md16:03
tlater[m]adds68: That means that buildstream is unable to contact your cache and/or you didn't set `push: true` in project.conf16:04
adds68omfg16:05
* adds68 facepalm16:05
adds68wow16:05
adds68i need beer16:05
adds68now16:05
ironfootwe both missed the "push: true" bit :)16:05
ironfootthanks16:05
tlater[m]Hehe16:05
* tlater[m] has been there before16:06
tlater[m]These error messages are in dire need of improving, both on the server and client.16:06
*** lachlan has joined #buildstream16:08
*** lachlan has quit IRC16:17
tme5benschubert, I reduced the number of test cases in that storage_vdir_import suite. Will open my MR tomorrow, hope my analysis of the cases ends up being ok haha16:18
tme5just an fyi in case you wanted to make sure16:18
*** lachlan has joined #buildstream16:21
*** narispo has quit IRC16:22
*** narispo has joined #buildstream16:23
*** narispo has quit IRC16:26
*** lachlan has quit IRC16:27
*** narispo has joined #buildstream16:27
*** jonathanmaw has quit IRC16:28
*** phildawson_ has quit IRC16:31
gitlab-br-bottmewett opened MR !1557 (tmewett/test-in-subprocess->master: Add in_subprocess pytest mark and modify tests which run in another process to use it) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/155716:33
*** lachlan has joined #buildstream16:34
*** lachlan has quit IRC16:51
*** lachlan has joined #buildstream16:52
*** tme5 has quit IRC16:59
*** lachlan has quit IRC17:09
*** toscalix has quit IRC17:14
*** lachlan has joined #buildstream17:21
*** johnward has quit IRC17:21
*** johnward has joined #buildstream17:25
*** adds68 has quit IRC17:25
*** lachlan has quit IRC17:25
*** ikerperez has quit IRC17:25
*** Becky has quit IRC17:25
*** samkirkham has quit IRC17:25
*** qinusty has quit IRC17:25
*** paulsherwood has quit IRC17:26
*** paulsherwood has joined #buildstream17:26
*** samkirkham has joined #buildstream17:26
*** ikerperez has joined #buildstream17:26
*** qinusty has joined #buildstream17:26
*** adds68 has joined #buildstream17:26
*** Becky has joined #buildstream17:26
*** adds68 has quit IRC17:33
*** qinusty has quit IRC17:33
*** samkirkham has quit IRC17:33
*** Becky has quit IRC17:33
*** ikerperez has quit IRC17:33
*** jennis has quit IRC17:34
*** laurence has quit IRC17:34
*** valentind has quit IRC17:34
*** johnward has quit IRC17:34
*** WSalmon has quit IRC17:34
*** bethw has quit IRC17:34
*** jennis has joined #buildstream17:36
*** johnward has joined #buildstream17:36
*** paulsherwood has quit IRC17:36
*** lachlan has joined #buildstream17:37
*** valentind has joined #buildstream17:37
*** bethw has joined #buildstream17:39
*** laurence has joined #buildstream17:39
*** WSalmon has joined #buildstream17:39
*** lachlan has quit IRC17:48
*** lachlan has joined #buildstream17:52
*** adds68 has joined #buildstream17:59
*** lachlan has quit IRC18:03
*** narispo has quit IRC20:51
*** narispo has joined #buildstream20:52
*** rdale has quit IRC22:18

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!