IRC logs for #buildstream for Thursday, 2019-08-08

*** rabbitear has joined #buildstream00:30
*** tristan has joined #buildstream00:31
*** tristan_ has joined #buildstream00:43
*** tristan has quit IRC00:51
*** tristan has joined #buildstream00:52
*** tristan has quit IRC01:50
*** tristan has joined #buildstream01:52
*** tristan has quit IRC01:53
*** tristan has joined #buildstream01:53
*** tristan has quit IRC01:55
*** tristan has joined #buildstream01:55
*** tristan has quit IRC01:56
*** tristan has joined #buildstream01:57
*** tristan has quit IRC01:59
*** rabbitear has quit IRC02:04
*** tristan has joined #buildstream04:01
*** tristan has quit IRC04:52
gitlab-br-botjuergbi opened MR !1527 (juerg/artifact-blobs->master: Artifact blob handling) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152705:26
*** tristan has joined #buildstream05:49
*** tristan has quit IRC05:54
*** tristan has joined #buildstream06:03
*** tristan has quit IRC06:52
*** tristan has joined #buildstream06:57
*** tristan has joined #buildstream07:06
juergbiKinnison: I've looked into generating a wheel for a static buildbox-casd binary, however, it doesn't seem ideal. afaict, I could only include the binary as data (file will not be marked as executable and not be put in 'bin') or script (pip attempts to insert #!/bin/python line, oops)07:11
juergbithe only alternative for automatic installation I can think of is to create a fake Python source tarball. i.e., a tarball that contains the static binary + a setup.py script that installs the static binary07:11
KinnisonUgh, that is not nice, is it?07:12
* Kinnison wonders what other python libraries do which need to install tools07:12
juergbithe build system interface expected by `pip install` is documented at https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface07:12
juergbiso this might actually be reliable but it doesn't seem ideal07:12
juergbiI checked grpcio-tools, which includes the protobuf compiler. they put the whole code into a .so Python C extension module. I don't really want to do this07:13
juergbinot sure about other packages07:13
juergbido you anticipate any actual issues with the tarball approach or does it simply feel odd?07:13
juergbiI'm not aware of any other pip-installable formats that would be suitable but maybe I'm missing something07:14
juergbii.e., the only other alternative I see would be manual download/installation of the binary07:15
KinnisonI guess short-term the tarball approach is acceptable.  Maybe benschubert might know of some other approach?07:16
juergbi(well, could also use a git repo instead of a tarball but that doesn't really improve the situation)07:17
*** toscalix has joined #buildstream07:48
*** tristan has quit IRC07:50
*** tristan has joined #buildstream07:52
*** tristan has quit IRC07:53
*** tristan has joined #buildstream07:56
*** tristan has joined #buildstream07:58
*** tristan has quit IRC07:59
*** tristan has joined #buildstream07:59
*** tristan has joined #buildstream08:03
*** tristan has quit IRC08:05
*** tristan has joined #buildstream08:05
qinustyI think nightly-extra is broken :( https://paste.gnome.org/pubilrn8v/v3mfie/raw08:10
laurencevalentind is on holiday, however we ensured last week that tpollard and jennis also had credentials to have a poke around the infra08:14
jjardonqinusty: probably because plugins from bst-plugings-experimental are broken at the moment https://gitlab.com/BuildStream/buildstream/issues/109008:16
qinustyah08:17
benschubertKinnison, juergbi I am not aware of any good solution. What other programs I know usually do it ask for the program to be installed on the host directly. I also think that this is in line with the rest of BuildStream (we do require bwrap, fuse), even though it might not be super simple due to buildbox not being commonly available on distributions yet. On our side, we would like to be able to build buildbox ourselves08:17
jjardonThere is a MR linked on that issue which tries to fix the plugins08:17
juergbibenschubert: adding buildbox-casd build instructions to the documentation is on my TODO. maybe we should just do that + as a convenience upload a static binary and write instructions how to download/install it for those who'd like to skip the build08:19
juergbithe second part could potentially follow post merge, although lack of that binary is an issue for WSL CI08:20
juergbiwe could always pursue the pip-installable tarball route later08:21
benschubertjuergbi: that seems good to me (as long as the pip-installable is not required :D)08:22
benschubertjuergbi: did I read the benchmarks correctly, the building is slower with casd?08:36
juergbiyes08:36
benschubertDo you have an idea where this is coming from? Is it expected?08:37
juergbithis might be due to IPC overhead without file batching08:37
juergbibatching / capturing whole directory with single call is already a planned optimization08:37
juergbihowever, I have not confirmed that this is the reason for the overhead08:37
benschubertok, thanks!08:38
jennisI'm trying to get the logs from the proto. Currently, we have the Artifact.get_files() and Artifact.get_buildtree() methods which "get" a field digest, and then create a CASBasedDirectory with this08:40
juergbiexpiry should be much faster, though :) will be interesting to get real user feedback in that aspect08:40
jennisThis works because files and buildtree's are both build.bazel.remote.execution.v2.Digest08:40
jennisHowever, now I'm trying to do something similar with the logs, which are defined differently in the proto08:40
jennisHow should I go about getting these?08:41
KinnisonI'd suggest returning a dict of name -> casbaseddirectory08:43
juergbithe digests are files, though, not directories08:43
Kinnisonoops, yes, erm08:43
juergbijennis: maybe it would make sense to create an empty CASBasedDirectory and add the log files08:43
*** alexandrufazakas has left #buildstream08:43
juergbii.e., create a virtual directory08:43
*** alexandrufazakas has joined #buildstream08:43
jennisok sure, but it's still the getting of each log file I'm not sure about08:44
juergbithe `logs` field should simply be a python list08:44
jennisoh, so when we get the proto (artifact = self._get_proto() ) artifact.logs  is just a list?08:45
juergbieach entry being a LogFile proto where you can get 'name' and [file] 'digest'08:45
juergbiyes08:45
jennis<class 'google.protobuf.pyext._message.RepeatedCompositeContainer'>08:45
jennisis type(artifact.logs)08:46
juergbiwell, not quite a standard list but you should be able to loop over it08:46
jennisok thanks, and with each logfile, I can create a use the digest and create a CASBasedDirectory with it08:47
juergbiwe might actually not have an API to add files by digest to a CASBasedDirectory08:48
juergbibut it should be fairly easy to add this, at least as internal API08:48
*** phildawson has joined #buildstream08:48
jennis        return CasBasedDirectory(self._cas, digest=buildtree_digest)08:48
juergbithat's for directories08:48
juergbifor logs we need to create an empty directory and then add the file digests08:48
jennisah, gotcha08:49
jennis09:43 <juergbi> the digests are files, though, not directories08:49
jennisI'm playing catch up08:49
juergbijennis: maybe it would have been better to just create a Directory for logs as well08:49
juergbinot sure why we didn't08:49
juergbi(in the proto, I mean)08:49
jennisI guess that'd be more consistent with "files"08:50
*** tristan has quit IRC08:51
jennisjuergbi, which would you prefer?08:52
Kinnisonjuergbi: We didn't because it was yet another object which we didn't want to have to create and push08:52
KinnisonI'd rather that Artifact.get_logs() returned a dict of name -> PutativeCasBlob08:53
KinnisonWhere PutativeCasBlob offers a .read()08:53
Kinnisonor somesuch08:53
juergbiwe could return name => objpath(digest), assuming we ensure the log files are local first08:54
KinnisonAlso acceptable08:54
juergbiobjpath being the real file path to the CAS blob08:54
* Kinnison nods08:54
juergbi(I'd rather not implicitly open files)08:55
KinnisonPutativeCasBlob in my proposal wouldn't until you tried to read it08:55
KinnisonBut I'm fine with `objpath`08:56
jenniscool, lets do that for now then08:57
*** tristan has joined #buildstream09:03
*** tristan has quit IRC09:05
*** tristan has joined #buildstream09:08
*** tristan has quit IRC09:10
*** tristan has joined #buildstream09:11
*** jonathanmaw has joined #buildstream09:12
jennisoh, just getting the object paths made the code much simpler09:12
jennisNo need for vdirs09:12
*** tristan has quit IRC09:13
*** tristan has joined #buildstream09:13
*** bochecha has joined #buildstream09:29
*** tme5 has joined #buildstream09:35
*** tme5 has quit IRC09:41
*** tme5 has joined #buildstream09:42
*** tme5 has left #buildstream09:42
*** tme5 has joined #buildstream09:46
*** tristan has quit IRC09:50
*** tristan has joined #buildstream09:51
tme5what do you think would be the best way to create a job that exits with a code not in _ReturnCode ?09:52
tme5trying to create a local test for https://gitlab.com/BuildStream/buildstream/issues/108909:52
* Kinnison grumbles that gitlab thinks his branch doesn't exist09:52
*** tristan has quit IRC09:52
*** tristan has joined #buildstream09:53
*** tristan has quit IRC09:55
*** tristan has joined #buildstream09:55
*** tristan has quit IRC09:56
KinnisonI particularly like how it can show the commits on the MR09:56
Kinnisonand the diffs09:56
KinnisonBut believes the branch doesn't exist09:57
*** tristan has joined #buildstream09:57
*** tristan has quit IRC09:59
*** tristan has joined #buildstream09:59
benschuberttme5: you could have the plugin raise SystemExit(137)   for example :)10:01
benschubertactually not 137, since this would be a segfault10:01
benschubertbut another number10:01
benschubert(otherwise we might think it is a real segfault)10:01
*** tristan has quit IRC10:02
*** tristan has joined #buildstream10:03
*** tristan has quit IRC10:05
KinnisonThe git repo itself is insistent it exists10:06
* Kinnison is very confused10:06
tpollardgitlab seems to be having a bad day10:07
KinnisonIt seems possible to review the MR, just not merge it :/10:07
tpollardit couldn't open a 3 line diff for me earlier10:07
KinnisonAnd no pipeline10:07
tpollardyep I can see that branch locally10:10
*** narispo has quit IRC10:11
*** narispo has joined #buildstream10:11
*** tristan has joined #buildstream10:24
tme5benschubert, thanks, I ended up doing that, just stuck a raise in ChildElementJob.child_process10:24
*** tristan has quit IRC10:25
benschubertthat seems good!, also please ping me once you have the MR up, I'm interested by this one :)10:25
*** tristan has joined #buildstream10:26
tme5benschubert, i have it in a local branch now, not sure how you want me to submit, do i need permissions to push to main repo?10:26
*** tristan has quit IRC10:27
*** tristan has joined #buildstream10:33
benschuberttme5: I'll add you as a dev, what's your gitlab handle?10:35
*** tristan has quit IRC10:35
tme5thanks, it's tmewett10:35
*** tristan has joined #buildstream10:35
benschubertdone10:36
*** tristan has quit IRC10:36
*** tristan has joined #buildstream10:37
tme5thanks10:41
*** tristan has quit IRC10:43
*** tristan has joined #buildstream10:43
*** tristan has quit IRC10:45
*** tristan has joined #buildstream10:45
*** narispo has quit IRC10:49
*** tristan has quit IRC10:51
*** tristan has joined #buildstream10:51
*** tristan has quit IRC10:58
*** tristan has joined #buildstream11:02
*** tristan has quit IRC11:03
*** tristan has joined #buildstream11:04
*** tristan has quit IRC11:05
*** tristan has joined #buildstream11:06
*** tristan has quit IRC11:07
*** tristan has joined #buildstream11:08
*** tristan has quit IRC11:09
*** tristan has joined #buildstream11:09
laurenceAhhh11:09
laurenceGitlab incident today, Kinnison11:10
laurenceand all11:10
laurencehttps://status.gitlab.com/11:10
*** tristan has quit IRC11:11
*** tristan has joined #buildstream11:11
*** tristan has quit IRC11:13
*** tristan has joined #buildstream11:13
*** tristan has quit IRC11:15
*** tristan has joined #buildstream11:16
*** tristan has quit IRC11:19
*** tristan has joined #buildstream11:20
*** tristan has quit IRC11:21
*** tristan has joined #buildstream11:22
*** tristan has quit IRC11:23
tlater[m]Is there a way to iterate over a mappingnode?11:23
*** tristan has joined #buildstream11:25
*** tristan has quit IRC11:27
benschuberttlater[m]: it supports .items(), .keys() and .values() like a dict11:27
tlater[m]Ah, but not `for x in node` then?11:27
benschuberttlater[m]: it wasn't implemented no, but you can do `for x in node.keys()` which is the same behavior11:29
tlater[m]Yep, I overlooked that11:29
tlater[m]Thanks benschubert11:29
tme5benschubert, as requested: https://gitlab.com/BuildStream/buildstream/merge_requests/153111:33
tpollardI think you need to set your committer email for git tme512:05
Kinnisonbenschubert: I've added that cdef as per your request, and pushed, but gitlab's dippytime still hasn't resolved so it might take a while12:11
tme5tpollard, yeah i realised, done so now, do i need to re-do anything?12:21
* Kinnison wonders why gitlab's UI takes so long to realise a push has happened12:21
juergbiI think normally it's fairly quick but occasionally it's slow12:25
KinnisonThey have an incident going on right now, but I'm just wondering what about that part is slow, when you can comment/deal with things on MRs instantly12:26
benschubertKinnison: cheers, LGTM then!12:33
Kinnisonbenschubert: Just have to wait for gitlab to catch up :D12:33
juergbihm, can I assign an MR to marge despite gitlab not seeing the pushed update yet?12:41
KinnisonI doubt that'd be safe12:41
juergbiwe need a bot to assign an MR to marge when gitlab sees the changes ;)12:42
tme5stupid question maybe, but where is the ostree source implemented?12:48
benschuberttme5: in https://gitlab.com/BuildStream/bst-plugins-experimental for master, in https://gitlab.com/BuildStream/bst-plugins-external for v1.X12:50
tpollardtme5: still looks like you need to ammend the author email on that commit, and then force push it12:53
*** narispo has joined #buildstream13:00
tme5tpollard, do I do that by resetting and re-commiting?13:03
tpollardtme5: "git commit --amend --author="Author Name <email@address.com>"13:04
tpollardshould be faster than recommitting :)13:05
*** narispo has quit IRC13:12
gitlab-br-botcoldtom closed MR !1523 (coldtom/cython-req->master: Add cython as a dev requirement) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152313:16
*** narispo has joined #buildstream13:21
gitlab-br-botmarge-bot123 merged MR !1520 (coldtom/move-cmake->master: Move cmake plugin to bst-plugins-experimental) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152013:25
*** tristan has joined #buildstream13:29
*** narispo has quit IRC13:36
*** narispo has joined #buildstream13:36
Kinnisonbenschubert: looks like the commits are showing up for my MR now, so as and when the pipeline goes green, I'll marge it if it has your 👍13:43
laurenceis all the opening and clsing of MRs today due to gitlab issues?13:47
*** tristan has quit IRC13:53
*** tristan has joined #buildstream13:53
*** tristan has quit IRC13:58
Kinnisonprobably14:05
*** tristan has joined #buildstream14:07
gitlab-br-botdanielsilverstone-ct opened MR !1528 (danielsilverstone-ct/load-reject-duplicates->master: Reject elements which have duplicated dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152814:11
tme5laurence, i screwed up mine due to unfamiliarity with GitLab14:12
tme5(i wish buttons would give more of a warning when they randomly create branches and MRs...)14:13
qinustyI find it best to do everything via git CLI, I only really use gitlab for MRs14:15
qinustyLocal branch creation can be done with `git checkout -b <branch-name> <base branch>`14:16
qinustypushing with `git push origin <remote branch name>` will create the branch for you, and then give you a URL to click for creating an MR (at least with gitlab)14:17
gitlab-br-botBenjaminSchubert approved MR !1500 (tpollard/messageobject->master: Message rework) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/150014:23
adds68Hi, does anyone know if "BST_ARTIFACTS_ONLY=1" has any effect when installing from pip?14:23
adds68or does that only work when installing from source14:23
gitlab-br-bottmewett opened MR !1531 (tmewett/report-weird-return-codes->master: Report error when job process unexpectedly dies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153114:32
tme5so to clarify, changes to ostree plugin go in bst-plugins-experimental ?14:34
benschuberttme5: correct14:36
laurencetme5, i see. I saw the same thing happen on another project you see. but perhaps that was also a hunman error14:37
benschubertadds68: if you use a source distribution, definitely, if using a wheel, I would guess so, but not 100% sure14:37
adds68benschubert, hey i think i am going to use instead: https://hub.docker.com/r/buildstream/artifact-cache14:38
benschubertthat also works :)14:39
adds68benschubert, i don't think it existed when fdsdk cache was set up, so less pain to stay in line with upstream now :D14:39
benschubertdefinitely14:40
*** alexandrufazakas has left #buildstream14:40
gitlab-br-botjennis opened MR !1533 (jennis/fix_artifact_log->master: Fix artifact log) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153314:40
adds68benschubert, if i'm going to be setting up the cache via systemd unit files, do you think it would be easier to install the bst-here script discussed here: https://buildstream.build/docker_install.html ?14:46
adds68I'm unsure if that would be easier than just calling bst from docker/podman14:47
benschubertadds68: if you want to start an artifact cache, I would just start the docker image instead of going through bst-here (which I'm not even sure supports this usecase)14:47
adds68cool thanks benschubert, i was hoping that to be the answer :)14:48
gitlab-br-botmarge-bot123 merged MR !1527 (juerg/artifact-blobs->master: Artifact blob handling) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152714:48
* Kinnison assigns his MR to marge in the hope it'll come before some other in the queue and actually merge14:48
Kinnisonsince I was clearly being silly rebasing it and waiting for a green pipe before merging, that's the sixth rebase in a row14:49
benschubertKinnison: marge will rebase worst case :)14:49
KinnisonIf she doesn't time out :D14:49
gitlab-br-botmarge-bot123 merged MR !1500 (tpollard/messageobject->master: Message rework) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/150014:51
gitlab-br-botBenjaminSchubert approved MR !1533 (jennis/fix_artifact_log->master: Fix artifact log) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153314:51
gitlab-br-botBenjaminSchubert approved MR !1528 (danielsilverstone-ct/load-reject-duplicates->master: Reject elements which have duplicated dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152814:52
gitlab-br-bottpollard approved MR !1533 (jennis/fix_artifact_log->master: Fix artifact log) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153314:52
*** tristan has quit IRC14:59
gitlab-br-botcoldtom opened (was WIP) MR !1532 (coldtom/move-meson->master: Move meson element to bst-plugins-experimental) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153215:01
*** tristan has joined #buildstream15:01
*** tristan has quit IRC15:03
juergbiI added the commit structure to the !1499 description as overview for reviewers15:11
gitlab-br-botMR !1499: WIP: Use buildbox-casd for CAS access https://gitlab.com/BuildStream/buildstream/merge_requests/149915:11
*** tristan has joined #buildstream15:13
tme5AttributeError: 'OSTreeSource' object has no attribute 'node_validate'. have I missed something?15:15
*** tristan has joined #buildstream15:15
tme5this is just trying to run flatpak-autotools examples with ostree plugin cloned locally15:16
benschuberttme5: the ostree plugin on bst-experimetnal is currently out of date with the latest buildstream master15:17
*** tristan has quit IRC15:17
*** tristan has joined #buildstream15:17
tme5ah, does it run against current release?15:18
benschubertI tried fixing it in https://gitlab.com/BuildStream/bst-plugins-experimental/tree/bschubert/fix-plugins-yaml, but tests were not running correctly. There is some cleanup to be done there15:18
benschuberttme5: bst-plugins-external would run against the current release15:18
*** tristan has quit IRC15:18
benschubertfor master, https://buildstream.gitlab.io/buildstream/buildstream.node.html is the documentation about nodes15:19
tme5benschubert, is that bst-external? it doesn't have ostree15:19
*** tristan has joined #buildstream15:19
benschubertah, ostree is bundled with the current release of buildstream15:20
tme5but from where? experimental at the time?15:21
*** tristan has quit IRC15:21
*** tristan has joined #buildstream15:21
*** tristan has quit IRC15:23
*** tristan has joined #buildstream15:23
benschuberttme5: no, in buildstream itself15:24
*** tristan has quit IRC15:25
*** tristan has joined #buildstream15:25
tme5maybe i'm being stupid but i can't find it15:26
*** tristan has quit IRC15:26
*** tristan has joined #buildstream15:27
tpollardbtw if anyone sees any bugs to do with frontend messaging that seem to be new please do raise an issue :)15:28
tpollardas the tests can only cover so much...15:28
tpollardespecially interactive stuff15:28
*** tristan has quit IRC15:29
benschuberttme5: https://gitlab.com/BuildStream/buildstream/blob/bst-1/buildstream/plugins/sources/ostree.py15:29
tme5interactive bst init is completely broken for me :)15:29
* Kinnison remains confused as to why that was ever a thing15:29
tme5oh it's not on master, i see15:29
*** tristan has joined #buildstream15:30
benschuberttme5: yep, currently released version15:30
tme5thanks very much15:30
*** tristan has quit IRC15:31
*** tristan has joined #buildstream15:32
gitlab-br-bottpollard approved MR !1533 (jennis/fix_artifact_log->master: Fix artifact log) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153315:32
*** tristan has quit IRC15:33
tpollardtme5: please can you raise an issue for the bst init error you're seeing? I know we recently changed some of that code15:41
tme5sure15:42
gitlab-br-botmarge-bot123 closed issue #1077 (Build skipped when element has duplicate build dependencies.) on buildstream https://gitlab.com/BuildStream/buildstream/issues/107715:50
gitlab-br-botmarge-bot123 merged MR !1528 (danielsilverstone-ct/load-reject-duplicates->master: Reject elements which have duplicated dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/152815:50
tpollardtme5: ta!15:51
tpollardit is indeed borked15:54
gitlab-br-bottmewett opened issue #1100 (Error when using interactive bst init) on buildstream https://gitlab.com/BuildStream/buildstream/issues/110016:01
*** narispo has quit IRC16:04
*** narispo has joined #buildstream16:06
*** tme5 has quit IRC16:27
gitlab-br-bottpollard opened MR !1534 (tpollard/initfix->master: _frontend/app.py: Fixup node._assert_symbol_name() positional args) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153416:30
*** narispo has quit IRC16:47
*** narispo has joined #buildstream16:47
*** tristan_ has quit IRC16:52
*** tristan_ has joined #buildstream16:52
*** bochecha has quit IRC17:05
*** xjuan has joined #buildstream17:17
*** tristan has joined #buildstream17:36
*** toscalix has quit IRC17:44
*** xjuan has quit IRC17:52
*** tristan has quit IRC17:55
*** jonathanmaw has quit IRC18:16
gitlab-br-botmarge-bot123 merged MR !1532 (coldtom/move-meson->master: Move meson element to bst-plugins-experimental) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/153223:27

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!