IRC logs for #buildstream for Tuesday, 2019-02-12

*** nimish has joined #buildstream00:55
*** nimish has quit IRC03:24
*** rdale has quit IRC04:40
gitlab-br-botjuergbi merged MR !1142 (juerg/buffer-size->master: Increase read buffer size to improve performance) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/114207:08
nielsdgjuergbi: The main use case is to connect to the user session (which is provided on the dbus interface "org.freedesktop.login1"07:36
nielsdgjuergbi: The practical use case is being able to run gnome-shell in a bst shell basically07:37
juergbinielsdg: you could try to add /run/dbus/system_bus_socket to `host-files` in the project.conf's `shell` configuration07:41
juergbinot sure whether that will suffice07:41
nielsdgdoesn't seem so :(07:58
nielsdgI also found out about the SYSTEMD_OFFLINE variable, so experimenting with that as well07:59
juergbiyou could also try strace on systemctl/loginctl in bst shell to figure out what's missing08:01
juergbivalentind: do you expect to have time today to think about !1140 (and possibly also its dependency !1139)?08:01
gitlab-br-botMR !1140: WIP: Do not resolve or mangle symlinks during staging https://gitlab.com/BuildStream/buildstream/merge_requests/114008:01
gitlab-br-botMR !1139: Return all directories in list_relative_paths() https://gitlab.com/BuildStream/buildstream/merge_requests/113908:01
juergbiit's not really about full code review at this point, but rather whether you anticipate issues with the change in behavior08:02
juergbiyou mentioned a potential issue with compose at the gathering, but that might actually be avoided by !113908:03
juergbiif you have a branch that you suspect may break, I can also do a test build here, let me know08:04
*** erle- has left #buildstream08:12
*** tristan has joined #buildstream08:12
valentindjuergbi, I am fine !1140. It think it is the right thing to do. That solves quite a lot of problems. But I do not remember what was the conclusion of the discussion with had with tristan.08:53
juergbivalentind: ok, good. tristan was not in favor of it at the gathering but I'll discuss this again with him with the latest findings - given that I got fdo-sdk and gnome-build-meta working with very little changes, and the split-rule issue I didn't think of before08:57
juergbispeaking of which, hi tristan :) arrived safely back home?08:57
KinnisonHmm, WSL runner trying to run the autotools examples08:57
Kinnisonis that right?08:57
KinnisonI'd have expected that to not run on WSL08:57
juergbiKinnison: no, one of the WSL runners didn't skip them for some reason08:58
KinnisonUrgh08:58
Kinnisonso spurious failure?08:58
juergbiwe anyway switched WSL CI to manual until the runner situation is sorted out08:58
KinnisonOK08:58
* Kinnison will rebase so that comes into his branch then08:58
juergbimaybe the platform check is not robust enough08:58
Kinnisonperhaps08:58
juergbiI think it worked correctly with the newly setup runner but broke with the old (manually installed?) runner08:59
valentindjuergbi, we could always have a top integration command to recreate symlinks after staging if we needed. I think it would be better. Maybe we could actually make a plugin that does that.08:59
juergbivalentind: replacing directory with a symlink in integration command should still work (fdo-sdk does that, iirc, and I could successfully build that)09:00
juergbiI don't currently see a need for a plugin for this but maybe I'm missing something09:01
juergbiit's not like we stop supporting symlinks anymore, we just don't follow them during staging09:01
KinnisonWSalmon: print(data) worked, thanks for taking the time to speak with me09:18
*** rdale has joined #buildstream09:23
juergbijennis: I think the code changes in !1101 look good now but please fix up the branch history as per our contributing guidelines. I've mentioned a few things in the MR but it applies to all the fixups09:57
gitlab-br-botMR !1101: Refactor artifact log command https://gitlab.com/BuildStream/buildstream/merge_requests/110109:57
*** solid_black has quit IRC09:57
*** solid_black has joined #buildstream09:58
jennisthanks juergbi09:58
*** jonathanmaw has joined #buildstream10:00
*** raoul_ has joined #buildstream10:06
*** raoul_ is now known as raoul10:20
*** alatiera has joined #buildstream10:22
raoulLooks like I've had a pipeline pass everything bar wsl, is this something I should worry about? Or has it been removed for now and I just need to rerun the whole pipeline for it to pass?10:23
laurenceraoul, think it's been temporarily removed for now until we sort out the hardware for WSL runners10:23
Kinnisonraoul: if you rebase you'll find WSL temporarily set to manual10:23
raoulAh cool, I'll do that10:24
raoulcheers10:24
tpollardraoul: have you got a link to the specific failure?10:25
laurencejonathanmaw, any news on that? /me goes to chase Codethink Ops10:25
raoultpollard, seems to be a few things https://gitlab.com/BuildStream/buildstream/-/jobs/15969059710:26
raouljust examples and integration tests that are failing it seems10:26
juergbiraoul, jonathanmaw: one WSL runner doesn't skip sandbox-requiring integration tests for some reason10:27
juergbithe other runner does/did10:27
raoulah well that'll be where it's going wrong10:27
jonathanmawhow odd. I'll have a peek10:28
valentindjuergbi, found an interesting thing. If we have 2 files in cache that are the same content, one should have execution access right, the other not, then at the end they will both have execution rights. And it will be written in cache at checkout.10:28
juergbivalentind: yes, that's a limitation of the hardlinking approach :/10:29
juergbinot an issue with FUSE10:29
juergbibuildbox-fuse, that is10:29
juergbinot sure how to best handle this long term for the non-FUSE case10:29
juergbiwe could copy all executable files instead of hardlinking them, but...10:29
juergbivalentind: do you see this issue in practice or just in a test case?10:30
valentindjuergbi, I do not think of anything that would be problematic. Because at the end execution rights win.10:32
juergbiyes, we never remove them10:33
valentindThough yes. Maybe for reproducibility.10:33
valentindBecause depending on the order of the build, you might have different right.10:33
valentindAnd, let's say, you have a script element that set +x on all files and make an artifact of all the files.10:35
valentindYou can seriously mess up a local cache like that.10:35
juergbiindeed, it's definitely not a good situation10:35
valentindWhat if we duplicate in the local cache. By default we start with the just the object in 0644. If we want it in 0744, we rename it in the cache with an extension .exec. If ever want it again in 0644 and it is only the .exec file, then we copy it.10:38
*** lachlan has joined #buildstream10:38
valentindAnd we rename it only if there is no other links.10:38
juergbia simple object existence check would get more expensive10:38
juergbianother option could be to always verify the permissions when creating hardlinks. if they do not match, update them only if the link count is 110:39
valentindRight.10:39
juergbiotherwise, create a full copy of the file10:39
juergbiin most cases (same file not changing executable bit) this should not require copies10:39
valentindWe need to do it atomically however.10:40
juergbiright, that's probably not possible10:40
*** WSalmon__ has joined #buildstream10:44
*** WSalmon_ has quit IRC10:45
valentindI think the hash of objects should have included its inode metadata.10:55
*** ChanServ sets mode: +o tristan11:02
tristanjuergbi, Yes thanks :)11:02
tristanjuergbi, I'm in that limbo between a nap and sleep where you try to aim your wakeup time for a sensible hour haha11:03
juergbi:)11:07
juergbiprobably best to discuss symlinks tomorrow ;)11:08
tristanyeah indeed, I couldnt wrap my head around it now11:09
tristanI do suspect losing that symlink argument, mostly I regret that alternative solutions might force the user to carefully place output files in paths which dont have symlinked directory components11:11
tristanbut anyway, if I tried to reason about the edge cases now I won't be able to11:13
juergbiI don't expect it to be painful in practice, but I can't be sure I'm not missing a use case, of course11:15
juergbialso, actually placing output files in symlinked directories breaks strip-rules, afaict11:16
juergbibut let's discuss this tomorrow :)11:17
tristanAt staging time we share some aspects of a package based system, but we are not... at the same time, we aren't an lfs style "build and install in this environment" thing either11:18
tristanseems there is a missing link11:18
tristan(like, rpm packages can say "this is a directory I provide", but we just have files)11:19
juergbifor package based systems, I think this would be an issue as well, at least if you allow the directory symlinks to ever change11:19
juergbias then the manifest of the other package no longer matches the reality, so you can't uninstall it properly11:19
tristanWell, if a dependency provided the symlink causing an install path to a file to be valid but redirected, that symlink is reliably existing as you cannot delete the dependency without deleting the package which needs it11:21
juergbibut what if that package is upgraded?11:21
tristanof course we have an algo which explicitly ignores the subdirectories in directory walks11:21
juergbiand in the new version the symlink doesn't exist or points somewhere else?11:21
juergbithis is not an issue for buildstream itself, as we don't have the concept of uninstalling package in the core11:22
tristanThat seems to be a breaking change indeed11:22
juergbithe skipping of non-empty directories is actually an issue and also something we need to discuss tomorrow11:25
laurencelachlan, jennis, I've added a load of issues to the benchmarks repo11:53
laurencemainly focused on what we need to do wrt the Debian-like project11:54
laurenceI also and a new milestone to capture those -11:54
laurencehttps://gitlab.com/BuildStream/benchmarks/milestones/111:54
laurencepls review / amend tickets as desired11:54
* lachlan is reviewing12:00
*** solid_black has quit IRC12:10
gitlab-br-botjjardon opened (was WIP) MR !1137 (jjardon/fedora_29->master: .gitlab-ci.yml: Test with current fedora release: 29) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113712:10
gitlab-br-botcs-shadow closed issue #890 (RFE: Allow `bst show` to print list of dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/issues/89012:19
gitlab-br-botcs-shadow merged MR !1121 (chandan/deps->master: _frontend: Allow printing dependencies using `bst show`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/112112:20
gitlab-br-botjuergbi approved MR !1137 (jjardon/fedora_29->master: .gitlab-ci.yml: Test with current fedora release: 29) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113712:22
gitlab-br-botjuergbi approved MR !1144 (valentindavid/pull-chmod-bug->master: buildstream/_cas/cascache.py: Set 0644 rights to pulled files) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/114412:22
jonathanmawraoul, juergbi: hmm, I can't reproduce the WSL runner not skipping that test when running tests manually12:26
juergbijonathanmaw: on the DESKTOP-7154KN6 runner?12:27
jonathanmawyep12:27
juergbiinteresting, no idea why this could happen sporadically12:28
juergbiI saw it here https://gitlab.com/BuildStream/buildstream/-/jobs/15955082012:28
*** raoul has quit IRC12:28
juergbijonathanmaw: oh, examples still use IS_LINUX/HAVE_BWRAP, not HAVE_SANDBOX12:29
juergbimaybe bwrap is installed on a WSL runner12:29
jonathanmawah, yes apparently12:30
juergbinot sure why your manual retest didn't trigger this issue, though12:30
* jonathanmaw uninstalls that12:30
juergbibuild-uid.py also has a couple of missing HAVE_SANDBOX checks12:30
juergbiand cachedfail.py and sandbox-bwrap.py12:31
juergbi(they might need HAVE_SANDBOX as additional constraint instead of as the only constraint, as they may be bwrap-specific)12:31
gitlab-br-botjjardon opened issue #906 (The logic to calculate the disk quota seems to be wrong) on buildstream https://gitlab.com/BuildStream/buildstream/issues/90612:37
jjardonpipelines are taking ~140 minutes to finish; is this the expected time?12:39
gitlab-br-botjuergbi approved MR !1101 (jennis/refactor_artifact_log->master: Refactor artifact log command) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/110112:40
juergbi140 minutes sounds very long, especially without WSL12:42
tpollardI've had a single pipeline take over 2 hours too12:43
juergbiwithout WSL?12:43
tpollardyep, one of the fedora ones12:43
juergbithe fedora update-deps one sometimes takes quite some time12:44
tpollardyep, for instance https://gitlab.com/BuildStream/buildstream/-/jobs/15842964312:45
KinnisonI have noticed it starts counting from when it tries to allocate a runner12:46
juergbiyes, timestamps in the log would be useful12:46
Kinnisonso sometimes it sits there counting when it isn't actually doing anything yet12:46
gitlab-br-botjjardon opened issue #907 (Pipelines are taking a lot of time to finish (~140 min)) on buildstream https://gitlab.com/BuildStream/buildstream/issues/90712:47
jjardonyeah, It's a feature requested a lot: https://gitlab.com/gitlab-org/gitlab-runner/issues/241212:48
juergbiin tpollard's example, actual tox tests took 96 min, out of 136 min total runtime12:50
juergbistill longer than normal12:50
jjardonwhat would be the normal? It used to be much less but I think more tests were added since then12:51
jjardonthe slowest test took ~5min, not sure that was faster before?12:51
*** raoul has joined #buildstream12:54
juergbijjardon: we used to share the local artifact cache directory across integration tests and we no longer do this for better isolation and to allow parallelism13:02
juergbinot sure how big the impact of that was13:02
raoulIs there a sensible way to instantiate sources for tests? I've been trying to do this in order test source cache methods I'm working on but seem to end up in a mess of instantiating various required objects13:06
juergbiraoul: can't you cover this indirectly via CLI-based tests?13:06
juergbithat's currently the preferred style13:07
raoulYeah, was just hoping to test the API before integrating it fully into the rest of buildstream13:07
juergbiah ok, don't know out of the top of my head how much would be involved for this13:08
raoulYeah seems to be a lot of faff, I've ended up trying to track down bits that aren't set up. Guess I'll just go for the full cli tests.13:10
*** lachlan has quit IRC13:12
*** lachlan has joined #buildstream13:21
gitlab-br-botjjardon merged MR !1137 (jjardon/fedora_29->master: .gitlab-ci.yml: Test with current fedora release: 29) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113713:31
*** nimish has joined #buildstream13:37
gitlab-br-botdanielsilverstone-ct opened (was WIP) MR !1131 (danielsilverstone-ct/further-optimisations->master: Further optimisations) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113113:41
*** aevri has joined #buildstream14:11
*** lachlan has quit IRC14:41
*** lachlan has joined #buildstream14:53
gitlab-br-botcs-shadow opened (was WIP) MR !998 (chandan/junction-dependency-format->master: loader: Allow dependencies to use ":" to refer to junctioned elements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/99815:01
gitlab-br-bottpollard opened issue #908 (Create Artifact abstraction class) on buildstream https://gitlab.com/BuildStream/buildstream/issues/90815:07
*** lachlan has quit IRC15:19
*** WSalmon__ has quit IRC15:20
*** lachlan has joined #buildstream15:23
*** mohan43u has quit IRC15:28
*** mohan43u has joined #buildstream15:33
*** lachlan has quit IRC15:34
gitlab-br-botLaurenceUrhegyi opened issue #909 (Artifact As A Proto) on buildstream https://gitlab.com/BuildStream/buildstream/issues/90915:46
laurenceraoul, tpollard, Kinnison, juergbi, anyone else who's interested in AaaP - please review the above ticket, trying to capture all the tasks15:46
laurencefeel free to amend as required15:46
Kinnison"the above ticket" ?15:48
laurenceyeah, the gitlab bot posted it above my comment15:48
laurence<gitlab-br-bot> LaurenceUrhegyi opened issue #909 (Artifact As A Proto) on buildstream https://gitlab.com/BuildStream/buildstream/issues/90915:48
Kinnisonaah I have the bot on ignore so that I don't ragequit the channel15:50
laurenceragequit, hahah15:54
*** lachlan has joined #buildstream15:55
*** lachlan has quit IRC16:03
*** lachlan has joined #buildstream16:11
laurencehhhhhmm, wonder if anyone can respond to Emerson on this ticket? https://gitlab.com/BuildStream/buildstream/issues/90016:13
laurencecryptographic authentication...16:14
gitlab-br-botBenjaminSchubert merged MR !1131 (danielsilverstone-ct/further-optimisations->master: Further optimisations) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113116:14
*** lachlan has quit IRC16:20
*** lachlan has joined #buildstream16:31
*** lantw44 has quit IRC16:50
*** lantw44 has joined #buildstream17:03
*** tristan has quit IRC17:20
tlater[m]<_gimpnet_lau "hhhhhmm, wonder if anyone can re"> laurence: That looks like a tough one17:33
* tlater[m] is curious where it will go17:33
laurenceindeed, tlater[m]17:34
gitlab-br-botvalentindavid opened MR !1146 (valentindavid/local-cache-exec-leak->master: Deduplicate files in local cache with or without exec rights) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/114617:47
valentindjuergbi, I have made !1146 to solve the issue with exec rights. Was there a ticket for that?17:48
juergbiI'm not aware of one17:48
valentindOK, I will create one then.17:48
juergbihm, it doesn't make me very happy17:52
valentindjuergbi, the issue or the fix?17:52
gitlab-br-botvalentindavid opened issue #910 (Execution rights on an object can corrupt local cache) on buildstream https://gitlab.com/BuildStream/buildstream/issues/91017:55
juergbithe fix. it makes the code less simple/elegant, but maybe there is no better way17:55
valentindjuergbi, yes. For sure. I think the idea of having executable rights stored on the directory was not really the right way to go.17:57
valentindAlso I wonder, what if we want to simulate links between files when using fuse? I think we would have needed inodes in the cache. Maybe there should be some refactoring to do in the protocol.17:59
gitlab-br-botjonathanmaw closed issue #895 (Buildstream needlessly stages junctions into  tmpdir) on buildstream https://gitlab.com/BuildStream/buildstream/issues/89517:59
gitlab-br-botjonathanmaw merged MR !1134 (jonathan/junction-no-tmpdir->master: Stage junctions into .bst instead of a tmpdir) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/113417:59
juergbiyou mean hardlinks? yes, we have a ticket open about this. this would require protocol extension18:00
valentindyes.18:00
valentindAnd will we store the access rights along with the file in this extension?18:01
juergbione aspect that bugs me is that it's not necessary on modern systems (when we use buildbox-fuse)18:01
juergbi(the exec thing, not hardlinks)18:01
juergbiand I generally prefer not to work around system limitations all over the code base18:02
juergbihardlinks: there is no design/proposal yet for this extension18:02
valentindjuergbi, I would be fine removing the local cache completely and always go through buildbox-fuse. But if we keep it, we should fix it.18:03
juergbiwe have to support systems that don't support FUSE, unfortunately18:03
*** jonathanmaw has quit IRC18:04
valentindjuergbi, I think we can simplify the code if we go for a different protocol for storing cache. If every file can have some metadata, then we can make simpler code.18:05
juergbipossibly, but I don't see such a fundamental protocol change happening upstream anytime soon18:06
juergbiand we want to be compatible, as far as possible18:06
valentindjuergbi, I think it is only for storing locally that we would need the extension. We could still support an older remote cache.18:08
juergbiyou want to rewrite things for every transfer? sounds potentially expensive18:09
valentindMaybe.18:09
juergbivalentind: regarding the MR, couldn't we reduce the impact by having our own Digest class that includes the proto Digest + the executable bit?18:10
juergbiso we don't need separate executable lists and sets etc.18:10
valentindjuergbi, sure.18:10
valentindBut we still need to populate that bit when pulling.18:11
juergbiyes, sure, but I would expect this to be much cleaner18:12
juergbibut maybe I'm missing something18:12
valentindI think we send the digest object to the server.18:12
valentindSo that would be a change in protocol.18:12
valentindCan we add a field in the message without breaking backward compatibility?18:12
juergbiwe would only send the Digest proto part, of course18:12
valentindAnd would the server answer back with the right bit?18:13
juergbiI wasn't proposing changing the protocol18:13
valentindOK, I have to look at that again. Because I thought that was not possible.18:13
valentindSo wrap the _CASBatchRead.18:14
valentindI see. I will try that.18:15
juergbiah, right, that's maybe not straight forward18:15
juergbias we don't currently track the requested digests18:16
juergbiso maybe this just shifts things around but doesn't simplify it18:17
juergbihave to think about it some more18:17
valentindjuergbi, I still simplifies the code to hide it in CASBatchRead.18:19
valentindCASBatchRead is relatively simple.18:19
valentindjuergbi, like that: https://gitlab.com/BuildStream/buildstream/commit/da80724e93acf662fb1b58b9f34918b91911437a18:27
valentindIt saves 13 lines of patch.18:30
valentindNot that big difference.18:30
*** raoul has quit IRC18:35
juergbiok, but we could still have this digest helper class to avoid requiring is_exec parameters everywhere18:38
valentindoh, I might have not understood all.18:40
valentindYes maybe I can do better.18:40
*** lachlan has quit IRC18:41
valentindThis is interesting. If we have multiple times the same file in a directory, we will make a batch request with multiple times the digest, and get multiple times the content.18:45
juergbideduplication in CASBatchRead.add() could make sense18:54
gitlab-br-botvalentindavid merged MR !1144 (valentindavid/pull-chmod-bug->master: buildstream/_cas/cascache.py: Set 0644 rights to pulled files) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/114419:13
*** adds68 has quit IRC20:13
*** ikerperez has quit IRC20:13
*** jennis has quit IRC20:13
*** mablanch has quit IRC20:17
*** mablanch has joined #buildstream20:18

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!