IRC logs for #buildstream for Tuesday, 2017-07-11

*** jude has joined #buildstream07:12
*** jonathanmaw has joined #buildstream08:24
*** tlater has joined #buildstream08:41
*** ssam2 has joined #buildstream09:06
*** tristan has joined #buildstream09:22
*** ChanServ sets mode: +o tristan09:22
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 1 commit (last: Artifacts documentation: Fixed a little typo) https://gitlab.com/BuildStream/buildstream/commit/917a1b9f54a2ec2342b91bdca4fb10f66bbfa6cf09:40
juergbitristan: i currently don't encode the key strength into the ostree ref itself, i.e., both still follow the format project/element/key09:41
juergbifor elements without dependencies, the weak and the strong key and therefore the ref will actually match09:41
juergbii don't see an issue with this and would keep it this way. unless you can think of an issue09:41
tristanjuergbi, that sounds fine to me09:42
juergbiok, ta09:42
tristanI think it would be nice to know, when building in weak planning mode... whether the strong key is the same as it would have been in strong planning mode09:42
tristanbut it's mostly a speculative desire to know that09:43
tristanseems interesting :)09:43
tristanas it can accidentally be the same in many cases, and can be interesting to know09:43
tristantlater, juergbi... since you guys are not on anything specific project-data wise, I would like for both of you to be testing with https://gnome7.codethink.co.uk/gnome-modulesets.git09:46
tristaninstead of some branch of buildstream-tests, if you're not using that already09:46
tristantlater, also, you could send me your public ssh key and become an authorized pusher to the GNOME artifact cache09:47
tlaterOk09:48
tristanfwiw: The master branch is an untracked conversion of GNOME modulesets which happens every 10 minutes09:48
tristanThe "tracked" has commits with all tracked sources, and is updated... every 2 hours I think09:49
tristanwith the "tracked" branch one can refer to a commit sha and other people should be able to reproduce exactly the same output09:49
juergbiok, will switch to that09:49
* tristan currently has 1cb9e7b3b0a1dda9414bc7fed694ec7b4433fc0c09:49
tristanhaha, oh crap ;-)09:51
tristanOk /me fixes the authorized keys09:51
tristaneasy to break09:51
tlatertristan: None of the simple gnome-moduleset elements seem to have archs :/09:53
juergbiregarding key display, yes, i agree. trying to wrap my head around how this should work given that there quite a few key variants in place now and not all may be known at the beginning of the session09:53
tristantlater, there is only the base system, which chooses the base runtime to build on09:54
tristantlater, you need arch conditionals ?09:55
tlaterWell, I'd like to make sure this branch doesn't break stuff somehow.09:55
tristantlater, they will probably come, but hopefully they should remain unneeded as much as possible09:55
juergbi        tar source at core-deps/mozjs38.bst [line 3 column 2]: Error mirroring https://people.mozilla.org/~sstangl/mozjs-38.2.1.rc0.tar.bz2: <urlopen error [Errno 111] Connection refused>09:55
tristantlater, oh *that* ? you mean how the arguments are parsed ?09:55
tristantlater, dont worry about that09:55
tristaneh, that can happen09:56
juergbialso got this, though09:56
juergbi        ostree source at base/base-system.bst [line 5 column 6]: Failed to fetch ref '9136b273bebba1316268fe943fbbc39b13fe104bd2a1b2a387117e9e2a83b0da' from origin: https://gnome7.codethink.co.uk/repo09:56
juergbi        Failed to fetch ref '9136b273bebba1316268fe943fbbc39b13fe104bd2a1b2a387117e9e2a83b0da' from 'origin': Socket I/O timed out09:56
tristansigh, so we're not out of the woods !09:57
gitlab-br-botpush on buildstream@arch_options (by Tristan Maat): 1 commit (last: main.py: Move --arch* options to the main command) https://gitlab.com/BuildStream/buildstream/commit/45d3be971aac9ad138a8c1438a6b7b0e70d5945109:57
tristanI haven't fetched sources in a while; I regularly nuke my artifact cache but not the sources; cause they take a long time09:57
juergbimakes sense09:57
tristanBut downloading from the remote used to work quite stably09:57
tristanI think originally I was downloading alot from sdk.gnome.org09:58
tristanpossibly gnome7 is under duress09:58
tristanFor artifact cache config:09:59
tristanartifacts:09:59
tristan  pull-url: https://gnome7.codethink.co.uk/artifacts09:59
tristan  push-url: artifacts@gnome7.codethink.co.uk:artifacts09:59
tristantlater, and in your ~/.ssh/config... you need:09:59
juergbinot compatible with the artifacts (metadata) required by my branch09:59
tristanHost gnome7.codethink.co.uk09:59
tristan        Port 1000709:59
tristanHmmm10:00
tlaterThanks :)10:00
juergbithat reminds me, we really need the cache key versioning10:00
tristanjuergbi, ok so... we need artifact revisioning for this kind of thing10:00
juergbi:)10:00
tristanjuergbi, yeah, I wrote an email to the list with thoughts on all kinds of revisions, not long ago10:00
juergbiyes, i saw that but haven't really looked into the details yet10:01
tristanyeah it's a bit complicated because plugins10:01
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5210:01
gitlab-br-botpush on buildstream@arch_options (by Tristan Maat): 3 commits (last: utils.py: Fix _tempdir for python3.4) https://gitlab.com/BuildStream/buildstream/commit/eecec59a93fba04fd9b84e474cf4237aea35ff1610:02
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5210:02
gitlab-br-botpush on buildstream@arch_options (by Tristan Maat): 1 commit (last: main.py: Move --arch* options to the main command) https://gitlab.com/BuildStream/buildstream/commit/580b2f331aa1db716f72b7aa57ef463fdc1db58810:04
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5210:04
tlaterCan I add a test that ensures permissions are 644? x)10:05
tlaterActually, pre-commit hooks can probably do that10:05
tristantlater, there are some things we want executable10:06
tristantlater, but I think they are quite limited, one of them is setup.py10:06
tristanalso doc/sphinx-build3 needs exec10:06
tristanbut looks like thats part of the weird docs building hack that would be nice to remove, we shouldnt be touching python 2 interpretors at all10:07
tlaterI'll just hack a script locally for now ;)10:07
tlaterLooks like this passes fine: https://gitlab.com/BuildStream/buildstream/merge_requests/52/pipelines10:28
tristantlater, did you get my comment ?10:29
tristanTo be honest, I also overlooked some of the wording of the arches stuff, but that was there since ssam2 put it anyway10:29
tlaterAh, alright10:30
tristan"Architecture of the machine running the build" for --arch is just misleading10:30
tristanMaybe we should take this opportunity to correct these strings10:31
ssam2oops, yes I meant to change that10:31
tlaterHm, how woudl you word that?10:31
tristanOk so... for --target-arch "Produce elements that execute on this architecture"10:31
tristanI think it can be more readable with "Machine architecture for build output"10:32
tristanThat way we can take --host-arch and say "Machine architecture for the sandbox", or "Machine architecture for build input"10:33
tristanbecause "Run as a native build for the given architecture" isnt really meaningful... to me at least10:33
tlaterI like the sandbox one, it explains what "build input" is10:33
ssam2yes that sounds better10:34
tristanThen --arch "Architecture of the machine running the build" can just be "Machine architecture", leave it ambiguous, it's anyway clarified by the other two which say "(defaults to --arch)"10:34
tristanOk agreed for the sandbox thing10:35
tristanOk I'm comfortable with those strings, unless there are objections in the next few seconds... tlater just roll with those10:36
tristanI'm still not _entirely_ clear about what this is going to mean for running full pipelines with virtualization capable sandboxes10:37
tristanbut it makes sense for what we do with it now anyway10:38
tristanand we'll just have to learn from experimentation when we get there10:38
* tristan is tempted to add a push-port to the artifacts configuration10:39
* tristan doesnt like sneaking into users' .ssh/config files10:39
tlaterMy linter shows an error on line 844 of main.py - There's a format string with too many arguments there. Not sure what the string is supposed to be, but that should probably be fixed?10:41
tristantlater, good call10:46
tristanit's on a different line here though10:46
tlaterProbably additions by me10:47
tristantlater, anyway just keep self.file_count10:48
tristanon that line10:48
gitlab-br-botpush on buildstream@arch_options (by Tristan Maat): 3 commits (last: main.py: Move --arch* options to the main command) https://gitlab.com/BuildStream/buildstream/commit/9e45372c687afd0dc378c36742dfe782912eb48a10:55
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5210:55
gitlab-br-botpush on buildstream@arch_options (by Tristan Maat): 3 commits (last: main.py: Move --arch* options to the main command) https://gitlab.com/BuildStream/buildstream/commit/86578b928f29697b26ab025278f7104d590974ff10:59
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5210:59
tristanoh weird, did I mess something up with artifacts ?11:02
tristansuddenly I cant push ?11:02
* tristan fixes11:03
gitlab-br-botpush on buildstream@workspaces (by Tristan Maat): 9 commits (last: utils.py: Fix _tempdir for python3.4) https://gitlab.com/BuildStream/buildstream/commit/eecec59a93fba04fd9b84e474cf4237aea35ff1611:08
gitlab-br-botbuildstream: merge request (workspaces->master: Workspaces) #50 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5011:08
tlatertristan: For the `workspace list` command, should there be a target? It would make sense just to display all workspaces in the project.11:16
tristantlater, agreed, no target11:19
gitlab-br-botbuildstream: merge request (arch_options->master: main.py: Move --arch* options to the main command) #52 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/5211:23
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 3 commits (last: main.py: Move --arch* options to the main command) https://gitlab.com/BuildStream/buildstream/commit/86578b928f29697b26ab025278f7104d590974ff11:23
tristanfreaky11:27
tristanSo now that I had 'ForceCommand bst-artifacts-receive' in the /etc/sshd_config... it wont take any arguments I give it11:28
tristanlike... ForceCommand has to specify more than just the program I'm gonna run, but also the arguments ?11:28
tristanThat strict ?11:28
tristanAlso...11:28
tristanNow that I am commenting it out... it's *still* not working11:29
tristanAfter systemctl restart sshd... it retains the same behavior11:29
tristanSo that means... yesterday when I added "ForceCommand bst-artifacts-receive" and restarted sshd...11:29
tristanI actually broke it11:29
tristanbut it kept working for a while, because it looks like... some unknown cosmic dust needs to be sprinkled on the sshd server11:30
tristanwhich happens, at some unknown time11:30
tristanjuergbi, you seem to be the one who has any clue about sshd, any idea what's going on ?11:31
juergbitristan: do you use a control master on the client and might this still be running with the old sshd?11:32
tristanAlso, is there a way to do "ForceCommand bst-artifact-receive *" or something ?11:32
tristanAhhh11:32
tristanThat can be the problem yes11:32
tristanits on my side11:32
juergbitristan: ForceCommand, maybe try this: http://at.magma-soft.at/sw/blog/posts/The_Only_Way_For_SSH_Forced_Commands/11:33
juergbi(wrapper script that feeds back the arguments)11:34
tristanOh weird11:34
tristanForceCommand actually takes whatever command I send to ssh11:34
tristanand runs that command *instead*11:34
tristanit doesnt say "Only allowed to run this command you idiot"11:35
juergbiyep11:35
tristan:-/11:35
juergbiyou get the SSH_ORIGINAL_COMMAND env variable instead11:35
tristanSo it would be nice to have some flexibility, but I guess that's not a feature of ssh11:35
juergbiso a wrapper script can do whatever you want11:35
juergbibut it doesn't appear to be built-in11:35
tristanokay11:35
tristanSo I'll amend the docs for this11:36
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 3 commits (last: _artifactcache/pushreceive.py: Changing up the bst-artifact-receive options) https://gitlab.com/BuildStream/buildstream/commit/cd781089d2be66c79837f59355b0710f7bb09b5312:00
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 1 commit (last: Update man pages for new placement of arch parameters) https://gitlab.com/BuildStream/buildstream/commit/fc7eb1d5cbaa448a6c6df9da78fe582a665235fd12:02
tristanjuergbi, ok so... I think you are changing the UI a bit for the weak keys and suchlike... I just wanted to say; that I'm concerned for the regular log lines; I think the log lines should continue to only display either the strong or weak abbreviated key (depending on build mode)12:05
tristanbecause those lines... I dont want them to be too long12:06
tristanjuergbi, I hope that we can restrict this to LogLine.show_pipeline()12:06
juergbiyes, for build logs i'm planning to use the abbreviated strong key of the actual build (i.e., the same that gets stored in metadata, may be different from the strong key in strong build planning mode)12:07
tristanAh well that's alright too12:07
juergbiunfortunately, we may not know that key yet in earlier stages12:07
juergbiso we'd either first have to display/use the weak key or no key at all12:08
tristanYes, currently with tracking queues enabled they default to ????????12:08
tristantrue12:08
juergbiyes, one option is to do the same before we're sure about the key we actually use12:08
juergbialternative would be to first display the weak key but this may also be confusing12:09
juergbi(i.e., displayed key changing in the session)12:09
tristanRight, well I think that if you are going to show strong key all the time it probably wont change the code that much ?12:09
tristanI.e. in weak planning mode; strong keys get resolved based on the dependencies which will be built against, correct ?12:09
tristanRight I'd rather stay with, if keys change, they change from ???????? -> real key12:10
juergbione source of possible confusion that we essentially have two strong keys. one is the ideal strong key, i.e., as per strong build planning mode. the other is what you just described12:10
tristantrue12:10
tristanheh12:10
juergbias buildstream will still attempt to pull the artifact using the ideal strong key even in weak build planning mode12:11
juergbiat first, that is. it will fall back to weak key12:11
tristanThat makes sense12:11
juergbiwe actually have a third strong key12:11
juergbithe one embedded in the artifact that was downloaded using the weak key12:11
tristanRight12:12
tristanWhoa, mind boggling12:12
tristanSo I guess the ultimate strong key is whatever it was built against, in weak mode12:12
juergbiyes12:12
tristanAnd I guess that we want to make an attempt to download strong ideal artifacts even when we have weak artifacts present12:12
juergbiyes, that's how i implemented it12:13
tristanSo I guess we want to resolve as late as possible, just like with tracking enabled12:13
juergbithat ultimate key, yes12:13
tristanjuergbi, ok well; I am going to add something of a session summary when a pipeline completes12:14
tristanSo it *could* be augmented to repeat the LogLine.show_pipeline() again at the end12:14
tristanMy plan was just to clean things up and show what was processed/skipped/failed for each queue12:14
juergbito show what has the ideal strong key in the end?12:14
tristanand have only one path for that, instead of those custom messages in the Pipeline() apis12:15
tristanRight, it *could* be useful; show what was the ultimate strong key produced for each element all in one place12:15
juergbiyes, could be interesting. we can also look into this after merging the actual weak build/key functionality12:16
juergbilet's first get it into mergeable shape12:16
tristanI'm thinking that... first of all; the formatting options (see Context.log_element_format), will need weak keys12:16
tristanthat is the same formatting options to `bst show`12:16
tristanand we can start by adding weak abbreviated keys beside strong ones by default12:17
juergbicould make sense, yes12:17
tristanso that there is always the two keys12:17
juergbithe two keys will never match, though, not even in strong build planning mode (except for elements without dependencies)12:17
tristanRight12:17
juergbimaybe it would make more sense to display two strong keys12:17
juergbithe ieal one and the one actually built/downloaded12:18
tristanHmmm12:18
tristanYeah all three seem to be interesting12:18
tristanTo be honest though, I feel like I'm more interested in this information:12:18
juergbior just some flag whether the two strong keys are identical12:18
tristan  o Strong key of created/used artifacts12:18
tristan  o Weak key of created/used artifacts12:18
tristan  o Whether the strong key was the ideal one or not12:19
tristanI.e. I'm not so interested in what the ideal key would have been right ?12:19
juergbiright12:19
tristanOr might I be ?12:19
tristanyeah, I feel initially I'm more interested in just whether they were identical or not12:19
juergbinot really sure about the use cases for the displayed keys12:20
juergbithe last point is definitely something interesting12:20
juergbiif you have the artifact cached, you can theoretically easily gof rom strong key to weak key12:20
juergbii.e., we could also just display strong key and make it easy to get to the weak key via bst show12:20
tristanIn log lines, we could portray that information with a color, like we could dim the keys which are non-ideal or such12:20
juergbiyes, could make sense12:21
juergbimaybe also some non-color indication when logging (non-colored) to file12:21
juergbilike a check mark next to the key12:22
tristanI'm thinking for the summaries at least we should do that12:22
tristanLike, if you have a whole log file, you can easily derive that from looking at the summary12:22
juergbiagreed12:22
juergbi12:23
tristanhmmm, if utf8 is not an issue, maybe using an extra char for a checkmark is good12:24
* tristan thinks there is still no exotic characters in the output so far12:25
juergbii don't expect any issues these days for UTF-8 output on linux12:27
tristannah it should be fine to have utf8 logs, in any case I think they technically already are12:38
tristani.e. we cannot predict what is in the commands shown12:38
tristanOr in the case of an error, what is captured and displayed from the stdout of the process12:38
jonathanmawtristan: After some fiddling with safe_copy, it seems the standard way to handle errors in utils.py (raising an exception) always leads to stack traces.12:48
jonathanmawI can give a more informative error message (which would have helped when I was having the problem initially)12:49
jonathanmawbut I'm not sure what to do if I want to get rid of the stack traces12:49
jonathanmawIs it a matter of going to all the things that call them and raising those exceptions as an appropriate type (ElementError, SourceError, etc)?12:50
tristanjonathanmaw, yes.13:13
jonathanmawtristan: ok, I'll have a look where it's being called.13:14
tristanjonathanmaw, in this case it's a special case though, cause it's the frontend that does it13:16
tristanjonathanmaw, so you will want to except OSError as e, and raise PipelineError(...) from e13:16
tristanin _pipeline.py13:16
*** xjuan has joined #buildstream13:48
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 6 commits (last: _artifactcache: Few changes) https://gitlab.com/BuildStream/buildstream/commit/aad3ff24cfecf3745abf5f69f53a9ee757024bf413:52
gitlab-br-botbuildstream: merge request (jonathan/36-fix-ugly-stack-trace->master: Fix ugly stack trace) #53 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5313:57
tlatertristan: Is there a way to get access to *all* elements in the project yet?13:57
gitlab-br-botbuildstream: merge request (jonathan/36-fix-ugly-stack-trace->master: Fix ugly stack trace) #53 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/5314:06
tlatertristan: For checking workspaces for elements that don't exist I'll have to load the entire set of elements - otherwise there's no way to tell if something has been removed14:14
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 1 commit (last: artifacts documentation: Now there is 'push-port', amend artifact docs for this) https://gitlab.com/BuildStream/buildstream/commit/5106173c3dbfed75831baaaab658b0ed9500937f14:27
tristantlater, no there is no way to do that14:28
tristantlater, but I thought it would make sense at least to issue a warning if there are open workspaces that are irrelevant to the current pipeline14:29
tristanmaybe I have to rethink that14:29
tristantlater, I guess you have a point that the user may use many different pipelines or be running subsets of their project14:30
tristanstill it would be nice to have a warning in place I think, perhaps without user prompting though14:30
tlaterThis is probably going to issue a warning most of the time - perhaps it's better to point out workspaces with missing elements in `workspace list`14:31
tristantlater, also I think we should be mentioning the active workspaces relevant to the pipeline when printing the heading ahead of running the pipeline14:31
tristanprobably that should just be wired in by default into `bst show`14:32
tristanso it's automatically a part of the default formatting (Context.log_element_format)14:33
* tristan has to leave closing coffee shop for now14:33
tlatero/14:34
tristantlater, if it's going to warn most of the time that may not be too much of a problem though, if its a brief warning listing the unused workspaces, and shows up as a WARNING message, after showing the summary and at the beginning of the scheduling14:34
tristanAlthough I honestly doubt that it would show up most of the time14:35
tristanif people opened a workspace, it stands to reason that it's going to be active for the majority of pipelines they run for that project14:35
tlaterYeah, that's fair14:35
tristancould be just a MessageType.WARN with "Unused workspaces" as the message, and a \n separated list of workspaces in the message detail14:36
tristannice and clean14:36
* tristan out14:37
*** tristan has quit IRC14:40
*** jonathanmaw has left #buildstream15:43
tlaterThe element-format value in the default userconfig doesn't seem to have an effect in `bst show` - though this may just be me not using it correctly. I think the --format option of `bst show` accidentally overrides it with a new default.17:02
tlater^ I'll just leave that here because I'm not confident enough to leave an issue, and don't know enough about output to simply fix it.17:05
*** tlater has quit IRC17:09
*** jude has quit IRC17:26
*** tristan has joined #buildstream17:54
*** ChanServ sets mode: +o tristan18:07
*** jude has joined #buildstream18:09
*** ssam2 has quit IRC18:52
*** jude has quit IRC19:25
*** jude has joined #buildstream19:36
*** jude has quit IRC21:30
*** xjuan has quit IRC21:43
jjardon[m]hi, I get a 404 error from https://buildstream.gitlab.io/ ; is this intended?23:39

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!