IRC logs for #buildstream for Friday, 2018-05-04

*** Prince781 has joined #buildstream01:18
*** Prince781 has quit IRC02:17
*** mhm527 has joined #buildstream03:30
*** mhm527 has quit IRC03:31
*** Prince781 has joined #buildstream03:40
*** Prince781 has quit IRC04:44
*** Prince781 has joined #buildstream05:06
*** Prince781 has quit IRC05:11
gitlab-br-botbuildstream: merge request (valentindavid/385_vte_notification->master: buildstream/_frontend/app.py: Notify VTE on failure in interactive mode.) #447 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/44705:16
*** Prince781 has joined #buildstream05:17
*** tristan has joined #buildstream05:24
gitlab-br-botbuildstream: issue #383 ("AssertionError when caching artifact") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/38306:28
gitlab-br-botbuildstream: merge request (juerg/non-strict-buildable->master: element.py: Fix buildable check in non-strict mode) #448 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/44806:28
*** Prince781 has quit IRC06:32
*** toscalix has joined #buildstream06:58
gitlab-br-botbuildstream: merge request (juerg/no-remote-summaries->master: Drop use of OSTree summary files and dynamically plan build dependencies) #446 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44608:31
*** aday has joined #buildstream08:35
toscalixvalentind: based on tristan mail, I assigned https://gitlab.com/BuildStream/buildstream/issues/359 to you08:40
valentindtoscalix, ok08:40
laurencecs_shadow, hello :)08:47
laurenceare you planning to go to GUADEC?08:47
*** jonathanmaw has joined #buildstream08:48
*** dominic has joined #buildstream08:51
tristanjuergbi, this rebase is going to be a real pain :-S08:53
jmacthanks juergbi08:53
tristanjuergbi, now we actually hand the pipeline to the elements and let the elements themselves do things... I dont like how loopy this is08:54
juergbitristan: that's why I mentioned it a few days ago08:54
tristanyeah08:54
*** aday has quit IRC08:54
juergbiyes, I was going forth and back on this one08:54
tristanI wonder how this should ultimately look08:54
tristanIs there an alternative without that ?08:54
juergbinot extremely happy about it either, but each element can only be in a single pipeline, it doesn't seem unreasonable08:54
juergbione potential alternative would be something like a callback to keep it decoupled08:55
tristan"a pipeline"08:55
tristanIs there now more than one pipeline too ?08:55
juergbino, there isn't. it's just hypothetical08:55
tristanSo, I was thinking that one way forward was to have something additional, like the QueueMaster or something, sitting in between... that might mesh well with what tlater is doing08:56
tristanBut08:56
tristanIt looks like this needs knowledge of the planner08:56
*** bethw has joined #buildstream08:56
tristanIf the segmented lists of things which "we only might build" were to be sorted and planned first, and only optionally built, then at least we wouldnt need that part08:57
tristanI'm trying very hard here to think of an approach which doesnt involve mangling... where Elements dont know anything about their owners08:58
tristanand we have a nice top-down ownership model08:58
*** aday has joined #buildstream08:59
juergbihm, not sure how that would work with planning everything but still keeping some parts optional08:59
cs_shadowlaurence: hey! Not sure yet. I’ll have to check with others at office about the plan09:00
juergbieach on-demand build dependency can have further dependencies, so in one way or another you have to do dynamic planning09:00
cs_shadowLet me get back to you on that09:00
tristanI thought that if we had scheduled the build-of-build dependencies *after* the reverse dependencies which pull them in, and then have them in WAIT state until their reverse dependencies pass through the PullQueue09:00
laurencecs_shadow, cool. would be good to see you all there :)09:01
tristanjuergbi, We could then decide whether to put those elements in READY or SKIP state09:01
tristanjuergbi, but I recall you said something about this approach being overly complex09:01
juergbitristan: how to organize this with indirect dependencies, e.g., runtime dependencies of these build dependencies?09:01
juergbiwhich could have multiple indirect reverse dependencies09:01
juergbiI don't see a straight forward way right now but there might be something simple that still makes this possible09:02
tristanWell, the knee-jerk response to that question is to filter those build-of-build branches properly, in fact; the --except logic does pretty much *exactly* this algorithm09:03
tristanjuergbi, i.e. excepting build-of-build from a specific element, does not filter out what is orthogonally depended on09:03
tristanthe result of that is the portion of the branch which needs to be queued after the dependency09:04
tristanbut...09:04
tristanthis is only a knee-jerk response to the question09:04
tristannot completely thought out I suppose09:04
juergbitristan: not sure I understand. my question was more how to dynamically queue up such indirect dependencies, not how to filter them out at the beginning09:05
juergbior maybe top-down that would actually be pretty easy09:06
tristanThey would be queued as segments *after* the depending element instead of before them09:06
juergbiso we would add everything to the pipeline/queues but keep some elements in an inactive/waiting state until we actually need them09:07
tristanat the *basic* level, we would decide to skip them if we never needed to pull/build them, instead of adding them to the lists dynamically on demand; that is the basic concept09:07
tristanexactly09:07
*** bochecha_ has joined #buildstream09:08
tristanit would introduce more iterations in the loop which searches for the next element that is ready to process09:08
tristanbut that seems like a negligible overhead09:08
juergbiand as the 'active' state would be an Element property, an Element could activate its dependency without going through a Pipeline object09:09
tristanI suppose it could be orchestrated with an active state09:09
tristanjuergbi, or with a _schedule_pull() and/or _schedule_assemble() sort of approach09:10
juergbi'active' was just a placeholder, point was that Elements would directly influence other Elements, not going through the Pipeline09:10
tristanYeah, basically I guess this comes down to... when an element fails in the PullQueue, it must propagate some state to it's dependencies09:11
tristantelling it's build-of-build dependencies that they are needed for this run09:11
juergbiwould the number of session elements always include those potentially queued elements?09:15
juergbijust a detail I guess09:15
*** bochecha_ has quit IRC09:16
tristanjuergbi, I think that queueing build-of-build *after* the depending elements is in fact a meaningless operation, the more I think of it09:17
tristani.e. it's not important to put them after, the build plan is still sorted in an optimal way for "what we need first comes first"09:17
tristanit's just a lot of elements get skipped at the beginning of the loop09:17
tristan(or temporarily ignored, to avoid confusion with the word "skipped")09:18
tristanjuergbi, the more I have been thinking about session / total... the more I think that it is a weird prediction we make; it's somehow useful but does not match perfectly with the underlying machine09:19
tristanif we have sessions where some elements are tracked, some elements are tracked and built, and other elements are only tracked (not extremely useful but it's an example): then session / total doesnt really capture something very meaningful09:20
*** tristan has quit IRC09:35
gitlab-br-botbuildstream: merge request (jennis/explanation-of-commands->master: commands.rst: Split up commands to easily allow for further description) #450 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45009:42
*** aday has quit IRC09:44
*** aday has joined #buildstream09:54
*** tristan has joined #buildstream09:58
jennistristan, if you have a moment !450 (^^) is a small docs fix re the patch allowing for adding further description to commands10:00
tristanjennis, let's merge10:03
*** aday has quit IRC10:04
jennisok great10:06
gitlab-br-botbuildstream: merge request (jennis/explanation-of-commands->master: commands.rst: Split up commands to easily allow for further description) #450 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/45010:07
gitlab-br-botbuildstream: merge request (valentindavid/etag->master: Store etag along with cache.) #441 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44110:09
finnIs there a variable I can use which contains the absolute project folder path?11:05
finne.g I have a file in files/whatever.tar11:05
finnI'd like to source it, what should the URL look like?11:05
finnAt the moment, I have an alias: project_dir: file:///home/finn/alpine11:06
finnThen in my .bst I have: url: project_dir:/files/whatever.tar11:06
Nexus        directory = self._main_options['directory']11:07
Nexus        directory = os.path.abspath(directory)11:07
Nexusis what i currently use11:07
Nexusnot sure if we store a hard path?11:07
tlaterNexus: you can use python code in .bst files? ;p11:07
*** bethw has quit IRC11:08
Nexussorry, i saw variable and assumed11:08
Nexusdon't our bst files just check relative to themselves?11:09
Nexuse.g. gnome-modulesets/elements/apps.bst points at apps/foo.bst11:10
Nexuswhere apps is a dir inside elements11:10
finnDirectory structure locally is:11:10
finn-elements11:10
finn-files11:10
gitlab-br-botbuildstream: merge request (valentindavid/253_better_missing_variable_errors->master: Fix provenance in error message for undefined variables.) #449 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44911:10
tlaterNexus: Yes, dependencies work that way11:10
tlaterBut tar *sources* need to specify an absolute directory in their `path`11:11
tlaterOr `url`11:11
Nexuswell that sucks11:11
tlaterWe work around it in the test suite using code, but I can't remember if I ever checked for a variable...11:11
finnDo you format the project.conf?11:14
finnproject_dir: file://{project_dir}11:14
tlaterfinn: yep, that's exactly what we do11:15
tlaterWe might want to create such a variable - this has been annoying me since I started working on buildstream11:16
tlater(Assuming it doesn't exist and I simply overlooked it)11:16
finnThis is basically for a docker workaround currently so it isn't vital. I'll keep absolute file path name until you make a decision :)11:17
jennisjuergbi, looks like !446 is ready to land? As far as I can see, there is no obvious blocker to my remote expiry, however I'd like to test against your changes to make sure11:34
gitlab-br-botbuildstream: merge request (valentindavid/253_better_missing_variable_errors->master: Fix provenance in error message for undefined variables.) #449 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44911:44
*** csoriano has quit IRC11:48
*** csoriano has joined #buildstream11:48
gitlab-br-botbuildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/43312:07
gitlab-br-botbuildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/43312:09
gitlab-br-botbuildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/43312:10
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34712:12
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34712:20
NexusI can't seem to come up with a way of omitting .bzr from the source.stage() function. The wya i understand it, i'd need to prevent .bzr from being checked out, without modifying the upstream branch (which i doubt we want to do?), i don't see how this is possible12:20
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34712:22
valentindtristan, Should we forbid relative path of elements to contain colon character? Or should we check for name with colon before we split with for junctions?12:25
*** bethw has joined #buildstream12:32
*** aday has joined #buildstream12:40
gitlab-br-botbuildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/43312:45
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34712:46
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44512:48
*** jonathanmaw has quit IRC13:02
*** xjuan has joined #buildstream13:21
gitlab-br-botbuildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/37214:09
jenniswhen writing docs, is it ok to refer to "name:" "kind:" "sources:" etc.. as nodes?14:21
*** toscalix has quit IRC14:44
*** mohan43u has quit IRC14:44
finnAre they points of intersection?14:46
finnTo me they look like keys14:47
finnor types?14:48
tlaterThey're certainly dictionary keys, but this seems like an odd term to use in user-facing documentation14:49
tlater"Options", or "settings" perhaps?14:49
finnas is a node, a node is generally something which has branches coming out of it14:49
finnThere is also an option called options14:49
tlaterI'm not opposed to nodes/keys, but I feel like it's not quite as obvious to users.14:49
finnsettings is probably better than options to avoid a name clash?14:50
tlaterYeah, definitely14:50
finnsettings it is14:50
tlaterHehe14:50
tlaterjennis: ^14:50
tlater(Obviously we'd need to make sure that this term is used accross the documentation, which I don't think it currently is)14:51
finnSo I've tarred up a docker image, sourced it in a .bst15:01
finnrun the build command successfully15:01
finnwhen I try to run the shell, I get this:15:01
finnbwrap: execvp sh: No such file or directory15:01
jmacThat means there's no /bin/sh in your artifact15:02
tlaterfinn: have you checked that the platform ends up in the sandbox properly? You can assert that using `bst export`15:02
jmacUsually it happens when you miss off a dependency15:02
tlaterIf you export the element with the tarfile you should get a nice sysroot :)15:02
finnError: No such command "export".15:02
tlatererr checkout15:03
tlaterSorry15:03
*** mohan43u has joined #buildstream15:03
finncheckout works15:04
tlaterDoes the resulting directory contain bin/sh?15:04
finn bst checkout base.bst directory/15:04
finndirectory/image-tools/bin/sh exists15:05
tlaterAh15:05
tlaterSee, this won't work15:05
tlaterBecause bin/sh isn't in the root directory :)15:05
finnriiight15:05
tlaterThis is a little tricky, I think you'll need to make sure buildstream doesn't extract it into a directory with the same name as the element15:06
tlaterThere was a note about that in the documentation for tar sources, I think15:06
*** tiago has quit IRC15:06
finnThanks15:13
finnthat worked15:13
finnbase-dir: '*'15:15
gitlab-br-botbuildstream: merge request (jennis/initialise-elements-sub-dir->master: app.py: Ensure an elements sub-dir is created upon project initialisation) #451 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45115:15
jennisOk, I agree with settings15:17
jennisCould someone give !451 a quick review, it's a pretty trivial change15:17
jennispls :)15:17
finnwhat are all these triangles?15:18
finn15:18
tlaterEum, where?15:18
jennisThat's my terminal settings :p15:18
tlaterAh15:19
tlaterx)15:19
finnvery pretty15:19
finnthough I'd remove them and have a simple shell result15:19
tlaterSo much vertical space though!15:19
finnAh so you're creating an elements dir with the init command now15:20
gitlab-br-botbuildstream: merge request (jennis/initialise-elements-sub-dir->master: app.py: Ensure an elements sub-dir is created upon project initialisation) #451 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45115:20
jennis^^ no more triangles15:21
finnhaha thanks15:21
jennisyes, just because our `bst init` command will by defauly create a project.conf which has the SETTING "element-path: elements"15:21
*** aday has quit IRC15:23
jennistlater, thoughts on scrapping the (broken) x86_image in examples and replace it with what finn has just come up with15:23
jennisas the maintainer of said repo :p15:23
tlaterjennis: Different purposes, we should instead add finn's15:24
*** aday has joined #buildstream15:24
jennisThought we didn't use that x86 image?15:24
tlaterBoth will be broken until the docker plugin is fixed15:24
jennisFinn's source is a tarball in files/15:24
tlaterjennis: The x86image example is an example that shows how you create an x86image using the x86image plugin15:24
jennisaha, makes sense...15:25
tlaterfinn's example is an example of how you create a simple, small buildstream project15:25
jennisok, I agree with add15:25
tlaterIn either case, I'm not sure how much I like the tarball in files/15:25
tlaterWe could also do that for the x86image plugin15:25
tlaterBut it makes the example hard to run because you'd need to modify the url first15:26
finnyou still want to add things to this -examples repo then?15:26
jennisHow about hosting it somewhere?15:26
finnI can create a workaround with a script15:26
finnor host it somewhere15:26
finncould host it on git...15:26
tlaterScript workaround isn't particularly user friendly, which is important for someone just starting off, I think15:27
tlaterI'm not sure what licensing we need to worry about with hosting it somewhere, or where we'd host it15:27
tlaterPreferably someone just fixes the docker plugin and we don't need to worry about this15:27
finnHow hard would this docker patch be>15:27
finnThis work isn't _essential_15:27
*** aday has quit IRC15:27
tlaterNot very, it's correcting the permissions of a few files15:28
finnWouldn't this make more sense then15:28
jennisWhy are you against storing it in files?15:28
jennisWe do this for our integration tests15:28
*** aday has joined #buildstream15:28
tlaterjennis: Because you need an additional script to be able to run it then15:28
finnI'd have to format my string15:28
tlaterI worry that this would be confusing to a beginning user.15:29
finnagreed15:29
finnit isn't the "normal" way15:29
tlater"Do we need to run this weird script on a new buildstream project?"15:29
jennisok fair enough15:29
finnok, what about I make this docker patch my task on Tuesday15:29
tlaterThat would be awesome :)15:29
finnand jennis - write the docs as if this example existed15:29
jennis+ 115:29
finnI can send you it if you really want it now15:29
jennisNo, it's ok :)15:30
finnI can tar it up15:30
finnok15:30
*** tiago has joined #buildstream15:36
*** tiago has quit IRC15:39
tlater^ As an alternative to fixing the docker script, we should probably also discuss adding a project-directory variable to core buildstream15:57
tlaterWill probably have to wait until we're at a sensible time to talk to tristan, though ;)15:58
* tlater sets up an issue for this in the mean time15:58
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44516:02
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44516:02
*** dominic has quit IRC16:05
gitlab-br-botbuildstream: issue #388 ("Allow including sources besides `local` from the project directory") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/38816:20
tlaterfinn: I think I managed to make a convincing case there :)16:20
tlater(This is all assuming it doesn't currently exist, but I can't find anything suggesting so, and nobody has told me it does yet)16:22
*** finn has quit IRC16:23
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44516:25
*** jennis has quit IRC16:47
*** aday has quit IRC16:50
*** aday has joined #buildstream16:50
*** bethw has quit IRC17:10
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34717:20
*** xjuan has quit IRC20:25
*** aday has quit IRC21:22
*** aday has joined #buildstream21:23
*** cs_shadow has quit IRC21:55
*** aday has quit IRC22:17
*** tristan has quit IRC23:38

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!