IRC logs for #buildstream for Thursday, 2018-05-03

*** Prince781 has joined #buildstream02:28
*** Prince781 has quit IRC03:39
*** tristan has joined #buildstream04:42
*** mohan43u has joined #buildstream06:43
*** jonathanmaw has joined #buildstream07:46
*** toscalix has joined #buildstream07:53
*** tristan has quit IRC08:02
gitlab-br-botbuildstream: merge request (valentindavid/workspacedir_config->master: Add 'workspacedir' configuration.) #444 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44408:16
*** bethw has joined #buildstream08:51
gitlab-br-botbuildstream: merge request (juerg/no-remote-summaries->master: WIP: Drop use of OSTree summary files and dynamically plan build dependencies) #446 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44609:11
*** tiago has quit IRC09:17
*** tiago has joined #buildstream09:40
*** aday has joined #buildstream09:46
jmacjuergbi: With regard to BST_VIRTUAL_DIRECTORY: anyone writing a new plugin without knowing about the virtual directory changes will be unaffected, right? They can still use get_directory10:03
juergbijmac: yes. I'd rather have it the other way round but that's not really an option for compatibility reasons10:03
jmacand if they are aware of the changes, and choose to just use get_virtual_directory, that's fine, so I don't see what BST_VIRTUAL_DIRECTORY adds in that case10:03
juergbithe issue is when we no longer have a regular FS directory as backend of the virtual directory10:04
juergbior are you suggesting to create that on demand?10:04
jmacNo10:05
juergbithe old plugins can't be used for remote execution, e.g.10:05
jmacBut if you use get_virtual_directory you shouldn't be able to see the underlying directory being used in any case10:05
juergbiright but the point of BST_VIRTUAL_DIRECTORY is that BuildStream core knows whether your plugin is compatible with purely virtual directories10:05
juergbiBuildStream core can't easily know at the start whether the plugins use get_directory() or get_virtual_directory() (or both)10:06
juergbiwithout that flag10:06
jmacRight, but does it need to? We can still provide a useful error message if you use get_directory and only have a CAS backend10:06
jmacThe only thing I can think of that being useful for is providing a list of which plugins use get_directory10:07
juergbian error message is not always sufficient, though10:08
juergbithe plan is that we'll always use the CAS backend, and prefer FUSE even for local execution10:08
juergbihowever, we can't use CAS-backed FUSE with get_directory() plugins10:08
juergbito avoid breaking existing projects/plugins, we have to fall back to non-FUSE for that10:08
jmacSo you want to dynamically choose to use the CAS backend for some elements?10:08
juergbiI think we have to10:09
jmacRight, that was the bit I was missing. Thanks!10:09
juergbiHow else can we cover backward compatibility?10:09
juergbiUnless we require opt-in on the project level10:09
jmacI thought it'd be set per-project10:09
tlaterjuergbi: Will we continue to support the TarCache for this purpose then?10:09
juergbijmac: For remote execution it will be opt-in, however, I wasn't planning on requiring opt-in for local execution with CAS-backed FUSE10:10
juergbitlater: no, CAS artifact cache can still be used with old plugins without FUSE10:10
juergbithey just can't profit from the FUSE on-demand staging10:10
tlaterAlright, that makes more sense then :)10:10
jmacjuergbi: Right, this makes sense now.10:11
juergbiOk. If anyone can come up with a better compatibility story, I'm all ears, of course :)10:11
juergbiInstead of BST_VIRTUAL_DIRECTORY we could consider a more generic BST_PLUGIN_API_VERSION or something like that10:12
juergbiDon't know whether that would be better long term10:13
jmacNo, I'm absolutely fine with BST_VIRTUAL_DIRECTORY10:14
juergbijennis: You asked about the scheduling changes a few days ago, they are now in !44610:14
jennisjuergbi, thank you10:19
gitlab-br-botbuildstream: merge request (valentindavid/385_vte_notification->master: buildstream/_frontend/app.py: Notify VTE on failure in interactive mode.) #447 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44710:24
gitlab-br-botbuildstream: merge request (valentindavid/385_vte_notification->master: buildstream/_frontend/app.py: Notify VTE on failure in interactive mode.) #447 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44710:46
*** persia has joined #buildstream11:06
*** awacheux has joined #buildstream11:16
*** bethw has quit IRC11:32
*** finn has quit IRC12:01
*** finn has joined #buildstream12:03
*** tristan has joined #buildstream12:24
juergbitristan: fyi: I've opened !446 which includes the dynamic planning. As expected it conflicts with your pipeline refactor branch, so I'd appreciate comments on the changes and then we'll have to decide how exactly to merge the two branches12:36
gitlab-br-botbuildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/42112:41
tristanjuergbi, ok so I ran into a wall... which maybe tlater can help me with12:46
tristantlater, --except logic is broken, especially for tests/frontend/buildtrack.py12:47
tristantlater, notice that Pipeline.remove_elements() accesses self.dependencies() and self.targets directly: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_pipeline.py#L50412:47
tristantlater, that means that... when you run `bst track --except baz.bst foo.bst bar.bst` ... the constructed tracking list is completely different than when you run `bst build --track foo.bst --track bar.bst --track-except baz.bst target.bst`12:49
*** bethw has joined #buildstream12:49
tristanI currently have a function: https://bpaste.net/show/090603dd3fc2 which doesnt pretend that "targets" is meaningful when I do the except algo, but I'm not sure the results are correct anymore12:50
tlaterAlright, taking a look12:52
* tlater has a meeting in a bit, might not have enough time :|12:54
tristanjuergbi, I'm taking a look now... made a comment but it's unrelated, lemme just look at the pipeline changes12:56
juergbita12:56
tristanwoosh12:57
tristanself._first_non_track_queue is first thing I'm seeing12:57
tristanjuergbi, I did something similar to your _run() function to remove the redundant scheduler runs...12:59
tristanstill not a pushed change in my branch I know12:59
juergbiah, I actually checked your branch whether you already fixed that...12:59
tristanYeah I got stuck :-S13:00
tristanself._first_non_track_queue goes kind of against what I wanted to achieve, while I can see how that lets you reduce lines of code13:01
*** finn_ has joined #buildstream13:01
tristanI am kind of shooting for... a bit more verbose Stream.build(), Stream.fetch(), Stream.track() etc... the main session running things13:02
*** finn has quit IRC13:02
*** finn_ is now known as finn13:02
tristanUmmm, lemme push a hint at where this is going... although it's stuck in a state of flux13:02
juergbitristan: it's not just to reduce lines of code, though, needed to know where to queue dynamically planned elements (build dependencies)13:07
tristanjuergbi, I thought it was for how we separate where we add the track elements in bst build --track13:08
juergbiit makes things easier there as well, yes, but for dynamic planning it's essential (or something like it)13:09
tristanHmmm13:09
tristanjuergbi, so this is where I was going: https://gitlab.com/BuildStream/buildstream/blob/tristan/pipeline-refactor/buildstream/_stream.py#L26313:10
tristanI only have `bst show` and `bst track` using the new pipeline loading stuff... notice in _pipeline.py I have some methods I've prefixed new_ ... so once I convert the higher level stuff, I will remove the old ones and then s/new_//13:11
tristanit's a process13:11
tristanjuergbi, it *looks* however like what you have done there... lands fairly easily in my refactor before this change... i.e. I've separated Stream/Pipeline completely by that point but havent untangled the weird shit that pipeline is actually doing yet13:12
tristanlike pretending that "there is only one list of targets", and that "except" means something different depending on the command13:13
tristanhowever, it's a bit of an ugly state of flux13:13
tristanMaybe I can clean it up, so that the only ugly part is that Stream.py still has init_pipeline()... then you could rebase on top of that... and I could continue with the hard part of making pipeline.py make sense after that13:14
tristannot sure, sounds ugly still13:15
juergbi(meeting)13:16
Nexusjuergbi: How long is it looking for CAS caching to go through?13:22
tristanjuergbi, I think that I can rebase on top of your patch instead btw, but I need to have a closer look at how it works13:22
tristanI dont think I can sort it out tonight13:22
tristanjuergbi, fwiw, I was wondering also (asides from your dynamic scheduling), since the calling/queueing layer needs to explicitly schedule tracking... whether it should also have to explicitly schedule assembly13:24
tristanI wonder if that would be an equivalent but opposite approach than what you took for marking the elements as "queued"13:24
tristan(but havent looked deep enough, so I could totally be spewing nonsense here)13:25
tristantlater, note that something I think is clearly wrong is this: https://gitlab.com/BuildStream/buildstream/blob/master/tests/frontend/buildtrack.py#L3913:26
tristantlater, here you say that "If we except 2.bst recursively, it should result in excepting 2.bst and 7.bst"13:26
juergbitristan: with assembly you mean BuildQueue? or the combination of all non-track queues or...?13:26
juergbiI'm not very happy with 'first_non_track_queue' but at least it's descriptive ;)13:27
tristantlater, looking at the graph you've constructed: https://gitlab.com/BuildStream/buildstream/blob/master/tests/frontend/buildtrack.py#L79 ... 2.bst clearly excepts 2.bst, 3.bst, 4.bst, 5.bst, 6.bst and 7.bst13:27
juergbiI'm wondering whether we should move a bit closer to two subpipelines, one for tracking, one for the rest13:27
*** finn_ has joined #buildstream13:29
*** finn has quit IRC13:31
*** finn_ is now known as finn13:31
*** finn has quit IRC13:32
*** finn has joined #buildstream13:32
*** bethw has quit IRC13:36
tristanjuergbi, I'm not sure what that means... i.e. elements need to pass through tracking before pull & build etc13:40
juergbitristan: they need to pass through tracking before pull and build, if they are tracked, yes13:41
tristanjuergbi, maybe what you mean by "pipelines" on the other hand is "lists of elements"13:41
tlatertristan: It's not quite that clear13:41
juergbihowever, besides supporting tracking of elements that we then don't build13:41
*** finn has quit IRC13:41
tlaterBecause you're building 0.bst13:41
tlaterSo, because you're not explicitly excepting 3.bst, it's still included13:42
*** finn has joined #buildstream13:42
tlaterBecause 0.bst depends on it13:42
tristantlater, yes but you are saying "build 0.bst ... and track 2.bst except for 3.bst"13:42
tristantlater, it seems to me entirely incorrect that the --except algorithm takes the build target(s) into consideration when calculating which elements to *track*13:42
juergbitristan: I meant, besides supporting building of elements that we don't track, we may also want to support tracking of elements that we don't build (because they are unused build-only dependencies)13:42
tristantlater, these are two entirely different selections happening13:43
tristantlater, I refer/repeat the above for scrutiny:13:43
tristan<tristan> tlater, that means that... when you run `bst track --except baz.bst foo.bst bar.bst` ... the constructed tracking list is completely different than when you run `bst build --track foo.bst --track bar.bst --track-except baz.bst target.bst`13:43
tristantlater, why should those two differ ?13:44
tristanjuergbi, *yes* agree ... one might say: bst build foo.bst --track bar.bst --track baz.bst --track-except pony.bst13:44
*** finn has quit IRC13:45
*** finn has joined #buildstream13:45
tristanjuergbi, now bar and pony might be dependencies of foo... but baz might be completely unrelated13:45
*** finn has quit IRC13:45
*** finn has joined #buildstream13:45
tristanjuergbi, my refactor would allow that accidentally, which is why I wondered if we should explicitly be calling Element._schedule_assemble() like we do for Element._schedule_tracking()13:46
tristan(otherwise, I would filter out unrelated tracking targets to conform to current behavior, as that isn't a big deal)13:46
tlatertristan: I agree, that inconsistency annoys me now.13:46
tlaterI don't quite remember why it was there in the first place13:46
tlaterBut it may just have been an accidental result of using 'Pipeline.target'13:47
tlaterPerhaps that happened when we started allowing multiple targets?13:47
tristantlater, yes, so I am in a predicament where it seems that instead of the test cases representing the desired behavior, and the code being made to conform, the test cases represent the ad-hoc behavior at the time13:47
tristanwhich is worrisome13:48
juergbitristan: that could make sense, yes. although please note we already have _schedule_assemble() which is specifically for the build job (i.e., Element.assemble()) while we'd need a new method to cover the combination of pull+fetch+build13:48
tristanjuergbi, indeed13:49
tristanjuergbi, it might be more verbose, but consistency would be more valuable (if we find that it makes more sense)13:49
tlaterI remember spending a lot of time thinking about what that tree should look like, but I certainly only considered it for `bst build`13:49
tristanjuergbi, not convinced it's the right avenue, just throwing the idea out there13:49
tristanjuergbi, note https://gitlab.com/BuildStream/buildstream/blob/tristan/pipeline-refactor/buildstream/_pipeline.py#L8413:49
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44513:49
tristanjuergbi, with Pipeline.new_load(), I just load/resolve the groups of targets, and return groups of elements (redundancy allowed)... this allows Stream.build() to say:  elements, track_elements, track_except_elements = Pipeline.new_load(targets, track_targets, track_except_targets)13:50
tristanThen we have 3 lists of elements which all point into the same loaded build graph13:51
tristanfrom where we go ahead and use planning features, except features, and other list manipulation features provided by Pipeline to decide what to do with elements and feed them into queues13:51
tristan*from there13:51
*** finn has quit IRC13:52
tristantlater, that is very unfortunate13:52
*** finn has joined #buildstream13:52
tristantlater, because right now, I currently need `bst track` to be correct, and make `bst build` behave the correct way13:52
tlatertristan: I *do* think that in that particular test case excluding "3.bst" would be an annoyance for the user13:53
tlaterBecause the user would only think about 0.bst, which clearly *includes* 3.bst13:53
tristantlater, bst track does not have the potential to confound stuff, so it is the prime example... in fact this same test should be run with @pytest.parameterize() an extra time, such that we test the same tracking plans with both `bst track` and `bst build`13:53
tristantlater, you are not excepting from the activity of building, you are only excepting from the activity of tracking13:54
tlaterEven in that case - why would excluding 2.bst exclude 3.bst, which is clearly depended on by 0.bst?13:54
tristantlater, lemme look again13:55
tlaterI think the immediate dependencies by the target(s) should always be explicitly excluded.13:55
tristantlater, ok so we're talking different things here...13:55
tristantlater, *if* you did --track 0.bst --track-except 2.bst... I should agree with you13:56
tristantlater, the way your test is constructed however... you are saying that "excepting 2.bst excepts only 2.bst and 7.bst"13:56
tristanlater on in the test you do a subtraction of sorts13:56
tristanto derive what you actually expect to see in the test case13:57
tlatertristan: Which exact combination are we looking at, then?14:00
tristanthat is a good question, if I fix one, a different one breaks14:01
tristanbecause I go ahead and separate them14:01
tristanso that tracking element selection is entirely cleanly separated from build target selection14:01
tristantlater, I am trying to figure out what to do14:01
tristantlater, are you telling me I am reading the test incorrectly ?14:02
tlatertristan: You seem to be interpreting what it does correctly14:02
tristantlater, I am expecting that on line 162, where you say that "The result of excepting 2.bst is that 2.bst and 7.bst", is true regardless of what main target has been selected for tracking14:02
tristanmaybe I need to change that so that it is more clear, and squash both parameterization into a single one with 4 parameters14:03
tlaterYeah, that might be better14:03
tlaterThe test case says "the result of excepting 2.bst is that 2.bst and 7.bst are not tracked"14:04
tlaterI.e., if they weren't tracked in the first place, they still wouldn't be. But I don't think you'd expect anything else.14:04
tristanright, but that implies context14:04
*** finn has quit IRC14:04
tristantlater, it implies that you have tracked 0.bst I think14:05
*** finn has joined #buildstream14:05
tristanI guess in that particular case, there is not much choice14:05
*** finn has quit IRC14:05
tlatertristan: Not necessarily. If you track 3.bst, for example, 2.bst and 7.bst are still not in the resulting set14:05
*** finn has joined #buildstream14:05
tlaterIf you track 3.bst and 2.bst simultaneously, 2.bst and 7.bst are not in the result14:05
*** finn has quit IRC14:06
*** finn has joined #buildstream14:06
tlaterI'd expect that if I wrote another test case which just tracks 7.bst, that would actually break - but I'm not sure about that14:06
tristanif you track another element which depends on 2.bst and on 8.bst... and --track-except 2.bst, then 2-7.bst will not be tracked14:07
tristaneven though the other element would otherwise track 2-8.bst without the except statement14:07
tlaterYeah, I think that's what is expected here14:08
tlaterIf there is inconsistency with `bst track`, I think `bst track` should be changed to follow this, instead14:08
tristantlater, I prefer to change this to one parameterize... with (track_elements, except_elements, expected_tracked_element)14:08
tlaterDefinitely, the double parameterization makes this hard to think about14:09
tristantlater, then I might put the whole thing into a separate parameterize... which does the whole thing with, or without build14:09
* tlater thought it was nice to be able to test a lot without writing much, but this seems too unclear.14:09
tristani.e. one mode of testing is with build, the other is with plain track14:09
tristanensuring that the semantics of tracking selection are the same14:09
tlaterYep, that sounds better - is there still a question about what the result of an exception should be?14:10
tristantlater, what I'd like though; is for you to take a look at my interpretation of Pipeline.remove_elements() closely: https://bpaste.net/show/090603dd3fc214:10
tristantlater, is there a question ?14:10
tristantlater, I agree with 2.bst and 7.bst, *if* --track 0.bst was provided, or the other examples you provided14:11
tlaterOk :)14:11
tlaterHrm, I really would like to run that function to see its output14:12
tlaterBut I'll have a look through it for now14:12
tristanyeah I understand14:12
tristantlater, I didnt modify any content, only removed the implicit relations to self.targets and self.dependencies()14:12
tristantlater, note that self.new_dependencies() is the same as the old one, except takes a list of targets instead of assuming that a single list of targets exist14:13
tristani.e. Pipeline becomes rather stateless14:13
tlaterYup, that makes sense14:14
gitlab-br-botbuildstream: merge request (juerg/non-strict-buildable->master: element.py: Fix buildable check in non-strict mode) #448 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44814:14
*** finn has quit IRC14:15
tristantlater, pipeline-refactor has the function in case you want to get your hands dirty and try it out, but I didnt include my broken implementation of Stream.build() in it14:17
tristanhttps://bpaste.net/show/dde6fc02e9b1 that is my broken Stream.build()14:17
tlaterWell, if I just except in the right spot I won't have to worry about that ;p14:18
tristantlater, I expect you're too busy with other things anyway to get too deep, you've provided me with some thoughts of how to proceed with the test14:18
* tlater probably shouldn't, yeah, I'll try and assert that the algorithm matches what the other one does in my mind 14:19
tristannote line 31 in the above paste, where I call Pipeline.except_elements() with 3 parameters14:19
tristantlater, yeah that's just sort of what I was hoping for, some verification that... what I am trying to do makes sense :)14:19
tristanjuergbi, I think what we're going to do is that we're going to land your changes ahead of my branch, to expedite remote cache expiry, and I will just deal with the nightmare on my side :)14:21
tristanjuergbi, but give me some time tomorrow to look it over just a bit ;-)14:22
juergbithanks14:22
*** bethw has joined #buildstream14:22
gitlab-br-botbuildstream: merge request (juerg/no-remote-summaries->master: Drop use of OSTree summary files and dynamically plan build dependencies) #446 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44614:23
tlatertristan: As far as I can tell there should be no difference between those two implementations14:34
tlaterIt also still matches what I had in my mind of what this looked like, so there should be no regressions14:34
tlaterUnit tests would be really nice for this sort of thing, especially now that the pipeline is becoming stateless...14:35
*** bochecha_ has joined #buildstream14:38
juergbiNexus: Sorry, forgot to answer your question. The goal is to have a full-featured/ready MR for CAS by mid of May14:44
Nexusok thanks14:46
gitlab-br-botbuildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44515:06
toscalixis it the next planned release 1.1.4 or 1.2 ?15:09
*** finn has joined #buildstream15:12
juergbitoscalix: next development release is 1.1.4, next stable release is 1.2.0 (in September)15:14
*** bochecha_ has quit IRC15:15
toscalixthanks15:18
jmacjuergbi: Do you know where I could find some better docs on Google CAS? I'm looking at https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteexecution/v1test/remote_execution.proto, but it doesn't seem to be complete16:21
*** dominic has quit IRC16:28
*** bochecha_ has joined #buildstream16:29
jmacOn closer inspection maybe the missing things are features you've proposed like symlinks16:40
jmacI'm not sure what directory.ParseFromString is meant to do, though16:40
*** Prince781 has joined #buildstream16:40
*** bethw has quit IRC17:05
*** xjuan has joined #buildstream17:05
*** jonathanmaw has quit IRC17:05
*** toscalix has quit IRC17:33
*** bochecha_ has quit IRC17:52
*** aday has quit IRC17:53
*** Prince781 has quit IRC17:54
*** Prince781 has joined #buildstream18:02
*** tristan has quit IRC18:48
gitlab-br-botbuildstream: merge request (valentindavid/253_better_missing_variable_errors->master: Fix provenance in error message for undefined variables.) #449 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44918:48
juergbijmac: for the symlink proposal, see https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64/edit18:50
juergbialso mentioned here: https://docs.google.com/document/d/12c3oAPgedckLpue2yj0xGgJTEOyUm4mXWWBJ157J-8I/edit#heading=h.nhgmkgkf44t218:51
juergbiParseFromString is a generic grpc method, see https://developers.google.com/protocol-buffers/docs/pythontutorial18:52
juergbiplease note that `String` refers to a binary string, not a text/unicode string18:53
juergbii.e., ParseFromString simply deserializes a protobuf message18:53
*** xjuan has quit IRC20:11
*** Prince781 has quit IRC21:36
*** Prince781 has joined #buildstream22:21
*** Prince781 has quit IRC22:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!