IRC logs for #buildstream for Wednesday, 2019-01-16

*** milloni has quit IRC00:00
*** milloni has joined #buildstream00:00
*** bilelmoussaoui has joined #buildstream01:23
*** nimish has joined #buildstream02:20
*** nimish has quit IRC02:27
*** nimish has joined #buildstream02:29
*** nimish has quit IRC02:44
*** nimish has joined #buildstream02:45
*** xjuan has quit IRC03:41
*** nimish has quit IRC03:51
*** nimish has joined #buildstream06:33
*** bochecha has joined #buildstream07:10
*** bilelmoussaoui has quit IRC07:19
*** alatiera has joined #buildstream07:56
*** nimish has quit IRC08:21
*** nimish has joined #buildstream08:21
*** toscalix has joined #buildstream08:52
*** mohan43u has joined #buildstream08:52
*** sambishop has joined #buildstream09:11
*** nimish has quit IRC09:41
*** nimish has joined #buildstream09:42
*** rdale has joined #buildstream09:51
*** nimish has quit IRC09:57
*** nimish has joined #buildstream09:57
*** nimish has joined #buildstream09:57
*** raoul has joined #buildstream10:02
*** jonathanmaw has joined #buildstream10:06
*** nimish has quit IRC10:22
*** nimish has joined #buildstream10:23
gitlab-br-botvalentindavid merged MR !1069 (valentindavid/git-reduced-history->master: buildstream/_gitsourcebase.py: Reduce git history for git describe.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/106910:33
*** alatiera has quit IRC10:37
*** alatiera has joined #buildstream10:38
*** bilelmoussaoui has joined #buildstream10:55
*** nimish has quit IRC10:58
*** nimish has joined #buildstream10:58
*** cs-shadow has quit IRC11:04
jennisjmac, I've made your suggested change to https://gitlab.com/BuildStream/buildstream/merge_requests/107211:06
jennisI assume we're happy to merge once pipeline passes?11:07
*** rdale has quit IRC11:12
*** nimish has quit IRC11:23
*** nimish has joined #buildstream11:23
*** rdale has joined #buildstream11:25
gitlab-br-botjennis merged MR !1072 (jennis/doc_fixes_in_context->master: Small documentation/comment fixes in context.py) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107211:33
*** cs-shadow has joined #buildstream11:35
jmacjennis: Yes, go for it11:40
jmacI approved it when I reviewed it as it was a very minor nit-pick11:40
jennisyeah I noticed :) thanks jmac11:40
*** nimish has quit IRC11:48
*** nimish has joined #buildstream11:49
*** nimish has quit IRC11:54
*** bilelmoussaoui has joined #buildstream11:54
*** nimish has joined #buildstream11:54
*** nimish has quit IRC12:14
*** nimish has joined #buildstream12:15
*** nimish has joined #buildstream12:15
*** abderrahim32 has quit IRC12:18
*** abderrahim32 has joined #buildstream12:18
*** nimish has quit IRC12:20
*** nimish has joined #buildstream12:21
*** slaf_ has joined #buildstream12:23
*** slaf has quit IRC12:25
*** slaf_ is now known as slaf12:25
*** nimish has quit IRC12:26
*** nimish has joined #buildstream12:26
*** bilelmoussaoui has quit IRC12:31
*** nimish has quit IRC12:41
*** nimish has joined #buildstream12:42
*** alatiera_ has joined #buildstream12:48
*** alatiera has quit IRC12:48
*** alatiera_ is now known as alatiera12:49
gitlab-br-botjuergbi merged MR !1071 (raoul/cas-refactor->master: Cas refactor) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107112:56
gitlab-br-botphildawson opened (was WIP) MR !1075 (phil/plugin-testing-api->master: Expose basic api for testing external plugins.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107513:17
philHi all, anyone able to take a look at !1075 for me. It's my thoughts on what a basic API for use when testing none-core plugins might look like so would appreciate any feedback :)13:19
cs-shadowphil: I was planning to have a look, this seems really useful. In the meantime, it seems like the lint and docs jobs are failing so they might need a little love :)13:26
*** abderrahim32 has quit IRC13:28
*** nimish has quit IRC13:32
*** nimish has joined #buildstream13:32
philcs-shadow, cheers. I'm looking into that now, but struggling to work out what's happening with the docs jobs.13:40
*** tristan has joined #buildstream13:58
*** sambishop has quit IRC14:00
*** lachlan has joined #buildstream14:14
*** sambishop has joined #buildstream14:16
*** nimish has quit IRC14:17
*** nimish has joined #buildstream14:17
*** nimish has quit IRC14:27
*** nimish has joined #buildstream14:28
*** alatiera has quit IRC14:32
*** nimish has quit IRC14:33
*** nimish has joined #buildstream14:33
*** alatiera has joined #buildstream14:36
*** lachlan has quit IRC14:37
*** nimish has quit IRC14:38
*** nimish has joined #buildstream14:39
*** lachlan has joined #buildstream14:41
gitlab-br-botaevri opened MR !1076 (chadnan->master: bst-docker-import: fix Chandan's name) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107614:44
*** nimish has quit IRC14:49
*** nimish has joined #buildstream14:49
*** alatiera has quit IRC14:56
*** sambishop has quit IRC14:56
*** sambishop has joined #buildstream14:58
gitlab-br-botaevri merged MR !1076 (chadnan->master: bst-docker-import: fix Chandan's name) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107615:13
*** sambishop has quit IRC15:17
*** sambishop has joined #buildstream15:22
*** nimish has quit IRC15:29
*** nimish has joined #buildstream15:29
*** sambishop has quit IRC15:32
*** nimish has quit IRC15:35
*** nimish has joined #buildstream15:38
jennistristan, are you around?15:38
*** tristan has quit IRC15:42
*** tristan has joined #buildstream15:44
*** sambishop has joined #buildstream15:46
*** ChanServ sets mode: +o tristan15:46
tristanjennis, yep15:46
tristanjennis, whats up15:46
tristanjennis, Can I land the test refactor ?15:46
jennisahh wonderful15:46
jennisno this is something unrelated :p15:47
jennisSo basically I've been working on the bst artifact subcommands15:47
jennisand while for delete and checkout, we've been able to handle artifact refs relatively easily, for pull, I'd had to add a whole load of questionable logic in cli.py so that we can directly pull an artifact ref15:48
*** nimish has quit IRC15:48
jennisAfter discussing this with juergbi, we've agreed that this is not the proper way to do it15:48
jennisHere is what I tried doing for `bst artifact pull`: https://gitlab.com/BuildStream/buildstream/blob/6e12c3fe5d1067b97e1850355f4e69cc6905f8d4/buildstream/_frontend/cli.py#L99215:49
gitlab-br-bottristanvb opened MR !1077 (tristan/detoxing-integration-tests->master: Make integration tests parallelizable) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107715:49
*** nimish has joined #buildstream15:49
jennishttps://gitlab.com/BuildStream/buildstream/blob/6e12c3fe5d1067b97e1850355f4e69cc6905f8d4/buildstream/_frontend/cli.py#L99215:49
*** nimish has joined #buildstream15:49
jennisSorry that was the wrong link15:49
tristanjennis, yeah that looks totally wrong, that logic should bleed out so high up15:50
jennisNow basically, when we're dealing with pulling elements, we pass the element to stream, this does the loading, and adds to the pull queue and so on15:50
tristanI dont think it even makes sense for the cli.py to every know what the artifact cache API is15:51
jennisyes, and the reason I did this is because I don't see how we can cleanly handle artifact refs15:51
tristancli.py/app.py should be only talking to stream15:51
jennisOk, thanks for that clarification15:51
jennisBut even if I moved this logic to stream, we're not really handling artifact refs until we hit CASCache15:51
tristanstream is the main interface of the bst core, and everything in _frontend/ should go through it...15:52
jennisok thanks15:52
tristaninitialization is currently messy (i.e. a frontend needs to load the Context() and Project() itself)... anyway, you get the idea15:52
jennisBut now it's a question of, how do I integrate artifact ref handling into the pipeline15:52
tristanWhat is the pipeline ?15:53
tristanYou mean the loaded element graph ?15:53
jennisinto the scheduler and queues, because these all work with ElementJob15:53
tristanI don't believe that you do, ok so basically you are pointing out that "pull" is different in the sense that we need to construct an element in order to do something ?15:54
jennisSo I'm extending pull to be able to take an artifact ref as well as an element name here15:55
tristanjennis, I think the cli probably determines whether a given target is an element or a ref first, and then uses a different Stream() pathway for pulling refs15:55
jennisok, so we need to create another Stream method to handle the pulling of refs?15:55
tristanWell, I don't see how you are going to do it the same way15:55
tristanA ref can be pulled directly15:56
tristanLoading an element basically produces that ref, which can then be pulled directly15:56
jennisBut then there is the loading of the project and gathering the user conf and project conf to get the remotes and then the initialising of the remotes as CASCache objects and then pulling the ref15:57
tristanWhen pulling element targets, you really need to load the project first of all, which you dont need for a ref15:57
jenniswhich is all the messy logic I've currently got in cli.py15:57
tristanAnd you also need to resolve the refs on the fly when you deal with elements15:57
jennisWell, you do need to load the project to get the config, and thus, the urls of the remotes?15:57
tristanor that is unclear actually15:58
tristanperhaps only at build time there is a question of which artifact is preferred depending on strict mode etc15:58
juergbithe current logic in the MR also allows specifying a mix of elements and artifact refs. shall we just disallow that? or should stream support this mix in a single session?15:58
tristanjennis, the Project perhaps, but not the full data model15:58
jennisok sure15:59
jennisSo basically, we're good to write a Stream method which figures out the remotes and pulls the refs directly?15:59
tristanjuergbi, Can we determine what it is by the format of the string ?15:59
juergbiyes15:59
tristanjennis, I would say yes16:00
jennisok that's good to hear, thanks tristan16:00
juergbithis should still go via scheduler, though, right?16:00
* jennis got the impression that it didn't16:00
juergbior are we saying we skip the regular UI if refs are specified?16:01
juergbiseems a bit odd from the user perspective16:01
juergbialthough I could understand it from the implementation perspective16:01
jennisjuergbi, do you think the 'odd from the user perspective' is still odd if we restrict to only refs or only elements?16:03
juergbiwould definitely be better than the mixed mode right now in the MR16:03
juergbiit would still not be ideal, imo16:03
jennisYeah, but still odd in the sense that if you use the command with an element, and then use it again with a ref, what you see as a user looks completely different?16:04
juergbihowever, we could say that we do it like that for now and consider moving this to the scheduler/pipeline later16:04
jennisThat's good16:04
juergbiup to tristan ;)16:04
jennisAnother thing I wanted to address is the '--remote' option16:04
jennisCan this only be a remote that exists in the project.conf(s) or user.conf?16:05
jennisOr should we just be able to specify any old url? (I just don't know how we deal with certificates and keys when this is the case) especially when it comes to pushing16:05
juergbiI thought I saw a discussion about this somewhere16:06
juergbifor pushing I think we can indeed not support CLI URLs due to typically requiring authentication16:06
juergbithis might be something to discuss at the gathering16:07
juergbiI proposed a topic with regards to artifact server configuration in projects and junctions16:07
juergbiwould probably fit in there16:07
tristansorry stepped out16:12
tristanhad a thought, reading backlog...16:12
tristanI had the same thought as juergbi16:12
tristanthat maybe these would bypass the scheduler16:12
tristanthats very un-nice16:13
tristanI was thinking it would be a dummy element per ref or something like that16:13
tristanjuergbi, jennis, it would be nice too if we didnt run the scheduler *twice* just because refs and elements were intermingled on the command line16:14
*** nimish has quit IRC16:14
*** nimish has joined #buildstream16:14
*** sambishop has quit IRC16:15
juergbiyes, I think we should either not allow intermingling or combine them16:15
tristanso we'd have a.) the cli.py passes the opaque targets to Stream.pull()... b.) Stream.pull() sorts the elements from the refs, and loads a build graph of those, and resolves `--deps` for only those... c.) Stream.pull() adds dummy elements for any ref which was specified and appends those to the list of elements... d.) Stream.pull() invokes the scheduler with a PullQueue()16:15
tristanWith that, we should also document explicitly that recursive `--deps` options are not supported on refs, each specified ref will be pulled with a `--deps none` criteria16:16
tristanThat is, whether we allow intermingling or not, we need to specify that for refs16:17
tristanit might not be true at push time, and it may not be true forever for pull (funky dynamic scheduling ?)16:17
jennisOh, they're not options, that's a proposed order of things16:19
jennistristan, could you expand on the dummy element? I'm not really sure I'm following this bit16:21
jennisIs this just *something* that's going to invoke the loading of the project and initialising of the caches?16:22
tristanjennis, We can create an internal dummy element which has no dependencies and it's name is derived from the artifact name (I think we can derive the original element name from the artifact name)16:24
tristanjennis, that element needs to hardwire itself to produce the right key16:24
tristanThis might be useful in other commands which involve handling refs and dont need project loading16:25
jennisok, currently that's a big grey area for me, but I'll dive into it and see where I get16:25
tristanjennis, Anyway... once you have a dummyelement which you can instantiate like DummyElement(project_name, element_name, cache_key)16:25
tristanwhich produces the correct artifact name16:26
jennisAnd parsing the ref should be easy enough16:26
tristanYou can just pop those into a PullQueue along with any other elements16:26
jennisok, thanks tristan, juergbi16:27
tristanjennis, It can be tough to do it through all the existing cache key mechanics... it might make more sense to move the ArtifactCache.get_artifact_fullname() -> Element.get_artifact_name()16:28
tristanjennis, and make sure everything asks Element.get_artifact_name() instead... then have DummyElement just override that16:28
gitlab-br-bottristanvb merged MR !1077 (tristan/detoxing-integration-tests->master: Make integration tests parallelizable) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107716:32
tristanjennis, How about the tests refactor... can we land that ?16:33
tristanhttps://gitlab.com/BuildStream/buildstream/merge_requests/106216:34
tristangitlab-br-bot, tell me about !106216:34
gitlab-br-botMR !1062: General refactor in tests directory https://gitlab.com/BuildStream/buildstream/merge_requests/106216:34
tristan:)16:34
cs-shadowtristan: hi, would you be open to toxifying the 1.2 branch? (The reason I ask is that there are chances that the dependencies will drift over time on that branch, especially since we don't work on that often)16:34
tristancs-shadow, I definitely think we should port all of the tox related things there *if* we're not going to do a stable in a while16:35
tristanthat discussion is going to happen at the gathering but I wouldnt object to being prepared in advance16:35
cs-shadowtristan: ok cool, I'll have a go at it. Just wanted to make sure that you don't have any reservations against it16:36
jennisoh nice, I'll bear that in mind. You can land the tests refactor, it won't take me long to change checkout -> artifact checkout16:36
tristanjuergbi, you said 1.9.x but I think 1.90.x rings a bell, and leaves a nicely paranoid namespace just incase16:36
juergbitristan: I meant 1.90.x16:37
tristanjennis, thanks :)16:37
tristanjuergbi, ah, yeah indeed16:38
tristanAre there things planned for monday in London ?16:44
tristanlaurence, ?16:44
tristanor it's really just from tuesday16:44
*** nimish has quit IRC16:44
*** nimish has joined #buildstream16:45
KinnisonThose of us from mcr don't get to London until late on Monday so I doubt there're plans16:45
KinnisonAt least plans which would involve us16:46
Kinnison:-D16:46
KinnisonI don't think I get to my hotel until gone 9pm :(16:46
tristanI see16:48
tristanAnd on thursday it's off to brussels16:48
KinnisonI thought that was Friday, but I'm going back to mcr anyway :-D16:50
* Kinnison thought tue/wed was buildstream gathering, and thu was this "build tools" hack day16:50
cs-shadowKinnison: right, 29-30 is more buildstream focused, similar to our previous events. And 31st will have a more diverse audience. This should be similar to the hackathon that we did in NY after BazelCon, in case that rings a bell in people's minds16:52
Kinnisoncool16:52
* Kinnison is looking forward to it all16:52
KinnisonThough monday night will suck16:52
laurencetristan, nothing on monday16:53
tristanGotcha :)16:57
*** tristan has quit IRC17:01
*** raoul has quit IRC17:15
*** nimish has quit IRC17:15
*** nimish has joined #buildstream17:15
*** nimish has quit IRC17:20
*** nimish has joined #buildstream17:21
*** nimish has quit IRC17:36
*** nimish has joined #buildstream17:37
jennisDoesn't look like the CASCache refactor took into account `bst-artifact-server`17:44
gitlab-br-botaevri opened MR !1078 (aevri/shell_separator_hint->master: cli.py: add a hint about '--' to 'bst shell' help) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107817:46
gitlab-br-botjennis opened issue #867 (`bst-artifact-server` no longer works) on buildstream https://gitlab.com/BuildStream/buildstream/issues/86717:49
*** nimish has quit IRC17:52
*** nimish has joined #buildstream17:52
*** lachlan has quit IRC17:52
jennisAhh, fixed it17:56
jennisI think17:56
jenniswow, how did I do that17:56
*** nimish has quit IRC17:57
*** jonathanmaw has quit IRC18:00
jennisQuick review for anyone that's up for it: https://gitlab.com/BuildStream/buildstream/merge_requests/107918:00
jmacApproved18:01
jmac(Take it out of WIP, though!)18:01
gitlab-br-botjennis opened (was WIP) MR !1079 (jennis/fix_bst_artifact_server->master: Ensure that we can execute `bst-artifact-server`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107918:02
jennisDone18:02
jennisSet to merge18:02
jmacThanks!18:02
jennisPerhaps we should have a test to ensure this command returns a help message or something18:02
*** lachlan has joined #buildstream18:16
*** lachlan has quit IRC18:29
gitlab-br-botjennis closed issue #867 (`bst-artifact-server` no longer works) on buildstream https://gitlab.com/BuildStream/buildstream/issues/86718:33
gitlab-br-botjennis merged MR !1079 (jennis/fix_bst_artifact_server->master: Ensure that we can execute `bst-artifact-server`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107918:33
*** toscalix has quit IRC18:53
*** lachlan has joined #buildstream19:12
*** oknf[m] has left #buildstream19:13
*** tristan has joined #buildstream19:25
*** lachlan has quit IRC19:37
*** xjuan has joined #buildstream19:43
*** lachlan has joined #buildstream20:05
gitlab-br-bottristanvb closed issue #712 (BuildStream spends a long time pulling/looking to pull before doing anything useful) on buildstream https://gitlab.com/BuildStream/buildstream/issues/71220:10
gitlab-br-bottristanvb merged MR !1067 (tristan/element-processing-order->master: Scheduler refactor, fix processing order) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/106720:10
gitlab-br-bottristanvb opened MR !1080 (tristan/cleanup-staging-tempdir->master: element.py: Cleanup temporary staging directories on termination) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108020:18
gitlab-br-bottristanvb opened MR !1081 (tristan/error-message-regression->master: sandbox/sandbox.py: Display failed commands in the detail string) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108120:19
*** lachlan has quit IRC20:49
gitlab-br-bottristanvb merged MR !1080 (tristan/cleanup-staging-tempdir->master: element.py: Cleanup temporary staging directories on termination) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108021:17
*** tristan has quit IRC21:27
*** tristan has joined #buildstream21:46
*** bochecha has quit IRC22:05
*** bochecha has joined #buildstream22:06
*** tristan has quit IRC22:39
*** johnward has quit IRC22:40
*** benbrown has quit IRC22:41
*** paulsherwood has quit IRC22:41
*** paulsherwood has joined #buildstream22:41
*** benbrown has joined #buildstream22:41
*** johnward has joined #buildstream22:41
*** WSalmon has quit IRC22:42
*** mablanch has quit IRC22:42
*** WSalmon has joined #buildstream22:42
*** mablanch has joined #buildstream22:43
*** tristan has joined #buildstream22:59
*** bochecha has quit IRC23:01
valentindI really do not understand test tests/integration/workspace.py::test_workspace_update_dependency_failed23:03
valentindI think it has passed until now by chance only.23:04
valentindAre all files from a workspace supposed to be older than staged dependencies?23:05
valentindAh the file is not supposed to change.23:07
valentindIt is. Just mtimes are supposed to be updated.23:19
*** ChanServ sets mode: +o tristan23:44
tristanLooks like the bzr plugin doesnt handle parallelism that well, I'm seeing the "atomic" moves not being very atomic23:45
*** tristan has quit IRC23:49
*** xjuan has quit IRC23:49
*** tristan has joined #buildstream23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!