IRC logs for #buildstream for Friday, 2019-03-22

*** swick has quit IRC00:43
*** nimish2711 has joined #buildstream01:27
*** nimish2711 has quit IRC02:55
*** nimish2711 has joined #buildstream02:58
*** nimish2711 has quit IRC03:08
*** nimish2711 has joined #buildstream03:08
*** nimish2711 has quit IRC03:19
*** swick has joined #buildstream03:51
*** mohan43u has joined #buildstream04:07
*** cs-shadow has quit IRC04:31
*** cs-shadow has joined #buildstream04:31
*** benschubert has quit IRC04:31
*** benschubert has joined #buildstream04:31
*** csoriano has quit IRC04:31
*** persia has quit IRC04:32
*** persia has joined #buildstream04:32
*** csoriano has joined #buildstream04:39
*** csoriano has quit IRC04:40
*** csoriano has joined #buildstream04:40
*** tristan has quit IRC05:06
*** tristan has joined #buildstream05:28
*** tristan has quit IRC07:22
*** toscalix has joined #buildstream08:16
*** toscalix has joined #buildstream08:18
*** tristan has joined #buildstream08:55
*** rdale has joined #buildstream09:21
*** raoul has joined #buildstream09:43
*** tpollard has joined #buildstream09:44
gitlab-br-botBenjaminSchubert opened MR !1254 (bschubert/optimize-dependencies->master: Rework Element.dependencies to be more efficient) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125409:49
benschuberttristan: ^ you might be interested09:49
*** toscalix has quit IRC09:59
*** mohan43u has quit IRC10:00
*** toscalix has joined #buildstream10:03
*** mohan43u has joined #buildstream10:04
*** mohan43u has quit IRC10:08
*** jonathanmaw has joined #buildstream10:09
*** mohan43u has joined #buildstream10:14
*** mohan43u has quit IRC10:24
*** mohan43u has joined #buildstream10:27
gitlab-br-botBenjaminSchubert opened MR !1255 (bschubert/profiler-as-cm->master: Cleanup profiler and make is as a context manager) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125510:36
*** lachlan has joined #buildstream10:40
*** toscalix has quit IRC10:48
*** toscalix has joined #buildstream10:53
*** csoriano has quit IRC10:59
*** mohan43u has quit IRC10:59
*** csoriano has joined #buildstream11:00
*** csoriano has joined #buildstream11:01
*** mohan43u has joined #buildstream11:03
*** nimish2711 has joined #buildstream11:03
gitlab-br-bottristanvb opened MR !1256 (tristan/backport-update-state-changes-1.2->bst-1.2: Tristan/backport update state changes 1.2) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125611:08
*** gokcennurlu_ has joined #buildstream11:11
*** mohan43u has quit IRC11:12
*** mohan43u has joined #buildstream11:16
benschubertjuergbi: there is mention in the profiler and https://docs.buildstream.build/CONTRIBUTING.html#profiling-the-artifact-cache-receiver of being able to profile the cas server. However the relevant code is not in the casserver.py. Is this a mistake or should the profile and the docs not exist anymore?11:17
*** ChanServ sets mode: +o tristan11:17
tristanjuergbi, raoul, I know you're waiting for feedback on !1214; I think I did have preference of queue separation in the past, and I still think it makes better sense than baking pushes into the fetch queue11:17
gitlab-br-botMR !1214: Remote source cache https://gitlab.com/BuildStream/buildstream/merge_requests/121411:17
tristanI might be overlooking some things though ?11:17
juergbibenschubert: this is a leftover from the pre-CAS artifact server11:18
benschubertjuergbi: so I can safely remove correct?11:18
juergbiyes. it might make sense adding profiling support in the future but let's just remove the leftovers from now11:19
juergbi(especially considering we might move to casd)11:19
benschubertthanks11:19
tristanjuergbi, I think separating the tasks is more inline with our architecture in general, and one benefit I can see right away is that we should not be blocking builds which can commence after a fetch completes because the fetch is spending time pushing to a remote11:19
raoultristan, The remaining points were what to name the new source push queue in the user interface, and whether to have the queue after the fetch stage or after artifact push.11:20
juergbitristan: yes, separation is also how it's implemented right now. and generally does make sense11:20
juergbibut to avoid blocking build, we need to reorder the queues. right now it's between fetch and build11:20
tristanraoul, I see, I thought juergbi linked me in specifically to discuss whether it should be separated11:20
juergbiit should probably be between build and [artifact] push, or after [artifact] push11:20
juergbitristan: that is/was one possible alternative11:21
tristanI see11:21
juergbijust wanted to mention the possibilities11:21
KinnisonOur WIP MR for the YAML rework is now posted.  jennis and I would appreciate tactful eyes on it - there's still plenty of TODOs for us to resolve, but it'll be interesting to see what others thing over-all11:21
tristanThat seems to be a bit flawed11:21
tristanjuergbi, Maybe in some future, queues could be changed to not be a flat list, and orthogonal queues could occur ?11:22
juergbiyes, that's a limitation with the current scheduler11:22
juergbihowever, I think we don't want to block the branch on lifting this restriction11:22
tristanit's not a huge change to implement11:22
tristanno no11:22
tristanI only mean, it's less of a concern if we consider that change on the roadmap11:22
juergbisure11:23
benschuberttristan: I'm looking in reworking how we handle the queues11:23
tristanyeah :)11:23
benschubertso that problem would go away11:23
juergbithe remaining blocking questions are mainly about the (UI) name of the new queue, to avoid confusion with artifact push11:23
tlater[m]We're seeing an interesting performance trend for master: https://buildstream.gitlab.io/buildstream-benchmarking/benchmarks/public/average_time_master_Build_of_Baserock_stage1-binutils_for_x86_64_20190322-023246.html11:24
tlater[m]We aren't quite sure if this is a measurement error or a real effect11:24
tlater[m]Does anyone know what features might be responsible for that trend?11:24
*** alatiera has joined #buildstream11:25
KinnisonWe recently got a few performance improvements merged I thought11:25
Kinnisonbenschubert: ^^ ?11:25
benschuberttlater[m]: trend being buildstream becomes more performant? (Am I reading the graph correctly?)11:25
tlater[m]benschubert: Yup11:25
benschubertyep, that's improvment from our performance work11:25
tlater[m]We're just not sure why it's that big of an improvement all of the sudden11:25
benschubertglad to see it's noticeable11:26
tlater[m]Cool I'll just take your word for it. Thanks benschubert :)11:26
tristantlater[m], I have a hard time believing that the optimizations improve performance that much for such a small set of elements11:26
tristanare those minutes ?11:26
tlater[m]tristan: I'm not sure11:26
tlater[m]I haven't looked much into how the graphs work yet11:26
tristanI mean, we're really talking about something like < 20 elements11:26
tlater[m]lachlan?11:26
juergbithe staging improvements might also have had an impact here, difficult to say without more details11:26
benschubert^ I second this11:26
Kinnisonthe first drop might be sourcecache populating11:26
lachlanUnits are seconds11:26
Kinnisonand the second it in use?11:26
tristanstaging improvements are more likely11:26
tlater[m]i don't think we have more details atm, unfortunately11:27
juergbiand if the fetch job is not part of the performance test, source cache should also have provided a speedup11:27
juergbi(in staging sources)11:27
Kinnisontlater[m]: If you want to review a WIP MR, https://gitlab.com/BuildStream/buildstream/merge_requests/1257 would enjoy some love11:27
tlater[m]I'd love to write some more specific tests so we can tell what causes these improvements |:11:28
tlater[m]Kinnison: I was about to have a look, actually :)11:28
benschuberttlater[m]: that would be nice11:28
jennispretty sure those benchmarks mount in a prepopulated source cache11:28
KinnisonWhile I was expecting some improvement, I think tlater[m] will be pleased with the numbers in the MR description11:28
tlater[m]Yes, they do, to avoid benchmarking network latency11:28
benschuberttristan: if you have time also, !1254 is the rework for Element.dependencies11:29
gitlab-br-botMR !1254: Rework Element.dependencies to be more efficient https://gitlab.com/BuildStream/buildstream/merge_requests/125411:29
juergbijennis, Kinnison: if this only reuses .cache/buildstream/sources and not CAS source cache, then source cache is expected to have a net negative effect (for now), though11:30
Kinnisonaah11:34
tristanbenschubert, Have been glancing at it11:35
*** lachlan has quit IRC11:35
tristanbenschubert, you might also take a glance at https://gitlab.com/BuildStream/buildstream/merge_requests/125611:36
benschuberttristan: gah, seems like we would need to update our gitlab-ci.yml for this, seems the old runners are gone11:36
tristanbenschubert, at a high level I dont have issues with the Element.dependencies() change - only I want to be sure that the behavior is the same11:37
tristanold runners gone ?11:37
* tristan doesnt get it11:37
tristanoh crap11:37
benschuberttristan: I'm pretty sure the behavior stayed the same since we pass all the tests11:37
tristaneverything is broken on gitlab ?11:37
tristanyanked our machines ?11:38
tristanhehe11:38
benschubertfor the old runners, it means the docker images11:38
benschubertwe changed the ways we were handling docker images for running tests, and apparently removed the previous ones11:38
tristanThey just went away ? I recall there was talk of reorganizing the runners11:38
benschubertcs-shadow: do you have more context about the runners?11:38
tristanbut didnt make sure the CI was transitioned before deleting stuff ?11:38
tristanSeems easily reparable11:39
benschubertyep, updating the gitlab-ci.yml to point to the new images should do it11:39
benschubertI don't recall any other changes11:39
cs-shadowah! this is 1.2. I remember updating it on the master branch, seems like 1.2 fell off the radar11:41
tristanbenschubert, I think the cache key test *should* be enough to ensure the order of dependencies is stable, however I don't believe that test is11:41
cs-shadowi'll put in a MR in a moment11:41
tristanbenschubert, i.e. we currently try to cover every feature (usually a yaml construct) in small test cases and ensure that every possible configuration of anything will result in the same key11:43
tristanAnd then we have a quite shallow project and `bst show` it11:43
benschuberttristan: not sure I understand your last comment. I meant for the Element.dependencies() changes, the test suite should cover all possible case so the result should bt stable11:43
tristanwhat it doesnt do, is construct a deep dependency graph11:43
tristanbenschubert, That's what I'm not sure of - I'm talking about the cache key test because I think it's most likely to catch changes in dependency graph walking orders11:44
tristan(since dependency order is still relevant in cache keys, but might not be if we change that as we discussed, as an optimization)11:45
tristanOther than that, it is mostly relevant in staging order11:45
tristanSo overlap tests should cover it a bit11:45
benschubertthe cache keys tests were improved when working on loading rework, so I would think they are pretty much covering everythign currently :)11:46
tristanThis actually reminds me11:46
gitlab-br-botcs-shadow opened MR !1258 (chandan/fix-ci-1.2->bst-1.2: .gitlab-ci.yml: Update CI to use images from GitLab Registry) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125811:47
tristanphil, you are working on making sure external plugins have good support from BuildStream for the purpose of testing correct ?11:47
tristanI have mixed some people up last week, hope I am not guessing the wrong person :)11:47
cs-shadowcan someone please sanity check ^^ before I hand it over to marge? :)11:48
*** alatiera_ has joined #buildstream11:48
*** alatiera has quit IRC11:48
tristanphil, I think we have a good framework with the update.py script and cache key test harness, and we probably want to expose that in a sane way for external plugins to test for cache key breakages11:48
*** alatiera_ is now known as alatiera11:49
gitlab-br-bottristanvb approved MR !1258 (chandan/fix-ci-1.2->bst-1.2: .gitlab-ci.yml: Update CI to use images from GitLab Registry) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125811:49
tristancs-shadow, looks good to me11:49
cs-shadowtristan: thanks!11:49
*** lachlan has joined #buildstream11:53
*** alatiera has quit IRC11:58
*** lachlan has quit IRC12:04
philtristan, You are correct. I'm not really familiar with our cache key tests, but from a brief look I think I agree that it would nice to expose them. I'll open an issue to that effect.12:05
tlater[m]Kinnison: The timing results definitely look promising - but a single commit that touches 105 files? :|12:05
* tlater[m] rolls up his sleeves12:05
tristanphil, Thanks !12:06
jennisheh, tlater[m], we were following the commit policy12:06
tristanphil, it is pretty simple once you look at it :)12:06
jennis"all commits must pass the tests", or something along those lines12:06
tristanHah12:06
tlater[m]This sounds like a good situation in which to break that policy x)12:07
KinnisonNot my call :D12:07
* Kinnison -> hospital for to have mould stuffed into a hole in his arm :D12:07
*** lachlan has joined #buildstream12:07
juergbitristan: thanks for the comment with regards to the scheduler. any opinion on the names for the two push queues? https://gitlab.com/BuildStream/buildstream/merge_requests/1214/#note_15157026512:10
tristanI have worse :-S12:11
tristanjuergbi, Commented12:18
juergbita12:18
tristanjuergbi, We probably dont want to block on that comment, but it is quite annoying :-S12:18
tristanyou'll see heh12:18
*** lachlan has quit IRC12:18
juergbitristan: raoul did increase the max queue name length from 5 to 8 to allow src-push12:19
juergbi(in frontend widget)12:20
raoulYeah was just about to say, I didn't seem to mess anything else up from what I saw12:20
juergbione question is whether this will break anything. the other question is what to call the artifact push queue12:20
juergbiin the branch it's currently art-push but I don't like that too much12:20
juergbiwe could just leave it as push but that could be confusing12:21
raoultristan, for reference made this asciinema of it working https://asciinema.org/a/OIObuXAhwhYi39nXQWkXAJkUW12:21
* juergbi would keep 'fetch' instead of 'src-pull'12:23
juergbiif we wanted to change that, we should consistently change it also for commands and documentation. but I don't think we want that, at least not in this MR12:23
juergbiand in the pipeline summary I wouldn't use any abbreviations12:24
juergbiSKIPPED Art-pull also seems odd. why do we abbreviate that?12:24
gitlab-br-botBenjaminSchubert approved MR !1258 (chandan/fix-ci-1.2->bst-1.2: .gitlab-ci.yml: Update CI to use images from GitLab Registry) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125812:25
raoulThe pipeline summary just uses the same name variables, I can extend it to be Artifact Pull etc.12:25
raoulsame for the SKIPPED12:26
raoulprobably just need to add a full_name to queues and use that where appropriate12:26
tristan(A) agree with juergbi that I would keep 'fetch' instead of 'src-pull'... (B) I don't mind 'art-pull' so much, but it would be nice to have long/short names now that it is becoming more important.. (C) I personally would not want to block this technical branch *too* much on UI improvements12:28
tristanit's probably a better strategy to do concentrated efforts on improving the UI separately from concentrated efforts in the core12:29
juergbimakes sense12:29
gitlab-br-botcs-shadow approved MR !1254 (bschubert/optimize-dependencies->master: Rework Element.dependencies to be more efficient) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125412:29
juergbiwondering whether we should just go with a new src-push queue for this MR and leave all existing queue names the same for now12:30
tristanI would be happy with that12:30
juergbithat way we reduce the risk of renaming the same queue multiple times12:30
tristanSad to increase the minimum length to 812:31
tristanthat screen realestate is precious12:31
raoulcool, I can change some of the names back and then I think that's ready to be merged?12:32
raoul(haven't push removing the push queue from source fetch and moving the source push queue after the build queue)12:33
*** alatiera has joined #buildstream12:41
*** CTtpollard has joined #buildstream12:43
*** tpollard has quit IRC12:43
*** tristan has quit IRC12:44
*** CTtpollard has quit IRC12:49
gitlab-br-botmarge-bot123 merged MR !1258 (chandan/fix-ci-1.2->bst-1.2: .gitlab-ci.yml: Update CI to use images from GitLab Registry) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125812:50
gitlab-br-botmarge-bot123 merged MR !1254 (bschubert/optimize-dependencies->master: Rework Element.dependencies to be more efficient) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125413:00
*** tristan has joined #buildstream13:16
*** alatiera has quit IRC13:37
*** alatiera has joined #buildstream13:41
*** nimish2711 has quit IRC13:48
*** nimish2711 has joined #buildstream13:51
*** tpollard has joined #buildstream14:12
gitlab-br-botBenjaminSchubert closed issue #864 (Local cache doesn't cleanup properly) on buildstream https://gitlab.com/BuildStream/buildstream/issues/86414:12
*** tpollard has quit IRC14:12
*** nimish2711 has quit IRC14:24
jjardoncs-shadow: when we talk about the deprecation of the docker images, we have always said they will be eventually removed, but not a specific date (I would imagine at least a year minimum) seems now they are removed and we are breaking the CI in some projects14:29
*** tpollard has joined #buildstream14:29
jjardonWhy the rush on removing them?14:29
*** nimish2711 has joined #buildstream14:30
*** mohan43u has quit IRC14:34
*** mohan43u has joined #buildstream14:37
*** lachlan has joined #buildstream14:46
raoulTristan, juergbi, I've pushed the remote source cache with just src-push name and everything else the same, (had to faff around with the tests that checked for names) and changed the PushQueue to ArtifactPushQueue, so I think it's ready to be merged14:49
raoulAfter I rebase onto master at least14:51
cs-shadowjjardon: I asked around about the dates in https://mail.gnome.org/archives/buildstream-list/2019-February/msg00047.html but didn't get any response so I assumed they were okay.14:53
cs-shadowThe reason for removing them is that it's not really recommended to use images that aren't getting updates.14:53
cs-shadowbuildstream-fedora should still be around until May as per the message, but I need to look into what happened to the older tags14:53
jjardoncs-shadow: agreed, but that's a problem to downstream to solve. We even broke our own project here (1.2 branch); eave the  images in dockerhub doesn't cost anything, no reason really to remove them without time to downstream to move to the new ones14:56
*** lachlan has quit IRC15:01
tlater[m]jjardon: tbf, this was announced a fair while ago. There was time to move to the new images.15:02
*** lachlan has joined #buildstream15:02
* tlater[m] imagines removing them later would just have broken downstream later15:03
benschubertand the test images were private to the project, so we are the only ones who should break15:04
jjardonAnd still we broke our own project :) I still argue that they should not be removed in a long long time15:04
jjardonbenschubert: not sure I understand, those images are public in dockerhub15:05
benschuberthttps://gitlab.com/BuildStream/buildstream-docker-images#testsuite-images jjardon15:06
tlater[m]jjardon: We've removed them from the public space specifically because they were supposed to be internal15:06
cs-shadowbst-1.2 was on oversight, for others, I am not sure if leaving it indefinitely is a better alternative. Personally, I'd be in favor of removing the image vs. shipping an image with vulnerabilities because it hasn't been rebuilt in ages15:08
jjardonRight, I guess buildstream-fedora images are not being removed then?15:08
cs-shadownot until May anyway :)15:09
cs-shadowfeel free to reply to the ML thread or https://gitlab.com/BuildStream/buildstream-docker-images/issues/13 if you think we should keep them around for longer15:09
tlater[m]Side note - because of this restructure we realized that the benchmarks were always testing buildstream master15:10
tlater[m]Because we were running on buildstream-fedora with a pre-installed buildstream15:10
jjardoncs-shadow: I didn't say keeping them forever, only saying removing them will break CI of projects we are not aware of, and that can make people not very happy15:10
*** lachlan has quit IRC15:11
* cs-shadow isn't sure how to avoid CI breakages of projects we don't know about15:11
jjardonMore if we take in account keeping them cost us literally nothing15:12
cs-shadowI disagree that it costs nothing. https://hub.docker.com/u/buildstream used to have two pages of images which made it harder for users browsing the page which one is relevant for their use-case15:13
jjardoncs-shadow: sure, but that's a documentation problem really15:15
jjardonBut still, give one month from the deprecation notice to actually removing the images from dockerhub is way too short in my opinion. Sorry I didn't said anything before, somehow I missed that email. BTW, this is in no means a criticism of the move itself! I think the organization of the images is much clear now15:20
jjardonI will reply to the email about buildstream-fedora one15:20
jjardonBut problem is not that much project can not update, but that they will not be able to build old releases anymore15:21
*** lachlan has joined #buildstream15:27
*** mohan43u has quit IRC15:33
*** tpollard has quit IRC15:49
*** lachlan has quit IRC15:52
*** lachlan has joined #buildstream15:54
*** lachlan has quit IRC16:03
*** lachlan has joined #buildstream16:12
gitlab-br-botraoul.hidalgocharman opened MR !1259 (raoul/965-AaaP-service->master: Artifact as a Proto: protos and service) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/125916:23
*** toscalix has quit IRC17:00
*** nimish2711 has quit IRC17:11
*** nimish2711 has joined #buildstream17:52
*** raoul has quit IRC17:59
*** jonathanmaw has quit IRC17:59
*** nimish2711 has quit IRC18:00
benschuberttristan: are you around? I'm trying to understand why "a.bst" is at this place for https://gitlab.com/BuildStream/buildstream/blob/master/tests/frontend/order.py#L76 (I would expect it to be first)18:06
*** alatiera_ has joined #buildstream18:12
*** alatiera has quit IRC18:12
*** alatiera_ is now known as alatiera18:13
benschubertnevermind, got it18:24
*** mablanch has quit IRC18:31
*** lachlan has quit IRC19:35
*** rdale has quit IRC19:57
*** alatiera has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!