| *** narispo has quit IRC | 02:14 | |
| *** narispo has joined #buildstream | 02:14 | |
| *** traveltissues has joined #buildstream | 06:32 | |
| *** traveltissues has quit IRC | 06:35 | |
| *** traveltissues has joined #buildstream | 06:47 | |
| *** traveltissues has quit IRC | 06:51 | |
| *** traveltissues has joined #buildstream | 07:47 | |
| *** santi has joined #buildstream | 09:43 | |
| *** santi has quit IRC | 09:47 | |
| *** santi has joined #buildstream | 09:51 | |
| *** santi has quit IRC | 09:51 | |
| *** santi has joined #buildstream | 09:54 | |
| *** tiagogomes has joined #buildstream | 09:58 | |
| *** phildawson_ has joined #buildstream | 10:08 | |
| *** bochecha has joined #buildstream | 10:19 | |
| *** lachlan has joined #buildstream | 10:29 | |
| benschubert | tlater[m]: you were looking at the benchmarking stuff a bit right? Is it 'available' for PRs? :) | 10:39 |
|---|---|---|
| *** lachlan has quit IRC | 10:40 | |
| tlater[m] | benschubert: Not sure what benchmarking stuff you mean | 10:42 |
| tlater[m] | Do you mean the bot? | 10:42 |
| benschubert | yes ! | 10:43 |
| tlater[m] | lachlan would know more, seems he's not here right now - the bot is still heavily WIP, might or might not be currently up, let me see if I can figure out whether it'd work right now :) | 10:43 |
| benschubert | thanks! | 10:43 |
| *** lachlan has joined #buildstream | 10:44 | |
| tlater[m] | ah, lachlan, we were just talking about you :D | 10:44 |
| benschubert | o/ | 10:49 |
| lachlan | benschubert: Which MR would you like to benchmark? | 10:49 |
| benschubert | lachlan: I haven't made it yet, give me a minute :) | 10:50 |
| lachlan | benschubert: If you can let me know when it is created and the buildstream pipeline completes successfully. | 10:51 |
| benschubert | sure, thanks :) | 10:51 |
| *** cs-shadow has joined #buildstream | 10:52 | |
| gitlab-br-bot | BenjaminSchubert opened MR !1755 (bschubert/optimize-scheduling->master: scheduler.py: Optimize scheduling by not calling it unnecessarily) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1755 | 10:54 |
| *** santi has quit IRC | 11:15 | |
| *** santi has joined #buildstream | 11:15 | |
| benschubert | lachlan: oh thanks! | 11:29 |
| coldtom | what is the recommended way to say "don't push during this build" these days? | 11:48 |
| tlater[m] | coldtom: afaik it's still --pushers 0 | 11:48 |
| *** lachlan has quit IRC | 11:49 | |
| coldtom | ah, that makes this job slightly concerning: https://gitlab.com/celduin/burn-sdk/-/jobs/371572011 | 11:53 |
| coldtom | (build is run with --pushers 0) | 11:54 |
| coldtom | on a semi-related note, is there any plan to specify some elements should not be pushed? e.g. I don't want my remote cache to be full of enormous top-level artifacts | 12:00 |
| persia | It's hard to have a sensible default for that. I have a use case where I mostly only care about having enormous top-level artifacts in my cache. | 12:02 |
| coldtom | sure, i meant a switch that can be put somewhere | 12:02 |
| coldtom | with default to push thigns | 12:02 |
| coldtom | maybe in element config a "push: false" | 12:02 |
| tlater[m] | coldtom: You mean something like `bst push --except`? | 12:03 |
| tlater[m] | Oh, I see | 12:03 |
| tlater[m] | No plans for that yet | 12:03 |
| tlater[m] | I think the current suggestion for that would be to disable pulling and push what you want manually | 12:03 |
| persia | coldtom: Any suggestions on "somewhere"? I don7t think it belongs in the project config, because differnt users of the same project may have different goals. I am also not sure it belongs in user config, because managing that for many projects quickly gets unwieldy. | 12:03 |
| coldtom | it would also be useful for if you need to depend on proprietary blobs you can't cache | 12:03 |
| benschubert | coldtom: I woul dbe very careful about putting that in the element config, because downstream people might be wanting different things. If we ever implement that I'd rather have that at user's config level | 12:03 |
| tlater[m] | But I agree that's a little cumbersome if it's a small set of elements you want to avoid pushing. | 12:03 |
| tlater[m] | benschubert: project config level could work too; users can always override that, and I suspect specific projects will dislike pushing certain things, rather than specific users. | 12:04 |
| benschubert | If you want to avoid pushing big artifacts: putting an nginx in front of your cache that only accepts requests <100Mo is one way, you'll always try to push 100Mo though | 12:04 |
| persia | (another viewpoint here is that of the cache administrator, who may have different costs/capacities/whatever for different size artifacts) | 12:04 |
| tlater[m] | Maybe project+artifact cache config | 12:05 |
| benschubert | one reason I could see this valid though is for proprietary blobs you cannot redestribute | 12:05 |
| benschubert | and those you would _never_ want them in a cache in the open | 12:05 |
| * tlater[m] thinks project + cache config therefore makes a lot of sense | 12:06 | |
| tlater[m] | So you can push your element to the local non-public cache | 12:06 |
| tlater[m] | But avoid throwing it into the open cache :) | 12:06 |
| persia | benschubert: Ooh. That's exciting. | 12:08 |
| benschubert | And currently, it would be a huge problem unless you use 'lcoal files', but there is a thread on the ML that wants to change this handling which would force us to have to consider this use case. Also, what about RE? :'D | 12:10 |
| tlater[m] | Oh, right, RE will push things to all kinds of places :| | 12:11 |
| persia | To my mind, RE implies everything is pushed to cache. | 12:11 |
| persia | But that might not be a public cache (depends on how the RE is hosted) | 12:11 |
| benschubert | Correct, then should we have a provision to prevent RE on some servers for some elements? It might become quickly cumbersome | 12:11 |
| gitlab-br-bot | traveltissues opened (was WIP) MR !1753 (traveltissues/remove-unused-functions->master: remove unused functions 1/2) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1753 | 12:12 |
| persia | I'm not sure how BuildStream could differentiate between RE-with-private-cache and RE-with-publlic-cache | 12:12 |
| persia | It's just an endpoint. Same interface either way. | 12:12 |
| coldtom | perhaps this config could be in cache configuration | 12:13 |
| tlater[m] | coldtom: Yeah, I think that'd be best | 12:14 |
| benschubert | juergbi: https://gitlab.com/BuildStream/buildstream/merge_requests/1756 I would still need to do some benchmarks, but I believe we should put this in to follow Python's documentatino | 12:17 |
| persia | coldtom: My concern is that the accessibility of a network cache can change without notice to the client. | 12:17 |
| juergbi | benschubert: haven't reviewed it in detail but makes sense. I don't think it's relevant for the current child watcher implementation (doesn't use threads, afaik) but we should follow the API spec | 12:21 |
| *** bochecha has quit IRC | 12:22 | |
| *** bochecha has joined #buildstream | 12:22 | |
| * tlater[m] agrees, assuming benchmarks don't show this is significantly slowr | 12:23 | |
| *** lachlan has joined #buildstream | 12:25 | |
| benschubert | tlater[m]: agreed, guess I'll have to do some heavy benchmarking in the next days :'D | 12:26 |
| benschubert | lachlan: https://gitlab.com/BuildStream/benchmarking/benchmarks/-/jobs/371620231 doesn't seem good? :'D | 12:27 |
| lachlan | Historical issues with the stable release and baserock build, expected | 12:29 |
| benschubert | oh :) good, thanks! | 12:29 |
| *** traveltissues has quit IRC | 12:32 | |
| *** santi has quit IRC | 13:14 | |
| *** santi has joined #buildstream | 13:51 | |
| lachlan | benschubert: Benchmark complete, MR updated | 14:08 |
| lachlan | I will be taking the bot offline though | 14:08 |
| *** lantw44 has quit IRC | 14:09 | |
| benschubert | lachlan: oh thanks! | 14:09 |
| lachlan | At the minute you will need to pick through the CSV file for the meaningful results | 14:10 |
| benschubert | ok! thanks :) | 14:13 |
| benschubert | mmh it seems to be 2 seconds slower :( | 14:13 |
| *** lachlan has quit IRC | 14:16 | |
| *** phildawson-ct has joined #buildstream | 14:20 | |
| tlater[m] | benschubert: Is that 2 seconds on the build? | 14:21 |
| benschubert | seems like it | 14:21 |
| * tlater[m] wonders if we can sacrifice 2 seconds for that | 14:21 | |
| tlater[m] | It's like a 0.5% increase, isn't it? | 14:21 |
| benschubert | the benchmarked PR was !1755 not the other one :( | 14:21 |
| gitlab-br-bot | MR !1755: WIP: scheduler.py: Optimize scheduling by not calling it unnecessarily https://gitlab.com/BuildStream/buildstream/merge_requests/1755 | 14:21 |
| tlater[m] | Ah | 14:21 |
| benschubert | so I _expected_ an optimization on this one for builds | 14:21 |
| *** phildawson_ has quit IRC | 14:21 | |
| benschubert | I still want to do my own testing there | 14:22 |
| tlater[m] | It's not a bad idea. We don't do warmup runs on the bot atm, so the variance is a little larger than what you're probably used to | 14:22 |
| tlater[m] | Yeah, standard deviation is about a second, so it might just be outliers | 14:23 |
| benschubert | and only 4 builders right? | 14:26 |
| benschubert | I'm more interested in 8-16-32-64 :D | 14:27 |
| * tlater[m] doesn't know about the builders | 14:28 | |
| *** lachlan has joined #buildstream | 14:54 | |
| *** bochecha has quit IRC | 15:06 | |
| *** bochecha has joined #buildstream | 15:06 | |
| *** lantw44 has joined #buildstream | 15:08 | |
| gitlab-br-bot | cs-shadow opened MR !1758 (chandan/remote-setuppy-test->master: Drop support for `setup.py test`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1758 | 15:22 |
| *** lantw44 has quit IRC | 15:24 | |
| *** lantw44 has joined #buildstream | 15:33 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1753 (traveltissues/remove-unused-functions->master: remove unused functions 1/2) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1753 | 15:35 |
| *** lachlan has quit IRC | 15:38 | |
| gitlab-br-bot | coldtom opened MR !1759 (coldtom/fix-junction-remotes->master: _project.py: Allow junctions to use parent remote) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1759 | 15:42 |
| *** santi has quit IRC | 15:43 | |
| *** lachlan has joined #buildstream | 15:56 | |
| *** bochecha has quit IRC | 16:04 | |
| *** lachlan has quit IRC | 16:15 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1759 (coldtom/fix-junction-remotes->master: _project.py: Allow junctions to use parent remote) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1759 | 16:16 |
| *** lachlan has joined #buildstream | 16:31 | |
| benschubert | tlater[m]: https://gitlab.com/BuildStream/buildstream/merge_requests/1755 look at the benchmarks, I need to rerun them on my desktop at home but... | 16:39 |
| *** lachlan has quit IRC | 16:40 | |
| tlater[m] | Oh, wow, that's a lot faster | 16:40 |
| benschubert | o/ | 16:41 |
| tlater[m] | I really wonder why the other benchmarks don't show that... | 16:41 |
| benschubert | that was on a VM though, I need real hardware to double check but I think that's one of the best perf improvement we got since february in terms of scheduling | 16:41 |
| tlater[m] | Especially because the parallelism seems to improve a fair bit :) | 16:42 |
| benschubert | Yep, I'll try 32 probably now :) | 16:43 |
| * tlater[m] wonders why the benchmarking bot's results don't show this | 16:44 | |
| tlater[m] | It might be less affected, iirc it's like a 2 core machine | 16:44 |
| benschubert | righ, I'm running on a 32 core one | 16:44 |
| *** santi has joined #buildstream | 16:52 | |
| *** phildawson-ct has quit IRC | 16:54 | |
| *** phildawson has joined #buildstream | 16:55 | |
| *** lantw44 has quit IRC | 17:16 | |
| *** lantw44 has joined #buildstream | 17:19 | |
| benschubert | coldtom: btw since you worked on the source tests, I sent an email to the ML, you might have other insights | 17:23 |
| *** lantw44 has quit IRC | 17:28 | |
| *** lachlan has joined #buildstream | 17:33 | |
| *** lantw44 has joined #buildstream | 17:38 | |
| *** lachlan has quit IRC | 17:42 | |
| *** rdale has quit IRC | 17:43 | |
| *** santi has quit IRC | 17:48 | |
| *** lachlan has joined #buildstream | 17:52 | |
| *** lachlan has quit IRC | 18:01 | |
| *** lachlan has joined #buildstream | 18:18 | |
| *** lachlan has quit IRC | 18:21 | |
| *** tiagogomes has quit IRC | 18:54 | |
| *** cs-shadow has quit IRC | 19:32 | |
| *** narispo has quit IRC | 21:49 | |
| *** narispo has joined #buildstream | 21:49 | |
| *** narispo has quit IRC | 21:58 | |
| *** narispo has joined #buildstream | 21:58 | |
| *** narispo has quit IRC | 22:01 | |
| *** narispo has joined #buildstream | 22:05 | |
| *** narispo has quit IRC | 23:09 | |
| *** narispo has joined #buildstream | 23:09 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!