| *** nimish has joined #buildstream | 01:38 | |
| *** nimish has quit IRC | 03:25 | |
| *** tristan has joined #buildstream | 03:29 | |
| *** ZULA has joined #buildstream | 07:38 | |
| *** ZULA has left #buildstream | 08:20 | |
| *** toscalix has joined #buildstream | 08:44 | |
| *** toscalix has quit IRC | 08:46 | |
| *** toscalix has joined #buildstream | 08:48 | |
| *** toscalix has quit IRC | 09:12 | |
| *** toscalix has joined #buildstream | 09:13 | |
| *** benschubert has joined #buildstream | 09:16 | |
| benschubert | Question about remote caches. when doing a "bst artifact push", the command line says "The default destination is the highest priority configured cache" Does that mean it's impossible to push to all caches? | 09:22 |
|---|---|---|
| *** ChanServ sets mode: +o tristan | 09:26 | |
| tristan | That is a good question, I'm not sure what the status of this is - there is (was ?) a problem where a downstream project is (was ?) unable to replicate the artifacts produced by an upstream (junctioned) project | 09:26 |
| tristan | I can say that the original design of multiple caches was for different use cases, we had it in mind that people would use separate artifact servers for long standing side branches, in which case we'd only want to push to the dev branch's server, while still falling back to pulling from the master server | 09:27 |
| tristan | I think that the configuration has evolved since then to have "push" options for each configured artifact server | 09:29 |
| tristan | and maybe the help output is outdated | 09:29 |
| benschubert | tristan: so it should push to everything correct? | 09:30 |
| tristan | Sorry for the lack of a straight answer :-S | 09:32 |
| tristan | lets look at what the config provides now | 09:32 |
| gitlab-br-bot | BenjaminSchubert closed issue #869 (Local cache size calculation is wrong) on buildstream https://gitlab.com/BuildStream/buildstream/issues/869 | 09:33 |
| tristan | User configuration can decide: https://docs.buildstream.build/using_config.html#artifact-server | 09:33 |
| benschubert | I'll look more into it. I've got troubles on the system and wanted to roll out hypothesis :) | 09:34 |
| tristan | And project configuration can provide recommendations: https://docs.buildstream.build/format_project.html#artifact-server | 09:34 |
| tristan | According to docs, project configuration cannot decide whether to push/pull | 09:35 |
| tristan | and it is not clear to me if this is by design | 09:35 |
| tristan | I.e. when building project foo, it does not make sense for it's junctioned dependency bar to decide that it is allowed to push | 09:35 |
| tristan | normally that will fail, as builders of project foo will lack credentials to push to dependency bar's server | 09:36 |
| tristan | yeah its a pretty tricky area, we need to clarify this; one thing is sure is that projects *recommend* artifact servers, but builders have the last word (i.e. user config) | 09:37 |
| benschubert | I was just confused, because "bst push" used to push to all available caches, and "bst artifact push" seems to only be able to push to one server | 09:37 |
| tristan | Oh that is strange, I don't see how the CLI change would be related to a behavioral change | 09:39 |
| benschubert | it's either a bug or a documentation error :) I'll look into that. It's sadly down in my list of current bugs | 09:41 |
| benschubert | the first one beinf, we still hit out of storage space errors when running BuildStream master | 09:41 |
| tristan | out of storage space errors cannot be avoided technically afaics, the best we can do is handle the out of space conditions gracefully | 09:42 |
| *** jonathanmaw has joined #buildstream | 09:42 | |
| tristan | We might want to make the "headroom" configurable, that would help avoid the situation | 09:42 |
| tristan | benschubert, i.e. the "headroom" is hard coded to 2GB right now, which means that if you dont have a limited quota setup, we try to cleanup if ever we observe there is less than 2GB remaining | 09:43 |
| benschubert | yes | 09:43 |
| benschubert | Second thing: the local cache cleanup took 38 hours on our system, the one time where it kicked in | 09:44 |
| tristan | We *could* technically have something declarative for an element to say how big it's temporary build directory would be, and how large of an artifact it would create (but that could still lie, so we'd still hit out of space conditions) | 09:44 |
| tristan | Yeah, that needs optimization for sure | 09:45 |
| tristan | it is horribly slow even for regular sized projects | 09:45 |
| tristan | I think juergbi and Kinnison were discussing different ref-based cleanup strategies | 09:45 |
| tristan | err, object based | 09:46 |
| Kinnison | We did, though I think it was juergbi and valentind who actually sorted things out | 09:46 |
| Kinnison | And partialcas will simplify/complicate matters (dunno which) | 09:46 |
| valentind | We did only for the server side. | 09:46 |
| Kinnison | is it not the same code in both cases? | 09:46 |
| tristan | Nope | 09:47 |
| valentind | There is some common code. | 09:47 |
| juergbi | BuildStream can't deal with partial artifacts in local CAS just yet | 09:47 |
| valentind | But it is a different strategy the the top level code is different. | 09:47 |
| juergbi | soon to be fixed | 09:47 |
| benschubert | So there is a plan to change how the cleanup works if I understand well (with the lcoal cas changes) correct? Or do I need to keep track of this? | 09:48 |
| tristan | It can be good to make sure there is already an issue tracking this | 09:49 |
| tristan | benschubert, I'm not worried that we forget, the cleanup is so obviously slow and painful, it's impossible to forget :) | 09:51 |
| *** raoul has joined #buildstream | 09:51 | |
| benschubert | ok | 09:52 |
| benschubert | in the meantime, I'll go back to wiping my entire cache every few runs | 09:52 |
| tristan | For smaller projects in the hundreds of elements, it takes a while to cleanup, but it doesnt happen so often and is at least less expensive than zapping (removing) the local cache | 09:54 |
| tristan | Well, assuming a sample project like freedesktop-sdk & gnome-build-meta, where there is not too much churn at the bottom of the stack | 09:54 |
| benschubert | We have a cache relatively close so that makes it less painful | 09:55 |
| tristan | (can take me 30min sometimes, which is less than downloading everything, and much much less than building everything) | 09:55 |
| *** phildawson has joined #buildstream | 10:12 | |
| gitlab-br-bot | phildawson opened (was WIP) MR !1215 (phil/consolidate-repo-tests->master: Consolidate templated source tests) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1215 | 10:35 |
| tristan | Anyone wanna take a look at !1216 ? | 10:36 |
| gitlab-br-bot | MR !1216: Improve error reporting when files are not found https://gitlab.com/BuildStream/buildstream/merge_requests/1216 | 10:37 |
| tristan | I'd like to include that, along with a fix for #919 in an upcoming bugfix release | 10:37 |
| gitlab-br-bot | Issue #919: 'bst build <elem>' does not assemble all required elements in some circumstances https://gitlab.com/BuildStream/buildstream/issues/919 | 10:37 |
| benschubert | tristan: do you want to include the fix form !1070 at the same time? I might have some time to work on it today | 10:38 |
| gitlab-br-bot | MR !1070: WIP: Refactor _update_state() to be called only when needed https://gitlab.com/BuildStream/buildstream/merge_requests/1070 | 10:38 |
| tristan | benschubert, I've been waiting for !1070 for exactly that reason yeah | 10:38 |
| benschubert | ah right, 919 is for this | 10:38 |
| benschubert | I'll probably probe you later today about it, have a few things to do first though | 10:39 |
| tristan | I'm mostly interested in having a clean and simple recursion algorithm for reverse dependencies | 10:40 |
| tristan | But anyway, I'm off to dinner and probably wont be around late | 10:40 |
| * tristan fixed his jetlag problem | 10:40 | |
| tristan | benschubert, I will be up earlier than everyone else tomorrow - and I am very willing to hammer out a version of !1070 which uses a hash to ensure that we can have a simpler algo which only stores direct reverse deps tomorrow afternoon | 10:45 |
| *** raoul has quit IRC | 10:45 | |
| tristan | But I won't do it unless you ask me, I don't want to steal your fire :) | 10:45 |
| tristan | anyway leave messages in the channel and I will pick them up when I wake up :) | 10:46 |
| *** raoul has joined #buildstream | 10:46 | |
| *** tristan has quit IRC | 10:56 | |
| *** tristan has joined #buildstream | 11:03 | |
| adds68 | when is the next release of bst-external happening? | 11:15 |
| adds68 | we need to merge/land https://gitlab.com/BuildStream/bst-external/merge_requests/74 ASAP for freedesktop-sdk | 11:15 |
| *** alatiera has joined #buildstream | 11:22 | |
| gitlab-br-bot | juergbi approved MR !1211 (jennis/move_node_get_project_path->master: Cleanup: Move _yaml.node_get_project_path() to Project._get_path_from_node()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1211 | 11:23 |
| * SotK hopes for "after https://gitlab.com/BuildStream/bst-external/merge_requests/71 gets merged too" | 11:23 | |
| benschubert | tristan: sure! I will see what I can do today and let you know where I'm at | 11:27 |
| gitlab-br-bot | marge-bot123 merged MR !1211 (jennis/move_node_get_project_path->master: Cleanup: Move _yaml.node_get_project_path() to Project._get_path_from_node()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1211 | 12:14 |
| raoul | I've just had a test under the unix pipeline have a temperamental test failure, where I've just retried the job and it's passed. This should be reason for concern right? | 12:16 |
| raoul | it's the tests/sandboxes/mounting/mount_simple.py::test_mount_proc, and you can see I've just retried it to get it to pass here: https://gitlab.com/BuildStream/buildstream/-/jobs/175225228 | 12:16 |
| *** raoul has quit IRC | 12:43 | |
| *** nimish has joined #buildstream | 13:17 | |
| *** raoul has joined #buildstream | 13:32 | |
| *** raoul has quit IRC | 14:03 | |
| *** raoul has joined #buildstream | 14:38 | |
| *** raoul has quit IRC | 14:38 | |
| *** jonathanmaw has quit IRC | 14:38 | |
| *** phildawson has quit IRC | 14:38 | |
| *** raoul has joined #buildstream | 14:39 | |
| *** phildawson has joined #buildstream | 14:39 | |
| *** jonathanmaw has joined #buildstream | 14:39 | |
| *** kapil___ has quit IRC | 14:51 | |
| benschubert | When storing data on a remote cache, I have a 1 to 4 ratio about disk space used (meaning a 1GB "build" would take 4GB on the remote cache as a mean), is that expected? I'm not caching build trees | 15:38 |
| juergbi | benschubert: no, bst-artifact-server doesn't use any compression, however, it should also not require substantially more space than the actual payload (metadata should not be that big) | 15:42 |
| juergbi | unless it has a huge number of tiny files, which would result in large Directory objects | 15:43 |
| benschubert | juergbi: any clues on how to investigate? | 15:47 |
| juergbi | benschubert: castool might help https://gitlab.com/jmacarthur/cas-inspector/ | 15:48 |
| jmac | That's very old now, I'd be surprised if it worked | 15:49 |
| juergbi | basic structure hasn't changed, though, iirc | 15:49 |
| juergbi | at least for read-only access | 15:49 |
| jmac | Yes, but the place we import GRPC from did iirc | 15:50 |
| gitlab-br-bot | raoul.hidalgocharman approved MR !1216 (tristan/missing-file-errors->master: Improve error reporting when files are not found) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1216 | 15:51 |
| benschubert | juergbi: I'll try this thanks | 15:51 |
| juergbi | benschubert: or more low level you could also try to find the largest blobs and check what they are | 15:52 |
| juergbi | maybe buildtrees are still accidentally cached | 15:53 |
| benschubert | juergbi: ok I'll look into this | 16:01 |
| *** ChanServ sets mode: +o tristan | 16:11 | |
| tristan | raoul, there are two known issues about spurious test failures (that I know of), one of them was introduced by juergbi parallelizing tests with `pytest .... -n 2` (still dont know why it sometimes crashes one of the threads) and the other has to do with mounts afaik | 16:11 |
| tristan | raoul, next time, better to post the link to the failed job than the successful one, but I think anyway in your case it is the mount failure and is unrelated to whatever patch you are testing | 16:12 |
| tristan | raoul, thanks for looking at !1216 btw :) | 16:13 |
| raoul | fair point, yeah I couldn't see it being caused by anything I'd done but felt it should be raised | 16:14 |
| raoul | nw :) | 16:14 |
| *** lachlan has joined #buildstream | 16:51 | |
| *** lachlan has quit IRC | 16:54 | |
| jennis | tristan, if you have some time, would you be able to revist https://gitlab.com/BuildStream/buildstream/merge_requests/1209 please? | 16:55 |
| *** toscalix has quit IRC | 16:55 | |
| *** toscalix has joined #buildstream | 16:56 | |
| tristan | jennis, dont think I visited that before but I'll check it first thing tmw | 16:59 |
| tristan | Kinnison, was talking about it I think | 16:59 |
| jennis | You've commented on it already ;) and thanks! | 17:01 |
| tristan | oh yeah I see and remember :) | 17:01 |
| tristan | cold cache | 17:01 |
| * tristan is really surprised if deep copies are faster than chainmap, and worries only that the replacement is shallow copies, which would "work under the majority of circumstances" but not be correct | 17:02 | |
| *** lachlan has joined #buildstream | 17:06 | |
| jennis | mhmm :/ can you think of any test we could write that would test this? | 17:06 |
| tristan | jennis, ummm, maybe something that runs bst track on an element/source which has deeply nested dicts, and asserting that only the ref changes as expected | 17:08 |
| tristan | jennis, shallow copies means only the dicts are copied and references to members are not, which leaves some margin of error if any dict is not recursively copied and stuff in there is modified | 17:09 |
| tristan | jennis, So deeply nested dicts which have project option conditionals in them or list directives | 17:09 |
| tristan | would in my estimation break if the copies are shallow | 17:10 |
| *** lachlan has quit IRC | 17:12 | |
| jennis | But we've replaced node_chain_copy() calls with node_copy() calls which is a recursive and deep copy, no? | 17:12 |
| tristan | I havent looked deeply enough to say :) | 17:13 |
| tristan | you would know better | 17:13 |
| tristan | jennis, however, I can say that I originally created ChainMap in order to replace python's deepcopy(), because it was really noticeably slower with large projects | 17:14 |
| tristan | where large projects are around 500 elements | 17:14 |
| tristan | lets not even talk about 1000 or 50K element super ginormous projects | 17:14 |
| jennis | I see, thanks for the context | 17:15 |
| jennis | So note that we've noticed a 20s improvement when loading 75k elements (admittedly, non-complicated elements) and none of the test suite breaking as a result | 17:16 |
| tristan | mhm, I'd be interested in seeing the sample elements | 17:17 |
| jennis | From what i'm inferring, it should like more complicated element declarations need to be tested with this change...? | 17:17 |
| jennis | tristan, sample elements in this project: https://gitlab.com/jennis/benchmark_debian | 17:17 |
| jennis | whoops | 17:17 |
| jennis | wrong one | 17:17 |
| jennis | This one: https://gitlab.com/jennis/debian-stretch-bst | 17:18 |
| jennis | They're all import elements which look like this: https://gitlab.com/jennis/debian-stretch-bst/blob/master/elements/base-files/base-files.bst for example | 17:18 |
| tristan | "Whoops, GitLab is currently unavailable. | 17:18 |
| tristan | " hehe | 17:18 |
| jennis | yeah it struggles to load the elements subdir :( | 17:19 |
| tristan | yeah, I guess it might be better to commit a generator script somewhere | 17:19 |
| * tristan will have to turn an eye to benchmarks one day | 17:19 | |
| tristan | we need to generate different shapes and sizes of graph | 17:20 |
| jennis | There is a nice patch from WSalmon which lets us do this | 17:20 |
| tristan | vertical, horizontal, mixed; and with randomized complexity of realistic project elements | 17:20 |
| jennis | If you're interested: https://gitlab.com/BuildStream/benchmarks/commit/2aff6ce964133a7358037ea18eb80f48728e11f9 | 17:21 |
| tristan | then run the benchmark 1000 times on different randomized samples of realistic elements, and grab the average of per-record times | 17:21 |
| tristan | one thing to note especially about import elements, is they have almost nothing to composite | 17:22 |
| tristan | i.e. their defaults are almost nil | 17:22 |
| jennis | This is true | 17:23 |
| tristan | using a mix of autotools/meson... real build elements, each with some overrides, will yield more realistic results | 17:23 |
| jennis | I will benchmark the path with gnome/freedesktop | 17:23 |
| jennis | patch* | 17:23 |
| *** lachlan has joined #buildstream | 17:40 | |
| *** lachlan has quit IRC | 17:49 | |
| *** lachlan has joined #buildstream | 17:53 | |
| gitlab-br-bot | jennis opened MR !1221 (jennis/assert_composition_failure->master: Add tests to ensure that overwriting on subsequent compositions does not fail) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1221 | 18:00 |
| *** phildawson_ has joined #buildstream | 18:28 | |
| *** phildawson has quit IRC | 18:28 | |
| *** raoul has quit IRC | 18:28 | |
| *** alatiera has quit IRC | 19:13 | |
| *** phildawson_ has quit IRC | 19:16 | |
| *** jonathanmaw has quit IRC | 19:28 | |
| *** nimish has quit IRC | 19:42 | |
| *** toscalix has quit IRC | 20:13 | |
| *** tristan has quit IRC | 20:55 | |
| *** lachlan has quit IRC | 21:04 | |
| benschubert | Tristan: haven't got time to look at the update_state today. Will do tomorrow unless you already did it, keep me updated | 21:57 |
| gitlab-br-bot | BenjaminSchubert opened MR !1222 (bschubert/linter-for-tests->master: Enable pylint on the tests) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1222 | 22:02 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!