| *** phoenix has joined #buildstream | 01:01 | |
| *** phoenix has quit IRC | 01:23 | |
| *** jward has quit IRC | 01:54 | |
| *** jward has joined #buildstream | 01:55 | |
| *** paulsherwood has quit IRC | 01:55 | |
| *** paulsherwood has joined #buildstream | 01:55 | |
| *** tristan has quit IRC | 02:34 | |
| juergbi | valentind, jjardon: seeing digitalocean rate limit errors in bastion logs: gitlab-runner[8629]: ERROR: Error creating machine: Error in driver during machine creation: GET https://api.digitalocean.com/v2/droplets/188887681: 429 Too many requests driver=digitalocean | 04:23 |
|---|---|---|
| juergbi | I think that's the root cause of our troubles. the droplets are already created but they aren't used because the rate limit error from the digitalocean API | 04:25 |
| juergbi | and with retries a single CI pipeline can then reach 100 droplets | 04:25 |
| juergbi | upgrading docker-machine might fix the issue | 04:28 |
| juergbi | gitlab maintains a fork at https://gitlab.com/gitlab-org/ci-cd/docker-machine | 04:28 |
| juergbi | which includes https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/commit/fa13d6b3a2188defc9dfa7ebfc2e4aebdc011418 and https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/commit/5050ac2b4f8256adc07ea888f4d355287e79c45e | 04:29 |
| *** hasebastian has joined #buildstream | 05:12 | |
| *** benschubert has joined #buildstream | 07:09 | |
| *** rdale has joined #buildstream | 08:02 | |
| benschubert | jjardon: may I take over https://gitlab.com/BuildStream/buildstream/-/merge_requests/1740/ ? Would be useful to test my implementation of https://mail.gnome.org/archives/buildstream-list/2020-April/msg00003.html | 08:03 |
| *** tpollard has joined #buildstream | 08:22 | |
| *** santi has joined #buildstream | 08:35 | |
| jjardon | benschubert: of course! | 08:51 |
| benschubert | cheers, thanks! I should be able to have something today :) | 08:52 |
| jjardon | juergbi: if we need more than 100 droplets there is something wrong going on; anyway I think someone is taking a look already | 08:52 |
| jjardon | benschubert: nice! :) | 08:52 |
| jjardon | juergbi: if not fixed, I will redeploy the whole bastion myself after work today, sorry pretty busy atm | 08:53 |
| juergbi | ok but we really want the fixes from the docker-machine fork, just redeploying bastion might not help. I should also have pinged benbrown, or is someone else taking a look already? | 08:57 |
| benbrown | Does need a redeploy really, but with the above gitlab fork | 09:01 |
| benschubert | juergbi: how long does it usually take to run the basic tests for you? (no --integration, no --plugins) | 09:04 |
| benschubert | (not on CI, locally) | 09:04 |
| juergbi | benschubert: 1238 passed, 157 skipped in 81.60s (0:01:21) | 09:06 |
| benschubert | what | 09:06 |
| benschubert | runing them all for me: (pytest -n 4) : 36 failed, 1089 passed in 5994s (1:39:54) | 09:07 |
| juergbi | eh, that's slower than CI job with integration | 09:08 |
| benschubert | yep | 09:08 |
| benschubert | tests/frontend/pull.py::test_build_remote_option takes 915s | 09:08 |
| benschubert | so 10 times the time to run all your tests | 09:08 |
| juergbi | 3.17s call tests/frontend/pull.py::test_build_remote_option | 09:08 |
| benschubert | I have 20 tests over 350s | 09:08 |
| juergbi | slowest test is: 36.44s call tests/frontend/large_directory.py::test_large_directory | 09:08 |
| benschubert | which python version? | 09:08 |
| juergbi | 3.7.6 | 09:09 |
| benschubert | ok I'm on 3.8 | 09:09 |
| benschubert | all the slow tests are in frontend/pull.py or frontend/remote-caches.py | 09:09 |
| benschubert | I dowloaded yesterday the statically compiled version of casd/fuse from buildstream's website | 09:10 |
| gitlab-br-bot | juergbi merged MR !1865 (juerg/cache-key->master: element.py: Fix strong cache key calculation in non-strict mode) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1865 | 09:10 |
| benschubert | oh -> "background threads are active" | 09:10 |
| benschubert | interesting | 09:10 |
| juergbi | benschubert: do you roughly know when it became that slow or has it always been like that for you? | 09:11 |
| benschubert | haven't run the tests locally for a few weeks | 09:11 |
| benschubert | so can't really say | 09:11 |
| juergbi | and it might be better to test on a branch that doesn't fail any tests | 09:11 |
| benschubert | however it seems to be related to python 3.8 | 09:12 |
| benschubert | I'm on master | 09:12 |
| benschubert | no changes at all | 09:12 |
| juergbi | I'm normally using my own local (non-static) builds of buildbox components, not the static builds | 09:12 |
| juergbi | however, I wouldn't expect a big difference | 09:12 |
| juergbi | I'm planning to upgrade to Python 3.8 relatively soon but don't have it installed at all right now | 09:13 |
| benschubert | OK I'll have a deeper look then, thanks! | 09:13 |
| juergbi | benschubert: but this is a recent CI run with python 3.8 with integration tests: 1378 passed, 15 skipped, 2 xfailed in 1254.53s (0:20:54) | 09:14 |
| *** tristan has joined #buildstream | 09:14 | |
| juergbi | that seems similar to other CI runs | 09:14 |
| benschubert | weird | 09:14 |
| juergbi | maybe the failing tests give you a hint what's wrong with the environment | 09:15 |
| benschubert | yep I'll dig, thanks :D | 09:15 |
| tpollard | we all need ryzen | 09:16 |
| juergbi | hehe, I don't think that's the issue here | 09:17 |
| juergbi | ryzen 4000 laptops look indeed nice for builds (and the ryzen 3000 desktop cpus anyway) | 09:19 |
| benschubert | I've got a 12 cores desktop with nvme ssd on which I'm trying, pretty sure the problem is elsewhere :'D | 09:21 |
| *** ChanServ sets mode: +o tristan | 09:23 | |
| tristan | So I was asked to look at !1867, which is actually an incarnation of !1002, and while looking at this, I see that #815 only appears to have patches for bst-1 | 09:23 |
| gitlab-br-bot | MR !1867: Handle grpc errors of type UNAVAILABLE and ABORTED https://gitlab.com/BuildStream/buildstream/-/merge_requests/1867 | 09:23 |
| gitlab-br-bot | MR !1002: WIP: Handle grpc errors of type UNAVAILABLE and ABORTED. https://gitlab.com/BuildStream/buildstream/-/merge_requests/1002 | 09:23 |
| gitlab-br-bot | Issue #815: Properly handle grpc exceptions https://gitlab.com/BuildStream/buildstream/-/issues/815 | 09:23 |
| tristan | So I wonder, are the retries already baked into master for these cases ? it doesnt appear so with a quick glance at casremote.py | 09:23 |
| juergbi | tristan: buildbox-casd (or rather, common) has some built-in retry functionality and all remote CAS access is going through buildbox-casd | 09:25 |
| tristan | I'm thinking, if it's already handled better in master and we're retrying at the optimal times, then we can just land the code in bst-1 and not care too much about the redundancy of how errors are handled spread out through the codebase | 09:25 |
| tristan | juergbi, I see | 09:25 |
| tristan | So while we might technically have such a scenario with buildbox-casd, it's insanely improbable | 09:25 |
| juergbi | it might still need work in bst master and/or buildbox, but we might not be able to usefully share code with bst-1 | 09:25 |
| tristan | iiuc | 09:26 |
| juergbi | yes, we obviously can't switch to buildbox-casd in bst-1 | 09:26 |
| juergbi | if we need some kind of generic gRPC retry infrastructure in bst master despite having buildbox, that infrastructure could probably be shared with bst-1 | 09:27 |
| tristan | Right of course, I'm just thinking... while looking at the patch a second time, the code still looks weird to me, my comment on !1002 applies | 09:27 |
| tristan | Like, why are we handling the same error with retries in the same way everywhere on the calling side if this is something frequent ? | 09:27 |
| tristan | But, if we're going to be retiring bst-1 anyway fairly soonish, I think we shouldn't block on any discussion around those lines | 09:28 |
| tristan | at least, I would be more concerned with writing more maintainable code in master | 09:28 |
| benschubert | tristan: agreed, I find the implementation weird, and think my comment on !1002 would be a good compromise between both (not sure you saw it?) | 09:29 |
| benschubert | right | 09:29 |
| tristan | benschubert, yeah I saw it indeed and agree | 09:29 |
| *** rdale has quit IRC | 09:29 | |
| tristan | I didn't write a replacement, and nobody appears interested in doing so, and I think there is not much point in insisting at this point | 09:30 |
| benschubert | fair | 09:31 |
| *** rdale has joined #buildstream | 09:32 | |
| * tristan comments and bounces the ball back to jjardon and valentind... | 09:36 | |
| benschubert | tristan: I'm working on allowing to have external plugins for some of our tests, and was wondering, do we want to keep the `test_cache_key` for external plugins like tar or not? | 09:38 |
| *** cs-shadow has joined #buildstream | 09:40 | |
| tristan | Good question, where to test these | 09:42 |
| tristan | We certainly want to have it tested either in BuildStream or where the tar plugin will live, and "do we need both" depends on whether modifications to BuildStream core can have an effect here | 09:42 |
| tpollard | the ostree cache key tests needs re-adding too | 09:43 |
| tristan | We certainly need at the very least, a test with 1 realistic source plugin | 09:43 |
| benschubert | Yeah, I like the idea of having it inside BuildStream, since we would not be breaking _core_, but still that would not be a nice workflow when changing the key for the plugin | 09:44 |
| tristan | so that we have cache keys calculated with a proper dict returned by Plugin.get_unique_key() | 09:44 |
| tristan | We certainly need these tests on the plugin repo side | 09:44 |
| tristan | At least for plugins which we continue to maintain and guarantee stability of | 09:44 |
| tristan | I'm still not deeply in touch with how plugins use buildstream.tests | 09:45 |
| tristan | Maybe I should try to document it to learn how it works and help iron out the kinks | 09:46 |
| *** phoenix has joined #buildstream | 09:48 | |
| benschubert | mmh the problem is that we have a single test that tests for both source and element plugins | 09:48 |
| benschubert | Ok, I'll move it to being run, as a `plugins` test, and we should also have a test _per plugin_ in the respective repos? | 09:48 |
| benschubert | Does that sound sensible? | 09:49 |
| benschubert | tristan: for plugins and buildstream.tests let me know if you have any questions, there will be 2 new MRs today hopefully that will change some things though | 09:49 |
| tristan | Does that mean BuildStream keeps testing them all, even after they migrate, and that the plugins also run the tests, but that the test continues to be maintained in one place ? | 09:50 |
| * tristan is really unsure how this works hehe | 09:50 | |
| benschubert | Re-reading the test it seems that we are doing too much | 09:52 |
| benschubert | Seems like I could split that | 09:52 |
| tristan | Could be, what we try to do is have a test which exercises each feature of each plugin | 09:52 |
| tristan | at least when there is new configuration added to an element or source which might affect cache key, we add an additional element to exercise it, record the cache key and then compare forevermore in the future | 09:53 |
| benschubert | so: | 09:54 |
| benschubert | 1) let's split the 'source' part of the test in one test per kind | 09:54 |
| benschubert | 2) Move this part to the source test, so that it can be run both by BuildStream and plugins without duplication (the test data would still be in BuildStream though, not sure that's great?) | 09:54 |
| benschubert | 3) the elements part of the test is for things that are kept in core anyways so we can keep them there | 09:54 |
| benschubert | or we can extend `register_kind` to give a list of elements for the cache key tests? | 09:56 |
| benschubert | which would solve the problem of keeping them in BuildStream | 09:56 |
| benschubert | that ways plugins can update it themselves | 09:57 |
| tristan | Yeah I'm sure there are parts of that I'm not grasping because I didn't pay enough attention to how the tests work | 10:02 |
| tristan | with regards to external plugins | 10:02 |
| tristan | I think it is better if the cache keys and element .bst files are maintained in the plugin repos, though | 10:03 |
| benschubert | yep, let me propose a MR so we have something tangible to discuss on? | 10:04 |
| tristan | The question is what happens when cache keys break intentionally | 10:04 |
| tristan | Until 2.0, this might happen a few times | 10:04 |
| tristan | After 2.0, it's good to have a story in place for plugins who are not going to be 'stable' | 10:05 |
| tristan | benschubert, Sure :) | 10:05 |
| tristan | there is a piece of information I've been holding onto... which is that, for cache key stability; especially for build element plugins and for project.conf defaults, we will most probably want to have a way for default YAML to evolve over time, where projects can "opt in" to new and improved defaults at the cost of a cache key break | 10:07 |
| tristan | paulsherwood, handled this in an interesting way in YBD, which was to have projects provide a defaults.yaml file, I'm not sure how we'd want to do this | 10:08 |
| *** phoenix has quit IRC | 10:23 | |
| *** hasebastian has quit IRC | 10:30 | |
| benschubert | cs-shadow, tristan, juergbi I was trying to push !1740 over the line, but we really have a lot of tests that require `tar`. I'm wondering whether we should not keep `tar` in core, as an exception, to allow both potential plugins fetching (pending proposal) and testing for us. Would that seem sensible to you? | 10:41 |
| gitlab-br-bot | MR !1740: WIP: tar plugin move https://gitlab.com/BuildStream/buildstream/-/merge_requests/1740 | 10:41 |
| benschubert | However, it shares code with the 'deb' plugin, so we'd need to be able to open some part of the API there | 10:43 |
| tristan | I'm ambivalent about whether we keep `tar` itself in core or whether we only use a simpler bare bones `bootstrap-tar` or such | 10:47 |
| tristan | It might be considered an advantage if we keep the implication of DownloadableFileSource class in core (more easily sharable with other plugins ?) | 10:49 |
| cs-shadow | i don't really mind either way, but i don't think it's worth ripping it out now since it's embedded in our tests. Even if we decide to move it out, I'd suggest defering that unitil we have a solid plan for how to split up plugins properly | 10:49 |
| tristan | s/implication/implementation | 10:49 |
| benschubert | yeah, I'd pause the moving of it until we have something for our tests (it's ued way more widely than any other plugin) | 10:49 |
| benschubert | and I'm also not usper confident about having it in both core and bst-plugins-experimental in the meantime | 10:50 |
| tristan | We probably need to sort this out soonish, though | 10:51 |
| benschubert | ah definitely | 10:51 |
| cs-shadow | tristan: what differences do you envision between the current tar plugin and the would-be bare bones `bootstrap-tar` ? | 10:51 |
| cs-shadow | (at a glance, I don't see much extra features in tar as it is) | 10:51 |
| tristan | cs-shadow, I'm not really sure to be honest, it's not a whole lot of code; afaiu the fancy stuff revolves mostly around supporting ETag | 10:52 |
| juergbi | the _find_base_dir() part is actually something that we might want to discuss for a core element | 10:53 |
| juergbi | I think it's not ideal as is | 10:53 |
| juergbi | very confusing for tarbombs | 10:53 |
| tristan | Ah yeah, the ability to extract subdirectories | 10:54 |
| benschubert | the problems I see: | 10:54 |
| benschubert | 1) it means keeping downloadablefilesource in core | 10:54 |
| benschubert | 2) the deb plugin depends on tar https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/src/bst_plugins_experimental/sources/deb.py#L56 | 10:54 |
| benschubert | But it is also solveable | 10:54 |
| tristan | I kind of thought the feature itself is nice | 10:54 |
| cs-shadow | subdirs are a fair point, but just to note - the etag business is in DownloadableSource already, so if we keep that and continue deriving tar from it, then we get that for "free" | 10:55 |
| juergbi | benschubert: 1) but we'd need that for barebones tar as well, wouldn't we? | 10:55 |
| benschubert | juergbi: unless we want to duplicate yes | 10:55 |
| juergbi | tristan: I agree but right now it's very confusing if the tarball does contain more than a single directory | 10:55 |
| juergbi | so it's fine for the vast majority of cases but not always | 10:56 |
| tristan | don't tarballs normally contain more than one directory ? | 10:56 |
| juergbi | I mean, one top-level directory | 10:56 |
| tristan | So the API is weird in that case ? | 10:56 |
| cs-shadow | i think `deb` deriving from tar is incorrect. It only works because the two plugins happen to be in the same package, but it's not kosher form BuildStream API point of view | 10:56 |
| tristan | I think a tarball containing more than one top-level directory, contains the directory `.` at the top doesn't it ? | 10:57 |
| juergbi | no, and we ignore ./ | 10:57 |
| juergbi | you can have a tarball with foo/bar.c and baz/bar.c | 10:57 |
| benschubert | cs-shadow: agreed, and I'm not sure the deb current implementaion is the best possible, but it does share quite a bit of code with tar | 10:57 |
| juergbi | or even just two files without any directory | 10:57 |
| juergbi | tristan: this I object to: "The first matching directory will be used." | 10:58 |
| tristan | juergbi, Well, maybe we need some churn in that API ? I think its good to have some API to be selective about what you want to extract | 10:58 |
| juergbi | I think we should error out if there are multiple matches to the pattern | 10:58 |
| benschubert | that seems sensible to me | 10:58 |
| juergbi | probably not a reason to keep it out of core. just wanted to mention this concern | 10:59 |
| tristan | Ah so... basically I think that was mostly so that we would have a default to match the "single toplevel directory" which is the majority of cases | 10:59 |
| cs-shadow | benschubert: i think a bit of code duplication is probably fine there as we don't have a mechanism to properly derive one plugin from another. We could obviously create a TarBase or something, but that seems overkill to me, just for `deb` plugin | 10:59 |
| tristan | So if it were to error out instead, we would still have a sane default | 10:59 |
| tristan | which would work for the majority of cases | 10:59 |
| juergbi | yes | 11:00 |
| benschubert | Ok, so if we were to: | 11:01 |
| benschubert | 1) Add the sane default to tar | 11:01 |
| benschubert | 2) Remove the dependency of `deb` on tar | 11:01 |
| benschubert | 3) Remove tar from bst-plugins-experimental | 11:01 |
| benschubert | Would that satisfy everything? | 11:01 |
| juergbi | fine with me | 11:02 |
| juergbi | we may want to review the DownloadableFileSource API and make that public | 11:02 |
| juergbi | or is it already? | 11:02 |
| benschubert | it is not yet | 11:03 |
| benschubert | and duplicated in core and bst-plugins-experimental | 11:03 |
| tristan | ...As I write docs about overlapping files... I notice that while we have a facility to whitelist files which an artifact overwrites in it's dependencies, we do not have a facility to do the opposite, and choose the overwritten files to be the "correct" ones | 11:13 |
| tristan | ... just an irrelevant observation ... | 11:13 |
| benschubert | tristan: like saying "don't overwrite it, keep the already present one" ? | 11:14 |
| tristan | yeah | 11:14 |
| benschubert | tristan: cs-shadow just to close the previous discussion does the proposed change seem good to you? | 11:14 |
| tristan | I suppose you could construct something with filter elements for this | 11:14 |
| benschubert | I actually be very happy if we could support that too | 11:14 |
| cs-shadow | benschubert: +1 from my side | 11:15 |
| tristan | benschubert, for your (1) (2) (3) proposal above, I have no objections and would be happy with this | 11:15 |
| tristan | +1 | 11:15 |
| benschubert | awesome! | 11:15 |
| benschubert | I'll create an issue for that | 11:16 |
| abderrahim[m] | tristan: what we're doing in that case is to append an `rm` command to install-commands | 11:17 |
| abderrahim[m] | but I guess this doesn't solve the issue if we still want the file when there are no ovelaps | 11:17 |
| tristan | abderrahim[m], so you've encountered situations where it would be useful ? How interesting ! | 11:19 |
| tristan | I mean, I guess that removing the file at install time is easy enough | 11:19 |
| tristan | What might we call the activity of modifying the artifacts at install time in order to avoid the conflict... "Artifact munging" ? | 11:30 |
| juergbi | I prefer structuring dependencies in such a way that there are no overlaps at all | 11:49 |
| tristan | juergbi, I think that overlaps are almost always unexpected, but I could be wrong | 11:51 |
| tristan | Maybe they are expected when you want to completely overwrite a library (provided across a junction) with another fresher one | 11:51 |
| tristan | In which case, the filter approach used in fdsdk/g-b-m is much better | 11:52 |
| juergbi | I personally don't like such overwriting | 11:52 |
| juergbi | but I can understand that it's not always simple to find another way | 11:52 |
| tristan | yeah I don't like that it exists, but yeah | 11:52 |
| tristan | I recall a veeeeerrrryyyy long time ago, considering the possibility of a "replaces" semantic for YBD | 11:54 |
| tristan | which would have done this same thing, but now I like the idea even less | 11:54 |
| juergbi | I wouldn't like 'replaces' on the file level | 11:57 |
| juergbi | however, 'replaces' on an element level could be interesting | 11:57 |
| juergbi | although, probably only safe in very specific scenarios | 11:57 |
| juergbi | in general it may be too confusing | 11:58 |
| tristan | It was indeed about the element level | 12:00 |
| tristan | But again, it gets into the territory of modifying the upstream project in place, and consequently consuming something that is unsupported by upstream | 12:00 |
| tristan | Then again, it *could* have an interesting additional property of saying... any reverse dependency of "this thing I'm replacing" should be built against "this element instead" | 12:01 |
| tristan | Which could plausibly have more interesting effects than the filter approach we've been seeing | 12:02 |
| *** tristan has quit IRC | 12:06 | |
| *** tristan has joined #buildstream | 12:24 | |
| *** ChanServ sets mode: +o tristan | 12:24 | |
| tristan | Grrrr, still have issues with CI it seems | 12:24 |
| tristan | the droplets are at it again I suppose | 12:24 |
| tristan | Looks like everything at https://gitlab.com/BuildStream/buildstream/pipelines is either failed cause they couldnt get a runner, or still running and waiting for a runner | 12:30 |
| juergbi | tristan: https://irclogs.baserock.org/buildstream/%23buildstream.2020-04-17.log.html#t2020-04-17T04:23:43 | 12:36 |
| *** hasebastian has joined #buildstream | 12:44 | |
| benschubert | juergbi: running the first `build` invocation on tests/frontend/pull.py::test_pull_artifact with `--debug` gives me the following buildbox-casd output: https://paste.gnome.org/pulr1iuvq Does that seem coherent to you? Do you have an idea of where I should look? | 12:49 |
| juergbi | benschubert: comparing it to here | 13:05 |
| juergbi | after Creating grpc channel there is a 20s gap in your log | 13:05 |
| juergbi | here it's 0.002s | 13:05 |
| benschubert | yep | 13:05 |
| benschubert | What.... | 13:06 |
| juergbi | benschubert: https://paste.gnome.org/p7e5rrr0x | 13:07 |
| benschubert | in `initializing remove caches`, I see "14:02:42 Using ares dns rsolver" then next GRPC log is "13:03:02 New connected subchannel", there's nothing in between either in GRPC or in BuildStrem | 13:08 |
| juergbi | maybe it attempts DNS lookup for localhost? | 13:08 |
| juergbi | do you have localhost in /etc/hosts or do you rely on separate localhost handling via /etc/nsswitch.conf? | 13:09 |
| juergbi | ares dns resolver doesn't really use the modules in nsswitch.conf | 13:09 |
| benschubert | let me trace the ares resolver | 13:10 |
| benschubert | nsswitch.conf: `hosts: files dns` | 13:10 |
| benschubert | and I definitely have localhost in /etc/hosts | 13:10 |
| juergbi | ok, then it shouldn't be related to !1836 | 13:10 |
| gitlab-br-bot | MR !1836: resolve hostname before sending to grpc https://gitlab.com/BuildStream/buildstream/-/merge_requests/1836 | 13:10 |
| juergbi | but my suspicion is a timeout somewhere that you're hitting | 13:11 |
| benschubert | just looked at the logs of my home dns resolver, I'm not asking for localhost, so it doesn't seem to be getting out of my laptop | 13:12 |
| juergbi | maybe strace with timestamps? | 13:12 |
| juergbi | that might be problematic with FUSE, though, so maybe better not test with buildbox-fuse | 13:12 |
| juergbi | (assuming you see the issue also without) | 13:13 |
| benschubert | mmh good point I'll try this | 13:13 |
| juergbi | testing without buildbox-run backend in general is probably good for comparison, if you haven't yet | 13:13 |
| benschubert | isn't the default still to use bwrap? | 13:14 |
| benschubert | (haven't set anything in particular there) | 13:14 |
| juergbi | it is | 13:14 |
| benschubert | then I should still be using vwrap | 13:15 |
| juergbi | ah, we get the FUSE stager debug message also if we don't use it | 13:15 |
| gitlab-br-bot | marge-bot123 merged MR !1867 (valentindavid/bst-1/retry-cas-calls->bst-1: Handle grpc errors of type UNAVAILABLE and ABORTED) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1867 | 13:15 |
| benschubert | which I do | 13:15 |
| juergbi | (it takes the same amount of time here) | 13:15 |
| juergbi | benschubert: and I just did a test against the buildbox static binaries, no time difference either | 13:18 |
| juergbi | as they statically link protobuf/gRPC, it shouldn't be an issue with your protobuf/gRPC version, assuming you see the slow times also with the static binaries | 13:18 |
| benschubert | cares -> "timeout while contacting DNS servers" | 13:18 |
| benschubert | ok that is then the problem | 13:18 |
| benschubert | it indeed takes 15 sec | 13:19 |
| benschubert | it doesn't make any sense though: c-ares resolver gets a AF_INET result: addr:127.0.0.1, port: 34661, and then it loops to try to find something | 13:19 |
| juergbi | maybe it also attempts IPv6 in parallel and that fails? | 13:20 |
| benschubert | aah right I have an IPv6 since a few weeks ago | 13:21 |
| benschubert | but I do have: ::1 ip6-localhost | 13:21 |
| benschubert | in my /etc/hosts | 13:21 |
| juergbi | I have ::1 localhost | 13:22 |
| benschubert | aaand the test is now instant | 13:23 |
| benschubert | -_-' | 13:23 |
| benschubert | thanks! | 13:23 |
| juergbi | yw | 13:23 |
| benschubert | However, we need to fix this, a default ubuntu installation apparently doesn't put 'localhost' on ipv6 | 13:23 |
| benschubert | my /etc/hosts was not touched after install | 13:23 |
| juergbi | that might be a bug either in ares or ubuntu | 13:24 |
| juergbi | I don't think we can do much on our side | 13:24 |
| juergbi | except report it, if it's not a known issue | 13:25 |
| benschubert | I'll try to see if I find something | 13:29 |
| juergbi | benschubert: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1644009 | 13:31 |
| juergbi | benschubert: btw: in an ubuntu 18.04 container here I have: ::1localhost ip6-localhost ip6-loopback | 13:31 |
| benschubert | weird | 13:31 |
| juergbi | I also still have a 14.04 container, same there | 13:32 |
| benschubert | I'm on ubuntu 20.04 | 13:32 |
| juergbi | and I've never touched those | 13:32 |
| juergbi | don't have 20.04 around right now | 13:32 |
| benschubert | ok, now the rest suite runs in 355s still something weird then :'D | 13:37 |
| juergbi | benschubert: I'm running with -n 12, though. with -n 4 it would likely be slower as well | 13:45 |
| benschubert | slowest test is: tests/artifactcache/config.py::test_only_one with 77s each Oo | 13:48 |
| juergbi | 150.87s here with -n 4 | 13:49 |
| benschubert | yeah, seems like the 4 test_only_one are the difference between us both | 13:50 |
| juergbi | 0.06s call tests/artifactcache/config.py::test_only_one[index-user] | 13:51 |
| benschubert | great x') on to debug those | 13:51 |
| WSalmon | Masive thanks to benbrown for updating the version of docker-machine on the bastion, with shiny ansible too!! the CI seems to be running much better than it was, the new driver has a limit on DO's API so that will mean that new runners wont just get spun up instantly but it should mean that they do come up eventually | 14:32 |
| WSalmon | jjardon, i know you were considering attacking the bastion later, it seems to be behaving its self atm! | 14:33 |
| benschubert | awesome thanks! | 14:35 |
| juergbi | great, thanks benbrown | 14:39 |
| WSalmon | https://gitlab.com/BuildStream/infrastructure/docker-machine-install-role/-/merge_requests/1 this is what we are now running in prod, if juergbi or jjardon fancy hitting the big green button | 14:40 |
| benbrown | juergbi: ta! | 14:41 |
| WSalmon | https://gitlab.com/BuildStream/infrastructure/docker-machine-install-role/-/merge_requests/2 not many people can merge to this repo, could some of the good ansible people be added maybe coldtom benbrown benbrown benschubert etc can juergbi jjardon do that? | 14:53 |
| WSalmon | or maybe to the infra group so it gets inherited? | 14:54 |
| WSalmon | valentind, also maybe? | 14:54 |
| WSalmon | oh valentind can already | 14:56 |
| WSalmon | very sensible | 14:56 |
| *** hasebastian has quit IRC | 14:58 | |
| benschubert | juergbi: ok the reason for the second problem is because cares tries to resolve example.com and fails :'D | 14:58 |
| *** hasebastian_ has joined #buildstream | 14:58 | |
| gitlab-br-bot | willsalmon opened MR !1868 (willsalmon/workspaceFix->master: _workspaces.py: Add option to update workspace version) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1868 | 15:17 |
| WSalmon | abderrahim[m], dose ^ work for you? | 15:27 |
| WSalmon | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1868/diffs#3012a7da425a1d025fe027616100d26c2f017337_485_491 seems to work nicely but wonders if benschubert will have a more canonical way to do it? | 15:29 |
| abderrahim[m] | I'll take a look later today | 15:32 |
| WSalmon | thanks | 15:48 |
| *** lachlan has joined #buildstream | 16:09 | |
| *** rdale has quit IRC | 16:14 | |
| *** tpollard has quit IRC | 16:59 | |
| *** hasebastian_ has quit IRC | 17:05 | |
| *** santi has quit IRC | 17:13 | |
| jjardon | WSalmon: nice! do you still need permissions? | 17:14 |
| WSalmon | jjardon, i can log in now but i dont have access to DO | 17:17 |
| WSalmon | it still seems a bit funny but its much better than it has been | 17:18 |
| WSalmon | i will go look on monday morning | 17:18 |
| jjardon | WSalmon: what's your email? | 17:19 |
| WSalmon | will.salmon@codethink.co.uk | 17:20 |
| jjardon | WSalmon: you should be able now | 17:20 |
| jjardon | It's literally one machine which needs some maintenance; much better now with that ansible though :) | 17:21 |
| *** mohan43u has quit IRC | 17:30 | |
| *** lachlan has quit IRC | 17:38 | |
| *** mohan43u has joined #buildstream | 17:40 | |
| *** tristan has quit IRC | 18:20 | |
| *** benschubert has quit IRC | 19:08 | |
| *** toscalix has joined #buildstream | 19:27 | |
| *** mohan43u_ has joined #buildstream | 19:48 | |
| *** mohan43u_ has joined #buildstream | 19:49 | |
| *** mohan43u_ has joined #buildstream | 19:49 | |
| *** mohan43u has quit IRC | 19:50 | |
| *** cs-shadow has quit IRC | 20:51 | |
| *** toscalix has quit IRC | 22:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!