| *** delli3 has quit IRC | 00:16 | |
| *** delli3 has joined #buildstream | 00:17 | |
| *** delli3 has quit IRC | 00:25 | |
| *** delli3 has joined #buildstream | 00:28 | |
| gitlab-br-bot | jjardon opened issue #1279 (CI is failing: "ERROR: Preparation failed: exit status 1") on buildstream https://gitlab.com/BuildStream/buildstream/-/issues/1279 | 00:49 |
|---|---|---|
| *** mohan43u_ has quit IRC | 00:50 | |
| *** mohan43u has joined #buildstream | 00:50 | |
| *** narispo has quit IRC | 01:02 | |
| *** narispo has joined #buildstream | 01:02 | |
| *** delli3 has quit IRC | 01:18 | |
| *** mohan43u has quit IRC | 02:06 | |
| *** mohan43u has joined #buildstream | 04:39 | |
| cphang | Looks like this was retriggered by WSalmon and it now passed. Still need to root cause why it's flaky | 07:24 |
| *** benschubert has joined #buildstream | 07:47 | |
| *** tpollard has joined #buildstream | 08:04 | |
| *** rdale has joined #buildstream | 08:27 | |
| gitlab-br-bot | juergbi opened MR !1845 (juerg/platform->master: Improve sandbox configuration handling) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1845 | 08:36 |
| *** santi has joined #buildstream | 08:39 | |
| *** tristan has quit IRC | 08:41 | |
| *** lachlan has joined #buildstream | 09:23 | |
| *** tristan has joined #buildstream | 09:24 | |
| *** lachlan has quit IRC | 10:06 | |
| *** lachlan has joined #buildstream | 10:22 | |
| *** lachlan has quit IRC | 10:33 | |
| *** lachlan has joined #buildstream | 10:43 | |
| *** ChanServ sets mode: +o tristan | 11:04 | |
| tristan | Only one runner these days ? | 11:04 |
| tristan | jjardon, ^^^ ? | 11:04 |
| jjardon | tristan: you need to ask anyone from the infra team https://gitlab.com/BuildStream/infrastructure/infrastructure/-/wikis/home | 11:06 |
| jjardon | tristan: but it seems like a bug | 11:06 |
| tristan | valentind, ^^^ ? | 11:06 |
| jjardon | the bastion should spin up runners on demand | 11:06 |
| tristan | I've got one pipeline I've pushed, and they are all waiting on digitalocean-bastion-x86_64 (#483969) | 11:07 |
| jjardon | ok, then maybe the bastion is down | 11:07 |
| tristan | Hmmm, maybe I'm wrong, looks like they are slowly coming to life, but they all seem to say the same runner :-S | 11:09 |
| *** lachlan has quit IRC | 11:09 | |
| jjardon | benbrown: are you working in the bastion now? ^ | 11:10 |
| juergbi | not sure what's going on but I just deleted old runners that somehow weren't shut down | 11:11 |
| tristan | jjardon, I think there is no issue, except the issue that all jobs are reporting the same runner | 11:11 |
| juergbi | it's not behaving properly for a few weeks now | 11:11 |
| benbrown | Yeah, getting my bearings. DO did list a total of 77 droplets, unsure if the bastion lost track. | 11:11 |
| tristan | They all say "on buildstream-bastion a334e492" | 11:12 |
| benbrown | Looks like this machine needs a bit of attention, I'm sure the 5000 directories that don't map to a running machine in ~/.docker/machine/machines doesn't help things | 11:12 |
| tristan | "Running with gitlab-runner 12.0.1 (0e5417a3)" | 11:12 |
| tristan | But... they *are* processing in parallel | 11:12 |
| tristan | They also all say "digitalocean-bastion-x86_64 (#483969)" in the UI | 11:13 |
| jjardon | benbrown: note there is a script to kill stale runners | 11:13 |
| tristan | Maybe there is only one runner but it is allowed to do a few jobs in parallel (in separate dockers) ? | 11:13 |
| jjardon | benbrown: https://gitlab.com/BuildStream/infrastructure/infrastructure | 11:14 |
| benbrown | Yeah I've noticed that, but it doesn't clean up the bastion | 11:14 |
| jjardon | benbrown: the bastion should always run, but not run any jobs; what you mean by cleaning the bastion? | 11:15 |
| *** lachlan has joined #buildstream | 11:18 | |
| benbrown | Tbh I might be barking up the wrong tree, but it's something worth tidying. docker-machine tracks machine config/ssh keys in the directory mentioned above, with a directory per machine, the 5000+ directories that are present but not removed since machines are removed in DO and not via the bastion are preventing `docker-machine ls` from running. Might not have any influence on docker-machines normal operations. | 11:19 |
| jjardon | benbrown: oh, by any means that needs to be cleaned up | 11:20 |
| tristan | jjardon, as I recall, the main reason to release 1.4.2 is for python 3.8 support (plus couple bugfixes) | 11:27 |
| tristan | But nobody added py38 to the CI, and it's looking like it's got failing tests :-S | 11:28 |
| tristan | https://gitlab.com/BuildStream/buildstream/-/jobs/492363875 | 11:28 |
| tristan | So lets put that on ice for a bit, maybe I can solve that today | 11:28 |
| jjardon | sure | 11:29 |
| * jjardon didn't know no python3.8 tests existed for the bst-1 branch | 11:29 | |
| tristan | jjardon, they are the same tests for all python versions | 11:30 |
| tristan | There is a py38 docker image in use in master | 11:30 |
| tristan | I added that to the list and it breaks (so far) on the cache key test | 11:30 |
| tristan | Ah, I read your last statement without the word "no" :) | 11:31 |
| jjardon | tristan: yeah, I meant that :) | 11:31 |
| * jjardon reads again and his sentence was not clear at all | 11:32 | |
| tristan | Nah I'm just a bit dyslexic :) | 11:32 |
| tristan | Aha, pip source tests also broken | 11:35 |
| tristan | did we land this without running the test suite ? seems unlikely | 11:35 |
| tpollard | There may changes landed in master that can be backported for 3.8 changes | 11:37 |
| *** santi has quit IRC | 11:39 | |
| *** santi has joined #buildstream | 11:39 | |
| tristan | tpollard, yes there were already but I'll try to search there | 11:54 |
| tpollard | indeed, but if it wasn't tested against the full test suite... | 12:05 |
| tristan | Yeah I just did a search through that | 12:06 |
| tristan | interestingly in 1.4, pip source plugin is whining about "Unable to find a suitable pip command" | 12:06 |
| tristan | And in master, _PYTHON_VERSIONS was not updated to add python3.8 | 12:07 |
| *** lachlan has quit IRC | 12:07 | |
| tristan | So for some reason it works in master, or the pip source may accidentally be untested in master, or smth | 12:07 |
| tristan | I suspect it's a similar issue for cache key, it bails out not because of cache key but the `bst show` command itself is failing | 12:08 |
| tpollard | pip source should be tests in master & bst-plugins-experimental currently | 12:08 |
| tpollard | *tested | 12:08 |
| tristan | probably all 4 tests will work after fixing that | 12:09 |
| valentind | Sorry, I was having lunch. Do you still need help with the builders? | 12:09 |
| * tristan is using the same docker image as in master | 12:09 | |
| tristan | valentind, as I understand, the same runner is indeed running multiple jobs in parallel, but I don't understand the runner setup correctly | 12:10 |
| tristan | at first I was worried because the messages all indicate the same runner | 12:10 |
| tristan | I don't know if that is intentional, others seem to think it is not | 12:10 |
| valentind | I will ssh to see. | 12:11 |
| valentind | Well it is creating them now. | 12:14 |
| valentind | It might just delete them. | 12:15 |
| tristan | valentind, Do you know why they all say "buildstream-bastion a334e492" in the beginning of their output, and they all say "igitalocean-bastion-x86_64 (#483969)" in the gitlab UI ? | 12:16 |
| valentind | This is normal. | 12:16 |
| tristan | valentind, e.g. for this: https://gitlab.com/BuildStream/buildstream/pipelines/131413524 | 12:16 |
| tristan | So how do we know if it's using more than one runner or not ? | 12:16 |
| benbrown | jjardon: I've not seen this before, have you? "cache adapter could not be initialized: missing S3 configuration" | 12:16 |
| benbrown | The S3 configuration appears to be present | 12:16 |
| jjardon | benbrown: yeah, I do not think buildstream is using that for cache | 12:17 |
| valentind | Ah right they are all on 0e5417a3. | 12:17 |
| tristan | valentind, At first, it *looked* like it was going to run them all sequentially, but it is processing a few jobs at once at least | 12:18 |
| tristan | valentind, again I don't know if it is setup correctly | 12:18 |
| valentind | ERROR: Docker machine "runner-a334e492-gitlab-runner-autoscale-1584551586-29da6e54" does not exist. Use "docker-machine ls" to list machines. | 12:18 |
| valentind | This is the issue I think. | 12:19 |
| tristan | So there is an issue :) | 12:19 |
| benbrown | I don't think that's the root issue | 12:20 |
| benbrown | Those messages seem to appear as a result of failure to create | 12:21 |
| benbrown | And the failure to create seems to be because it "Could not create cache adapter" | 12:21 |
| jjardon | benbrown: IIRC, Ive seen that error when you dont configure S3 cache in the runner toml, but then someone try to use cache: in the the CI YAML | 12:23 |
| benbrown | jjardon: S3 cache is configured in the runner toml, I'm unsure if the credentials are invalid | 12:24 |
| jjardon | benbrown: oh, then we need to take a look indeed | 12:25 |
| valentind | Probably expired. | 12:25 |
| valentind | The key is still there | 12:26 |
| jjardon | maybe a problem with the API (bst uses DO S3, not AWS one; it took me ages to configure exactly as it wanted the first time) | 12:26 |
| valentind | We have to move the configration entries | 12:27 |
| valentind | The file format for the configuration seems to have changed. | 12:27 |
| valentind | It has to be moved to [runners.cache.s3] I think. | 12:27 |
| jjardon | I see | 12:27 |
| valentind | We should check the documentation. | 12:28 |
| jjardon | guess https://wiki.gnome.org/Projects/BuildStream/Infrastructure is not up-to-date | 12:28 |
| valentind | But this is not a big deal. I remember seeing this error message before. | 12:28 |
| jjardon | maybe better to put it in a repo somewhere | 12:28 |
| jjardon | I think benbrown already have some plans for it :) | 12:28 |
| valentind | The error messages about not being able to create are just old. | 12:29 |
| valentind | Maybe restarting the gitlab-runner would reduce the error messages of old missing machines. | 12:30 |
| benbrown | I was getting the error messages about create during `journalctl -f` | 12:31 |
| valentind | benbrown, Yes, but I think it tried to create them a long time ago. | 12:31 |
| benbrown | Restarting might help reduce error messages about missing machines though | 12:31 |
| valentind | And then it failed with a very long timeout. | 12:31 |
| valentind | There are no message in the log about the creations of those machines. | 12:32 |
| valentind | OK, I restart to see. | 12:32 |
| tristan | heh, there goes my pipeline ;-) | 12:33 |
| valentind | I retried it. | 12:33 |
| tristan | but, test_cache_key() passed after adding python3.8 to pip.py's _PYTHON_VERSIONS | 12:34 |
| tristan | Good sign :) | 12:34 |
| tristan | Can't really say why this is not needed in master though | 12:34 |
| tristan | Should probably patch master too for consistency | 12:35 |
| valentind | Does not make sense. | 12:35 |
| valentind | It is building. But not on new builders. | 12:36 |
| tristan | Hmmm | 12:37 |
| tristan | It seems it was more snappy to start everything at once | 12:37 |
| tristan | but I could just be hallucinating | 12:37 |
| valentind | It seems to be running on existing machines. | 12:37 |
| valentind | Maybe that is fine. | 12:37 |
| valentind | All the builds seem to be running on a334e492 howeve. | 12:38 |
| valentind | Ah that is the bastion. Not an issue. | 12:38 |
| valentind | tristan, It is OK, they run on different builders. | 12:40 |
| tristan | valentind, cool, thanks for popping in there :) | 12:41 |
| valentind | Look at the line 7: Running on runner-a334e492-project-1975139-concurrent-0 via runner-a334e492-gitlab-runner-autoscale-1585656852-00e32fad... | 12:41 |
| valentind | The last name is the name of the builder. | 12:41 |
| tristan | I do think performance is better since you restarted | 12:41 |
| valentind | This is what you have to look at. | 12:41 |
| tristan | ah ok | 12:41 |
| valentind | So here it is 00e32fad | 12:41 |
| valentind | But if you look at another build, then it has a different name. | 12:42 |
| tristan | 5ffde9e4 in the python3.8 test I was interested in | 12:42 |
| valentind | I think it was just gitlab-runner had too much cleanup to do probably. | 12:42 |
| valentind | And it was taking a long time. | 12:42 |
| tristan | 00e32fad for debian-9 | 12:42 |
| valentind | But let's fix the S3 cache. | 12:43 |
| valentind | Someone is editing the configuration, benbrown? jjardon? | 12:43 |
| jjardon | not me | 12:44 |
| valentind | https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscache-section | 12:45 |
| valentind | It explains here what to change, | 12:45 |
| jjardon | you were rigth: "With GitLab Runner 12.0.0 the old configuration syntax was removed and is no longer supported" | 12:49 |
| valentind | I just wait for whoever is editing config.toml to be finished. | 12:50 |
| benbrown | valentind: Ah sorry, just had it open in vim | 12:50 |
| * benbrown may as well have used less for his purposes | 12:50 | |
| gitlab-br-bot | tristanvb opened MR !1846 (tristan/bst-1/testsuite-py38->bst-1: Pass CI with python3.8) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1846 | 12:50 |
| tristan | There is a merge request for a pip source fix and addition of python 3.8 (for bst-1 branch) | 12:51 |
| valentind | OK, let's see if works better. Maybe we have to restart though. | 12:51 |
| valentind | That did not fix it. I will restart. | 12:53 |
| gitlab-br-bot | tristanvb opened MR !1847 (tristan/pip-source-3.8->master: pip source plugin: Add support for python 3.8) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1847 | 12:57 |
| tristan | And there is a merge request for doing the same change on pip source for master | 12:57 |
| valentind | ERROR: Error creating machine: Error in driver during machine creation: GET https://api.digitalocean.com/v2/droplets/186853180: 429 Too many requests | 13:01 |
| valentind | At least the S3 error is gone. | 13:02 |
| *** tristan has quit IRC | 13:07 | |
| gitlab-br-bot | tpollard approved MR !1847 (tristan/pip-source-3.8->master: pip source plugin: Add support for python 3.8) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1847 | 13:09 |
| *** lachlan has joined #buildstream | 13:32 | |
| *** lachlan has quit IRC | 13:51 | |
| *** lachlan has joined #buildstream | 13:55 | |
| *** mohan43u_ has joined #buildstream | 14:00 | |
| *** mohan43u has quit IRC | 14:02 | |
| *** mohan43u_ has quit IRC | 14:03 | |
| *** mohan43u has joined #buildstream | 14:03 | |
| *** lachlan has quit IRC | 14:06 | |
| *** lachlan has joined #buildstream | 14:16 | |
| *** lachlan has quit IRC | 14:21 | |
| *** lachlan has joined #buildstream | 15:17 | |
| *** lachlan has quit IRC | 15:42 | |
| *** lachlan has joined #buildstream | 16:08 | |
| tpollard | I've got an issue where I can wget a tar url on a machine without an issue, but when bst tracks the same url it gets a 403? | 16:28 |
| *** tpollard has quit IRC | 16:56 | |
| *** lachlan has quit IRC | 17:02 | |
| *** mohan43u has quit IRC | 17:10 | |
| *** lachlan has joined #buildstream | 17:14 | |
| *** mohan43u has joined #buildstream | 17:14 | |
| gitlab-br-bot | juergbi opened MR !1848 (juerg/buildbox-run-error->master: _sandboxbuildboxrun.py: Check for buildbox-run initialization errors) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1848 | 17:15 |
| *** santi has quit IRC | 17:57 | |
| *** tristan has joined #buildstream | 18:01 | |
| *** lachlan has quit IRC | 18:15 | |
| *** toscalix has joined #buildstream | 18:55 | |
| *** toscalix has quit IRC | 19:03 | |
| *** tpollard has joined #buildstream | 19:05 | |
| *** mohan43u_ has joined #buildstream | 19:08 | |
| *** tpollard has quit IRC | 19:08 | |
| *** mohan43u has quit IRC | 19:09 | |
| *** rdale has quit IRC | 19:55 | |
| *** benschubert has quit IRC | 19:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!