| juergbi | jjardon: the new Debian droplets (both permanent and autoscale) break non-setuid bubblewrap as they have a Debian-specific patch disabling unprivileged user namespaces | 05:23 |
|---|---|---|
| juergbi | we either need to set kernel.unprivileged_userns_clone=1 or switch to e.g. Fedora | 05:24 |
| juergbi | I could manually change the former (sysctl) on the permanent ones but if they weren't set up manually, it would be better to update the script | 05:26 |
| juergbi | and not sure what's the best way to change it for autoscale droplets | 05:26 |
| juergbi | theoretically, I think we could even set it from within the (privileged) docker containers. however, as it affects the whole system, that wouldn't be very clean | 05:27 |
| jjardon | juergbi: let me change all to fedora | 06:46 |
| jjardon | juergbi: well no, the permanents are debian, so please go ahead with that manual fix | 06:47 |
| jjardon | I will change the autoscale ones | 06:47 |
| juergbi | ok, will change the sysctl in the permanent ones | 06:49 |
| jjardon | juergbi: done | 06:49 |
| *** tristan has quit IRC | 06:49 | |
| jjardon | (sorry I would do the permanents ones but I have a meeting in 5 min) | 06:50 |
| juergbi | no problem. I've created and applied /etc/sysctl.d/local.conf on the permanent ones | 06:54 |
| *** tristan has joined #buildstream | 06:55 | |
| *** narispo has quit IRC | 07:09 | |
| *** benschubert has joined #buildstream | 07:09 | |
| *** narispo has joined #buildstream | 07:09 | |
| *** ChanServ sets mode: +o tristan | 07:13 | |
| tristan | Interesting, so because of all the CI madness on master, the website is not updating... https://gitlab.com/BuildStream/docs-website/-/jobs/517869182 | 07:14 |
| tristan | I think this will go away by itself though | 07:14 |
| tristan | Lets try manually running a normal, documentation publishing pipeline on master (not a scheduled nightly known-to-fail pipeline) | 07:15 |
| tristan | and then rerun docs-website | 07:15 |
| gitlab-br-bot | tristanvb opened MR !1869 (tristan/remove-old-version-annotations->master: Remove old version annotations) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1869 | 07:42 |
| gitlab-br-bot | BenjaminSchubert approved MR !1869 (tristan/remove-old-version-annotations->master: Remove old version annotations) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1869 | 07:47 |
| tristan | What is the current way for checking if a MappingNode has a key ? | 08:07 |
| tristan | if "key" in node.keys(): ... ? | 08:07 |
| juergbi | tristan: should be without the .keys() part | 08:24 |
| juergbi | i.e., it directly implements __contains__ | 08:25 |
| *** santi has joined #buildstream | 08:26 | |
| tristan | Alrighty | 08:28 |
| *** narispo has quit IRC | 08:30 | |
| *** narispo has joined #buildstream | 08:30 | |
| juergbi | valentind: !1843 now also fixes the FileNotFoundError in push that you've encountered (incl. test case). I hope that covers related missing blob issues and will hopefully be merged soon | 08:35 |
| gitlab-br-bot | MR !1843: Fix handling of missing blobs in `ArtifactCache.pull()` https://gitlab.com/BuildStream/buildstream/-/merge_requests/1843 | 08:35 |
| juergbi | *all related missing blob issues | 08:35 |
| valentind | OK, I will update the docker image | 08:36 |
| *** rdale has joined #buildstream | 08:40 | |
| *** tpollard has joined #buildstream | 08:46 | |
| jjardon | juergbi: meeting finish; all ok with the CI so far? | 08:50 |
| juergbi | jjardon: I think it's working ok right now, will see more over time | 08:51 |
| juergbi | a hardlink-related tar tests appears to be somewhat flaky now. not sure why but haven't seen this before. will keep an eye on it | 08:53 |
| *** lachlan has joined #buildstream | 08:56 | |
| jjardon | juergbi: on the elastic or permanent runners? or both? | 08:57 |
| juergbi | on a permanent runner | 08:57 |
| tristan | Pipelines seem to be failing on master, it seems to only happen sometimes but, I've got two failures from ubuntu 18 in a row: https://gitlab.com/BuildStream/buildstream/-/jobs/518074334 | 08:58 |
| jjardon | ah ok; that is running debian10 instead fedora30 so maybe there are some differences | 08:58 |
| tristan | Seems unrelated to runners | 08:58 |
| juergbi | that's indeed the tar hardlink issue. this one is also on a permanent runner | 08:59 |
| juergbi | tristan: have you seen the same issue on an autoscale runner? | 08:59 |
| tristan | test_out_of_basedir_hardlinks | 08:59 |
| tristan | Not today, today I just have these | 08:59 |
| jjardon | permanent runners are new (and based in debian10 instead fedora30 as the elastic ones), so take that into account. It would be interesting to know what is the diference that makes them fail | 09:00 |
| juergbi | ok, wondering what kernel (or docker host) aspect could cause that failure | 09:00 |
| tristan | Ok so, perhaps they succeeded in pre-merge before the runner configuration changed ? | 09:01 |
| jjardon | I think those test still run in the same docker container so the only difference should be the kernel and maybe the docker version | 09:03 |
| juergbi | pre-merge was on autoscale runner from the old buildstream-bastion | 09:03 |
| juergbi | jjardon: old buildstream-bastion also used fedora 30 droplets? | 09:03 |
| jjardon | oh, is the docker image to use specify in the job? | 09:03 |
| jjardon | or is taking the default one? /me checks | 09:03 |
| juergbi | the docker image is specified in the job | 09:03 |
| jjardon | yeah, we have a global image: in CI | 09:04 |
| juergbi | nothing should have changed with regards to the docker images | 09:04 |
| juergbi | we do have a global image: as well but for tests such as ubuntu 18.04 we override it in the job | 09:05 |
| jjardon | ah cool; better. we are extra safe as we have the global image: so no image is taking by default from the runners config | 09:06 |
| jjardon | so only difference should be kernel and maybe docker version/config in the host? Not sure how that can affect the hardlink tests though | 09:06 |
| jjardon | juergbi: old buildstream-bastion also used fedora 30 droplets? --> yes | 09:06 |
| juergbi | yes, don't have an idea right now why it would fail on the debian 10 kernel | 09:07 |
| jjardon | basically buildtream-bastion is the same, only change is the name to bastion-debian9 to differentiatie from the new one (bastion-debian10) | 09:07 |
| juergbi | especially as it seems to fail in python code, not the actual hardlink call | 09:08 |
| WSalmon | Do we know what is causing the failed to upload artifacts stuff? https://gitlab.com/BuildStream/buildstream/-/jobs/517587022 | 09:16 |
| juergbi | WSalmon: what makes you think there is an issue uploading artifacts? | 09:17 |
| juergbi | tox is reporting failures | 09:17 |
| juergbi | tristan: regarding workspaces, always supporting older versions is nice in theory but with the workspace revamp it's not really possible | 09:19 |
| *** tristan has quit IRC | 09:19 | |
| WSalmon | oh ok, i thought i had had one that did work and then didnt upload, sorry | 09:19 |
| WSalmon | i just grabed that one without looking closely enough | 09:20 |
| juergbi | jjardon: I think the hardlink issue is a bug in the test case. working on fixing it | 09:24 |
| juergbi | it seems to incorrectly rely on the order of directory entries. maybe a kernel or filesystem difference causes a relatively consistent readdir() order difference in this particular case | 09:28 |
| WSalmon | juergbi, would this be cause if a the file system had mtimes to the second? i seem to rember one time valanteen changed something use having issues. | 09:30 |
| juergbi | WSalmon: do you mean the errors in the log you posted? | 09:37 |
| juergbi | that was `bwrap: No permissions to creating new namespace`. this was a temporary CI issue, has been fixed in the meantime | 09:38 |
| WSalmon | juergbi, so those jobs just need rerunning? | 09:44 |
| juergbi | yes, you need to retry jobs with bwrap errors | 09:45 |
| juergbi | only happened in a time frame of 24 hours or so | 09:45 |
| *** lachlan has quit IRC | 09:47 | |
| WSalmon | juergbi, ta | 09:49 |
| gitlab-br-bot | juergbi opened MR !1870 (juerg/tar-hardlinks->master: tests/sources/tar.py: Fix flaky test_out_of_basedir_hardlinks) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1870 | 09:52 |
| *** lachlan has joined #buildstream | 09:54 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1870 (juerg/tar-hardlinks->master: tests/sources/tar.py: Fix flaky test_out_of_basedir_hardlinks) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1870 | 09:59 |
| *** lachlan has quit IRC | 09:59 | |
| *** tristan has joined #buildstream | 10:03 | |
| *** ChanServ sets mode: +o tristan | 10:03 | |
| jjardon | juergbi: do you think https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/issues/60 makes sense ? | 10:07 |
| juergbi | if we want to have a long-running buildbox-casd, socket activation would definitely be the best approach | 10:09 |
| juergbi | however, that would require some additional work in buildbox-casd (e.g., dynamic protect-session-blobs) and also work in buildstream | 10:10 |
| juergbi | and there is a question how to handle buildstream-configured quota with a longer running casd | 10:11 |
| juergbi | and we can't hard depend on systemd, let alone systemd user session | 10:11 |
| juergbi | first we should have clearer motivation. not quite sure about "easier for local CAS" | 10:12 |
| juergbi | overall the setup with systemd user session is more complex | 10:12 |
| juergbi | it has some advantages but I wouldn't say it's easier in general | 10:12 |
| *** lachlan has joined #buildstream | 10:13 | |
| douglaswinship | has anyone else experienced the 'buildstream' and 'buildstream-install' directories appearing when you check an element out? I'm fairly sure that's an error. From what I can tell, it happens whenever integratoin commands are run during checkout. The effect dissapears if there are no integration commands or if the '--no-integrate' option is used. Seems to affect manual elements and anything | 10:15 |
| douglaswinship | descended from them, but it doesn't affect import, compose or filter elements. | 10:15 |
| douglaswinship | I assumed it was a bug that had been introduced at some point, but I checked back and I seem to be able to replicate it in older versions, all the way back to 1.0.0, so maybe it's an intended behaviour for some reason? (Or just hasn't been deemed important enough to fix?) | 10:16 |
| douglaswinship | So i'm questioning whether it's worth raising an issue on gitlab. | 10:16 |
| juergbi | douglaswinship: that does sound like a bug to me. please raise it on gitlab | 10:17 |
| douglaswinship | Thought so. | 10:18 |
| douglaswinship | Are there any standards / formats i need to know about for raising issues on this project? This'll be my first one. | 10:18 |
| coldtom | there's a template when you open the issue douglaswinship | 10:20 |
| *** lachlan has quit IRC | 10:21 | |
| douglaswinship | perfect. Thanks | 10:21 |
| *** lachlan has joined #buildstream | 10:35 | |
| tristan | Interesting... according to gitlab's python API, there has not been any "docs" job which happened at all, since commit c39c12ec | 10:35 |
| tristan | which is December 10 | 10:36 |
| juergbi | that seems wrong | 10:37 |
| tristan | Indeed | 10:38 |
| tristan | There is still clearly a docs job | 10:38 |
| tristan | This is with some debug tracing added: https://paste.centos.org/view/572626ab | 10:40 |
| tristan | This is the file which crawls artifacts: https://gitlab.com/BuildStream/docs-website/-/blob/master/generate_pages.py | 10:41 |
| *** phoenix has joined #buildstream | 10:42 | |
| tristan | since juergbi's juerg/buildbox-run branch landed, there are no more "docs" artifacts reachable via gitlab restful API | 10:43 |
| tristan | juergbi, I have a feeling that this might be related to having failing jobs perhaps ? | 10:43 |
| tristan | Like, do we have some jobs which are allowed to fail, or some similar side effect since landing that branch ? | 10:44 |
| juergbi | buildbox-run is not allowed-to-fail, though | 10:44 |
| juergbi | we do have allowed-to-fail but not related to buildbox | 10:44 |
| juergbi | e.g., fedora with updates is currently failing due to a Python dep (or pytest) issue | 10:44 |
| * tristan is thinking there is *some* behavioral change around then which might allow us to figure out why we have this side effect :-S | 10:44 | |
| tristan | Or ! | 10:45 |
| tristan | juergbi, Do we have a bunch of more jobs since then ? | 10:45 |
| tristan | Maybe... you know these restful API people love to not return all results for each query | 10:46 |
| tristan | Maybe we need more than one request for each commit in order to collect all the jobs for a given commit | 10:46 |
| juergbi | it might be related to the very confusing way gitlab shows pipelines on the website | 10:47 |
| juergbi | linking to the overnight pipeline instead of the post-merge master pipeline | 10:47 |
| juergbi | however, here I don't see any fatal pipeline issues https://gitlab.com/BuildStream/buildstream/-/merge_requests/1864 | 10:47 |
| tristan | Maaaaybe but as I recall the docs has always been "except: schedules" | 10:48 |
| juergbi | fedora-update fails as expected right now, and remote-execution fails (have to check that) | 10:48 |
| juergbi | but docs job looks fine and it executed pages deployment in the same pipeline | 10:48 |
| juergbi | tristan: yes but the scheduled pipeline is a pipeline for the same commit | 10:49 |
| juergbi | so there are cases where gitlab only shows the latest pipeline for a commit and that can be a scheduled one (and thus, without docs) | 10:49 |
| juergbi | maybe there is a similar issue on programmatic access | 10:49 |
| tristan | I see what you mean... so it could be a bug where since we had many commits per day, we had a better chance to land on a commit which was not "scheduled" | 10:50 |
| tristan | Ah | 10:50 |
| tristan | Nice point | 10:50 |
| tristan | So yes, the API is currently looking up jobs from commits, so it would make sense that we have similar structure as viewing the pipeline which succeeded/failed for a given MR | 10:51 |
| tristan | But in the pasted output, it starts looking up master at line 63 | 10:52 |
| tristan | first unsuccessful | 10:52 |
| tristan | those look like regular CI jobs | 10:52 |
| tristan | And 20 is a suspicious number | 10:53 |
| tristan | oddly round, like we only got one batch of results... | 10:53 |
| *** lachlan has quit IRC | 10:53 | |
| tristan | gitlab also likes to only display 20 issues/merge requests per page | 10:54 |
| juergbi | ah, that could make sense | 10:55 |
| juergbi | only see 20 jobs of that commit | 10:55 |
| * tristan is perusing https://python-gitlab.readthedocs.io/en/stable/gl_objects/pipelines_and_jobs.html#jobs ... | 10:55 | |
| juergbi | so maybe the buildbox-run job put it over the 20 | 10:55 |
| tristan | could be that the python wrapper is not all that smart... | 10:55 |
| tristan | or we're not using the right cursor-like API | 10:56 |
| juergbi | tristan: you can pass kwarg all=True on the list operation | 10:58 |
| juergbi | alternatively, you might also be able to filter with `name="docs"` | 10:59 |
| tristan | Oh nice find | 10:59 |
| tristan | juergbi, where would I try filtering, did you find proper API reference ? | 11:00 |
| * tristan still swimming... | 11:00 | |
| juergbi | https://python-gitlab.readthedocs.io/en/stable/api/gitlab.html#gitlab.mixins.ListMixin.list | 11:00 |
| juergbi | **kwargs – Extra options to send to the server (e.g. sudo) | 11:00 |
| juergbi | and here it says `name` for filter: https://docs.gitlab.com/ee/api/commits.html#list-the-statuses-of-a-commit | 11:00 |
| juergbi | documentation is not very coherent | 11:01 |
| juergbi | benschubert: with a new CI runner test_casd_redirects_stderr_to_file_and_rotate is flaky for some reason. do you have an idea why? https://gitlab.com/BuildStream/buildstream/-/jobs/518213278#L2170 | 11:03 |
| benschubert | let me have a look | 11:03 |
| tristan | juergbi, that worked, thanks for finding that in the docs :) | 11:03 |
| tristan | filter by name="docs" directly in the list() method | 11:03 |
| juergbi | great | 11:04 |
| benschubert | juergbi: the 'time.sleep' might be too little for those new runners. that's the only explanation I would have | 11:04 |
| benschubert | .1s might be better or .2s | 11:04 |
| juergbi | ah, right, missed that | 11:04 |
| juergbi | that makes sense. the runner executes multiple jobs in a single (big) VM, so timing might be less consistent | 11:05 |
| benschubert | ah, then .5s then? :D | 11:05 |
| juergbi | yes, let's set it to .5s | 11:06 |
| juergbi | will add this to my other flaky test MR !1870 | 11:06 |
| gitlab-br-bot | MR !1870: tests/sources/tar.py: Fix flaky test_out_of_basedir_hardlinks https://gitlab.com/BuildStream/buildstream/-/merge_requests/1870 | 11:06 |
| juergbi | thanks for the hint | 11:06 |
| *** phoenix has quit IRC | 11:07 | |
| tristan | https://gitlab.com/BuildStream/docs-website/-/merge_requests/4 | 11:11 |
| tristan | gah, the pipeline will fail here | 11:12 |
| *** lachlan has joined #buildstream | 11:13 | |
| tristan | heh, we can't really do pre-merge CI on the docs-website repo due to the possibility of exposing the token ? | 11:13 |
| *** lachlan has quit IRC | 11:17 | |
| tristan | test_failed_build_quit[continue-interactive/failed-build.bst] | 11:17 |
| tristan | This is different than hardlink issue right: https://gitlab.com/BuildStream/buildstream/-/jobs/518024312 ? | 11:18 |
| juergbi | yes but seems timing related like the casd log issue | 11:19 |
| juergbi | *sigh* | 11:19 |
| juergbi | may have to increase timeouts in more places | 11:20 |
| tristan | its housecleaning week :) | 11:21 |
| tristan | coincidentally in spring | 11:21 |
| WSalmon | <juergbi> may have to increase timeouts in more places <- when i added a time out for the file system i added a function so all the time outs were set in one place, i presume juergbi is talking about something else | 11:22 |
| WSalmon | * not a time out a slepp | 11:23 |
| juergbi | WSalmon: I'm talking about not filesystem-related timeouts we have in certain tests | 11:23 |
| WSalmon | sorry for the noise | 11:24 |
| juergbi | the filesystem-related granularity sleeps shouldn't be affected by shared CI runner | 11:24 |
| juergbi | no problem | 11:24 |
| tristan | tada ! https://docs.buildstream.build/master/main_using.html | 11:24 |
| juergbi | \o/ | 11:24 |
| tristan | propagation of docs changes working again :) | 11:24 |
| juergbi | tristan: is it intentional that the example sessions are not automatically regenerated? https://docs.buildstream.build/master/examples/flatpak-autotools.html#build-the-hello-bst-element | 11:26 |
| tristan | juergbi, nope, that's another issue | 11:27 |
| tristan | juergbi, two issues actually... for some reason, the session generation failing is not causing the docs job to fail, the other being that that session fails | 11:28 |
| juergbi | ah | 11:28 |
| tristan | it fails because of lack of ostree plugin in the required plugins I believe | 11:28 |
| tristan | probably best to fix both at once... | 11:29 |
| * tristan does more housecleaning | 11:29 | |
| tristan | in tox.ini we have: git+https://gitlab.com/BuildStream/bst-plugins-experimental.git@be5ac19e5062bc23a46ed8ce7aa2958a2653c917#egg=bst_plugins_experimental[ostree] | 11:31 |
| tristan | for the docs | 11:31 |
| * tristan suspects it is not working | 11:31 | |
| tristan | weird, it's there in .tox/docs/lib/python3.7/site-packages/bst_plugins_experimental-0.12.0.dist-info/entry_points.txt, but running that session bails out with: Pip package bst-plugins-experimental does not contain a plugin named 'ostree' | 11:35 |
| traveltissues | tristan: i think ostree wasn't in that version | 11:47 |
| *** lachlan has joined #buildstream | 11:47 | |
| traveltissues | it's registered in 0.13.0 | 11:47 |
| * WSalmon wonders why the docs use a fixed commit but the tests have ` py{36,37,38}-plugins: git+https://gitlab.com/buildstream/bst-plugins-experimental.git@{env:BST_PLUGINS_EXPERIMENTAL_VERSION:{[config]BST_PLUGINS_EXPERIMENTAL_VERSION}}#egg=bst_plugins_experimental[ostree,deb] | 11:49 | |
| WSalmon | ` | 11:49 |
| gitlab-br-bot | DouglasWinship opened issue #1288 (Prevent buildstream sandbox folders from appearing in checked out artifacts) on buildstream https://gitlab.com/BuildStream/buildstream/-/issues/1288 | 11:51 |
| tristan | traveltissues, I'm a bit further along debugging | 11:52 |
| tristan | traveltissues, ostree is certainly in that version, it's installed at .tox/docs/lib/python3.7/site-packages/bst_plugins_experimental/sources/ostree.py | 11:52 |
| tristan | I have a checkout of that version beside bst | 11:52 |
| *** lachlan has quit IRC | 11:53 | |
| tristan | Also, .tox/docs/lib/python3.7/site-packages/bst_plugins_experimental-0.12.0.dist-info/entry_points.txt lists it, and I'm also looking at setup.py from that version of bst-plugins-experimental | 11:54 |
| traveltissues | i don't see ostree here: https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/0.12.0/setup.py maybe i'm overlooking something | 11:54 |
| tristan | Looking at the loader code, I'm suspecting that we have some disparity in what we want the "group" to be called, i.e. in pkg_resources.get_entry_info() https://setuptools.readthedocs.io/en/latest/pkg_resources.html | 11:55 |
| tristan | tox.ini says be5ac19e5062bc23a46ed8ce7aa2958a2653c917 | 11:56 |
| tristan | So I'm looking at https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/be5ac19e5062bc23a46ed8ce7aa2958a2653c917/setup.py essentially | 11:57 |
| tristan | Now here's the thing... BuildStream considers sources and elements as separate namespaces | 11:57 |
| tristan | Same for separate projects | 11:57 |
| tristan | One project can have the word 'git' used as a source and as an element, and they will not conflict, a junctioned project may also have 2 additional 'git' plugins, one source and one element | 11:58 |
| tristan | So, BuildStream asks pkg_resources.get_entry_info('bst-plugins-experimental', 'buildstream.plugins.sources', 'ostree') and thus fails to find it | 11:59 |
| tristan | But this version at least of bst-plugins-experimental, as we can see here: https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/be5ac19e5062bc23a46ed8ce7aa2958a2653c917/setup.py#L56 | 11:59 |
| tristan | Crams all of it's plugins into a single "buildstream.plugins" group, thinking it's okay to use the same namespace | 12:00 |
| tristan | This would presumably work fine, if setup.py was putting elements into "buildstream.plugins.elements" and sources into "buildstream.plugins.sources" groups | 12:01 |
| tristan | The key here is to find out what went wrong where | 12:01 |
| tristan | it looks like bst-plugins-experimental has been using the word "buildstream.plugins" since the initial commit | 12:03 |
| tristan | and it looks like bst-external is also using that | 12:04 |
| tristan | So the namespaces should still "just work" regardless of this I suspect, because we load sources and elements from separate contexts (using pluginsbase) | 12:04 |
| tristan | hrrrm | 12:04 |
| tristan | cs-shadow broke this with b4d472e9c2 | 12:06 |
| *** lachlan has joined #buildstream | 12:07 | |
| tristan | seems like the "right thing", but now where are we supposed to load ostree from ? | 12:07 |
| tristan | oh, well it appears master has it | 12:08 |
| valentind | juergbi, your branch is running now on this pipeline: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/pipelines/137847189 | 12:14 |
| valentind | We will see how it goes. | 12:14 |
| tristan | I see, and now some Makefile foo, it looks like GNU Make's special $(foreach ...) is not appropriate for this | 12:14 |
| juergbi | valentind: ok, ta | 12:15 |
| gitlab-br-bot | juergbi merged MR !1870 (juerg/tar-hardlinks->master: tests/sources/tar.py: Fix flaky test_out_of_basedir_hardlinks) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1870 | 12:26 |
| gitlab-br-bot | tristanvb opened MR !1871 (tristan/fix-doc-builds->master: Documentation build fixes) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1871 | 12:47 |
| tristan | juergbi, ^^^ That'll fix the flatpak example, and ensure that we don't have unnoticed breakage there in the future | 12:47 |
| juergbi | great, taking a look | 12:48 |
| tristan | oh wait... we still use marge right ? or do we ? | 13:03 |
| juergbi | tristan: yes, we normally do, it should normally be working fine now (assuming no CI issues) | 13:06 |
| juergbi | tristan: had a second comment, sorry, was a bit late | 13:06 |
| tristan | No worry, I did have that thought that I could just depend on sessions-prep too, but did not consider parallelism | 13:08 |
| *** lachlan has quit IRC | 13:30 | |
| *** lachlan has joined #buildstream | 13:44 | |
| gitlab-br-bot | seanborg opened MR !1872 (seanborg/documentation-typos->master: Fix typo namspaceing -> namespaceing) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1872 | 14:23 |
| gitlab-br-bot | coldtom approved MR !1872 (seanborg/documentation-typos->master: Fix typo namspaceing -> namespaceing) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1872 | 14:24 |
| gitlab-br-bot | marge-bot123 closed issue #1276 (BuildStream build fails if the CAS is missing blobs) on buildstream https://gitlab.com/BuildStream/buildstream/-/issues/1276 | 14:57 |
| gitlab-br-bot | marge-bot123 merged MR !1843 (juerg/artifact-blob-not-found->master: Fix handling of missing blobs in `ArtifactCache.pull()`) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1843 | 14:57 |
| valentind | juergbi, The cache server does not seem to work. So it is hard to know if it is working. | 15:03 |
| valentind | It pushes, but it does not pull the artifacts back. | 15:03 |
| valentind | They seem to go to /dev/null. | 15:04 |
| juergbi | valentind: that's with the buildbarn CAS server, right? | 15:07 |
| juergbi | I don't know too much about it. iirc, it uses a ring buffer. I assume it's big enough for the current purpose? | 15:08 |
| juergbi | our push/pull tests in CI are working but that's with bst-artifact-server | 15:09 |
| juergbi | the remote execution tests are working as well with buildgrid CAS server | 15:09 |
| juergbi | (the latter still use bst-artifact-server for push/pull, though) | 15:10 |
| valentind | I think it is buildbarn. I did not set it up. | 15:10 |
| juergbi | it would be good to test with buildbarn as well but it may not be trivial in the test suite | 15:11 |
| coldtom | juergbi, valentind, it's currently using Redis for small blobs and S3 for big blobs | 15:14 |
| juergbi | thanks for the info | 15:15 |
| coldtom | where small is <4MiB and big is >=4MiB | 15:15 |
| juergbi | coldtom: have you (or someone else) tested basic buildstream push/pull functionality with buildbarn as CAS? | 15:16 |
| coldtom | juergbi, yes, and it does work generally | 15:16 |
| coldtom | using redis and s3 like this has proven a bit fragile though | 15:16 |
| *** lachlan has quit IRC | 15:19 | |
| *** lachlan has joined #buildstream | 15:32 | |
| *** lachlan has quit IRC | 15:38 | |
| douglaswinship | I've got a plugin to propose adding to bst-plugins-experimental, (and then back-port to bst-external), but I could use some advice on what to do for testing. | 15:46 |
| douglaswinship | https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/92 | 15:46 |
| douglaswinship | The plugin is called url_manifest. It generates a manifest of elements and their recursive dependencies, listing url-related information about their sources. | 15:47 |
| douglaswinship | So a good test would be to provide a basic buildstream project with an url_manifest element, and build the manifest. | 15:47 |
| douglaswinship | But I've never used pytest before and I don't understand the testing framework that's already in place. | 15:48 |
| douglaswinship | Contributing.rst says that "At minimum, instructions | 15:48 |
| douglaswinship | Contributing.rst says that "At minimum", there should be "instructions on how the maintainer can test it for themselves." But where would I put those instructions? | 15:49 |
| juergbi | douglaswinship: at first glance the main concern with this plugin is that it's not generic, it relies on source plugin-specific variables and assumes those internals existing based on the source kind string, which is not guaranteed | 15:51 |
| juergbi | the same applies to collect_manifest, even though that's already merged | 15:51 |
| WSalmon | douglaswinship, many tests have a little exmple project, i would just copy a existing test and modify it | 15:51 |
| juergbi | however, it may make sense to first have a general discussion about such *_manifest plugins before working on tests | 15:52 |
| juergbi | e.g., maybe the generic Source API can be extended to cover these use cases | 15:52 |
| juergbi | and if it no longer contains plugin-specific code, testing should be much easier | 15:53 |
| coldtom | i guess the Source API could require some form of URI field, i think that's the main reason why plugin-specific code exists in *_manifest | 15:55 |
| WSalmon | look how this test runs bst, then checks out a element and then inspects what was checked out https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/tests/integration/collect_integration.py | 15:56 |
| *** lachlan has joined #buildstream | 15:56 | |
| valentind | coldtom, Multiple uris. | 15:56 |
| juergbi | coldtom: the Source API extension we need for #1275 might help for this use case as well | 15:56 |
| gitlab-br-bot | Issue #1275: Introduce tracking of individual sources, using the Remote Asset API https://gitlab.com/BuildStream/buildstream/-/issues/1275 | 15:56 |
| valentind | juergbi, Can we backport that to 1.4? | 15:57 |
| juergbi | we don't even have it in master yet, so difficult to say, but probably not | 15:57 |
| juergbi | although 1.4 plugins might still be able to follow a similar pattern | 15:58 |
| juergbi | e.g., 1.4 plugins could optionally implement these extra methods and 1.4 *manifest plugins could check whether that method is implemented for each plugin | 15:59 |
| juergbi | for details we need more concrete plans for master first | 15:59 |
| *** rdale has quit IRC | 16:03 | |
| douglaswinship | WSalmon: thanks for the link, but this is where I expose a critical lack of understanding about pytest and how these tests are being run. From what I can see, the file that you sent me doesn't actually _do_ anything at all. It defines a lot of functions, but I don't see where it ever actually calls any of them. | 16:03 |
| douglaswinship | And a quick grep doesn't seem to suggest that there's any other files that call them. | 16:05 |
| WSalmon | so when you run pytest it look is a diretory strucuture, this case `test/` for python files with functions that start `test` | 16:05 |
| douglaswinship | I'll do some reading on it in future. | 16:05 |
| douglaswinship | It's not sounding like this particular MR is likely to be accepted in anything like it's current form anyway, so I may not create any tests for it right now. | 16:07 |
| WSalmon | douglaswinship, its east to run tox in the root of the plugins folder, that will trigger https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/tox.ini#L16 that will run pytest with all the right bits installed in a venv | 16:07 |
| coldtom | anyone seen anything like this before: https://paste.gnome.org/pvquqas7h using latest master | 16:07 |
| juergbi | coldtom: no, this might be an issue with a particular plugin | 16:08 |
| juergbi | I vaguely recall a ujson issue but I think that was resolved a long time ago | 16:10 |
| juergbi | or maybe you can still hit it by using too old version of ujson | 16:10 |
| coldtom | looks like an issue with flatpak_image | 16:17 |
| coldtom | or git_tag | 16:17 |
| *** lachlan has quit IRC | 16:20 | |
| coldtom | flatpak_image is the issue, i'll figure out a patch | 16:24 |
| *** CTtpollard has joined #buildstream | 16:32 | |
| *** tpollard has quit IRC | 16:32 | |
| *** lachlan has joined #buildstream | 16:37 | |
| *** lachlan has quit IRC | 16:46 | |
| *** lachlan has joined #buildstream | 17:00 | |
| *** lachlan has quit IRC | 17:04 | |
| *** lachlan has joined #buildstream | 17:18 | |
| *** lachlan has quit IRC | 17:21 | |
| *** mohan43u has quit IRC | 17:22 | |
| *** mohan43u has joined #buildstream | 17:26 | |
| *** CTtpollard has quit IRC | 17:26 | |
| *** lachlan has joined #buildstream | 17:35 | |
| *** mohan43u has quit IRC | 17:41 | |
| *** lachlan has quit IRC | 17:42 | |
| *** toscalix has joined #buildstream | 17:43 | |
| *** mohan43u has joined #buildstream | 17:44 | |
| *** santi has quit IRC | 17:50 | |
| *** lachlan has joined #buildstream | 17:55 | |
| *** lachlan has quit IRC | 18:04 | |
| *** toscalix has quit IRC | 18:21 | |
| *** toscalix has joined #buildstream | 18:27 | |
| *** narispo has quit IRC | 19:37 | |
| *** narispo has joined #buildstream | 19:37 | |
| *** benschubert has quit IRC | 19:38 | |
| gitlab-br-bot | marge-bot123 merged MR !1847 (tristan/pip-source-3.8->master: pip source plugin: Add support for python 3.8) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1847 | 19:40 |
| *** narispo has quit IRC | 20:18 | |
| *** narispo has joined #buildstream | 20:18 | |
| *** narispo has quit IRC | 21:07 | |
| *** narispo has joined #buildstream | 21:07 | |
| *** tristan has quit IRC | 21:13 | |
| *** toscalix has quit IRC | 21:33 | |
| *** juergbi has quit IRC | 22:25 | |
| *** juergbi has joined #buildstream | 22:25 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!