| gitlab-br-bot | buildstream: merge request (jjardon/misc_fixes->master: Dome little documentation fixes) #326 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/326 | 00:13 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (jjardon/install_debian->master: source/install.rst: put Debian version under the same subsection) #325 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/325 | 00:14 |
| gitlab-br-bot | buildstream: merge request (jjardon/misc_fixes->master: Some little documentation fixes) #326 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/326 | 00:14 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 00:34 |
| gitlab-br-bot | buildstream: merge request (jjardon/theme_change->master: WIP: Change theme to sphinx_rtd_theme) #327 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/327 | 01:05 |
| gitlab-br-bot | buildstream: merge request (jjardon/theme_change->master: WIP: Change theme to sphinx_rtd_theme) #327 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/327 | 01:08 |
| gitlab-br-bot | buildstream: merge request (jjardon/theme_change->master: Change theme to sphinx_rtd_theme) #327 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/327 | 01:08 |
| *** tristan has quit IRC | 02:33 | |
| *** tristan has joined #buildstream | 04:01 | |
| *** valentind has joined #buildstream | 08:27 | |
| *** toscalix has joined #buildstream | 08:57 | |
| gitlab-br-bot | buildstream: issue #283 ("New sequence ID in messages cannot work") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/283 | 09:20 |
| toscalix | tristan: the main reason for mirroring the sources is not tactical, it is legal (compliance) in embedded since you are always the distributor when providing gpl software on a device | 09:31 |
| toscalix | in services the situation is different | 09:32 |
| tristan | toscalix, it's a good point you raised | 09:35 |
| tristan | toscalix, I would argue that "the main reason" is going to be different, depending on use cases | 09:35 |
| toscalix | ... use case AND license. | 09:38 |
| *** dominic has joined #buildstream | 09:38 | |
| tristan | toscalix, true :) | 09:38 |
| toscalix | good morning | 09:38 |
| toscalix | :-) Waking up with compliance stuff on Monday is.... far from ideal | 09:39 |
| tristan | toscalix, in fact this is probably something interesting to consider for distribution mechanisms like.. flatpak | 09:39 |
| *** abderrahim[m] has quit IRC | 09:39 | |
| *** jonathanmaw has joined #buildstream | 09:41 | |
| toscalix | flatpak is just another way of distributing software so yes, the source code should be shipped with the binaries and/or made available if the code has copyleft licenses | 09:41 |
| toscalix | so mirroring is a must in those cases | 09:41 |
| toscalix | interesting is the point "publically available" | 09:42 |
| toscalix | when working at SUSE, we received a few letters with every release of people requesting the CD. | 09:43 |
| tristan | toscalix, just saying, because compliance issues surrounding distribution in the context of flatpak was something discussed at GUADEC | 09:43 |
| toscalix | was it recorded? | 09:44 |
| tristan | Surely it was | 09:44 |
| toscalix | I haven seen a video about it. I would watch it | 09:44 |
| toscalix | will look for it, thanks | 09:44 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 09:44 |
| *** abderrahim[m] has joined #buildstream | 09:45 | |
| tristan | toscalix, https://2017.guadec.org/talks-and-events/index.html#abstract-1-resurrecting_dinosaurs_what_can_possibly_go_wrong | 09:45 |
| toscalix | ah, thanks | 09:54 |
| toscalix | ah, wait, I saw this one from Richard | 09:55 |
| toscalix | already | 09:55 |
| *** tiago has joined #buildstream | 10:00 | |
| *** aday has joined #buildstream | 10:03 | |
| *** tiago has quit IRC | 10:08 | |
| *** tiago has joined #buildstream | 10:10 | |
| *** valentind has quit IRC | 10:15 | |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 10:18 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 10:27 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 10:29 |
| gitlab-br-bot | buildstream: merge request (jjardon/theme_change->master: Change theme to sphinx_rtd_theme) #327 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/327 | 10:34 |
| gitlab-br-bot | buildstream: merge request (jjardon/misc_fixes->master: Some little documentation fixes) #326 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/326 | 10:36 |
| gitlab-br-bot | buildstream: merge request (jjardon/misc_fixes->master: Some little documentation fixes) #326 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/326 | 10:39 |
| gitlab-br-bot | buildstream: issue #306 ("Follow-up from "Some little documentation fixes"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/306 | 10:39 |
| gitlab-br-bot | buildstream: merge request (tristan/project-refs->master: WIP: Optionally load and store source references in project.refs) #314 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/314 | 10:41 |
| gitlab-br-bot | buildstream: merge request (tristan/project-refs->master: Optionally load and store source references in project.refs) #314 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/314 | 10:41 |
| tristan | juergbi, if you can take a look at MR 314 above that would be cool :) | 10:41 |
| juergbi | will do | 10:42 |
| tristan | juergbi, I wont be around for the evening, going to leave soonish for dinner, so... take your time anyway | 10:46 |
| Nexus | DId something change on the docs over the weekend? My CI keeps failing on them now | 10:46 |
| Nexus | Can someone please run the CI pipeline on a know working + up to date with master branch please? | 11:06 |
| ltu | Nexus, https://gitlab.com/BuildStream/buildstream/merge_requests/322 | 11:06 |
| ltu | are you subscribed to the email notifications? you can then see all the updates | 11:07 |
| Nexus | ltu: i don't get emails about merges for some reason | 11:10 |
| Nexus | but from what i can see, it looks like all CI's are failing the same way atm | 11:10 |
| juergbi | yes, not sure what's happening | 11:10 |
| juergbi | maybe some issue with the CI runner cache due to changes in branches? | 11:11 |
| ltu | Nexus, go here and ensure it's set to 'Watch' for the whole project, all repos - https://gitlab.com/BuildStream | 11:11 |
| Nexus | oh good, not just me then. Maybe? It seemed to start at around the time the docs were updated? | 11:11 |
| juergbi | can clear the runner cache but if we haven't fixed the cause, it will happen again | 11:12 |
| juergbi | i don't see a connection with the changes in master | 11:12 |
| Nexus | ltu: was set to global :) | 11:12 |
| juergbi | i'll flush the cache. first pipeline after that will take a bit longer, though | 11:12 |
| juergbi | let's see: https://gitlab.com/BuildStream/buildstream/-/jobs/58161841 | 11:13 |
| Nexus | failed | 11:14 |
| Nexus | can we run CI on master? | 11:14 |
| Nexus | see if that passed | 11:14 |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 11:14 |
| tlater | Nexus: Yeah, just open a pipeline | 11:15 |
| Nexus | Insufficient permissions for protected ref 'master | 11:15 |
| tlater | Nexus: My CI is also failing, just rebased: https://gitlab.com/BuildStream/buildstream/-/jobs/58148164 | 11:15 |
| Nexus | grrrr | 11:15 |
| tlater | And here's a pipeline on master: https://gitlab.com/BuildStream/buildstream/pipelines/19118837 | 11:15 |
| Nexus | that awkward moment when master fails... | 11:25 |
| tlater | Yep, definitely some screwup with the pytest config | 11:25 |
| tlater | jennis: Did you last touch this? I saw some commits from over the weekend, didn't check exactly what they did, but they seemed to be unrelated. | 11:26 |
| tlater | otoh, my branch from above passed fine before I rebased, so perhaps I should double check | 11:26 |
| juergbi | latest commit on master did succeed, though | 11:26 |
| juergbi | in original pipeline | 11:26 |
| juergbi | could it be some Fedora or PyPI update? | 11:27 |
| juergbi | would be odd | 11:27 |
| jennis | tlater: nope :/ | 11:27 |
| tlater | juergbi: Our fedora version is fixed | 11:27 |
| juergbi | $ dnf install -y python2 | 11:28 |
| juergbi | Fedora 27 - x86_64 - Updates 3.5 MB/s | 21 MB 00:05 | 11:28 |
| tlater | And we have a flag set that tells pip not to install new versions of python packages | 11:28 |
| tlater | We also get the same error on debian: https://gitlab.com/BuildStream/buildstream/-/jobs/58148166 | 11:28 |
| tlater | (debian-8, that is) | 11:29 |
| juergbi | right, just saw that | 11:29 |
| tlater | The only changes I made to my branch since the last passing CI were docs-related :| | 11:32 |
| tlater | So something must have changed in the CI image | 11:33 |
| *** tristan has quit IRC | 11:34 | |
| tlater | Tests also fail offline | 11:34 |
| jennis | `./setup.py test` is failing for you? | 11:38 |
| jennis | on master branch? | 11:38 |
| tlater | jennis: Hm, no, I think I missed a flag | 11:38 |
| jennis | ahh ok | 11:38 |
| jennis | phew | 11:39 |
| tlater | So tests seem to be running fine outside the CI | 11:39 |
| * tlater wonders what in the world could cause this error | 11:39 | |
| Nexus | my local tests fail | 11:39 |
| tlater | Didn't check for docs, but the other ones pass fine | 11:39 |
| tlater | Freshly-cloned buildstream repo :) | 11:39 |
| tlater | Also up-to-date fedora image | 11:40 |
| tlater | Hrm | 11:40 |
| jonathanmaw | tried running tests in my branch with `python3 setup.py test --addopts --integration` and it's gotten a lot further than it does in CI | 11:42 |
| jonathanmaw | I suspect that in CI it's not reading conftest.py, but I have no idea why that would be the case | 11:42 |
| tlater | Only thing that would cause this is a breaking change in pytest-runner, but our index-url should mean that that's impossible | 11:43 |
| tlater | There *was* a new release of that yesterday | 11:45 |
| juergbi | tlater: have you already tried local run after sdist? | 11:47 |
| juergbi | maybe sdist suddenly fails somehow | 11:47 |
| tlater | juergbi: No, that's definitely worth a try | 11:48 |
| * tlater wonders if perhaps setup.cfg isn't caught in the sdist | 11:48 | |
| tlater | Err, config.py | 11:48 |
| juergbi | and possibly source/plugin.rsttemplate | 11:49 |
| juergbi | usage: setup.py [options] [file_or_dir] [file_or_dir] [...] | 11:51 |
| juergbi | setup.py: error: unrecognized arguments: --integration | 11:51 |
| juergbi | i see the same issue locally when using sdist | 11:51 |
| juergbi | various files are missing in the dist tarball | 11:52 |
| tlater | That would explain it | 11:52 |
| juergbi | do we need to add them to the MANIFEST? | 11:52 |
| juergbi | wondering why this suddenly happens, though, change in setuptools? | 11:52 |
| tlater | Shouldn't be possible, that's part of the fedora image | 11:53 |
| tlater | Unless, as part of sdist, we happen to upgrade python packaged | 11:53 |
| tlater | *packages | 11:53 |
| tlater | Ah, we actually install pytest-pylint in the .gitlab-ci.yaml | 11:54 |
| tlater | That's probably where this goes wrong | 11:54 |
| tlater | That should definitely be in the base image | 11:54 |
| juergbi | ok but it doesn't update setuptools, afaict | 11:54 |
| tlater | juergbi: It *does* install a new version of pytest-pylint | 11:56 |
| tlater | Although we've kind of asserted that this is an sdist issue... | 11:56 |
| juergbi | i see this new sdist error in the failing pipelines | 11:57 |
| juergbi | /usr/bin/python3: Error while finding module specification for 'setuptools_scm.git' (ModuleNotFoundError: No module named 'setuptools_scm') | 11:57 |
| juergbi | warning: no files found matching 'buildstream.doap' | 11:58 |
| juergbi | and it no longer copies .coveragerc, .gitignore, .gitlab-ci.yml, .pylintrc, conftest.py... | 11:59 |
| tlater | Are you sure those are new? I think I've seen them before | 11:59 |
| juergbi | does setuptools_scm ignroe MANIFEST and use the files from git | 11:59 |
| juergbi | so for some reason that fails and that is the cause for the rest? | 11:59 |
| juergbi | the question is why setuptools_scm.git is no longer available | 11:59 |
| juergbi | i don't see the setuptools_scm error in sdist of older (working) pipelines | 12:00 |
| tlater | Did you install pytest-pylint on your local machine before running sdist? | 12:01 |
| juergbi | ah no | 12:02 |
| tlater | Well, at least that can't be the cause | 12:03 |
| juergbi | docker sha changed for fedora:master-42. is this normal? | 12:07 |
| * tlater certainly doesn't think so | 12:07 | |
| juergbi | Using Docker executor with image buildstream/buildstream-fedora:master-42-571406d ... | 12:08 |
| juergbi | Using docker image sha256:4f4b0d740d75bd96300511258623d8f3e71a4adcdf7e5d9cc44a6cfd1accaf67 for predefined container... | 12:08 |
| tlater | To do so someone would have had to change the docker image tag | 12:08 |
| juergbi | Using Docker executor with image buildstream/buildstream-fedora:master-42-571406d ... | 12:08 |
| juergbi | Using docker image sha256:77217aff628255bf8a33973d43c55fa071bc2595acff0335d8d7b59de99aef97 for predefined container... | 12:08 |
| tlater | Which is very strictly verboten in our repo | 12:08 |
| juergbi | Using Docker executor with image buildstream/buildstream-fedora:master-42-571406d ... | 12:08 |
| juergbi | Using docker image sha256:c9367966a6dacb964e576ab6c6338bbb7d31137d7766904dabace398f9f81387 for predefined container... | 12:08 |
| juergbi | it seems to change all the time | 12:08 |
| tlater | Hm | 12:08 |
| tlater | It definitely shouldn't | 12:08 |
| tlater | We *did* have a commit to the image a few days ago | 12:09 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 12:09 |
| tlater | Nexus: When you changed the fedora image, did you also change the tag buildstream is using? | 12:11 |
| Nexus | i believe so | 12:11 |
| tlater | I'm assuming that change hasn't made it to master yet? | 12:11 |
| juergbi | no, it hasn't | 12:11 |
| juergbi | tlater: i can fix the issue locally by pip3 install setuptools_scm before sdist | 12:12 |
| juergbi | we should probably add that to the image | 12:12 |
| juergbi | (or put it in .gitlab-ci) | 12:12 |
| * tlater updates the image to include that and add pytest-pylint | 12:12 | |
| juergbi | i have no idea why this used to work, though, and suddenly fails | 12:12 |
| tlater | :[ | 12:12 |
| tlater | Even weirder that it also happens in the debian-8 image | 12:13 |
| juergbi | some dependency change in pypi | 12:14 |
| juergbi | i.e., it was implicitly pulled in before but no longer | 12:14 |
| tlater | This probably means that our --index-url hack doesn't actually work | 12:15 |
| juergbi | no idea about that part | 12:16 |
| gitlab-br-bot | buildstream: issue #307 ("Follow-up from "Add getting started section"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/307 | 12:27 |
| gitlab-br-bot | buildstream: issue #308 ("Follow-up from "Add getting started section"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/308 | 12:28 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 12:32 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 12:34 |
| Nexus | Hows it going? | 12:38 |
| tlater | Waiting for CI to finish | 12:39 |
| tlater | Should be able to then merge to bst-external | 12:39 |
| tlater | And then try on buildstream | 12:39 |
| Nexus | kk, who's the best person to ask about https://gitlab.com/BuildStream/buildstream/issues/21 ??? | 12:39 |
| tlater | After that all branches experiencing this will have to rebase to master | 12:39 |
| tlater | Nexus: juergbi knows most about the incremental build features, I think | 12:40 |
| Nexus | juergbi: o/ | 12:40 |
| tlater | Don't think anyone else about was involved in the discussions about it, unfortunately | 12:40 |
| tlater | Though if you have specific questions, I can maybe help a bit | 12:41 |
| Nexus | tlater: 1) How should it be cached? (copy the dir/contents, or something more complex ) 2) Where should it be cached? 3) Is it something that should be optional? if so how? 4) Max size of cache? 5) Max num of versions in cache? 6) Cache everything or specific files? (customisable?) 7) How should cache deletion be handled? | 12:43 |
| tlater | On everything caching-related - you'd probably like to read into this https://gitlab.com/BuildStream/buildstream/tree/master/buildstream/_artifactcache | 12:44 |
| tlater | Max size of the cache is not your concern, if you're using that interface | 12:45 |
| tlater | Since the point of this is to share these artifacts, they should probably have some metadata telling us that they should not be used for normal builds | 12:46 |
| tlater | So, you'd probably want to change element.py, modify it to *also* push workspaced artifacts. I think you can reuse the metadata that says they are workspaced. | 12:48 |
| tlater | You won't need to worry about what to cache or anything, this is already done | 12:49 |
| * tlater isn't sure whether the remote caches already support reading artifact metadata to make sure we don't download an artifact we don't want for a build - this is something juergbi will have to answer :| | 12:50 | |
| tlater | Nexus: for reference, this is where we create element artifacts: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1097 | 12:53 |
| tlater | We copy everything that it puts in "collectdir" (used to be /buildstream/install in the sandbox): https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1118 | 12:53 |
| tlater | And then we add a little bit of metadata: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1137 | 12:54 |
| tlater | These days we *also* do that for workspaced elements that have partially-done builds, which is what that issue is about - so you won't need to worry about how we get these sorts of artifacts into the cache :) | 12:55 |
| tlater | You just need to think about how they are distributed | 12:55 |
| tlater | Which there's the whole "push" interface for - I know a bit less about it, but it should be fairly accessible. | 12:55 |
| tlater | jonathanmaw: Would you mind having a look at https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/28 ? | 12:58 |
| tlater | With any luck that and an appropriate commit to buildstream should fix our CI issues | 12:59 |
| gitlab-br-bot | buildstream: issue #309 ("All builds are failing") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/309 | 13:03 |
| juergbi | tlater: why do you want to push workspaced artifacts? | 13:03 |
| tlater | juergbi: I think https://gitlab.com/BuildStream/buildstream/issues/21 is about that? | 13:04 |
| juergbi | i don't think so. the only reference to workspaces is that we want to be able to create a workspace from a previously successful build | 13:05 |
| juergbi | copying the cached build tree | 13:05 |
| juergbi | but that previous build would have been a regular non-workspace build | 13:05 |
| juergbi | or do you mean something else? | 13:05 |
| tlater | I was referring to the "on another machine" part | 13:06 |
| juergbi | that part is not related to workspaces, afaik | 13:06 |
| tlater | Hm, yeah, your interpretation does make more sense | 13:06 |
| juergbi | i.e., you can download build trees from a previous build on a build server | 13:06 |
| juergbi | but i don't think it makes sense to share build trees of workspaces | 13:07 |
| tlater | Yep, sorry, I misunderstood the issue | 13:07 |
| * Nexus is no closer to understanding his issue | 13:08 | |
| * tlater should leave explaining to juergbi, as he has a better overview | 13:08 | |
| juergbi | Nexus: 3) due to the artifact size increase (affecting not just local cache but also push upload size and remote cache), we might have to make it optional, although it would obviously be nice to have it always on | 13:09 |
| juergbi | (and later pull is also affected) | 13:10 |
| juergbi | if so, it should likely be a project config | 13:10 |
| Nexus | juergbi: opt out or opt in? | 13:10 |
| juergbi | overridable in per-project user config, but it may get tricky with remote artifact caches | 13:11 |
| juergbi | i.e., if we make it optional per client, we can't rely on remote artifact cache having build trees | 13:11 |
| juergbi | presence of build tree has to influence the cache key, i suppose | 13:11 |
| juergbi | Nexus: opt in/out, not sure | 13:12 |
| juergbi | we need to figure out whether we can make it manageable for regular community projects with limited resources | 13:12 |
| juergbi | Nexus: is (1) / (2) clear based on tlater's response or do you still need clarifications there? | 13:12 |
| Nexus | My understanding was that both of those would be answered by understanding how the existing caching works | 13:13 |
| juergbi | to be clear, the idea is to make it part of the artifact | 13:14 |
| juergbi | i.e., cache it the same way as build output | 13:14 |
| Nexus | can you expand on that please? | 13:14 |
| juergbi | an artifact currently consists of build output (files directory), logs, and metadata | 13:14 |
| juergbi | this would add a fourth directory for the build tree | 13:15 |
| juergbi | but wouldn't change anything in the basic caching approach | 13:15 |
| Nexus | so the artifact would be "Build Output", "Logs", Metadata" and "Build Tree Cache" ? | 13:15 |
| juergbi | yes | 13:16 |
| Nexus | kk | 13:16 |
| juergbi | while that is what we discussed a while ago, it's possible that another solution would be better in the end | 13:16 |
| juergbi | e.g., use the artifact cache but cache the build tree with a separate ref (branch/tag) | 13:17 |
| juergbi | this would e.g. allow avoiding pulling the build tree unless the client really needs it | 13:17 |
| juergbi | reducing bandwidth. but it would complicate things | 13:18 |
| juergbi | i'm not sure how big of an issue pull/push bandwidth is | 13:18 |
| juergbi | build trees can be pretty large | 13:19 |
| Nexus | 5) and 7)? | 13:21 |
| tlater | Nexus: On 7, there's a deletion interface for artifacts, iirc, or it will be added very soon by the various cache expiry branches | 13:23 |
| tlater | This issue isn't concerned about deleting though, that will happen automatically depending on conditions yet-to-be-introduced | 13:23 |
| tlater | Not sure what you mean by 5, frankly. Every build should store its tree along with the artifact - again, cache expiry will be handled by another part of buildstream, although perhaps we could think about just removing build trees at first. | 13:25 |
| Nexus | 4 and 5 were similar. Basically meaning, will we limit how many times it is cached before the old one gets removed. If i build the same thing 50x in a row, will i have 50 caches? if so, that could be huge | 13:27 |
| tlater | Nexus: Yep, that's a currently open issue - #135 | 13:27 |
| tlater | Currently that's buildstream's behavior, and why you need to rm -rf ~/.cache/buildstream every once in a while | 13:27 |
| tlater | The solution to #135 will solve that by automatically deleting artifacts when we don't think they're needed anymore - but that's entirely unrelated to #21 | 13:28 |
| juergbi | Nexus: please note that the (ostree) artifact doesn't store identical files multiple times. with reproducible builds 50 similar builds shouldn't take that much space in total | 13:28 |
| juergbi | *(ostree) artifact cache | 13:28 |
| Nexus | ah i see | 13:29 |
| tlater | This is a bigger issue for the tar cache - we're considering obsoleting it, though, with the bazel CAS | 13:29 |
| juergbi | yes, tar cache will also be very slow with build tree caching | 13:30 |
| tlater | Hm, maybe another one of the "disable when using tar cache" things | 13:32 |
| juergbi | if we anyway make it optional, i don't think we need to do anything special | 13:32 |
| tlater | Nexus: Are you clear on the issue now? | 13:34 |
| Nexus | i think i have as much as i can get before reading through code :) | 13:38 |
| tlater | Cool :) | 13:38 |
| juergbi | tlater: and now alpine linux server is down, breaking build of the docker image :/ | 13:41 |
| juergbi | Error 503 Backend is unhealthy | 13:42 |
| tlater | Gah | 13:42 |
| tlater | Not our day :( | 13:42 |
| Nexus | mondays. | 13:42 |
| juergbi | CF continuous failing | 13:42 |
| tlater | Don't think there's much we can do beyond hoping that their server goes back up | 13:43 |
| juergbi | we should start building CI base images using buildstream itself... | 13:43 |
| tlater | That would be nice | 13:45 |
| * tlater is watching gitlab's issue on supporting OCI images for their CI | 13:45 | |
| tlater | Once they finally do an OCI plugin is really high up in my todo list | 13:46 |
| * tlater is tempted to depend on this to parse stuff like "5G": https://pypi.python.org/pypi/humanfriendly | 13:51 | |
| juergbi | https://github.com/gliderlabs/docker-alpine/issues/279 | 13:52 |
| *** sstriker has joined #buildstream | 14:11 | |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 14:12 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 14:26 |
| *** toscalix has quit IRC | 14:29 | |
| *** awacheux[m] has joined #buildstream | 14:48 | |
| *** awacheux has joined #buildstream | 14:51 | |
| gitlab-br-bot | buildstream: merge request (test-integration-flag->master: Fix lack of setuptools_scm causing CI failure) #328 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/328 | 14:55 |
| gitlab-br-bot | buildstream: merge request (test-integration-flag->master: Fix lack of setuptools_scm causing CI failure) #328 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/328 | 14:56 |
| tlater | juergbi: ^ That should hopefully fix our CI, is there any way for us to ask devs of various branches to rebase against this? | 14:57 |
| *** toscalix has joined #buildstream | 14:58 | |
| Nexus | what am i rebasing against? | 14:59 |
| tlater | Nexus: Master, once that MR lands (should be quite quick, it's basically a one-liner) | 14:59 |
| Nexus | kk, let me know :) | 14:59 |
| tlater | jjardon[m]: You will also have to rebase against master once that lands, unfortunately we can't fix this retroactively :( | 15:00 |
| jjardon[m] | no problem | 15:01 |
| gitlab-br-bot | buildstream: merge request (test-integration-flag->master: Fix lack of setuptools_scm causing CI failure) #328 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/328 | 15:02 |
| jjardon[m] | tlater: so any idea how come this start failing if we are using a fixed buildstream docker image? | 15:03 |
| *** aday has quit IRC | 15:04 | |
| *** aday has joined #buildstream | 15:05 | |
| tlater | jjardon[m]: I wish. juergbi and I spent a little while trying to debug that, I suspect setup.py automatically updates some dependencies, but then again, we couldn't actually find any evidence of that doing anything to setuptools. | 15:06 |
| jjardon[m] | well, at least is fixed now :) | 15:07 |
| gitlab-br-bot | buildstream: issue #309 ("All builds are failing") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/309 | 15:08 |
| gitlab-br-bot | buildstream: merge request (test-integration-flag->master: Fix lack of setuptools_scm causing CI failure) #328 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/328 | 15:09 |
| jennis | \o/ | 15:09 |
| jennis | nice one tlater ;P | 15:09 |
| tlater | Well, juergbi found the actual bug ;) | 15:09 |
| jennis | *and juergbi | 15:09 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 15:13 |
| jjardon[m] | Can I have some reviews on https://gitlab.com/BuildStream/buildstream/merge_requests/327 , please? 😆 | 15:14 |
| Nexus | lets see if this works then | 15:14 |
| tlater | Oh, nice jjardon[m] | 15:15 |
| gitlab-br-bot | buildstream: merge request (issue-243_dpkg_import_source_plugin->master: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 15:18 |
| gitlab-br-bot | buildstream: merge request (jjardon/misc_fixes->master: Some little documentation fixes) #326 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/326 | 15:27 |
| gitlab-br-bot | buildstream: merge request (jjardon/theme_change->master: Change theme to sphinx_rtd_theme) #327 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/327 | 15:27 |
| gitlab-br-bot | buildstream: merge request (jjardon/getting_started->master: Add getting started section) #323 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/323 | 15:28 |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 15:29 |
| *** Prince781 has joined #buildstream | 15:35 | |
| * tlater isn't sure how to calculate the size of an artifact before it's commited | 15:40 | |
| tlater | I could implement something with os.walk, but that's very slow in python3.4 | 15:41 |
| tlater | Alternatively, I could call `df`, but I'm not sure how portable that is | 15:41 |
| tlater | And finally, I could depend on this: https://pypi.python.org/pypi/scandir - it would make os.walk a bit quicker | 15:41 |
| skullman | you'd need `du`, not `df` | 15:42 |
| tlater | Ah, right, always confuse them, ta skullman | 15:42 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 15:45 |
| skullman | no idea about portability, but I'd avoid parsing the output of `du` in general since file names may include newlines and spaces, and the --null option to make it separate reports with NUL bytes rather than newlines is probably not portable | 15:46 |
| tlater | Hm, annoying. So it's between growing another dependency or accepting a bit of slowness on some systems | 15:47 |
| tlater | I suppose it would be acceptable to have a slow implementation for python3.4, and use the faster method in python3.5+ | 15:48 |
| skullman | if scandir exposes a replacement os.walk, then if scandir is present use it, otherwise fall back to the native slow version | 15:48 |
| tlater | Yep, that makes enough sense. Though it does create some annoying conditional imports | 15:49 |
| skullman | `try: from scandir import walk\nexcept ImportError: from os import walk` | 15:49 |
| tlater | ta skullman | 15:49 |
| jonathanmaw | juergbi: could you have a look at https://gitlab.com/BuildStream/buildstream/merge_requests/317 please? | 15:50 |
| *** Prince781 has quit IRC | 15:52 | |
| tlater | Probably a bit late in the day to ask, but I'm assuming we'll want to keep some metadata to get usage information for the cache expiry functionality? | 17:14 |
| tlater | I don't see a way to do this with just ostree, unfortunately | 17:15 |
| juergbi | tlater: if we ignore ostree API, one idea was to 'touch' the ref file whenever we call extract() and then we expire refs based on mtime | 17:24 |
| tlater | Oh, you can do that? | 17:24 |
| tlater | I suppose there's no reason why you couldn't | 17:24 |
| juergbi | this would be possible on the file system level. not sure how well that would work ostree API wise | 17:24 |
| juergbi | but maybe there are better possibilities. that's just a simple idea | 17:25 |
| juergbi | what kind of metadata were you thinking of? | 17:25 |
| tlater | Simply recording an atime in the artifact metadata whenever it's used | 17:25 |
| tlater | This would be a bit more flexible long term, in case we want to be smarter than LRU | 17:25 |
| tlater | But for now I think I like touching ref files better | 17:26 |
| juergbi | i.e., creating new ostree tree and commit objects on every access? | 17:26 |
| tlater | Ah, right, you can't even write into the repofile directly :| | 17:26 |
| juergbi | yes, we don't want to update the commit/tree hashes on read | 17:26 |
| juergbi | would lead to repushing etc. | 17:26 |
| tlater | There's a commit comment that we could abuse, but that feels wrong. | 17:27 |
| tlater | Also not sure it can be modified without changing the hash | 17:28 |
| juergbi | eh, no, it's a hash | 17:29 |
| tlater | Well, in my defense, this doc isn't very clear about what that has actually hashes | 17:30 |
| juergbi | everything should be hashed, just like in git | 17:31 |
| juergbi | unless something is explicitly marked as out of band data or so. don't know whether ostree has that | 17:32 |
| juergbi | theoretically we could have a second ref that we don't push just for access time management | 17:32 |
| juergbi | but i prefer the mtime hack | 17:32 |
| * tlater agrees, will attempt it | 17:33 | |
| *** toscalix has quit IRC | 17:35 | |
| juergbi | hm, should we hard-depend on arpy in setup.py to get it pulled in as part of buildstream installation even though it's an optional source plugin-specific dependency? | 17:38 |
| tlater | Do we currently? Sounds like a mistake. | 17:42 |
| juergbi | no, the branch doesn't. just wondering whether the missing package warning is friendly enough | 17:48 |
| *** awacheux has quit IRC | 17:56 | |
| *** dominic has quit IRC | 18:07 | |
| *** valentind has joined #buildstream | 18:16 | |
| *** tristan has joined #buildstream | 18:19 | |
| *** ltu has quit IRC | 19:16 | |
| *** aiden has quit IRC | 19:16 | |
| *** paulsherwood has quit IRC | 19:16 | |
| *** tlater has quit IRC | 19:17 | |
| *** jmac has quit IRC | 19:17 | |
| *** milloni has quit IRC | 19:17 | |
| *** benbrown has quit IRC | 19:17 | |
| *** adds68 has quit IRC | 19:17 | |
| *** Nexus has quit IRC | 19:17 | |
| *** adds68 has joined #buildstream | 19:20 | |
| *** laurence has joined #buildstream | 19:23 | |
| *** benbrown has joined #buildstream | 19:23 | |
| *** paulsherwood has joined #buildstream | 19:24 | |
| *** jmac_ has joined #buildstream | 19:25 | |
| *** aiden has joined #buildstream | 19:25 | |
| *** tlater has joined #buildstream | 19:25 | |
| *** Nexus has joined #buildstream | 19:25 | |
| *** toscalix has joined #buildstream | 20:16 | |
| *** toscalix has quit IRC | 20:20 | |
| *** toscalix has joined #buildstream | 20:20 | |
| *** toscalix has quit IRC | 20:31 | |
| *** tristan has quit IRC | 20:41 | |
| *** aday has quit IRC | 20:45 | |
| *** jonathanmaw has quit IRC | 20:54 | |
| *** sstriker has quit IRC | 21:12 | |
| *** valentind has quit IRC | 21:13 | |
| *** Prince781 has joined #buildstream | 21:43 | |
| gitlab-br-bot | buildstream: issue #299 ("Build sometimes hangs forever after "Unknown exception in SIGCHLD handler"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/299 | 22:42 |
| *** Prince781 has quit IRC | 23:20 | |
| *** Prince781 has joined #buildstream | 23:21 | |
| *** Prince781 has quit IRC | 23:36 | |
| *** Prince781 has joined #buildstream | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!