| *** sstriker has quit IRC | 00:09 | |
| *** ptomato[m] has quit IRC | 00:11 | |
| *** connorshea[m] has quit IRC | 00:11 | |
| *** asingh_[m] has quit IRC | 00:11 | |
| *** pro[m] has quit IRC | 00:11 | |
| *** bochecha has quit IRC | 00:12 | |
| *** abderrahim[m] has quit IRC | 00:12 | |
| *** mrmcq2u[m] has quit IRC | 00:12 | |
| *** kailueke[m] has quit IRC | 00:12 | |
| *** cgmcintyre[m] has quit IRC | 00:12 | |
| *** inigomartinez has quit IRC | 00:12 | |
| *** waltervargas[m] has quit IRC | 00:12 | |
| *** jjardon[m] has quit IRC | 00:12 | |
| *** ernestask[m] has quit IRC | 00:12 | |
| *** m_22[m] has quit IRC | 00:12 | |
| *** mattiasb has quit IRC | 00:12 | |
| paulsherwood | where are the bst pipelines that are building gnome? | 01:17 |
|---|---|---|
| *** connorshea[m] has joined #buildstream | 01:19 | |
| mcatanzaro | paulsherwood: Is https://gitlab.gnome.org/GNOME/gnome-build-meta what you're looking for? | 01:58 |
| paulsherwood | mcatanzaro: yes!!! thanks a lot :) | 02:02 |
| *** mcatanzaro has quit IRC | 02:09 | |
| *** bochecha has joined #buildstream | 02:21 | |
| *** cgmcintyre[m] has joined #buildstream | 02:45 | |
| *** bochecha has quit IRC | 02:53 | |
| *** cgmcintyre[m] has quit IRC | 02:53 | |
| *** connorshea[m] has quit IRC | 02:53 | |
| *** ernestask[m] has joined #buildstream | 03:01 | |
| *** ernestask[m] is now known as Guest43691 | 03:02 | |
| *** waltervargas[m] has joined #buildstream | 03:05 | |
| *** mattiasb has joined #buildstream | 03:39 | |
| *** inigomartinez has joined #buildstream | 03:42 | |
| *** asingh_[m] has joined #buildstream | 03:50 | |
| *** ptomato[m] has joined #buildstream | 04:05 | |
| *** abderrahim[m] has joined #buildstream | 04:29 | |
| *** kailueke[m] has joined #buildstream | 04:43 | |
| *** bochecha has joined #buildstream | 04:55 | |
| *** jjardon[m] has joined #buildstream | 06:17 | |
| juergbi | jjardon[m]: that error seems like the hardware has no support for armv7 (not all aarch64 hardware does). or the kernel has CONFIG_COMPAT disabled | 06:21 |
| juergbi | i would also recommend running bst via linux32 (or setarch) to make sure the 'wrong' uname output doesn't produce any issues | 06:23 |
| juergbi | the plan is to properly support non-host execution environments in bst itself in the future without linux32/setarch | 06:24 |
| * persia thinks linux32 on aarch64 generates armv8l, and it requires setarch to reach armv7l | 07:33 | |
| *** m_22[m] has joined #buildstream | 07:39 | |
| *** ernestask has joined #buildstream | 07:50 | |
| *** mrmcq2u[m] has joined #buildstream | 07:54 | |
| *** connorshea[m] has joined #buildstream | 08:00 | |
| *** pro[m] has joined #buildstream | 08:21 | |
| *** cgmcintyre[m] has joined #buildstream | 08:25 | |
| *** aday has joined #buildstream | 08:37 | |
| *** Guest43691 is now known as ernestask[m] | 09:25 | |
| gitlab-br-bot | buildstream: merge request (juerg/225->master: Fix staging of runtime dependencies with overlaps) #263 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/263 | 09:44 |
| *** tristan has joined #buildstream | 09:47 | |
| *** noisecell has joined #buildstream | 09:55 | |
| *** adds68 has joined #buildstream | 09:56 | |
| *** jonathanmaw has joined #buildstream | 09:58 | |
| *** ssam2 has joined #buildstream | 10:05 | |
| *** adds68 has quit IRC | 10:17 | |
| gitlab-br-bot | buildstream: merge request (image-authoring->master: Image authoring documentation) #262 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/262 | 10:18 |
| gitlab-br-bot | buildstream: merge request (image-authoring->master: Image authoring documentation) #262 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/262 | 10:20 |
| tlater | Hrm, I wonder if that can be merged to make our tests pass again | 10:21 |
| tlater | The page isn't included anywhere currently, so all it would do is complete the half-done documentation page | 10:22 |
| ssam2 | it seems to me we should resolve accidental pushes by reverting what was pushed, not by hastily pushing more stuff | 10:23 |
| ssam2 | commit history is littered with mistakes, i wouldn't worry about making it "messy" with a revert | 10:24 |
| juergbi | oh, didn't realize tests were failing because of this | 10:24 |
| juergbi | i'll revert it | 10:24 |
| tlater | tyvm juergbi | 10:24 |
| juergbi | done | 10:26 |
| gitlab-br-bot | buildstream: merge request (cache-keys-os-arch->master: Cache keys to consider OS and machine arch) #261 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/261 | 10:26 |
| tristan | jjardon[m], it should be possible to build armv7 with aarch64, you want to have an armv7 tuned native compiler and runtime for that, and you may need to pass --target / --build to some modules (mostly those which use inline assembly) | 10:27 |
| juergbi | tristan: we should not have to modify the config due to exposed details of the host machine | 10:29 |
| tristan | juergbi, in build instructions | 10:29 |
| tristan | juergbi, not bst --option options | 10:30 |
| *** adds68 has joined #buildstream | 10:30 | |
| tristan | juergbi, what we've found is that when building in a targetted yocto runtime in x86_64 targetting i586, or same with arm... is you need to tell some things like webkit about which assembly to use | 10:30 |
| juergbi | build instructions should not need to change whether the host is aarch64 or armv7, imo | 10:30 |
| tristan | with their ./configure --build ${host_triple} | 10:31 |
| juergbi | tristan: that's only the case if you run without linux32/setarch | 10:31 |
| tristan | I was just going to say that | 10:31 |
| juergbi | you may want to generally be specific about this but i don't think we should require people to do that | 10:31 |
| tristan | If you run with linux32, it shouldnt need that really | 10:31 |
| tristan | but I havent tried it with our stacks | 10:31 |
| tristan | (with flatpak / gnome stack) | 10:31 |
| juergbi | and i think we've already agreed that bst should allow specifying the execution environment (anyway needed for remote builds) | 10:31 |
| juergbi | so i'd rather recommend linux32/setarch for now than fiddling with config arguments | 10:32 |
| juergbi | as that's closer to 'the future' | 10:32 |
| tristan | still, buildstream currently wont run itself under linux32 for you | 10:32 |
| tristan | although we should look into that | 10:32 |
| juergbi | it won't? | 10:32 |
| tristan | not currently no, we dont understand machine arches either, yet; we just let the user configure options | 10:33 |
| tristan | so we wouldnt know when to run the sandbox under linux32 either | 10:33 |
| juergbi | yes, we definitely need to work on this for cross-build support | 10:33 |
| tristan | but along the same lines as remote and virtualized building, it's a good feature to consider fitting into that api | 10:33 |
| juergbi | however, i would expect linux32/setarch in front of bst to be identical to running on physical 32-bit machine | 10:33 |
| juergbi | (in this regard) | 10:34 |
| tristan | Well, currently you can run it under linux32 and try, or you can use specific args and a targetted native compiler in the runtime | 10:34 |
| tristan | I wouldnt say that cross-build on compatible host is completely broken because we dont have extra convenience features for it yet | 10:35 |
| juergbi | i didn't say that | 10:35 |
| * tristan on his way out the door... | 10:35 | |
| tristan | right, I did reorder your words on the cross-build support line :-) | 10:36 |
| tristan | we need to work on this indeed; it fits into other plans as well | 10:36 |
| tristan | juergbi, we might want to start using gitlab milestones instead of wikis for roadmapping... | 10:37 |
| tristan | might make putting things on roadmaps easier | 10:37 |
| tristan | anyway I gotta run... | 10:37 |
| juergbi | might be useful | 10:37 |
| juergbi | enjoy your holiday | 10:37 |
| gitlab-br-bot | buildstream: merge request (image-authoring->master: WIP: Image authoring documentation) #262 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/262 | 10:39 |
| tlater | Is there anywhere I could put a sample project for buildstream documentation? | 10:42 |
| juergbi | maybe a separate repo in the buildstream group? | 10:43 |
| tlater | Perhaps we should have a buildstream-examples repo in general | 10:43 |
| tlater | I imagine this will come up a few more times | 10:44 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 11:01 |
| gitlab-br-bot | buildstream: merge request (cache-keys-os-arch->master: Cache keys to consider OS and machine arch) #261 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/261 | 11:19 |
| *** toscalix has joined #buildstream | 11:19 | |
| gitlab-br-bot | buildstream: merge request (juerg/225->master: Fix staging of runtime dependencies with overlaps) #263 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/263 | 11:29 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 11:39 |
| *** aday has quit IRC | 11:47 | |
| *** rafaelff[m] has joined #buildstream | 11:48 | |
| *** aday has joined #buildstream | 11:49 | |
| tlater | ssam2: Could you have a look at this MR? https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/21 | 11:49 |
| tlater | It's another one on the testsuite image... | 11:49 |
| *** aday has quit IRC | 11:50 | |
| juergbi | pages:deploy failing again, hm | 11:59 |
| *** aday has joined #buildstream | 12:02 | |
| gitlab-br-bot | buildstream: issue #225 ("bst shell crashes when staging dependencies") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/225 | 12:04 |
| gitlab-br-bot | buildstream: merge request (juerg/225->master: Fix staging of runtime dependencies with overlaps) #263 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/263 | 12:04 |
| ssam2 | tlater, merged | 12:10 |
| tlater | ta :) | 12:10 |
| *** aday has quit IRC | 12:13 | |
| *** aday has joined #buildstream | 12:14 | |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 12:26 |
| gitlab-br-bot | buildstream: merge request (patch-1->master: install.rst: Update instructions for Arch) #264 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/264 | 12:38 |
| *** mcatanzaro has joined #buildstream | 12:54 | |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 13:03 |
| tlater | juergbi: Yeah, looks like I missed the __version__.py, I thought I explicitly included it. Another 30 minutes wait it is, then :| | 13:06 |
| juergbi | tlater: i assume CI caching is working, right? (there were some issues a while ago, afaik) | 13:07 |
| tlater | Yeah, I spent a while fixing that | 13:07 |
| tlater | I just double checked too, it seems to be working | 13:08 |
| juergbi | ok, good | 13:08 |
| * tlater wishes he could *inspect* the cache, but a guess will have to do | 13:09 | |
| *** nexus has quit IRC | 13:10 | |
| ssam2 | we can ssh into the runners while builds are running | 13:10 |
| *** nexus has joined #buildstream | 13:11 | |
| tlater | Hm, I'd need to see it after one completes | 13:11 |
| ssam2 | they get deleted after that | 13:11 |
| tlater | Either way, worst case the caches simply don't contain anything relevant, so we can investigate that after a merge | 13:11 |
| tlater | Where do the caches/artifacts go then? | 13:13 |
| ssam2 | gitlab uploads them to a separate machine before deleting the runner | 13:14 |
| ssam2 | artifacts go to ... i'm not sure for this project | 13:15 |
| ssam2 | gnome7.codethink.co.uk perhaps ? | 13:15 |
| tlater | Could we download from there? | 13:15 |
| ssam2 | yeah | 13:15 |
| ssam2 | oh, you mean gitlab artifacts. i think those are hosted by gitlab.com somewhere | 13:15 |
| ssam2 | but the cache is hosted by us, we can dig into it | 13:15 |
| tlater | Nice | 13:16 |
| tlater | Any way I could gain access to that server? | 13:16 |
| ssam2 | maybe; might be easier if you just come to my desk though | 13:16 |
| juergbi | tlater: now the test failed in cache-related way: | 13:24 |
| juergbi | PermissionError: [Errno 13] Permission denied: '/builds/BuildStream/cache/integration-cache/artifacts' | 13:24 |
| tlater | Yeah, I saw that | 13:25 |
| *** ssam2 has quit IRC | 13:25 | |
| tlater | I suspect it's confusing the unix/linux caches | 13:26 |
| * tlater will set a cache key | 13:26 | |
| tlater | This hasn't happened before, good thing we ran the tests a few times here | 13:26 |
| *** ssam2 has joined #buildstream | 13:26 | |
| persia | Doesn't the merge of 261 mean that it shouldn't confuse them? | 13:27 |
| tlater | persia: It doesn't confuse individual branches anymore, it looks like it still confuses different jobs | 13:27 |
| persia | Ah. | 13:27 |
| juergbi | i would never expect any confusion between ostree and tar caches. they use separate directories (and different cache keys even before !261) | 13:27 |
| tlater | juergbi: Extracted sources are still shared, afaik | 13:28 |
| persia | I'm remembering that in Debian, there is an interesting distinction between "Arch: any" and "Arch: all", in that, in cases where a given source is known not to result in different artifacts on different architectures (e.g. pure python libraries), there is metadata to indicate this, so that the artifact is only built in one place and reused everwhere. | 13:29 |
| juergbi | ah, right but cache key shouldn't match | 13:29 |
| persia | Dunno if that is optimisation for the way .debs are stored in Debian archives, or something useful for us. | 13:29 |
| juergbi | due to our non-use of host tools, it would be difficult to support this optimization | 13:30 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 13:31 |
| juergbi | i don't think it's very important for us at this point | 13:31 |
| * tlater crosses his fingers that this works | 13:31 | |
| tlater | On the up side, this means that the cached sources are actually accessed by the integration tests, so the cache *is* working | 13:31 |
| tlater | When it doesn't cause permission errors, of course | 13:31 |
| juergbi | tlater: hm, so CI_COMMIT_REF_NAME means that the cache is never used for new branches/MRs? | 13:32 |
| tlater | Yes, each one gets a new, clean cache | 13:32 |
| tlater | I also know why we got an error last time, it looks like the unix test completed first, which it usually doesn't | 13:35 |
| tlater | Presumably gitlab takes the first artifact | 13:35 |
| *** nexus has quit IRC | 13:37 | |
| *** nexus has joined #buildstream | 13:37 | |
| tlater | Hm, same error with a clean cache | 13:38 |
| juergbi | tlater: no cache reuse across branches is already the case with master or why the change? | 13:38 |
| tlater | I thought we shared caches across all runners | 13:39 |
| tlater | At least that caused the cache issues a while ago | 13:39 |
| juergbi | my question is about sharing across branches, not runners | 13:40 |
| tlater | Yeah, as far as I am aware currently we share across all tests, including those on different branches | 13:40 |
| * tlater may have missed a change | 13:40 | |
| juergbi | and your latest branch changes that, right? hence me asking why | 13:41 |
| juergbi | as this would mean that the SDK is downloaded for every new branch/MR, if i'm not missing anything | 13:41 |
| juergbi | or is that cached separately? | 13:42 |
| tlater | The key I added just now makes it so that every branch downloads its own copy of the sdk | 13:42 |
| juergbi | exactly. and as that takes quite some extra time, i'm still wondering why you're changing this | 13:44 |
| tlater | Right, I suppose it could be done only per-job | 13:45 |
| tlater | But we had issues in the past with doing it across all branches, so I thought it might not be a bad addition anyway | 13:45 |
| tlater | juergbi: I take it your opinion is that we should only do it per-job, to avoid unix/linux mingling? | 13:46 |
| juergbi | yes. afaik, this only broke when you were working on earlier versions of this branch, not with regular branches | 13:47 |
| juergbi | so i would rather avoid redownloading the SDK all the time | 13:47 |
| juergbi | it's slow enough as it is | 13:47 |
| tlater | Yeah, that makes sense | 13:48 |
| juergbi | if we see breakage, we may have to reevaluate but in general i expect it to work | 13:48 |
| * tlater suspects the breakage here is down to never having attempted this with an empty cache | 13:49 | |
| juergbi | tlater: may mkdir -p be required as root (i.e., before su)? | 13:50 |
| tlater | Yeah, I think so | 13:50 |
| *** aday has quit IRC | 13:51 | |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 13:51 |
| *** aday has joined #buildstream | 13:56 | |
| tlater | The `mkdir -p` looks to have fixed it | 13:57 |
| * tlater wonders what caused the cache to be cleared | 13:57 | |
| jonathanmaw | juergbi: Do you have any suggestions on how I can test the general case that I can make project.conf override source config? The only thing that comes to mind for me is to write a custom source to output the changes | 14:00 |
| juergbi | tlater: you added a key... | 14:00 |
| juergbi | jonathanmaw: let me take a look | 14:01 |
| tlater | juergbi: This broke before I added the key, though | 14:01 |
| jonathanmaw | thanks juergbi | 14:01 |
| juergbi | tlater: ah, right, but before that you noticed that the unix test finished first | 14:01 |
| tlater | Yeah, it might have been both | 14:02 |
| tlater | Interesting, it looks like the unix platform is slightly faster at downloading the sdk | 14:06 |
| jmac | I need to run through a few things to make sure I've understood the build process properly | 14:13 |
| jmac | We start off with a base system image, and then build an element using that, but any changes during the build phase are discarded, and only changes made during the install phase persist, is that right? | 14:14 |
| juergbi | jmac: the path is what matters, not the phase | 14:17 |
| tlater | By default anything in /buildstream/install is persisted | 14:17 |
| jmac | Right, so the build process can modify other bits of the file system but those changes are lost | 14:18 |
| jmac | Is that right? | 14:18 |
| juergbi | the rootfs is read-only during a regular build | 14:18 |
| juergbi | only source and install directories are read-write | 14:18 |
| jmac | Even /tmp? | 14:18 |
| juergbi | no, /tmp should be read-write as well | 14:19 |
| juergbi | what i meant was that the staged dependencies will be read only | 14:19 |
| juergbi | /tmp is mounted on top, if i'm not mistaken | 14:19 |
| jmac | OK so far. Are the files in /buildstream/install then overlaid on the rootfs after building? | 14:20 |
| jjardon[m] | juergbi: tristan thanks for the comments: I will test and come back to you | 14:20 |
| juergbi | jmac: only for future build jobs where these files may be staged as dependency | 14:21 |
| juergbi | jonathanmaw: if i see this right, coverage should already be pretty decent. i.e., you're testing project overrides. your concern is that it's only tested indirectly by observing the effect of a git config override instead of a more direct test. correct? | 14:22 |
| jonathanmaw | correct | 14:22 |
| juergbi | adding a possibility to show source config via the CLI could make sense. however, given that you already have test coverage, i don't think this should be a blocker | 14:23 |
| jmac | juergbi: And `bst checkout` does the compositing of the original rootfs + all dependency output + element output? | 14:23 |
| juergbi | jonathanmaw: and for that we'd want to consider real (non-test) use cases | 14:23 |
| * jmac will ask more later as we're crossing conversations here | 14:23 | |
| jonathanmaw | juergbi: okie doke. I'll tidy up my branch some more and it'll be ready for review. | 14:24 |
| juergbi | ta | 14:24 |
| juergbi | jmac: yes | 14:26 |
| juergbi | well, original rootfs not necessarily | 14:26 |
| paulsherwood | how do i do bst track on lots files at once? | 14:26 |
| * paulsherwood has encountered 'inconsistent pipeline, and a list of 50 files' | 14:27 | |
| juergbi | jmac: it only stages the (recursive) runtime dependencies | 14:27 |
| jmac | Ah, yes | 14:27 |
| ssam2 | paulsherwood, `bst track --deps=all` | 14:27 |
| ssam2 | or you can pass multiple individual args to `bst track`, but you probably want to say 'track all deps of this element' ? | 14:28 |
| juergbi | or bst build --track-all to do it as part of the build command | 14:28 |
| juergbi | (optionally with --track-save to update the .bst files) | 14:28 |
| paulsherwood | vm | 14:29 |
| paulsherwood | tvm | 14:29 |
| tlater | ^ That is debatable, since you cannot access the artifact if you don't --track-save | 14:29 |
| tlater | We didn't quite think through that workflow | 14:29 |
| juergbi | indeed. we will hopefully soon add the external ref metadata | 14:29 |
| juergbi | tlater: are you intentionally frequently retriggering the pipeline? | 14:31 |
| tlater | Yup, sorry, I need to check if the cache is actually working; changes were made to it, after all | 14:31 |
| juergbi | no problem, was just wondering whether there was a CI issue | 14:31 |
| tlater | Hm, looks like gitlab is not creating a cache anymore :| | 14:33 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 14:44 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 14:52 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 14:57 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 15:11 |
| tlater | That's weird... | 15:14 |
| tlater | Setting the cache key to a variable seems to completely disable caching for the run | 15:14 |
| tlater | Maybe I have to add a character? | 15:15 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 15:15 |
| tlater | ... That did it | 15:16 |
| * tlater wonders if he should file a gitlab issue | 15:16 | |
| tlater | Assuming this passes the integration tests should be ready, juergbi | 15:17 |
| tlater | Sorry for the delay, I tested this quite a lot on my private branch, must just never have hit this edge case | 15:17 |
| juergbi | tlater: i guess at least the two top commits should be squashed, possibly even apply these changes to the 'merge integration tests into general tests' commit? | 15:18 |
| tlater | Right, that's a point | 15:19 |
| tlater | I'll do so when the current run finishes, just to make sure I test the cache one last time | 15:19 |
| juergbi | ok | 15:19 |
| juergbi | oddly enough this is even in gitlab examples: | 15:20 |
| juergbi | key: "$CI_JOB_NAME" | 15:20 |
| tlater | Yeah, I know | 15:20 |
| tlater | I think I'll file a bug and see where it goes | 15:20 |
| tlater | Ugh, still getting permission errors | 15:22 |
| tlater | I guess I should run the chown in the actual job | 15:22 |
| tlater | Oh, I do | 15:22 |
| tlater | Hrm | 15:22 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 15:32 |
| gitlab-br-bot | buildstream: issue #236 ("Elements not building when using "bst build --track-all --track-save"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/236 | 15:32 |
| gitlab-br-bot | buildstream: issue #237 ("Check that elements produced by BuildStream are bit-for-bit reproducible.") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/237 | 15:33 |
| gitlab-br-bot | buildstream: merge request (juerg/dbus->master: Inherit user id and group id for bst shell) #265 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/265 | 15:39 |
| * paulsherwood wonders if there's some hitch with the fosdem talk video... still no sign, but most others are up | 15:44 | |
| persia | I would have expected it at https://video.fosdem.org/2018/K.3.201/ | 15:45 |
| juergbi | that's odd | 15:45 |
| persia | I don't see the following talk either | 15:46 |
| persia | https://video.fosdem.org/2018/K.3.201/developing_enterprise_and_community_distros.webm was the immediately preceeding talk | 15:47 |
| paulsherwood | to be fair they've been landing every day, so maybe soon... | 15:48 |
| persia | Video team does excellent work, but there are never enough folk who volunteer to do the post-processing and upload, plus post-fosdem is a time of illness and sleep (too many people bring germs from too many corners of the globe). | 15:50 |
| * nexus wonders if it was put in the wrong dir | 15:51 | |
| nexus | ah, found it | 15:52 |
| nexus | waiting review | 15:53 |
| nexus | it's in preview atm | 15:53 |
| * tlater wonders if we should throw an exception if we cannot find a plugin that's defined in project.conf | 15:54 | |
| nexus | Don't we? | 15:54 |
| nexus | what happens? | 15:55 |
| tlater | Well, bst seems to just carry on until we crash because we can't find a plugin required for an element | 15:56 |
| tlater | I suppose technically some paths might not require the plugin | 15:56 |
| tlater | As it stands I can't even get ssam2's docker plugin to run :| | 15:57 |
| ssam2 | we should raise an exception if the config is bad | 15:57 |
| jmac | I think if there's a missing plugin defined in project.conf then it should be reported right away | 15:57 |
| gitlab-br-bot | buildstream: merge request (212-git-source-needs-a-way-to-disable-checking-out-submodules->master: WIP: Resolve "Git source needs a way to disable checking out submodules") #259 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/259 | 15:59 |
| tlater | It would also be useful to list installed plugins when one is missing - a lot easier to debug that way | 15:59 |
| * tlater will write an issue in a bit | 16:00 | |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 16:14 |
| tlater | ssam2: (or anyone who knows about dnf) Does this look like a dnf error? https://gitlab.com/BuildStream/buildstream/-/jobs/51536648 | 16:23 |
| ssam2 | yep, dnf randomly fails occasionally like that | 16:25 |
| * tlater hates fiddling with CI | 16:26 | |
| tlater | I guess I'll just have to wait a bit and see if that clears up | 16:26 |
| ssam2 | it usually does | 16:27 |
| tlater | Ah, it passed! | 16:28 |
| tlater | So, if at any point you get permission errors with the new integration tests, it seems "retry after a while" is a valid strategy to resolve that | 16:29 |
| * tlater has no idea what causes it, some files seem to randomly change permissions during test runs | 16:30 | |
| jonathanmaw | juergbi: I've checked my changes, rebased and pushed my submodule checkout changes. Can you review them when you have time? | 16:31 |
| juergbi | will do, ta | 16:32 |
| ltu | re fosdem, we also need to upload the slides to https://fosdem.org/2018/schedule/event/introducing_buildstream/ | 16:32 |
| ltu | i suspect they were missed off as they were being perfected right up until the talk | 16:33 |
| tlater | ssam2: I seem to be missing a 'SaveFile' in buildstream.utils - iirc that's something you wanted to merge into buildstream but was eventually refused | 16:38 |
| tlater | What happened to that? | 16:38 |
| ssam2 | it got merged | 16:39 |
| ssam2 | i thought | 16:39 |
| ssam2 | but renamed | 16:39 |
| tlater | Ah | 16:39 |
| ssam2 | do you have an old version of the branch ? i thought i updated it already | 16:39 |
| tlater | Hm, lemme check | 16:39 |
| ssam2 | its in there as utils.save_file_atomic() | 16:40 |
| tlater | Oh, yeah, looks like I had an old version | 16:40 |
| tlater | Odd, I rebased it against master | 16:40 |
| tlater | Ah, looks like I don't have your remote in my branch | 16:43 |
| *** ruben has joined #buildstream | 16:48 | |
| *** ruben has quit IRC | 17:03 | |
| *** ruben has joined #buildstream | 17:05 | |
| gitlab-br-bot | buildstream: issue #175 ("Refactor integration tests") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/175 | 17:10 |
| gitlab-br-bot | buildstream: merge request (175-refactor-integration-tests->master: Resolve "Refactor integration tests") #233 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/233 | 17:10 |
| tlater | ta juergbi :) | 17:20 |
| ltu | wahey! | 17:20 |
| tlater | I'll scour master for non-migrated integration tests for the next few days | 17:21 |
| juergbi | tlater: actually, workspace mount test was not removed | 17:23 |
| tlater | Ack | 17:23 |
| juergbi | we can just remove the integration-tests directory, right? | 17:24 |
| juergbi | as it's already migrated | 17:24 |
| tlater | I did apparently migrate it | 17:24 |
| *** toscalix has quit IRC | 17:24 | |
| tlater | Yeah, the integration-tests directory has been completely purged from CI files | 17:24 |
| juergbi | ok, i'll remove it | 17:24 |
| tlater | ta | 17:24 |
| jmac | Does BuildStream reduce max-jobs if it's running several build jobs at once? | 17:32 |
| ssam2 | no | 17:32 |
| juergbi | the idea is that the kernel scheduler can handle this well enough. it's not always ideal but generally works as long as you're not running out of RAM | 17:33 |
| juergbi | maybe we should look into implementing a make-compatible job server at some point | 17:34 |
| tlater | Looking at ssam2 's code, I think we could definitely use a better linter in the CI - the fetching failure was entirely due to a missing variable | 17:37 |
| * tlater suspects that this fell through since the method in question was mostly unused before the recent elements state refactoring | 17:37 | |
| ssam2 | no CI can completely prevent me from making stupid mistakes | 17:38 |
| ssam2 | many have tried | 17:38 |
| tlater | Heh | 17:38 |
| persia | ssam2: CI is not your taskmaster, only your tutor, helping you learn which of your mistakes are stupid :) | 17:38 |
| jmac | Actually I was wondering why my job was using too few cores, rather than too many | 17:40 |
| * tlater thinks missing variable declarations are so obvious that CI should definitely point them out | 17:41 | |
| persia | I think a make-compatible job server is probably wise. The other case where the kernel scheduler fails is when one runs out of IO bandwidth. | 17:41 |
| persia | tlater: Many projects put linters and similar in their CI: you'd just need to write the wrapper to do that. | 17:41 |
| tlater | I suspect there already is one for pylint | 17:42 |
| persia | A hundred cores trying to use a disk connected by a single x4 PCIe link tends to cause unfortunate performance if not throttled, as an example I have seen tried before. | 17:42 |
| tlater | This comes up from a quick Google: https://pypi.python.org/pypi/pytest-pylint | 17:42 |
| persia | tlater: Just needs an MR to add it :) I wonder if a pep8 suite would also be useful | 17:43 |
| tlater | We have a pep8 suite already | 17:43 |
| persia | In CI? Ah. I should look at things more carefully. | 17:43 |
| * tlater creates an issue and will probably create an MR for adding pytest-pylint tomorrow | 17:44 | |
| juergbi | i'd definitely welcome more static checking | 17:44 |
| juergbi | might it actually be a superset of the pep8 suite, i.e., it could replace that? | 17:44 |
| tlater | Using it here it has a lot of hints for code practices that we don't follow, so its usefuleness will be a bit more limited than usual | 17:44 |
| tlater | juergbi: No, unfortunately it doesn't do multiple line checking | 17:45 |
| tlater | It does *most* of pep8, but not all | 17:45 |
| juergbi | :/ | 17:45 |
| * tlater might spend a bit of time seeing if it could replace pep8 though, possible someone's written a plugin at this point | 17:45 | |
| tlater | My knowledge of pylint-pep8 state is 5 months outdated | 17:46 |
| tlater | ssam2: I don't mind if this happens tomorrow, but it would be nice if you could give me an ok on these changes: https://gitlab.com/BuildStream/bst-external/merge_requests/9/diffs?diff_id=10959286&start_sha=9f28b6dd49ab2a9099d5eb6cd99ddcef9542655b | 17:49 |
| ssam2 | tlater, looks OK | 17:51 |
| *** valentind has joined #buildstream | 17:56 | |
| jjardon[m] | sorry for the question, but is there any reason to use pytest-pep8 instead pycodestyle? | 17:57 |
| ssam2 | we should update to pycodestyle, but i think there will be extra warnings that someone will need to fix | 17:57 |
| jjardon[m] | also, seems buildstream needs the pep8 python package but its not used anywhere? | 17:57 |
| jjardon[m] | sure, only asking as I think we do not use pep8 at all, but pytest-pep8; is that correct? | 17:58 |
| tlater | Hrm, when I built my debian-8 image I had to install both individually | 17:58 |
| jjardon[m] | ah, does pytest-pep8 depends on pep8 python package then? | 18:01 |
| *** jonathanmaw has quit IRC | 18:03 | |
| gitlab-br-bot | buildstream: issue #238 ("BuildStream does not abort when a configured plugin is not installed") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/238 | 18:03 |
| gitlab-br-bot | buildstream: issue #239 ("Use pylint for linting") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/239 | 18:05 |
| *** tristan has quit IRC | 18:12 | |
| tlater | jjardon[m]: I'm not entirely sure, it's a very vague memory at this point. | 18:14 |
| tlater | Probably could take some experimenting | 18:14 |
| * tlater puts it in his backlog | 18:14 | |
| *** ssam2 has quit IRC | 18:31 | |
| *** ruben has quit IRC | 19:08 | |
| *** aday has quit IRC | 19:47 | |
| juergbi | mcatanzaro: did some epiphany testing here and we're getting closer. 8 out of 10 tests are now passing with a bit of additional help. one test is timing out and the migration test is failing | 20:19 |
| mcatanzaro | juergbi: It's always possible that the remaining failures are epiphany regressions rather than BuildStream issues... but the only way to check that is for me to reinstall jhbuild. Do you want to risk that? :P | 20:20 |
| mcatanzaro | Historically, I've made a habit of just commenting out failing epiphany tests instead of fixing them... my main concern here is not that epiphany specifically works, but that we can tell maintainers to follow some process to generate a tarball using BuildStream and expect it to probably work. (And being able to run a functional application would be nice too!) | 20:20 |
| juergbi | i wasn't expecting dist tests to rely on system/session services | 20:21 |
| juergbi | doesn't sound ideal to me | 20:21 |
| juergbi | i also had to run 'ninja install' before 'ninja dist' because the tests require system-installed gsettings schemas | 20:22 |
| juergbi | that sounds like an epiphany/meson issue, not a buildstream one. but easy to workaround as ninja install inside the sandbox works (will be lost when leaving but that's ok) | 20:22 |
| juergbi | my uid patch indeed allows access to dbus now, however, the tests complain with: getpwuid_r(): failed due to unknown user id | 20:23 |
| juergbi | adding a /etc/passwd entry avoids this issue | 20:24 |
| juergbi | obviously not a useful solution | 20:24 |
| mcatanzaro | System-installed gsettings schemas are required by most GNOME applications... easy to workaround indeed, though | 20:26 |
| mcatanzaro | juergbi: Why is adding an /etc/passwd entry not useful? | 20:26 |
| juergbi | i mean, we don't want to ask developers to do this all the time | 20:27 |
| persia | Maybe we can have an element that adds a few things to /etc that are normal on most systems, with static values, and use that as a dependency if we need to run that sort of test? That would be mostly developer-invisible, except for them needing to add the dependency. | 20:30 |
| persia | Or we could put an /etc/passwd in a platform (e.g. the GNOME SDK), so that it was already there. | 20:30 |
| juergbi | we'd need the host uid of the developer in /etc/passwd, that's not fixed | 20:31 |
| persia | Probably safe to assume 1000, and developers with a different uid would have to do something special. | 20:33 |
| persia | But yeah, not ideal, and approaches leakage of host environment | 20:33 |
| juergbi | with bst shell some leakage is intentional. we wouldn't have to weaken the build sandbox | 20:34 |
| juergbi | however, we may need a way to control it | 20:34 |
| persia | Maybe either RO hardlink to real /etc/passwd or cp /etc/passwd into the environment? | 20:36 |
| persia | Also /etc/group, /etc/shadow | 20:36 |
| juergbi | system groups should probably not be copied. so copying just a single line (or generating it) is probably better | 20:41 |
| juergbi | /etc/shadow is not accessible by non-root but should also not be needed | 20:41 |
| juergbi | mcatanzaro: i tried running ninja dist with dbus-run-session as possible alternative but that again produces the accessibility registration error | 20:42 |
| persia | Maybe `touch /etc/shadow`? But yeah, not usually necessary. | 20:42 |
| juergbi | i guess epiphany dist generally doesn't work on headless systems? | 20:42 |
| mcatanzaro | juergbi: I've never had any reason to try it on a headless system | 20:43 |
| juergbi | CI systems would have the same issue | 20:43 |
| mcatanzaro | Epiphany itself doesn't have any a11y-related code at all; I assume that's coming from GTK+ (or, less-likely, maybe WebKit) | 20:44 |
| juergbi | i.e., you also can't run tests in gitlab CI | 20:44 |
| mcatanzaro | I could run it under fatal-criticals to see where it's coming from, but then I have to switch back to the old version of Buildstream (and presumably rebuild everything again?) | 20:44 |
| persia | When X was the standard, we used to use a special fake X server for CI, which acted like an X server, except nothing was rendered (just state stored in RAM). Is there anything like that for Wayland? | 20:44 |
| juergbi | mcatanzaro: no, i don't think that's necessary | 20:45 |
| juergbi | (rebuild of dependencies would not be required, though, the rebuild was necessary because of a change in master, not my patch) | 20:45 |
| mcatanzaro | persia: That's probably xvfb? | 20:46 |
| juergbi | persia: weston has a headless backend | 20:46 |
| mcatanzaro | I'll switch to master and see where the a11y error is coming from, then | 20:46 |
| juergbi | ok, ta | 20:47 |
| persia | mcatanzaro: It was xvfb, yes :) I doubt that's the current state of the art today | 20:47 |
| mcatanzaro | xvfb is how we run WebKit tests | 20:47 |
| *** ernestask has quit IRC | 20:54 | |
| gitlab-br-bot | buildstream: issue #240 ("Make a release of BuildStream") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/240 | 21:02 |
| mcatanzaro | juergbi: I'm trying to debug to see where the a11y warnings are coming from. I ran export G_DEBUG="fatal-warnings" | 21:24 |
| mcatanzaro | Now when I run "ninja test" it fails badly because it runs out of fds (even with your fix), so I run "ninja test -j1" instead. | 21:25 |
| mcatanzaro | Now all the tests crash, as desired. Problem: how do I generate a backtrace from the core dumps? | 21:25 |
| mcatanzaro | coredumpctl on the host collects the core dump, but it can't read it | 21:26 |
| mcatanzaro | I mean, it can't generate a usable backtrace | 21:26 |
| juergbi | missing debug symbols? | 21:26 |
| mcatanzaro | And there doesn't seem to be coredumpctl inside the sandbox. There is gdb inside the sandbox. | 21:26 |
| mcatanzaro | I guess I should try manually running one of the tests inside gdb, but that's a lot harder than just running 'coredumpctl gdb'. | 21:27 |
| juergbi | gdb works but you're missing the function names in the backtrace? | 21:27 |
| juergbi | gdb /path/to/crashed-executable core.1234 | 21:27 |
| juergbi | should be equivalent | 21:27 |
| juergbi | if you have a core file | 21:28 |
| mcatanzaro | juergbi: core file is managed by coredumpctl, on the host | 21:28 |
| mcatanzaro | It tries to generate a coredump for e.g. /newroot/buildstream/build/_builddir/tests/test-ephy-uri-helpers | 21:28 |
| juergbi | coredumpctl dump > epiphany-workspace/core | 21:29 |
| juergbi | and then access it in the bst shell | 21:29 |
| mcatanzaro | That works, excellent! | 21:31 |
| mcatanzaro | It's really hard to do without *bash completion* though ;) ;) ;) | 21:31 |
| juergbi | mcatanzaro: you didn't see my comment on that bug? | 21:32 |
| mcatanzaro | I did! | 21:32 |
| juergbi | run bash instead of the default sh and it works | 21:32 |
| mcatanzaro | Oh, manually run bash. Yes... OK | 21:32 |
| juergbi | i.e., just append bash as argument to 'bst shell' | 21:32 |
| juergbi | a bit annoying but easy enough as workaround | 21:33 |
| mcatanzaro | Anyway, from the backtrace I judge that *any* use of GTK+ is going to try to use the a11y bus... it's called a few levels inside gdk_display_manager_open_display() | 21:33 |
| persia | Probably should get fixed, but workarounds are useful when two days behind schedule :) | 21:33 |
| mcatanzaro | So yes, I'd say running tests on a headless server is not going to work. I wonder how useful GitLab CI will be for us, then.... | 21:34 |
| mcatanzaro | https://paste.gnome.org/piullxchz/38yrhm/raw | 21:35 |
| mcatanzaro | For WebKit we actually manually spin up an a11y bus before running tests. | 21:35 |
| juergbi | hm, interesting. wondering why gtk3-demo works | 21:36 |
| mcatanzaro | Anyway, your uid branch fixed the a11y problem, anyway | 21:42 |
| juergbi | we still have the getpwuid/passwd issue | 21:43 |
| juergbi | but yes, if we could find a suitable workaround for that, we're pretty close | 21:43 |
| *** luvspamchrono has joined #buildstream | 21:52 | |
| *** luvspamchrono has quit IRC | 21:53 | |
| *** mcatanzaro_ has joined #buildstream | 22:05 | |
| *** mcatanzaro has quit IRC | 22:07 | |
| *** mcatanzaro_ is now known as mcatanzaro | 22:07 | |
| *** valentind has quit IRC | 22:25 | |
| gitlab-br-bot | buildstream: issue #241 ("Allow name resolution inside bst shell") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/241 | 23:21 |
| *** ruben has joined #buildstream | 23:27 | |
| *** mcatanzaro has quit IRC | 23:41 | |
| *** Prince781 has joined #buildstream | 23:44 | |
| *** tristan has joined #buildstream | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!