| *** narispo has quit IRC | 01:24 | |
| *** narispo has joined #buildstream | 01:25 | |
| *** narispo has quit IRC | 01:55 | |
| *** narispo has joined #buildstream | 01:55 | |
| *** narispo has quit IRC | 04:20 | |
| *** narispo has joined #buildstream | 04:20 | |
| *** narispo has quit IRC | 04:49 | |
| *** narispo has joined #buildstream | 04:50 | |
| *** persia has quit IRC | 05:49 | |
| *** persia has joined #buildstream | 06:00 | |
| *** persia has quit IRC | 06:03 | |
| *** persia has joined #buildstream | 06:05 | |
| *** persia has quit IRC | 06:05 | |
| *** persia has joined #buildstream | 06:05 | |
| *** narispo has quit IRC | 06:10 | |
| *** narispo has joined #buildstream | 06:11 | |
| *** narispo has quit IRC | 06:41 | |
| *** narispo has joined #buildstream | 06:41 | |
| juergbi | Kinnison: are you happy with my replies in !1499? | 07:02 |
|---|---|---|
| gitlab-br-bot | MR !1499: Use buildbox-casd for CAS access https://gitlab.com/BuildStream/buildstream/merge_requests/1499 | 07:02 |
| * Kinnison was in the process of checking through it all, yep, 👍 | 07:08 | |
| juergbi | ta, let's hand it off to marge, then | 07:10 |
| juergbi | and then I'll hide for the rest of the week ;) | 07:11 |
| Kinnison | hehe | 07:11 |
| *** narispo has quit IRC | 07:11 | |
| gitlab-br-bot | marge-bot123 merged MR !1499 (juerg/casd->master: Use buildbox-casd for CAS access) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1499 | 07:12 |
| *** narispo has joined #buildstream | 07:12 | |
| Kinnison | juergbi: while you wait, maybe you can get back to !1542? | 07:13 |
| gitlab-br-bot | MR !1542: Support strict build dependencies https://gitlab.com/BuildStream/buildstream/merge_requests/1542 | 07:13 |
| benschubert | Nice juergbi ! | 07:14 |
| juergbi | I will | 07:14 |
| *** phoenix has joined #buildstream | 07:21 | |
| *** tpollard has joined #buildstream | 07:30 | |
| *** phoenix has quit IRC | 07:32 | |
| *** phoenix has joined #buildstream | 07:32 | |
| *** phoenix has quit IRC | 07:35 | |
| *** phoenix has joined #buildstream | 07:38 | |
| *** narispo has quit IRC | 07:43 | |
| *** narispo has joined #buildstream | 07:43 | |
| *** rdale has joined #buildstream | 07:57 | |
| benschubert | phil: do you remember the rational behind https://gitlab.com/BuildStream/buildstream/merge_requests/1291/diffs#b907e842cc9fd582114f61e5cb32cc3c87d9c258 ? :) | 08:01 |
| *** narispo has quit IRC | 08:13 | |
| *** narispo has joined #buildstream | 08:13 | |
| *** ikerperez has joined #buildstream | 08:15 | |
| jennis | Uhh nice juergbi! | 08:19 |
| jennis | Kinnison, I've replied to your review comments | 08:19 |
| * Kinnison responds to the only one which needs a response | 08:20 | |
| jennis | ok thanks, mhmm passing the ArtifactCache into the Project | 08:22 |
| jennis | Would anyone object to this? | 08:22 |
| Kinnison | Just into that function, not into Project's __init__ | 08:23 |
| jennis | oh right, I misinterpreted | 08:23 |
| *** tme5 has joined #buildstream | 08:33 | |
| *** phoenix has quit IRC | 08:33 | |
| *** raoul has joined #buildstream | 08:56 | |
| *** jonathanmaw has joined #buildstream | 08:57 | |
| *** traveltissues has joined #buildstream | 09:05 | |
| *** narispo has quit IRC | 09:07 | |
| *** narispo has joined #buildstream | 09:07 | |
| jennis | Kinnison, the function in ArtifactCache takes the element and the key, not the ref | 09:12 |
| Kinnison | jennis: Oh? | 09:12 |
| jennis | ArtifactCache.contains()... Not sure if it's worth adding a contains_ref() method? | 09:12 |
| jennis | Or keeping as is | 09:12 |
| Kinnison | Bleh, skip it for now | 09:12 |
| Kinnison | add a comment saying why it's open-coded there for now though please | 09:12 |
| jennis | aye aye | 09:12 |
| *** phoenix has joined #buildstream | 09:37 | |
| *** phoenix has quit IRC | 09:39 | |
| tpollard | Is anyone happy to approve? https://gitlab.com/BuildStream/website/merge_requests/124 | 09:42 |
| traveltissues | tpollard, i don't have approval rights there sorry | 09:44 |
| benschubert | tpollard: It's on the website? But isn't the website for bst 1.2 currently? | 09:44 |
| benschubert | Because if so, No mention of BuildBox casd should be made on the website until the 2.0 release | 09:45 |
| tpollard | I suppose that's a wider question | 09:47 |
| juergbi | it's on the website but like this: "BuildStream master also requires buildbox-casd: Install BuildBox" | 09:48 |
| juergbi | the latter being a link | 09:48 |
| juergbi | we don't currently have separate install instructions for 1.2 and master | 09:48 |
| juergbi | and master documentation points to the shared install instructions: https://docs.buildstream.build/master/index.html | 09:49 |
| benschubert | juergbi: ah, that's unfortunate, ok :) | 09:50 |
| juergbi | I didn't feel like restructuring the whole website for the casd branch ;) | 09:50 |
| juergbi | and I think it's not too bad with that note | 09:50 |
| * tpollard tends to agree | 09:50 | |
| laurence | it does say 'for buildstream master you'll need buildbox' | 09:53 |
| benschubert | laurence: I was just very confused because https://buildstream.build/install.html doesn't have any reference to master | 09:58 |
| benschubert | and that is the official page for installation guides | 09:58 |
| benschubert | so we maybe should not speak about source install there but link to the other page? | 09:58 |
| tme5 | could there just be a link to the install instructions in the docs? then there is no confusion as you choose the docs version you want | 10:00 |
| laurence | remove the install guide from the website entirely, and have two install guides, one in each version of the docs - master and stable ? | 10:04 |
| laurence | and link to the docs from the website | 10:04 |
| * tlater[m] thinks this is best | 10:05 | |
| laurence | i think for us it probably is | 10:05 |
| laurence | it's a bit sad because install should be on the website | 10:05 |
| tlater[m] | We could even host the docs on the website | 10:05 |
| laurence | but it's not that important | 10:05 |
| tlater[m] | It's all static HTML anyway | 10:05 |
| *** narispo has quit IRC | 10:07 | |
| *** narispo has joined #buildstream | 10:07 | |
| gitlab-br-bot | aevri opened (was WIP) MR !1554 (aevri/nomp->master: Remove uneccesary _platform.multiprocessing) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1554 | 10:32 |
| tlater[m] | This image needs buildbox, guys: https://gitlab.com/BuildStream/buildstream-docker-images/blob/master/buildstream/nightly.Dockerfile | 10:32 |
| * tlater[m] scrambles | 10:32 | |
| tlater[m] | We should have a "installed new dependency" checklist ;) | 10:32 |
| tlater[m] | This makes me wonder if we should start depending on our own docker images with FROM | 10:45 |
| tlater[m] | It would make updating these things a lot simpler, only need to do it in one place... | 10:45 |
| benschubert | tlater[m]: normally the jinja templates should be able to take care of most of things. Moreover, we test on way more than just the fedora image used for the image we publish | 10:47 |
| tlater[m] | Yeah, that's fair enough, but it's a little difficult to add a new dependency, no? | 10:48 |
| tlater[m] | Maybe I'm reading the repository wrong | 10:48 |
| benschubert | tlater[m]: the problem comes when you need to install differently per OS | 10:50 |
| tlater[m] | Yeah, but that's always a problem. I don't like that we need to do this per-image even when they share an OS | 10:50 |
| benschubert | that's not the case | 10:51 |
| benschubert | we need to do it once for the testsuite once for the public image | 10:51 |
| benschubert | because they don't share all dependencies | 10:51 |
| tlater[m] | Ok, I'm definitely reading the repo wrong then :) | 10:51 |
| benschubert | and if we want, we can always define a jinja include and extract a specific part away :) | 10:51 |
| tlater[m] | That said, it'd be nice if the public image and testsuite shared the part of the image that is the same | 10:52 |
| tlater[m] | Because then we'd be forced to fix our docker images before publishing buildstream with a new dpeendency | 10:52 |
| benschubert | tlater[m]: I was pretty sure tests were run in the image before publishing. Am I mistaken? | 10:52 |
| benschubert | tlater[m]: https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/275651069 ah no, we just do bst --version | 10:53 |
| tlater[m] | Yeah | 10:53 |
| benschubert | updating the check here could be a good thing if you have an idea though :) | 10:53 |
| * tlater[m] should sit down and think through that :| | 10:54 | |
| tlater[m] | I still think building the test images FROM the production images would make more sense | 10:55 |
| benschubert | but that would complicate the whole pipeline a lot :) | 10:55 |
| benschubert | and we don't want to publish the images if anything failed | 10:55 |
| tlater[m] | Hm, yeah, that's fair enough. | 10:58 |
| * tlater[m] starts building the buildstream docker images with buildstream | 10:58 | |
| *** phil has quit IRC | 10:59 | |
| *** phil has joined #buildstream | 10:59 | |
| tme5 | benschubert, thanks for your comments, do you think it's ok to run the entire test in a subprocess, for those that use them? | 11:17 |
| tme5 | and also I was considering trying to get a pytest marker to do what you want, as I don't think loaders are really part of the user API | 11:18 |
| tme5 | I can't see any reason why the whole test couldn't be, in fact it would probably simplify things, as there is a lot of duplicate arg passing and setup between the parent and child | 11:21 |
| juergbi | tme5: for the internal storage tests, I expect this to be fine. not sure about the artifactcache tests, though, as they also use the `cli` API | 11:22 |
| benschubert | tme5: One problem I can see would be with fixtures that would be required to run in the subprocess. Otherwise I think we can run everythin in the sp. | 11:22 |
| benschubert | You can customize pytest test collection and build the tests on the fly :) | 11:22 |
| benschubert | I also wonder if we could use pytest-xdist to give it to us for free actually, juergbi tme5 | 11:24 |
| benschubert | since it runs tests in subprocesses already | 11:24 |
| benschubert | if we could specify that some tests need an isolated runner | 11:25 |
| benschubert | that would be stopped and restarted after the test | 11:25 |
| benschubert | we would be able to remove all the rest and just run tests normally :) | 11:25 |
| juergbi | maybe, don't know xdist that well | 11:25 |
| benschubert | https://pypi.org/project/pytest-xdist/ we already use it for the multiprocessing of tests on Master | 11:26 |
| benschubert | on CI | 11:26 |
| benschubert | if we have a way of telling xdist to kill after the test... | 11:27 |
| benschubert | https://github.com/pytest-dev/pytest-forked we could also get some inspiration from that | 11:27 |
| benschubert | tme5: having a custom collection and replicating part of pytest-forked should give us a nice, correct reporting and such I guess | 11:32 |
| ironfoot | I'm trying to create some symlinks in a script element, and I'm currently running `ln -sf /ostree "%{SOME_PATH}"/ostree` in one of the commands. | 11:45 |
| ironfoot | symlink is created, but in a weird way: | 11:46 |
| ironfoot | lrwxrwxrwx 1 pedrotegra pedrotegra 12 Aug 20 12:41 ostree -> ../../ostree | 11:46 |
| ironfoot | what magic is this :/ | 11:47 |
| benschubert | ironfoot: are you thinking aobut the "../.." ? We are translating every symlink to a relative one as part of putting files in the CAS | 11:47 |
| ironfoot | yes, that's exactly what I'm talking about | 11:48 |
| ironfoot | and that would explain it | 11:48 |
| ironfoot | I guess to avoid that I can make them relative | 11:53 |
| juergbi | ironfoot: is this on 1.x stable or master? | 11:58 |
| juergbi | there should be no symlink mangling in master anymore | 11:58 |
| ironfoot | juergbi: 1.x | 12:06 |
| ironfoot | What was the reason for this? | 12:06 |
| juergbi | the reason for the mangling in 1.x or the reason why we stopped doing that? | 12:06 |
| juergbi | see https://gitlab.com/BuildStream/buildstream/merge_requests/1140 | 12:07 |
| gitlab-br-bot | marge-bot123 merged MR !1554 (aevri/nomp->master: Remove uneccesary _platform.multiprocessing) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1554 | 12:08 |
| *** phil has quit IRC | 12:17 | |
| ironfoot | the reason for the mangling in 1.x | 12:29 |
| ironfoot | I wonder if I should be moving to master | 12:29 |
| ironfoot | thanks for the information though, I got it doing what I wanted :) | 12:37 |
| *** alexandrufazakas has joined #buildstream | 13:06 | |
| *** qinusty has left #buildstream | 13:07 | |
| *** qinusty has joined #buildstream | 13:07 | |
| qinusty | . | 13:08 |
| *** alexandrufazakas has left #buildstream | 13:08 | |
| qinusty | Hey has anyone ran through the buildbox setup for fedora? https://buildstream.build/buildbox_install.html | 13:09 |
| qinusty | despite installing uuid-devel I'm getting "Package 'uuid', required by 'virtual:world', not found" | 13:09 |
| * tpollard just lazily took the binary | 13:09 | |
| qinusty | hmm, that could work but would require someone to manually update it D: | 13:11 |
| traveltissues | qinusty, your pkg-config path is set correctly? | 13:18 |
| qinusty | Not sure, I just followed the docs :/ It's not having issues with the other packages e.g. "Found grpc++, version 1.18.0" | 13:20 |
| cs-shadow | Question about our CI agents - I have a periodic job to build bst-plugins-container at 0400 UTC, that fails almost regularly due to Docker not running on the agents. | 13:20 |
| cs-shadow | I was wondering if that's coinciding with some other cleanup on workers? Am I just getting unlucky or do we have some such schedule maintenance for the workers? | 13:20 |
| tlater[m] | ironfoot: Do you know? ^ | 13:21 |
| tpollard | jennis: when does the cleanup scrip cron run? | 13:21 |
| *** traveltissues has quit IRC | 13:22 | |
| jennis | tpollard, every hour | 13:23 |
| jennis | but removes stuff older than 1 day | 13:23 |
| tpollard | Yep that's what I thought... | 13:23 |
| *** traveltissues has joined #buildstream | 13:23 | |
| *** phildawson has joined #buildstream | 13:24 | |
| cs-shadow | jennis: do we also restart the Docker daemon while doing that? | 13:28 |
| cs-shadow | (Otherwise removing old stuff shouldn't cause this particular failure) | 13:28 |
| jennis | cs-shadow, not that I'm aware of. The script looks for any droplet labelled "runner-" that is older than a day and destroys it | 13:30 |
| qinusty | Does anyone here make use of the buildstream:nightly docker image? | 13:31 |
| cs-shadow | jennis: okay, thanks for checking | 13:31 |
| qinusty | Running `bst --version` gets me "Illegal instruction (core dumped)" | 13:31 |
| juergbi | qinusty: it builds fine in Fedora when building the docker image: https://gitlab.com/BuildStream/buildstream-docker-images/blob/master/testsuite/fedora-30.Dockerfile | 13:31 |
| * cs-shadow will change the schedule time as a hail mary | 13:31 | |
| qinusty | huh strange juergbi, I was taking https://gitlab.com/BuildStream/buildstream-docker-images/blob/master/buildstream/nightly-extra.Dockerfile and attempting to build buildbox-casd ontop | 13:32 |
| qinusty | running into issues with the dependencies listed at https://buildstream.build/buildbox_install.html | 13:32 |
| juergbi | have you checked the cmake log? there might be some more info | 13:33 |
| jennis | ah, qinusty, I've just reproduced this :/ | 13:35 |
| qinusty | Which? | 13:35 |
| qinusty | illegal instruction or fedora ? :D | 13:35 |
| jennis | https://paste.gnome.org/pgn1hatrh | 13:36 |
| jennis | Yeah, just trying to run buildstream in the nightly | 13:36 |
| qinusty | Yeah, I had noticed this a week or so ago but forgot to raise it | 13:36 |
| tme5 | i was getting that! | 13:36 |
| tme5 | on my machine, when i first installed buildstream | 13:37 |
| qinusty | oh | 13:37 |
| qinusty | not inside docker? | 13:37 |
| tme5 | no | 13:38 |
| benschubert | tme5: do you remember how you fixed it? | 13:38 |
| tme5 | oh, actually it was while installing | 13:39 |
| jennis | qinusty, tme5: I've filed https://gitlab.com/BuildStream/buildstream-docker-images/issues/46 to keep track of this | 13:40 |
| tme5 | i have absolutely no recollection of how i fixed it | 13:40 |
| tme5 | i'm sorry lol | 13:40 |
| tlater[m] | juergbi: On the client end, we check if an artifact already exists on the server based on its proto before we send its cas blobs. | 13:51 |
| tlater[m] | As far as I understand CAS, this shouldn't be necessary, because the blobs won't upload if they already exist anyway | 13:52 |
| tlater[m] | is this correct, or am I going to tank performance by removing that check? | 13:52 |
| juergbi | tlater[m]: can you point me to the code in question? this might be a check to avoid uploading lots of bytes the other end already has | 13:55 |
| tlater[m] | juergbi: https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_artifactcache.py#L408 | 13:56 |
| tlater[m] | If this is just about protos, it doesn't matter, but we upload CAS blobs after that check | 13:57 |
| juergbi | ah ok, so removing that would retain the findmissingblobs call done by send_directory | 13:58 |
| juergbi | so it shouldn't actually upload blobs that aren't missing on the server | 13:58 |
| tlater[m] | Right, gotcha | 13:58 |
| tlater[m] | Thanks :) | 13:59 |
| *** CTtpollard has joined #buildstream | 14:00 | |
| juergbi | the artifact proto presence check should still be faster, though | 14:00 |
| juergbi | not sure whether the difference would be significant | 14:00 |
| *** tpollard has quit IRC | 14:01 | |
| tlater[m] | Well, once we start thinking about split artifacts that will have to be thrown out the window anyway | 14:05 |
| tlater[m] | I could first check if the artifact proto is somewhere, but that doesn't guarantee that a cas server actually has the blobs. | 14:05 |
| *** CTtpollard is now known as tpollard | 14:08 | |
| tpollard | indeed | 14:08 |
| juergbi | yes, there isn't really any other way with completely independent artifact and CAS server | 14:11 |
| tme5 | benschubert, it's a bit messy (because of use of undocumented internals), but I'm pretty sure pytest-forked is very easily adaptable to run on appropriately marked tests! | 14:29 |
| tme5 | which is nice | 14:29 |
| tme5 | i'll have a look and see if there are any issues with running the whole test in a subproc | 14:30 |
| tme5 | i suppose we are fine with using pytest internals, as testing/runcli.py already uses some | 14:38 |
| tlater[m] | tme5: It's fine in the test suite | 14:39 |
| tlater[m] | Just make sure it's well marked as something that may break. | 14:39 |
| tme5 | ok, will do | 14:41 |
| benschubert | tme5: they're public API on pytest even though not super documented, so that's fine :) | 14:44 |
| tme5 | great | 15:00 |
| *** rocklee has joined #buildstream | 15:03 | |
| *** rocklee has quit IRC | 15:04 | |
| *** narispo has quit IRC | 15:54 | |
| *** narispo has joined #buildstream | 15:54 | |
| *** tpollard has quit IRC | 16:03 | |
| *** tpollard has joined #buildstream | 16:08 | |
| *** tpollard has quit IRC | 16:10 | |
| *** tme5 has quit IRC | 16:32 | |
| *** phildawson_ has joined #buildstream | 16:34 | |
| *** phildawson has quit IRC | 16:36 | |
| *** traveltissues has quit IRC | 16:36 | |
| *** jonathanmaw has quit IRC | 17:00 | |
| *** phildawson_ has quit IRC | 18:18 | |
| *** narispo has quit IRC | 18:24 | |
| *** narispo has joined #buildstream | 18:24 | |
| *** narispo has quit IRC | 18:55 | |
| *** narispo has joined #buildstream | 18:55 | |
| *** traveltissues has joined #buildstream | 18:59 | |
| adds68 | tlater[m], have you ever come across: Handshake failed with fatal error SSL_ERROR_SSL: error:1000009c:SSL routines:OPENSSL_internal:HTTP_REQUEST. ? :D | 19:25 |
| adds68 | tlater[m], on a pull server | 19:26 |
| *** phoenix has joined #buildstream | 19:33 | |
| *** phoenix has quit IRC | 19:38 | |
| *** phoenix has joined #buildstream | 19:38 | |
| *** phoenix has quit IRC | 19:42 | |
| *** phoenix has joined #buildstream | 20:12 | |
| *** phoenix has quit IRC | 20:16 | |
| *** phoenix has joined #buildstream | 21:06 | |
| *** narispo has quit IRC | 21:09 | |
| *** narispo has joined #buildstream | 21:09 | |
| *** phoenix has quit IRC | 21:09 | |
| *** phoenix has joined #buildstream | 21:11 | |
| *** phoenix has quit IRC | 21:18 | |
| *** rdale has quit IRC | 22:17 | |
| *** narispo has quit IRC | 23:37 | |
| *** narispo has joined #buildstream | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!