| *** aminbegood has joined #buildstream | 02:41 | |
| *** aminbegood has quit IRC | 03:06 | |
| *** tristan has quit IRC | 04:43 | |
| *** tristan has joined #buildstream | 05:50 | |
| *** ChanServ sets mode: +o tristan | 06:15 | |
| tristan | We might have a minor testing issue | 06:16 |
|---|---|---|
| tristan | I'm looking at bst-1 for now so it's possible (but doubtful) that we don't have this in master | 06:16 |
| tristan | We have a bunch of tests which rely on `bst workspace list` to catch things, especially validating the project loading... but then we've changed the behavior of loading the project such that it is lazily loaded, even the main toplevel keys are not validated until `Project.ensure_fully_loaded()` gets called, which it doesn't in the case of `bst workspace list` | 06:18 |
| tristan | Might have some incorrect coverage, but maybe it's fine | 06:18 |
| *** hasebastian has joined #buildstream | 07:00 | |
| *** jude has joined #buildstream | 07:10 | |
| *** benschubert has joined #buildstream | 07:17 | |
| *** tristan has quit IRC | 07:38 | |
| *** phildawson has quit IRC | 07:46 | |
| benschubert | Can someone familiar with fdk tell me what https://gitlab.com/BuildStream/buildstream/-/jobs/547188942 means? I'm tempted to build a lower target given the number of failures and the time take to build all of this | 07:46 |
| benschubert | *fsdk | 07:46 |
| *** phildawson has joined #buildstream | 07:46 | |
| WSalmon | i think its a ostee saying the disk is about to run out of space | 07:48 |
| WSalmon | ie ostree is being run in the sandbox and is detecting that if it writes any more files then the disk it is on will fill up beond its threshold | 07:48 |
| benschubert | So our overnight runner would not have enough space for building the overnight anymore? Do we have metrics for this runner? Or could anyone check this runner? | 07:49 |
| * WSalmon looks | 07:51 | |
| WSalmon | benschubert, it looks like its a docker machine with the default harddrive attached, i will look up what that is | 07:57 |
| benschubert | cheers | 07:57 |
| WSalmon | afaict it has 80gb | 07:59 |
| WSalmon | i dont know if this is sensible | 07:59 |
| WSalmon | hango on | 07:59 |
| WSalmon | it looks like a digitalocean-size=s-8vcpu-32gb which by default has a 640gb hardrive if you make it in the gui | 08:02 |
| WSalmon | i found the runner and it clames to have a 640 volume attached | 08:04 |
| WSalmon | but its using the docker machine key so i cant ssh in and snoop around | 08:05 |
| *** rdale has joined #buildstream | 08:05 | |
| WSalmon | docker machine thinks its closed that runner down too even tho it seems to be up | 08:08 |
| *** seanborg_ has joined #buildstream | 08:09 | |
| WSalmon | so its 80 for most runners but 640Gb for the overnight one, valentind dose 640Gb seem enough? | 08:15 |
| *** hasebastian has quit IRC | 08:24 | |
| *** tpollard has joined #buildstream | 08:25 | |
| *** tristan has joined #buildstream | 08:28 | |
| WSalmon | benschubert, you might get more luck asking #freedesktop-sdk on freenode if valentind is afk atm. | 08:36 |
| benschubert | ah I can wait for him to come back :) thanks! | 08:37 |
| *** santi has joined #buildstream | 08:42 | |
| radiofree | hi, is there any element i can use to achieve the same as doing bst artifact checkout -f --deps none something.bst output/ && bst artifact checkout -f --deps none something-else.bst --directory output/ ? | 08:56 |
| radiofree | i.e i just want the artifacts ouputs and none of its build/runtime dependencies | 08:56 |
| benschubert | you should be able to specify multiple artifacts to `artifact checkout` if I remember correctly | 08:59 |
| benschubert | radiofree: ^ | 08:59 |
| radiofree | doesn't look like it, also was wondering if there was an element type i could use for this so i could just do `bst checkout combined.bst` | 09:03 |
| *** ChanServ sets mode: +o tristan | 09:03 | |
| tristan | There isn't, it's a feature I've been wanting to add to compose for a very long time though | 09:03 |
| tristan | latest example of something I think it could be useful for: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1913#note_339922579 | 09:04 |
| tristan | compose is for creating compositions, and we originally created it with support for filtering split domains, but never gave it the capability of selecting specifically which elements should be included | 09:05 |
| benschubert | ah yeah good point | 09:05 |
| benschubert | tristan: when you have time, I'd appreciate your thoughts on https://gitlab.com/BuildStream/buildstream/-/merge_requests/1919 I just need to finish the benchmarking (marginal changes from what I see in my original benchmarks) (ignore the two #FIXME comments, I need to remove that) | 09:06 |
| radiofree | ok thanks | 09:11 |
| tristan | benschubert, will be a moment | 09:12 |
| radiofree | oh btw, has anyone tried building/running buildbox-casd on fedora 32? when it launches it sits there using 100% of a core and never seems to do anytihng | 09:12 |
| tristan | radiofree, factoring this into compose can be quite tricky (but I think is worthwhile), on the other hand, writing a plugin for your own purpose which does _only that_ should not be too difficult | 09:13 |
| tristan | not sure it's worth your while | 09:13 |
| traveltissues | can someone with merge access please take a look https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/merge_requests/267 | 09:13 |
| radiofree | ok i'll take a look at plugins | 09:13 |
| tristan | benschubert, ooooooh this will be a fun review... nice to see that in the pipeline :D | 09:13 |
| valentind | benschubert, 3% error in ostree is annoying. If you have a huge disk and plenty of place it does not mean much. But in this case I suppose the builders do not have much space. | 09:14 |
| valentind | Let me see the size they have. | 09:14 |
| benschubert | tristan: ;) and this enables me to add variables resolving in the sources for "free" afterwards :D | 09:14 |
| tristan | yup | 09:14 |
| benschubert | actually, do you want it in the same MR to have a full picture? I already have the branch | 09:14 |
| valentind | 80gb. The build should not take that much. I suppose it is because builders are re-used. | 09:16 |
| traveltissues | santi: since i have previous commits at buildbox-common can i have dev permission please | 09:16 |
| benschubert | valentind: yeah :/ Not sure what happens there | 09:16 |
| valentind | benschubert, could you send the buildstream logs the gitlab artifacts for this job? | 09:17 |
| valentind | So it is easier to see issues. | 09:17 |
| benschubert | valentind: not sure I understand? | 09:18 |
| benschubert | https://gitlab.com/BuildStream/buildstream/-/jobs/547188942 was the build | 09:18 |
| valentind | You know that gitlab ci can have artifacts? | 09:18 |
| benschubert | ah yeah | 09:18 |
| valentind | Yes, it would be nice that the logs of all builds go there. | 09:19 |
| valentind | Also does bst2 have some cache logs like bst 1.4? | 09:19 |
| benschubert | we could get casd debug logs yep, but thei're very verbose | 09:19 |
| valentind | That shows the cache usage. | 09:19 |
| benschubert | ah no, I don't think we have that | 09:19 |
| valentind | OK, it would have been nice to have some information on the disk used by cache. Just to make sure the issue is about cache artifacts that are bigger than they should. | 09:21 |
| santi | traveltissues: you should have dev permissions across buildbox. is it not working? | 09:21 |
| benschubert | valentind: ostree seems to ask for 160k, that seem small though | 09:21 |
| valentind | At least temporarily you could have some "df" in the commands so that we can see what it starts with. | 09:21 |
| valentind | benschubert, That is just one object. | 09:21 |
| valentind | I do not think it is finished after clinfo. | 09:22 |
| benschubert | ah I see | 09:22 |
| traveltissues | santi: i think i might just have needed an approval and misread it | 09:23 |
| benschubert | valentind: would you have by any chance some monitoring on those machines? The gitlab-runner metrics would give us everything we'd need | 09:23 |
| santi | radiofree: casd could be calculating the disk usage. or does that happen for an empty cache directory? | 09:25 |
| radiofree | empty cache directory | 09:26 |
| benschubert | valentind: in the meantime, I'll add a df -h at the end of the run and rerun. Expect results tonight/tomorrow x) | 09:27 |
| radiofree | santi: 100% reproducible for me on fedora 32 | 09:28 |
| radiofree | /usr/local/bin/buildbox-casd /home/james/test/ | 09:28 |
| *** aminbegood has joined #buildstream | 09:28 | |
| tristan | benschubert, Ok I'll look at that now, you can look at https://gitlab.com/BuildStream/buildstream/-/merge_requests/1923 ;-) | 09:28 |
| benschubert | tristan: deal :) | 09:28 |
| santi | interesting, radiofree. can you try running it with `--log-level=trace`? | 09:36 |
| valentind | benschubert, On failure it would be nice to du also the path the artifact cache and source cache. | 09:38 |
| radiofree | santi: i'll open an issue later with some more debugging, using the prebuilt binaries for now | 09:39 |
| WSalmon | WSalmon> it looks like a digitalocean-size=s-8vcpu-32gb which by default has a 640gb hardrive if you make it in the gui | 09:39 |
| WSalmon | <WSalmon> i found the runner and it clames to have a 640 volume attached | 09:39 |
| WSalmon | <WSalmon> but its using the docker machine key so i cant ssh in and snoop around | 09:39 |
| WSalmon | valentind, ^ | 09:39 |
| benschubert | valentind: ok I'll add this to the job. | 09:40 |
| valentind | WSalmon, you can with "docker-machine ssh" | 09:40 |
| radiofree | santi: but it seems to stop on 2020-05-12T10:39:47.931+0100 [1279544:139770070870144] [buildboxcasd_localcasinstance.cpp:52] [INFO] Using `FuseStager` as staging backend | 09:40 |
| WSalmon | WSalmon> docker machine thinks its closed that runner down too even tho it seems to be up | 09:40 |
| WSalmon | i didnt mange | 09:40 |
| WSalmon | i think you can ssh in too | 09:40 |
| valentind | Right, because it is off. | 09:40 |
| radiofree | i don't think it's the buildbox-fuse causing issues btw, since that's the prebuilt one which works with the prebuilt buildbox-casd | 09:40 |
| WSalmon | it was up in the DO gui | 09:40 |
| radiofree | (~/.local/bin/buildbox-casd works fine) | 09:41 |
| santi | yes, casd should be only looking for the path to buildbox-fuse at that point, not invoking it, radiofree. | 09:44 |
| santi | but it's definitely hanging before starting to listen to requests | 09:44 |
| santi | please open that issue and we'll have a look | 09:45 |
| tristan | benschubert, I'll cancel some of your older pipelines if that's alright | 09:57 |
| tristan | getting crowded in there | 09:58 |
| benschubert | tristan: ah sure | 09:58 |
| benschubert | just not the fsdk one please :) | 09:58 |
| tristan | jeez I hope I didn't, which one is that ? | 09:59 |
| tristan | benschubert, they look like all MRs, only cancelled recent ones (started in last hour) which are not tagged "latest" | 10:00 |
| tristan | resolve-variables and cache-key-helper anyway | 10:01 |
| benschubert | tristan: ah you didn't kill it, perfect :) | 10:01 |
| benschubert | https://gitlab.com/BuildStream/buildstream/-/jobs/548037489 this one. I'm hopiing it will finally pass | 10:01 |
| tristan | probably not on the first page of the pipelines page :) | 10:03 |
| benschubert | definitely not x) | 10:06 |
| * tristan noticed that utils._parse_version() didn't guard against non-string versions and goes to fix that in master, now that the bst-1 version is safer | 10:08 | |
| tristan | Well, good start to this week, what's next... have to complete the juncle work | 10:09 |
| * tristan notices juergbi made a reply today | 10:09 | |
| benschubert | in a few minutes we'll have variables expanded in sources too ;) | 10:10 |
| tristan | breakthrough ! | 10:11 |
| tristan | heh | 10:12 |
| tristan | I guess there is no way around the deprecation step, or, things will just go more smoothly this way | 10:12 |
| tristan | update such external plugin repos, and then come back and remove the APIs altogether | 10:13 |
| benschubert | yep that's my plan | 10:13 |
| benschubert | should not be too hard | 10:13 |
| tristan | juergbi, am I missing something ? or aren't warnings which are allowed to be fatal, all non-fatal by default ? | 10:14 |
| tristan | let me give this some thought, I think I'm liking the simplicity of making these (possibly fatal) warnings with the possibility of extending this in the future | 10:16 |
| benschubert | tristan: all warnings can be set as fatal or not in Python if that answers your question? | 10:17 |
| benschubert | not sure I got everything though | 10:17 |
| juergbi | tristan: warnings: they are indeed all non-fatal by default right now. I'm not sure whether it would be a good idea to make one fatal by default | 10:19 |
| tristan | benschubert, I mean BuildStream has Plugin.warn(), and we can set them as fatal in project.conf with fatal-warnings: | 10:19 |
| tristan | juergbi, right, I just thought your mail insinuated that we should "make it a fatal warning" | 10:20 |
| tristan | juergbi, I didn't receive the email yet :-S | 10:20 |
| tristan | heh | 10:20 |
| tristan | Just reading it off mailman | 10:20 |
| benschubert | tristan: ah sorry, thought we were still on the previous thing sorry :) | 10:21 |
| tristan | context-switch :) | 10:21 |
| * tristan wants to close the book on the juncle | 10:21 | |
| tristan | the juncle book ! | 10:21 |
| tristan | Am I the only one who didn't received it ? so strange | 10:23 |
| benschubert | tristan: for source variables usage: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1924 if you're interested, still need to add a test | 10:25 |
| jjardon | tristan: hi, did you have the time to create a bst 1.4.3? | 10:30 |
| tristan | jjardon, on the way, I've got the ostree fix in there now | 10:32 |
| tristan | juggling juncles | 10:32 |
| jjardon | coolio, thanks! | 10:34 |
| tristan | benschubert, did you receive https://mail.gnome.org/archives/buildstream-list/2020-May/msg00006.html by mail ? | 10:34 |
| benschubert | tristan: I did, an hour ago almost | 10:34 |
| tristan | Ok, so it's not a problem with mailman | 10:35 |
| tristan | jjardon, did you receive that ? | 10:35 |
| jjardon | tristan: yup | 10:36 |
| tristan | ok that's doubly strange | 10:38 |
| tristan | I'll writeup a reply and hope the mail comes, otherwise I'll post it awkwardly | 10:38 |
| juergbi | tristan: mailman probably doesn't deliver it to you because you're a direct recipient | 10:43 |
| juergbi | i.e., I think the issue is between our mail servers | 10:43 |
| *** seanborg_ has quit IRC | 10:45 | |
| *** seanborg_ has joined #buildstream | 10:46 | |
| juergbi | tristan: a test mail worked fine, though. let me resend the mail to you only | 10:49 |
| juergbi | although, I guess that's still not ideal for reply | 10:50 |
| juergbi | resent | 10:50 |
| *** cs-shadow has joined #buildstream | 10:52 | |
| WSalmon | if i use bst-artifact-server for my index and artifacts (without index only) is the cas bit manged by casd? | 10:54 |
| juergbi | yes | 10:55 |
| WSalmon | cphang, ^ | 10:55 |
| juergbi | bst-artifact-server acts as a thin proxy | 10:55 |
| cphang | well that explains things | 10:55 |
| WSalmon | juergbi, i have seen lots of exmaples of casd talking to a "upstream" casd, are there any stress tests of it talking to anything else? | 10:56 |
| WSalmon | stress/smoke etc | 10:56 |
| cphang | juergbi for context we're still trying to root cause https://gitlab.com/celduin/infrastructure/celduin-infra/-/issues/144 right now. It seems to work fine in examples where bst-artifact-server is used only, but we're hitting the described issue when using an external CAS service. | 10:57 |
| juergbi | WSalmon: in our regular test suites we only test casd against casd as everything else obviously requires more setup. our remote execution tests do test buildbox-casd talking to buildgrid CAS server, though, iirc. it's more of a side effect, though, not meant as a CAS protocol test | 10:59 |
| juergbi | cphang: ok, certainly possible that there is an interoperability bug somewhere | 11:00 |
| cphang | juergbi this error only appears under load, so it's understandable you wouldn't have caught it in your test suites. | 11:01 |
| juergbi | have you been able to print the exception message that causes the 'unknown' gRPC error? | 11:01 |
| juergbi | or is it not an exception in buildbox-casd after all? | 11:01 |
| cphang | That's what makes it difficult, it's an unhandled exception we think at buildbox level, which means it only gets caught here https://github.com/grpc/grpc/blob/master/include/grpcpp/impl/codegen/method_handler_impl.h#L44 | 11:02 |
| juergbi | cphang: but if we know in which gRPC method the exception happens, wouldn't be as simple as adding a general catch clause with better log to buildbox-casd code? | 11:03 |
| juergbi | to print the actual exception | 11:03 |
| tristan | juergbi, I've received it now, I'm not sure it was your send either | 11:04 |
| cphang | yep but https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/merge_requests/267 so https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/merge_requests/268 yielded no useful additional log messages | 11:04 |
| tristan | juergbi, i.e. I've only now, got a copy with the list and Chandan also on CC | 11:04 |
| tristan | Anyway | 11:04 |
| juergbi | interesting, so not the resend | 11:05 |
| cphang | traveltissues you did a little more digging this morning? | 11:05 |
| juergbi | I would add a try catch at the top-level method handler of the relevant method(s) | 11:06 |
| juergbi | cphang, traveltissues: could even add it to the CALL_INSTANCE macro definition in buildboxcasd_casserver.cpp | 11:07 |
| juergbi | covering all regular operations | 11:07 |
| traveltissues | we could, i wonder if it's not even unknown but not passed correctly | 11:08 |
| traveltissues | well, not forwarded | 11:08 |
| cphang | At this point a stack trace would be really handy | 11:11 |
| traveltissues | it would be good to at least have the logs from buildbox | 11:14 |
| traveltissues | if those exist somewhere | 11:14 |
| cphang | Yeh we're saving them in fdsdk runs, one second | 11:14 |
| traveltissues | i should have asked earlier | 11:15 |
| traveltissues | i think the obvious interesting points are BatchUpdateBlobs and FindMissingBlobs | 11:15 |
| traveltissues | and these should have basic logging | 11:16 |
| cphang | e.g. https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/547130278/artifacts/file/cache/buildstream/logs/_casd/1589212610.2864647.log | 11:16 |
| traveltissues | ty | 11:16 |
| traveltissues | so the last logged call is FindMissingBlobs | 11:16 |
| traveltissues | and `2020-05-11T16:09:06.168+0000 [31:281473366094064] [buildboxcommon_client.cpp:678] [ERROR] Batch download failed: Retry limit exceeded. Last gRPC error was 14: Socket closed` | 11:17 |
| cphang | yeh | 11:19 |
| cphang | also see https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/546340524/artifacts/file/cache/buildstream/logs/_casd/1589185623.1610744.log | 11:19 |
| traveltissues | why do we believe this is not a server issue? | 11:21 |
| cphang | because we can get this occurring without these Batch download errors. Because those errors point to a pull, not a push, and that we never see these errors sever side. | 11:22 |
| traveltissues | code 1 is cancelled i think | 11:22 |
| cphang | https://gitlab.com/celduin/infrastructure/celduin-infra/-/issues/144#note_338058901 | 11:23 |
| cphang | and we never see this with bst1.4.x either | 11:24 |
| traveltissues | but would you see a server-side error if the request were never communicated to the server? | 11:25 |
| traveltissues | for some network related issue? | 11:25 |
| cphang | so we're using AWS, so we wouldn't see it if there were an error at the ingress level say with an NLB, but errors at that level would be catastrophic, I would expect to see no traffic at all, this is much more transient than that. | 11:28 |
| coldtom | hmm, but we're successfully communicating with the server most of the time, and we use the same runners for 1.4.x which just seems to work | 11:29 |
| traveltissues | what versions are you using now? | 11:30 |
| cphang | both bst 1.4.x and 1.9.x both make REAPI calls and they both communicate successfully with the server. It's just that with 1.9.x clients we get this transient error | 11:30 |
| coldtom | 1.93.3 traveltissues | 11:30 |
| cphang | 1.93.x clients sorry | 11:30 |
| coldtom | the server is 1.93.1 i think, but the artifact communication seems fine afaik (plus i don't think the artifact server has changed between these versions) | 11:31 |
| traveltissues | and did you also see this problem with 1.93.1 client? | 11:36 |
| coldtom | not afaik, but it's possible that we missed it and assumed the error was caused by something else | 11:37 |
| traveltissues | if you were able to use 1.93.1 without that issue then that's a good sign | 11:38 |
| tristan | juergbi, looks like I now received the other... it's as if your explicit email bumped into the stuck one on the mail server and pushed it forward | 11:40 |
| tristan | juergbi, maybe our mail server has a narrow pipe and is not used to large emails, so some force needs to be applied | 11:40 |
| tristan | Anyway I've managed a reply | 11:41 |
| tristan | juergbi, not easy subject material | 11:41 |
| cs-shadow | tristan: hi, I'd appreciate your feedback on !1904 if you've got some time | 11:48 |
| *** aminbegood has quit IRC | 11:48 | |
| cs-shadow | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1904 (about --deps build and --deps run) | 11:48 |
| radiofree | santi: i did a bit of digging and it looks the issue is caused by grcp-c++-1.26.0 (the version in fedora) | 11:51 |
| radiofree | santi: building against 1.25.0 and it works | 11:51 |
| tristan | cs-shadow, I see, so we basically just backed out `plan` from most of those ? | 11:52 |
| tristan | cs-shadow, what do you think about removing `plan` entirely ? | 11:52 |
| tristan | cs-shadow, I wonder if, in the cases where `plan` exists at all, it is just the default behavior if `--deps` is not specified at all ? | 11:53 |
| tristan | cs-shadow, I'm only asking cause, I wonder if we could refer to the `Scope` documentation for an explanation of `plan` and possibly have more coherent documentation | 11:53 |
| juergbi | radiofree: ah, right, Fedora is still on broken gRPC :-/ | 11:54 |
| juergbi | https://gitlab.com/BuildStream/buildstream/-/issues/1283#note_328679088 | 11:54 |
| juergbi | if anyone has Fedora connections, please push https://bugzilla.redhat.com/show_bug.cgi?id=1812743 / https://bugzilla.redhat.com/show_bug.cgi?id=1794566#c9 | 11:55 |
| radiofree | yep, still very much broken | 11:55 |
| santi | thanks, radiofree. I'll have a look at the changelog | 11:55 |
| tristan | cs-shadow, anyway I think I'm happy with the patch, I'll comment there | 11:56 |
| WSalmon | is there a setting that turns of pushing elements you just pulled? i seem to rember talking about this a while ago? | 11:59 |
| cs-shadow | tristan: yeah, i removed `plan` for now and a couple of other options that didn't seem to work as expected, like `build --deps none` and `build --deps run` | 12:02 |
| cs-shadow | other than that, it's mostly complete. I think plan needs some special treatment, i don't know exactly what though :) | 12:02 |
| tristan | cs-shadow, Looks mostly good to me, I suggested a change in how the tests are written, I think they'd be much better that way | 12:04 |
| WSalmon | cs-shadow, there seems to be a bug with plan that i am trying to make a nice test case for so we can fix, i dont think we want to remove the surounding code because we havnet fixed the bug... | 12:04 |
| tristan | cs-shadow, e.g. see tests/frontend/show.py::test_show_except() for another example | 12:05 |
| WSalmon | but i may have the wrong context from your comment | 12:05 |
| cs-shadow | tristan: that seems much better, i'll change it to that style. thanks | 12:05 |
| cs-shadow | WSalmon: the current patch won't remove plan (or any other option for that matter) so i think it shouldn't hurt any new tests etc | 12:06 |
| WSalmon | ie, buildstream seems to be building build-deps build-deps when it should only do that for a all plan | 12:06 |
| cs-shadow | however i'd be curious to know that the bug is | 12:06 |
| WSalmon | *when they are cased | 12:07 |
| WSalmon | cached | 12:07 |
| WSalmon | remotely | 12:07 |
| cs-shadow | if ^ is true, it's definitely a bug | 12:07 |
| WSalmon | obciusly it needs to build everything if nothing is cached anywere | 12:07 |
| WSalmon | it seems to be but i am trying to make a simple test case | 12:08 |
| benschubert | WSalmon: please ping me once you have a test case, I can have a look | 12:08 |
| WSalmon | will do | 12:08 |
| benschubert | tristan: we don't seem to have many tests around variables. Mainly: https://gitlab.com/BuildStream/buildstream/-/blob/master/tests/format/variables.py#L48 For the source variables, should I just enter a line here to ensure they are resolved or do you have a better idea? | 12:21 |
| tristan | benschubert, I'm out for the day hehe ;-) | 12:24 |
| * tristan goes for dinner | 12:24 | |
| benschubert | oh! sure, I'll find a place for the tests then ;) have a good evening | 12:24 |
| tristan | benschubert, at a glance, I didn't think we have a way to show source configuration with `bst show`... if we do now, I guess that should be sufficient ? | 12:26 |
| tristan | anyway, gotta run | 12:26 |
| benschubert | yep, will see, thanks | 12:26 |
| WSalmon | benschubert, is there a way to run show and only get my direct dependencies? | 12:27 |
| benschubert | WSalmon: I don't think so no | 12:28 |
| *** tristan has quit IRC | 12:29 | |
| cs-shadow | WSalmon: you can use `--format %{deps} %{build-deps} %{runtime-deps}` to get direct dependencies. Use one or more of all-deps, build-deps and runtime-deps for the desired effect | 12:33 |
| cs-shadow | with `bst show` that is | 12:33 |
| benschubert | *goes and re-read `bst show --help` | 12:33 |
| cs-shadow | i've also just found a bug in `show --format`. Give it something like `bst show --format '%{name} ${build-deps}` and watch it blow up. | 12:36 |
| cs-shadow | note the `$` near build-deps, which was a typo i made while trying this | 12:36 |
| *** seanborg_ has quit IRC | 12:37 | |
| *** seanborg_ has joined #buildstream | 12:37 | |
| benschubert | #define blows up? | 12:37 |
| cs-shadow | https://gitlab.com/snippets/1975901 | 12:39 |
| benschubert | oh great. Mind creating an issue? | 12:40 |
| cs-shadow | it's probably our formatting stuff. incidentally `--format ${build-deps}` is fine on its own, but not if there's another subsitution variable in front | 12:40 |
| cs-shadow | will do. there's yet another issue I think | 12:41 |
| cs-shadow | `--format %{deps}` appears to be duplicating build and runtime deps if some element is present is in both sets | 12:41 |
| benschubert | my guess is, our regex are greedy, thus it matches 'name} ${build-deps' | 12:41 |
| cs-shadow | that sounds plausible | 12:42 |
| *** aminbegood has joined #buildstream | 13:29 | |
| WSalmon | juergbi, could we add pkg-config to the deb and ubuntu build deps of https://buildstream.build/buildbox_install.html | 13:45 |
| WSalmon | it catches me out everytime i build on ubnutu | 13:45 |
| juergbi | ah, is this not an indirect dependency of the corresponding -dev packages? | 13:46 |
| juergbi | in which case, sure, let's add it | 13:46 |
| juergbi | WSalmon: btw: I've commented on !1909 | 13:46 |
| WSalmon | i assume not as i keep hitting it | 13:48 |
| benschubert | juergbi: I like your solution to !1909 | 13:49 |
| WSalmon | so it should be fixed? i will verifiy afterwork and close my issue if it works | 13:50 |
| benschubert | WSalmon: if you try juerg's branch mentionned in the comment | 13:50 |
| WSalmon | i see you have rebase my test, i assume my test is good enough | 13:51 |
| WSalmon | might be good to add a bst show style test or have you done that too? | 13:51 |
| benschubert | WSalmon: the test with just adding a 'sandbox' key? I commented on the PR and asked for the test to be a separate test :) | 13:55 |
| WSalmon | sorry i got the MR and the branch mixed up, i see now. | 13:58 |
| WSalmon | juergbi, both times i have had that was from 19.10 havent tried older maybe something changed | 14:09 |
| *** aminbegood has quit IRC | 14:13 | |
| juergbi | WSalmon: I assume you're referring to the pkg-config issue. I'm fine adding it to the instructions in any case | 14:15 |
| WSalmon | juergbi, yes | 14:15 |
| * WSalmon would have appreciated a fyi about https://gitlab.com/BuildStream/buildstream/-/merge_requests/1919 | 14:31 | |
| benschubert | WSalmon: is it breaking something? | 14:39 |
| WSalmon | no but it changes code i was looking at to try and understand 1909 | 14:41 |
| benschubert | ah yeah, was planning into pingin the 1909 MR once it was reviewed, but juerg was quicker | 14:42 |
| benschubert | WSalmon: slightly tangential but also concerning variables: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1924 | 14:59 |
| WSalmon | thanks benschubert | 15:01 |
| *** tristan has joined #buildstream | 15:05 | |
| *** ChanServ sets mode: +o tristan | 15:05 | |
| radiofree | am i right in thinking i could use buildbarn as a cas? | 15:49 |
| radiofree | using buildbox as a proxy? | 15:49 |
| coldtom | radiofree, you can use the split artifact cache config to do this, yup | 15:49 |
| radiofree | any links to an example? | 15:50 |
| WSalmon | radiofree, the simles thing is to use bst's own cache server but it only scales so far | 15:51 |
| WSalmon | simplest | 15:51 |
| coldtom | https://docs.buildstream.build/master/format_project.html#split-cache-servers | 15:51 |
| radiofree | our client has an existing buildbarn deployment, would be good to use that since they want to play around with some stuff | 15:56 |
| coldtom | radiofree, note, you'll still need to deploy a bst-artifact-server to hold the artifact protos | 15:56 |
| coldtom | oh, and the split cache config is master only, so if you want to use 1.4 you may need to shove them both behind a webserver | 16:00 |
| cphang | radiofree I would look at https://gitlab.com/celduin/infrastructure/celduin-infra where we do exactly this. | 16:00 |
| radiofree | briiiiliant | 16:01 |
| radiofree | oh btw i should say great work to everyone here working on these projects - i'd never used buildstream before friday and by saturday i was running gpu accelerated docker containers on a tegra | 16:01 |
| radiofree | pretty much all from reading through freedesktop/jetbot/celduin etc... | 16:02 |
| radiofree | cphang: so there's some additional deployment work required? | 16:05 |
| * radiofree will take this to celduin | 16:06 | |
| WSalmon | juergbi, https://gitlab.com/celduin/bsps/example-app/-/jobs/549001570 this job just pulled every single element, when in bst1.4 it would have just pulled the top most element | 16:19 |
| juergbi | WSalmon: ok, thanks, will take a look | 16:20 |
| WSalmon | the celduin cache has everything | 16:20 |
| WSalmon | but the fdsdk cache dosent cache the boot strap | 16:21 |
| WSalmon | so bst starts fetching those bits | 16:21 |
| WSalmon | benschubert, you asked about this too | 16:21 |
| *** devcurmudgeon has quit IRC | 16:26 | |
| *** Frazer has quit IRC | 16:26 | |
| *** cphang has quit IRC | 16:26 | |
| *** ikerperez has quit IRC | 16:26 | |
| *** cphang has joined #buildstream | 16:27 | |
| benschubert | WSalmon: thanks | 16:31 |
| *** tpollard has quit IRC | 17:18 | |
| juergbi | WSalmon, benschubert: I think this fixes this regression: https://gitlab.com/BuildStream/buildstream/-/commits/juerg/dynamic-plan | 17:21 |
| juergbi | we might want to further improve the code but I think we should merge these reverts as soon as we have a regression test | 17:21 |
| benschubert | juergbi: mind doing a benchmark on this before? I have a bad feeling for other cases | 17:24 |
| benschubert | in which case I'd rather try to really finally move to a dynamic pipeline | 17:24 |
| tristan | juergbi, late night random thought: I wonder if a different dependency type might add a different dynamic to the junctions discussion | 17:24 |
| tristan | for instance, what if there was a build dependency that did not have the property of needing to run | 17:25 |
| juergbi | benschubert: sure, can run a benchmark but maybe tomorrow | 17:25 |
| benschubert | juergbi: sure | 17:26 |
| juergbi | tristan: how would this help? | 17:26 |
| tristan | it would enable one to distinguish an aspect of what an element might want to do with this artifact | 17:26 |
| *** santi has quit IRC | 17:27 | |
| tristan | I'm not entirely sure it would help honestly, as possibly such a dependency type might make no difference, one might agrue that adding an element with only build dependencies could express the same thing | 17:29 |
| juergbi | yes, that's what I was thinking | 17:29 |
| tristan | Today radiofree asked how to "make an artifact out of artifact A and B, without their dependencies" | 17:30 |
| tristan | I wonder if we have a straight forward answer to that with an approach which adds elements | 17:31 |
| tristan | (quotes mine, he didn't phrase it that way) | 17:31 |
| tristan | Anyway, it's a different optic, and just a random thought which I think is a different perspective | 17:32 |
| juergbi | right now it can be done either with a filter or with a custom element that doesn't stage indirect dependencies | 17:33 |
| juergbi | *custom plugin | 17:33 |
| tristan | right, I recommended the latter, and I think it's a feature missing from `compose` | 17:33 |
| tristan | `compose` should be able to be selective about the elements it includes I mean | 17:33 |
| juergbi | could make sense | 17:34 |
| tristan | filters do strange and unexpected things heh | 17:35 |
| tristan | which turn out to be useful | 17:35 |
| *** phildawson has quit IRC | 17:49 | |
| *** phildawson has joined #buildstream | 17:49 | |
| *** phildawson has quit IRC | 17:53 | |
| *** pointswaves has joined #buildstream | 18:25 | |
| *** aminbegood has joined #buildstream | 18:36 | |
| *** aminbegood has quit IRC | 18:51 | |
| *** narispo has quit IRC | 19:09 | |
| *** jude has quit IRC | 19:12 | |
| *** cs-shadow has quit IRC | 19:22 | |
| *** chipb has quit IRC | 19:34 | |
| *** chipb has joined #buildstream | 19:35 | |
| chipb | is there a straightforward way to seed an 'sdk' rootfs using, say, debootstrap in lieu of depending on a central ostree repository? | 19:36 |
| chipb | I assume I'd probably need a custom element plugin. | 19:37 |
| chipb | but are there any particular gotchas I should be aware of? | 19:37 |
| benschubert | chipb: do you mean creating an initial chroot to build? You can use docker images through `docker source` (from bst-plugins-experimental), or from a tar file, etc | 19:39 |
| chipb | docker itself is largely a no-go for my particular environment. I guess folks could depend on a centrally-located tarball to bootstrap, but at that point I may just do the ostree thing anyway. | 19:41 |
| benschubert | yep that would work | 19:42 |
| pointswaves | chipb, yep, have you seen freedesktop-sdk? thats a common option but if docker is a no go then a tar for a bootstrap is very common. gitlab.com/freedesktop-sdk/freedesktop-sdk/ | 19:55 |
| pointswaves | freedesktop use a tar as a starting point and then build every thing you need on top so you never touch the bootstrap, you could use them or use them as a template | 19:55 |
| pointswaves | they also do good audit stuff you may find useful | 19:56 |
| *** toscalix has joined #buildstream | 19:57 | |
| pointswaves | chipb, if you are keen on ostree there is a ostree source element in bst-plugins-experiment IIRC | 20:03 |
| *** seanborg_ has quit IRC | 20:48 | |
| *** aminbegood has joined #buildstream | 21:32 | |
| *** pointswaves has quit IRC | 21:35 | |
| *** aminbegood has quit IRC | 21:42 | |
| *** aminbegood has joined #buildstream | 21:43 | |
| *** toscalix has quit IRC | 21:43 | |
| *** aminbegood has quit IRC | 22:03 | |
| *** aminbegood has joined #buildstream | 22:04 | |
| *** aminbegood has quit IRC | 22:09 | |
| chipb | ultimately, I was trying to stay away from magic tarball. | 22:58 |
| chipb | and I want build flows to mostly Just Work without third parties needing to set up ostree for themselves. | 22:58 |
| chipb | of course, I've got more annoying constraints anyway. heh. | 23:06 |
| chipb | some of my builds require a rather large EDA toolchain. | 23:07 |
| chipb | it's probably not reasonable to copy it about as a true input artifact, so I'll probably need to bind mount it in a reasonably straightforward [to the developer] way. | 23:11 |
| *** benschubert has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!