| *** Prince781 has joined #buildstream | 00:03 | |
| *** Prince781 has quit IRC | 00:20 | |
| *** Prince781 has joined #buildstream | 02:53 | |
| *** tristan has joined #buildstream | 03:54 | |
| *** Prince781 has quit IRC | 04:50 | |
| juergbi | jjardon: Not sure, ssssam[m] set this up, iirc. I assume you have access to the username/password used by CI via secret variables in the CI settings, could you just use those? | 04:55 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (tristan/fix-track-semantics->master: Fix track semantics) #422 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/422 | 05:50 |
| tristan | juergbi, I dont think those are retrievable | 05:54 |
| tristan | well, I recall playing with those settings once and not being able to retrieve what I had previously set | 05:54 |
| tristan | (which made enough sense to me at the time, albeit inconvenient) | 05:55 |
| juergbi | I can see them, afaict | 05:55 |
| tristan | Ah, it may very well have changed, or I may have missed some setting | 05:55 |
| juergbi | there is a 'Reveal values' button | 05:56 |
| tristan | Aha | 05:56 |
| tristan | I would probably have missed that :) | 05:56 |
| tristan | or, it could be a new feature, this was probably a year ago and gitlab is pretty fast moving | 05:56 |
| juergbi | indeed | 05:58 |
| tristan | hmmm, I mucked it up | 06:06 |
| tristan | aha | 06:10 |
| gitlab-br-bot | buildstream: merge request (tristan/fix-track-semantics->master: Fix track semantics) #422 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/422 | 06:12 |
| tristan | sigh, I'd really like to refactor Pipeline | 06:12 |
| tristan | instead I'm slowly nudging it in the direction I want, but it's not enough | 06:13 |
| gitlab-br-bot | buildstream: issue #367 ("bst track refuses to track only one element") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/367 | 06:23 |
| gitlab-br-bot | buildstream: merge request (tristan/fix-track-semantics->master: Fix track semantics) #422 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/422 | 06:23 |
| gitlab-br-bot | buildstream: merge request (tristan/pylint-enable-unused-variable->master: Enable linting for unused variable) #423 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/423 | 06:39 |
| gitlab-br-bot | buildstream: merge request (tristan/pylint-enable-unused-variable->master: Enable linting for unused variable) #423 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/423 | 06:51 |
| *** aday has joined #buildstream | 08:08 | |
| *** toscalix has joined #buildstream | 08:08 | |
| *** jonathanmaw has joined #buildstream | 08:22 | |
| jjardon | juergbi: I tried but it doesnt seem to work; it only allow me to create new repos in <usernameci>/ namespace | 08:31 |
| *** noisecell has joined #buildstream | 08:45 | |
| *** bethw has joined #buildstream | 08:57 | |
| *** dominic has joined #buildstream | 09:06 | |
| juergbi | tristan: I didn't realize that it's so crucial in practice to avoid planning build dependencies of pulled elements | 09:07 |
| juergbi | these queueing changes will be quite big, we'd have to completely reverse planning. we'd have to queue all build-only dependencies on demand instead of via plan | 09:09 |
| juergbi | not completely reverse as we still keep runtime dependencies in the plan | 09:09 |
| juergbi | however, from the perspective of build dependencies, we'd start with the toplevel target only and then queue build dependencies as needed | 09:10 |
| tristan | juergbi, My original email had a much stronger paragraph there, but then I re-read your original text and decided I had misinterpreted it. | 09:34 |
| gitlab-br-bot | buildstream: issue #369 ("Run tests on aarch64") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/369 | 09:35 |
| juergbi | thinking about this a bit more, it might indeed not be that difficult to implement. we just don't have dynamic queueing like that so far, afaik | 09:35 |
| tristan | juergbi, I think though, if we A.) Plan in regular order, and chop off build-of-build... B.) If not locally cached, append these build-of-build dependencies *after* the elements which depend on them... C.) Implement Queue.status() to report "skip" correctly in the Queues, we can get it done | 09:35 |
| tristan | juergbi, do we need dynamic queuing at all ? well; it depends where you put the logic I guess | 09:36 |
| juergbi | tristan: I don't think this works, we have to stop them from being pulled, e.g. | 09:36 |
| juergbi | unless you say you don't care about this | 09:36 |
| gitlab-br-bot | buildstream: issue #370 ("Setup aarch64 runners") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/370 | 09:36 |
| tristan | juergbi, if they are *after* the depending elements in the Queue, then we can skip them if we have already determined that previous depending elements are being downloaded | 09:37 |
| juergbi | things happen in parallel, though | 09:37 |
| juergbi | it's not so clear cut | 09:37 |
| gitlab-br-bot | buildstream: issue #371 ("Run tests in Debian stable") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/371 | 09:37 |
| tristan | juergbi, Ah right, you'd need to block them until you know the child task is actually pulling something | 09:38 |
| juergbi | I think dynamically queueing them is cleaner and might not be that hard | 09:38 |
| tristan | juergbi, this *could* technically be done with some additional IPC, i.e. tell the parent "Ok I'm downloading the element" | 09:38 |
| tristan | but maybe dynamic queueing is cleaner indeed | 09:38 |
| juergbi | yes, but this additional interaction sounds more complex overall, harder to maintain | 09:39 |
| tristan | juergbi, it's important to not download them; because I dont want to download the binary sysroot that was needed to bootstrap the stage1 and stage2 binaries of the bootstrap, plus the stage1 and stage2 binaries | 09:39 |
| juergbi | right, I somehow forgot about half the bootstrapping part. I thought it'd be sufficient to skip source fetching for that but it's not | 09:40 |
| tristan | And, it is *very much* a goal to have developers building on their local machines, and deploying to VMs and rigs locally, *before* pushing their own patches for review to an integration pipeline | 09:40 |
| tristan | juergbi, the same thing happens with debootstrap-ostree | 09:40 |
| juergbi | yes, completely agree, it just slipped my mind | 09:40 |
| tristan | :) | 09:40 |
| juergbi | one question is what happens if such an element is tracked | 09:40 |
| juergbi | optimally, we'd have to dequeue it after tracking | 09:41 |
| tristan | currently, track queue is placed before pull queue, or any other queue | 09:41 |
| tristan | but yeah things may need to change | 09:41 |
| juergbi | mark these elements somehow and then drop them off the pipeline after tracking. only readd them if needed | 09:42 |
| jjardon | cs_shadow: do you have access to the https://hub.docker.com/u/buildstream/ account, by any chance? | 09:56 |
| tlater | jjardon: Pretty sure I do | 09:56 |
| tlater | In case you need someone to push something or somesuch, if you're just checking if cs_shadow already does, nevermind :) | 09:57 |
| *** noisecell has quit IRC | 09:58 | |
| jjardon | tlater: ah you are the man! can you create a "buildstream-debian" repo there please? so the new debian-9 images are pushed there | 09:59 |
| tlater | Sure, I'll ping you when it's up | 09:59 |
| jjardon | tlater: tvm | 10:00 |
| * tlater wonders if we have a debian image just to satisfy distro wars | 10:00 | |
| jjardon | I though the ci user will have enough powers to create it but now that part of the CI is failing: https://gitlab.com/BuildStream/buildstream-docker-images/commit/730276be22e49359820087c930f3143d651d5e64/pipelines | 10:01 |
| jjardon | tlater: distro reality I would say | 10:01 |
| tlater | Hehe, that was tongue-in-cheek, jjardon. Just curious why we need one, given this is a docker image so the actual distro matters very little | 10:02 |
| * tlater still thinks it should all be alpine | 10:02 | |
| jjardon | tlater: I have seens exactly the same python code that work in stock Debian but not in Fedora, and viceversa | 10:03 |
| tlater | Oh, I get that, but that's what the testsuite images are for, no? | 10:03 |
| *** noisecell has joined #buildstream | 10:03 | |
| tlater | jjardon: Repo is up as buildstream/buildstream-debian | 10:04 |
| tristan | tlater, confounding images again I think | 10:04 |
| tristan | tlater, jjardon is talking about the docker images we use *in gitlab* to test that BuildStream installs and works on various real distros | 10:05 |
| jjardon | tristan: thanks! | 10:05 |
| tristan | not the image we use as a base runtime for the sandbox in integration tests | 10:05 |
| jjardon | tlater: I meant | 10:05 |
| tlater | Ah, I see | 10:06 |
| tlater | Again, though, I believe the test suite images cover that | 10:06 |
| tlater | buildstream-fedora is exclusively used for bst-here at this point | 10:06 |
| tlater | Why do we need a buildstream-debian? | 10:06 |
| tristan | uhhh | 10:08 |
| tristan | tlater, we're running CI and we want to have plenty of different environments we run the same test suite in ? | 10:09 |
| tristan | tlater, i.e. every major point version of every relevant mainstream distro ? | 10:09 |
| tristan | tlater, so that our tests dont only test "Ok this worked on fedora, yay" | 10:10 |
| tlater | tristan: Not all yet, only fedora-27 and debian-8 on Linux | 10:10 |
| tlater | But that interface exists | 10:10 |
| tristan | tlater, right, that is a work in progress | 10:10 |
| tlater | If we want to add debian-9, I think it should be one of the testsuite images | 10:10 |
| tlater | *Not* a general-purpose image only used for bst-here | 10:10 |
| tristan | Ummm, maybe I dont understand the reason of this conversation; *I* dont really understand why we would need more than one *repo* to hold all of our images | 10:10 |
| tristan | tlater, agree | 10:11 |
| tristan | I.e. the bst-here script only needs one image, and serves a different purpose | 10:11 |
| * tlater is perhaps just picky about naming here | 10:11 | |
| tlater | jjardon: What do you intend to use that image for? I might be completely oblivious to something we use our buildstream-fedora image for. | 10:12 |
| tlater | But I suspect we should name that image testsuite/debian-9 | 10:13 |
| cs_shadow | jjardon: I personally don't but is it the same account as the one that's used in GitLab CI? | 10:14 |
| tlater | cs_shadow: I think you need a separate account for this, the account we use in gitlab CI uses keys created by ssssam[m] afaik | 10:14 |
| tlater | Do you have a docker hub account? I can add you to the group :) | 10:15 |
| cs_shadow | tlater: okay. I probably don't have an account yet but I'll get back to you once i manage to create one | 10:16 |
| jjardon | tlater: can you add me to the group as well, please? | 10:21 |
| tlater | jjardon: Sure, I'll need your docker id, I think | 10:22 |
| jjardon | tlater: jjardon | 10:22 |
| tlater | Ah, big surprise ;p | 10:22 |
| tlater | jjardon: I think you should be an owner now | 10:23 |
| jjardon | thanks! | 10:25 |
| jjardon | BTW, in case you didnt notice, we are generating aarch64 docker images of all the docker images but image-tools, as its x86-specific | 10:26 |
| tlater | Ah, nice | 10:26 |
| * tlater should keep better track of the docker images repo | 10:27 | |
| tlater | Hm, I suppose we should make bst-here check what arch we're on then | 10:27 |
| gitlab-br-bot | buildstream: merge request (ps-add-example-formats->master: Add example formats) #424 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/424 | 10:30 |
| gitlab-br-bot | buildstream: merge request (ps-add-example-formats->master: Add example formats) #424 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/424 | 10:32 |
| gitlab-br-bot | buildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 10:36 |
| *** aday has quit IRC | 10:42 | |
| jjardon | tlater: seems the CI user is still not able to push: can you check if you have to activate some permission or something? https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/63848777 | 10:43 |
| cs_shadow | tlater: checking arch in bst-here sound like a good idea. | 10:43 |
| cs_shadow | As an aside, we might also need an option to specify a different base image | 10:44 |
| tlater | cs_shadow: I have a sort-of fork in a snippet in the repo that allows that | 10:44 |
| cs_shadow | tlater: oh cool! I think i might have a half-baked branch somewhere too :) | 10:45 |
| tlater | I don't think it's very useful to end-users to allow using different images, so I kind of left it as something I use for testing. | 10:45 |
| *** aday has joined #buildstream | 10:45 | |
| tlater | Perhaps we should combine efforts actually, get this stuff into the main repo :) | 10:45 |
| paulsherwood | ok to merge https://gitlab.com/BuildStream/buildstream/merge_requests/424/pipelines ? | 10:46 |
| paulsherwood | (please squash :-) | 10:46 |
| cs_shadow | tlater: definitely. I think that option will be useful for people like us behind corporate firewalls who don't have direct access to docker hub | 10:46 |
| laurence | tlater, did we say that the work on the scheduler is worthy of it's own issue on gitlab? | 10:47 |
| laurence | In fact there may be one already...lemme have a look | 10:47 |
| cs_shadow | tlater: I have to grab some food now but I'll catchup with you about bst-here improvements | 10:47 |
| tlater | laurence: No issue yet, might be | 10:47 |
| cs_shadow | soon (TM) | 10:47 |
| tlater | Heh, ciao cs_shadow :) | 10:47 |
| jjardon | paulsherwood: what "chroot-able SDKs" actually means? not sure there is a specific plugin for that? | 10:47 |
| tlater | laurence: I kept track of what I'm doing in !347 for now | 10:48 |
| paulsherwood | jjardon: what tristan said :) | 10:48 |
| paulsherwood | the text says e.g., which means for example. bst can be used for that, and is being iiuc? | 10:48 |
| jjardon | paulsherwood: not sure if we are talking about the same, but chroots tarballs can not be created yet: https://gitlab.com/BuildStream/buildstream/issues/348 seems https://gitlab.com/BuildStream/buildstream/issues/263 still needs to be implemented | 10:53 |
| laurence | tlater, ah, there's some notes on it here which may also overlap too - https://gitlab.com/BuildStream/buildstream/issues/188 | 10:53 |
| tlater | Oh, I'd forgotten about those comments | 10:54 |
| tlater | Yeah, perhaps I should open a new issue then | 10:54 |
| tlater | Maybe do this work on a fresh branch, too | 10:54 |
| laurence | ok, either way, let's link them, for provenance | 10:54 |
| * tlater will start after figuring out what's wrong with jjardon 's pipeline | 10:54 | |
| laurence | cool :) | 10:54 |
| tristan | jjardon, well, theres no reason you cannot chroot into a `bst checkout` directory of something you've created | 11:00 |
| tristan | jjardon, the only advantage you really get from getting a tar stream, is file ownership = root, and more goodies only once we've sorted out the famous issue #38 | 11:02 |
| jjardon | sure, I was only a bit confused about the "chroot-able SDKs" term; maybe "chroot-able linux system trees" is better? | 11:03 |
| tristan | jennis, I dont understand what you mean by "I think this patch should also go into a separate MR (e.g. Further "Commands" documentation) and we'll keep this one solely for Getting started." | 11:04 |
| tlater | jjardon: Yep, was a permission thing, I restarted the CI for you: https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/63867246 | 11:05 |
| tristan | jjardon, linux system trees is a very weird wording I've never heard of before | 11:05 |
| tristan | sysroots ? | 11:05 |
| tristan | runtimes ? | 11:05 |
| jjardon | sysroots! yeah | 11:05 |
| tristan | both are a bit more recognizable | 11:05 |
| tristan | all ambiguous, but recognizable | 11:06 |
| tristan | runtimes is a bit more specific as it probably doesnt contain a kernel or services | 11:06 |
| jjardon | tlater: thanks! | 11:06 |
| tristan | although both are fairly possible | 11:06 |
| tristan | (you could chroot into a runtime with a kernel in it, the kernel would just not be useful inside your chroot) | 11:07 |
| tristan | jennis, I'm asking you because I'm worried about going off the rails here, and I dont want things documented redundantly (that's sort of the point of the exercise of sorting out a good layout) | 11:08 |
| tristan | jennis, there is one section where we can talk specifically about commands, one by one, and that can live in the same place as the generated documentation, as demonstrated in the patch I linked | 11:09 |
| jennis | So, tristan, the patch which you mentioned allows us to add documentation under every single command here. The MR we've been discussing this on is for a Getting started section | 11:10 |
| gitlab-br-bot | buildstream: merge request (ps-add-example-formats->master: Add example formats) #424 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/424 | 11:11 |
| jennis | And yes, I am totally on board with ALL command related documentation being in the one section | 11:12 |
| tristan | jennis, Ok I think I understand... "this patch" is totally ambiguous without that clarification | 11:13 |
| jennis | tristan, all I was suggesting was that we do this section by section so the MRs are much more manageable | 11:13 |
| tristan | jennis, the general gist, is that "Getting Started" is a series of simple exercises which after completed, the user has learned the things we want to teach them | 11:13 |
| tristan | and the things we want to teach them, are not "sections" | 11:13 |
| gitlab-br-bot | buildstream: merge request (valentindavid/workspacedir_config->master: Add 'workspacedir' configuration.) #420 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/420 | 11:14 |
| jennis | ok, thanks for making that clear | 11:14 |
| tristan | jennis, I would be hesitant in this case about "section by section", specifically for this patch | 11:14 |
| tristan | Rather, it would be better to make a very rough draft, and do some rounds of iteration on it a few times before merging it all at once | 11:15 |
| jennis | So you would rather merge it all at once? | 11:15 |
| jennis | To be clear ^ | 11:16 |
| tristan | I would rather see a draft of a dummy section, which has consistent structure for an exercise; along with a vision of what the exercises will be, and how they will teach the user the things we want them to learn | 11:17 |
| tristan | before seeing text, actually | 11:17 |
| tristan | I.e. lets look at another good example and copy some things... What is the structure... starting with "In this exercise we're going to bla bla bla" | 11:18 |
| tristan | How do we format notes, where do they go in relation to what they are noting ? | 11:18 |
| tristan | Conclusions of each exercise, are they important ? | 11:19 |
| tristan | Summary of what we've learned ? | 11:19 |
| tristan | jennis, as a rough idea, I think we can all we want to teach done in 3 or 4 exercises | 11:19 |
| tristan | jennis, and maybe we can aim for just 2 as a barrier to landing | 11:20 |
| tristan | then observe | 11:20 |
| jennis | tristan, ok I agree, we need this to be manageable | 11:21 |
| tristan | The two which I have good ideas how they can be very instructive, are starting with paulsherwood's example of A.) Creating a project, B.) Introduction of the manual element (be sure to have the user navigate over to the manual element reference docs, so that they can *see* the command lists, make sure the manual element declares empty command lists just to make sure it's clear)... C.) Perform a build... D.) Check out the result | 11:21 |
| tristan | The second one would be to move on to using a runtime, explaining how variables work, etc | 11:22 |
| jennis | I was just about to suggest something very similar :) | 11:22 |
| tristan | looking at the autotools element, and composing an autotools element by setting some variables, let the user run the build... afterwards point out what has happened | 11:22 |
| tristan | etc | 11:22 |
| gitlab-br-bot | buildstream: merge request (ps-add-example-formats->master: Add example formats) #424 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/424 | 11:23 |
| tristan | jennis, this has evolved a lot since I used close to 10 years ago, but may be a source of inspiration of how to structure things: https://developer.apple.com/library/content/referencelibrary/GettingStarted/DevelopiOSAppsSwift/BuildABasicUI.html#//apple_ref/doc/uid/TP40015214-CH5-SW1 | 11:24 |
| tristan | I think that's become a lot more complicated since the original cocoa | 11:25 |
| gitlab-br-bot | buildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 11:25 |
| jennis | tristan, ok, this seems like a good idea to me. Aha, that link is great, thanks | 11:25 |
| Nexus | tristan: do you have time to look at my MR for "opening a workspace with a cached build tree"? I believe you were uncertain about what to do with it | 11:26 |
| Nexus | https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 11:26 |
| tristan | jennis, one thing I'm curious about technically, is if we can A.) Run `bst something` with forced colors dumped into a file... B.) Collect that output with the ANSI color codes and use some kind of easy way to render that output in the reStructuredText | 11:27 |
| jennis | mhmm yes, I currently have no idea, but will have a look around | 11:27 |
| tristan | Nexus, that is titled "WIP: Issue 21 caching build trees", does it have anything to do with workspaces ? | 11:28 |
| jennis | because I guess the alternative is screenshots, but that's ugly | 11:28 |
| Nexus | tristan: yes it's the combination of #21 and #311 | 11:29 |
| Nexus | the title and comment are just something i quickly typed up at one point because i needed an MR | 11:29 |
| Nexus | i'll reword/name it | 11:29 |
| tristan | jennis, that would suck whenever it comes time to update the document yeah, it would also be good to have a way that the paths printed and the username on the shell prompt is always the same | 11:29 |
| tristan | Nexus, ok I dont really remember off hand, and I dont have hours to throw at it right now... ask me a question :) | 11:30 |
| tristan | And refresh my memory :) | 11:30 |
| tristan | I recall we talked about some things which we'd have to sort out after... and we talked about the minimal first implementation, and what that would have to do | 11:30 |
| jennis | mhmm tristan, perhaps I'll just tweak docker container and create an examples user for that | 11:31 |
| tristan | And we were uncertain if we could merge that when it was complete or if we would have to wait for other stuff | 11:31 |
| Nexus | tristan: so that implementation has been written, it now works and allows you to open a workspace from a cached build tree. However nothing specific was done regarding the "special sauce" as that was the thing you said to leave out for now | 11:31 |
| Nexus | tristan: so basically, i just need to know if you want it merged as is, with a follow up MR for special sauce stuff later(if at all) or if you want it put on hold for now | 11:32 |
| *** dominic has quit IRC | 11:33 | |
| tristan | Ah the "special sauce" has to do with how the structure of Source.init_workspace() goes ahead and replaces Source.stage(), instead of augmenting it | 11:34 |
| tristan | juergbi, give me a hand here, we discussed this in detail last time, and there was also some questions of optimizations in the air | 11:35 |
| tristan | I think something about staging without the .git/ subdir in regular circumstances, but not for workspaces ? | 11:35 |
| juergbi | yes, we concluded that we should normally avoid staging the .git subdir and similar | 11:36 |
| juergbi | however, it is required for some projects that need `git describe` working when not building from a dist tarball | 11:37 |
| juergbi | it may be possible to specifically handle the `git describe` case without having to stage the .git directory, but we haven't really looked into this yet, afaik | 11:38 |
| juergbi | and yes, for workspaces we want to keep .git | 11:39 |
| juergbi | not sure how to handle the cached build dir case for workspaces. probably a lot less important in practice | 11:40 |
| juergbi | I mean, caching the build dir of a workspace build | 11:40 |
| juergbi | we want to create a workspace from a cached build dir (#311), which may require a combination of cloning the .git and copying the cached build dir into it | 11:41 |
| juergbi | I don't think we've discussed that aspect in detail either | 11:41 |
| tristan | juergbi, Ok so... to the point at hand | 11:42 |
| tristan | We want to have workspace open work with cached build trees by default if one exists for the current cache key | 11:43 |
| tristan | juergbi, And `init_workspace()` is being a royal pain in the ass about it | 11:43 |
| tristan | Now, if we were to later optimize staging to omit the .git directory, such that we could avoid caching it, we still need to reproduce it for a workspace | 11:45 |
| tristan | Which, also sucks | 11:45 |
| tristan | One temptation is to say: Lets deprecate init_workspace() and force Sources to do the init_workspace() thing unconditionally | 11:45 |
| tristan | But that ignores that we might end up caching build trees without the obnoxiously large .git/ subdir | 11:46 |
| tristan | So this is a sort of hot potato | 11:46 |
| tristan | juergbi, if we were to deprecate init_workspace() now, and have workspaces *expect* them to be there until we do further optimizations, do you think we could live with that ? | 11:47 |
| tristan | juergbi, one idea I have is... let's say you checkout a cached build tree for a workspace... and then you stage the source into a temp directory... and then you link the files in from the temp directory, only creating files which dont yet exist in the workspace | 11:48 |
| tristan | How about that ? | 11:48 |
| juergbi | also have to keep in mind that the plan is to store sources in CAS | 11:49 |
| juergbi | but we don't want to store the .git directory in CAS either | 11:50 |
| gitlab-br-bot | buildstream: issue #372 ("Allow queues to run auxilliary jobs after an element's job finishes") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/372 | 11:50 |
| juergbi | if we assume that we only ever create workspaces from cached build dirs with the exact same source version, linking in missing files should work (well, some projects will probably modify VCS-controlled files :/) | 11:51 |
| juergbi | that's under the assumption that the cached build dir doesn't have the .git directory. otherwise it's pretty odd | 11:52 |
| juergbi | I have too many other things on my mind right now, not sure I can come up with a clear plan at this point | 11:53 |
| Nexus | tristan: is this mergeable without the "special sauce" stuff for now? | 11:55 |
| tristan | Nexus, be patient. | 11:58 |
| Nexus | kk | 11:58 |
| tristan | juergbi, A.) I dont think we *store* sources in CAS, we do so temporarily for the sake of staging, as an intermediate step... B.) Of course we dont make that assumption, we need to open workspaces for which there was never a build | 12:00 |
| juergbi | A) it's only a cache, of course, yes | 12:01 |
| tristan | juergbi, ok; if we can "fill in the blanks" for a matching version of checked out sources, this is going to work regardless of where the cached build tree is coming from | 12:02 |
| juergbi | B) the assumption I mentioned above is that cached build dirs don't have the .git directory, not that we necessarily have a cached build dir | 12:02 |
| tristan | juergbi, remember that this is a *later* step, it means that; when we get around to omitting the .git dir... we would have to do this blank filling in | 12:02 |
| tristan | But... holy shit webkit | 12:02 |
| tristan | Sigh | 12:02 |
| juergbi | yes, I think we need to handle it before we mandate anything going through CAS or similar | 12:03 |
| tristan | juergbi, https://gitlab.com/BuildStream/buildstream/issues/294#note_68634241 | 12:05 |
| tristan | Ummm | 12:05 |
| tristan | So, now that we know that build directories are about 17GB, we want to cache them in artifacts :-S | 12:05 |
| juergbi | I definitely don't want to store any .git directories in artifacts | 12:06 |
| tristan | Right | 12:06 |
| tristan | Ok so we can't land this under any circumstance without fixing stage | 12:07 |
| jmac | I'm guessing we'd need some extra flag in .bst files to say when a build requires the .git directory | 12:07 |
| tristan | I feel rather hostile towards the 2 or 3 modules in the whole world who have the nerve to *require* a .git/ directory | 12:08 |
| juergbi | *cough* buildstream itself *cough* | 12:08 |
| tristan | Rather I suspect that should be workaroundable | 12:08 |
| tristan | juergbi, we work from an install dont we ? | 12:08 |
| jmac | I don't think we should make any assumptions about the way projects lay out source; they may not use git and put all their source files in .git/, if they want | 12:08 |
| tristan | we do generate tarballs ? | 12:08 |
| juergbi | we use the git describe version number trick | 12:08 |
| juergbi | jmac: yes but source plugins could provide extra information | 12:09 |
| jmac | True | 12:09 |
| tristan | jmac, I agree, but I dont think the VCS counts in how projects lay out source, they are rather just the container | 12:09 |
| jmac | Yes, that's what I'm saying | 12:10 |
| jmac | You can have a .git directory which is not part of VCS | 12:10 |
| tristan | Oh yeah, and of course any of these techniques is specific to the Source implementation | 12:10 |
| tristan | so if we're staging without it, it's because GitSource.stage() decided to | 12:10 |
| jmac | In which case we'll know it's git so only that plugin strips .git; quite right | 12:11 |
| tristan | Well, I'm not sure we can have a solution to this immediately (already 9pm ?!) :-S | 12:13 |
| juergbi | I agree, this needs further thoughts | 12:14 |
| tristan | Nexus, your answer for today is "No, this is not ready.", it's not related to your patch, it's just a very hard problem | 12:15 |
| jmac | It's clear workspaces affect this so I need to go and read up on what they actually do | 12:15 |
| tristan | I'm going to comment on the issue (not the MR) | 12:15 |
| tristan | caching of workspace builds can be a double whammy too | 12:16 |
| tristan | but I think we need a design which prevents that | 12:16 |
| tristan | WebKit's git repo is relatively very small, only around 6GB, it's just that people who work on WebKit, typically have an additional 20GB of stuff in their workspace | 12:17 |
| juergbi | I think we really want the Source-specific optimizations for workspaces in the end | 12:29 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues/311#note_69267797 | 12:32 |
| tristan | That's my summary of the day | 12:32 |
| * Nexus looks at the 500 word "comment" with something resembling horror, then resigns himself to start reading | 12:40 | |
| *** aday has quit IRC | 12:56 | |
| *** aday has joined #buildstream | 12:56 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 13:03 |
| *** Prince781 has joined #buildstream | 13:10 | |
| * tlater really appreciates pylint's suggested method names | 13:10 | |
| tristan | tlater, I see you opened #372... and I wonder do you already have a picture of what local LRU expiry is going to look like at a high level ? | 13:12 |
| tlater | Yep, I do, I thought it wasn't relevant to #372 though | 13:13 |
| tlater | I can add my notes for that there, if you like | 13:14 |
| tristan | Is there an issue open for LRU expiry ? | 13:14 |
| tlater | #135 should be that issue | 13:14 |
| tlater | There is an MR for it too in !347 | 13:14 |
| tristan | would be nice to see the plan in there | 13:15 |
| tristan | right, MRs are probably at least a bit premature | 13:15 |
| tlater | Well, I'd hoped that MR wasn't premature, but then concurrency hit :) | 13:16 |
| tristan | (this one at least) | 13:16 |
| tristan | yeah, understood :) | 13:16 |
| * tlater will try to adopt the plan from that MR a bit and add it to #135, pretty clear on what I want to do once I have the extra jobs working | 13:16 | |
| * paulsherwood wonders how folks would feel about the https://www.coreinfrastructure.org/programs/badge-program for buildstream | 13:17 | |
| *** xjuan has joined #buildstream | 13:21 | |
| tristan | paulsherwood, a quick look at that page makes it seem github related | 13:42 |
| tristan | but maybe it was a joke :) | 13:43 |
| cs_shadow | quick clarification - When writing a source plugin, am I allowed to make the `ref` field a dict instead of a string? | 13:47 |
| tristan | cs_shadow, yes | 13:47 |
| cs_shadow | tristan: thanks :) | 13:47 |
| tristan | cs_shadow, if you change it later, you must support earlier versions | 13:47 |
| cs_shadow | sure, that makes sense | 13:48 |
| tristan | and set BST_FORMAT_VERSION on your Source plugin class (if that is an external plugin to BuildStream) | 13:48 |
| tristan | if you want people to be able to assert the format from their project.conf | 13:48 |
| cs_shadow | good idea, it is indeed an external plugin | 13:48 |
| paulsherwood | tristan: no, it's not github related, nor was i joking | 13:49 |
| tristan | right, you only need to set BST_FORMAT_VERSION if the format your plugin accepts is ever enhanced | 13:49 |
| cs_shadow | there's a very good chance that it might change, especially in the early phases | 13:50 |
| tristan | paulsherwood, ah ok - it looks a bit like begging for praise though doesnt it ? | 13:50 |
| paulsherwood | i understand your point. however when i attempted the checklist for other projects i found it mostly worthwhile to think about | 13:50 |
| paulsherwood | i'm ok not to do it, of course | 13:51 |
| tristan | its all a matter of perception, either way | 13:51 |
| tristan | getting an external approval will be a positive for some and a negative for others, I suppose | 13:52 |
| paulsherwood | yup | 13:52 |
| tristan | on a personal level, I'm not the kind of person to chase a certificate, or to award certificate distributors with the gratification of feeling like their certificates are meaningful | 13:53 |
| tristan | but it's a very personal standpoint :) | 13:53 |
| tristan | I wouldnt be against it at a project level if people really want that, is what I'm trying to say | 13:54 |
| * tristan has to skip out on closing shop again | 13:54 | |
| jonathanmaw | hrm, is it correct that element passed into Pipeline.initialized() using the except_ arg will be loaded into the pipeline? | 13:57 |
| jonathanmaw | I was planning to use except_ for elements that don't have element files (but we still know about them because they have an open workspace) | 13:57 |
| *** tristan has quit IRC | 13:57 | |
| jonathanmaw | but that doesn't seem to work, because we explicitly load them along with the other elements https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_pipeline.py#L123 | 13:58 |
| tlater | jonathanmaw: They have to be, unfortunately, although I don't remember the detail of why | 13:58 |
| jonathanmaw | the description on https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_frontend/app.py#L236 might be explaining why, but I don't understand it | 13:58 |
| jonathanmaw | I was expecting except_ to be treated different under different circumstances according to that text, but that doesn't seem to be the case. | 13:59 |
| tlater | jonathanmaw: It's treated differently depending on the command | 13:59 |
| tlater | Though they will still be loaded before we start processing the pipeline in either case | 14:00 |
| tlater | It doesn't explain why we need to load them, unfortunately :| | 14:00 |
| tlater | Maybe the loader explains it? | 14:00 |
| tlater | Hm, no | 14:02 |
| tlater | jonathanmaw: I think we need to know what the dependencies of an excepted element are, which we can only know if we've loaded it - but I don't quite recall *why* | 14:03 |
| *** Prince781 has quit IRC | 14:50 | |
| *** tristan has joined #buildstream | 14:55 | |
| tristan | oh jaysus, except_ | 15:00 |
| tristan | ok, that is the part which is in most need of a refactor, we need to tear apart the pipeline and make it _only_ do this targets/except/loading stuff | 15:00 |
| tristan | jonathanmaw, the most confusing part of except_, is that that it is treated differently for a command | 15:01 |
| tristan | so, the way the code is setup, you cannot say... "Lets start with a list of toplevel resolved elements, and then process them with except, and then decide what to do with them" | 15:01 |
| tristan | That is the horrible | 15:01 |
| tristan | jonathanmaw, the second most confusing part of except, is the part you dont understand, which is most probably because it's not documented well | 15:02 |
| tristan | What does --except mean... well, it's simple enough if you get down to what `bst fetch/track/show --except` *should* do | 15:02 |
| tristan | jonathanmaw, what except does, is, it takes some arguments (toplevels), and subtracts it from another build graph (other toplevels) | 15:03 |
| *** noisecell has quit IRC | 15:04 | |
| tristan | jonathanmaw, now, it's possible that one or more of the except elements, are not in the build graph you want to subtract it from, but remember that excepting is a recursive activity | 15:04 |
| tristan | so, I can take two build graphs, with two different toplevels, and except one from the other... the other will be chopped off *where the two graphs intersect* | 15:05 |
| *** bethw has quit IRC | 15:05 | |
| tristan | jonathanmaw, that first part, is because you want to say... bst track <something-from-gnome-world> --except core-deps.bst... where core-deps.bst is an unrelated stack element (because things in world depend directly into it, instead of generalizing) | 15:06 |
| tristan | jonathanmaw, it gets a *bit* more complicated, because build graphs have multiple toplevels | 15:06 |
| tristan | so, when you subtract one element recursively from the other build graph... you stop subtracting when you hit another explicit toplevel | 15:07 |
| tristan | this is mostly tlater's domain really, he should be able to explain it ;-P | 15:08 |
| * tristan thinks this part actually works beautifully, albeit the surrounding usage is a complete mess, one of the last messes which desperately need a cleanup | 15:09 | |
| tlater | tristan: I forgot that we have multiple toplevels and was scratching my head over a situation in which we wouldn't just rip out the element from the tree | 15:10 |
| tristan | right, in fact `bst track --except` was not going to be useful without considering the intersection | 15:10 |
| tristan | because of the way stack elements are employed there | 15:11 |
| tristan | and, they turn out to be really useful that way | 15:11 |
| * tlater still likes how the intersection algorithm works :) | 15:11 | |
| tristan | tlater, we had a lot of interesting conversations about how that would work, yes | 15:11 |
| tristan | I would love for you to be the one who rips _pipeline.py apart and makes it "just be able to load pipelines, and manipulate lists of elements, such that they can be planned and intersect-subtracted and such" | 15:12 |
| * tlater would like to, but is tied up in other stuff atm :| | 15:13 | |
| tristan | and then contributors wouldnt need to run away arms flailing when trying to navigate it | 15:13 |
| tristan | to be honest, it's the kind of thing that makes you think "Wait... is it really right to be adding more features here without fixing this ?" | 15:13 |
| *** noisecell has joined #buildstream | 15:14 | |
| * tristan back from the harvest at the fried chicken farm... | 15:44 | |
| tristan | tlater, one part I forgot to mention in all of this interesting intersection logic, is the diamond case; which is also beautiful | 15:44 |
| tristan | this really merits a diagram somewhere in the docs | 15:44 |
| tristan | When you except an element, the excepted element is excepted unconditionally; the part of the build graph which it removes however, is not removed when orthogonal elements outside of the except intersection independently depend on dependencies which would otherwise be subtracted | 15:45 |
| tristan | jonathanmaw, ^^^ there is the complete story :) | 15:46 |
| * tristan wish he could document this with a diagram, and fix the pipeline, I will try to find the time | 15:47 | |
| *** toscalix has quit IRC | 15:47 | |
| *** Prince781 has joined #buildstream | 15:48 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 16:02 |
| *** noisecell has quit IRC | 16:36 | |
| jmac | juergbi: One more question about remote execution if you're still around - you said "The build of a single element may involve execution of multiple actions" - is that Actions as in the Bazel Remote execution API? and if so, couldn't we do all our builds in one action object? | 16:52 |
| juergbi | jmac: Yes, Action as per API | 16:53 |
| juergbi | For many cases we could do everything in a single Action | 16:53 |
| juergbi | However, not reasonably well with the current plugin API | 16:54 |
| jmac | I can't think what the exception would be | 16:54 |
| juergbi | And some plugins do additional stuff between commands | 16:54 |
| juergbi | Also, integration is run before plugin action with a slightly different sandbox: read-write rootfs (and no sources) | 16:55 |
| jmac | Oh, so the plugin needs to do some logic which is outside the Action | 16:55 |
| juergbi | Not BuildElements but some others, yes | 16:55 |
| jmac | Yes, OK, that makes sense. | 16:55 |
| juergbi | E.g. use the (virtual) filesystem API to move stuff around etc. | 16:55 |
| jmac | Thanks juergbi. | 16:56 |
| juergbi | See also 'command batching' at the end of the proposal mail | 16:56 |
| juergbi | I expect us to add this but it's an optimization | 16:57 |
| jmac | Yes, you do explain it there, apologies! | 16:57 |
| *** jonathanmaw has quit IRC | 17:00 | |
| *** slaf has quit IRC | 17:10 | |
| *** slaf has joined #buildstream | 17:21 | |
| *** solid_black has joined #buildstream | 18:15 | |
| *** Prince781 has quit IRC | 19:09 | |
| *** xjuan has quit IRC | 19:13 | |
| *** tristan has quit IRC | 19:35 | |
| *** solid_black has quit IRC | 20:08 | |
| *** aday has quit IRC | 21:45 | |
| gitlab-br-bot | buildstream: merge request (jjardon/debian-9->master: .gitlab-ci.yml: Run test in current Debian stable (stretch)) #425 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/425 | 23:29 |
| gitlab-br-bot | buildstream: merge request (jjardon/debian-9->master: .gitlab-ci.yml: Run test in current Debian stable (stretch)) #425 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/425 | 23:30 |
| jjardon | paulsherwood: even if we do not get the badge; I think is worth check we meet the requisites, all the criteria points seems reasonable: https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/criteria.md | 23:37 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!