| albfan[m] | instructions for archlinux: I need to install lzip too. | 00:05 |
|---|---|---|
| albfan[m] | and lauch bst track base/freedesktop-sdk-junction.bst | 00:06 |
| albfan[m] | before bst build --track-all core-deps/gtk+-3.bst | 00:06 |
| gitlab-br-bot | buildstream: merge request (jjardon/codequality->master: WIP: Add job to analyze project code quality with Code Climate CLI) #431 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/431 | 01:25 |
| gitlab-br-bot | buildstream: merge request (jjardon/codequality->master: Add job to analyze project code quality with Code Climate CLI) #431 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/431 | 01:34 |
| gitlab-br-bot | buildstream: merge request (jjardon/install_deps->master: Update install instructions) #406 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/406 | 01:44 |
| gitlab-br-bot | buildstream: merge request (jjardon/install_deps->master: Update install instructions) #406 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/406 | 01:45 |
| *** tristan has joined #buildstream | 06:11 | |
| albfan[m] | 18,1 Gb in ~/.cache/buildstream, how much it takes to build gtk3? | 06:41 |
| tristan | albfan[m], not sure exactly, but certainly much less... there are a few factors here; and they are mostly related to the work jennis and tlater are doing right now... | 06:54 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues/136 <-- expiring artifacts in remote artifact cache | 06:55 |
| tristan | Currently we've not enabled the shared artifact caches because they get full quickly and need manual maintenance | 06:55 |
| tristan | This means, you are not benefiting from downloading the already built artifacts yet :'( | 06:56 |
| tristan | That also means that, ... if an artifact is already built... you would not need to download the source code needed to build it | 06:56 |
| tristan | for the gnome-build-meta project, this would mean you would never need to download any source code to build the base freedesktop-sdk runtime | 06:57 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues/135 <-- expire artifacts in local cache | 06:57 |
| tristan | This will basically mean that for at least the artifact cache (~/.cache/buildstream/artifacts), we would automatically delete the least recently used build results | 06:58 |
| tristan | it's possible that it's amounting to that much right now for GTK+, but it really should be much less | 06:58 |
| *** tristan has quit IRC | 07:12 | |
| *** toscalix has joined #buildstream | 07:51 | |
| albfan[m] | tristan: I can inspect what is taking that space later (afk now) | 07:56 |
| jennis | albfan[m], have you just done a full build of GNOME from scratch? iirc, the gnome newcomers guide recommends having ~20GB of free space | 08:09 |
| albfan[m] | jennis: yes, from scratch. Is there other way? | 08:10 |
| jennis | I don't think so, jjardon may know more regarding this | 08:12 |
| *** jonathanmaw has joined #buildstream | 08:15 | |
| tlater | Hm, if the cache is filled only with one build the cache expiry wouldn't help | 08:24 |
| *** noisecell has joined #buildstream | 08:36 | |
| jennis | juergbi, we determine the size of an object in OSTreePusher here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_artifactcache/pushreceive.py#L202 | 08:48 |
| jennis | in send_putobjects* | 08:49 |
| jennis | this size, "tar_info.size" can then be re-accessed in receive_putobjects(), right...? | 08:49 |
| juergbi | jennis: yes, this should be in the tar header of the embedded file and should be accessible there even before actually extracting the file | 08:51 |
| jennis | fantastic | 08:52 |
| jmac | jjardon: Yeah, that's probably the best idea | 09:01 |
| *** bethw has joined #buildstream | 09:02 | |
| *** aday has joined #buildstream | 09:25 | |
| *** toscalix has quit IRC | 09:43 | |
| *** toscalix has joined #buildstream | 09:45 | |
| gitlab-br-bot | buildstream: issue #378 ("Make `bst-here` more flexible") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/378 | 10:03 |
| *** tristan has joined #buildstream | 10:17 | |
| tlater | juergbi: Do we need to worry about forged headers at all? | 11:57 |
| gitlab-br-bot | buildstream: merge request (jjardon/another_codequality->jjardon/codequality: WIP: test codequality) #432 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/432 | 11:58 |
| juergbi | tlater: it seems we don't have any validation at all right now, afaict | 12:05 |
| juergbi | push is a privileged operation, so it's not quite as critical | 12:05 |
| juergbi | I don't think we should worry about it as part of this work as it wouldn't be a regression | 12:06 |
| juergbi | the CAS server performs validation, though | 12:06 |
| tristan | jjardon, it looks like a cool feature, but I dont like how much of a black box it is | 12:26 |
| tristan | jjardon, the codeclimate thing I mean | 12:26 |
| tristan | jjardon, this says that radon is an available plugin, so maybe we're running it redundantly: https://docs.codeclimate.com/docs/list-of-engines | 12:26 |
| tristan | jjardon, also maybe it makes sense to install codeclimate ourselves, generate more reports with it in a single task, not use the docker image, and actually understand what is going on in the delivery process of this json file that gitlab likes to read | 12:27 |
| * tristan should probably comment on the issue | 12:28 | |
| *** xjuan has joined #buildstream | 12:53 | |
| gitlab-br-bot | buildstream: merge request (jjardon/install_deps->master: Update install instructions) #406 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/406 | 12:55 |
| *** Prince781 has joined #buildstream | 13:05 | |
| *** tristan has quit IRC | 13:34 | |
| *** tristan has joined #buildstream | 13:37 | |
| *** Prince781 has quit IRC | 14:01 | |
| jennis | Hi guys, I've just completely cleaned both my remote and local caches and have rebuilt a simple bst project. `du -sh` in ~/.cache/buildstream/artifacts/ostree turns out to be a lot larger than `du -sh` in /path/to/remote/artifacts, both are identical ostree repos, executing `ostree refs` in each repo shows them to be the same | 14:30 |
| jennis | any ideas? | 14:30 |
| *** Prince781 has joined #buildstream | 14:30 | |
| jennis | (Seems like this shouldn't be the case) | 14:31 |
| tlater | jennis: By 'cleaning' do you mean `rm -rf <repopath>`? | 14:31 |
| jennis | yeah | 14:32 |
| jmac | and `ostree refs` shows exactly the same refs for both? | 14:32 |
| jennis | yes | 14:32 |
| jmac | Hmm! | 14:33 |
| tlater | Good question ;p | 14:34 |
| jennis | https://paste.baserock.org/ovuseqereq | 14:34 |
| tristan | jennis, the remote cache uses compressed artifacts | 14:34 |
| tristan | jennis, the local cache does not | 14:34 |
| tlater | tristan: Oh, is there a reason for that? | 14:35 |
| tristan | hardlink checkout optimization | 14:35 |
| tristan | on the server it's unimportant | 14:35 |
| tlater | Right, that makes sense | 14:35 |
| jennis | tristan, aha! Thank you | 14:35 |
| tlater | Locally the user might get annoyed... | 14:35 |
| jmac | How does the compression work? I thought we sent uncompressed tars to the remote cache. | 14:36 |
| tristan | jmac, we locally create a compressed repo, and we send the compressed objects through a tar stream | 14:36 |
| tristan | there are no tarballs | 14:36 |
| jmac | Ah yes | 14:36 |
| tristan | TarStream is simply an optimization to avoid round tripping for sending a lot of files | 14:36 |
| jjardon | tristan: codeclimate is AGPLv3, impossible to be more open :) . But yeah we can use codeclimate directly instead the image created from gitlab but we will probably lost the nice MR integration | 14:37 |
| tlater | I.e., to enable people pushing objects from South Korea to the UK | 14:37 |
| tristan | (when pushing to an artifact server as far as say, manchester, tarstream turns a 1GB push from 13hours into about 20min upload) | 14:37 |
| tristan | jjardon, it's not really that important to spend time on, though | 14:39 |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 14:39 |
| tristan | I mean, it would be indeed cute to have things rendered in the gitlab UI | 14:39 |
| tristan | jjardon, maybe we can also easily just find the right code to convert radon json output into codeclimate json format; and avoid pulling in the whole machine | 14:40 |
| tristan | it's the "black box" nature of this third party machine which we've not configured ourselves that I dislike, mostly | 14:41 |
| * tristan wishes gitlab would be more open about letting users contribute to analytics | 14:41 | |
| tristan | seems they want to have built-in things for everything; while I just want to take the things I've created and plug them into gitlab's analytics in a configurable way | 14:42 |
| tristan | jjardon, ok I'll tell you what, lets do this.. | 14:48 |
| tristan | jjardon, lets add this in "post", and also move the radon job into "post", it doesnt make sense in tests, and it doesnt make sense to block tests on these extra jobs | 14:49 |
| tristan | jjardon, and lets add a comment in the codeclimate job YAML that this needs refactoring, and that we probably want to just provide the codeclimate.json directly as an output of radon, with some conversion | 14:50 |
| tristan | it will still be a black box, but it might add some fun to CI pipelines, and it will have a documented path for improving things later | 14:50 |
| tristan | jjardon, sound good ? | 14:50 |
| *** toscalix has quit IRC | 14:54 | |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 15:01 |
| gitlab-br-bot | buildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/317 | 15:05 |
| tlater | Hm, it looks like my issue was that `Element` objects can't be pickled | 15:14 |
| tlater | Is there any way to share elements across the process boundary? | 15:14 |
| tlater | Hm, no, I suppose the element is just passed along on the parent thread. | 15:18 |
| tristan | tlater, we are *very* careful about what we serialize through the IPC, simple objects _only_ | 15:28 |
| tristan | regardless if pickling is performed on our cucumbers | 15:29 |
| tristan | tlater, the Element is indeed copied *into* the child by simple virtue of fork() | 15:29 |
| tristan | but anything passed over the IPC is hand crafted | 15:29 |
| tlater | Yep, I've figured that out by now | 15:30 |
| tlater | I assumed the element was passed back by the child, but that turned out not to be the case | 15:30 |
| tlater | Grr, only reference to `self.element` left is the one passed to `action_cb()` | 15:32 |
| tlater | Easily the hardest one... | 15:32 |
| tristan | benbrown, valentind; git source staging has to change because of https://gitlab.com/BuildStream/buildstream/issues/376 | 15:34 |
| tristan | benbrown, or rather, that is a side effect of https://gitlab.com/BuildStream/buildstream/issues/311 | 15:35 |
| tristan | So, when we stage for a build, we should exclude all history except for the minimum which might be required for a `git describe` | 15:35 |
| tristan | benbrown, for a workspace checkout however, it needs A.) The regular clone history ... B.) A git remote set-url origin to point to the real upstream | 15:36 |
| tristan | B.) is done as special sauce in `git` plugin's implementation of Source.init_workspace() currently | 15:36 |
| benbrown | Right, makes sense | 15:39 |
| benbrown | WRT preserving all history of git mirrors, disabling auto gc ought to be enough (I'm sure skullman could correct me) | 15:41 |
| tristan | it certainly sounds that way, if things work in such a way that I sort of suspect they might | 15:42 |
| benbrown | But any unreachable objects won't be exposed by default, so if you cloned from the bare repository (which I believe is how bst currently operates) certain shas won't be present | 15:42 |
| tristan | benbrown, i.e. if you could mirror all git repositories into one repo (ignoring hash collision possibilities) | 15:42 |
| tristan | then it would work at least for any repo with rewritten history | 15:43 |
| tristan | I see | 15:43 |
| tristan | benbrown, well `git` plugin's Source.fetch() implementation *should* be taking care of this | 15:43 |
| tristan | we currently dont implement things exactly to spec, but we meet the requirements | 15:44 |
| tristan | i.e. Source.track() need not download or clone anything, it only needs to report the most recent commit sha for a tracking branch | 15:44 |
| tristan | and Source.fetch() need not clone the whole repo, it only needs to obtain the required ref | 15:44 |
| skullman | benbrown: you can configure a git server to let you ask it for unreachable objects, but yeah, you wouldn't get them if you did a mirror fetch, since that goes by reachable refs | 15:45 |
| benbrown | see uploadpack.allowAnySHA1InWant in the git config man page | 15:45 |
| benbrown | skullman: yeah | 15:45 |
| tristan | the implementation uses clone, which normally "works" | 15:45 |
| tristan | right, so we might want to ensure that a BuildStream mirror implementation ensures it's mirrored repos allow this | 15:46 |
| benbrown | Indeed | 15:46 |
| * tristan notices that he hasnt had time to look at all of the issue comments today | 15:46 | |
| tristan | sorry about that, I think there was a big one from skullman I still have to look at... | 15:46 |
| skullman | benbrown: IIRC there's also an option to allow SHA1s that are reachable, but it's disrecommended since it's expensive to calculate, hence the big fat warning on allowAnySHA1InWant that it allows fishing for alternate histories | 15:46 |
| skullman | tristan: it's not urgent, I've got to write a follow-up about how we could add metadata to cache servers | 15:47 |
| skullman | s/cache server/artifact caches/ | 15:47 |
| *** sstriker has joined #buildstream | 15:57 | |
| *** sstriker has quit IRC | 16:06 | |
| *** Prince781 has quit IRC | 16:28 | |
| juergbi | skullman: I haven't really looked into #76 yet but please keep non-OSTree artifact cache in mind (specifically, the CAS plans for remote execution) | 17:02 |
| *** jonathanmaw has quit IRC | 17:04 | |
| *** bethw has quit IRC | 17:31 | |
| gitlab-br-bot | buildstream: merge request (jjardon/debian-9->master: WIP: .gitlab-ci.yml: Run test in current Debian stable (stretch)) #425 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/425 | 17:55 |
| *** tristan has quit IRC | 17:58 | |
| gitlab-br-bot | buildstream: merge request (jjardon/debian-9->master: .gitlab-ci.yml: Run test in current Debian stable (stretch)) #425 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/425 | 18:02 |
| *** tristan has joined #buildstream | 18:15 | |
| * albfan[m] continue to build gtk3 on buildstream to hack GNOME! | 18:48 | |
| cs_shadow | Hi. one of our builds is failing with "OSError: [Errno 39] Directory not empty: <path-to-bst-cache>" and I'm also seeing "BUG" in the click output. Have I messed up something on my side or is it actually a bug? | 19:00 |
| cs_shadow | I don't have any workspaces open, in case it matters | 19:00 |
| *** xjuan has quit IRC | 19:35 | |
| *** tristan has quit IRC | 19:44 | |
| *** aday has quit IRC | 19:48 | |
| *** slaf has quit IRC | 22:27 | |
| *** slaf has joined #buildstream | 22:29 | |
| *** slaf has joined #buildstream | 22:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!