| *** tristan has quit IRC | 04:24 | |
| gitlab-br-bot | push on buildstream@juerg/recursive-pipelines (by Jürg Billeter): 17 commits (last: Only initialize remote artifact cache connections if needed) https://gitlab.com/BuildStream/buildstream/commit/f17ef1e47cfce604bcd93373eae9646a96e9122d | 08:52 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (juerg/recursive-pipelines->master: WIP: Junctions and subprojects) #160 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/160 | 08:53 |
| *** WSalmon has joined #buildstream | 09:18 | |
| *** bethw has joined #buildstream | 09:24 | |
| *** valentind has joined #buildstream | 09:33 | |
| *** jonathanmaw has joined #buildstream | 09:48 | |
| *** tpollard has joined #buildstream | 10:05 | |
| *** ssam2 has joined #buildstream | 10:16 | |
| ssam2 | the bst-external tests seem to have broken | 10:19 |
| ssam2 | looks perhaps like an api break inside buildstream: https://gitlab.com/BuildStream/bst-external/-/jobs/41823885 | 10:20 |
| *** valentind has quit IRC | 10:21 | |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 10:21 |
| *** jude has quit IRC | 10:32 | |
| *** jude has joined #buildstream | 10:48 | |
| gitlab-br-bot | push on buildstream@132-loading-external-plugins-works-without-explicit-requirement-in-project-conf (by Jonathan Maw): 11 commits (last: setup.py: Add pytest-xdist dependency) https://gitlab.com/BuildStream/buildstream/commit/841f4e9e47ed4d7d14d8526a657600a6a67d7312 | 10:59 |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 11:00 |
| *** semanticdesign has joined #buildstream | 11:12 | |
| *** bochecha has joined #buildstream | 11:16 | |
| ssam2 | gitlab CI logs render so much faster with Firefox 57 :-) | 11:26 |
| bochecha | everything is better with Firefox 57 :) | 11:29 |
| *** bochecha has quit IRC | 11:32 | |
| tpollard | oooh I need to try that | 11:58 |
| *** cs_shadow has joined #buildstream | 12:12 | |
| *** bochecha has joined #buildstream | 12:36 | |
| tlater | Gah, multiprocessing.Pool doesn't reap its processes when it's terminated T.T | 13:01 |
| tlater | *And* you can't actually access the processes it controls | 13:02 |
| * tlater wonders who thought this would be a good idea | 13:02 | |
| *** bochecha has quit IRC | 13:21 | |
| *** nexus_ has joined #buildstream | 13:55 | |
| *** mcatanzaro has joined #buildstream | 13:56 | |
| *** nexus_ has quit IRC | 13:56 | |
| *** tristan has joined #buildstream | 14:00 | |
| *** tristan has quit IRC | 14:04 | |
| *** semanticdesign has quit IRC | 14:12 | |
| *** semanticdesign has joined #buildstream | 14:13 | |
| gitlab-br-bot | buildstream: Jonathan Maw created branch 114-give-better-warnings-on-overlaps | 14:15 |
| gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 14:15 |
| *** tristan has joined #buildstream | 14:28 | |
| *** bochecha has joined #buildstream | 14:44 | |
| gitlab-br-bot | push on buildstream@event-hooks (by Tristan Maat): 4 commits (last: _hooks.py: Implement event hooks) https://gitlab.com/BuildStream/buildstream/commit/45027a041b63999f26d97439e35f02ba2216999d | 14:45 |
| tristan | tlater, added some comments to your branch now | 15:27 |
| tlater | tristan: ta :) | 15:27 |
| tristan | mostly nit picks, but one thing I'm more confused and curious about | 15:28 |
| tristan | I *think* I understand what happened in _pipeline.py | 15:28 |
| tristan | but it's getting hard to follow, so I dont really know | 15:28 |
| tlater | Alright, reading right now | 15:28 |
| tristan | the "I'm skeptical" comment yeah | 15:29 |
| tristan | If it's becoming spaghetti because Planner.plan() doing 2 quite orthogonal activities in the same function, as I suspect, then we should change that function into two functions and make that more clear | 15:30 |
| tlater | Ah, yes | 15:30 |
| tlater | I was thinking about that | 15:30 |
| tristan | in any case, as far as I can see there can only be "one plan" | 15:30 |
| tristan | and this makes it look like there are two, very confusing to the reader | 15:30 |
| tlater | It doesn't quite work that way here unfortunately, because the track plan is different from the build plan | 15:30 |
| tlater | Because we can track completely different things from the things we build | 15:31 |
| tlater | I did think that perhaps splitting plan into two methods would work | 15:31 |
| tristan | That just means that "the plan" encapsulates more elements in the case of building + tracking | 15:31 |
| tristan | doesnt it ? | 15:31 |
| tristan | Right, I think that is a very good idea to split that | 15:31 |
| tlater | tristan: Not entirely, because one of the plans is fed to the tracking queue and the other bit is fed to whatever comes after | 15:32 |
| tlater | So we really have two plans | 15:32 |
| tristan | Oh ? | 15:32 |
| tlater | That just come together at some point | 15:32 |
| tristan | Why would we have that ? | 15:32 |
| tlater | Because we don't always want to track everything we build | 15:32 |
| tlater | And also not build everything we track (though this is debatable) | 15:32 |
| tristan | Oh right and... ok I recall, but this worries me a lot | 15:32 |
| * tlater also didn't like how this looked too much, franly | 15:33 | |
| tristan | just like fran | 15:33 |
| tristan | tlater, ok so... the big issue which worries me about this | 15:33 |
| tristan | tlater, is that I dont feel reassured that what we feed directly to the build queue is not discovering a cached artifact for it's dependencies, and building against *that* | 15:34 |
| tristan | instead of being forced to wait for that cached dependency to become tracked first | 15:34 |
| tristan | tlater, that has to be 200% certain | 15:34 |
| * tristan has meeting... | 15:35 | |
| * tlater might be off a bit sooner than usual (not before 5/1.5 hours from now though) | 15:37 | |
| gitlab-br-bot | push on buildstream@event-hooks (by Tristan Maat): 4 commits (last: _hooks.py: Implement event hooks) https://gitlab.com/BuildStream/buildstream/commit/e6a3d3672c7ce71a54aadd2d191dd01734fc9e1c | 16:10 |
| gitlab-br-bot | push on buildstream@event-hooks (by Tristan Maat): 2 commits (last: Add hook cleaning step after buildstream finishes) https://gitlab.com/BuildStream/buildstream/commit/9c2e38597709beb9ce890436d5a251a93de93f60 | 16:10 |
| *** bethw has quit IRC | 16:20 | |
| *** ssam2 has quit IRC | 16:20 | |
| *** WSalmon has quit IRC | 16:20 | |
| *** bethw has joined #buildstream | 16:20 | |
| *** WSalmon has joined #buildstream | 16:20 | |
| *** ssam2 has joined #buildstream | 16:20 | |
| tristan | tlater, probably me as well... anyway I think for the most part that branch is ready; it would be nice to clean that part up a bit if possible | 16:24 |
| tristan | I expect we have some assurance that building will not skip over something that needs tracking first | 16:24 |
| * tlater will at least leave a giant comment explaining the situation | 16:24 | |
| tlater | And add a test for this... | 16:24 |
| tristan | i.e. that depends on something being tracked first | 16:24 |
| tristan | but maybe not | 16:24 |
| tlater | I'm not 200% sure either right now. | 16:24 |
| tristan | because the old way assumed that we ran track unconditionally on everything | 16:24 |
| tristan | yeah there is a good chance it's not covered | 16:25 |
| tristan | because I think in the old way... we can simply say "dont build this unless it's passed through the tracking queue" | 16:25 |
| tristan | and then we just know | 16:25 |
| tristan | this seems to be getting *again* into the issue that we need something more dedicated and reliable for managing pipeline element transient state | 16:26 |
| tristan | not just a scattered hand full of checks on private variables exposed by Element class | 16:26 |
| * tlater is pretty curious how you would solve that | 16:28 | |
| tristan | tlater, after this coming up in a few places, I am starting to think we need to delegate events to Elements in the main thread | 16:30 |
| tristan | I think we add a private ElementState() object to centralize logic | 16:30 |
| tristan | logic around reporting state, and handling of important events | 16:31 |
| tristan | So, as things happen in the pipeline, we have the scheduler delegate some very generic events to the element (which internally delegates that to ElementState()) | 16:31 |
| tristan | generic events such as, successful completion of a given task (like pulling/pushing/building/fetching/tracking, etc, success of any of these is an event) | 16:32 |
| tristan | When we ask an element for a cache key, it consults the element state | 16:32 |
| tristan | where everything involved in reporting all of this stuff resides, closely together | 16:33 |
| tlater | Right, that would make it less scattered | 16:33 |
| tristan | generally I'm thinking that's the right approach, plus of course; analyze all of the possibilities, and handle it there | 16:33 |
| *** tristan has quit IRC | 16:37 | |
| *** tristan has joined #buildstream | 16:38 | |
| tristan | oops | 16:38 |
| tristan | <tristan> generally I'm thinking that's the right approach, plus of course; analyze all of the possibilities, and handle it there | 16:38 |
| tristan | <tristan> also, this should totally completely kill the approach we currently have for "force recalculation" of things | 16:38 |
| tristan | <tristan> tlater, build & track heavily relies on the principal that the cache key is unresolved initially, and force recalculated after tracking completes | 16:38 |
| tristan | <tristan> while this holds up pretty well in practice, its certainly delicate and not very future proof | 16:38 |
| * tlater finds the new tests oddly satisfying to write | 16:50 | |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 7 commits (last: Issue #113: Split tracking and saving in `bst build`) https://gitlab.com/BuildStream/buildstream/commit/fd188b94eb873e54c1615d3842686656e72b3d2e | 16:53 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 16:53 |
| tristan | tlater, good ! | 16:57 |
| tristan | tlater, also satisfying to see the aggregated coverage report raise it's percentage :D | 16:58 |
| tlater | :) | 16:58 |
| gitlab-br-bot | push on buildstream@sam/multiple-caches (by Sam Thursfield): 2 commits (last: Remove the push-port config option) https://gitlab.com/BuildStream/buildstream/commit/bb6c5a5542198c97c83b41b1c55d11b09c884b25 | 17:04 |
| gitlab-br-bot | push on buildstream@sam/multiple-caches (by Sam Thursfield): 1 commit (last: Add support for multiple remote caches) https://gitlab.com/BuildStream/buildstream/commit/efb70b3af625221a2719065efb5d7b4b170b9597 | 17:05 |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 17:06 |
| ssam2 | juergbi, here's my multiple-cache support branch so far: https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 17:06 |
| ssam2 | the fundamentals seem correct to me (review welcome of course) | 17:07 |
| ssam2 | needs work on the test suite before it can be merged | 17:07 |
| *** valentind has joined #buildstream | 17:07 | |
| * tristan thinks juergbi is at the airport getting ready to board a flight... | 17:07 | |
| gitlab-br-bot | push on buildstream@sam/multiple-caches (by Sam Thursfield): 1 commit (last: Add missing keyword arg) https://gitlab.com/BuildStream/buildstream/commit/e04f703021b119d42fc1e687bab79776ca5ba2a4 | 17:08 |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 17:09 |
| ssam2 | me too, but he has a bouncer :-) | 17:09 |
| *** jonathanmaw has quit IRC | 17:30 | |
| tristan | So here is a goal | 17:38 |
| tristan | I want to have a buildstream project for packaging buildstream | 17:39 |
| *** valentind has quit IRC | 17:39 | |
| ssam2 | of course ! | 17:39 |
| tristan | I want one which uses multistrap to generate a base image against which we can automate builds | 17:39 |
| tristan | and output debian packages | 17:39 |
| tristan | similar to the debootstrap-ostree | 17:39 |
| tristan | and I want the same for fedora | 17:39 |
| tristan | which means we need an rpm plugin similar to the dpkg one | 17:40 |
| tristan | and maybe we can use something like the docker thingy to generate the base that we use to package rpms | 17:40 |
| ssam2 | yeah | 17:40 |
| tristan | so we can make a package targetting the right version of fedora or whatnot | 17:40 |
| tristan | And... I want it to be done by end of january | 17:40 |
| *** bethw has quit IRC | 17:40 | |
| tristan | Possible ? | 17:40 |
| tristan | ssam2, I'm talking about this right now, cause I want to submit a buildstream talk for distros devroom at FOSDEM | 17:41 |
| ssam2 | hmm, i have more questions | 17:41 |
| tristan | and that seems like exactly the right demo to make there | 17:41 |
| ssam2 | is the goal a "nice" package which depends on distro pkgs to provide all our deps ? | 17:41 |
| ssam2 | or a "fat" package where we can bundle things like recent OSTree releases, if needed | 17:41 |
| ssam2 | the latter would be simpler | 17:42 |
| tristan | the former is better, i.e. I dont need to roll it out for older distros | 17:42 |
| tristan | just rather, something I can easily provide a ppa for debian based distros and a similar thingy for rpm based distros | 17:43 |
| ssam2 | the former is better, but the latter is OK if needed? | 17:43 |
| ssam2 | i also wonder about stuff that's in pip but not packaged in fedora/debian | 17:43 |
| tristan | well; what would the latter be ? | 17:43 |
| ssam2 | like ruamel.yaml | 17:43 |
| ssam2 | the latter would be a package like 3rd-party app vendors often do, where it bundles dependencies | 17:43 |
| ssam2 | only those needed of course | 17:43 |
| tristan | yeah that would be fine | 17:44 |
| tristan | that sounds like the "correct" thing actually | 17:44 |
| ssam2 | ok, seems feasible then | 17:44 |
| tristan | because doing your definition of the former, sounds like it requires "being part of the distro" itself | 17:45 |
| ssam2 | that, or packaging everything in separate packages | 17:45 |
| tristan | not something that people who write apps and want to deliver onto a distro, normally are | 17:45 |
| ssam2 | so we'd do a ruamel.yaml package for debian and upload it to our PPA, for example | 17:45 |
| ssam2 | but that's more work | 17:45 |
| tristan | at this stage, just having a nice pipeline to show off, that we are reusing mostly the same dependency model and pipeline to build, but just a few conditionals on both ends (conditional base system and conditional package deployment elements) | 17:46 |
| ssam2 | i made a start on a source plugin that can import from docker hub, which might help a bit | 17:46 |
| ssam2 | as it saves us from putting distros into ostree repos that we have to host | 17:47 |
| ssam2 | distros mostly already publish themselves to Docker hub | 17:47 |
| tristan | ... would be a great demo to show off distro agnostic build system nature | 17:47 |
| ssam2 | yeah, it could be cool | 17:47 |
| tristan | and push the narrative/agenda... that decoupling of build system and package delivery/deployment is a good idea | 17:47 |
| ssam2 | this is what Omnibus already does | 17:48 |
| ssam2 | although Omnibus doesn't do sandboxing, so you have to run it on the distro you are making a package for | 17:48 |
| ssam2 | and its definitions language is just Ruby in disguise | 17:48 |
| tristan | (and yeah, docker source plugin + docker output are both good ideas !) | 17:48 |
| tristan | and both probably quite simple to do | 17:49 |
| ssam2 | yeah, it would probably need only a few days to finish off what i started | 17:49 |
| ssam2 | this all sounds feasible to me. and I guess the side effect is we can package BuildStream for everyone :-) | 17:50 |
| tristan | yep | 17:50 |
| tristan | also maybe makes an eventual move from setuptools -> meson, easier | 17:51 |
| tristan | maybe buildstream 1.2 wont be installed with pip, but via a ppa, or regular meson `ninja install` command or such | 17:51 |
| ssam2 | sounds good! | 17:53 |
| * tristan not sure that that part can happen so fast, but desperately hopes we can sever ties with setuptools | 17:53 | |
| * tristan is also wondering about releasing tarballs of buildstream | 17:54 | |
| tristan | we dont have a distcheck in our CI, I should probably try to add that this week | 17:54 |
| tristan | Or, maybe I should make the default CI do that | 17:54 |
| ssam2 | is that a thing for python projects? surely `git archive` is enough ? | 17:54 |
| tristan | instead of `pip install .` in the CI... do `./setup.py sdist` | 17:55 |
| ssam2 | oh yeah, sdist | 17:55 |
| tristan | and extract the generated tarball, and ensure that it works | 17:55 |
| tristan | yeah, since the beginning I had tested that, but the MANIFEST.in may be missing some things | 17:55 |
| gitlab-br-bot | push on buildstream@sam/fixed-ci-image (by Sam Thursfield): 1 commit (last: .gitlab-ci.yml: Use specific version of buildstream-fedora Docker image) https://gitlab.com/BuildStream/buildstream/commit/5c37208c5c5df1925a0bb280933772feb04cb674 | 17:57 |
| gitlab-br-bot | buildstream: merge request (sam/fixed-ci-image->master: .gitlab-ci.yml: Use specific version of buildstream-fedora Docker image) #167 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/167 | 17:57 |
| *** ssam2 has quit IRC | 17:58 | |
| *** semanticdesign has quit IRC | 18:03 | |
| *** semanticdesign has joined #buildstream | 18:03 | |
| *** tristan has quit IRC | 18:04 | |
| *** valentind has joined #buildstream | 18:47 | |
| gitlab-br-bot | buildstream: merge request (sam/fixed-ci-image->master: .gitlab-ci.yml: Use specific version of buildstream-fedora Docker image) #167 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/167 | 18:53 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: .gitlab-ci.yml: Use specific version of buildstream-fedora Docker image) https://gitlab.com/BuildStream/buildstream/commit/5c37208c5c5df1925a0bb280933772feb04cb674 | 18:53 |
| gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/fixed-ci-image | 18:53 |
| gitlab-br-bot | buildstream: merge request (lzip->master: Add support for lzip in tar source) #162 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/162 | 19:47 |
| *** bochecha has quit IRC | 19:50 | |
| *** bochecha has joined #buildstream | 19:51 | |
| *** semanticdesign has quit IRC | 19:53 | |
| gitlab-br-bot | buildstream: merge request (fix-compose-delete-with-symlink-in-path->master: Remove non canonical path from manifest after integration commands in compose plugin.) #161 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/161 | 20:02 |
| *** bochecha has quit IRC | 22:40 | |
| *** tristan has joined #buildstream | 22:46 | |
| *** tristan has quit IRC | 23:12 | |
| *** mcatanzaro has quit IRC | 23:51 | |
| *** tristan has joined #buildstream | 23:52 | |
| *** valentind has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!