| *** mcatanzaro has quit IRC | 03:43 | |
| *** bochecha has joined #buildstream | 06:52 | |
| *** bochecha has quit IRC | 07:37 | |
| *** bochecha has joined #buildstream | 07:38 | |
| *** jude has joined #buildstream | 08:42 | |
| * persia is experiencing frustration with yet another electronic calendar that tries to comprehend timezones and fails to present this to the user effectively | 08:42 | |
| persia | Is the meeting today at 14:00 or 15:00 UTC? | 08:43 |
|---|---|---|
| *** anahuelamo has joined #buildstream | 08:45 | |
| *** givascu has joined #buildstream | 08:48 | |
| juergbi | persia: 14:00 UTC | 09:02 |
| juergbi | https://www.timeanddate.com/worldclock/fixedtime.html?iso=20171114T14 | 09:02 |
| persia | Thanks. | 09:02 |
| *** givascu has quit IRC | 09:05 | |
| *** givascu has joined #buildstream | 09:05 | |
| *** valentind has joined #buildstream | 09:16 | |
| *** bochecha has quit IRC | 09:20 | |
| *** noah has joined #buildstream | 09:25 | |
| *** WSalmon has joined #buildstream | 09:41 | |
| *** jonathanmaw has joined #buildstream | 09:49 | |
| *** WSalmon has quit IRC | 10:02 | |
| *** WSalmon has joined #buildstream | 10:02 | |
| *** noah has quit IRC | 10:03 | |
| *** bethw has joined #buildstream | 10:04 | |
| *** adds68 has quit IRC | 10:09 | |
| *** ssam2 has joined #buildstream | 10:12 | |
| *** givascu has quit IRC | 10:19 | |
| *** csoriano has quit IRC | 10:30 | |
| *** csoriano has joined #buildstream | 10:31 | |
| ssam2 | this test failure is frying my head: https://gitlab.com/BuildStream/bst-external/-/jobs/40161531 | 11:04 |
| ssam2 | because I can't reproduce it locally | 11:04 |
| ssam2 | I guess I can try running that test inside the Docker image that we run the tests in | 11:04 |
| ssam2 | ah ok, it occurs in the docker image | 11:11 |
| *** valentind has quit IRC | 11:12 | |
| ssam2 | i didn't add an __init__.py ! of course. | 11:13 |
| tlater | Heh | 11:19 |
| ssam2 | juergbi, is your recursive-pipelines branch pushed anywhere ? | 11:20 |
| ssam2 | i'm interested to try it out a little if possible | 11:21 |
| ssam2 | do people like having the bot notify on every push ? | 11:33 |
| ssam2 | or should we limit it to merge requests and issue changes ? | 11:33 |
| ssam2 | i'm looking at the bot config to add bst-external and buildstream-docker-images repos to its config | 11:33 |
| ssam2 | as well | 11:33 |
| ssam2 | but seems like buildstream.git is the only repo where the bot sends messages for each push. personally i find that quite noisy | 11:34 |
| juergbi | ssam2: not pushed yet, planning to do that soon | 11:44 |
| ssam2 | ok. valentind's branch of freedesktop-sdk has two projects, one which depends on the other, so i thought it'd be a fun testcase | 11:45 |
| juergbi | bot notifying on every push to master (and stable branches) might be ok, but the notifications of pushes to other branches definitely sounds too much to me | 11:46 |
| juergbi | good idea, might set up a test for this | 11:46 |
| ssam2 | hmm, i'm not sure if we can limit push notifications just to master | 11:48 |
| ssam2 | currently it notifies about every branch | 11:48 |
| *** gitlab-br-bot has quit IRC | 11:53 | |
| *** gitlab-br-bot has joined #buildstream | 11:53 | |
| ssam2 | i've left notifications for the buildstream.git repo as they are, but we should now be notified on issues and merge requests for the bst-external and buildstream-docker-images repos | 11:56 |
| ssam2 | meanwhile, review of https://gitlab.com/BuildStream/bst-external/merge_requests/7 would be appreciated :-) | 11:56 |
| *** tristan has joined #buildstream | 12:09 | |
| * tlater is getting very annoyed at docker :| | 12:21 | |
| tlater | It takes like 10 minutes to install buildstream | 12:21 |
| ssam2 | could you be more specific about what you're doing ? | 12:22 |
| ssam2 | docker has multiple storage backends, some of which are very slow | 12:22 |
| ssam2 | so might be a config issue | 12:22 |
| tlater | ssam2: Hm, fair, I'm installing buildstream from a volume mounted on my host | 12:22 |
| tlater | Do you know how to specify a somewhat faster backend for volumes than whatever the default is? | 12:23 |
| ssam2 | https://docs.docker.com/engine/userguide/storagedriver/selectadriver/ | 12:23 |
| tlater | Ta :) | 12:23 |
| ssam2 | overlayfs probably a good one | 12:23 |
| *** bethw has quit IRC | 12:25 | |
| gitlab-br-bot | push on buildstream@97-apply-pep-3102-to-all-public-api-surfaces (by Jonathan Maw): 14 commits (last: Refactor, remove some unused imports) https://gitlab.com/BuildStream/buildstream/commit/cc6e09ce63c7576bd3f258515946e38a945903f4 | 13:04 |
| gitlab-br-bot | buildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/127 | 13:04 |
| gitlab-br-bot | buildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/127 | 13:05 |
| juergbi | reminder, monthly meeting in 55 min in #buildstream-meetings | 13:06 |
| tristan | Oh! | 13:23 |
| tristan | juergbi, one hour before I expected it, is that true ? | 13:23 |
| tristan | or 1h and 55min ? | 13:24 |
| juergbi | the meeting is at 14:00 UTC and it's now 13:24 UTC | 13:24 |
| juergbi | ok for you? | 13:24 |
| tristan | oh yeah, it's great | 13:24 |
| tristan | 11pm instead of midnight is a bonus :D | 13:24 |
| juergbi | :) | 13:25 |
| *** bethw has joined #buildstream | 13:34 | |
| *** tristan has quit IRC | 13:45 | |
| *** tristan has joined #buildstream | 13:55 | |
| * persia peers around | 13:59 | |
| juergbi | meeting imminent in #buildstream-meetings | 13:59 |
| tlater | persia: You mean pers around? | 14:00 |
| persia | Heh, but no, I was in the wrong place. I didn't realise we had enough meetings to need a meeting channel (although the lack of bot notices during the meeting might be nice) | 14:01 |
| juergbi | yes, that was the main motivation for the separate channel, iirc | 14:01 |
| *** xjuan has joined #buildstream | 14:08 | |
| juergbi | xjuan: are you here for the meeting? please join #buildstream-meetings | 14:09 |
| xjuan | juergbi, no, not really :) | 14:09 |
| juergbi | ok, nm | 14:10 |
| *** hashpling has joined #buildstream | 14:23 | |
| *** mcatanzaro has joined #buildstream | 14:29 | |
| *** jonathanmaw has left #buildstream | 14:31 | |
| tlater | Gah, I'm struggling to get buildstream's tests to run under docker | 14:45 |
| tlater | Currently I'm getting ldconfig errors | 14:45 |
| tlater | "Can't find source path /root/modified-buildstream/buildstream/integration-tests/tmp/build/amhello-kzogbjmt/scratch/_/mount: Permission denied" | 14:45 |
| tlater | While running things as root in docker :/ | 14:46 |
| ssam2 | that's weird. it works for me | 14:49 |
| * tlater starts again with just ssam2's image | 14:51 | |
| ssam2 | it looks like a FUSE or possibly bubblewrap issue | 14:51 |
| ssam2 | also, are you running with seccomp disabled ? | 14:51 |
| tlater | ssam2: Yup, that was my first intuition | 14:52 |
| tlater | It's possible I just borked a container enough for it to freak out like that | 14:53 |
| *** hashpling has quit IRC | 15:01 | |
| *** sstriker has joined #buildstream | 15:03 | |
| persia | ssam2: On artifact certification: My initial thought is that artifact should be considered certified if any one of the signatures on the artifact matches any one of the signatures on a whitelist that is configured for the user (possibly inherited from project configuration), and that for many use cases, this is satisfied by a single signature. The part I don't understand is how the UI is complicated if it permits adding a new signature rather | 15:11 |
| persia | than replacing a signature if someone wants to certify a previously certified artifact. | 15:11 |
| ssam2 | how does someone certify a previously certified artifact in the first place ? | 15:12 |
| ssam2 | the initial certification would be done implicitly as part of the build process | 15:13 |
| persia | I presume there exists some command like `bst certify foo` with optional arguments to indicate the key and/or identity to be used. | 15:13 |
| persia | Is there an optional argument to *not* certify a result? | 15:14 |
| ssam2 | i don't see what purpose that would serve | 15:14 |
| * persia can imagine many use cases related to iterative debugging where one would actively not want to certify an artifact. | 15:14 | |
| ssam2 | i don't think users should really be certifying artifacts in the first place | 15:14 |
| ssam2 | the goal is for automated builders to be doing certification | 15:15 |
| persia | In the Debian world, developers are strongly discouraged from certifying any artifact that would be unsuitable for distribution, because doing so breaks using certifications for authentication. | 15:15 |
| ssam2 | if you're developing a new buildstream feature, certainly don't certify or push your artifact anywhere | 15:15 |
| ssam2 | but i don't think it's great to add special UI just for BuildStream developers | 15:15 |
| tristan | ssam2, I think we can have users signing artifacts, and I suspect that we may also want QA teams certifying already built artifacts | 15:15 |
| persia | So, there is a mechanism by which a user can choose to certify or not certify? (where autobuilders are considered as a special class of user)? | 15:16 |
| ssam2 | there has to be some kind of mechanism, yes | 15:16 |
| tristan | ssam2, which might involve a pull from one server and promotion to another server, or perhaps promotion of an artifact to a "tested" branch on the same server | 15:16 |
| ssam2 | like, just don't have a private key, for example | 15:16 |
| tristan | Also, I dont think we want to consider that the automated builder and the regular user are "different things" | 15:16 |
| persia | I don't think it is acceptable to require a user not to have a GPG key in order to not certify things. | 15:17 |
| ssam2 | ok, let's step back a bit | 15:17 |
| ssam2 | what cases are there for producing an artifact that's "unsuitable for distribution" ? | 15:17 |
| ssam2 | bear in mind that if it's in a cache, nobody is going to pull it unless they want to | 15:17 |
| tristan | only artifacts which we dont push anyway | 15:17 |
| tristan | I.e. workspaced | 15:18 |
| tristan | or depending on workspaced | 15:18 |
| persia | In the Debian environment, I usually did it by adding random debug statements into some source code while debugging. I wanted to use the debug artifacts locally, but distributing them would have been bad. | 15:18 |
| persia | They usually also had garbage changelogs, etc. | 15:18 |
| ssam2 | ok. That doesn't apply in BuildStream | 15:18 |
| persia | Why not? | 15:18 |
| ssam2 | someone would have to have exactly the same random debug statements as you in order to pull that artifact | 15:18 |
| ssam2 | in which case, pulling that artifact would be correct | 15:18 |
| ssam2 | (if they did happen to have exactly the same random debug statements in their code, I mean( | 15:19 |
| persia | Ah, right, because artifacts are inherently unordered, so publishing garbage is safe. | 15:19 |
| * tristan will leave you to it and wait for another proposal, at which point I'll probably spend at least half a day thinking about it all over again, and a new discussion can start | 15:20 | |
| persia | Next hurdle: there exist a number of use cases for developers to have more than one GPG key. Is there a useful mechanism yb which the user can specify *which* key to use? | 15:20 |
| ssam2 | yes, i think we should make it configurable which key you use to sign | 15:20 |
| persia | So, yes, now that I realise that publishing garbage is actively useful, I no longer can think of any reasons not to certify, and if I can choose with which key I am certifying, then all is good. | 15:21 |
| tristan | Or, since the *project* probably comes with the full public keys which are acceptable for validation of that project from that project's artifact share | 15:21 |
| tristan | It can probably be deduced from the public keys stored in the project itself | 15:21 |
| tristan | without burdening the user with config options | 15:21 |
| persia | tristan: That isn't a safe choice, as it blocks key transition. | 15:21 |
| tristan | I'll take your word for it now, but suspect that it might be convenient in a lot of cases | 15:22 |
| persia | Also makes it hard for people to publish stuff if they aren't identified as project developers, which breaks various downstreaming use cases. | 15:22 |
| tristan | but I know you have put a lot of thought into this | 15:22 |
| persia | I've no issues with adding it as a possible shortcut for convenience, but think it isn't a safe *only* mechanism. | 15:22 |
| tristan | persia, I dont mean *dont have a config option* | 15:22 |
| persia | tristan: Then I'm not arguing with you :) | 15:23 |
| tristan | persia, I mean, if you *are* a recognized pusher anyway, why not | 15:23 |
| tristan | I mean why not deduce it, if you are already a well known signer, but anyway... | 15:23 |
| tristan | problem can be bigger than that, but might be a nice added convenience | 15:23 |
| tristan | I dont expect most developers who are not on a special list, to be regularly pushing stuff, but that might also happen | 15:24 |
| persia | ssam2: So, I suppose the thing I want is to choose a signature storage mechanism that would potentially allow multiple signatures to exist if someone later wrote separate tooling to support `bst certify ${ARTIFACT_HASH}`, but there's no reason for the implementation you have described above not to be deployed without having written that. | 15:24 |
| persia | ssam2: Do you think that will cause any issues for your implementation? | 15:24 |
| ssam2 | there are various ways it could work internally, so i doubt that would be an issue | 15:25 |
| ssam2 | and makes sense to make it flexible internally | 15:25 |
| persia | Having the internals flexible enough to potentially later support multiple signatures is my only concern. The UI you describe is so amazingly clean that I'm delighted that the first implementation won't have any sensible way for the user to add multiple signatures as it would only clutter the UI. | 15:26 |
| persia | Also, thank you for disabusing me of the concern that there be a requirement for uncertified artifacts: I've spent too many years working with tooling that treated certified artifacts as sacred and meaningful because of the signature rather than the content: this is a far better design. | 15:27 |
| ssam2 | i guess the main question about the internals is whether to store signatures in or out of the artifact | 15:27 |
| ssam2 | storing them within seems simpler but less efficient. so I guess we should probably prototype that first and see how well it works in practice | 15:28 |
| tristan | I still tend to think signatures belong inside the "artifact"; however once you get into an artifact; there are different things to sign | 15:31 |
| persia | My only concern with storing within is whether that breaks hashes: in a content-aware world, I fear changing content for fear it changes the reference to the object. | 15:31 |
| tristan | but, I could be convinced otherwise given strong arguments | 15:31 |
| tristan | also, transmission of the signature inside the logical artifact unit ensures an end-to-end nature | 15:31 |
| tristan | where the artifact server is not a point of failure in the equation | 15:32 |
| persia | In Debian, an "artifact" consists of several files. Certifications are stored as signatures within the "description" and "changes" files, where these files have checksums for all the other files (as a means of proxying certification). I don't know if any of the ideas from that can be reused. | 15:32 |
| ssam2 | we'd have to sign everything except the signatures themselves, of course | 15:32 |
| persia | This is subject to problems in the event of a server failure mid-download, as one might not have the right content, but any mirror of the full set of files in an "artifact" is self-consistent and the certification can be validated. | 15:33 |
| tristan | In the ostree artifact server, if we had multiple branches to store the same artifacts under, we can "promote" an existing artifact by resigning it and pushing it onto a known branch | 15:33 |
| tristan | or a user branch | 15:33 |
| tristan | at no cost of extra disk space on the artifact server, except for the new signature | 15:33 |
| ssam2 | we can also do that with commit metadata | 15:33 |
| ssam2 | however, with commit metadata the identity of the commit stays the same | 15:33 |
| ssam2 | which could be a problem, as how do you know if there are new signatures ? | 15:33 |
| persia | tristan: I don't understand that use of "promote" after developing my appreciation of the content-aware nature of buildstream artifacts. | 15:33 |
| tristan | I am afraid this conversation might be getting deep into treating ostree as something we require, trust and are tightly coupled with | 15:34 |
| tristan | which we are not. | 15:34 |
| ssam2 | that does affect the design, indeed | 15:35 |
| tristan | I just mean, if we *did* add signatures to the abstract artifact storage unit (the definition of the content of an artifact), with ostree, it would be costless to store it multiple times | 15:35 |
| persia | I like the idea of the identity of commits staying the same, as it makes it easier to reuse existing data, despite the complications of not automatically providing guidance on signatures. | 15:35 |
| ssam2 | my main concern is time to hash all 700MB of GCC | 15:35 |
| ssam2 | hopefully that turns out to not matter | 15:35 |
| ssam2 | if it does matter, the fact that OSTree already has a hash of it would come in handy | 15:36 |
| tristan | I am very afraid that a proposal might require ostree to calculate GPG | 15:36 |
| persia | hashing large binaries turns out not to take very long when contrasted to the time required to compute a certificate for a large binary. | 15:36 |
| ssam2 | we can't *require* ostree, as we have a tarball backend | 15:37 |
| persia | This is part of why Debian chooses to run multiple hashes over content, and then only certify the set of hash values (and filesizes) in a separate text file. | 15:37 |
| tristan | ssam2, We dont know what the user has when the user has an artifact | 15:38 |
| tristan | All we know, is the user has the content | 15:38 |
| tristan | that content, should be self verifiable, ideally, also | 15:38 |
| tristan | that seems quite within our capability | 15:38 |
| *** sstriker has quit IRC | 15:39 | |
| tristan | the computation time for generating that signature, I'm not hugely concerned with; if we can assert that our home grown, self contained algorithm for signing/verifying *always matches* that which has assistance from an artifact backend, that would be a nice optimization | 15:39 |
| tristan | that may be a matter of just asking the backend for the hashes and filenames and sizes | 15:40 |
| tristan | or such | 15:40 |
| ssam2 | makes sense | 15:41 |
| persia | I think such an algorithm shouldn't be self-contained: let's depend on GPG or at least on some well-known frequently accelerated crypto hashes, so we aren't a) really slow and b) hopelessly insecure as are all first implementations of new hashes | 15:42 |
| persia | Other than that, works for me :) | 15:42 |
| tristan | persia, yes, in that sense I agree; lets not rewrote GPG in python | 15:42 |
| tristan | *rewrite | 15:42 |
| tlater | ssam2: Could you try running the buildstream integration tests in your container if you have one? Dw if you don't, but I just can't get them to work right now. | 15:57 |
| gitlab-br-bot | buildstream: merge request (130-interactive-prompt-prefix->master: Accept the first characters as shortcut on interruption prompts) #151 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/151 | 16:09 |
| nexus | Is there a doc i can read that explains clearly: What buildstream is, What is does, how, and why? i'm trying to go through all of the docs and add detail, clarify points and fix errors, but i can't find any sign of a generic "This is what Buildstream is" | 16:11 |
| *** givascu has joined #buildstream | 16:29 | |
| tristan | nexus, probably we need to create something to that effect; possibly the right place for that to live would be on the website which currently doesnt exist | 16:43 |
| tristan | nexus, I could send you the original architecture / project pdfs if you'd like to draw inspiration from that | 16:44 |
| nexus | tristan: That'd be great :) | 16:45 |
| persia | While the website is an excellent place for it, the first paragraph of the top-level README would benefit from an abstract, for folk who end up with the repo and not the website. | 16:52 |
| ssam2 | buildstream solves a problem that you only have if you're working on a large project that's split up into many small, different components | 16:52 |
| ssam2 | well, they don't have to be small components, just numerous and different from each other | 16:52 |
| persia | But there are a vast number of projects that fall into this category: building distros, building distributed systems, building integrated computing environments, etc. | 16:53 |
| persia | ssam2: It's a matter of scale. Even vegastrike is a "small" project in the context of a project that consumes vegastrike. | 16:53 |
| persia | Or maybe s/consumes/integrates/ | 16:54 |
| ssam2 | right. I think the key is that it's split up | 16:54 |
| * persia picked vegastrike as it was once on the shortlist of projects that took up so much space on Debian mirrors that it was considered for ejection from Debian | 16:54 | |
| ssam2 | e.g. some companies have enourmous code bases which they keep in a single repo, with a single build system | 16:55 |
| ssam2 | and BuildStream isn't useful there | 16:55 |
| tlater | It does make it quite hard to boil down what it does in a sentence or two, because it's hard to explain to a single person. | 16:55 |
| persia | Unless the company wants to transition to a split model, yes. | 16:55 |
| persia | For me, "BuildStream is a tool that provides a consistent interface for integration developers working on projects that involve many discrete codebases with potentially variant build systems to identify component versions, build components, and generate artifacts suitable for testing or deployment." | 16:57 |
| persia | That sentence needs work, and would probably benefit from a couple more sentences to follow. | 16:57 |
| persia | Also, I suspect there are a number of important BuildStream features not captured there. | 16:57 |
| persia | If nothing else, the workgroup support features (shared caches, shared project config, etc.) deserve some highlighting. | 17:00 |
| * tlater thinks the fact that it uses no host tools is also pretty important | 17:01 | |
| persia | Maybe s/build/bootstrap and build/? | 17:02 |
| persia | Or maybe an additional dependent clause, replacing the full stop with ", from a specified sysroot for improved reproducibility." or something? | 17:03 |
| * persia leaves fixing the sentence to people still in Tuesday | 17:03 | |
| tlater | I like the second one better, because 'bootstrap' sounds too buzzwordy - on the other hand, that may be what we want ;P | 17:03 |
| *** anahuelamo has quit IRC | 17:04 | |
| *** jonathanmaw has joined #buildstream | 17:18 | |
| *** tpollard has quit IRC | 17:33 | |
| *** valentind has joined #buildstream | 17:55 | |
| *** jude has quit IRC | 18:00 | |
| *** jonathanmaw has quit IRC | 18:03 | |
| *** ssam2 has quit IRC | 18:09 | |
| *** bethw has quit IRC | 18:18 | |
| *** givascu has quit IRC | 20:38 | |
| *** tristan has quit IRC | 20:58 | |
| *** bethw has joined #buildstream | 21:21 | |
| *** xjuan has quit IRC | 22:30 | |
| *** mcatanzaro has quit IRC | 23:03 | |
| *** valentind has quit IRC | 23:03 | |
| *** mcatanzaro has joined #buildstream | 23:05 | |
| *** bethw has quit IRC | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!