| *** tristan has quit IRC | 04:22 | |
| *** tristan has joined #buildstream | 04:28 | |
| tristan | Implementing declarative array composition directives (<), (>) and (=) is a damn PITA... but starting to have passing tests \o/ | 07:40 |
|---|---|---|
| *** tiagogomes has joined #buildstream | 08:27 | |
| adds68 | hi, would anybody here know how to compile OSTREE with a http back-end? | 08:28 |
| adds68 | I can't find any flags to enable http at build time | 08:29 |
| *** jude has joined #buildstream | 08:29 | |
| tristan | adds68, http backend ? I didnt even know that exists, what is it supposed to do ? | 08:31 |
| adds68 | tristan, i'm not too sure, but it's used by Flatpak | 08:32 |
| adds68 | error: This version of ostree was built without libsoup or libcurl, and cannot fetch over HTTP | 08:32 |
| adds68 | I recieve this error when trying to pull anything from a http/https url | 08:32 |
| tristan | Ahhh right that | 08:33 |
| tristan | The pulling side | 08:33 |
| tristan | Hmmm, I wonder why you would have an ostree that lacks that | 08:34 |
| tristan | kind of abnormal, yes we also require that | 08:34 |
| tristan | adds68, if you compiled ostree yourself, then you need to install the right dependencies (usually means the corresponding -dev packages with your package manager) | 08:35 |
| tristan | so you probably want something like libcurl-dev to be installed | 08:35 |
| tristan | then reconfigure/rebuild/reinstall | 08:35 |
| tristan | ./configure should normally tell you in the output summary what it's going to build | 08:35 |
| tristan | if it says something like: http backend: None | 08:35 |
| tristan | then you dont have the deps | 08:35 |
| tristan | consult with configure.ac as usual to know exactly what it's looking for | 08:36 |
| adds68 | tristan, yep that would make sense, the last time i built it it reported a http backend, but i had issues with my build, so now i'm rebuilding again it can't seem to find the http backend | 08:36 |
| adds68 | tristan, i shall try that and report back, thanks :) | 08:36 |
| *** jonathanmaw has joined #buildstream | 08:40 | |
| *** tiagogomes has quit IRC | 08:53 | |
| *** tiagogomes has joined #buildstream | 08:54 | |
| adds68 | tristan, even with libcurl4-openssl-dev install, i still get 'none' in the http backend when i configure the install | 09:04 |
| adds68 | tristan, i think it's the openssl curl that is installed on debian, but i've also tried the gnu variant with no luck | 09:05 |
| tristan | and what did a reading of configure.ac tell you ? | 09:06 |
| *** tlater has joined #buildstream | 09:08 | |
| tristan | adds68, to me, https://github.com/ostreedev/ostree/blob/master/configure.ac#L120 this looks like it's not going to even check for your curl if you dont tell it to with --with-curl | 09:08 |
| tristan | maybe it prefers libsoup, I personally would prefer curl | 09:09 |
| *** ssam2 has joined #buildstream | 09:11 | |
| adds68 | tristan, not much that i can deduce | 09:11 |
| adds68 | tristan, but i think you're correct as it's saying ' configure: error: libsoup is enabled but could not be found' when i do ./configure --with-curl | 09:12 |
| * tlater finds that very amusing - why even give the curl option if you won't allow it? | 09:13 | |
| adds68 | Yep libsoup is now enabled, deduced by the --with-curl option :D | 09:15 |
| adds68 | Ostree is awful to build | 09:15 |
| *** bochecha has joined #buildstream | 09:17 | |
| tristan | I very much doubt that it cannot be used | 09:20 |
| tristan | the configure.ac specifies a minimal bound version of curl that it needs if it's going to use it, that might not have been found | 09:20 |
| tristan | Or, your distro may have not provided a .pc file for pkg-config to find it with | 09:21 |
| *** jude has quit IRC | 09:24 | |
| gitlab-br-bot | buildstream: merge request (sam/no-install-python-deps->master: .gitlab-ci.yml: Use `pip3 install --no-index` when installing BuildStream) #110 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/110 | 09:28 |
| *** jonathanmaw has quit IRC | 09:29 | |
| *** jonathanmaw has joined #buildstream | 09:29 | |
| tristan | Does using a declarative list composition directive to apply to a list which doesnt exist cause an error ? | 09:33 |
| tristan | ssam2, thoughts ^^^ ? | 09:34 |
| tristan | Doesnt really make the code more easy or more difficult to write, just a decision to make | 09:34 |
| bochecha | tristan: you mean the >> to append, etc.. ? | 09:35 |
| tristan | yeah it's going to be (<), (>) and (=) | 09:35 |
| tristan | Where (=) is the same as not using any, but just for the sake of tedious completeness | 09:36 |
| tristan | There is some added safety I guess in raising an error | 09:36 |
| bochecha | IMHO, if I try to append to a list which doesn't exist, it's because I expect the list to exist | 09:36 |
| bochecha | it could happen if I made a typo in the list name | 09:37 |
| tristan | yes | 09:37 |
| bochecha | at the very least, there should be a very visible warning about the list not existing | 09:37 |
| tristan | Yeah ok I'm sold, I think better start with an error | 09:37 |
| tristan | I dont like it being a gray area of warnings | 09:37 |
| bochecha | otherwise, people could waste hours trying to figure out why the build doesn't do what they expect, and debugging this would be hell | 09:37 |
| bochecha | errors work to :) | 09:38 |
| tristan | bochecha, what can potentially happen, is that you eventually modify your project.conf to remove something that elements are compositing to | 09:38 |
| tristan | which means you need to explicitly update elements in that case | 09:38 |
| tristan | I guess that's the main downside to the error | 09:38 |
| bochecha | it's also easier to later open it up by making it a warning, than transforming a warning into an error (which breaks existing projects) | 09:38 |
| tristan | Also, with validation currently in place, there is not much room for typos, but they could indeed happen in public data | 09:39 |
| tristan | which is more free form... any element/source configuration is validated quite strongly | 09:39 |
| tristan | And variables/environment are not allowed to have lists anyway | 09:39 |
| tristan | meh -> error for now :) | 09:40 |
| bochecha | what about configure-commands? isn't that a list in a variable? | 09:41 |
| tristan | yes but if you put comfigure-corrands into a BuildElement's config... you will get an error | 09:41 |
| tristan | Oh but... no that is not a "variable" | 09:42 |
| tristan | bochecha, by variable I mean, what you declare as "variables:" and can expand as %{foo} | 09:42 |
| tristan | variables:, like environment: is a flat list of strings | 09:43 |
| tristan | bochecha, so yeah... this could probably be documented better (and I think I have libreoffice draw images from my initial designs which could help) | 09:44 |
| tristan | bochecha, basically; configure-commands is a part of an elements "configuration", which appears strictly under the "config:" section of any element | 09:45 |
| tristan | bochecha, what that configuration is, is in the domain of the element implementation to document and validate | 09:45 |
| tristan | depends, sources, description (if specified), and core features like conditionals... is in the domain of buildstream core... "config:" is the domain of the plugin defining it, and is private to that plugin... while "public:" is data that any element may read off of it's dependencies | 09:46 |
| *** jude has joined #buildstream | 09:48 | |
| tristan | sources themselves are a bit different; instead we delete some things that buildstream core parses (i.e. 'kind') and just give the rest of the dictionary to the source plugin | 09:48 |
| bochecha | ah, so if in `config:` I mistype `configure-comands` I'll get a validation error? | 09:49 |
| tristan | yes | 09:49 |
| *** anahuelamo has quit IRC | 09:49 | |
| bochecha | col | 09:49 |
| bochecha | * cool | 09:49 |
| *** jonathanmaw has quit IRC | 09:49 | |
| tristan | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/buildelement.py#L119 | 09:49 |
| tristan | linking that just cause I think it's cool how it works :) | 09:50 |
| *** jonathanmaw has joined #buildstream | 09:50 | |
| * tristan not sure that comment is worth a "FIXME" prefix | 09:52 | |
| tlater | You should prefix it with a FIXME to remove the FIXME | 09:55 |
| *** anahuelamo has joined #buildstream | 10:03 | |
| *** givascu has joined #buildstream | 10:24 | |
| tlater | tristan: About the integration tests... Any .bst file that imports a local tar file has to be modified to load from $(pwd), since buildstream only supports the 'file://' protocol for those. Avoiding to dirty some of the source tree seems to be impossible. | 10:30 |
| tristan | tlater, I'm losing context of what you are commenting on, maybe better to comment on that inline with discussion and link to it here ? | 10:31 |
| tlater | Sure | 10:31 |
| tlater | tristan: https://gitlab.com/BuildStream/buildstream/blob/master/integration-tests/autotools-test/elements/amhello.bst#L12 | 10:37 |
| tristan | tlater, ummm that's a link to something... your message here implies you are responding to something I said, no ? | 10:38 |
| tlater | Whoop, sorry, had the wrong thing in my clipboard | 10:39 |
| tlater | https://gitlab.com/BuildStream/buildstream/issues/108#note_43409599 | 10:39 |
| tristan | ah yeah, found it via inbox | 10:39 |
| tristan | tlater, that is not unavoidable at all | 10:39 |
| * tristan comments inline, better | 10:40 | |
| tlater | Ah, right, that's a way to do it. ta :) | 10:42 |
| tristan | Ok... another question for the team ! ... Shall we just drop the buildstream-tests repository entirely, after having addressed https://gitlab.com/BuildStream/buildstream/issues/96 ? | 10:43 |
| tristan | Does anyone have any work floating around in there ? | 10:43 |
| ssam2 | we already merged the gnu-toolchain branch into baserock/definitions.git | 10:43 |
| tlater | tristan: It has some useful old test branches, such as the one detailing how x86images work | 10:43 |
| * tristan starts by archiving it | 10:44 | |
| ssam2 | tlater, is that not defs2bst.git ? | 10:44 |
| tlater | ssam2: It refers to a branch in buildstream-tests | 10:44 |
| ssam2 | ah right | 10:44 |
| tristan | So at least buildstream-tests is now read-only | 10:44 |
| tristan | the defs2bst.git readme should probably new refer to baserock stuff instead | 10:45 |
| tristan | *now | 10:45 |
| * tlater should hopefully clean the x86image documentation soon anyway | 10:45 | |
| tristan | x86image had an .omit test, all that omitted stuff, and omitting mechanics, has been removed | 10:46 |
| tristan | And whatever it was, if revived, should appear in bst-external, which jonathanmaw is looking into | 10:46 |
| ssam2 | tristan, i've been thinking about artifact caches more | 11:51 |
| ssam2 | getting ostree-push to work over http(s) would be pretty non-trivial | 11:51 |
| ssam2 | so trying to picture how things would look if #111 (tarball push & receive) got fixed ... | 11:52 |
| ssam2 | i'd say we should have a http endpoint that serves tarballs, and a change to bst-artifact-receive such that tarballs can be pushed over SSH | 11:52 |
| ssam2 | and we keep 'push over https' as a separate task entirely. I guess my question is, do you care enough about push-over-https that it's worth me writing up my thoughts on that? | 11:53 |
| tristan | ssam2, I think push over https is orthogonal, and adding such support in the future would not break any user configuration right ? | 12:00 |
| tristan | ssam2, if we have one kind of url that we can push/pull from, and another that can only be pulled from; that seems reasonable | 12:00 |
| tristan | and in the future if we handle https push, that becomes a feature | 12:01 |
| ssam2 | that makes sense | 12:01 |
| ssam2 | i think the main arguments for push-over-https are (a) push behind firewalls that block ssh, and (b) more flexibility with user auth | 12:02 |
| ssam2 | but (b) can be achieved by customizing the ssh daemon too... and (a) can be achieved with corkscrew | 12:02 |
| tristan | It's better to think about enclosing this into a nice black box that we can reimplement behind the scenes without breaking peoples workflow on upgrades | 12:03 |
| tristan | yeah; other features will be nice, and having that box should let us grow features :) | 12:04 |
| ssam2 | i think "black box" isn't really practical server-side | 12:05 |
| tristan | I guess in any world, push/pull over ssh will always require some admin (or maybe one day a web interface ?) has to obtain a user's private key | 12:05 |
| ssam2 | it's always going to require some http server config | 12:05 |
| ssam2 | and we don't want to force people to use a specific http server | 12:05 |
| tristan | yeah, as horrible as that is; we'll have to live with it | 12:05 |
| tristan | so I guess, some upgrades might require a sysadmin on the artifact server side | 12:05 |
| ssam2 | yeah | 12:06 |
| ssam2 | but sysadmins are used to stuff like this https://git-scm.com/docs/git-http-backend | 12:06 |
| tristan | but user/client configuration and such is always stable and back compat | 12:06 |
| ssam2 | right | 12:06 |
| *** semanticdesign has joined #buildstream | 12:31 | |
| tristan | whew... ok so now I have test cases asserting all the declarative list composition works; think I have the whole surface covered, with single / double compositions; i.e. one can append to a list where the intermediate layer is prepending to the same list, or double append/prepend | 12:45 |
| tristan | That all works; what I am left with... is a huge mess in _yaml.py with waaay to much procedural code :) | 12:46 |
| tristan | so boil it down to about half the loc count... and should be good to close the book on that | 12:46 |
| gitlab-br-bot | push on buildstream@108-integration-tests-not-idempotent-and-self-contained (by Tristan Maat): 20 commits (last: tests/format/assertion.py: Added tests for the (!) assertion directive) https://gitlab.com/BuildStream/buildstream/commit/f9df02735253ce530a7fa6c5fb11a4d9ded97a29 | 14:11 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 16 commits (last: Ensure that artifact file permissions are set in the right order) https://gitlab.com/BuildStream/buildstream/commit/a968bab76201856a67c876be25745fee912c136b | 14:28 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: WIP: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 14:28 |
| *** tristan has quit IRC | 14:36 | |
| *** tristan has joined #buildstream | 14:40 | |
| *** jude has quit IRC | 14:48 | |
| jjardon[m] | Hi, FYI I have completed https://wiki.gnome.org/jjardon/BuildSystemComponentBuildstream a little more, but unfortunatelly Im blocked by https://gitlab.com/BuildStream/buildstream/issues/92 | 15:12 |
| jjardon[m] | Any help from anyone using a distro different from Arch would be appreciated! | 15:12 |
| tristan | jjardon[m], ohh that's a shame, but arch is like, highly configurable thing right ? there is no user namespaces available for arch ? | 15:30 |
| * tristan finds this very surprising, as he thinks jjardon[m] normally runs blood-on-the-floor-bleeding-edge stuff :) | 15:30 | |
| tristan | jjardon[m], I wanted to give it a go but... I actually did a ton of other work this weekend, and am closing in on solving the last "hard feature" that needs to land for 1.0 | 15:31 |
| tristan | jjardon[m], ssam2... others who have baserocking history: I feel like just dropping the pre-/post- notations for all commands in BuildElements | 15:32 |
| jjardon[m] | tristan: Its possible, but I have to recompile my kernel | 15:33 |
| tristan | Seeing as you can now prepend, append or override lists declaratively | 15:33 |
| tristan | Any thoughts on that ? | 15:33 |
| tristan | it would be a bit annoying to change the migrations, but would give us something cleaner after I think | 15:33 |
| jjardon[m] | not sure we can ask that to all the Arch users; convince arch kernel maintainer is probably the best approach | 15:33 |
| tristan | jjardon[m], I agree | 15:33 |
| tristan | jjardon[m], I'm surprised though, seeing as debian even now has it in "stable" | 15:34 |
| tristan | with a pretty slow release cadence | 15:34 |
| tristan | I bet if we look for it, arch probably already has a bug where that is being discussed | 15:34 |
| jjardon[m] | tristan: is not about having the latest features, but about security | 15:35 |
| tristan | we could at least weigh in there | 15:35 |
| jjardon[m] | Arch consider that option to not be secure | 15:35 |
| tristan | jjardon[m], I know, but it's not enabled by default on debian either | 15:35 |
| tristan | i.e., you have to either be superuser to use it; or, you have to explicitly enable it through /proc or /sys as superuser | 15:35 |
| tristan | still the kernel lets you do it, and one can either enable it, or decide to install setuid binaries they trust, to do it | 15:36 |
| tristan | weighing in on a probably-existing discussion in arch linux is probably a good idea | 15:37 |
| ssam2 | tristan, mm... how would that look? something like this? https://pastebin.com/s3Puj1nJ | 15:37 |
| jjardon[m] | ah yeah; seems Arch doesnt like that neither : "# CONFIG_USER_NS is not set" | 15:37 |
| tristan | jjardon[m], note that; that also would mean no flatpaks for arch | 15:37 |
| jjardon[m] | in my /proc/config.gz | 15:37 |
| tristan | ssam2, approximately yes, you dont have the indent at the right location I think | 15:38 |
| ssam2 | oh yeah | 15:38 |
| jjardon[m] | tristan: I know, I know; lets not block on this. I think is Arch what it needs to change, only wanted let you know | 15:38 |
| ssam2 | so it's slightly more typing than before, but probably worth it for consistency | 15:38 |
| tristan | https://bpaste.net/show/b58549224441 | 15:38 |
| tristan | yeah | 15:38 |
| tristan | it removes an unneeded feature | 15:39 |
| tristan | and makes things probably all around less confusing | 15:39 |
| tristan | jjardon[m], so you dont have flatpak on arch ? | 15:39 |
| ssam2 | yes, people did often use pre-configure-commands for things they weren't intended for | 15:39 |
| tristan | I guess not | 15:39 |
| jjardon[m] | https://wiki.archlinux.org/index.php/Bubblewrap and https://bugs.archlinux.org/task/36969 | 15:40 |
| jjardon[m] | tristan: I have | 15:40 |
| tristan | hmmm, how do you have flatpak without user namespaces I wonder ? | 15:40 |
| tristan | does flatpak fallback to only seccomp ? | 15:40 |
| tristan | maybe | 15:40 |
| * jjardon[m] wonders now as well | 15:41 | |
| tristan | jjardon[m], it looks anyway like that discussion is on the right track and coming to the same conclusions as debian | 15:42 |
| tristan | i.e. not enabled by default, with warning | 15:42 |
| * tristan doesnt think you can do an unprivileged chroot without user namespaces and only seccomp | 15:42 | |
| gitlab-br-bot | push on buildstream@sam/use-via-docker-2 (by Sam Thursfield): 1 commit (last: Add `bst-here` script and document it) https://gitlab.com/BuildStream/buildstream/commit/c52a57d0c789164b99d91ada7ded373f2d82a2e0 | 15:57 |
| tristan | ssam2, is there any significant overhead in running bst-here / docker ? | 15:58 |
| tristan | asides from downloading the image and the disk space it takes ? | 15:58 |
| * tristan never tried docker in any way whatsoever | 15:59 | |
| tristan | clueless | 15:59 |
| ssam2 | if set up properly, overhead is pretty negligible | 15:59 |
| tristan | maybe memory and diskspace but processing is just fine ? | 15:59 |
| ssam2 | docker has multiple storage backends, some of which are pretty inefficient | 15:59 |
| tristan | ah | 15:59 |
| ssam2 | but the buildstream cache lives in a separate volume independent from the copy-on-write storage backend stuff in any case | 16:00 |
| tristan | so a passthrough, especially buildstream is filesys intensive, can get annoying | 16:00 |
| ssam2 | in my limited testing the bst-here wrapper worked fine | 16:00 |
| tristan | Anyway sounds great - but still I guess unfortunately, we cannot recommend that as an alternative for arch users right ? | 16:00 |
| ssam2 | no, because it still uses the host kernel | 16:01 |
| tristan | ssam2, if you can make a note somewhere in your docs, as to whether or not it drops the requirement of CONFIG_USER_NS that would be good :) | 16:01 |
| tristan | yeah I guess | 16:01 |
| * tristan was wishful thinking | 16:02 | |
| gitlab-br-bot | push on buildstream@sam/use-via-docker-2 (by Sam Thursfield): 1 commit (last: Add `bst-here` script and document it) https://gitlab.com/BuildStream/buildstream/commit/e3462b664f035e38fc558eb33ec4729c50aa0d03 | 16:07 |
| gitlab-br-bot | buildstream: merge request (sam/use-via-docker-2->master: Add `bst-here` script and document it) #113 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/113 | 16:08 |
| tlater | Is there an artifact share for the baserock artifacts? /me is getting tired of building gcc | 16:18 |
| tlater | Unfortunately testing means frequently flushing caches :/ | 16:19 |
| tlater | ssam2: I recall you saying something about fuse, do you have an idea what might be causing this: https://gitlab.com/BuildStream/buildstream/-/jobs/36402818? | 16:21 |
| ssam2 | that's weird | 16:23 |
| ssam2 | missing `fuse` package would cause it ... but i'm sure that's present in the buildstream-fedora image | 16:24 |
| tlater | I just rebased it, so I'm a little confused. Note this does *not* run as root anymore | 16:25 |
| tlater | Maybe we can't run fuse in the CI without privileges? | 16:25 |
| ssam2 | maybe, but that wouldn't explain that error | 16:25 |
| ssam2 | actually, fusermount *isn't* present in the iamge | 16:25 |
| ssam2 | *image | 16:25 |
| ssam2 | but I guess it's only called in an exception handler codepath | 16:26 |
| ssam2 | so we didn't notice til now | 16:26 |
| tlater | Hm. It seems that exception also ends up in a loop, since gitlab doesn't detect it failing | 16:26 |
| ssam2 | about your previous question, https://ostree.baserock.org/cache/ might have what you need | 16:26 |
| ssam2 | also the ci that populates that is currently broken | 16:26 |
| tlater | ssam2: Doesn't matter, at least the actual artifacts should work. ta :) | 16:27 |
| * tlater will try to find the exception handler that causes that error then | 16:27 | |
| * ssam2 presents tlater with https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/3 | 16:28 | |
| tlater | \o/ | 16:28 |
| *** ssam2 has quit IRC | 17:01 | |
| tristan | fusermount is setuid and called by libfuse in the case of not being root | 17:01 |
| tristan | that's the conclusion I think juergbi said | 17:01 |
| tristan | Maybe we need to also require fusermount in setup.py | 17:02 |
| juergbi | yes, while libfuse attempts to mount without the setuid helper, mainline kernels don't allow this for unprivileged users, afaik, so libfuse falls back to the setuid helper | 17:08 |
| juergbi | afaik, it's not allowed in user namespaces either (where some mount(2) calls are allowed) | 17:08 |
| gitlab-br-bot | buildstream: issue #116 ("BuildStream usually requires fusermount") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/116 | 17:17 |
| tristan | it's filed... so it happened. | 17:17 |
| tristan | :) | 17:17 |
| *** jonathanmaw has quit IRC | 17:18 | |
| *** tlater has quit IRC | 17:20 | |
| *** givascu has quit IRC | 18:22 | |
| *** jude has joined #buildstream | 20:35 | |
| *** jude has quit IRC | 22:16 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!