| *** tristan has quit IRC | 03:36 | |
| *** tristan has joined #buildstream | 04:11 | |
| *** tristan has quit IRC | 07:29 | |
| *** tristan has joined #buildstream | 07:40 | |
| *** ChanServ sets mode: +o tristan | 07:40 | |
| *** ssam2 has joined #buildstream | 08:55 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: context.py: Make _get_overrides() take a project name, not a project.) https://gitlab.com/BuildStream/buildstream/commit/3a0db8e0b45426c901b0967ae37f9c15a186ba6e | 09:05 |
|---|---|---|
| *** tlater has joined #buildstream | 09:09 | |
| tristan | juergbi, I think this lamba thing which I couldnt easily make heads or tails of is something you added: https://gitlab.com/BuildStream/buildstream/commit/cced3006a39fd8be6e4fa926e68666c672db3d6d#36aa50e17b25fc1aabcf9e316de484ba4592aeae_584_577 | 09:10 |
| tristan | juergbi, did it have a purpose ? | 09:10 |
| juergbi | tristan: that was tlater. in commit 44862d5c he wrote: Prevent the PROVENANCE_KEY from being sorted | 09:12 |
| * tlater reads logs | 09:13 | |
| tristan | oh | 09:13 |
| tristan | strange | 09:13 |
| tristan | Sorry :) | 09:13 |
| tristan | tlater, I think it was always pointless, _yaml.node_sanitize() wants a sorted list, of a dict which _omits_ the PROVENANCE_KEY... I suppose that was a workaround for sorting dicts with int keys | 09:14 |
| tristan | So yeah as I recall that's what it must have been for | 09:14 |
| tlater | tristan: Yes, that's what it was | 09:14 |
| tristan | Removing it in this case is safe because now we strip the provenance before the sort | 09:14 |
| tlater | That makes sense, I thought _yaml.node_sanitize() was supposed to remove the PROVENANCE_KEY | 09:15 |
| tristan | It does | 09:15 |
| tristan | this is just a result of pushing some of my refactoring out of my branch and onto master - that was just house cleaning, removing all the `if PROVENANCE_KEY continue` statements in favor of _yaml.node_items() | 09:17 |
| tlater | Ah, alright. Well, anyway, the unit tests would be screaming at least if the workaround was still necessary :) | 09:18 |
| tristan | yeah, I figured it was safe because cache key tests pass | 09:18 |
| tlater | tristan: I think buildstream!100 and buildstream-tests!16 are ready for a review | 09:58 |
| tlater | Hum... buildstream-tests!16 is supposed to fail. Does gitlab CI somehow remove RO permissions in tar files? | 10:09 |
| tristan | tlater, doesnt it run as root ? | 10:10 |
| tlater | Ah, right | 10:10 |
| tlater | So I suppose that test case can only work as a user... Should I just ignore the test case then or try to switch to a user for those tests? | 10:10 |
| tristan | rather, if we can get the whole test suite to run as a regular user first, at least when running tests for linux platform; that would be best | 10:12 |
| tlater | Yeah, that seems like the best approach. I'll see if that works. | 10:12 |
| adds68 | hi all, i ran successful build of baserock last week using buildstream, i've been side tracked until today, when i've gone back into my sandbox and looked for the build i cannot find it? | 10:35 |
| adds68 | Is there a specific directory where bst outputs to? | 10:35 |
| tlater | What's your 'sandbox'? The build does not produce files, it produces an artifact managed by buildstream. | 10:36 |
| adds68 | tlater, just a pip env | 10:36 |
| tlater | Although, if you really want to look at files, look inside $HOME/.cache/buildstream/artifacts | 10:36 |
| tlater | And in there ostre/refs/heads/<something> | 10:36 |
| adds68 | tlater, ahh ok thank you | 10:36 |
| tlater | adds68: If you want to actually look at the artifacts without checking the cache directory, use bst shell - you can then play with a terminal that has those artifacts built :) | 10:38 |
| tlater | s/terminal/shell/ | 10:38 |
| adds68 | tlater, so if i wanted to load my baserock build into Qemu, i would use the shell? | 10:38 |
| tlater | adds68: For that you would use bst checkout | 10:39 |
| tristan | adds68, note that since we are pretty fast moving these days; it's possible that entire builds appear to go missing after a git update of buildstream | 10:39 |
| tristan | adds68, i.e. tlater's cross platform branch having landed will have done this, as it effects cache keys | 10:39 |
| adds68 | tristan, ahhh ok | 10:39 |
| adds68 | When i try and run build/checkout, i get this error "Error loading project: Could not find file at /home/adamjones/git-sources/buildstream/project.conf" | 11:18 |
| adds68 | Is this a new required file? | 11:18 |
| tristan | adds68, a project.conf is required at the root of any project | 11:22 |
| tristan | adds68, if you did not cd into the project directory, then you can use the -C/--directory option | 11:22 |
| tlater | adds68: Think git repositories, except it's bst x) | 11:23 |
| tristan | Well, I was rather thinking 'directories with makefiles' | 11:23 |
| tristan | but sure :) | 11:24 |
| tlater | bstfile? | 11:25 |
| *** semanticdesign has joined #buildstream | 12:22 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: sandbox.py: Fixup placement of main Sandbox docstring) https://gitlab.com/BuildStream/buildstream/commit/03a646e708d3559b3e0f06c0103b76ef8290872e | 12:36 |
| *** semanticdesign has quit IRC | 12:40 | |
| tlater | tristan: On !100, list_relative_paths() doesn't list directories - _force_rmtree specifically only operates on directories (and has to). I don't think any function in our utils currently covers that. | 13:09 |
| adds68 | tristan, if i receive an "Error loading pipeline" error, would this most likely be due to upgrading to the latest commit? | 13:11 |
| ssam2 | you should get more detail than just those 3 words | 13:12 |
| tristan | adds68, as ssam2 says, I doubt that's all you got | 13:13 |
| tristan | if it is, that's a bug | 13:13 |
| adds68 | tristan, oops sorry "Error loading pipeline: Could not find file at /home/adamjones/git-sources/definitions/./elements/baserock" | 13:14 |
| tristan | adds68, and what was the command you typed to get that ? | 13:15 |
| adds68 | yet in my bst .cache baserock is there | 13:15 |
| adds68 | i did bst checkout baserock ~/buildstream-artifacts | 13:15 |
| tristan | adds68, are you sure you didnt mean `bst checkout baserock.bst` ? | 13:16 |
| tristan | adds68, fwiw, I recommend following these instructions: http://buildstream.gitlab.io/buildstream/completion.html#completion | 13:16 |
| tlater | Lots of people seem to get confused by being allowed to omit elements/ but needing the extension :) | 13:17 |
| tristan | completions should really solve this, though. | 13:17 |
| adds68 | tristan, ah ok i didn't think the extension was needed if it was already in the cache | 13:18 |
| adds68 | tristan, that still produces the same error | 13:19 |
| tristan | and omitting the relative 'element-path' is not allowed, it's rather required | 13:19 |
| tristan | adds68, are you sure the error does not now say "Could not find file at /home/adamjones/git-sources/definitions/./elements/baserock.bst" ? | 13:19 |
| adds68 | tristan, well yes it does, but i mean it's the same error | 13:20 |
| tristan | adds68, and is there a baserock.bst there ? | 13:20 |
| adds68 | I should have added "produces the same error but with baserock.bst" | 13:20 |
| ssam2 | there is no baserock.bst that i'm aware of | 13:21 |
| adds68 | tristan, nothing in the definitions | 13:21 |
| adds68 | So does this mean i have to rebuild or am i missing something out? | 13:21 |
| tristan | ssam2, I suspected as much, but was leaving time for adds68 to figure it out :) | 13:22 |
| tristan | adds68, Well, what command did you originally run to build it with ? | 13:22 |
| adds68 | bst build systems/build-system-content.bst | 13:22 |
| tristan | So maybe you want to check that out ? | 13:23 |
| adds68 | tristan, ah yes i'm an idiot | 13:25 |
| adds68 | thanks | 13:25 |
| tristan | tlater, regarding !100 - my concerns were that A.) We're looking at problems stemming from directories lacking write permissions and ignoring problems stemming from directories lacking read permissions, and B.) we *might* be leaving code paths behind to fend for themselves by not handling it in one place | 13:30 |
| tristan | the latter concern is just difficult to ascertain from a brief reading | 13:30 |
| tristan | tlater, in addition to A and B, I am also concerned that people might expect same day reviews, which would mean I will have a very hard time getting any work done myself :D | 13:31 |
| tristan | so lets sleep on this one at least | 13:32 |
| tlater | tristan: I'm expecting nothing, trying to be respectful :) I just throw things out hoping you'll respond when you have time. | 13:32 |
| * tlater should perhaps get used to using gitlab issues for that | 13:32 | |
| tristan | tlater, yeah... I didnt mean to bark or anyting by that... | 13:33 |
| tlater | ta though, your reviews are much appreciated - also as part of my coding experience :) | 13:34 |
| tristan | flattery will get you everywhere... but yeah, I guess I'm particularly under a lot of pressure with project options dangling above my neck | 13:35 |
| * tristan has option classes allowing definition of different types of option, an option pool which loads from the project option definitions, and does the jinja2 expression resolution based on that, and also setting the options from user config file | 13:36 | |
| * tristan needs to get CLI support for overriding options, implement a function to resolve a dictionary with conditionals using a recursive scan (and fully defining the (?) statements) | 13:37 | |
| tristan | need to remove all the arch options from the cli, config file and loader | 13:38 |
| tristan | and provide a magick (and documented) arch option which is automatically resolved to uname's guess | 13:38 |
| tristan | then, declarative array prepend/append/override remains to be done too | 13:39 |
| * tristan guesses the documentation updates will also be intense :-/ | 13:40 | |
| ssam2 | awful hack du jour : https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/34776023 | 13:54 |
| ssam2 | `docker build` running inside a GitLab CI docker container | 13:54 |
| tlater | \o/ | 13:54 |
| tlater | tbf, that's probably not to uncommon practice. | 13:54 |
| ssam2 | probably not | 13:55 |
| ssam2 | just funny that Docker has this client/server architecture which makes sense for running containers, but really doesn't make sense for building images or pushing them around | 13:55 |
| tlater | s/to/too <- I've been doing that constantly lately | 14:06 |
| ssam2 | i'm trying to work out how the automation of docker image builds should work now | 14:12 |
| ssam2 | should we push a new image every time someone commits to master of buildstream-docker-images.git ? | 14:12 |
| ssam2 | or have a "release new image" button ? | 14:12 |
| ssam2 | should we also push a new image every night with the latest version of BuildStream, or something ? | 14:12 |
| ssam2 | once buildstream 1.0 hits, I guess we'll want a "stable" docker image | 14:13 |
| ssam2 | and possibly an "unstable" one, although i guess that's less important | 14:13 |
| ssam2 | but for now buildstream doesn't do any releases, so we can only do "unstable" docker images | 14:13 |
| tristan | ssam2, as far as gitlab testing goes; what I want ideally is: Never call apt-get install or dnf whatever, never invoke pip install with --upgrade, something explicit to validate a new docker image before we start using it for CI | 14:15 |
| * tristan has to leave closing coffee shop, I'm not the last one, but I know the guy is on holiday tomorrow for the rest of the week and must really be itching to leave | 14:16 | |
| tlater | o/ | 14:17 |
| *** tristan has quit IRC | 14:20 | |
| *** tristan has joined #buildstream | 14:23 | |
| *** xjuan has joined #buildstream | 14:27 | |
| ssam2 | mm i wasn't thinking about the gitlab CI use case | 14:28 |
| ssam2 | but it does matter | 14:28 |
| ssam2 | are you imaging a pipeline that goes: commit to buildstream.git; new docker image produced; gitlab CI for buildstream.git runs tests on the new docker image? | 14:29 |
| ssam2 | that seems pointless though | 14:29 |
| ssam2 | I guess you mean, no `pip install --upgrade` for dependencies | 14:29 |
| ssam2 | all this makes sense | 14:30 |
| ssam2 | I think pushing a new Docker image on every commit to buildstream-docker-images.git is the way to go | 14:30 |
| ssam2 | then buildstream.git can pull a specific version of that image, or 'latest', either way when buildstream adds a new dependency, it's just 1 or 2 pull requests to get that integrated in the CI pipeline | 14:31 |
| tristan | ssam2, yeah basically that would be enough for me | 14:34 |
| tristan | ssam2, if we wanted to be fancy, we could run CI against buildstream master against the most recent docker image, and be able to know that we can safely move to the new one | 14:34 |
| tristan | but meh, not important at all I guess, and maybe convoluted for nothing | 14:35 |
| *** xjuan has quit IRC | 14:35 | |
| tristan | ssam2, what I would really like eventually, say 5 years from now, would be nice if our CI ran against various docker images from various epochs | 14:35 |
| tristan | heh | 14:36 |
| ssam2 | to see if buildstream still works on Ubuntu 17.04 in 2025, and such ? | 14:36 |
| tristan | Yeah | 14:36 |
| ssam2 | ybd does something like that ... https://gitlab.com/baserock/ybd/blob/master/.gitlab-ci.yml#L61 | 14:37 |
| tristan | of course, that presumes we dont start requiring bleeding edge stuff | 14:37 |
| ssam2 | oh wow, I just noticed you can set a different docker image for different gitlab-ci jobs in the same .yml file | 14:37 |
| tristan | also it would be nifty if, starting soon, we could start running a different CI on BSD | 14:37 |
| tristan | ssam2, of course these are all just "wants" :) | 14:38 |
| ssam2 | it looks like some of the current tests actually depend on `dnf remove` being possible | 14:54 |
| ssam2 | i guess that's needed because we don't have an actual BSD runner | 14:55 |
| ssam2 | i might want to switch that to work some other way, like symlinking `bwrap` to /bin/false or something rather than calling `dnf` | 14:56 |
| ssam2 | would be faster to do it that way as well | 14:57 |
| ssam2 | given the extra deps that the test suite requires, i guess I'll add a `devel` variant of the docker image | 15:01 |
| ssam2 | which can contain stuff like sphinx | 15:01 |
| tlater | ssam2: Technically it doesn't depend on dnf remove, but I've found that python can import packages it shouldn't be able to if ostree isn't installed | 15:02 |
| ssam2 | i don't follow | 15:03 |
| tlater | When ostree isn't installed some import statements will cause buildstream to crash. Using the flags to call buildstream that disable linux-specific things, it will still attempt to call those imports, but won't fail if ostree is installed. | 15:04 |
| tlater | So just symlinking might hide bugs | 15:04 |
| tlater | For ostree in specific | 15:04 |
| tlater | Although *techinically* the tests can run just fine without that. | 15:05 |
| tristan | Ah, for ./setup.py test | 15:05 |
| tristan | ssam2, because setup.py does the assertions for ostree/bwrap regardless, and is also the correct entry point for pytest tests | 15:06 |
| tristan | ssam2, also, it's explicitly desirable to not "use a fallback plaform in the absence of bwrap + ostree" | 15:07 |
| tristan | in otherwords, on Linux, we _require_ the linux dependencies | 15:07 |
| tristan | intentionally | 15:07 |
| ssam2 | i agree with you, but https://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml#L60 seems to contract | 15:07 |
| ssam2 | *contradict | 15:07 |
| ssam2 | if BST_FORCE_BACKEND="unix" is enough to test the non-bwrap/ostree fallback, why uninstall them ? | 15:07 |
| tristan | ah right | 15:08 |
| tristan | setup.py recognizes this | 15:08 |
| tristan | sorry | 15:08 |
| tristan | then honestly, I dont follow either :) | 15:08 |
| tristan | but it is a hairy little thing | 15:08 |
| ssam2 | ok | 15:08 |
| ssam2 | well, I can leave it as-is if that's better | 15:08 |
| ssam2 | I was thinking of opening a "avoid package manager calls in .gitlab-ci.yml bug" as part of this docker work i'll be doing | 15:09 |
| ssam2 | a bug entitled "avoid package manager calls in .gitlab-ci.yml", I mean | 15:09 |
| tristan | ssam2, I'm not concerned with calling dnf/apt-get to remove/uninstall things, although I hope it does not result in implicitly upgrading, installing something of an ad-hoc version as a side effect :) | 15:09 |
| ssam2 | right, OK | 15:10 |
| tristan | The point is just to be sure what we're testing against, and that it remains the same thing | 15:10 |
| ssam2 | ok | 15:13 |
| tlater | Egh, sorry for being awful at explaining that | 15:13 |
| ssam2 | so I'll aim at "avoid package upgrades in .gitlab-ci.yml" for this week | 15:13 |
| tlater | ssam2: Basically, an accidental 'import ostree' anywhere can cause buildstream to crash if ostree isn't installed. | 15:14 |
| tlater | But *only* if it isn't installed - it can't be hidden through the environment variable. | 15:14 |
| tlater | Just in case you can think of a better way to do this... | 15:14 |
| tlater | Perhaps something fancy with the modules global could do it | 15:14 |
| ssam2 | oh I see | 15:15 |
| ssam2 | so you remove it in order to catch cases where `import ostree` happens | 15:15 |
| tlater | Yup :) | 15:15 |
| ssam2 | we could install a fake 'ostree' package earlier in PYTHONPATH that would raise an exception when imported | 15:16 |
| ssam2 | sounds like the `dnf remove` calls are fine though anyway | 15:16 |
| tlater | A comment explaining those calls is probably necessary though - really should have had one. | 15:17 |
| *** xjuan has joined #buildstream | 15:22 | |
| ssam2 | i've added one | 15:27 |
| gitlab-br-bot | push on buildstream@master (by Sam Thursfield): 1 commit (last: .gitlab-ci.yml: Add comment after IRC discussion) https://gitlab.com/BuildStream/buildstream/commit/4231a5e6d277ee0450d2359f064901b4be37edc7 | 15:27 |
| * tlater can't believe os.walk's onerror call has exactly one line of documentation and not a single stackoverflow question regarding it. | 15:31 | |
| gitlab-br-bot | buildstream: issue #99 ("Automate builds of official Docker images") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/99 | 15:34 |
| gitlab-br-bot | buildstream: issue #100 ("Avoid package upgrades during CI") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/100 | 15:40 |
| tlater | Ugh, it looks like os.walk can't handle permission errors nicely - you have to restart the entire call | 15:44 |
| gitlab-br-bot | buildstream: issue #101 ("Improve convenience for users running BuildStream through `docker`") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/101 | 15:49 |
| *** jude has quit IRC | 15:49 | |
| *** bochecha has joined #buildstream | 15:50 | |
| *** jonathanmaw has joined #buildstream | 16:34 | |
| *** ssam2 has quit IRC | 16:39 | |
| *** tlater has quit IRC | 17:00 | |
| *** jonathanmaw has quit IRC | 17:19 | |
| *** xjuan has quit IRC | 18:23 | |
| *** xjuan has joined #buildstream | 18:40 | |
| *** bochecha has quit IRC | 20:21 | |
| *** jude has joined #buildstream | 20:57 | |
| *** tristan has quit IRC | 21:37 | |
| *** xjuan has quit IRC | 22:21 | |
| *** jude has quit IRC | 22:34 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!