| *** phoenix has quit IRC | 01:00 | |
| *** rdale has quit IRC | 01:01 | |
| *** krichter[m] has quit IRC | 02:40 | |
| *** tchaik[m] has quit IRC | 02:42 | |
| *** krichter[m] has joined #buildstream | 02:55 | |
| *** tchaik[m] has joined #buildstream | 03:10 | |
| *** tristan has quit IRC | 05:01 | |
| gitlab-br-bot | juergbi opened (was WIP) MR !1878 (juerg/vdirectory->master: storage: Improve Directory API) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1878 | 05:20 |
|---|---|---|
| *** tristan has joined #buildstream | 05:30 | |
| *** ChanServ sets mode: +o tristan | 05:30 | |
| *** traveltissues has quit IRC | 05:36 | |
| *** traveltissues has joined #buildstream | 05:36 | |
| juergbi | tristan: after merge of !1878 I'm planning to tag a 1.93.2 snapshot, so we can depend on it in bst-plugins-experimental and complete the conversion to the virtual directory API | 05:40 |
| tristan | Makes sense | 06:48 |
| tristan | juergbi, I'm not sure there are API breaks in this, but it looks like your plan is to do some API improvements first, then depend on that version and upgrade any plugins to make best use of API, and then later we make virtual directory API mandatory ? | 06:50 |
| juergbi | yep, no API breaks in this one, just extensions | 06:51 |
| juergbi | in future API review/cleanup there might be breaks, though. to be discussed | 06:51 |
| juergbi | first want to get the overdue virtual directory migration done | 06:52 |
| tristan | yup | 06:52 |
| juergbi | with this I can build the main parts of freedesktop-sdk with buildbox-run-bubblewrap and buildbox-fuse, so we should also be close to that migration | 06:52 |
| tristan | juergbi, after our immediate goings on, we might want to touch base and prioritize on any blocking core work which finalizes and improves the plugin stories | 06:53 |
| tristan | Doing that would help unblock/clarify any easier work surrounding plugin migrations and such | 06:54 |
| juergbi | tristan: yes. have we already checked that we have gitlab issues for the points from the bst 2.0 planning mail, and assigned them to the 2.0 milestone? | 06:55 |
| juergbi | if so, we can probably also make use of a gitlab board | 06:55 |
| tristan | Just a thought, I have a sneaking suspicion there are some people looking at plugins which are planned to be moved, and are just itching to do the busy work in moving them around and setting up repos and such and such | 06:55 |
| tristan | Not yet no, that's probably a good thing to do | 06:56 |
| tristan | wait, the milestone or the "board" ? | 06:56 |
| tristan | Oh both, well - no objections maybe a board would be helpful in this instance to prioritize things in orders which unblock more parallelizable work | 06:56 |
| juergbi | we can create a board for the 2.0 milestone, might be useful to coordinate | 06:58 |
| tristan | then we'd use those fancy features like "assignee" and using tags like "Todo, Doing, Done" so we could visualize it in this super modern fancy way | 07:00 |
| tristan | in general I feel that it's all way overkill in normal times, but no objections :) | 07:01 |
| juergbi | yep but maybe for 2.0 blockers it would be useful to keep track | 07:01 |
| tristan | Lets anyway start with milestoning the issues | 07:02 |
| tristan | I think that we will have a good 3 rounds of blockers before the release, where we start scavenging into the tedium near the end | 07:03 |
| *** rdale has joined #buildstream | 07:11 | |
| *** benschubert has joined #buildstream | 07:27 | |
| WSalmon | So i think this branch may need spliting in to other tests and git_lfs stuff for the MR but for now its easist just to do all the hard work in one go. after some quick greping it looks like core bst tests can create git repos `tests/frontend/track.py: repo = create_repo("git", str(tmpdir))` but when i try to do it from bst-plugins-experimental then git is not a valid option https://gitlab.com/BuildStream/bst-plugins-experimental/-/jobs/523264319 | 07:37 |
| WSalmon | any one got any hints? | 07:37 |
| benschubert | WSalmon: you don't register a `kind` for `git` in your tests | 07:39 |
| benschubert | for getting access to `create_repo`, you should `register_repo_kind` in your conftest as done by buildstream: https://gitlab.com/BuildStream/buildstream/-/blob/master/tests/conftest.py#113 | 07:41 |
| benschubert | so you'd need to create a `git_lfs` repo and register it | 07:41 |
| WSalmon | that test has nothing to do with git-lfs thats one of the submodule tests that never got moved from bst-external to bst-plugins-experimental | 07:43 |
| WSalmon | i will make a seperat MR for the git-lfs stuff and the `other tests` | 07:43 |
| WSalmon | but just getting everything to work is a faff for two branches | 07:44 |
| WSalmon | all the git-lfs stuff seems to work now | 07:44 |
| WSalmon | i just want to use create_repo as its done in places like `tests/frontend/track.py` i cant see how that one registers the `git` repo | 07:45 |
| benschubert | See the link above to how the buildstream test register it | 07:46 |
| benschubert | but the git repo is afaik private to buildstream, so we should probably make that public/move it first | 07:46 |
| WSalmon | yep | 07:52 |
| *** phildawson-ct has joined #buildstream | 07:58 | |
| gitlab-br-bot | BenjaminSchubert opened MR !1879 (bschubert/fix-mktemp-usage->master: tests: Stop using tmpdir_factory.mkdtemp("")) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1879 | 08:05 |
| gitlab-br-bot | BenjaminSchubert opened MR !1880 (bschubert/no-warnings->master: Fix some warnings happening during the tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1880 | 08:06 |
| *** santi has joined #buildstream | 08:07 | |
| benschubert | abderrahim[m]: would you have time to reply to my question in !1840 please? :) If you think the MR is actually not useful, I'll just close it | 08:08 |
| gitlab-br-bot | MR !1840: _stream.py: Make `_enqueue_plan` a timed activity https://gitlab.com/BuildStream/buildstream/-/merge_requests/1840 | 08:08 |
| traveltissues | thanks for having a look at !1810 benschubert what was the specific issue with the cachekey tests? | 08:24 |
| gitlab-br-bot | MR !1810: WIP: Remove the pip build element & source plugins https://gitlab.com/BuildStream/buildstream/-/merge_requests/1810 | 08:24 |
| *** tpollard has joined #buildstream | 08:43 | |
| benschubert | traveltissues: ah sorry about the confusion, I was working on a way of having tests marking that they require external plugins so we don't loose such tests :) | 08:49 |
| traveltissues | i understand, how far did you get with that? | 08:50 |
| benschubert | I am almost there, so could probably have it for review by tomorrow evening | 08:51 |
| traveltissues | nice, i wasn't sure how difficult that would be with the current test env | 08:51 |
| benschubert | not that much actually, but I needed a goot test case and you just provided it to me :D | 08:53 |
| traveltissues | the commit i posted should become obsolete then and replaced with your work | 08:54 |
| benschubert | yeah I could revert the commit and add the tag and that should work | 08:56 |
| traveltissues | that commit was just on a separate testing branch so there's no impact | 08:57 |
| benschubert | ah ! awesome I misunderstood everything then x') | 08:58 |
| traveltissues | i suspect there'll be a similar issue regarding removing tar source from bst-plugins-experimental | 08:58 |
| traveltissues | np benschubert, no i just cherry-picked tom's commits on top of master | 08:58 |
| benschubert | I actually am writing an ML thread which proposes to move tar back to core (among other things :/) | 08:59 |
| traveltissues | there seems to be some long-running uncertainty about this issue, since some of these seem to have been moved out of core in the past | 08:59 |
| benschubert | ah I just mean bringing back tar | 09:00 |
| traveltissues | i'm guessing it was never firmly decided what the criteria was for which plugins had first-class support | 09:00 |
| traveltissues | tar element as well? | 09:00 |
| benschubert | no | 09:00 |
| benschubert | just tar source | 09:00 |
| benschubert | as we hugely rely it on for tests and it would allow a bootstrap of plugins iff we have plugins that can be elements :) | 09:01 |
| benschubert | which is the core of my proposal | 09:01 |
| benschubert | but I still need to finish writing it | 09:01 |
| traveltissues | i started fiddling with this https://gitlab.com/BuildStream/bst-plugins-experimental/-/tree/traveltissues/remove-tar | 09:02 |
| traveltissues | btw is there some reason why tox hangs locally? sometimes it gets stuck at sdist-make step until i remove the cache | 09:05 |
| benschubert | no idea | 09:05 |
| traveltissues | it's my main complaint with tox | 09:06 |
| benschubert | running on windows? | 09:07 |
| traveltissues | debian 10 | 09:07 |
| benschubert | ok no idea then | 09:08 |
| *** lachlan has joined #buildstream | 09:11 | |
| WSalmon | any one know why we cant run the plugin tests in parallel? there are peobally going to be a load more of these soon so i imaging sorting this might be we worth it. https://gitlab.com/BuildStream/bst-plugins-experimental/-/commit/cd71f966f280d40b4a27370048675f95ce6a6f2c https://gitlab.com/BuildStream/bst-plugins-experimental/pipelines/139038014 | 09:17 |
| WSalmon | I am happy to look in to this but wanted to check some one didnt already know the issue | 09:18 |
| benschubert | WSalmon: you might need something like https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/testing/_fixtures.py#L25 | 09:21 |
| benschubert | if it works, we could probably make it public | 09:21 |
| juergbi | WSalmon: it seems like the buildstream fixtures are not used src/buildstream/testing/_fixtures.py | 09:21 |
| juergbi | ah :) | 09:21 |
| WSalmon | it looks like quite a bit more of src/buildstream/testing/ needs to public is there a good reason to keep some of it private? | 09:22 |
| benschubert | because nobody needed anything from it | 09:23 |
| benschubert | and it's simpler to make things public than to make things private | 09:23 |
| benschubert | so opening as needed :) | 09:23 |
| juergbi | tristan: I think we should review the default project config for 2.0, do you agree? I'd like to strip it down to a minimum. I prefer sane linux defaults to be provided by projects that also provide a good starting point for bootstrap/toolchain etc. (e.g., fd-sdk) | 09:24 |
| tristan | juergbi, I agree - but at the same time I'm quite convinced that we need a strategy for changing defaults in the future (both for project.conf defaults and for plugins), which changing defaults needs to be opt-in such as to retain cache key stability | 09:26 |
| tristan | Making it more bare bones is good but doesnt solve the cache key/changing defaults problem | 09:26 |
| *** santix has joined #buildstream | 09:27 | |
| * tristan finally passed his tests for the "min-version" branch, will have to check with integration tests and all tests enabled | 09:27 | |
| *** santi has quit IRC | 09:27 | |
| *** lachlan has quit IRC | 09:27 | |
| tristan | I'll probably submit an MR today... | 09:27 |
| juergbi | tristan: remaining issue after making it mininmal: are you referring to possible changes in the minimal default config or changes in the 'defaults' provided by an include in fd-sdk or similar? | 09:27 |
| juergbi | great | 09:27 |
| tristan | juergbi, I'm referring to things like... people recently "fixed" the defaults of the cmake plugin in order to use typed configurations ( -DFOO:PATH=/bar/baz or whatever it is ) | 09:28 |
| tristan | Those changes will just keep coming over time, wherever they are | 09:28 |
| * tristan heads out for some food and will be back soon... | 09:29 | |
| juergbi | that's plugin-specific, but I guess we could come up with a solution that covers project and plugin config | 09:30 |
| *** tristan has quit IRC | 09:32 | |
| *** phildawson-ct has quit IRC | 09:39 | |
| *** lachlan has joined #buildstream | 09:41 | |
| benschubert | has anyone seen errors like https://gitlab.com/BuildStream/buildstream/-/jobs/523326120#L2512 ? | 09:43 |
| traveltissues | yes, i've been seeing them on those hosts, they sometimes fail | 09:44 |
| benschubert | juergbi: is it the new runner which is slower? | 09:45 |
| juergbi | benschubert: yes, well, the 'permanent-runners' are two larger machines that run multiple jobs in parallel, so there is more variance | 09:46 |
| *** cs-shadow has joined #buildstream | 09:46 | |
| benschubert | I see, so we should make those timeouts longer ? | 09:46 |
| juergbi | the plan is to switch to kubernetes but I don't know how soon we can do that and how the variance will be with kubernetes | 09:46 |
| benschubert | I see | 09:47 |
| juergbi | we probably should, yes | 09:48 |
| juergbi | for one such failure, the timeout is already about 10x what the test takes on my local machine, though | 09:48 |
| juergbi | I suspect maybe some I/O bottleneck or so | 09:48 |
| benschubert | :/ | 09:49 |
| juergbi | there might also be some tuning possibilities for fairer scheduling within the VM but it probably doesn't make too much sense to do this now before moving to kubernetes | 09:50 |
| juergbi | e.g., not sure whether cgroups are used for the different jobs | 09:50 |
| benschubert | yeah, bumping the timeout instead os not too bad | 09:50 |
| juergbi | no, shouldn't really hurt | 09:51 |
| benschubert | and overall tests are running much more consistently :) | 09:51 |
| juergbi | maybe 5 minutes instead of the 60 second timeout for long pexpect | 09:51 |
| *** phildawson-ct has joined #buildstream | 09:56 | |
| *** phoenix has joined #buildstream | 10:03 | |
| *** narispo has joined #buildstream | 10:09 | |
| jjardon | juergbi: not sure that is the plan, only sometime I setup on a Sunday morning :) | 10:22 |
| jjardon | we can decrease the concurrency of the permanent runners if you think that can help | 10:23 |
| *** lachlan has quit IRC | 10:25 | |
| juergbi | jjardon: maybe, or at least running the jobs in separate systemd slices/scopes (and thus, cgroups), if that's not already the case | 10:26 |
| juergbi | don't know whether that's easy to configure | 10:26 |
| juergbi | btw, if autoscale is not crucial: hetzner now offers dedicated 32-core servers with 128 GB RAM and 1 TB NVMe for USD 140 a month | 10:31 |
| jjardon | juergbi: jobs run in separate docker containers, so I think they run in different slices | 10:35 |
| jjardon | not sure if some further tweak is possible | 10:35 |
| juergbi | right, should be the case, assuming docker properly integrates with systemd | 10:36 |
| juergbi | it might indeed be an I/O issue and I/O might not be limited in the cgroups | 10:36 |
| jjardon | juergbi: nice, if time I can setup those over the weekend; let me decrease the concurrency in the meantime | 10:37 |
| *** lachlan has joined #buildstream | 10:39 | |
| *** lachlan has quit IRC | 10:42 | |
| *** phoenix has quit IRC | 10:44 | |
| *** tristan has joined #buildstream | 10:46 | |
| *** ChanServ sets mode: +o tristan | 10:48 | |
| tristan | juergbi, yes that specific patch was plugin specific, I just suspect we'll have similar issues with project.conf | 10:48 |
| tristan | anyway, would be quite restrictive if we had to say "Ok, now the default project.conf configuration will never change !" | 10:49 |
| juergbi | tristan: if it's really minimal, it might not be completely unrealistic. but yes, I agree, we should have a mechanism in case we do need to change something | 10:51 |
| juergbi | but the only thing I can think of in this regard is feature flag, or different defaults for different format/min-version | 10:52 |
| *** lachlan has joined #buildstream | 10:56 | |
| gitlab-br-bot | BenjaminSchubert opened (was WIP) MR !1879 (bschubert/fix-mktemp-usage->master: tests: Stop using tmpdir_factory.mkdtemp("")) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1879 | 11:14 |
| robjh | did you remove the --max-jobs option? | 11:14 |
| traveltissues | not afaik, i used this recently | 11:14 |
| robjh | i had things in the wrong order | 11:15 |
| traveltissues | on 1.93.1+209.g505f77f5 | 11:15 |
| traveltissues | ah | 11:15 |
| tristan | juergbi, min-version doesnt live in the defaults (format-version did, because it was not a required field) | 11:16 |
| tristan | anyway | 11:16 |
| juergbi | tristan: what I meant was that theoretically min-version (or a similar field) could influence which set of defaults to use, but I don't think you want this either | 11:17 |
| tristan | juergbi, do we want to keep the default pathing in place ? I think it's convenient to be offered there | 11:17 |
| tristan | No I don't think it's appropriate, however if there is a versioning of defaults, that could be exposed separately | 11:18 |
| tristan | (and more clearly that it is a version of project.conf defaults) | 11:18 |
| tristan | Or alternatively, one could allow the user to override the whole file somehow ? | 11:18 |
| *** lachlan has quit IRC | 11:19 | |
| tristan | That could get sticky though with junction relationships | 11:19 |
| * tristan hasn't given enough thought into what the best solution would be, just there should be some solution | 11:19 | |
| juergbi | tristan: I would drop the default path variables. my reasoning is that if you want a convenient starting point instead of defining everything yourself, you want to use something like fd-sdk as a base anyway. and if you want to define everything yourself, you really want to be in control of the complete filesystem hierarchy | 11:20 |
| juergbi | i.e., for these two scenarios I don't see much of a practical difference | 11:21 |
| juergbi | but I might be missing something | 11:21 |
| juergbi | it might not be ideal for the 'hello world'-like examples where the example simply points to a bootstrap tarball and the rest is all in the example project | 11:23 |
| juergbi | although, 'hello world' would need very few variables, so maybe it would actually be good to define them in the hello world project itself | 11:24 |
| tristan | ok well, if we're going to do that, I think we need to prepare ground work | 11:30 |
| tristan | for instance (sec...) | 11:30 |
| juergbi | tristan: if we go for project-configurable cache key versions (to account for bug fixes in cache key algorithm etc.), maybe we could generalize that to also cover other core changes, and then support version conditionals in our default configs | 11:30 |
| juergbi | (should be kept to a minimum, ideally never used) | 11:31 |
| gitlab-br-bot | cs-shadow approved MR !1879 (bschubert/fix-mktemp-usage->master: tests: Stop using tmpdir_factory.mkdtemp("")) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1879 | 11:31 |
| tristan | juergbi, I think we need to backport !1851 to bst-1, which is a technically breaking change (but shouldn't have bad side effects considering what it fixes), and roll out a bst 1.6.x series | 11:31 |
| gitlab-br-bot | MR !1851: Process options in includes files with the options of their junction https://gitlab.com/BuildStream/buildstream/-/merge_requests/1851 | 11:31 |
| paulsherwood | juergbi: i feel like i've waited for this conversation for 2 years | 11:32 |
| *** lachlan has joined #buildstream | 11:32 | |
| tristan | juergbi, in other words, I'd like to have fdsdk/g-b-m actually implement something sensible at the time of a 2.0 release | 11:32 |
| gitlab-br-bot | BenjaminSchubert opened (was WIP) MR !1880 (bschubert/no-warnings->master: Fix some warnings happening during the tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1880 | 11:32 |
| tristan | that it a lot of churn for some projects to figure out a nice layout of public includes for their downstream/junctioning projects | 11:33 |
| gitlab-br-bot | marge-bot123 closed issue #1290 (Some tests break with pytest 5.4.1) on buildstream https://gitlab.com/BuildStream/buildstream/-/issues/1290 | 11:33 |
| gitlab-br-bot | marge-bot123 merged MR !1879 (bschubert/fix-mktemp-usage->master: tests: Stop using tmpdir_factory.mkdtemp("")) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1879 | 11:33 |
| tristan | s/that it/that is/ | 11:34 |
| *** lachlan has quit IRC | 11:35 | |
| juergbi | tristan: that could make sense | 11:38 |
| juergbi | should check with the fd-sdk maintainers but my guess is they would be happy about it | 11:39 |
| abderrahim[m] | tristan: I'd very much like it, and I have another couple changes I'd like to have a 1.6 release for | 11:40 |
| * abderrahim[m] is currently working on using fd-sdk includes in g-b-m | 11:40 | |
| abderrahim[m] | regarding includes, I have one more concern | 11:42 |
| juergbi | 1.6 would also get the min-version detection from the start, of course | 11:42 |
| tristan | yup | 11:42 |
| abderrahim[m] | for me 1.6 would be the occasion to add some warnings too | 11:43 |
| tristan | I'm thinking 1.6 is the segway into 2.x land, we should add things which make it easier to lay out ground work to ease later transition | 11:43 |
| abderrahim[m] | for things that would break in 2.x | 11:43 |
| tristan | good point | 11:43 |
| abderrahim[m] | so shall we branch for 1.4 now? | 11:44 |
| tristan | I think it makes sense to | 11:48 |
| tristan | juergbi, ^^ ? | 11:49 |
| juergbi | no objections | 11:49 |
| tristan | that frees up bst-1 branch for landing things like this towards 1.6 | 11:49 |
| juergbi | we could also wait until the first MR towards 1.6 is ready | 11:49 |
| tristan | and bst-1.4 "just incase" we need hotfixes | 11:49 |
| tristan | Right | 11:50 |
| juergbi | but I don't expect lots of bst-1 MRs between now and when that's the case, so it might not matter | 11:50 |
| tristan | yeah, I think branching it is a better signal for some contributors who might want to work on transitional support in 1.6 | 11:51 |
| gitlab-br-bot | cs-shadow approved MR !1880 (bschubert/no-warnings->master: Fix some warnings happening during the tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1880 | 11:53 |
| cs-shadow | hi, I have a couple of patches for the website repo in case someone has spare cycles to review (and potentially merge, I don't have privs for that): | 11:56 |
| cs-shadow | - https://gitlab.com/BuildStream/website/-/merge_requests/133 | 11:56 |
| cs-shadow | - https://gitlab.com/BuildStream/website/-/merge_requests/134 | 11:56 |
| tristan | nice | 11:57 |
| * tristan merges the logo move since that seems a no brainer... | 11:58 | |
| tristan | s/logo/mascot | 11:59 |
| tristan | cs-shadow, I've been giving some background cycles to thoughts about this venv approach btw | 12:00 |
| cs-shadow | tristan: I just created this as a very basic guide for how to use venvs. Is there anything in particular you'd like to see? | 12:01 |
| tristan | cs-shadow, one thing I was thinking is... do we want to (A) Rename default buildstream.conf file path location to buildstream2.conf ? ... or (B) Do we want to support an environment variable (our first user/distro facing one ?) to set a default location for the buildstream.conf file ? | 12:01 |
| tristan | I was thinking, if (B)... then I was hoping that entering a venv could automagically setup your conf file to somewhere specific to your venv or such | 12:02 |
| juergbi | hm, I'd like to keep user configuration out of the venv by default | 12:03 |
| juergbi | I want to be able to delete it and start over without losing my config | 12:03 |
| tristan | Or, maybe in the venv we can just override the correct XDG path for configuration ? | 12:03 |
| juergbi | but it's a good point | 12:04 |
| tristan | There will be a period where people will work on the same project with both bst-1 and bst-2, I think that's inevitable | 12:04 |
| tristan | same directory, checkout old stable branch and work with bst-1... now checkout master of fdsdk and work with bst-2, for example | 12:05 |
| juergbi | buildstream2.conf wouldn't be the worst idea | 12:05 |
| juergbi | I think right now my config works with both branches. but that's definitely not guaranteed | 12:05 |
| tristan | yeah and even then, I suspect it depends what you have in your config | 12:06 |
| cs-shadow | i'm leaning towards recommending people to set `XDG_CACHE_something` during the transition period to point to their respective config files | 12:06 |
| juergbi | what about first looking for buildstream2.conf and if that doesn't exist, go for buildstream.conf | 12:07 |
| juergbi | same for buildstream1.conf on bst 1.6 | 12:07 |
| tristan | right, I would like it though, that we don't just tell people "set XDG_FOO_BAR when entering your bst-2 venv so that you have the right config" | 12:07 |
| tristan | I'd rather say "start the venv with this command, and it will now relocate everything to this place" | 12:07 |
| tristan | juergbi, I like that | 12:08 |
| tristan | that lets you nuke your venvs as you like without losing anything precious | 12:08 |
| tristan | for that matter, what about source cache directories and local CAS ? | 12:09 |
| juergbi | we also need to keep cache in mind, though. our cache directory structure is not 100% the same. on the other hand, some parts are actually sharable | 12:09 |
| juergbi | :) | 12:09 |
| cs-shadow | just to note about venv features - by default there's no mechanism to inject our own environment variables or code, so we'll have to have this logic in BuildStream itself. So, I'm not hugely in favor of this approach. | 12:09 |
| cs-shadow | I agree that it's good to keep venvs disposable | 12:09 |
| tristan | cs-shadow, that makes me sad :'( | 12:10 |
| tristan | heh | 12:10 |
| tristan | cs-shadow, you'd think you might be able to say "venv --setup=buildstream.setup" | 12:10 |
| tristan | and then we'd just create a nice setup file for people to use :) | 12:11 |
| tristan | Ok well, I think that none of this discussion is really blocking https://gitlab.com/BuildStream/website/-/merge_requests/133 | 12:11 |
| tristan | thoughts ? shall we just merge that docs thing right now and keep improving things separately ? | 12:11 |
| * tristan thinks so... | 12:12 | |
| cs-shadow | I'd agree. I do think we can sort out the namespacing of config files and caches before 2.0 though | 12:12 |
| cs-shadow | s/can/should/ | 12:12 |
| tristan | yeah, config file thing we can do very quickly... simple patch with docs included can be easily backported for bst 1.6 | 12:14 |
| tristan | I think juergbi's idea of buildstream2.conf in priority and then look for buildstream.conf is a good one | 12:14 |
| abderrahim[m] | I also like the idea of buildstream1.conf and buildstream2.conf with a fallback to buildstream.conf | 12:14 |
| juergbi | tristan: I forgot about bst shell --sysroot. it is the last feature (afaik) that doesn't work yet with buildbox-run. it's not quite straight forward to implement as buildbox-run is CAS-oriented. so I'm wondering whether we even still have a use case for it now that we store (failed) buildtrees in CAS | 12:19 |
| tristan | Ahhh that | 12:24 |
| tristan | Interesting | 12:24 |
| tristan | juergbi, So from a user perspective working on a project, when they encounter a build error they are given an option to drop into a functional shell of that build right ? | 12:25 |
| tristan | And it is also equally easy to exit the session and debug a failed build afterwards ? | 12:25 |
| juergbi | yes, exactly, that continue to works with buildtrees | 12:26 |
| tristan | juergbi, the only other use case I could imagine is wanting to do a `bst shell --sysroot <the directory I just checked it out to>` | 12:26 |
| juergbi | *continues to work *sigh* | 12:26 |
| juergbi | i.e., a glorified user chroot | 12:26 |
| tristan | which I think didnt really work before either | 12:26 |
| tristan | yes | 12:26 |
| tristan | juergbi, I think we don't need --sysroot, and it's nice that we don't expose raw paths to host directories to the user anymore anyway | 12:27 |
| tristan | "Here's your failed build directory, go clean it up yourself later" | 12:27 |
| juergbi | yep | 12:28 |
| juergbi | immediately after checkout bst shell --sysroot would be identical to just running bst shell. so I don't see a reason for it | 12:28 |
| tristan | juergbi, on a similar note about removing useless cruft which has only been a thorn in our side... what about source-bundle ? | 12:28 |
| juergbi | the only advantage of bst shell --sysroot would be that you could manipulate it (persistently) | 12:28 |
| juergbi | tristan: source-bundle has already been removed | 12:29 |
| tristan | nice | 12:29 |
| juergbi | we still have support for '--include-build-scripts', though | 12:29 |
| juergbi | which is the replacement | 12:29 |
| tristan | I'm pretty sure I ... oooohhhhh | 12:29 |
| tristan | That's what I noticed in the code thinking "oh damn, this annoying stuff still exists ?" | 12:29 |
| tristan | oh well | 12:30 |
| juergbi | we could recheck with users whether it's still needed. I don't know | 12:30 |
| tristan | there must have been some reason for keeping that, someone must have cared a great deal | 12:30 |
| tristan | if that's the case we can leave it alone | 12:30 |
| juergbi | would be nice to get rid of generate_script() methods | 12:30 |
| tristan | exactly | 12:30 |
| tristan | weird dual standard anyway | 12:30 |
| juergbi | it might actually still be possible when we review the API with regards to batching | 12:31 |
| juergbi | the generated script for batching is somewhat similar | 12:31 |
| juergbi | maybe we can unify it | 12:31 |
| tristan | That would be very nice | 12:31 |
| juergbi | yes, so let's revisit this as part of that review | 12:32 |
| tristan | now would be a great time to already have an issue to note this down on ;-) | 12:34 |
| cs-shadow | > there must have been some reason for keeping that, someone must have cared a great deal | 12:42 |
| cs-shadow | tristan: I remember you being this certain someone when I proposed to remove it initially :D | 12:42 |
| cs-shadow | I think the rationale was that it allowed bootstrapping for a new architecture or something like that | 12:43 |
| tristan | cs-shadow, yes I was that guy for a while | 12:43 |
| tristan | I think later it got out of control and became something about "being able to run BuildStream with no sandbox" | 12:44 |
| tristan | The bootstrapping case is a good one to keep in sight though now that you raise it | 12:45 |
| juergbi | well, with remote execution / buildbox-run there are now alternatives | 12:45 |
| juergbi | a single bst instance can orchestrate builds across architectures | 12:46 |
| * cs-shadow wonders how many people are using it for real | 12:46 | |
| tristan | juergbi, I think the idea was for the cases you really didnt have anything on that arch yet | 12:46 |
| juergbi | or alternatively, build e.g. x86-64 part on one system, push it to artifact server. then build the ARM part on another system, pulling the x86-64 artifacts from the server | 12:47 |
| tristan | but I think nowadays that starts with qemu | 12:47 |
| juergbi | ah, ok | 12:47 |
| tristan | like if you can get as much as a busybox shell and a kernel running on a new arch, you can run the build scripts to build a bigger runtime from there and debug things as they arise | 12:48 |
| tristan | rather than lots of trial and error | 12:48 |
| * tristan is not sure that it's justified to keep such a big wart in the codebase to cover that feature though | 12:49 | |
| tristan | there must be better ways | 12:49 |
| juergbi | I concur | 12:49 |
| juergbi | traditional cross compilation covers this in principle | 12:49 |
| juergbi | as do qemu-user-based solutions | 12:50 |
| juergbi | if that's the only use case, I wouldn't keep it | 12:50 |
| juergbi | (unless we can get it practically free with batch-related API changes) | 12:51 |
| tristan | nod | 12:51 |
| tristan | if we can get it very literally the same with batch related API, there can be other values, for instance just providing concise reports of how a thing was built | 12:52 |
| tristan | automatically generating a website like lfs from your buildstream project... | 12:52 |
| juergbi | hehe | 12:53 |
| WSalmon | ubuntu installs git-lfs diffrently to the other os's so i could do with adding a tiny bit of ubuntu spesific config to the testing docker image, how do i do this without upseting the templating stuff? cs-shadow benschubert ? | 13:02 |
| gitlab-br-bot | tristanvb opened (was WIP) MR !1881 (tristan/min-version->master: Replace format-version with min-version) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1881 | 13:04 |
| tristan | juergbi, wanna take a look at that one ? there may be some questionable aspects in there | 13:09 |
| tristan | The bulk of it is: _project.py now understands min-version... and cli.py/app.py is changed for `bst init` | 13:11 |
| tristan | I might want to remove the check for "format-version" in _project.py, seems sufficient to only check for absence of "min-version" | 13:12 |
| tristan | Also, I have some scattered statements like this one: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1881/diffs#5de28618ea7bb36c58e2b7dda21be3f35ffc64bd_582_600 | 13:14 |
| tristan | For version checks and creating projects, we want to pretend that we are 2.0 until we really are, and then remove those code blocks after releasing 2.0 | 13:14 |
| * tristan considered doing this directly in utils.get_bst_version() but thought that might have unintended consequences | 13:15 | |
| cs-shadow | WSalmon: where's this needed? (none of the docker images currently have lfs anyway) | 13:15 |
| *** phoenix has joined #buildstream | 13:17 | |
| tristan | the rest of the patch is of course lots of tedious updating of project.conf files and ensuring tests pass | 13:17 |
| WSalmon | i have a branch that adds gitlfs to them all and the packages create a /etc/gitconfig or equivilant but not ubuntu if i add a /etc/gitconfig with the magic lines then it passes in ubuntu | 13:17 |
| WSalmon | i think ubuntu is trying to do something cleaver by having the gitlfs config in the users ~/.gitconfig | 13:18 |
| WSalmon | but it dosent quite work in docker | 13:18 |
| cs-shadow | WSalmon: I think it might be best to defer the setup of git lfs to the projects that need it, like plugins-experimental | 13:19 |
| cs-shadow | adding requirements for _all_ plugins to buildstream-docker-images will result in bloated images IMO | 13:19 |
| WSalmon | we install a load of plugin spesific things atm, ostree for instance.. we have a hole `# plugins dependencies` section | 13:22 |
| *** lachlan has joined #buildstream | 13:23 | |
| WSalmon | i gues we could just seperat docker images create for bst-plugins- | 13:23 |
| cs-shadow | all those dependencies were dependencies of core plugins at the time they were added, it was never the intention to support requirements for external plugins in the same images | 13:24 |
| cs-shadow | yeah, either a separate image or if it's lightwerght, might as well do it on the fly | 13:25 |
| tristan | cs-shadow, how do we intend to continue testing these plugins outside of core ? | 13:25 |
| tristan | I'm unsure what the use cases are of lightweight images too, as I recall we have images meant for "bst-here", and images meant for testing | 13:26 |
| tristan | consumers of bst-here will undoubtedly have different expectations and needs, which doesnt seem to fit very well with a desire to keep them lightweight | 13:27 |
| *** lachlan has quit IRC | 13:27 | |
| *** phoenix has quit IRC | 13:29 | |
| cs-shadow | bst-here and teststuite images are completely separate, that's not an issue. the way I see it the testsuite images should come with all system requirements to build and run BuildStream. Nothing prevents plugin repositories to install more stuff on top as needed. | 13:29 |
| *** phoenix has joined #buildstream | 13:30 | |
| cs-shadow | for example, when I'm testing container plugins, I don't need to install ostree etc and when I'm testing ostree, i don't need to install docker | 13:30 |
| tristan | Errrm... except that distros don't usually allow reproducible installations of exactly a specific version of a package | 13:31 |
| WSalmon | yep but as they all share the same runners then having the same image is going to be a advantage | 13:31 |
| tristan | So when you run a testsuite, you don't have absolute certainty that it broke because of your code, and might wonder if a system dependency minor change introduced breakage instead | 13:31 |
| juergbi | tristan: will take a look at min-version | 13:32 |
| cs-shadow | tristan: hows' that going to change if they are pre-installed in the image? they are built nightly anyway | 13:32 |
| tristan | (which was kind of the point of having preinstalled stuff in dockers instead of ever running `apt-get install` in gitlab-ci.yml`) | 13:32 |
| tristan | cs-shadow, Ok so you mean, external plugins can install more stuff on top as needed: In their own separate docker containers ? | 13:33 |
| tristan | cs-shadow, I didn't realize we were doing that :) | 13:33 |
| tristan | juergbi, I'm done for the day... so I'll check comments in the morning :) | 13:33 |
| juergbi | btw: freedesktop-sdk build time is almost identical with the classic bwrap backend and buildbox-run+fuse right now (144min vs. 142min). I expect that to improve with further buildbox optimizations, though | 13:34 |
| WSalmon | tristan, we dont, i was asking if we are going to need to do that | 13:34 |
| cs-shadow | to me, this seems like the whole blessed plugins conversation again. Like which plugins | 13:35 |
| cs-shadow | *Like which plugins' dependencies we install and which we don't | 13:35 |
| cs-shadow | "all of them" isn't an answer because then we'll end of enormously sized images that'll take ages to pull | 13:35 |
| tristan | all of them isn't an answer because of incompatibilities across systems as I understand it (e.g.: cant test win32 plugins on linux images) | 13:36 |
| *** phildawson-ct has quit IRC | 13:36 | |
| tristan | I'm not sure how enormous these images are, but if they are like 500MB today, and all of them = 700MB, sounds like a useful thing to have | 13:37 |
| tristan | cs-shadow, I think what I'm considering most here is: Where does one want to draw the line in terms of maintenance burden ? | 13:37 |
| cs-shadow | also, incompatible dependencies (two plugins can depend on conflicting packages for instance) | 13:38 |
| tristan | if separating all the things neatly into separate images for all of the plugin repos we maintain is not a huge overhead and does not cause friction in testing, then surely we can keep everything neatly separated | 13:38 |
| cs-shadow | tristan: yes, previously we used to blindly publish docker images without any testing that they would actually pass BuildStream tests. | 13:39 |
| cs-shadow | Now every buildstream-docker-images runs BuildStream tests to ensure that when we push, we are sure that the tests won't fail due to issues in the image | 13:39 |
| tristan | at least, this aspect concerns me more than the size of the images :-S | 13:39 |
| cs-shadow | so, if we add any more dependencies, we must test them as well | 13:39 |
| cs-shadow | and the pipelines on docker-images already take hours so I'm wary of adding more stuff | 13:40 |
| tristan | While that testing is interesting, in any case we'll need to have an explicit commit to .gitlab-ci.yml on the BuildStream side (or plugin repo side) to consume the precise new image sha256 | 13:40 |
| cs-shadow | that's already the case in all repos we maintain. This is to guard against breakages while upgrading | 13:41 |
| tristan | right, otherwise you discover the breakage a bit later and have to go fix again | 13:41 |
| cs-shadow | exactly and you realize that universe has moved on since then so it can be come harder :) | 13:42 |
| tristan | About conflicting dependencies, I think that is a very pythonic thing, and I wonder if it's realistic that that would ever happen with the plugins we maintain (separate or not) | 13:43 |
| tristan | We probably dont want stories where our blessed plugin repos have conflicting dependencies, as we'd expect users to be able to safely combine them | 13:44 |
| cs-shadow | I've definitely come across conflicting dependencies in some cases, when there's a "tool" and a "tool-next-generation" and you can only have one of them at a time | 13:47 |
| cs-shadow | but I'd agree that we should avoid it in our blessed plugin repositories | 13:47 |
| juergbi | benschubert, tlater[m]: I just noticed that !1787 broke the generated man pages, as in: The dependencies to checkout [default: _PipelineSelection.RUN] | 13:48 |
| gitlab-br-bot | MR !1787: Make PipelineSelection a proper enum type https://gitlab.com/BuildStream/buildstream/-/merge_requests/1787 | 13:48 |
| tristan | cs-shadow, I'm just trying to reason about what's the most practical thing for *us* to maintain with least effort | 13:48 |
| *** lachlan has joined #buildstream | 13:48 | |
| tristan | cs-shadow, I suppose we'd want stories for third party plugin authors to maintain their own docker images, but for ours; there may even be value in sharing an image (i.e. ensuring that there are no such collisions) | 13:49 |
| benschubert | juergbi: ouch, any way I can reproduce locally? | 13:49 |
| juergbi | tox -e man | 13:49 |
| juergbi | at least I assume it was that MR, haven't verified yet | 13:51 |
| benschubert | ok, are you having a look or do you want me to have a go at it? | 13:51 |
| cs-shadow | tristan: for user-facing image, https://hub.docker.com/r/buildstream/buildstream comes in various variants. So, we can add our plugins to the "extra" variants and external plugins can derive from the one the find most sensible. | 13:52 |
| cs-shadow | for testsuite images, I'm thinking installing dependencies on the fly may be the simplest, given that they will usually be small and domain-oriented | 13:52 |
| tristan | cs-shadow, anyway I doubt we're getting to the bottom of it now, but as I discussed with juergbi earlier on in the day, we probably want to prioritize smoothing out the plugin story in order to optimize parallelism of work towards 2.0 | 13:52 |
| juergbi | benschubert: I'm not looking into it right now and you probably know more about these enums | 13:52 |
| benschubert | ok I'll take a look | 13:53 |
| cs-shadow | tristan: agreed, I can try starting a conversation about the actual breakdown of plugins | 13:53 |
| juergbi | ta | 13:53 |
| gitlab-br-bot | marge-bot123 merged MR !1880 (bschubert/no-warnings->master: Fix some warnings happening during the tests) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1880 | 13:54 |
| tristan | cs-shadow, right, maybe I'm too perfectionist but I really don't like having anything modify the docker environment in a .gitlab-ci.yml on the fly, as I cannot trust it to be deterministic | 13:54 |
| tristan | as I recall, we trust it for python dependencies only, because we install specific versions | 13:54 |
| tristan | cs-shadow, regarding conversations... that one probably has to happen (actual breakdown), but I was more thinking of all of the pending work that touches plugins... changes to plugin API, and maybe a review of buildstream.testing and how consumable/practical that whole story is | 13:58 |
| tristan | the more we can get done before plugins actually migrate on that front, the less hassle we have updating stuff in various repos | 13:59 |
| *** lachlan has quit IRC | 13:59 | |
| cs-shadow | I was actually thinking that we might discover real issues when we do start that process | 13:59 |
| cs-shadow | (and fix them as we go along) | 14:00 |
| benschubert | cs-shadow: we already have some issues that we can fix before starting more | 14:00 |
| benschubert | what I think we are lacking there is a more coordinated effort, instead of working each on a bit without looking at the bigger move :) and doing it slowly but surely might be the easiest road path. As the swiss president said last week about the lockdown: "As fast as possible, as slow as necessary" | 14:01 |
| gitlab-br-bot | juergbi opened MR !1882 (juerg/shell-sysroot->master: Remove bst shell --sysroot) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1882 | 14:02 |
| * tristan likes napoleon's "don't hurry, we have no time to lose !" | 14:03 | |
| benschubert | juergbi: on latest master, runnign `tox -e man` works ? | 14:04 |
| * cs-shadow will roll his own "as fast as possible, but not any faster" | 14:04 | |
| juergbi | benschubert: and the diff looks fine? no _PipelineSelection in there? | 14:04 |
| benschubert | oh I se | 14:05 |
| juergbi | as I've mentioned to tristan earlier today, a gitlab board might be useful for 2.0 | 14:05 |
| juergbi | or at least I would give it a try | 14:06 |
| benschubert | that seems good, we should also cleanup the milestone :) | 14:06 |
| juergbi | and make sure everything from the mail is in there | 14:06 |
| robjh | i have a build that fails on make, but it doesnt look like its actually trying to do anything. whats more odd, is on a run where it did look like it was doing something, it got killed with no other errors, traveltissues, you were seeing this before, right? | 14:16 |
| juergbi | can you put the exact console messages on a pastebin? | 14:18 |
| traveltissues | robjh: i needed to install libgirepository1.0-dev | 14:20 |
| benschubert | juergbi: do we commit the updated manpages afterwards? | 14:21 |
| robjh | traveltissues, i'll try that in a moment | 14:21 |
| robjh | juergbi, https://mcr.robjh.com/tmp/bst.log | 14:21 |
| juergbi | benschubert: yes, I don't remember the reason why don't just do this in CI but we periodically update the man pages | 14:21 |
| juergbi | robjh: hm, no other output, that's indeed odd | 14:22 |
| juergbi | I'd expect an error output from make | 14:22 |
| robjh | i know! right | 14:22 |
| WSalmon | juergbi, https://gitlab.com/BuildStream/buildstream/-/boards/1693466 | 14:23 |
| WSalmon | like that? | 14:23 |
| gitlab-br-bot | BenjaminSchubert opened MR !1883 (bschubert/fix-manpages->master: types.py: Add a __str__ to PipelineSelection to fix man pages) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1883 | 14:23 |
| benschubert | juergbi: ^ here you go | 14:23 |
| juergbi | robjh: do you see anything else with this command? bst artifact log freedesktop-sdk.bst:bootstrap/build/gcc-stage1.bst | 14:24 |
| juergbi | or alternatively, by manually checking ~/.cache/buildstream/logs | 14:24 |
| juergbi | WSalmon: yep | 14:24 |
| juergbi | benschubert: ta | 14:24 |
| benschubert | and it was indeed this mr that broke it | 14:25 |
| gitlab-br-bot | juergbi approved MR !1883 (bschubert/fix-manpages->master: types.py: Add a __str__ to PipelineSelection to fix man pages) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1883 | 14:25 |
| robjh | juergbi, yes, that gives output similar to the first time it failed to build (before it started failing with no output) | 14:25 |
| juergbi | robjh: ah, that wasn't the first time. right, was already marked as 'failed' in the first pipeline. bst caches failures. it should output the same error log as the first time but that sounds like a bug | 14:26 |
| juergbi | robjh: to retry, explicitly delete the cached result with: bst artifact delete freedesktop-sdk.bst:bootstrap/build/gcc-stage1.bst | 14:27 |
| robjh | okay, thanks. | 14:27 |
| juergbi | we should probably be more explicit about this | 14:27 |
| juergbi | and provide a hint how to retry | 14:27 |
| robjh | the first thing i tried was passing --retry or --force | 14:27 |
| cs-shadow | juergbi: maybe we should print the same user prompt as well as we do when build fails the first time around | 14:28 |
| juergbi | in interactive mode, we actually should provide interactive (r)etry option | 14:28 |
| *** qd39bf has joined #buildstream | 14:28 | |
| *** qd39bf has left #buildstream | 14:28 | |
| juergbi | exactly, do we intentionally not do this right now? | 14:28 |
| cs-shadow | i don't think it's intentional; it's probably that we are utilizing the same codepaths for successfull cached builds | 14:29 |
| robjh | i suspect it failed because it ran out of ram or something | 14:30 |
| traveltissues | possible, i needed to bump my vm up to finish the build | 14:30 |
| robjh | ie, something that can be fixed without changing any of bst's inputs | 14:31 |
| juergbi | right, unfortunately, we can't always detect this. so we should make it clear to the user what they can do | 14:31 |
| robjh | bst 1.4.1 didnt cache failures, right? why the change? | 14:32 |
| juergbi | caching failures can be very useful because we also cache the failed buildtree by default. this allows entering a shell with the failed buildtree to debug | 14:33 |
| juergbi | helpful if you want to debug webkit build failure after 1h or so | 14:34 |
| robjh | i seee | 14:36 |
| robjh | thanks for letting me know | 14:36 |
| valentind | juergbi, Looking at your patch. It would maybe make sense to have a temporary file thing in the sandbox. | 14:38 |
| juergbi | valentind: I was considering this but wasn't sure what the best API would be | 14:40 |
| juergbi | the oci case is also relatively special as you don't know the filename before writing the file | 14:40 |
| juergbi | most can simply do dir.open_file("final-filename", mode="w") | 14:40 |
| juergbi | and internally this will already use a temporary file | 14:40 |
| juergbi | (for atomic replace, even though that's not too relevant in this context) | 14:41 |
| juergbi | we're planning to do a review of the plugin API incl. the Directory API before 2.0, maybe streamline a few things | 14:42 |
| juergbi | do you think it's important to add a tempfile API before that? | 14:42 |
| *** phoenix has quit IRC | 14:44 | |
| juergbi | valentind: have you already had a chance to build it and try it out? | 14:46 |
| valentind | I am building it now. | 14:46 |
| valentind | But I am looking at the patch while it is building. | 14:47 |
| valentind | I do not find the definition of Directory.remove() | 14:48 |
| valentind | juergbi, Does that need a branch of buildstream? | 14:48 |
| juergbi | yes, also juerg/vdirectory | 14:48 |
| juergbi | !1878 | 14:48 |
| gitlab-br-bot | MR !1878: storage: Improve Directory API https://gitlab.com/BuildStream/buildstream/-/merge_requests/1878 | 14:49 |
| valentind | Wanted to make sure that recursive deletion of a symlink to a directory did not remove the directory recursively. | 14:51 |
| valentind | juergbi, Maybe that should be documented. | 14:51 |
| valentind | (and tested). | 14:52 |
| juergbi | yes, will expand that. the default behavior of the Directory API is to never follow symlinks | 14:52 |
| valentind | I think the only change of behavior is for ".wh..wh..opq". But I am actually not sure what the correct behavior is. | 14:53 |
| juergbi | what difference did you spot? | 14:53 |
| juergbi | I tried to replicate the logic | 14:53 |
| valentind | I think of the case were a layer has "foo/bar" and "baz -> foo". And then second layer "baz/.wh..wh..opq". | 14:53 |
| valentind | But I think your code makes more sense anyway. | 14:54 |
| valentind | It will remove the symlink and create a new directory not touching the previous directory. | 14:54 |
| valentind | But I think it is probably invalid layers. | 14:54 |
| valentind | So probably not important. | 14:54 |
| juergbi | ah ok | 14:56 |
| valentind | Line 766 | 14:57 |
| valentind | if not layerdir.exists(f): | 14:57 |
| valentind | I think it was parentdir | 14:57 |
| valentind | Maybe it was wrong already... | 14:57 |
| juergbi | that's in a loop iterating within the parent dir | 14:57 |
| juergbi | so parentdir.exists(f) should always be True | 14:58 |
| valentind | ah no it is correct | 14:58 |
| valentind | Sorry. | 14:58 |
| valentind | Yes, if it does not exist in the layer dir, then we have to white it out. | 14:58 |
| juergbi | yep | 14:58 |
| juergbi | do you know how much of that code is exercised by the fd-sdk oci images? | 14:58 |
| juergbi | i.e., I hope we have at least basic coverage | 14:59 |
| valentind | juergbi, I hope export_to_tar does not prepend all paths with ./ | 15:01 |
| valentind | Because that does not work well with podman. | 15:01 |
| juergbi | no, I think it doesn't | 15:01 |
| juergbi | the string argument is the prefix | 15:01 |
| valentind | By the way, it would be nice to make "checkout --tar" not do that. | 15:01 |
| juergbi | ah, right, there we pass "." | 15:02 |
| juergbi | I wonder why | 15:02 |
| valentind | Maybe to be "compatible" with bst1.4. | 15:02 |
| valentind | I do not know. | 15:02 |
| valentind | OK, I think I did not see any mistake. | 15:03 |
| valentind | I still have to test it. | 15:03 |
| juergbi | thanks for the review | 15:03 |
| valentind | Thank you for importing the virtual directories. | 15:03 |
| juergbi | yw | 15:04 |
| juergbi | fd-sdk repo also has a few plugins, which I didn't port | 15:04 |
| valentind | So now it will be actually usable for simple plugins that do not really need to execute things remotely. | 15:04 |
| juergbi | however, I think they are pretty easy | 15:04 |
| juergbi | yes, and in some cases it might be a lot faster with the switch to buildbox-run as there all the directory operations are done in memory | 15:05 |
| juergbi | no actual staging | 15:05 |
| valentind | I think they are not very big. If you can handle oci, then the rest should be do-able. | 15:05 |
| valentind | git lfs does not work. | 15:06 |
| juergbi | that's not related to the virtual directory API, is it? | 15:06 |
| valentind | I cannot use "use-lfs: false" on components/icu.bst. | 15:06 |
| valentind | But then it fails. | 15:06 |
| valentind | It is not related. | 15:07 |
| juergbi | ok, I think WSalmon is looking into git lfs on bst2 | 15:07 |
| *** devcurmudgeon has joined #buildstream | 15:07 | |
| valentind | I will just try to build the docker/oci images without icu. | 15:08 |
| juergbi | in your bst2 branch you commented it out | 15:09 |
| juergbi | and I didn't notice any build issue with `make build` and the oci images | 15:09 |
| valentind | Yes, because that does not work. But then it still fails. | 15:10 |
| valentind | Either way it fails. | 15:10 |
| valentind | Might be my local installation then. | 15:10 |
| juergbi | the build didn't fail, though | 15:10 |
| juergbi | but I haven't actually tested the result | 15:10 |
| WSalmon | valentind, this needs a little bit of tidying up and working out how to do the dependancys for testing but seems to work just fine https://gitlab.com/BuildStream/bst-plugins-experimental/-/tree/willsalmon/git_tag_gitlfs | 15:11 |
| valentind | Wow, fetch is so slow. | 15:14 |
| valentind | It takes several minutes to apply a patch. And it freezes my computers because of all the IO. | 15:14 |
| *** lachlan has joined #buildstream | 15:14 | |
| juergbi | yes, we definitely need to optimize fetch | 15:15 |
| valentind | I do not think applying a patch on a nvme disk should do that. | 15:15 |
| juergbi | patch might be the worst case right now | 15:15 |
| juergbi | well, not sure, have to recheck | 15:15 |
| juergbi | it's obviously not about applying the patch itself. it's about getting the sources into CAS | 15:16 |
| valentind | I wonder if "systemd-run --user --slice=buildstream.slice --shell" will help not taking up all the IO. | 15:16 |
| juergbi | it would be nice if reflink was pervasively available | 15:21 |
| juergbi | that allows some safe optimizations when importing into CAS | 15:22 |
| *** ca46 has joined #buildstream | 15:28 | |
| *** ca46 has left #buildstream | 15:28 | |
| *** lachlan has quit IRC | 15:32 | |
| *** lachlan has joined #buildstream | 15:55 | |
| jjardon | hi, I have a bst project with a lot of elements. I want to run a static code analyzer on all of them with a command line option; is there any way to do that? | 16:17 |
| jjardon | or what would be the cleanest? | 16:17 |
| tristan | I could imagine an element which has a runtime + static analyzer as dependencies to stage, and other dependencies on the elements you want to analyze... without ever staging those artifacts, you could iterate over all the elements you want to analyze and only stage the sources and run the analyzer ? | 16:21 |
| tristan | of course a caveat is that you can only analyze things which have already built in that approach | 16:22 |
| juergbi | yes, that's probably the only option as a bst element right now | 16:23 |
| juergbi | an alternative would be to use `bst source checkout --deps all stack.bst` | 16:23 |
| juergbi | but that would be with tooling outside bst | 16:23 |
| juergbi | tristan: besides the caveat that you mentioned, the element approach might actually fail because buildstream doesn't check whether sources are available if the artifact is already cached | 16:25 |
| juergbi | so we'd need some kind of special source dependency flag for this to work properly | 16:25 |
| *** lachlan has quit IRC | 16:26 | |
| *** lachlan has joined #buildstream | 16:42 | |
| *** lachlan has quit IRC | 16:48 | |
| *** hasebastian has joined #buildstream | 17:01 | |
| *** tpollard has quit IRC | 17:07 | |
| *** lachlan has joined #buildstream | 17:20 | |
| *** lachlan has quit IRC | 17:24 | |
| *** rdale has quit IRC | 17:25 | |
| *** lachlan has joined #buildstream | 17:49 | |
| *** santix has quit IRC | 17:55 | |
| *** cs-shadow has quit IRC | 17:59 | |
| *** lachlan has quit IRC | 18:01 | |
| gitlab-br-bot | marge-bot123 merged MR !1883 (bschubert/fix-manpages->master: types.py: Add a __str__ to PipelineSelection to fix man pages) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1883 | 18:23 |
| *** hasebastian has quit IRC | 18:43 | |
| valentind | juergbi, librsvg did not build. There is a missing fix in bst-plugins-experimental for the crate plugin. | 19:32 |
| juergbi | valentind: it built fine here but I didn't delete the sources directory | 19:35 |
| valentind | It should have failed at build I think. | 19:36 |
| valentind | The url set in the generate configuration file is wrong. | 19:37 |
| valentind | I am disabling librsvg. And I continue to build. | 19:37 |
| valentind | I should soon have a docker and oci images. | 19:37 |
| valentind | I will test it tommorrow morning. | 19:37 |
| juergbi | hm, odd. I can send you the build log if that helps with anything | 19:39 |
| juergbi | files in ~/.cache/buildstream/sources/cargo/crates_crates all seem to be from a couple of days ago, the time of my first test build with bst2 | 19:40 |
| *** narispo has quit IRC | 19:40 | |
| juergbi | so it shouldn't have been in the cache from bst1 | 19:40 |
| valentind | juergbi, It should be something that has never worked on bst2. | 19:40 |
| *** narispo has joined #buildstream | 19:41 | |
| valentind | ah... | 19:41 |
| valentind | juergbi, maybe I did not get the latest version of your branch. | 19:41 |
| valentind | Maybe you rebased. | 19:41 |
| valentind | Right... | 19:42 |
| juergbi | which commit did you use? | 19:42 |
| valentind | I did not do git fetch. | 19:42 |
| valentind | 1803c07446a64734bc1645ddbf7b0cb7c16c3fba | 19:43 |
| valentind | How do we force rebuild of an element? | 19:44 |
| valentind | artifact delete. | 19:44 |
| *** narispo has quit IRC | 19:44 | |
| juergbi | yep | 19:44 |
| juergbi | ah, that's really an old version | 19:45 |
| *** narispo has joined #buildstream | 19:45 | |
| juergbi | if it cached the wrong source, you may also have to delete ~/.cache/buildstream/sources/cargo/ | 19:46 |
| valentind | Still same error | 19:49 |
| valentind | The problem is the .cargo/config file is wrongly generated. | 19:49 |
| valentind | It should have been fixed by a705890e27ed223e05fe7f20a25277f6821baccf | 19:55 |
| valentind | Is there some source artifact cache? | 19:55 |
| valentind | I removed .cache/buildstream/source_protos/cargo | 19:59 |
| *** benschubert has quit IRC | 20:05 | |
| *** toscalix has joined #buildstream | 20:07 | |
| *** foobaz has joined #buildstream | 20:08 | |
| *** foobaz has left #buildstream | 20:08 | |
| *** rdale has joined #buildstream | 21:13 | |
| *** toscalix has quit IRC | 21:30 | |
| *** narispo has quit IRC | 22:20 | |
| *** narispo has joined #buildstream | 22:20 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!