| *** tristan has quit IRC | 00:52 | |
| *** jennis has joined #buildstream | 01:50 | |
| *** noisecell has joined #buildstream | 04:03 | |
| *** noisecell has quit IRC | 04:03 | |
| *** noisecell has joined #buildstream | 04:15 | |
| *** noisecell has quit IRC | 04:18 | |
| *** Prince781 has quit IRC | 04:34 | |
| *** bochecha_ has joined #buildstream | 05:00 | |
| *** bochecha_ has quit IRC | 05:09 | |
| *** bochecha_ has joined #buildstream | 05:25 | |
| *** Prince781 has joined #buildstream | 05:49 | |
| *** Prince781 has quit IRC | 06:54 | |
| *** Prince781 has joined #buildstream | 07:11 | |
| *** ernestask has joined #buildstream | 07:28 | |
| *** toscalix has joined #buildstream | 08:00 | |
| toscalix | valentind: I need your help to understand some details about how BuildStream work. Can we have a video chat this morning? It shouldn't be long | 08:08 |
|---|---|---|
| toscalix | well, some are not details | 08:08 |
| toscalix | I read the docu and the architechtural document but still.... | 08:08 |
| *** jonathanmaw has joined #buildstream | 08:15 | |
| valentind | toscalix, sure. | 08:26 |
| valentind | jonathanmaw, Sorry I did not answer yesterday. I was not working. | 08:27 |
| valentind | jonathanmaw, I think for the mirrored URI for plugins, we should let the plugin decide it. Maybe just have method Source.get_mirrored_uri(), and then we override it in some places. Would that help? | 08:40 |
| valentind | or more get_mirrored_path. | 08:41 |
| jonathanmaw | valentind: yep, that works for me | 08:41 |
| valentind | We could even make it a "generator" that does the counting. | 08:43 |
| valentind | So that we can have the {n} anywhere in the path. | 08:43 |
| jonathanmaw | ah, a generic function where sources can define their own format string? | 08:49 |
| valentind | Well, either it returns a format string, or it returns a generator that you can iterate. | 08:56 |
| valentind | Generator as in Python generator. | 08:56 |
| valentind | With "yield" | 08:57 |
| *** noisecell has joined #buildstream | 09:02 | |
| jmac | Today I'm getting inconsistent results from pylint - either no errors, or 4 import errors, on the same source | 09:12 |
| tlater | juergbi: There are ways to do that | 09:22 |
| tlater | I think it's as simple as including '# pylint: disable=all' at the top of the file or somesuch | 09:22 |
| * tlater isn't sure about the exact syntax, but our fuse file has the correct header | 09:23 | |
| tlater | jmac: Run in CI? Locally? What source? | 09:23 |
| juergbi | ah, that analysis is also done via pylint? the files are already ignored via .pylintrc, though | 09:23 |
| juergbi | also, I don't want to modify the generated files | 09:23 |
| jmac | tlater: Locally, just running ./setup.py test --addopts "-m pylint" | 09:23 |
| tlater | Oh, juergbi, were you asking about the checks tristan recently introduced? | 09:24 |
| juergbi | jmac: I actually also get one import order issue locally for a few days now. consistently, but I don't see that in CI | 09:24 |
| tlater | Pylint does code quality checks too, they aren't as neat though. | 09:24 |
| juergbi | tlater: yes, the 'code quality' check | 09:24 |
| juergbi | as per gitlab terminology | 09:24 |
| tlater | Yeah, sorry, not sure about that :/ | 09:25 |
| tlater | jmac: Don't specify -m pylint | 09:25 |
| tlater | That is enabled by default, specifying enables scanning files that aren't checked by CI, I think. | 09:26 |
| tlater | Oh, you're running *just* pylint | 09:26 |
| jmac | tlater: So how do I run only ... yep | 09:26 |
| jmac | Even if it were scanning extra files, it shouldn't be inconsistent | 09:26 |
| tlater | Not sure where the inconsistency comes from, but we'll perhaps want a nicer way to specify "just pylint". | 09:27 |
| *** aday has joined #buildstream | 09:35 | |
| tlater | jmac: For the time being, `pylint --rcfile .pylintrc buildstream` in the main project directory seems to work. | 09:35 |
| jmac | I'll try that | 09:36 |
| jmac | I wonder if the inconsistency comes from directory ordering | 09:36 |
| * tlater wonders if we should recommend using pylint directly anyway | 09:41 | |
| tlater | I find the setup.py syntax is more confusing than helpful here | 09:41 |
| tlater | Also really easy to forget --addopts | 09:41 |
| jmac | My default test is 'setup.py test', which I think should be the standard for local testing | 09:43 |
| tlater | Oh, certainly | 09:43 |
| jmac | ...and my goal is to run a subset of those tests, and I want them to be as close to how setup.py runs them as possible | 09:44 |
| tlater | Hm, that's a point | 09:44 |
| tlater | It's harder to configure sensibly, unfortunately | 09:44 |
| tlater | It might be better to add a custom command, instead | 09:44 |
| tlater | './setup.py lint' for example | 09:44 |
| jmac | That would be good | 09:45 |
| *** bethw has joined #buildstream | 09:52 | |
| *** Prince781 has quit IRC | 10:03 | |
| *** mohan43u has quit IRC | 10:15 | |
| jmac | Running `pylint` directly can give very different results to setup.py - on my system, /usr/bin/pylint is a script which uses Python2, and has a completely different version of the pylint library | 10:16 |
| jmac | I've hacked around that but it's a bit of a trap | 10:16 |
| tlater | Right, I forgot about that... Why can't python3 just be the default for everybody? |: | 10:17 |
| tlater | Well, certainly just a temporary measure, I'll open up an issue about this | 10:17 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:31 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:32 |
| tlater | Is the CI server down? | 10:32 |
| * tlater 's MR complains about not being able to connect :| | 10:33 | |
| jmac | Worse, I installed pylint3 while experimenting and now pylint via setup.py reports a tonne of new errors :\ | 10:37 |
| jmac | tlater: CI seems to be OK now | 10:41 |
| tlater | Yeah, I noticed :) | 10:41 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:50 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 11:03 |
| *** bethw has quit IRC | 11:33 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 11:49 |
| *** jennis has quit IRC | 12:13 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 12:29 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 12:35 |
| *** bethw has joined #buildstream | 12:41 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 12:43 |
| jmac | Can anyone help me with an import issue? element.py has `from . import SandboxFlags`, despite SandboxFlags being part of the 'sandbox' package. When I try to use the same syntax to import from my own new module, it fails - how does that import get resolved? | 13:17 |
| bochecha_ | jmac: can you show some details? (file layout and content) | 13:20 |
| jmac | It's from current buildstream master, but I'll try and paste some examples | 13:21 |
| tlater | jmac: The SandboxFlags may be imported in __init__.py | 13:21 |
| tlater | Your new module would have to be in the same module as element.py - I assume it is? | 13:21 |
| jmac | My new module is called 'storage' and is in the same directory as element.py and sandbox/ | 13:21 |
| jmac | tlater: I saw the import in buildstream's top-level __init__.py, and duplicated it for 'storage' | 13:22 |
| jmac | Didn't make any difference | 13:22 |
| tlater | If you can push a WIP branch it would make debugging a lot easier :) | 13:23 |
| jmac | I can but it's full of other patches | 13:26 |
| jmac | https://pastebin.com/7TPnU2fy are some of the relevant bits; I'll push a branch later | 13:27 |
| tlater | Imports rely so much on file structure that it's hard to help out without looking at the actual files | 13:27 |
| tlater | jmac: I'm not being particularly helpful here, but is that break of naming conventions (Directory.py) deliberate? | 13:28 |
| jmac | It was deliberate as in I meant to do it, but not deliberate as in I expect it to be committed that way | 13:30 |
| tlater | Alright, just wondering :) The issue can be in the modules inside storage, as well, I can't spot it immediately, sorry |: | 13:32 |
| jmac | https://gitlab.com/BuildStream/buildstream/commits/jmac/cas_backed_vdir is the branch, but I can't figure out how to get GitLab to show a diff vs master | 13:34 |
| *** aday has quit IRC | 13:38 | |
| *** aday has joined #buildstream | 13:40 | |
| tlater | Hrm, that branch looks like the import should work perfectly fine - simplest way to debug would be to launch an interactive shell and attempt to `import Directory` from various modules, to see where it isn't picked up | 13:45 |
| jmac | ok... | 13:45 |
| jmac | I shall try that shortly. Thanks for taking a look. | 13:46 |
| Nexus | Just to confirm a theory of mine. Do we determine the current buildstream version by the latest tag? | 14:01 |
| Nexus | i.e. if i run `git tag 9.9` will buildstream think it is version 9.9? | 14:01 |
| gitlab-br-bot | buildstream: merge request (valentindavid/update_mirror->master: WIP: update mirror) #440 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/440 | 14:13 |
| toscalix | tlater: I am processing the tasks associated to the Inmediate priority, to be able to erase the milestone. Are you currently working on #188 https://gitlab.com/BuildStream/buildstream/issues/188 | 14:14 |
| *** xjuan has joined #buildstream | 14:15 | |
| toscalix | adds68: are you working on #189 https://gitlab.com/BuildStream/buildstream/issues/189 ? | 14:15 |
| toscalix | abderrahim[m]: are you currently working on #316 or is it idle https://gitlab.com/BuildStream/buildstream/issues/316? | 14:15 |
| adds68 | toscalix, no i believe jmac was looking into that | 14:15 |
| toscalix | adds68: thanks | 14:16 |
| tlater | toscalix: I'm not currently working on #188 - I'm not sure when/if I will get to that, either. | 14:16 |
| toscalix | tlater: thanks | 14:16 |
| toscalix | jmac: are you working on #189 https://gitlab.com/BuildStream/buildstream/issues/189 ? | 14:16 |
| jmac | No, but I can link to the remote exec work now and close that issue | 14:17 |
| toscalix | ah, the issue is not valid anymore? | 14:18 |
| toscalix | there is another one that substitute this one? | 14:18 |
| gitlab-br-bot | buildstream: issue #189 ("Performance issues with running the artifact cache") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/189 | 14:18 |
| jmac | Yes, it says so in the discussion | 14:18 |
| toscalix | thanks | 14:18 |
| toscalix | juergbi: are you working on #136 https://gitlab.com/BuildStream/buildstream/issues/136 ? | 14:21 |
| toscalix | abderrahim[m]: it seems there is a MR merged associated to #316 but the ticket is not closed. | 14:24 |
| abderrahim[m] | toscalix: I'm not working on it. | 14:25 |
| *** aday has quit IRC | 14:25 | |
| abderrahim[m] | I just tried to provide enough info. | 14:26 |
| abderrahim[m] | It was closed but I reopened it as I got the error once again | 14:26 |
| toscalix | abderrahim[m]: thanks | 14:26 |
| abderrahim[m] | btw, #383 looks very similar, and the fix there might have fixed #316 as well | 14:27 |
| toscalix | ah tlater you are working on https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 14:27 |
| toscalix | checking #383 | 14:27 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 14:32 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 14:32 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 14:35 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 14:35 |
| *** xjuan has quit IRC | 14:35 | |
| tlater | toscalix: Yes indeed - hope to get it finished up soon, too :) | 14:35 |
| tlater | Though progress has been quite a lot slower than I'd hoped, for lots of reasons. | 14:35 |
| toscalix | ah, so when you close the MR, issue #135 can be closed | 14:36 |
| gitlab-br-bot | buildstream: merge request (jennis/136-expire-remote-cache-artifacts->master: WIP:136 expire remote cache artifacts) #364 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/364 | 14:37 |
| toscalix | inmediate priorities milestone closed. All tickets has been moved to the current milestone with priority Urgent. | 14:40 |
| toscalix | https://gitlab.com/BuildStream/buildstream/boards?= | 14:40 |
| tlater | Thanks toscalix :) | 14:41 |
| tlater | This looks quite pretty too, should help pointing people to low hanging fruite. | 14:41 |
| tlater | s/fruite/fruit/ | 14:41 |
| toscalix | I will be pinging most of you in the coming days to try to assign the right severity and status to the tickets you are working on | 14:41 |
| finn | oo very nice | 14:42 |
| toscalix | I will publish the policy but the idea is that those things you consider people should be paying attention to, should be labeleed as Important. Those absolutely relevant are Urgent. Please do not use Critical. This is assigned in extraordinary cases only. | 14:43 |
| toscalix | finn: this is the status board: https://gitlab.com/BuildStream/buildstream/boards/134373?= | 14:43 |
| toscalix | when you move the tickets around, labels change automatically | 14:44 |
| toscalix | this way we can see who is doing what and the severity of the tasks, so people can pick the most important ones | 14:44 |
| toscalix | right now we have tons of critical stuff. We will need to process them | 14:45 |
| toscalix | critical is... "leave what you are doing and solve this" kind of situation | 14:45 |
| tlater | toscalix: I suspect some of those are slightly mislabeled, and should be urgent instead. | 14:46 |
| toscalix | agree, and most of them simply Important, but those who evaluate them should decide | 14:47 |
| finn | Otherwise the project is very much on fire | 14:47 |
| toscalix | yep | 14:47 |
| * finn prepares the hose | 14:47 | |
| tlater | Well, some of them are certainly still problems we just haven't managed to reproduce ;) | 14:48 |
| * tlater recalls spending days on 128 | 14:48 | |
| tlater | And is now running into a *very* similar issue | 14:48 |
| toscalix | I will create an IRC or video chat triage meeting to assign the righ priority to bugs and tasks | 14:48 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 14:53 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:03 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:11 |
| *** noisecell has quit IRC | 15:19 | |
| *** tristan has joined #buildstream | 15:23 | |
| tlater | Oof, we have about a million workspace tests all of a sudden | 15:26 |
| * tlater is starting to worry that the test suite is becoming too large to sensibly run during development | 15:26 | |
| tristan | tlater, it takes a few minutes here, I wouldnt say it's sensible to run the whole thing in edit/compile/test cycles (I'd rather be selective about which tests to run in that scenario), but still very reasonable to run once you're done with your changes before submitting the MR | 15:28 |
| tlater | tristan: It's about 10 minutes on my machine | 15:29 |
| tristan | maybe around 8 minutes here actually, hopefully could get optimized | 15:29 |
| tristan | tlater, exactly, so not good for edit/compile/test, but reasonable before pushing an MR I still think | 15:29 |
| * tlater might just need to get more efficient in narrowing down which tests he should run. | 15:30 | |
| tristan | edit/compile/test should always be `./setup.py test --addopts 'tests/path/to/test'` | 15:30 |
| tlater | It's a little harder when you find that there are bugs in the broader scope just before a merge, but yeah, that's more sensible :) | 15:31 |
| tristan | in refactors, you generally need to run them all, but when doing a particular feature, it's usually pretty obvious what to run | 15:31 |
| tristan | Demos[m], I would call the generated debian stuff fairly experimental stages now, it mostly needs more usage to ensure that bases are covered and the public data API (input used to generate debian package metadata) is sane... for RPM, I feel like the problem is a little bit simpler (bias perhaps for my understand RPM packages a bit better than debian), there is this issue open https://gitlab.com/BuildStream/buildstream/issues/170 | 15:39 |
| valentind | It seems that we allow staging with some artifacts making symbolic links to some directories, and other artifacts using those path without symbolic links. I am not sure this is what we want because to me it sounds like depending on the order of staging, you will either get something working or an error. | 15:40 |
| tristan | Demos[m], also contains a link to a prototype which does it, the gist of it being basically A.) Generate a spec with only the parts that define the distribution and runtim dependencies, _not_ the build parts... B.) Run `rpmbuild -bb` to collect the artifact output and create a package with it | 15:41 |
| tristan | valentind, the order in which dependencies are staged is certainly very meaningful, both in terms of how symlinks work and in terms of overlaps | 15:42 |
| tristan | valentind, in some cases it is necessary to at least have `type: runtime` dependencies for otherwise orthogonal elements in order to declare the intended staging order | 15:43 |
| valentind | Because of dependencies, order might actually not be obvious for the user. | 15:44 |
| tristan | valentind, the question of symbolic links is mostly that, usually only one element is the one which "create" / "provides" it, so you might need to have something that links `/opt/foo -> /usr/local/orgname` staged *first*, before building a ton of stuff that configures into /opt/foo, with the desire of partitioning something else in /usr/local/orgname or suchlike | 15:45 |
| tristan | dependencies provide the technique for declaring staging order | 15:45 |
| tristan | if it is not clear to users, it should be made more clear, possibly with helpful examples and such | 15:46 |
| valentind | tristan, OK, but I think we should detect when we get the wrong order and properly give an error message. | 15:46 |
| valentind | We did get cryptic error with freedesktop SDK. | 15:47 |
| tristan | We give errors in some symlink edge cases, and we have the overlap errors (which I think could be less verbose and more clear) | 15:47 |
| valentind | Now in freedesktop SDK we try to never mix up symbolic links and directories. Because it is source of issue. | 15:47 |
| tristan | But we cannot give any error or hint for something which we cannot realistically know what was, or was not intended (we cannot read the project author's mind) | 15:48 |
| valentind | One thing you can do is make an element that builds all symbolic link and make all depend to it, so that they all use the symbolic links. | 15:48 |
| tristan | that is recommended for symlinked directories | 15:49 |
| valentind | Oh, yes I am talking about sym links to directories. | 15:49 |
| valentind | The problem of being only recommended is that people will get into issue late, and they will not understand what is happening. | 15:49 |
| tristan | however it is almost necessarily not something one should enforce, i.e. you generally have a "something to link to", then you "link to it", and everything else which wants that link "sees the link" | 15:49 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:50 |
| valentind | For instance, #390. I think this is due to a symlink to directory. | 15:50 |
| valentind | tristan, we have warning on overlap. I think we should have warnings for sym links to directories that are not consistant there as well. | 15:51 |
| valentind | Because it gives a good information on what 2 artifacts are in conflicts. | 15:51 |
| tristan | valentind, if you are installing into /opt/foo, which you "think" is supposed to be a symbolic link, but forget to depend on it to ensure it exists, BuildStream has no way of knowing at *that time* that you thought /opt/foo "should be a symlink" | 15:51 |
| valentind | which | 15:51 |
| tristan | So there is nothing to say about that at *that time* | 15:51 |
| tristan | However, when you eventually stage the element which creates the /opt/foo link, and it is an already existing directory instead, there is an opportunity to report a sensible warning, which could be optionally an error (like how overlaps are also optionally errors) | 15:52 |
| valentind | tristan, Sure. But when you compose, we should say that one artifact considered /opt/foo as a directory and the other as a symlink | 15:52 |
| tristan | That might be enhanceable | 15:52 |
| jjardon | hi, using junctions so project A depends on project B; does that mean that project B will use the cache server from project A? | 15:53 |
| tristan | valentind, if you know where the warning/optional-error is possible to report to make it more helpful, then please document it thoroughly in an issue :) | 15:53 |
| valentind | tristan, OK, I will look at it a bit more and do that. | 15:53 |
| tristan | valentind, symlinks is the trickiest thing about staging, so thoroughly here is key :) | 15:54 |
| valentind | But I think it belongs along with overlap warnings. | 15:54 |
| jjardon | If yes, Is there a way to disable that? | 15:54 |
| tristan | valentind, it sounds sensible, but it seems to me we might want a separate class of warning-optional-error than overlaps, to make it clear that it was a symlink disagreement | 15:54 |
| tristan | instead of just a "file which overlapped another" | 15:55 |
| Demos[m] | tristan: Doing RPMs that way would make them incompatible with koji/obs right? | 15:55 |
| valentind | tristan, It will be different warning. Just the code will be around the same place. | 15:56 |
| tristan | Demos[m], it's possible to "run rpmbuild on your own spec file and reuse all the rpmbuild mechanics", but it's counter the general mission I think; we ultimately want the "build story" to be decoupled from the "package / deploy / distribute" story | 15:57 |
| tristan | Demos[m], i.e. we want to use the same build element to build an artifact, and then use an element to deploy it through deb, and another element to deploy it through rpm | 15:58 |
| tristan | valentind, sounds sensible I think yes | 15:58 |
| gitlab-br-bot | buildstream: issue #401 ("If a project B use project A as a junction, B should not try to use the bst remote cache of A (at least by default)") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/401 | 15:58 |
| tristan | jjardon, by default "A project declares it's artifact cache server" indeed, for instance we benefit from that in GNOME, we use the same revisioned ref of freedesktop-sdk and rely on freedesktop-sdk to provide the built artifacts for the version we depend on | 15:59 |
| tristan | jjardon, I think overrides need to be enhanced for more dynamic setups, I think the only overrides beyond that which are currently available, are user config and not project config | 16:00 |
| jjardon | that can be a problem if suddenly that cache is used by 100 people, instead only 3 | 16:00 |
| jjardon | I'd prefer that GNOME can use it's own cache server | 16:01 |
| Demos[m] | tristan: hmmm yeah. Unfortinately rpms want to handle dependencies their own way | 16:01 |
| jjardon | only use the freedesktop one if there are updates | 16:01 |
| tristan | jjardon, I see this as a performance problem, not a design problem, honestly. | 16:01 |
| tristan | jjardon, ultimately I think we want to have less repetition of builds and redundant artifacts, and more sharing; if that is a problem, then we need better performance and/or better host machines | 16:02 |
| jjardon | tristan: if a project have declared its own cache server, then is that used or still the cache or the junction is used directly, without any caching? | 16:02 |
| tristan | Demos[m], True, public-data allows for some measure of overriding for the specific cases, currently the debian plugin uses the BuildStream dependency model for generating dependency metadata but I think that also for debian, that needs more enhancements to be more flexible | 16:03 |
| tristan | jjardon, I dont understand the phrase | 16:03 |
| tristan | " then is that used or still the cache or the junction is used directly, without any caching " | 16:04 |
| Demos[m] | tristan: I'm thinking in terms of my experience with CPack, which does a similar thing for RPMs/DEBS (as does the meson rpm plugin). For CPack, the generated debs and RPMS don't really reduce work much because they are super primitive and throw away lots of data. | 16:05 |
| tristan | Demos[m], I think that the current debian plugin was originally designed with the thought in mind "Lets take this organizations custom set of debian packages which define their custom OS, and make a script which converts it all into BuildStream elements, with the build / deploy steps separate" | 16:05 |
| tristan | Demos[m], While that is useful for some edge cases, I don't think that's essentially the best, most popular use of it; so that's another reason to consider it experimental :) | 16:06 |
| Demos[m] | yeah, certantly having debs/rpms as a possible binary artifact source makes sense | 16:07 |
| tristan | Oh, as a source that is a completely different topic to me | 16:08 |
| tristan | i.e. importing packaged data into a runtime to build on top of; is a completely separate goal | 16:08 |
| tristan | also useful :) | 16:08 |
| Demos[m] | it would be nice to be able to leverage bst's dependency info in some cases. Like a ton of projects have really quite complex setups for building their official rpm/apt repos (puppet, foreman, rdo(openstack)). One reason is because if you have a large number of dist packages building them on a dev machine tends to involve like mockchain, which is very, very primitive | 16:09 |
| *** finn has quit IRC | 16:10 | |
| *** finn has joined #buildstream | 16:10 | |
| tristan | Right, so I think there are cases where say, you are an org and you release an App, and you want to release 2 deb packages and 2 rpms (say one for LTS and one for latest of each distro) | 16:10 |
| tristan | In that case, the BuildStream dependency model is not interesting to you | 16:11 |
| tristan | There are cases where you want to build a fully functional / bootstrapped system from the ground up for your own embedded product, and you want to use RPM or Debian as a deployment / upgrade mechanism | 16:12 |
| tristan | In that case the dependency model can automate much or potentially all of the work | 16:12 |
| tristan | There are cases where you have something in the middle too, I guess; like building a "suite" which includes separate packages for some middleware | 16:13 |
| tristan | jjardon, trying to understand your question, "or the junction used directly", makes me think that maybe you think a junction is a cached entity which contains content; if that is your line of thinking: no it is not like that | 16:15 |
| tristan | jjardon, actually junctions never produce artifacts themselves, they are just a window into the artifacts provided by the project you depend on (whether you need to build them locally, or are available and downloadable in that project's cache) | 16:15 |
| Demos[m] | Maybe something like having rpm macros that are able to pull metadata from buildstream? so you still write the spec yourself but you don't have to duplicate much info | 16:21 |
| tristan | Ummm, one could potentially have an integration command (which runs before any build stuff after staging), which takes public data in context and generates something in /var/rpm/macros (dont recall if that is the correct directory but you get the idea) | 16:23 |
| tristan | Not sure if you have access to dynamic context there, though; would need to investigate that idea further | 16:23 |
| tristan | (i.e. dynamic context of variables and such about the element being build, rather than the lower level element which declared the integration command) | 16:24 |
| tristan | Or, a customized element could have a pre configure command which generates a macros file based on variables in context before doing anything else | 16:24 |
| tristan | Demos[m], I dont think my head is 100% in the game for this topic though right now, and I think there is a chasm between our viewpoints (as in, it's hard for me to understand in a very precise way, what you want to have accomplished) | 16:26 |
| tristan | Demos[m], might be a topic better suited for the mailing list (also it's 1:30am here :)) | 16:26 |
| jjardon | Tristan sorry, let's try again; what I'd like is that if project B uses A as a junction, and project B have defined a cache server, then project B will use it's own cache server for everything (even its own copy of the artifacts produced by A), only accessing the A cache server if there are updates from project A | 16:27 |
| Demos[m] | yeah, I'm not sure I understand what I want. | 16:28 |
| Demos[m] | I may tinker with some stuff and come to the mailing list | 16:28 |
| Demos[m] | However: I have to go deal with an /extremely/ needy server right now | 16:28 |
| *** xjuan has joined #buildstream | 16:29 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:29 |
| *** bethw has quit IRC | 16:38 | |
| *** jonathanmaw has quit IRC | 16:40 | |
| tristan | jjardon, I think we can do something similar to what you ask, as I said, I think overrides need to be enhanced for more dynamic setups... however I doubt that your specific request makes perfect sense | 16:48 |
| tristan | jjardon, i.e. I can envision that a B never uses A's cache server, meaning that B needs to rebuild it's copy of A artifacts itself, that is very straight forward | 16:49 |
| tristan | jjardon, I can imagine something more complex and "feature territory", whereas there are mirroring functionalities baked into the cache server (e.g. B's server actually "pulls" artifacts from A's server automatically and mirrors them) | 16:50 |
| tristan | jjardon, but I think you're losing me at "only accessing the A cache server if there are updates from project A", not exactly sure how that would work | 16:51 |
| tristan | the first option, is really only a matter of enhanced configuration options | 16:51 |
| tristan | the second option is a heavy duty "feature" | 16:51 |
| jjardon | Tristan user of project B checks its own cache first , the if its not there it uses cache from A. And it will only push to B | 16:52 |
| tristan | that said, thinking about mirroring services for artifact caches "on it's own" sounds interesting, given the load issues we've been seeing | 16:52 |
| tristan | jjardon, right, ok that part makes a bit more sense to me now | 16:53 |
| tristan | jjardon, that is fairly within reach with some configuration data, remember that "Multiple artifact caches for a project" is a feature introduced before junctions existed | 16:54 |
| tristan | and junctions did not go very far towards extra enhancements of this | 16:54 |
| tristan | so the first "multi cache" already works exactly how you describe | 16:54 |
| jjardon | Yup | 16:55 |
| tristan | jjardon, this was mostly thought up for cases of workflows where there is a "stable" or "dev / side branch" activity for a project, where a branch might pull artifacts from "mainline", but only push the differing artifacts to "dev" | 16:55 |
| *** tiago_ has quit IRC | 16:55 | |
| tristan | jjardon, What you ask for is fairly within reach, but... I think it might not solve what you want it to | 16:56 |
| tristan | jjardon, if for instance, B sets itself up to push only to B, and download first from B and then A if not found in B | 16:57 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:57 |
| tristan | Then, as long as artifacts existed in A, B will never contain them | 16:57 |
| tristan | jjardon, so basically, since B never had to build stuff that was downloaded from A, you dont get to reduce the server load for A | 16:58 |
| tristan | Unless you implement this mirroring stuff in *some* way | 16:58 |
| tristan | which is less trivial | 16:58 |
| tristan | Also, I'm not sure that that is the best way to address load; if we need mirrors for load, maybe they should just strictly be mirrors, and not be tied into this so deeply | 16:59 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 17:00 |
| tristan | jjardon, I guess one could say that: If not exists in B, download from A, if still not exists... download sources and "Build it"... regardless of how the artifact was obtained: Push it to B | 17:01 |
| jjardon | Tristan how difficult would be to always check if the artifact exist in the server and push if it us not there? | 17:01 |
| tristan | jjardon, still, if this is really a question about server load, I dont know if this is a "solve", it just pushes the problem from one place to another | 17:02 |
| jjardon | Well | 17:02 |
| tristan | jjardon, that is less difficult now that I think of it | 17:02 |
| *** toscalix has quit IRC | 17:04 | |
| tristan | jjardon, you can file an issue for this feature, or raise discussion on the ML, either way to get the ball rolling; I agree we need more configuration and control of artifact caches now that we have junctions in play | 17:04 |
| tristan | I just think we need to really think of what problems we're solving, and if we're really solving them the right way | 17:05 |
| jjardon | As I said, free desktop project can have 10 users and everything is ok; I think I makes sense gnome / KDE / others take care on their own cache , instead relay all of then in the freedesktop one | 17:05 |
| jjardon | I have to take a plane now; I have filled an issue already | 17:05 |
| jjardon | I will try to expand there later | 17:05 |
| tristan | That would mean to never resort to download from A, if you wanted control | 17:06 |
| tristan | maybe that should *also* be an option | 17:06 |
| tristan | jjardon, and maybe it is project A which needs to declare this for people who depend on it, rather than allowing depending projects to configure in a way which spams your servers | 17:06 |
| tristan | I do still feel skeptical about acknowledging "the load is too much" argument, things may slow down with hundreds of thousands of parallel requests, if they happen to occur at the same time | 17:07 |
| tristan | but things "should not break" | 17:07 |
| tristan | if we're designing around a broken implementation: That is bad - better to fix the implementation | 17:08 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 17:10 |
| *** Prince781 has joined #buildstream | 17:38 | |
| *** Prince781 has quit IRC | 17:41 | |
| *** Prince781 has joined #buildstream | 17:42 | |
| *** bochecha_ has quit IRC | 18:07 | |
| *** Prince781 has quit IRC | 18:14 | |
| *** Prince781 has joined #buildstream | 18:16 | |
| *** ernestask has quit IRC | 18:39 | |
| *** xjuan has quit IRC | 19:17 | |
| *** Prince781 has quit IRC | 19:31 | |
| *** Prince781 has joined #buildstream | 19:52 | |
| *** bochecha_ has joined #buildstream | 19:56 | |
| *** tristan has quit IRC | 20:06 | |
| *** Prince781 has quit IRC | 20:07 | |
| *** bochecha_ has quit IRC | 21:55 | |
| *** noisecell has joined #buildstream | 22:32 | |
| *** Prince781 has joined #buildstream | 22:47 | |
| *** noisecell has quit IRC | 23:18 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!