| *** bethw has joined #buildstream | 00:04 | |
| *** bethw has quit IRC | 00:49 | |
| gitlab-br-bot | buildstream: merge request (reset-all-workspaces->master: Add option to reset multiple workspaces) #374 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/374 | 01:13 |
|---|---|---|
| *** Prince781 has joined #buildstream | 02:40 | |
| *** Prince781 has quit IRC | 04:07 | |
| *** tristan has quit IRC | 05:06 | |
| *** tristan has joined #buildstream | 05:35 | |
| gitlab-br-bot | buildstream: issue #352 ("Race condition: Incorrect saving of "running files" in workspaces.yml local state file") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/352 | 07:13 |
| tristan | tlater, juergbi ^^^^ issue 352, oops. | 07:13 |
| juergbi | ah, crap, yes, that should definitely be in the main process | 07:16 |
| tristan | juergbi, I'm also looking at extending the workspaces.yml for https://gitlab.com/BuildStream/buildstream/issues/346 | 07:17 |
| tristan | which needs something similar | 07:17 |
| juergbi | yet another case of complexity due to missing separation between pristine workspace sources and build directory | 07:18 |
| juergbi | I really don't want to keep that long term | 07:18 |
| tristan | the running files thing ? | 07:18 |
| juergbi | yes | 07:18 |
| tristan | Yeah I can't say I really understand completely what it's for | 07:18 |
| juergbi | it's needed with the current way we handle workspaces | 07:18 |
| tristan | I noticed the commit says we can remove it once we have failed build artifacts | 07:18 |
| juergbi | yes, if we also create artifacts for failed workspace builds where we record the dependencies, we don't need the running files | 07:19 |
| juergbi | but we'd still have to record the failed builds | 07:20 |
| tristan | So my case fwiw, is that basically: The plugin decides where to stage it's sources in the sandbox; a natural choice is `self.get_variable('build-root')` but that is not the law afaics | 07:20 |
| juergbi | hm, isn't this recorded in the sandbox instance? | 07:21 |
| tristan | So, now that workspaces are mounted, `bst shell --build --sysroot /path/to/bst/created/sandbox` ... doesnt know how to retrieve that information | 07:22 |
| juergbi | ah, after the fact | 07:22 |
| tristan | I.e. the same thing that happens when we give the user the opportunity to debug interactively | 07:22 |
| tristan | So, I think recording that in workspace.yml would be a good solution | 07:23 |
| tristan | While looking at how I might serialize that, I stumbled on #352 :) | 07:23 |
| tristan | So in any case, whether it is running files, or a failure or something, there is still data we'll want to record for whatever running files is for | 07:23 |
| juergbi | if we allow specifying the sysroot, the path could theoretically have changed | 07:24 |
| tristan | Could it ? | 07:24 |
| tristan | juergbi, that path is a sandbox relative path | 07:24 |
| tristan | it could have changed with code changes to the plugin | 07:25 |
| tristan | which is plausible but rather an outlier | 07:25 |
| juergbi | the element could have switched kind | 07:26 |
| juergbi | unlikely, though | 07:26 |
| tristan | heh quite | 07:27 |
| tristan | It could be re-inforced with better error checking for that, if we eventually need that | 07:27 |
| tristan | But in *any* case, I think it's clear that we need to send back information about workspaces to the main process for serialization-upon-task-completion | 07:28 |
| tristan | Which could be handled by the Queues at .done() time | 07:28 |
| tristan | Maybe we can leverage the already existing Workspace._to_dict() stuff | 07:28 |
| tristan | for that | 07:28 |
| tristan | just send the whole workspace serialized back over the wire, replace it inline and save | 07:29 |
| juergbi | yes, probably makes more sense than inventing another interface for that | 07:31 |
| tristan | Also has the advantage of "just working transparently" for whatever changes might be made to a workspace in a task | 07:32 |
| tristan | no matter how workspace metadata evolves | 07:32 |
| *** toscalix has joined #buildstream | 07:35 | |
| *** noisecell has joined #buildstream | 07:48 | |
| *** noisecell has quit IRC | 07:59 | |
| *** noisecell has joined #buildstream | 08:03 | |
| *** noisecell has quit IRC | 08:03 | |
| *** jonathanmaw has joined #buildstream | 08:46 | |
| tlater | tristan: On #352, do we have the same issue if we try to run two instances of buildstream in the same project? | 08:51 |
| tlater | Is that even a use case we care about? | 08:51 |
| tristan | tlater, I care a lot less about that | 09:00 |
| tristan | tlater, multiple projects using the same caches is important to consider; multiple parallel running instances much less so, parallel instances running for the same project _and_ same directory... very low chance of that ever happening | 09:01 |
| *** noisecell has joined #buildstream | 09:01 | |
| tristan | maybe we have to eventually try using locks to ensure better robustness in those possibilities; but I dont think it's pressing | 09:02 |
| tlater | Hm, I have no idea how to neatly propagate this state to the main process | 09:11 |
| tlater | It certainly must be calculated when staging the dependencies | 09:12 |
| *** dominic has joined #buildstream | 09:15 | |
| tristan | tlater, I'm doing it now | 09:19 |
| tristan | tlater, I need it anyway to solve https://gitlab.com/BuildStream/buildstream/issues/346 | 09:19 |
| tlater | Alright, would you mind giving me a heads up when you have something pushed? I really want to see what it *should* look like, just to actually understand buildstream's multiprocessing properly. | 09:21 |
| *** dominic has quit IRC | 09:22 | |
| *** dominic has joined #buildstream | 09:22 | |
| tristan | tlater, sure I intend to :) | 09:23 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 09:50 |
| *** aday has joined #buildstream | 09:59 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:06 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:09 |
| *** valentind_ has quit IRC | 10:10 | |
| gitlab-br-bot | buildstream: merge request (richardmaw/fix-unit-test-hang-on-crash->master: Fix unit tests hanging when there's an exception in a sandbox run) #375 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/375 | 10:11 |
| tristan | tlater, you do have a test case in place ensuring that this 'running_files' mumbo jumbo is doing it's thing right ? | 10:16 |
| tlater | I thought I did, but I'm not sure I trust it anymore | 10:17 |
| tristan | I just want to make sure I didnt break anything while pushing this | 10:18 |
| tristan | it's "just about done" now, so I can move on and fix 346 after this | 10:18 |
| tlater | tristan: Well, the test I wrote is in integration/workspace.py | 10:19 |
| tlater | It doesn't try building multiple elements, though | 10:19 |
| tlater | So it won't capture the issue you're solving | 10:19 |
| tristan | tlater, will it break if running files mumbo jumbo doesnt get serialized ? | 10:19 |
| tlater | Yeah, it should | 10:19 |
| tristan | 346 is a different issue | 10:19 |
| tristan | Ok good, thanks :) | 10:20 |
| * tristan is thinking of adding a utils._is_main_process() utility, and peppering the codebase with some assertions based on that | 10:22 | |
| tristan | would be at least good to add that assertion to Workspaces.save_config() | 10:22 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:25 |
| tlater | Yeah, I think this might be handy. I've been bitten a few times while working with workspaces now, I can see this being incredibly easy to overlook. | 10:26 |
| * tlater realized his two most recent branches *also* have this problem | 10:26 | |
| juergbi | we also talked about moving away from the fork() model at some point | 10:27 |
| jmac | What's the lifetime of logfile stored in /.cache/buildstream/logs/ ? | 10:27 |
| tlater | jmac: I *think* that directory isn't flushed currently, but it might be flushed with non-failing elements like $cache/build/ | 10:29 |
| tlater | Other than that I don't think logs have a lifetime | 10:30 |
| tristan | juergbi, interestingly, passing data around the way we do would not get in the way of moving to a threading model :) | 10:30 |
| tristan | jonathanmaw, just a thought passing through my mind... regarding junctioned element addressing: It would also unlock the ability to open and work with cross-junction workspaces :) | 10:30 |
| tristan | tlater, what is the desired functionality for these running file thingamabobs when you run `bst shell --build workspaced/element.bst` ? | 10:35 |
| tlater | tristan: It depends on the dependencies of that element | 10:36 |
| tlater | If any of its dependencies had files that were updated since the last time that element was built, those files should be recorded - if I recall the functionality correctly. | 10:37 |
| tlater | `tests/integration/workspace.py::test_workspace_update_dependency_failed` tries to test that | 10:38 |
| tristan | tlater, it tries to test it using `bst shell` ? | 10:38 |
| tlater | Oh, no, it doesn't | 10:39 |
| tlater | Hm | 10:39 |
| tristan | I'm asking about bst shell, and what is expected. | 10:39 |
| tristan | seems to me strange that a bst shell invocation should modify the state | 10:39 |
| gitlab-br-bot | buildstream: merge request (tristan/workspace-serialize-writes->master: serialize writes to workspaces.yml) #376 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/376 | 10:40 |
| tlater | Hm, is the workspace mounted for bst shell? | 10:40 |
| tlater | If it is, then I think it's still required | 10:40 |
| tristan | tlater, that is issue #346 | 10:40 |
| tristan | tlater, i.e. shells for workspaces regressed ever since we started mounting them | 10:41 |
| tlater | Ah, so we don't want to mount it anymore | 10:41 |
| tristan | it's all in the backlog... | 10:41 |
| tristan | Yes we want to mount it | 10:41 |
| tristan | tlater, also, 346 is specifically about running bst shell on an existing sandbox; it is already mounted correctly without that case (sorry wasnt clear) | 10:42 |
| tristan | using bst `shell --build --sysroot ...` on a workspaced failed build sandbox, will fail to mount the workspace | 10:42 |
| tlater | If `bst shell` mounts, I think the expectation is that running_files is updated with any files from updated dependencies. | 10:43 |
| * tlater needs to double check though | 10:43 | |
| tristan | tlater, so basically you are saying that A.) Running build, track, build... or B.) Running build, track, shell, build... is going to make the last 'build' of the workspaced element behave differently ? | 10:45 |
| tristan | and that is intentional ? | 10:45 |
| juergbi | strictly speaking, this is required, yes | 10:46 |
| juergbi | if we assume that the user might run make itself in the shell | 10:46 |
| tlater | tristan: The point of running files is to keep track of any file updates in dependencies between builds | 10:46 |
| tlater | We only have access to successful artifacts atm, so if the dependency changes twice between successful builds, we might end up missing a file update | 10:47 |
| tlater | If you want to be able to run `make` in a shell and get a compilation with updated dependencies, you need to also keep track of those updates | 10:47 |
| * tlater still finds it hard to wrap his head around this edge case every time he has to express it. | 10:48 | |
| tristan | I dont understand this. | 10:48 |
| juergbi | tristan: it's essentially the 'git checkout' approach with regards to timestamps for build dependencies | 10:49 |
| tristan | Remember that there is no way on earth that a `bst shell` invocation is ever going to change Workspace.last_successful | 10:49 |
| juergbi | whenever you update dependencies, you have to update the timestamps of all modified files (like git checkout) | 10:49 |
| juergbi | you can only reset all timestamps to the same after a successful build | 10:50 |
| tristan | juergbi, I think bst shell does not need this, but I still might not understand | 10:50 |
| juergbi | tristan: correct, last_successful should definitely not be updated, but running_files should theoretically be | 10:50 |
| juergbi | it depends on what bst shell is used for | 10:50 |
| tristan | we dont know what it's used for of course | 10:50 |
| juergbi | if you just look at things, it's not needed, if you actually do a partial build, it's needed | 10:50 |
| tristan | but... the point is; when you open a bst shell... it's going to go through the same codepaths | 10:51 |
| tristan | So, it's going to bump timestamps and do it's boogie down if dependencies have changed *anyway* | 10:51 |
| tristan | The question is... should the result of that boogie down be stored in workspaces.yml for a *next build* | 10:51 |
| juergbi | yes, a (failed) make in bst shell is not different from a failed `bst build` | 10:52 |
| tristan | So, running files are lists of files for each dependency | 10:52 |
| juergbi | that have been updated since the last successful build | 10:52 |
| tristan | So, running files have nothing to do with the element being built, they dont record anything from the workspaced element in question, as far as I can see | 10:53 |
| juergbi | correct | 10:53 |
| juergbi | as the workspace is stored in a regular directory, there are no timestamp issues | 10:53 |
| juergbi | to be precise, they have nothing to do with the files of the element being built itself | 10:54 |
| juergbi | they are still specific to the workspace | 10:54 |
| tristan | "it's essentially the 'git checkout' approach with regards to timestamps for build dependencies" <-- I dont understand that statement and havent read the code behind git checkout | 10:55 |
| tristan | So, lemme try to understand what this is doing | 10:55 |
| tristan | "whenever you update dependencies, you have to update the timestamps of all modified files" | 10:56 |
| tristan | juergbi, What is "all modified files" ? | 10:56 |
| juergbi | e.g., a header file of a library dependency | 10:56 |
| tristan | Certainly, not the modified files in the workspace right ? | 10:56 |
| juergbi | if the header file changes, we need to make sure it has an updated timestamp | 10:56 |
| tristan | Right | 10:57 |
| tristan | That much I get.... so if there is never a "successful build", what harm does it do if `bst shell` does not modify this metadata ? | 10:57 |
| tristan | We have a list of files, which have changed since the last successful build | 10:58 |
| tristan | We use that to bump timestamps, such that any files which have changed since the last successful build are "new" | 10:58 |
| juergbi | the issue is how do we know which files have changed | 10:58 |
| juergbi | e.g., you might update a library dependency from version 1 to version 2 | 10:59 |
| tristan | bst shell cannot modify these files, certainly | 10:59 |
| juergbi | correct, but you could invoke bst shell after updating a dependency without running bst build | 10:59 |
| tristan | juergbi, I assume you are using the artifacts.diff codepaths to discover this | 10:59 |
| juergbi | yes | 10:59 |
| juergbi | but they can only compare between two versions | 11:00 |
| tristan | Right, .... so stop there a moment | 11:00 |
| tristan | <juergbi> correct, but you could invoke bst shell after updating a dependency without running bst build | 11:00 |
| tristan | Circle back... | 11:00 |
| tristan | basically, I cant find my line above but what I am saying... is that `bst shell` *is going to do this dance anyway* | 11:01 |
| tristan | It's just not going to *save* the result of it | 11:01 |
| tristan | Why should it *save* the result ? | 11:01 |
| juergbi | example | 11:01 |
| tristan | Even if the dependencies change two or three times between two builds | 11:01 |
| tristan | And bst shell gets invoked in between a number of times | 11:01 |
| tristan | The next time you run the build, you still use the last successful build + the list of files which have changed in dependencies | 11:02 |
| * tristan awaits example | 11:02 | |
| juergbi | you have element foo in a workspace that depends on library libbar. libbar is currently at version 1. all has been built successfully | 11:02 |
| juergbi | now you update libbar to version 2 and bst build just libbar. all still fine | 11:02 |
| juergbi | then you invoke bst shell for the element foo and call make in there | 11:03 |
| juergbi | that builds a few object files but fails at some point, so it's only a partial build | 11:03 |
| tristan | Right | 11:03 |
| juergbi | you then switch libbar back to version 1 and call bst build on foo | 11:03 |
| juergbi | just checking the artifact diff, buildstream will see no differences | 11:03 |
| juergbi | so header files of libbar will have an old timestamp | 11:04 |
| juergbi | and make might nto do anything | 11:04 |
| tristan | Ah, ok I got it | 11:04 |
| tlater | Hm, I wonder if the `bst shell` case means that we'll have to keep a running file list even with failed artifacts | 11:05 |
| juergbi | I suppose so, unless we create a fake artifact for bst shell | 11:05 |
| tlater | Perhaps we should also note that example down in a comment somewhere | 11:06 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 11:25 |
| tlater | tristan: Would you mind adding your thoughts about this comment? https://gitlab.com/BuildStream/buildstream/merge_requests/347#note_67638476 | 11:29 |
| tlater | I discussed the default cache quota mostly with you, but I do agree with juergbi here. | 11:29 |
| juergbi | tlater: btw: I don't have a very strong opinion on this, just thought it's a bit odd and thus commented | 11:31 |
| * tlater understands, but wants to nail these configuration options so we don't have to break API when we decide they're not good enough | 11:32 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 11:37 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 11:44 |
| jjardon | Hi, can I have a second review of https://gitlab.com/BuildStream/buildstream/merge_requests/373 , please? | 11:59 |
| juergbi | jjardon: these python libraries aren't all base system requirements, are they? i.e., at least some of them are automatically pulled in by pip3, iirc | 12:10 |
| gitlab-br-bot | buildstream: merge request (tristan/workspace-serialize-writes->master: serialize writes to workspaces.yml) #376 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/376 | 12:12 |
| jjardon | juergbi: I have a look to the code; seems all of them are part of the bst code (not test or anyythig else). Similar to the already listed ruamel.yaml | 12:13 |
| tristan | jjardon, fwiw, the reason I had ruamel.yaml there was because at a certain point, I found that trying to install buildstream with it not being installed with your package manager on debian, was causing the installation to fail, that was a long time ago | 12:16 |
| tristan | jjardon, rather, if it can be verified that this is no longer the case, it would be better to remove that from the base system requirements | 12:16 |
| jjardon | tristan: but they are required, install them with pip or from your package manager | 12:16 |
| juergbi | the point is that the user doesn't have to manually install them before installing buildstream with pip | 12:17 |
| tristan | jjardon, not so... if you follow the instructions, it will not fail if you run `pip3 install .`; it will automatically take care of downloading it | 12:17 |
| tristan | jjardon, so the user doesnt have to be burdened with this knowledge | 12:17 |
| juergbi | tristan: I actually don't have a ruamel system package here. However, I'm not on Debian, so there might still be an issue there for some reason | 12:17 |
| tristan | jjardon, similarly, if you are going to use a package manager, then the distro has solved this problem for you already | 12:17 |
| tristan | i.e. installing a package on any system should not require that the user be aware of the dependencies | 12:18 |
| jjardon | tristan: how come? | 12:18 |
| tristan | jjardon, because distro packages have the dependencies encoded | 12:18 |
| jjardon | if bst is not packaged, the package manager will not knw what to install | 12:18 |
| tristan | you dont have to install things first, except for the things listed there | 12:18 |
| tristan | jjardon, pip will do it for you | 12:19 |
| tristan | jjardon, `pip install --user -e .` will automatically get the pure python dependencies for you, so it's not needed | 12:19 |
| jjardon | lets decide if use pip or not to the user; this is listing what dependencies are needed for running bst | 12:19 |
| tristan | jjardon, however, it will not install ostree, or bubblewrap | 12:19 |
| jjardon | exactly, lets list the dependencies, then say "hey, you could install these ones with pip automatically if you want" | 12:20 |
| tristan | jjardon, No, it is not listing that - it is a guide for a *user* to use, with the minimum amount of information for a user to get off the ground | 12:20 |
| tristan | jjardon, if you want to list all of the dependencies which are needed, that would be part of a more terse documentation targetted at say, distro packagers. | 12:20 |
| tristan | jjardon, the point is to guide the user safely with some instructions to install buildstream, the more information which goes there, the lower quality that guide is | 12:21 |
| tristan | it should ideally be quick and painless, not a daunting and frightening wall of dependencies | 12:22 |
| jjardon | tristan: no, someone can prefer to use distro packages than depend in the latest random version available in pip; listing all the dependencies helps with that | 12:22 |
| tristan | it looks like bubblewrap is missing there, though | 12:22 |
| tristan | jjardon, no, it does not, as I explained above | 12:22 |
| jjardon | tristan: bubblewrap is there | 12:22 |
| tristan | <tristan> jjardon, similarly, if you are going to use a package manager, then the distro has solved this problem for you already | 12:22 |
| tristan | <tristan> i.e. installing a package on any system should not require that the user be aware of the dependencies | 12:22 |
| tristan | jjardon, those dependencies are *only* useful for someone who is going to install with pip | 12:23 |
| tristan | jjardon, if you are installing with a package manager, then the package manager / distro is responsible for encoding the dependencies into the buildstream package - the user doesnt need to know. | 12:23 |
| jjardon | ok ,should we remove all the python packages and also python-ruamel-yaml and python-gobject from the packages you should install then? | 12:24 |
| juergbi | can pygobject be installed with pip? | 12:24 |
| jjardon | yup | 12:24 |
| tristan | jjardon, we should remove everything that the user need not know - pygobject cannot be installed with pip | 12:25 |
| jjardon | https://pypi.python.org/pypi/PyGObject | 12:25 |
| tristan | I looked into that, it cannot; there is a commit in pygobject history saying "We made it work" | 12:25 |
| tristan | But it doesnt work at all. | 12:25 |
| tristan | at least, it did not work at all when I tried it | 12:25 |
| juergbi | it's not currently in our setup.py, so it will not work right now | 12:26 |
| juergbi | no idea whether it could work | 12:26 |
| tristan | It would have to be experimented with various platforms, I know it did not work for me last year with debian stretch | 12:26 |
| jjardon | ok, I wil try and come back to you | 12:26 |
| tristan | I think it might be safe to remove ruamel.yaml from that list, though | 12:26 |
| juergbi | might also be an issue with our ostree version check as that already requires pygobject early on | 12:27 |
| tristan | I just recall I had incidents with it, which is why I recommended installing that one first with package manager | 12:27 |
| jjardon | it would be good for that to be documented | 12:27 |
| jjardon | to avoid confusion | 12:27 |
| tristan | juergbi, that is also a separate issue, the problem when I tried installing pygobject was related to it trying to compile itself inside a pip install session | 12:27 |
| tristan | which was totally borked. | 12:28 |
| gitlab-br-bot | buildstream: issue #352 ("Race condition: Incorrect saving of "running files" in workspaces.yml local state file") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/352 | 12:28 |
| gitlab-br-bot | buildstream: merge request (tristan/workspace-serialize-writes->master: serialize writes to workspaces.yml) #376 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/376 | 12:28 |
| tristan | tlater, for your reference, see the commits in: https://gitlab.com/BuildStream/buildstream/merge_requests/376 | 12:29 |
| tristan | tlater, specifically: https://gitlab.com/BuildStream/buildstream/merge_requests/376/diffs?commit_id=a9cbb0f7074286f5323954d327bc923c0028e7c0 | 12:29 |
| * jjardon would like to test the dependency installation as part of the CI, so we are sure we keep them up-to-date | 12:29 | |
| tristan | jjardon, I would very much like that too; actually I want multiple distros and versions of those to be tested | 12:30 |
| tristan | jjardon, And I want it to happen for each major distro version that we run CI on | 12:31 |
| jjardon | tristan: let me work on something :) | 12:31 |
| tristan | jjardon, I think it starts with the buildstream-docker-images repo | 12:31 |
| tristan | there is mostly some sorting to do there to understand which image is used for what | 12:31 |
| tristan | originally, we tried to use the same fedora image for CI and for the convenience `bst-here` script | 12:32 |
| tristan | But that was not a great idea because; we want to test against python3.4 (earliest supported python), and the `bst-here` script naturally wants an up to date and featureful image | 12:32 |
| jjardon | lets support several distros in a generic way, then we can add the instructions of the CI to the docs, so no need to keep thing in sync | 12:33 |
| tristan | The important thing to not lose here, is that buildstream should not use a "tag/branch" to refer to the docker images it tests against | 12:34 |
| jjardon | any reason to limit to 3.4? even Debian stable have 3.5 nowadays | 12:34 |
| tristan | Personally, I think comments in the .gitlab-ci.yml is the best form of documentation for that, it doesnt belong in user facing docs really | 12:35 |
| jjardon | tristan: it should, if not how do you test against fedora 27 and fedora 28? | 12:35 |
| tristan | jjardon, No reason, but if we're only going to test with one image, it should be python3.4 | 12:35 |
| jjardon | agree, but users doesnt go to .gitlab-ci to check what they need to install | 12:35 |
| tristan | jjardon, We should be testing against every major release of every supported distro | 12:35 |
| tristan | "then we can add the instructions of the CI to the docs" I dont see why users need to know how the upstream project maintainers maintain their CI | 12:36 |
| jjardon | tristan: all major current releases (debian, arch, fedora, suse ...) have python 3.5, only saying | 12:36 |
| tristan | jjardon, I think you are talking about different forms of CI | 12:36 |
| tristan | I am talking about CI *of buildstream* | 12:36 |
| jjardon | yes | 12:37 |
| tristan | jjardon, They do now, and maybe in 1.2 we can consider bumping the python requirement, this was however not true a year ago | 12:37 |
| tristan | jjardon, which means.... we should be testing a *previous* release of major distros with python3.4 from last year for current buildstream | 12:38 |
| tristan | but that is getting less important | 12:38 |
| tristan | I think python3.5 minimum for 1.2 will probably be a good thing | 12:38 |
| tristan | currently we support down to 3.4, so we should be testing that | 12:39 |
| tristan | jjardon, Regarding instructions for people to run their *own* CI for their own buildstream projects; I worry about scope creep, I dont think it's really relevant in our docs | 12:39 |
| jjardon | tristan: me neither | 12:40 |
| jjardon | dont think I have suggested that? | 12:40 |
| tristan | One main reason for that is, we cannot really assume a specific system, some people use gitlab, others use jenkins, etc etc | 12:40 |
| tristan | <jjardon> lets support several distros in a generic way, then we can add the instructions of the CI to the docs, so no need to keep thing in sync | 12:40 |
| tristan | jjardon, I dont understand the above phrase about documentation then :) | 12:41 |
| jjardon | tristan: part of the CI is to install buildtream in different distros and run test to check it works | 12:41 |
| tristan | We've now established it is not about instructing users about how upstream buildstream runs it's own CI, and we've established that buildstream should not document how projects use buildstream in their own CI :) | 12:41 |
| jjardon | then we use those same instructions as a doc in https://buildstream.gitlab.io/buildstream/install.html | 12:41 |
| tristan | Ah, so you want to have some file describing those instructions, which we literally include into the docs ? | 12:42 |
| jjardon | we are not instructing user, we are chekcing that is actually possible install buildstream correctly in different distros | 12:42 |
| tristan | and run in the CI ? | 12:42 |
| jjardon | yes | 12:42 |
| tlater | Hm, I wonder if the docker images count as that | 12:42 |
| tristan | That sounds like a very nice idea :) | 12:43 |
| tlater | Or at least help with it | 12:43 |
| tristan | jjardon, however it's not a good idea for our CI | 12:43 |
| tristan | jjardon, for another reason | 12:43 |
| tristan | jjardon, we make *damn sure* that the only thing that every changes from one run of CI to another, is buildstream code | 12:43 |
| jjardon | doesnt matter really | 12:43 |
| tristan | jjardon, that is why we make sure the docker image already comes with things pre-installed | 12:43 |
| tristan | and we dont use a moving tag | 12:44 |
| jjardon | tristan: I have seen pure python changes in code that work in Ubuntu and not in Fedora | 12:44 |
| tristan | the docker image should not install things from distros during CI of buildstream | 12:44 |
| tristan | Sigh, we are talking apples and oranges again | 12:44 |
| tristan | jjardon, A.) We want many images of different versions of major distros... B.) We dont want those images to change *at all* between CI runs... C.) Changing of the backing image is done manually from our own .gitlab-ci.yml | 12:45 |
| jjardon | tristan: agree with that, its what me who changes the use to a specific docker image in the builstream CI instead using always the "latest" | 12:45 |
| tristan | Installing a package during the CI process, means we might get a subtly different version of some system dependency, which may falsify results | 12:45 |
| jjardon | tristan: as I said, I agree on that | 12:46 |
| tristan | right, so that's why I dont know if it's feasible to literally include those instructions into our generated documentation | 12:46 |
| tristan | I.e. the places where distro specific installation things happen, are in the buildstream-docker-images repo, and are not available when building documentation | 12:47 |
| tlater | tristan: Those instructions could be used as the Dockerfiles in buildstream-docker-images | 12:47 |
| tlater | We'd just need a neat way to make them a bit more human-readable | 12:47 |
| tlater | (The docker images, that is) | 12:47 |
| jjardon | let me see if I can think on something, taking in account above comments | 12:48 |
| gitlab-br-bot | buildstream: merge request (jjardon/update_depencies->master: docs: Update list of dependecies) #373 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/373 | 12:50 |
| tristan | jjardon, you may want to coordinate with cs_shadow, who iiuc maintains buildstream-docker-images | 12:50 |
| tristan | certainly sounds like an interesting idea, it's just a bit technically challenging | 12:51 |
| gitlab-br-bot | buildstream: merge request (jmac/logfile-widget-correction->master: WIP: Correct log line if logdir is empty) #377 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/377 | 12:53 |
| jjardon | it would be good as well to produce more docker images, not only one based in a specific version of Fedora. Then we can check that builstream works ok in all of the ones we care | 12:54 |
| tristan | yes, that is desired :) | 12:55 |
| tristan | jjardon, honestly, I want to start with versions of major distro of last year, and keep adding versions to CI for every stable release of BuildStream | 12:55 |
| tristan | jjardon, such that we can install BuildStream 1.0 on whatever major distro is available in 10 years, and it still works | 12:56 |
| jjardon | 1.0 ? | 12:56 |
| tristan | this is sort of an implicit requirement as we cannot guarantee cache key stability when reving major point versions of BuildStream | 12:57 |
| tristan | Maybe starting with 1.2, and then 1.4, 1.6, 1.8... like that... if there is no interest in 1.0 | 12:57 |
| tristan | jjardon, the point is that in a stable version of BuildStream we guarantee cache key stability | 12:57 |
| tristan | jjardon, and people want to be able to reproduce exactly the same artifact with the same key 10 years later | 12:58 |
| tristan | So the only thing we can do for that, is say "Use the version of BuildStream which produces that cache key", and for us to say that; we have to at least ensure it keeps working | 12:58 |
| jjardon | tristan: Id say if you have the resources to maintain a stable branch, sure go for it. If not do not create false expectation creating stable branches and instead try all your users to use the latest | 12:59 |
| tristan | We dont really have to keep every version of BuildStream under LTS support for backporting every bug fix down to 1.0, but we can at least make sure it keeps working on new distros. | 12:59 |
| tristan | jjardon, We need the CI first | 12:59 |
| jjardon | (latest version will support all cache keys or there is a path to upgrade from older cache keys to the new ones) | 13:00 |
| tristan | It should be a no brainer that an old version of BuildStream still runs on a modern distro | 13:00 |
| jjardon | tristan: no when modern distros doesnt have python3 anymore | 13:00 |
| tristan | jjardon, there is a lot of questions around being able to do that; it's currently not feasible without a lot of work | 13:00 |
| jjardon | but only python4, for example | 13:00 |
| tristan | I dont think we'll ever see a distro that doesnt have python2 | 13:01 |
| jjardon | tristan: do you want a bet? :) | 13:01 |
| tristan | maybe, but let's burn that bridge when we cross it, it will be a long time from now | 13:01 |
| tristan | Mr homegrown lazy distro who doesnt care about being able to run python2 programs might | 13:02 |
| tristan | but not serious distros | 13:02 |
| jjardon | python2 is already not included by default in several modern distros. And if it included there are plan to remove it | 13:02 |
| jjardon | removing it from the archive will take more time, but it will happen | 13:02 |
| tristan | jjardon, I dont see why anyone would port an already working python2 application to python3 unless it really needs long term development | 13:06 |
| tristan | jjardon, which is why I would not predict say, GTK+2 ever disappearing, for probably at least another 10 years | 13:06 |
| jjardon | tristan: well, because python2 is not supported anymore upstream and one day you will get a bug nobody will fix? | 13:07 |
| jjardon | yeah, python2 will still be in that distros archive for a long time, but you say "I don't think we'll ever see a distro that doesnt have python2" | 13:08 |
| jjardon | Fedora for example is working on porting everything to python3: http://fedora.portingdb.xyz/ when that happen they will not have any strong reason to keep python2 in the archives | 13:10 |
| tristan | jjardon, the result of that will more probably be "Users can no longer run that program anymore" and "The maintainers of that program have moved on", it's highly unlikely to get a port; so it's unreasonable for distros to stop supporting something (like a python2 app) unless their user base really doesnt need it anymore (there is new software which replaces it) | 13:10 |
| tristan | anyway, this is quite off topic by now, and I need to get food before restaurants start closing shop | 13:11 |
| * jjardon goes for quesadilla XD | 13:12 | |
| *** tristan has quit IRC | 13:21 | |
| Nexus | i can't tell if my laptop is terrible or if i'm getting hanging tests | 13:22 |
| *** tristan has joined #buildstream | 13:25 | |
| *** bethw has joined #buildstream | 13:25 | |
| jjardon | Should arpy be added to setup.py ? | 13:37 |
| tlater | jjardon: No, it's an optional dependency | 13:46 |
| tlater | I.e., it is only used for a specific plugin | 13:46 |
| jjardon | rigth | 13:46 |
| tlater | Nexus: Rebase on top of skullman's branch and find out :) | 13:47 |
| tlater | Nexus: https://gitlab.com/BuildStream/buildstream/merge_requests/375 | 13:47 |
| Nexus | it's just reaaaaallly slow | 13:52 |
| gitlab-br-bot | buildstream: merge request (jmac/logfile-widget-correction->master: Correct log line if logdir is empty) #377 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/377 | 13:53 |
| jjardon | Is needed to have a gcc compiler on the host for the buildstream tests? | 14:01 |
| tlater | jjardon: Not as far as I am aware, just as buildstream the buildstream tests never touch host tools | 14:02 |
| tlater | Well, unless you're playing with specific plugins. | 14:03 |
| gitlab-br-bot | buildstream: merge request (jmac/logfile-widget-correction->master: Correct log line if logdir is empty) #377 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/377 | 14:03 |
| jjardon | tlater: you will touch host tools in that case? | 14:03 |
| tlater | jjardon: Arpy is used to extract debs, for example | 14:03 |
| tlater | Host tools never contribute to actual builds, though | 14:04 |
| jjardon | ok, what aout patch, findutils, lzip, bzr ... Im seeing in the docker image ? | 14:05 |
| tlater | Also used for various extraction purposes :) | 14:05 |
| tlater | `patch` is used to apply patches to sources | 14:05 |
| tlater | gcc is used exclusively to build ostree | 14:06 |
| tristan | jjardon, those are optional dependencies, so we include them in the docker image to test the optional functionality | 14:06 |
| tristan | tlater, gcc might be required for psutils, actually; that is a very strange one | 14:06 |
| jjardon | tristan: sure, but are they host tools used in the build process ? | 14:06 |
| tristan | jjardon, they are host tools used in the process of obtaining sources | 14:06 |
| tristan | not in the build process | 14:07 |
| jjardon | should we document this somewhere? the fact that we are using "patch" from the host can be relevant? | 14:07 |
| tristan | gcc is a strange thing actually; for pip install to pass, if the host does not have python psutils, the host needs a C compiler, because psutils compiles C code as a part of it's installation process | 14:07 |
| tristan | jjardon, I think we should document it in the relevant plugins | 14:08 |
| jjardon | yeah, Im thinking on installing those libraries with pip, then remove gcc from the image | 14:08 |
| tristan | e.g. here: http://buildstream.gitlab.io/buildstream/sources/patch.html#module-sources.patch | 14:08 |
| tristan | possibly in installing.rst, we should link to http://buildstream.gitlab.io/buildstream/main_authoring.html#plugins, mentioning that depending on the project, you may need additional host tools | 14:10 |
| tristan | not sure, it would be a nicer experience to mention some things in installing.rst, but it would not be very maintainable to document plugin requirements in a central location | 14:11 |
| jjardon | I see | 14:11 |
| tristan | in any case; the user is supposed to get an error indicating what is needed *immediately* when running a command which requires a host tool | 14:11 |
| tristan | at least, very very early in the load process | 14:11 |
| tristan | so, you cannot find out that you need patch after 30 mins of building | 14:12 |
| jjardon | ok; let me see first what is actually plugin deps/tools to build pip packages/real depencies first | 14:12 |
| tristan | for patch specifically, we should have a harder requirement than "is patch there", this is a subtle bug I think - various implementations of patch have different quirks | 14:14 |
| tristan | gnu patch is very permissive with fuzzy features, but allowing fuzzy patch can have bad results, too | 14:14 |
| jjardon | yup | 14:15 |
| jjardon | I guess the git used is from the host as well? | 14:16 |
| jjardon | also tar? | 14:17 |
| gitlab-br-bot | buildstream: merge request (jjardon/update_depencies->master: docs: Update list of dependecies) #373 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/373 | 14:20 |
| gitlab-br-bot | buildstream: merge request (jjardon/update_depencies->master: docs: Update list of dependecies) #373 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/373 | 14:20 |
| tlater | jjardon: Tar actually depends, some of that is done by python | 14:21 |
| gitlab-br-bot | buildstream: issue #353 ("Document what parts of Buildstream / plugins depend on host packages") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/353 | 14:22 |
| tlater | juergbi: If we're going with the systemd-style quota definitons, do we want to adopt their 'infinity' and percentages of the full disk size as well? | 14:42 |
| * tlater checked humanfriendly and they aren't particularly specific about their parsing | 14:43 | |
| tlater | So I'll go with the suggested systemd implementation, even if it doesn't use SI units (grr...) | 14:43 |
| *** bethw has quit IRC | 14:44 | |
| *** hergertme has quit IRC | 14:58 | |
| juergbi | tlater: the systemd reference was just a suggestion, we don't have to follow it to the letter | 14:59 |
| tlater | Hm, I don't like it particularly much, personally | 14:59 |
| juergbi | but avoiding unnecessary differences with other config files is generally useful | 14:59 |
| juergbi | for storage, SI units might indeed make more sense | 15:00 |
| * tlater thinks it's handy to be able to say "please stick to this well-known format" | 15:00 | |
| juergbi | (it's somewhat different for something like RAM) | 15:00 |
| juergbi | it definitely is | 15:00 |
| tlater | And I think there's actually value in percentages for this | 15:00 |
| tlater | Just perhaps not 'infinity' | 15:01 |
| juergbi | infinity would just be no quota, wouldn't it? | 15:01 |
| tlater | That doesn't help very much, as it's guaranteed to crash buildstream during a build once we grow beyond disk space | 15:02 |
| tlater | Although I suppose we could still allow that? | 15:02 |
| juergbi | I suppose it depends whether we will add the second option that tristan suggested | 15:03 |
| tlater | Ah, right | 15:03 |
| juergbi | in which case cache-quota itself could default to infinity | 15:03 |
| juergbi | although tristan suggested default to a large number instead, but that's also somewhat odd | 15:03 |
| tlater | Leaving 'infinity' headroom makes absolutely no sense though x) | 15:03 |
| juergbi | agreed, that would mean never cache anything, which would fail builds | 15:03 |
| tlater | Yeah, I think defaulting to 'infinity' and disallowing 'infinity' on headroom would make this quite neat | 15:04 |
| juergbi | fine with me | 15:04 |
| tlater | Percentages are also relatively easy to do, so I'll just add those for convenience | 15:04 |
| *** Prince781 has joined #buildstream | 15:05 | |
| * juergbi is still worried about the performance impact | 15:06 | |
| *** bethw has joined #buildstream | 15:07 | |
| tlater | Yeah, I'll try using jmac's performance testing branch if I can | 15:07 |
| juergbi | sounds good | 15:08 |
| tlater | Otherwise I'll just run some tests on sample repositories | 15:08 |
| * tlater is more concerned about communication with the main thread | 15:08 | |
| juergbi | it will require a somewhat populated cache, though. it will be fast with small caches, of course | 15:08 |
| tlater | Yep, running these tests might take a while on this machine, too | 15:08 |
| * tlater considers doing this overnight on his home PC or something | 15:09 | |
| jjardon | about psutil: should we install it from the package manager or install it from pip? (that will require gcc and python-devel) | 15:12 |
| tlater | jjardon: In a recent discussion we decided "Whichever works best" | 15:13 |
| tlater | It's a bit vague, though usually we've sticked to pip | 15:14 |
| tlater | Unless there was a reason not to use the pip version on the distro | 15:14 |
| jjardon | then we should have to update the install guide to add gcc and python-devel as base dependencies | 15:14 |
| * tlater suspects it's not too bad to stick to the OS package manager for psutil where it is available | 15:15 | |
| jjardon | I think for this case is better to use the distro package to avoid those dependencies | 15:15 |
| jjardon | ok | 15:15 |
| tristan | tlater, juergbi performance is indeed a big concern I think; running `du -hs` on an artifact cache takes *ages*, we should be able to have ostree just report a number (possibly cached, possibly immediately) | 15:27 |
| tlater | tristan: I'm caching a number currently (well, intend to, but multithreading got me) | 15:27 |
| tlater | It's just that that cache will continuously be outdated | 15:27 |
| tlater | So it's probably still a fairly large overhead | 15:28 |
| juergbi | yes, the main issue of caching this on disk is concurrency, I suppose | 15:28 |
| juergbi | atomic replacement is not sufficient for this | 15:28 |
| * tristan doesnt like to see the word multithreading again... | 15:28 | |
| tlater | *multiprocessing? | 15:28 |
| tristan | concurrency is a better general word indeed :) | 15:29 |
| tlater | I'll have to figure out concurrency for the current implementation before testing its performance then. | 15:30 |
| gitlab-br-bot | buildstream: merge request (jjardon/update_depencies->master: docs: Update list of dependecies) #373 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/373 | 15:31 |
| tristan | tlater, juergbi maybe splitting up build + caching of artifacts into separate queues might let you synchronize some things | 15:32 |
| tristan | just a thought | 15:32 |
| tlater | Hmm | 15:32 |
| tlater | That might also allow pruning multiple things at the same time | 15:32 |
| juergbi | that might conflict with possible future on-demand staging | 15:32 |
| tristan | eh, not if you make builds contingent on the artifacts being cached | 15:33 |
| *** bethw has quit IRC | 15:33 | |
| tristan | although | 15:33 |
| tristan | juergbi, I still feel like https://gitlab.com/BuildStream/buildstream/issues/296 (lightweight artifacts) is the right thing architecturally speaking | 15:34 |
| tristan | not only as a matter of slight optimization | 15:34 |
| *** toscalix has quit IRC | 15:34 | |
| tristan | if it's not a huge overhead I dont really mind leaving things as is, but it does seem backwards the way we force everything to be an artifact before extracting it | 15:35 |
| juergbi | with on-demand staging lightweight artifacts would feel backward, imo | 15:36 |
| tlater | The artifact cache is running into more and more database issues ;) | 15:36 |
| tristan | juergbi, why ? you would just on-demand stage from the extract directory | 15:36 |
| *** Prince781 has quit IRC | 15:36 | |
| tristan | with a possible on-demand stage from a backing artifact storage, if you happen to be interacting with something remote | 15:37 |
| *** bethw has joined #buildstream | 15:37 | |
| juergbi | supporting both should be possible but it would add complexity | 15:37 |
| juergbi | we might need it anyway for workspaces, not sure | 15:38 |
| tristan | while it does feel backwards, the good argument I think is that, with ostree at least, forcing: Build result -> ostree -> extract -> hardlink to next build... | 15:39 |
| tristan | means we get to leverage deduplication | 15:39 |
| tristan | I guess that is the best argument for keeping it the way it is | 15:39 |
| juergbi | yes and we don't need a separate expiration mechanism etc | 15:40 |
| juergbi | fewer structural differences with remote execution | 15:41 |
| tristan | unrelated, but https://gitlab.com/BuildStream/buildstream/issues/294 is reaaaaaaly bad :-( | 15:41 |
| juergbi | and the overhead of commit shouldn't be very high | 15:41 |
| tristan | 5 minutes of caclulating cached state, wow | 15:41 |
| juergbi | you saw my (old) comment, right? https://gitlab.com/BuildStream/buildstream/issues/294#note_62585246 | 15:41 |
| tristan | for a project as small as gnome-build-meta | 15:41 |
| juergbi | well, it includes whole freedesktop-sdk via junction now | 15:41 |
| tristan | I dont think it does no | 15:42 |
| tristan | it does on someones WIP branch | 15:42 |
| tristan | right, time is being spent in ruamel.yamk | 15:43 |
| tristan | yaml | 15:43 |
| tristan | juergbi, maybe we should bit Angelos's bullet and do some of the heavy-weight internal stuff with json though the faster C library | 15:43 |
| tristan | *bite | 15:43 |
| * tristan finds it rather silly that we use parsers implemented in interpreted languages :-S | 15:44 | |
| jjardon | Does this look legit? https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/32 | 15:44 |
| juergbi | we need to read very little, so if those parts were in a separate file, I wouldn't expect a performance issue in that aspect | 15:45 |
| * tristan will have to think of it in the day time | 15:46 | |
| *** Prince781 has joined #buildstream | 15:46 | |
| skullman | I don't suppose someone could take a quick look at https://gitlab.com/BuildStream/buildstream/merge_requests/375 would be nice to be able to base work off it so the tests pass rather than hang. | 15:55 |
| *** Prince781 has quit IRC | 16:02 | |
| gitlab-br-bot | buildstream: merge request (jjardon/ci_fedora_27->master: Run test in Fedora27, apart from Debian 8) #378 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/378 | 16:07 |
| jjardon | tristan: something like that? ^ | 16:08 |
| tristan | jjardon, nice ! | 16:11 |
| tlater | jjardon: You need to be careful about removing bwrap/ostree in the unix tests | 16:12 |
| jjardon | tristan: more to common: Debian stable, Fedora 28 ,maybe Ubuntu | 16:12 |
| tlater | That's the main reason the CI isn't currently implemented like that, even though we have the images and everything ready | 16:12 |
| jjardon | tlater: Im not touching the unix test job (for now) | 16:12 |
| tlater | Though that yaml stuff is really neat :) | 16:12 |
| tlater | Ah, sorry | 16:12 |
| *** Prince781 has joined #buildstream | 16:12 | |
| tlater | I missed the diff break thing | 16:13 |
| jjardon | :) | 16:13 |
| tristan | it's esthetically yucky that the tests are in alphabetical order: https://gitlab.com/BuildStream/buildstream/pipelines/20276042 | 16:13 |
| tristan | tiny nitpick, maybe we could use a common prefix so that docs doesnt get mixed up in there | 16:13 |
| *** valentind has joined #buildstream | 16:13 | |
| jjardon | any reason why the debian docker image was called testsuite-debian instead buildstream-debian (as buildstream-fedora) | 16:14 |
| tlater | jjardon: I think the fedora image also comes with buildstream pre-installed | 16:14 |
| tlater | Yeah | 16:14 |
| jjardon | tlater: yes | 16:14 |
| tlater | fedora has buildstream pre-installed, whereas the debian image doesn't | 16:14 |
| jjardon | rigth | 16:14 |
| tlater | We wanted to create a second fedora image that doesn't come with buildstream installed | 16:14 |
| tlater | So that we can have nice CI images | 16:14 |
| tlater | And one to use for bst-here | 16:14 |
| jjardon | Any reason to not do that? | 16:15 |
| tlater | Time, mostly | 16:15 |
| tlater | x) | 16:15 |
| jjardon | oh! :) | 16:16 |
| jjardon | tristan: do you like test-debian-8, test-unix, test-fedora-27 more? | 16:17 |
| tristan | skullman, made just a little comment, I think the code looks perfectly sane indeed | 16:17 |
| tristan | had to check a moment what we are doing in terminator() | 16:17 |
| tristan | skullman, I think we have a few context managers which fail to try/except, resulting in weird behavior, at least it was important in a few which I recently fixed | 16:18 |
| tristan | jjardon, yeah that would be nice :) | 16:19 |
| tristan | jjardon, really just a tiny silly unimportant nitpick though :) | 16:19 |
| tristan | will still make pipelines look sexier | 16:19 |
| gitlab-br-bot | buildstream: merge request (jjardon/ci_fedora_27->master: Run test in Fedora27, apart from Debian 8) #378 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/378 | 16:20 |
| jjardon | tristan: let make it sexier | 16:20 |
| jjardon | lets | 16:20 |
| jjardon | I will do current Debian stable next, then maybe we should do Ubuntu | 16:21 |
| tristan | ubuntu is very debian like, but it wont hurt | 16:22 |
| tristan | I'm more interested in doing bsd, but that requires actually writing the platform for it :) | 16:22 |
| skullman | tristan: there's a test in !329 that will trigger the hang without the patch. It could be ported in, but I'm not sure it's worth the effort if it'd already be covered by something else after !329 is merged | 16:24 |
| tlater | Hm, I wonder if there's value in having a cache quota that refuses to ever delete artifacts | 16:25 |
| tristan | cs_shadow, what's your gitlab handle again ? | 16:25 |
| gitlab-br-bot | buildstream: merge request (jjardon/ci_fedora_27->master: Run test in Fedora27, apart from Debian 8) #378 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/378 | 16:25 |
| tlater | I.e., it will complain if you run out of disk space, rather than trying to delete artifacts if you set it to 'infinity' | 16:25 |
| tristan | ah it's a dash ! | 16:26 |
| tlater | Actually, no, that should be added with the future prompting feature | 16:26 |
| cs_shadow | Yep, cs-shadow | 16:26 |
| cs_shadow | (GitHub didn’t used to allow underscores) | 16:27 |
| tristan | I like dashes better in many cases anyway :) | 16:29 |
| tristan | they look more balanced | 16:29 |
| laurence | cs_shadow, aha, was just responding to the mail, all sorted now for the access though i see :) | 16:30 |
| tlater | Does anyone happen to know where we test userconfig settings? I can't find anything suchlike :| | 16:31 |
| jjardon | tristan: if you fancy reviewing the sexy version is all yours: https://gitlab.com/BuildStream/buildstream/merge_requests/378 | 16:32 |
| tristan | tlater, we test more about how the app behaves under different settings, scattered throughout tests | 16:32 |
| * jjardon goes to see the football | 16:32 | |
| tlater | Hm, right | 16:32 |
| tristan | tlater, but we have the original load test with is tests/context.py | 16:32 |
| tlater | o/ jjardon | 16:32 |
| tristan | tlater, which has to be thrown in the garbage if unneeded, or ported to proper CLI test for what we can salvage | 16:33 |
| tlater | Hm, I'd like to write something that asserts that my parser works properly, without going through the entirety of a pipeline | 16:33 |
| tristan | I ported over the remaining project tests | 16:33 |
| tristan | I think besides that we still have loader tests which need to be ported | 16:34 |
| tristan | tlater, I'm using `bst workspace list` for that sort of thing, if possible | 16:34 |
| tristan | it's the shortest codepath I think, but I dont know if you can assert anything meaningful with it | 16:35 |
| tlater | Ah, I guess that should do it, yeah | 16:35 |
| tristan | tlater, `bst show` is usually the way to go, with a one element pipeline | 16:35 |
| tlater | Perhaps a `bst show --format=%{config}` could be useful? | 16:35 |
| tristan | uhhh | 16:35 |
| tristan | nah, what's it doing ? | 16:35 |
| tlater | Show the user how buildstream interprets their config | 16:35 |
| tristan | tlater, if you are adding meaningful config, it means you have to test how bst behaves under that config | 16:36 |
| tristan | which means you are already implicitly testing the config, right ? | 16:36 |
| tlater | Yeah, that's true, but I'm concerned with invalid values for that config. | 16:36 |
| tristan | that you can use `bst workspace list` for | 16:37 |
| * tlater will go with that, ta tristan :) | 16:37 | |
| tristan | and result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason...) | 16:37 |
| cs_shadow | tristan: thanks! It seems like the MRs on buildstream-docker-images repo also require an approval. Do I need to be added to the approvers group too? (You can see settings on https://gitlab.com/BuildStream/buildstream-docker-images/edit) | 16:38 |
| tristan | I've never used "approvers" and I dont really care about what they are for | 16:39 |
| tristan | I think jjardon likes them :) | 16:39 |
| tristan | cs_shadow, I've set you as "Master" so you should be able to do what you want | 16:39 |
| tristan | I hope you dont get bogged down in pesky approvers :) | 16:39 |
| tristan | cs_shadow, if you cannot edit the settings, let me know what you want me to do there for you | 16:40 |
| tristan | I might only get to it tomorrow, but it's your call really, if you like approvers, you can have em :) | 16:40 |
| gitlab-br-bot | buildstream: merge request (richardmaw/fix-unit-test-hang-on-crash->master: Fix unit tests hanging when there's an exception in a sandbox run) #375 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/375 | 16:41 |
| cs_shadow | tristan: thanks! I'll familiarize myself with the settings and get back to you tomorrow then :) | 16:42 |
| tristan | cs_shadow, my personal take on "approvers" in general, is that it is a way for people to push things upstream without assuming any kind of liability for their merges | 16:43 |
| tristan | people get to "share the blame", and as such, tend to get nonchallant | 16:43 |
| tristan | so I dont like approvers :) | 16:43 |
| *** dominic has quit IRC | 16:45 | |
| *** aday has quit IRC | 16:46 | |
| *** Prince781 has quit IRC | 16:49 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:56 |
| *** aday has joined #buildstream | 17:00 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 17:02 |
| *** jonathanmaw has quit IRC | 17:08 | |
| *** Prince781 has joined #buildstream | 17:22 | |
| *** Prince781 has quit IRC | 18:08 | |
| *** bethw has quit IRC | 18:26 | |
| *** bethw has joined #buildstream | 19:37 | |
| *** tristan has quit IRC | 19:45 | |
| *** hergertme has joined #buildstream | 20:02 | |
| *** jsgrant has joined #buildstream | 20:14 | |
| *** Prince781 has joined #buildstream | 20:50 | |
| *** Prince781 has quit IRC | 20:54 | |
| *** Prince781 has joined #buildstream | 20:54 | |
| *** jsgrant has quit IRC | 20:55 | |
| *** jsgrant has joined #buildstream | 21:10 | |
| *** bethw has quit IRC | 21:21 | |
| *** aday has quit IRC | 21:30 | |
| *** jsgrant has quit IRC | 21:47 | |
| *** Prince781 has quit IRC | 21:48 | |
| gitlab-br-bot | buildstream: merge request (issue-21_Caching_build_trees->master: WIP: Issue 21 caching build trees) #372 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/372 | 23:20 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!