| *** tristan has joined #buildstream | 07:04 | |
| *** ChanServ sets mode: +o tristan | 07:41 | |
| *** jonathanmaw has joined #buildstream | 08:18 | |
| tristan | jjardon[m], I tried reproducing it again to be doubly sure, btw | 08:37 |
|---|---|---|
| tristan | jjardon[m], fwiw, I used commands almost _exactly_ as you have them specified currently in https://gitlab.com/baserock/definitions/blob/jjardon/bst2/.gitlab-ci.yml#L19 | 08:38 |
| tristan | jjardon[m], except for the fact that I have a definitions/ybd already checked out in some other directory and already have the gits ybd wants mirrored in some location | 08:38 |
| tristan | I used defs2bst master... definitions repo master... | 08:39 |
| tristan | And ybd master actually (with some configuration file changes to point to my local gits mirror) | 08:40 |
| tristan | And not reproducing that missing bluetooth or other issue, seems to work well here | 08:40 |
| * tristan checks the gitlab, maybe it's passing now ? | 08:40 | |
| ironfoot | tristan: I believe we managed to solve the missing bluetooth issue | 08:41 |
| ironfoot | and it's fixed in that branch | 08:41 |
| tristan | Ah | 08:41 |
| tristan | I'm seeing an issue with using ostree now | 08:41 |
| tristan | permission denied on the runner for file extended attributes ? | 08:42 |
| tristan | https://gitlab.com/baserock/definitions/builds/14084258 | 08:42 |
| ironfoot | the missing bluetooth.bst problem was solved with: https://gitlab.com/baserock/definitions/commit/440e4ed35ba6b3704f188765262646941a042dd7 | 08:42 |
| tristan | that looks correct yeah | 08:42 |
| * tristan pushes some changes to the gnu-toolchain branch of buildstream-tests | 08:43 | |
| tristan | and rebases build-gnome on top of those | 08:43 |
| tristan | Now we dont depend on the entire GNOME SDK for building the base toolchain (instead we just use org.freedesktop.[BasePlaform/BaseSdk]) | 08:44 |
| tristan | which should be much less downloading | 08:45 |
| ironfoot | kewl | 08:45 |
| jjardon[m] | tristan yep about the bluethoot problem I filled https://gitlab.com/BuildStream/defs2bst/issues/1 | 08:45 |
| ironfoot | regarding permissions for file extended attributes... if that's the case, we could try to enable that in our runners | 08:47 |
| ironfoot | iiuc, it's possible to add some privileges to the runner. It's also possible to use gitlab runner binary to test .gitlab-ci locally without pushing branches | 08:48 |
| tristan | jjardon[m], :-S | 08:52 |
| tristan | That should not be | 08:52 |
| tristan | jjardon[m], the output should really be going into the elements/ subdir | 08:56 |
| tristan | And I'm trying it here, and it's working, something else is up | 08:56 |
| tristan | And indeed | 08:57 |
| tristan | After your change, it produces an undesirable conversion | 08:57 |
| tristan | i.e. look at your `find` output | 08:57 |
| tristan | buildstream-tests/sound-server-pulseaudio.bst | 08:57 |
| tristan | buildstream-tests/weston-common | 08:57 |
| tristan | buildstream-tests/weston-common/weston.bst | 08:57 |
| tristan | buildstream-tests/dlna-services | 08:57 |
| tristan | buildstream-tests/dlna-services/rygel.bst | 08:57 |
| tristan | buildstream-tests/dlna-services/gupnp-dlna.bst | 08:57 |
| tristan | buildstream-tests/dlna-services/gssdp.bst | 08:57 |
| tristan | buildstream-tests/dlna-services/gupnp.bst | 08:58 |
| tristan | buildstream-tests/dlna-services/gupnp-igd.bst | 08:58 |
| tristan | buildstream-tests/dlna-services/gupnp-av.bst | 08:58 |
| tristan | ... | 08:58 |
| tristan | Instead of putting all the elements into buildstream-tests/elements, its making the root directory full of elements | 08:58 |
| ironfoot | hm.. I suggested that change. These were the instructions that were failing: https://gitlab.com/baserock/definitions/blob/768d4adfc8e58a31ec7baf8e66a76c4e5b282374/.gitlab-ci.yml#L30 | 08:58 |
| tristan | ironfoot, in that case, you should only have to change: bst -C buildstream-tests build elements/gnome/gnome-system-x86_64-content.bst | 08:59 |
| tristan | To be bst -C buildstream-tests build gnome/gnome-system-x86_64-content.bst | 08:59 |
| jonathanmaw | tristan: looking at source.py, the track() method doesn't mention what it raises. Should I assume that it might raise a SourceError? | 08:59 |
| tristan | ironfoot, elements are interpreted as "project element directory relative paths" | 08:59 |
| tristan | jonathanmaw, Yes | 09:00 |
| tristan | jonathanmaw, it can possibly raise other errors though, when using some of BuildStream's internal helper methods | 09:00 |
| tristan | jonathanmaw, from the point of view of a Source plugin, you should only really be concerned with raising SourceError | 09:01 |
| ironfoot | tristan: but that exactly is what jjardon[m] tried before: https://gitlab.com/baserock/definitions/blob/dc2986ae68f18dd708f0bc8a7ce49e5eda8c2ef0/.gitlab-ci.yml#L30 | 09:01 |
| tristan | ironfoot, so because of this line: https://gitlab.com/BuildStream/buildstream-tests/blob/gnu-toolchain/project.conf#L12 | 09:02 |
| tristan | everything should be interpreted as elements/ relative | 09:02 |
| ironfoot | right, that flag is new to me | 09:02 |
| tristan | ironfoot, *that* commit should work | 09:03 |
| tristan | https://gitlab.com/baserock/definitions/blob/dc2986ae68f18dd708f0bc8a7ce49e5eda8c2ef0/.gitlab-ci.yml#L30 | 09:03 |
| tristan | That should definitely work, if not, something I'm not aware of is going on | 09:03 |
| tristan | that is in fact working fine here | 09:04 |
| * ironfoot re-triggers the pipeline: https://gitlab.com/baserock/definitions/builds/14109914 | 09:04 | |
| tristan | ironfoot, was that pipeline triggered with --output buildstream-tests/elements ? | 09:11 |
| tristan | because... that worked | 09:11 |
| tristan | might be nice to add `bst show --deps all gnome/gnome-system-x86_64-content.bst` in advance of building | 09:12 |
| ironfoot | tristan: well, your theory was correct :) | 09:13 |
| ironfoot | although I don't understand. that flag in the config file was there yesterday | 09:14 |
| tristan | I dont understand how it failed before either | 09:14 |
| ironfoot | anyway, it's working now | 09:14 |
| ironfoot | jjardon[m]: this version is the one that is right: https://gitlab.com/baserock/definitions/blob/dc2986ae68f18dd708f0bc8a7ce49e5eda8c2ef0/.gitlab-ci.yml#L30 | 09:15 |
| tristan | now just have to fix that xattrs issue | 09:17 |
| tristan | seems quite strange though, there is nothing ultra privileged about what ostree is doing here | 09:17 |
| ironfoot | I've seen the same error before, when trying buildstream in docker. When acceessing the same container twice at the same time, one building, and the other trying to do a `show` | 09:27 |
| ironfoot | so, really odd error, I don't think my comment gives any clues | 09:28 |
| tristan | yeah, seems to be docker related ? gitlab runners also use docker ? | 09:28 |
| tristan | ironfoot, I looked at your docker docs patch by the way, and looks generally good | 09:29 |
| tristan | What I've been thinking... is I'd like to have a "Installing and Running" section around there | 09:29 |
| tristan | And the docker docs would fit better as a section of that | 09:30 |
| tristan | (currently it looks like a promotion for using docker, like that is the primary way buildstream should be run) | 09:31 |
| ironfoot | yes, gitlab runners are also docker containers (i believe) | 09:41 |
| ironfoot | i may be wrong here though | 09:41 |
| * tristan just remarked that because he quite often runs `bst show` on a project while it's building and never ran into that | 09:42 | |
| ironfoot | tristan: I agree with your suggestion for the docs. I didn't want to modify lots of things without having your input | 09:42 |
| tristan | yeah, I should probably just merge your patch as a first step, though | 09:42 |
| tristan | Also, I want to look into generating man pages and docs for the CLI itself, but I'm not sure exactly how to do that with Click | 09:43 |
| *** franred has joined #buildstream | 09:45 | |
| jjardon[m] | runners use docker, yes | 09:46 |
| tristan | still, it seems ironfoot was able to use docker without encountering that error | 09:47 |
| tristan | but then, if the issue has to do with concurrency + docker (i.e. `bst show` triggered it ?), then it may be happening at random | 09:48 |
| ironfoot | it might be that the runners do this kind of thing when using docker | 09:48 |
| ironfoot | one session to run commands, other to fetch the logs, etc, (i don't have a clue) | 09:49 |
| jjardon[m] | Maybe the underlying is that is runners the docker ages is related? I will check and see if I can use something else (its coreos atm) | 09:49 |
| *** ssam2 has joined #buildstream | 09:50 | |
| tristan | ironfoot, but buildstream will do a lot of concurrent ostree anyway, just have to fall on some part where artifacts are getting committed/extracted simultaneously | 09:50 |
| ironfoot | we could do with some investigation :_) | 09:54 |
| *** Chris has joined #buildstream | 09:54 | |
| *** Chris is now known as chrispolin | 09:55 | |
| tristan | ironfoot, I dont know, operation not supported looks like an fs thing | 09:55 |
| * tristan thinks he misread that as "operation not permitted" before | 09:56 | |
| jonathanmaw | Turns out I'm an idiot. I had some engimatic errors which turned out to be because I didn't notice that class variables and class methods use the same namespace, so I can't have self.track() the method, and self.track the variable! | 11:51 |
| tristan | jonathanmaw, happened to me before :) | 11:52 |
| * tristan is trying to hook up man page generation into setup.py but it's a pain :-/ | 11:53 | |
| jonathanmaw | tristan, can you give me some insight into how I'm breaking things here https://pastebin.com/K4KU7Y7G | 12:04 |
| jonathanmaw | I can see what's probably the offending line | 12:05 |
| jonathanmaw | but I don't see the reason why it's broken | 12:05 |
| *** brejoc has joined #buildstream | 12:05 | |
| jonathanmaw | It looks like the error message that got through was "Unknown exception in SIGCHLD handler" | 12:05 |
| tristan | yeah | 12:06 |
| tristan | jonathanmaw, you already downloaded the latest master from this weekend right ? | 12:06 |
| jonathanmaw | tristan: I don't think so | 12:07 |
| tristan | Oh | 12:07 |
| * jonathanmaw is not used to fast-moving projects :/ | 12:07 | |
| tristan | jonathanmaw, rebase | 12:07 |
| jonathanmaw | erm, maybe from the weekend, not sure about monday. | 12:07 |
| tristan | jonathanmaw, I believe I fixed at least how the frontend is handling failures from sources this weekend | 12:07 |
| tristan | I think it was already there monday though | 12:07 |
| jonathanmaw | tristan: looks like the only commit I was missing was "main.py: Custom initial loading feedback when not connected to a tty" | 12:08 |
| tristan | Ok I'll try to fix it better | 12:08 |
| tristan | yeah that's not it | 12:08 |
| tristan | jonathanmaw, there are 2 things going on here | 12:09 |
| jonathanmaw | tristan: when I run bst with --no-interactive, the traceback is less huge, but doesn't seem any more enlightening, btw. I even lose the "Unknown exception in SIGCHLD handler" | 12:09 |
| tristan | the part from the frontend doing that traceback is what I'm talking about, which needs some additional fixing | 12:10 |
| tristan | the source of the bug I believe is with open(self._get_mirror_file(), 'w') as f: | 12:10 |
| tristan | jonathanmaw, to explain a bit further, we expect that any errors from plugins get normalized into a SourceError (or any _BstError in fact) | 12:11 |
| tristan | That ensures that if there is something which can go wrong, the user will get an intelligible error message for it | 12:11 |
| tristan | normalized exceptions, will show up in the output log as an ERROR | 12:12 |
| jonathanmaw | tristan: ok, so it might have been a type of error I wasn't expecting. I should expand my except statements to be broader, and then I ought to get better logging. | 12:12 |
| tristan | instead of throwing an ugly stack trace | 12:12 |
| tristan | jonathanmaw, the problem with the "open" context managers are that they will throw IOError or OSError (forget which) exceptions | 12:13 |
| jonathanmaw | aha, I was getting a FileNotFoundError | 12:13 |
| tristan | right, or that | 12:13 |
| tristan | python3 | 12:13 |
| tristan | so what's happening I think, is that the leading directories to the file you want to create are missing | 12:13 |
| tristan | note however, that both the self.get_mirror_directory() method... _and_ the self.tempdir() context manager I pointed at the other day, will ensure that those returned directories exist so you dont have to | 12:15 |
| jonathanmaw | tristan: yep. though it looks like urllib lets you download files straight into memory, as well as to a filename of your choosing. | 12:16 |
| jonathanmaw | which at least saves me opening it up to read it. | 12:17 |
| jonathanmaw | I hope it is downloading to memory, instead of an anonymous temporary file somewhere, but I couldn't find anything about whether it uses a temporary file or not. | 12:18 |
| tristan | Is it desirable to download the whole file in memory ? | 12:18 |
| jonathanmaw | hrm | 12:18 |
| jonathanmaw | it saves me downloading it only to open it up | 12:18 |
| tristan | I think you want with self.tempdir(): | 12:18 |
| tristan | and download it to a file in that dir | 12:18 |
| tristan | afterwards, use shutil.move() or os.rename() to move it into the desired location in self.get_mirror_directory() | 12:19 |
| tristan | That is guaranteed anyway to be on the same filesys | 12:19 |
| tristan | so there is no copy | 12:19 |
| tristan | jonathanmaw, Ah I think I get your meaning by "it saves me downloading it only to open it up", because you want to do the sha256sum on it without downloading it ? | 12:20 |
| tristan | That is rather moot I think in this case | 12:20 |
| jonathanmaw | tristan: I don't know of any way to sha256sum without downloading | 12:21 |
| jonathanmaw | it's more about guessing it'd be marginally more efficient | 12:21 |
| tristan | jonathanmaw, you will anyway want to mirror it, so for the tarball case, fetch() and track() both just have to ensure that it's mirrored | 12:21 |
| tristan | So I guess, what you want is a.) ensure_mirror(uri) to download it if it's not there; b.) track() will call that and then do the sha256sum... c.) fetch() will also call that and raise a SourceError if the sha256sum specified in self->ref does not match that of the downloaded tarball | 12:23 |
| jonathanmaw | hrm, python docs recommend not using tempfile.mktemp(), which returns a pathname that did not exist when the call was made, but is insecure for making temporary files because something else might create a file with that name before you use it. | 12:43 |
| jonathanmaw | Since I want an anonymous name for where I'm downloading the file to, that'll do, however | 12:43 |
| jonathanmaw | unless there's a specific function just for generating anonymous filenames. | 12:43 |
| jonathanmaw | aha! it doesn't matter, since self.tempdir gives me the insulation I need. | 12:46 |
| tristan | yeah you want self.tempdir() context manager | 12:51 |
| tristan | if there is really an issue with tempfile module, then we should be able to fix it entirely in one place; within Plugin.tempdir() | 12:52 |
| tristan | Ah right, that's not an issue, as we use mkdtemp internally | 12:53 |
| tristan | no racing | 12:53 |
| *** brejoc has quit IRC | 13:25 | |
| jonathanmaw | \o/ got a plugin that seems to work for me https://gitlab.com/jonathanmaw/buildstream/commit/8f51d91ce04d166c5cfb74e2bf5e1ce24841ae75 | 13:56 |
| jonathanmaw | now to figure out the tests and make sure I actually have all the bases covered | 13:56 |
| tristan | jonathanmaw, :D | 14:02 |
| jonathanmaw | tristan, is there a way to run buildstream's test suits only for one set of tests? | 14:11 |
| jonathanmaw | i.e. if I created tests/sources/tar.py, would I be able to have it run only for those tests? | 14:11 |
| tristan | jonathanmaw, I dont know how, I suspect pytest may have some way | 14:12 |
| tristan | ./setup.py test --addopts [...] adds options to pytest | 14:12 |
| ssam2 | `py.test -k pattern` runs only tests whose name matches 'pattern' | 14:14 |
| tristan | So ./setup.py test --addopts -k pattern "should" do it | 14:15 |
| tristan | it may require quotes | 14:15 |
| tristan | Anyway, the tests take only 10 or 20 seconds to complete I think | 14:15 |
| tristan | jonathanmaw, so I think that A.) get_unique_key() should include the unresolved origin_url | 14:16 |
| tristan | and B.) take a look at git.py's implementation of get_consistency() | 14:17 |
| tristan | I think for the right experience, consistency should be CACHED if the ref exists (i.e. ref is not None) | 14:17 |
| tristan | RESOLVED if the tarball is downloaded _and_ the ref exists | 14:18 |
| tristan | otherwise INCONSISTENT | 14:18 |
| tristan | Oh sorry I got that backwards | 14:18 |
| tristan | RESOLVED if ref is not None, and CACHED if ref is not None _and_ tarball is downloaded | 14:18 |
| *** tristan has quit IRC | 14:23 | |
| *** tristan has joined #buildstream | 14:30 | |
| *** ChanServ sets mode: +o tristan | 14:31 | |
| jonathanmaw | tristan: in get_unique_key, I previously stuck with the sha256sum because sha256sum collisions are rare, and if the file is identical it oughtn't to make any difference. | 15:00 |
| jonathanmaw | are these assumptions correct? | 15:00 |
| *** franred has quit IRC | 15:02 | |
| tristan | jonathanmaw, it cant hurt | 15:09 |
| tristan | to include it | 15:10 |
| tristan | but | 15:10 |
| tristan | jonathanmaw, what is more important to me, than which way is ultimately more correct, is that all of the plugins we do include do it the same way | 15:10 |
| jonathanmaw | tristan: I see. works for me. | 15:15 |
| tristan | jonathanmaw, note that the test case structure for sources predates the existence of the pipeline, as such it's possible that the fixture there is more unwieldy than it needs to be | 15:15 |
| tristan | We will want to add a few more sources soon, starting with hg and bzr | 15:16 |
| tristan | jonathanmaw, if you judge that you will save time overall by reworking, and possibly simplifying that fixture stuff, then please feel free to go ahead :D | 15:17 |
| * tristan thinks there is a lot of potential to restructure and simplify the test cases now that the core is pretty much all written out, but wont be spending time doing that while more pressing things need doing | 15:19 | |
| *** jonathanmaw has quit IRC | 17:07 | |
| *** brejoc has joined #buildstream | 17:56 | |
| *** ssam2 has quit IRC | 18:09 | |
| *** tristan has quit IRC | 19:50 | |
| *** brejoc has quit IRC | 19:56 | |
| albfan[m] | tristan: flatpak usage for test suite ready See https://github.com/flatpak/flatpak/issues/699 | 22:38 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!