| *** mcatanzaro has quit IRC | 01:20 | |
| *** mcatanzaro has joined #buildstream | 01:20 | |
| *** mcatanzaro has quit IRC | 02:38 | |
| *** bochecha has quit IRC | 05:28 | |
| *** bochecha has joined #buildstream | 05:29 | |
| *** bochecha has quit IRC | 05:33 | |
| *** bochecha has joined #buildstream | 05:34 | |
| *** jude has joined #buildstream | 08:54 | |
| *** noah has joined #buildstream | 09:21 | |
| *** bethw has joined #buildstream | 09:26 | |
| *** jonathanmaw has joined #buildstream | 09:41 | |
| *** bethw has quit IRC | 09:50 | |
| *** tpollard has joined #buildstream | 10:05 | |
| tlater | bochecha: (If you are here) Currently you need to include a plugin from pip by specifying the plugin repository explicitly, probably 'bst-external:git-local' in your case | 10:05 |
|---|---|---|
| bochecha | tlater: I tried that | 10:06 |
| bochecha | reading the buildstream source code I figured something like that was needed | 10:06 |
| bochecha | I got further by specifying `BuildStream-external:git-local` | 10:06 |
| bochecha | but then it fails on requiring a .yaml file (source plugins usually don't have one?) | 10:06 |
| bochecha | I created an empty one, then it fails because it tries to import the buildstream.plugins.git_local module | 10:07 |
| tlater | Hm, that's odd | 10:07 |
| bochecha | at some point it completely ignores wherever the module comes from, and tries to import it from buildstream.plugins | 10:07 |
| bochecha | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_plugincontext.py#L133 | 10:07 |
| bochecha | here | 10:07 |
| tlater | The docker plugin works just fine for me, have you had a look at what sam did for that? | 10:08 |
| tlater | It's possible you're missing a __init__.py | 10:08 |
| tlater | This might help? https://gitlab.com/BuildStream/bst-external/tree/sam/docker-source/bst_external/sources | 10:09 |
| adds68 | Hi, when i run bst pull, in a directory, i'm getting an error "Error loading project: Could not find file at /home/adds68/git-repos/project.conf" | 10:10 |
| bochecha | I did add that __init__.py | 10:10 |
| adds68 | However my conf files is stores in ~/.config/buildstream.conf | 10:10 |
| bochecha | tlater: and really, I can manually load the entry point and import the plugin in a python shell | 10:10 |
| adds68 | Does bst try to look for a project specific config file or one stored in a .conf directory? | 10:10 |
| bochecha | tlater: but BuildStream doesn't try to load it from bst_external.sources, it tries to load it from buildstream.plugins | 10:11 |
| adds68 | I'm following the steps from https://gitlab.com/BuildStream/buildstream-docker-images which tells me to add my file under ~/.config/ | 10:11 |
| tlater | adds68: You also need a project.conf - the documentation for those is here: https://buildstream.gitlab.io/buildstream/projectconf.html#projectconf | 10:11 |
| adds68 | tlater, thank you, i shall take a look | 10:12 |
| tlater | bochecha: Perhaps I missed a MR... Let me have a look, some changes were about to happen | 10:13 |
| * tlater pulls current master and sees if his docker plugin still works | 10:13 | |
| bochecha | tlater: that `source.load_plugin(kind)` line, source comes from the pluginbase module, and its code just goes to search for any module in buildstream.plugins | 10:13 |
| tlater | bochecha: Not quite correct, its code is set up to look from a pip-defined temporary directory if the plugin is external | 10:14 |
| tlater | The decision whether to read from buildstream.plugins or somewhere else is made a bit earlier in the code. | 10:14 |
| bochecha | when creating that "source" object I suppose? | 10:14 |
| bochecha | what I'm saying is that in my case, it decides to load it from buildstream.plugins :) | 10:15 |
| bochecha | tlater: here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_plugincontext.py#L110 | 10:16 |
| bochecha | `location` is correct at that point | 10:16 |
| bochecha | (it's in `~/.cache/Python-Eggs/` because the .egg file had to be extracted) | 10:16 |
| tlater | That's where it *should* go - now just to figure out why it doesn't... | 10:16 |
| bochecha | yeah, I figured that's what it should do, but I couldn't understand why it doesn't... that pluginbase module looks like magic :x | 10:17 |
| tlater | bochecha: Is your repository pushed somewhere yet? | 10:18 |
| tlater | If it isn't a regression I'd like to have a look at it | 10:18 |
| bochecha | tlater: which one? the new source plugin? | 10:18 |
| tlater | Ah, yeah | 10:18 |
| tlater | gitlab-local, iirc | 10:18 |
| tlater | *git-local | 10:19 |
| bochecha | ah, seems I can't push to bst-external | 10:19 |
| * bochecha forks | 10:19 | |
| bochecha | tlater: https://gitlab.com/bochecha/bst-external/tree/git-local-source | 10:20 |
| bochecha | it's a first draft, I haven't been able to test whether it actually works, since I couldn't get builstream to load it :) | 10:21 |
| tlater | Yeah, the docker plugin still works, so this is entirely due to plugin importing requiring far too much magic | 10:21 |
| bochecha | the docker plugin works? huh | 10:21 |
| *** bethw has joined #buildstream | 10:22 | |
| tlater | Ah, nevermind, I used a tar plugin temporarily | 10:22 |
| bochecha | I don't see what I did differently :-/ | 10:22 |
| tlater | bochecha: Yeah, I think plugin loading changed overnight | 10:22 |
| tlater | Let me check the commit log... | 10:23 |
| tlater | The downsides of using pre-release software ;P | 10:23 |
| bochecha | heh, I don't mind, everyone is reactive enough in here :) | 10:23 |
| bochecha | (when I don't ask at undue hours of course) | 10:23 |
| tlater | You would have had an answer if tristan wasn't on holiday ;) | 10:24 |
| bochecha | hehe | 10:24 |
| bochecha | does he ever sleep? | 10:25 |
| tlater | I think he lives on coffee | 10:25 |
| * tlater is a bit stumped, no recent commits regarding this | 10:26 | |
| tlater | I just pulled current master, let's see what commit I used beforehand, I suppose... | 10:26 |
| bochecha | if you're busy with something else, and you can tell me which commit you used that worked, I'm happy to run the bisection :) | 10:31 |
| bochecha | (I'm also happy to let you run it and just reap the benefits without doing any of the work, of course) | 10:31 |
| tlater | Heh, I'm not sure which commit I used, because I was using the docker image | 10:31 |
| tlater | Unfortunately I didn't consider actually checking before I pulled | 10:31 |
| tlater | So, yeah, I'll look into it ;) | 10:32 |
| bochecha | :) | 10:36 |
| tlater | bochecha: Have you tried 'buildstrem-external:git-local'? | 10:47 |
| tlater | I.e. not CamelCase | 10:47 |
| bochecha | same error | 10:48 |
| bochecha | by the way, the error message is: | 10:49 |
| bochecha | Error loading pipeline: Failed to load Source plugin 'git_local': No module named 'pluginbase._internalspace._spfe78a8c11fe4c7df4ea275fc9bf81bf0.sources' | 10:49 |
| bochecha | I don't remember whether I had pasted it here | 10:49 |
| tlater | bochecha: Aha! | 10:49 |
| bochecha | is that important? it made no sense to me, so I just dug with pdb /o\ | 10:50 |
| tlater | Are you sure it's still loading from buildstream.plugins? | 10:50 |
| tlater | Because iirc, that's the path pluginbase uses to load from pip extracted sources - I might be misremembering | 10:50 |
| bochecha | that's what I found with pdb | 10:51 |
| bochecha | let me try again | 10:51 |
| bochecha | so | 10:52 |
| bochecha | I set a break point just before this line https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_plugincontext.py#L133 | 10:52 |
| bochecha | then I step into the function | 10:52 |
| bochecha | that's its code: https://paste.gnome.org/p3hxcanrs | 10:52 |
| bochecha | see how it tries to `__import__(self.base.package + '.' + name, ...)` ? | 10:53 |
| bochecha | well: | 10:53 |
| bochecha | (Pdb) self.base.package + '.' + name | 10:53 |
| bochecha | 'buildstream.plugins.git_local' | 10:53 |
| bochecha | and that __import__ call fails | 10:54 |
| tlater | Only if self.base.package = buildstream.plugins, which it might not be? | 10:54 |
| bochecha | but it is | 10:54 |
| bochecha | I'm in the pdb shell right now | 10:54 |
| tlater | Alright, unfortunate | 10:54 |
| bochecha | do you have the same problem I do? or does it work for you? | 10:56 |
| * tlater just got the docker plugin to load | 10:57 | |
| tlater | I'll check your branch next, not sure what I did to fix it | 10:57 |
| tlater | This is with the current master | 10:57 |
| bochecha | do you have an example I can use to try the docker source? | 10:58 |
| bochecha | (I'm wondering if this wouldn't be caused by different versions of pluginbase, e.g I have a newer/older than you, and its behaviour changed) | 10:59 |
| tlater | That's possible... Hang on, I'll push the project I'm working on somewhere | 10:59 |
| tlater | bochecha: The project is here: https://gitlab.com/tlater/image-test | 11:02 |
| tlater | It will *definitely* fail to build | 11:03 |
| tlater | But if the docker plugin loads it should be able to fail tracking, so it can be used to test that at least | 11:03 |
| tlater | It seems that the docker plugin is broken due to some API change too, so you will have to remove a line | 11:03 |
| tlater | `sed 's/from buildstream.utils import SaveFile//' -i docker.py` should do it | 11:04 |
| bochecha | ah right, that's because of a yet-unmerged change in BuildStream | 11:04 |
| tlater | I just used `bst track --deps all base.bst` to test, it requires an element that uses the docker plugin | 11:05 |
| bochecha | KeyError: 'bst_external/sources/docker.yaml' | 11:06 |
| bochecha | loading the plugin requires it to have a yaml file, even an empty one | 11:06 |
| bochecha | once I add that yaml file, tracking works :-/ | 11:07 |
| tlater | Hm | 11:08 |
| bochecha | (well, it fails on the URL being wrong, but that means the docker external source plugin was loaded fine) | 11:08 |
| tlater | Any differences between how you specify plugins and what base/base-system.bst does? | 11:08 |
| bochecha | not that I can see | 11:08 |
| * tlater tries the same element with git-local | 11:09 | |
| bochecha | there's an underscore in the name of my source, that's the only difference I see | 11:09 |
| tlater | Same error | 11:09 |
| bochecha | huh | 11:10 |
| bochecha | (Pdb) self.base.package + '.' + name | 11:10 |
| bochecha | 'buildstream.plugins.docker' | 11:10 |
| bochecha | so that's not the problem then | 11:10 |
| tlater | Surely it can't be that, the element plugins do the same thing | 11:10 |
| tlater | Well, that's impressive | 11:10 |
| tlater | I guess pluginbase overloads import or something? | 11:10 |
| bochecha | maybe :x | 11:11 |
| * tlater had a fight or two with pluginbase deciding they want garbage collection at random times, so he wouldn't be surprised | 11:12 | |
| bochecha | I don't get it :-/ | 11:14 |
| tlater | It's not the underscore either | 11:15 |
| tlater | Hmm | 11:16 |
| tlater | You're importing this | 11:16 |
| tlater | from buildstream.plugins.sources.local import LocalSource, unique_key | 11:16 |
| tlater | I don't think that's officially suported, and I frankly have no idea what happens | 11:16 |
| tlater | Let's see what happens if I remove that line... | 11:17 |
| tlater | Yeah, that's it | 11:17 |
| bochecha | circular imports? | 11:17 |
| tlater | bochecha: You can't import from other plugins at present | 11:17 |
| tlater | Perhaps not circular, but pluginbase definitely gets confused | 11:17 |
| bochecha | oh | 11:17 |
| bochecha | hum, I'd rather not copy-paste all of LocalSource for the few modifications I'm making :) | 11:18 |
| tlater | Unfortunately that's what we have to do atm, it would be a bit of a headache to allow that | 11:19 |
| bochecha | (in fact I was about to send a tiny refactoring to LocalSource, so that my GitLocalSource would have even less code) | 11:19 |
| * tlater can't remember exactly what blocked this, but it's not considered pre-1.0 work | 11:19 | |
| bochecha | is it something that might come at some point though? | 11:20 |
| tlater | Maybe, we've had a handful of people run into this | 11:20 |
| bochecha | ok | 11:20 |
| tlater | tristan seems to be opposed to it though, so who knows | 11:20 |
| bochecha | so if I have to completely copy-paste the code, maybe I should keep that out of bst-external? | 11:21 |
| * paulsherwood wonders if he's missed the 1.0 announcement... are we there yer? | 11:21 | |
| tlater | paulsherwood: Not yet, but soon (trademark) | 11:21 |
| tlater | bochecha: No, keep it in bst-external. It's unfortunate, but it's currently best practice. | 11:21 |
| * bochecha can't wait for juergbi's branch on junctions to land | 11:21 | |
| bochecha | tlater: alright | 11:21 |
| bochecha | tlater: so what about the need for an empty .yaml file along with the .py file, is that a restriction we should lift in BuildStream? | 11:31 |
| bochecha | it seems a bit bizarre to enforce that, since source plugins generally don't have accompanying yaml files | 11:32 |
| tlater | bochecha: It would be nice, but we wouldn't be able to actually include the element yaml files if we didn't do that - pip needs to move them about | 11:32 |
| *** adds68 has quit IRC | 11:32 | |
| tlater | Though I think with jonathanmaw's branch being merged that might be possible | 11:32 |
| bochecha | I don't understand what pip has to do with it | 11:33 |
| tlater | bochecha: pip finds and collects the source files - but pip doesn't know whether the plugin we're importing needs a yaml file, and we can only know whether it does once it's loaded | 11:33 |
| bochecha | the issue is here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_plugincontext.py#L104-107 | 11:33 |
| bochecha | wrapping these lines in a `try: ... / except KeyError` would make the yaml file optional, wouldn't it? | 11:34 |
| tlater | Right, it would | 11:34 |
| tlater | Duh | 11:34 |
| * tlater feels very stupid and forgot that pip isn't just a magical bux | 11:35 | |
| tlater | *box | 11:35 |
| tlater | well, entirely | 11:35 |
| bochecha | ): | 11:35 |
| bochecha | oops | 11:35 |
| bochecha | I meant to use this one: :) | 11:35 |
| bochecha | stupid fingers typed in the wrong order | 11:35 |
| tlater | Heh | 11:35 |
| tlater | Well, I don't see any reason not to do that | 11:35 |
| bochecha | I'll send a merge request then | 11:36 |
| tlater | Cool, ta :) | 11:36 |
| bochecha | that being said, is this _plugincontext.py responsible for loading both internal and external plugins? | 11:36 |
| tlater | bochecha: Yes, it is | 11:37 |
| bochecha | then how does it work with internal sources, which also don't have a .yaml file? | 11:37 |
| tlater | bochecha: That specific line is the pip-specific bit | 11:37 |
| bochecha | maybe it's a difference due to the fact bst_external gets installed as a packed egg which needs to be extracted in cache... | 11:38 |
| tlater | Yes, that's exactly why we do that :) | 11:38 |
| bochecha | (setting zip_safe=False in bst_external's setup.py removes all the pain with packed eggs, by the way) | 11:38 |
| tlater | I think we didn't want to have to require people to set that | 11:38 |
| tlater | Because we can't possibly control all plugin authors | 11:39 |
| bochecha | true | 11:39 |
| *** adds68 has joined #buildstream | 11:42 | |
| *** adds68 has quit IRC | 11:44 | |
| *** adds68 has joined #buildstream | 11:45 | |
| *** noah has quit IRC | 12:01 | |
| adds68 | tlater, are there any obvious reasons why a remote cache would say it's not configured for pulling? | 12:14 |
| adds68 | tlater, specifically the official artifact-docker | 12:14 |
| tlater | adds68: The remote cache wouldn't be saying that, buildstream would | 12:15 |
| tlater | The most obvious reason is a mistake in your configuration, what exactly are you trying to do? | 12:16 |
| adds68 | tlater, i've set up a remote cache using the buildstream-docker image on a remote server | 12:16 |
| adds68 | tlater, it currently empty, but i'm just trying test that buildstream can actually reach/use the cache | 12:17 |
| tlater | adds68: So the server is up and running, and you're trying to create a project that can push to it? | 12:17 |
| adds68 | tlater, correct! | 12:17 |
| adds68 | tlater, just out of context: http://37.153.172.201:8080/ | 12:18 |
| adds68 | i'm the only one with ssh access at the moment though | 12:18 |
| tlater | Alright, that should be fine then | 12:18 |
| tlater | Hm, I'm a bit out of the loop on configuration of these... | 12:18 |
| tlater | I think currently you define the push-and-pull url as a single url in project.conf? | 12:19 |
| adds68 | tlater, understood, i just thought i'd see if there was anything abvious it could be | 12:19 |
| adds68 | tlater, i was thinking maybe because it was empty, that could be why it's returning an error | 12:19 |
| tlater | adds68: I don't think so, no, that error is definitely a message buildstream normally gives you | 12:19 |
| *** adds68 has quit IRC | 12:19 | |
| *** adds68 has joined #buildstream | 12:20 | |
| gitlab-br-bot | buildstream: merge request (optional-yaml-for-source-plugins->master: plugincontext: Let plugins not have YAML defaults) #180 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/180 | 12:22 |
| *** bethw has quit IRC | 12:24 | |
| gitlab-br-bot | buildstream: merge request (optional-yaml-for-source-plugins->master: plugincontext: Let plugins not have YAML defaults) #180 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/180 | 12:29 |
| *** brlogger has joined #buildstream | 13:01 | |
| *** adds68 has joined #buildstream | 13:03 | |
| *** gitlab-br-bot has joined #buildstream | 13:04 | |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 13:35 |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 13:41 |
| bochecha | huh, so the tests for my merge-request in bst-external didn't fail on the missing git_local.yaml... how come? o_O | 13:48 |
| *** bochecha_ has joined #buildstream | 13:52 | |
| *** bochecha has quit IRC | 13:55 | |
| *** bochecha_ is now known as bochecha | 13:55 | |
| *** bochecha_ has joined #buildstream | 13:57 | |
| *** bochecha has quit IRC | 13:58 | |
| *** bochecha_ is now known as bochecha | 13:58 | |
| bochecha | the test fails when run with --cov | 14:13 |
| bochecha | it looks like in that case bst is run twice | 14:17 |
| tlater | Hm, odd | 14:18 |
| bochecha | oh no | 14:20 |
| bochecha | it builds first | 14:20 |
| bochecha | then checks out | 14:20 |
| bochecha | the checkout fails when the test is run with --cov | 14:20 |
| bochecha | but it succeeds otherwise | 14:20 |
| bochecha | hahahahahahahahaha | 14:27 |
| bochecha | it's the .coverage.${HOSTNAME}.${SOMETHING} file which gets added | 14:27 |
| bochecha | it trips bst into thinking there's a new file, so the project needs to be rebuilt | 14:27 |
| bochecha | except the file was added by the build | 14:27 |
| bochecha | this is one more reason why I want this git-local source :D | 14:28 |
| tlater | Oh, hah | 14:28 |
| tlater | It's like buildstream is developing a personality :) | 14:29 |
| bochecha | I only had to add `.coverage*` to the .gitignore, and the git-local source does its job, ignoring the coverage file, telling buildstream to ignore it, making the test for git-local pass | 14:29 |
| bochecha | it does feel a bit circular... | 14:29 |
| tlater | "Oh no, I just finished building that! Now there's another one..." | 14:29 |
| bochecha | yeah | 14:29 |
| bochecha | bst is the broom in the sorceror's apprentice | 14:29 |
| bochecha | except it doesn't keep filling up the water, it keeps building, never realizing its job is done :] | 14:30 |
| * tlater is imagining this poor cute beaver with its hands up its hear every time it turns its back to look at its twigs | 14:30 | |
| tlater | *hair | 14:30 |
| bochecha | but then that's really a problem with local sources using `path: .`, any file created by the build will trip bst | 14:30 |
| tlater | Yeah, I don't think the local was really intended to be used like that | 14:31 |
| tlater | Perhaps we don't need a .bstignore file | 14:32 |
| tlater | But add something to the element config? | 14:32 |
| tlater | ignored-files: [] | 14:32 |
| bochecha | yeah, it's very possible I'm not using the right tools here... the use-case seems important though (having the bst recipes in the source code repo) because it enables lots of nice things (like the dev/CI stuff we were discussing yesterday) | 14:32 |
| bochecha | maybe | 14:32 |
| bochecha | although reusing .gitignore seems easier :P | 14:33 |
| tlater | Definitely, but it won't work for a general project | 14:33 |
| tlater | Allowing local source plugins to also except files will probably be important | 14:33 |
| * tlater figures this is at least enough to raise it as an issue | 14:34 | |
| gitlab-br-bot | buildstream: merge request (workspaces-yml-version-1->master: Add support for version 1 workspaces.yml format) #176 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/176 | 14:35 |
| bochecha | maybe, yes | 14:35 |
| tlater | Hm, I guess for the time being we have git-local to test how this works out in practice. We can see how it pans out with things like recursive pipelines | 14:37 |
| bochecha | absolutely | 14:38 |
| bochecha | (I'm going to use recursvice pipelines as soon as they get merged) | 14:38 |
| *** bethw has joined #buildstream | 14:38 | |
| bochecha | I'm also happy to keep this git-local source separate for now, if needed (if it ends up being awful, I wonder whether having it in the bst-external repository implies some kind of commitment of maintenance) | 14:40 |
| tlater | bochecha: The bst-external repository is for plugins intentionally left outside of buildstream because they aren't ready to be included, but so generally useful that we want to make them publicly available | 14:43 |
| tlater | There will be some maintenance, but considering how few there currently are I don't think it would be an issue - you wouldn't have to take that on ;) | 14:43 |
| tlater | If it ends up being unmaintainable we'll remove it in the future, perhaps in the future we allow extending plugins and we can neaten it up | 14:44 |
| tlater | Perhaps it becomes obsolete | 14:44 |
| tlater | Either way, for now it is useful, so I think it should be available | 14:44 |
| *** mcatanzaro has joined #buildstream | 14:44 | |
| bochecha | > If it ends up being unmaintainable we'll remove it in the future | 14:45 |
| bochecha | that was my fear, that we wouldn't remove it | 14:45 |
| bochecha | (not that I'd have to maintain it... if I use it I don't mind) | 14:45 |
| tlater | I don't think we keep bst-external to the same standards we keep buildstream | 14:46 |
| tlater | I guess there's no explicit statement for that yet, though. | 14:47 |
| bochecha | ok, so the tests passed, and I'm confused: https://gitlab.com/bochecha/bst-external/-/jobs/43765411 | 14:54 |
| bochecha | they should have failed on the missing yaml file, as per https://gitlab.com/BuildStream/buildstream/merge_requests/180 | 14:54 |
| * tlater unfortunately knows pretty much nothing about the bst-external test suite | 14:55 | |
| tlater | Ah, it uses the same scripts | 14:55 |
| bochecha | it's very similar to the integration-tests one in buildstream | 14:55 |
| tlater | Hm, yeah, that should definitely throw an error | 14:57 |
| bochecha | with bst master it does (I just tried it here) | 14:58 |
| bochecha | could the CI run a bst version before this commit? | 14:59 |
| bochecha | 032d0f3 Add support for YAML default config loading | 14:59 |
| bochecha | it's the one that added loading the YAML | 14:59 |
| bochecha | from september... could the bst-external CI really be that late? :-/ | 14:59 |
| tlater | Seems very unlikely | 14:59 |
| * tlater thinks bst-external came into existence in October/November | 15:00 | |
| bochecha | but it uses a docker image which might be older? | 15:01 |
| tlater | No, that docker image is also newer than that commit | 15:01 |
| tlater | Perhaps we had a short time in which that was ignored for some reason, it would explain why the docker plugin worked for me despite lacking a .yaml file | 15:01 |
| bochecha | tristan last changed the _plugincontext.py file, doing something with exceptions | 15:02 |
| tlater | That's also a month ago though, and I don't think gitlab caches docker images | 15:02 |
| bochecha | no, that's not it | 15:02 |
| bochecha | yeah, the CI uses master | 15:04 |
| bochecha | $ su buildstream -c 'bst --version' | 15:04 |
| bochecha | bst, version 0.1.dev1591+g3d97314 | 15:04 |
| bochecha | that's what I get here as well, 3d97314 is the latest master | 15:04 |
| bochecha | so why does it fail here but works in the CI? o_O | 15:05 |
| tlater | Ugh, it sounds like a bug in the CI, but I have no idea how that would happen | 15:05 |
| bochecha | script: | 15:06 |
| bochecha | - pip3 install --no-index . | 15:06 |
| bochecha | the only thing I can think of is that a previous version of my MR installed the module with the .yaml file | 15:06 |
| * tlater still doesn't know what `--no-index` does | 15:06 | |
| bochecha | and `pip install` just installs over it, without removing what already exists, which means the .yaml file lingers | 15:07 |
| tlater | bochecha: iirc we keep some cache directories, that is possible | 15:07 |
| bochecha | but that only holds if modifications by one pipeline are kept for another one... | 15:07 |
| tlater | Try to remove the cache for a commit? | 15:07 |
| bochecha | how to do that? | 15:07 |
| tlater | Comment line 3-5 in the .gitlab-ci.yaml | 15:08 |
| bochecha | - export XDG_CACHE_HOME="$(pwd)/cache" | 15:08 |
| bochecha | so, the module gets installed correctly by pip | 15:08 |
| bochecha | but the **unpacked egg** is kept from one run to another | 15:08 |
| tlater | Yeah, but also; | 15:08 |
| bochecha | since it goes in ${XDG_CACHE_HOME}/Python-Eggs/ | 15:09 |
| tlater | - cache/buildstream/sources/ | 15:09 |
| tlater | We explicitly cache only buildstream sources | 15:09 |
| tlater | So unless that is broken somehow, this shouldn't happen | 15:09 |
| bochecha | yeah... | 15:09 |
| tlater | Worth a try to rm -r ${XDG_CACHE_HOME} though | 15:09 |
| tlater | (I take no responsibility for the CI taking longer ;) ) | 15:10 |
| bochecha | :) | 15:10 |
| bochecha | it's on my fork though, that shouldn't impact anybody else, right? | 15:10 |
| bochecha | last thing I want is to slow everybody else down ^_^ | 15:10 |
| tlater | Nah, it shouldn't, and I don't think anyone is working on bst-external atm anyway | 15:11 |
| nexus | Please could someone give me a nice definition of what an `element` is in the context of bst that I can use for the docs? ty | 15:27 |
| bochecha | nexus: an element is a unit of "thing to build" | 15:28 |
| bochecha | for example, if you're building an application which has 3 dependencies, you would most likely have 4 elements | 15:28 |
| jonathanmaw | tristan: I sorted out the tests issue with https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 15:34 |
| jonathanmaw | I forgot to commit the directories full of test data :/ | 15:34 |
| bochecha | tlater: ok, so definitely not a leftover git_local.yaml (I ran `find / -name "*.yaml"` as part of the CI) | 15:37 |
| bochecha | I don't understand why the tests pass | 15:37 |
| bochecha | I think this is the first time ever I'm spending so long trying to get some tests to **fail** | 15:38 |
| * tlater really hoped that would come up with something | 15:38 | |
| bochecha | me too | 15:39 |
| tlater | Perhaps we can access the machines these tests run on and try and debug from there, I think sam has access to them? | 15:41 |
| tlater | Otherwise, short of running the literal CI commands in docker, I really don't know how to figure this out :| | 15:41 |
| tlater | Sam is not about today, unfortunately | 15:42 |
| bochecha | I'm trying one last thing, then I'll give up and go get the bottle of génépi | 15:44 |
| bochecha | tlater: I got it to pass with master on my laptop | 15:49 |
| tlater | How? | 15:49 |
| *** starmad has joined #buildstream | 15:50 | |
| bochecha | I was installing bst-external with `python setup.py install` | 15:51 |
| bochecha | which installs it as a compressed egg file | 15:51 |
| bochecha | when we try to load the yaml file (which doesn't exist) from the egg, that raises the KeyError which makes the tests fail | 15:51 |
| bochecha | the CI installs bst-external with `pip install .` | 15:51 |
| bochecha | which installs it as a regular package, no egg | 15:52 |
| tlater | And that raises no error? | 15:52 |
| bochecha | when we try to load the yaml file (which still doesn't exist), then no error is raised | 15:52 |
| tlater | Ack | 15:52 |
| bochecha | the KeyError must come from the file list in the zip (eggs are just zips) | 15:52 |
| bochecha | so I still think my MR which try/except the KeyError is needed, for those who install with `python setup.py install` | 15:53 |
| bochecha | but at least now I know I'm not going crazy | 15:53 |
| tlater | Yeah, and perhaps we should have the CI install in all ways you can with python | 15:53 |
| tlater | Which is enormously cumbersome... | 15:53 |
| bochecha | an afternoon trying to get these tests to pass, and eventually I realize that they really should succeed ^_^ | 15:53 |
| bochecha | well, pip is becoming the defacto install tool | 15:54 |
| bochecha | it's learning to handle other build tools than distutils/setuptools (e.g flit), it's growing the pyrpoject.toml support, etc... | 15:54 |
| bochecha | hopefully, at some point the recommended way to install python modules will be `pip install` and everyone will really do that | 15:54 |
| tlater | Well, for now we know what's going wrong when we run into this again | 15:55 |
| bochecha | that's why I added a pip element to buildstream in fact, to be future-proof, because the distutils element can't install all python modules, whereas (at least in the future) the pip one will be able to | 15:55 |
| bochecha | I loath Python packaging every day a bit more | 15:57 |
| *** tpollard has quit IRC | 16:50 | |
| *** adds68 has quit IRC | 17:00 | |
| *** valentind has joined #buildstream | 17:07 | |
| *** starmad has quit IRC | 17:29 | |
| gitlab-br-bot | buildstream: merge request (workspaces-yml-version-1->master: WIP: Add support for version 1 workspaces.yml format) #176 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/176 | 17:34 |
| *** adds68 has joined #buildstream | 17:35 | |
| *** adds68 has quit IRC | 17:40 | |
| *** bethw has quit IRC | 17:47 | |
| gitlab-br-bot | buildstream: merge request (workspaces-yml-version-1->master: Add support for version 1 workspaces.yml format) #176 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/176 | 17:53 |
| *** jude has quit IRC | 18:56 | |
| gitlab-br-bot | buildstream: merge request (164-minimise-overlaps-by-having-overlaps-raise-exceptions-unless-configured-not-to->master: WIP: Resolve "Minimise overlaps by having overlaps raise exceptions unless configured not to") #181 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/181 | 19:23 |
| *** bochecha has quit IRC | 19:30 | |
| gitlab-br-bot | buildstream: merge request (164-minimise-overlaps-by-having-overlaps-raise-exceptions-unless-configured-not-to->master: WIP: Resolve "Minimise overlaps by having overlaps raise exceptions unless configured not to") #181 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/181 | 19:54 |
| *** jonathanmaw has quit IRC | 19:55 | |
| *** valentind has quit IRC | 23:11 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!