| *** tristan has joined #buildstream | 01:49 | |
| *** bochecha has quit IRC | 07:35 | |
| *** bochecha has joined #buildstream | 07:36 | |
| bochecha | hurray for bash completion! ♥ | 07:47 |
|---|---|---|
| *** jonathanmaw has joined #buildstream | 08:11 | |
| bochecha | has anybody given some thoughts about how to run tests for software build with BuildStream? | 08:14 |
| bochecha | hmmm, I guess I could `bst shell --scope build --command py.test app.bst` | 08:25 |
| *** uajain has joined #buildstream | 08:46 | |
| *** tlater has joined #buildstream | 09:17 | |
| gitlab-br-bot | push on buildstream@validation (by Tristan Maat): 2 commits (last: Add source plugin node validations) https://gitlab.com/BuildStream/buildstream/commit/be94c716210b7e9530be67dd20a8c625d7b731ae | 09:46 |
| bochecha | I'm reading this: https://buildstream.gitlab.io/buildstream/artifacts.html | 10:03 |
| bochecha | one thing I don't understand, is how to actually push the artifacts to the shared cache :-/ | 10:03 |
| gitlab-br-bot | buildstream: Tristan Maat created branch 24-better-validation-for-loaded-yaml | 10:04 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: WIP: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 10:04 |
| bochecha | do I have to run the bst-artifact-receive command over ssh? or does buildstream push them automatically once built? | 10:04 |
| gitlab-br-bot | push on buildstream@24-better-validation-for-loaded-yaml (by Tristan Maat): 7 commits (last: Add _yaml.validate_node) https://gitlab.com/BuildStream/buildstream/commit/2f545d57b982d97638951bca7facb45f43f63169 | 10:06 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: WIP: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 10:07 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 10:08 |
| jonathanmaw | Hi bochecha, I'm out of the loop when it comes to the shared artifact cache, but I'll see what I can find out. | 10:15 |
| jonathanmaw | bochecha: buildstream should have a `push` command | 10:21 |
| bochecha | jonathanmaw: ah, right | 10:22 |
| jonathanmaw | e.g. `bst push --deps=all core.bst` | 10:22 |
| bochecha | thanks, somehow I missed it :x | 10:22 |
| jonathanmaw | I think, anyway | 10:22 |
| jonathanmaw | I haven't used it, but that seems to be what it does, unless "target" means something different when pushing. | 10:22 |
| gitlab-br-bot | buildstream: Tristan Maat created branch 70-third-party-plugin-sharing | 10:37 |
| juergbi | bochecha: bst build will automatically push as well, though, if a push url is configured. bst push is to push already built artifacts | 10:43 |
| *** tristan has quit IRC | 12:03 | |
| *** tlater has quit IRC | 12:08 | |
| bochecha | juergbi: that's great :) | 12:16 |
| bochecha | although, I'd like only the builds made by the builder to be pushed, not the test builds I'd make on my laptop | 12:17 |
| bochecha | I guess I need to set the pull url in project.conf (so that anybody cloning the repo will reuse the artifacts from the shared cache) and the push url only in the user config for the builder | 12:28 |
| *** tlater has joined #buildstream | 12:30 | |
| *** tristan has joined #buildstream | 13:11 | |
| * tristan catches up with backlog | 13:24 | |
| tristan | bochecha, when you run `bst build` or `bst push`, only artifacts which are pertinent to the resolved pipeline will be pushed | 13:24 |
| tristan | we're thinking with multiple artifact shares one can come up with a strategy where only production builds ever land in a production share, and secondary caches could be created for dev (where those stores could be wiped at times) | 13:26 |
| tristan | pruning existing caches is a possibility but also a tricky problem | 13:26 |
| tristan | regarding testing; ideally we should be doing that in the pipeline (not that there is anything wrong with running them in a shell from time to time) | 13:28 |
| tristan | I left out local 'make check' stages from build elements for now | 13:28 |
| tristan | because it will be almost impossible to make huge 400+ element projects boot, and also cause I have some wild ideas I'd like to try | 13:29 |
| tristan | for instance; it would be very nice if we could cache the element's build directory in the artifact; which we could potentially use in a workspace (i.e. create a workspace on 'this build of webkit', with all your .o files and source files in place) | 13:30 |
| tristan | with the same thing we could create test elements which consume this and run the `make check` phases; by making that a separate element, you orthogonalize local tests and avoid blocking builds of reverse dependencies on `make check` | 13:31 |
| tristan | of course similar can already be done for installed tests, we'll want to create an element for just 'running installed tests' at some point | 13:31 |
| * tristan is glad somebody noticed completions \o/ | 13:32 | |
| tristan | :) | 13:32 |
| bochecha | tristan: about the artifacts caching, what I'm thinking is that everybody who will contribute to the project will need to pull the pre-built artifacts | 13:33 |
| bochecha | but then most builds people would do on their workstation are completely throwable test stuff which really have no reason to ever get to the shared cache | 13:34 |
| bochecha | (plus, most people wouldn't even have an SSH access to push to the shared cache) | 13:34 |
| tristan | true, I think auto builders are more interesting for contributing to caches | 13:34 |
| bochecha | exactly | 13:34 |
| bochecha | which is what I meant to say above | 13:34 |
| tristan | so with multiple artifact shares, you can have one that is only contributed to by a builder of the stable branch | 13:35 |
| tristan | and that one never expires | 13:35 |
| tristan | so you keep every build that was ever released (if you want) | 13:35 |
| tristan | and then you can have one for your dev work, which another builder would build from dev branches; or you could also let developers push to | 13:35 |
| tristan | this way I can try builds of your branch and such | 13:35 |
| bochecha | "I can try builds of your branch" hmm, maybe | 13:36 |
| tristan | in a dev setup, buildstream would have the production artifact cache as a fallback location for looking up artifacts | 13:36 |
| tristan | so you never have to end up pushing stuff that has not diverged from stable | 13:37 |
| bochecha | can I specify multiple pull urls, so that one is a fallback? | 13:37 |
| tristan | that's a not yet done feature no haha | 13:37 |
| tristan | sorry :) | 13:37 |
| tristan | but it's in the woodworks | 13:37 |
| bochecha | no problem, I'm not there yet anyway :P | 13:37 |
| bochecha | one thing that's not very clear from the docs, though... do the pull/pull urls have to be specified in the user config? or do they work as well in the project.conf? | 13:38 |
| bochecha | (I'm testing a build right now to see if it does, but it will take some time before it finishes :P) | 13:38 |
| tristan | I filed a bug about that I believe | 13:38 |
| bochecha | err, I meant pull/push urls | 13:38 |
| tristan | They are originally in user config, but I want to put them in project.conf | 13:39 |
| tristan | and have a possibility of overriding them, cause currently it's backwards | 13:39 |
| bochecha | yeah | 13:39 |
| bochecha | basically the pull url would be great in project.conf | 13:39 |
| bochecha | you clone the repo, run buildstream, it fetches the cached artifacts | 13:39 |
| tristan | If I have multiple projects, I can accidentally end up pushing proprietary things to the wrong place too easily | 13:39 |
| bochecha | you don't need to configure anything | 13:39 |
| bochecha | that too | 13:40 |
| tristan | Another approach I was thinking is managing state in <project>/.bst/ (we already track workspaces there) | 13:40 |
| tristan | so we could have stuff like `bst remote add/remove` for artifact caches ? | 13:40 |
| bochecha | I really think that at least the main pull url should be in git with the rest of the project | 13:41 |
| bochecha | means random contributors just clone your project, and they are ready to go | 13:41 |
| tristan | yes as a first step | 13:41 |
| bochecha | whether that's in project.conf or in a .bst folder doesn't really change anything for me :) | 13:41 |
| tristan | I'm trying to boil down a list of blockers to a stable release and that's one of them | 13:41 |
| bochecha | the .bst folder has the advantage of being able to contain the elements folder as well, though | 13:42 |
| tristan | basically anything that absolutely needs an api/ux change | 13:42 |
| bochecha | this way people who just want to hack on the code don't need to see the elements folder all the time | 13:42 |
| tristan | I'm not sure I follow about putting elements into <project>/.bst/ | 13:43 |
| bochecha | this way all the bst stuff is in the same place | 13:43 |
| tristan | the .bst/ directory should really be .gitignore | 13:43 |
| bochecha | oh | 13:43 |
| tristan | because it's local state, we store workspaces (like what workspaces you have open) there | 13:44 |
| bochecha | I thought it would contain things like the configuration for buildstream? you were talking about remotes for the artifact cache above | 13:44 |
| tristan | dont want that committed in your project | 13:44 |
| bochecha | and remotes for the artefacts cache are something I'd want committed in my project :) | 13:44 |
| tristan | yeah I mean for what the user wants to configure, like if they want to add anything | 13:44 |
| bochecha | oh, ok | 13:44 |
| tristan | first thing is project.conf has to either decide of provide a good recommendation | 13:44 |
| tristan | s/of/or | 13:45 |
| tristan | fancy ways of extra artifact caches dont block 1.0 :) | 13:45 |
| bochecha | (by the way, completely unrelated, but https://buildstream.gitlab.io/ is 404) | 13:45 |
| bochecha | (it should probably redirect to /buildstream/ ) | 13:45 |
| tristan | oh yeah I know | 13:45 |
| tristan | this comes up every now and then | 13:46 |
| tristan | so, we dont have a real website :) | 13:46 |
| tristan | And a temporary redirect to the docs would be nice | 13:46 |
| tristan | I'm not sure I know how to do it with gitlab and havent taken the time myself | 13:46 |
| tristan | nobody else did either haha :) | 13:46 |
| bochecha | and neither do I :P | 13:47 |
| bochecha | found the ticket: https://gitlab.com/BuildStream/buildstream/issues/87 | 13:48 |
| tristan | I suspect you need to make another static website for the buildstream gitlab "group" and in it; you need an index.html with javascript that provokes a redirect ? | 13:48 |
| tristan | something like that, cause I dont think it's apache | 13:48 |
| bochecha | in the ticket you don't talk about having a way to commit the pull url to version control (the ticket only talks about .bst/), do you want me to add a note about that? | 13:49 |
| tristan | bochecha, feel free to clarify that yeah, I thought I sort of implied that in the first of the 2 points but seems unclear :) | 13:50 |
| bochecha | oh, ok, I didn't understand it that way | 13:51 |
| bochecha | that's fine then, I subscribed to the issue | 13:51 |
| tlater | tristan: I have a prototype for the third party plugins, could you have a look to see if it's what you're expecting? | 13:51 |
| bochecha | I'll see if I get a moment to try and work on this eventually | 13:52 |
| tristan | tlater, ok will do, I am backed up with your branches I see | 13:53 |
| tristan | tlater, did you change --force-platform to an internal env var in your cross platform branch ? | 13:53 |
| tlater | Ah, no, I haven't yet | 13:54 |
| tristan | I still have to give it another lookover, that's just the first point that came to mind | 13:54 |
| tlater | Do you mean environment variable as in process.env or as a variable in project.conf? | 13:55 |
| tristan | juergbi, did you get a chance to look through my extraneously long email btw ? any thoughts ? | 13:55 |
| tristan | tlater, I mean like os.environ | 13:56 |
| juergbi | i started but haven't read it in full yet | 13:56 |
| tristan | tlater, like BST_FORCE_BACKEND="unix" bst build ... | 13:56 |
| bochecha | ERROR: Not configured for pushing artifacts | 13:56 |
| tristan | to select the catch all unix fallback backend | 13:56 |
| bochecha | I get that when running `bst push --deps=all TARGET` | 13:57 |
| tlater | Alright gotcha (process.env might be nodejs?) | 13:57 |
| tristan | hmmm | 13:57 |
| *** bochecha has left #buildstream | 13:57 | |
| *** bochecha has joined #buildstream | 13:57 | |
| bochecha | oops, wrong keyboard shortcut :-° | 13:57 |
| bochecha | I was about to say I added the urls in ~/.config/buildstream.conf | 13:58 |
| gitlab-br-bot | push on buildstream@70-third-party-plugin-sharing (by Tristan Maat): 1 commit (last: _plugincontext.py: Add support for pip-installed plugins) https://gitlab.com/BuildStream/buildstream/commit/75937d2a7c10410cf3754d67cc4274ebebc92d10 | 13:58 |
| tristan | bochecha, let me guess | 13:59 |
| tristan | hmm | 13:59 |
| tristan | you have a pull-url also | 13:59 |
| tristan | it specifies an unreachable service | 14:00 |
| tristan | or | 14:00 |
| tristan | points to an http/https location that is not serving an ostree repo ? | 14:00 |
| bochecha | it specifies a URL which returns 404... because I've never pushed anything yet | 14:00 |
| bochecha | do I have to create an empty ostree repo beforehand? | 14:00 |
| tristan | You need to yeah | 14:00 |
| bochecha | ok | 14:00 |
| bochecha | obviously I don't have sudo access on that ost to install ostree... | 14:01 |
| * bochecha creates an ostree repo and rsyncs it to the server | 14:01 | |
| tristan | it even has to be specifically archive-z2 http://buildstream.gitlab.io/buildstream/artifacts.html#initializing-the-cache | 14:01 |
| tristan | bochecha, I am suspecting there is going to be post release churn in how artifact shares work | 14:02 |
| * tristan hopes we can migrate this to something different without too much trouble | 14:02 | |
| bochecha | migrate to what? | 14:02 |
| tristan | Originally I wanted it to be "just ostree" | 14:02 |
| tristan | because there is no moving parts | 14:02 |
| tristan | but I already now need this `bst-artifact-receive` program in order to do pushes | 14:03 |
| tristan | which means I need a separate url to specify artifact caches | 14:03 |
| tristan | and I need something more tweaked out than just "run this" for setting up artifact caches (the plan was "Just host an ostree repo") | 14:03 |
| tristan | Also now I'm looking at platforms that dont do ostree and I'm thinking, I still want to use ostree artifact shares; so I might want layer in between to store artifacts built on non-Linux in a linux ostree share | 14:04 |
| tristan | Anyway, /me puts all that aside and thinks that we can find a way to migrate to something a bit cleaner without too much trouble | 14:05 |
| bochecha | huh | 14:07 |
| bochecha | I can't install bst-receive-artifacts on the server, because bst checks for bwrap, which I don't have on the server | 14:07 |
| bochecha | but it doesn't feel like something which would be needed on the server hosting the artifacts caches | 14:07 |
| tristan | bochecha, you are entirely correct indeed | 14:08 |
| tristan | we dont need that at all, but as we have some code share in the push/pull... its lumped together | 14:08 |
| tristan | we should probably have a way to bypass that for only installing bst-artifact-receive but not bst | 14:08 |
| gitlab-br-bot | buildstream: issue #91 ("bst-artifact-receive does not require bubblewrap") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/91 | 14:13 |
| bochecha | oh | 14:15 |
| bochecha | but bst-artifact-receive needs ostree anyway | 14:16 |
| bochecha | so I'll have to wait for the admin of this server to wake up :( | 14:16 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 2 commits (last: Add a way to force using the fallback platform) https://gitlab.com/BuildStream/buildstream/commit/3388db0d4f942e73d3e5541431fe64f20bf4a479 | 14:18 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: WIP: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 14:18 |
| tlater | tristan: Should I create a gitlab repository for buildstream-plugins already, or hold off on that until it's working? | 14:20 |
| *** xjuan has joined #buildstream | 14:22 | |
| tristan | tlater, go ahead, we can delete/rename repositories on gitlab I believe anyway | 14:25 |
| tlater | Hmmm... Looks like I can't create it in the buildstream group, but I suppose we can move it later | 14:29 |
| bochecha | anyway to run a `bst shell` which would have access to the network? (I understand why the network is cut by default, this is for a punctual test) | 14:34 |
| tlater | Alright, the buildstream-plugins repo is here for now: https://gitlab.com/tlater/buildstream-plugins/tree/master | 14:39 |
| tlater | The only notable thing about it is that setup.py defines a set of entry_points under 'buildstream.plugins' - that's how setuptools can locate the plugins. | 14:40 |
| tristan | bochecha, currently network should be reachable by default in `bst shell` | 14:43 |
| tristan | although it needs to be changed | 14:43 |
| bochecha | is dnf resolution not working then? | 14:44 |
| bochecha | sorry, I meant dns (damn muscle memory) | 14:44 |
| tristan | I guess, it should be disabled by default and have a flag; otherwise debugging failed builds can be problematic | 14:44 |
| tristan | not sure | 14:44 |
| bochecha | because I can't use pip inside a shell | 14:44 |
| tristan | Hmmm | 14:44 |
| tristan | I wonder if you have write access to / | 14:44 |
| bochecha | probably not, but I shouldn't need that to run a network query? | 14:45 |
| tristan | well, our understanding of enabling the network doesnt go further than passing --unshare-net or such to bwrap | 14:45 |
| tristan | so it could be many things, lets see what about /dev, I think we're mounting a fresh dev there | 14:46 |
| bochecha | /dev does contain a few things | 14:46 |
| tristan | bochecha, are you using a shell or launching it with bst shell --command ? | 14:47 |
| bochecha | bst shell --scope=run element.bst | 14:47 |
| tristan | yeah hmmm, so interactive is on | 14:47 |
| bochecha | Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f3c56ca5da0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/zope-interface/ | 14:48 |
| bochecha | pip just keeps saying that | 14:48 |
| bochecha | "name or service not known" :-/ | 14:48 |
| tristan | right and that means you get a `--dev /dev` appended to bwrap's argv | 14:48 |
| bochecha | no /etc/resolv.conf inside the sandbox, could that be it? | 14:49 |
| bochecha | as in, pip doesn't know which name server to use | 14:49 |
| tristan | Possibly | 14:49 |
| tristan | What is your runtime ? | 14:49 |
| bochecha | something I built | 14:49 |
| bochecha | but surely /etc/resolv.conf shouldn't come from the runtime | 14:50 |
| tristan | I mean, your runtime is doing everything... is it a part of something that boots ? | 14:50 |
| bochecha | no, it doesn't boot | 14:50 |
| tristan | cause if it is, it would be interesting to know if pip works in a booted one | 14:50 |
| bochecha | it's just to run apps inside it, with bwrap | 14:50 |
| tristan | yeah | 14:50 |
| tristan | so I'm not really sure off hand heh | 14:50 |
| bochecha | I wonder if bst shouldn't just mount the host's /etc/resolv.conf inside the sandbox? | 14:50 |
| tristan | no | 14:51 |
| tristan | haha | 14:51 |
| bochecha | why not? | 14:51 |
| tristan | I'm not averse to adding some mount options to bst shell though | 14:51 |
| bochecha | I mean, I can't add `nameserver 192.168.1.1` to the /etc/resolv.conf file inside the runtime | 14:52 |
| bochecha | that woudln't work when someone else tries to build | 14:52 |
| tristan | no because that is largely presumptuous about the content of you build and your host environment | 14:52 |
| bochecha | because they'd have a different nameserver on their network | 14:52 |
| tristan | which is the opposite of where I want to go | 14:52 |
| bochecha | sure, but then... how can the /etc/resolv.conf file inside the sandbox ever contain the right nameserver? | 14:52 |
| tristan | Well, good question; it can be read-write | 14:53 |
| bochecha | without dns, having network enabled is much less useful :) | 14:53 |
| tristan | Well, maybe we need to consider some scaffolding options for shells, they are clearly a risky affair to begin with | 14:55 |
| bochecha | yes, absolutely | 14:55 |
| bochecha | but they are so useful to hack/debug :) | 14:55 |
| bochecha | (obviously, builds should not have any network access, so this whole conversation doesn't apply to builds) | 14:55 |
| tristan | right, they get you what you need 99% of the time for debugging a failed build, and most of the time that you need to run gdb & valgrind as well | 14:56 |
| tristan | so I'm a bit worried about scope creep of course, like if we pour our energy a lot into `bst shell`, it could turn into some kind of production environment | 14:57 |
| tristan | which is also counter concept; a goal is to decouple what you build from how you deploy | 14:58 |
| bochecha | for production I would just use directly bwrap (probably with a wrapper tool), not bst | 14:58 |
| bochecha | the run/production sandbox really doesn't have the same constraints as the build one, it seems fair to have different tools | 14:59 |
| tristan | well hope we can generate docker images and flatpacks and things which have some things in place for deployment :) | 14:59 |
| bochecha | sure | 14:59 |
| bochecha | in my case I'm looking at straight ostree+bwrap for deployment | 14:59 |
| bochecha | it's not desktop apps, so flatpak is out, and docker is a bit too big for our target (embedded appliances) | 15:00 |
| tristan | well still, if we could have a nice scaffolding, perhaps some user/project supplied modules which attempt to bridge sandbox <--> host, if it were not too crazy... | 15:00 |
| tristan | it would mean you could like, run the epiphany you just built and it would work | 15:01 |
| bochecha | I'd be happy with a simple "make the network work in the shell" option :) | 15:01 |
| tristan | and having access to your /home would also be good | 15:01 |
| bochecha | the thing I wanted to do right now... I have python modules inside the sdk, built from tarballs | 15:01 |
| bochecha | `bst track` doesn't know how to update to a newer release of those modules | 15:02 |
| tristan | Well, do we know for certain that it's because of the resolve.conf ? | 15:02 |
| bochecha | so I wanted to get into a shell, and run `pip list --outdated` | 15:02 |
| bochecha | and pip would tell me which modules were outdated | 15:02 |
| bochecha | at which point I'd just update the elements | 15:02 |
| tristan | Did you manage to edit the resolve.conf and have it work ? | 15:02 |
| bochecha | but maybe I looked at this wrongly... should `bst track` be able to update those python module elements? :) | 15:03 |
| tristan | It should if they are in your project yeah, if you have reliable tracking branches for them in their repositories | 15:03 |
| bochecha | well, not branches | 15:04 |
| bochecha | urls to the release tarballs | 15:04 |
| tristan | Ok so that is a much wanted feature | 15:04 |
| bochecha | also, yes, sorry, I forgot to mention: I added the right line to the /etc/resolv.conf and now dns works fine in the shell | 15:04 |
| tristan | and it seems jjardon[m] pointed me to some script which does it... but we currently dont `bst track` tarballs that well | 15:04 |
| bochecha | ah, so maybe we can forget about having the network working in the shell and focus on that much wanted feature, then, because with it I don't need the network in the shell any more :) | 15:05 |
| tristan | it involves a heuristic / guess about how tarballs are stored, or perhaps a policy expressed by the user in the tar Source configuration | 15:05 |
| bochecha | or how about something different... | 15:05 |
| bochecha | right now I'm using elements of kind `pip` | 15:05 |
| bochecha | and each has a source of kind `tar` | 15:06 |
| bochecha | what if instead they had a source of kind `pypi` | 15:06 |
| bochecha | and bst track would automatically fetch the sources from pypi.python.org? | 15:06 |
| tristan | I'm not sure that you can easily create a reliable pypi source | 15:06 |
| bochecha | then the tracked ref would be the version of the release | 15:06 |
| bochecha | why wouldn't it be reliable? | 15:07 |
| tristan | One problem with creating a dpkg source plugin is that debian repos typically expire packages, so one would have to use an archive | 15:07 |
| bochecha | I don't think pypi expires tarballs though, but that could indeed be an issue | 15:08 |
| tristan | The tarball thing though, would be very neat and much wanted :) | 15:09 |
| bochecha | yeah | 15:10 |
| bochecha | but tarballs are hard | 15:10 |
| bochecha | you pretty much need to regex directory listings on web servers | 15:10 |
| bochecha | unless you use something like anitya | 15:10 |
| tristan | yep but I think there is a reference though | 15:10 |
| bochecha | https://release-monitoring.org/ | 15:10 |
| bochecha | which has an API, and already tries to do all the work of monitoring new releases and regexing those directory listings | 15:11 |
| tristan | hhhhmmmmmmmmm | 15:11 |
| bochecha | e.g https://release-monitoring.org/api/projects/?pattern=pytest to get the list of projects containing "pytest" in their name | 15:13 |
| bochecha | along with versions | 15:13 |
| bochecha | which seem to be reverse sorted (i.e last version first) | 15:13 |
| bochecha | and then https://release-monitoring.org/api/project/3765 to have all the details on the pytest project (including package names in a couple of distros) | 15:14 |
| tristan | oddly their responses seem to supply consistent version numbers, but not urls to released tarballs or bundles | 15:14 |
| bochecha | indeed | 15:14 |
| tristan | well, not oddly; I guess they specialize in the release number and thats it, not everyone does a tarball | 15:15 |
| bochecha | well, the service was made by the Fedora team | 15:15 |
| bochecha | it's used to track upstream releases of Fedora packages | 15:16 |
| bochecha | and Fedora is very much built entirely from tarballs | 15:16 |
| tristan | makes sense | 15:16 |
| tristan | the purpose is for humans to know when to manually intervene I guess | 15:16 |
| bochecha | yeah | 15:16 |
| bochecha | we have a bot in Fedora which listens to events emitted by anitya about new releases, then it will try building a new version of the package, and dump the results in a bug report | 15:17 |
| bochecha | as a packager, you know there's a new version upstream, you get a patch to update your package, and build logs | 15:17 |
| tristan | but yeah the release team has a script they use to update jhbuild moduleset xml with newest tarballs | 15:17 |
| tristan | so I *think* something rudimentary can be worked out of that | 15:17 |
| bochecha | it doesn't remove the need for human judgement about whether the update is safe, but it saves some of the grunt work :) | 15:17 |
| bochecha | I'm asking the anitya developer (he's a friend) whether they'd consider adding tarball urls in their API responses | 15:18 |
| tristan | well, there is also relying on a service vs not relying on one | 15:19 |
| bochecha | indeed | 15:20 |
| bochecha | adding tarball urls to their responses makes sense anyway, whether bst uses it or not | 15:20 |
| tristan | Perhaps both could be okay, if the service brings something great over directly poking ftp directories | 15:20 |
| bochecha | well, in some cases its http/ftp directories | 15:21 |
| bochecha | in other it's enumerating releases in a github project | 15:21 |
| bochecha | there are lots of different cases :) | 15:21 |
| tristan | In the cases you want to download a tarball from github ? | 15:22 |
| bochecha | some people do | 15:22 |
| tristan | well, you could use the associated git sha for that but yeah you might want the tarball for some reason | 15:22 |
| tristan | and sourceforge has always been a bit weird | 15:22 |
| bochecha | and then there's cpan, pecl, pypi, rubygems, ... | 15:22 |
| bochecha | I count 25 backends in anitya | 15:23 |
| bochecha | most of them I've never heard about ^_^ | 15:23 |
| tristan | So you would think of having different handlers for all of these ? Or shouldnt just a sort of try pattern, or list of try patterns for URIs should do the trick no ? | 15:24 |
| bochecha | to try what? | 15:24 |
| bochecha | no all of these actually give you a directory listing | 15:24 |
| tristan | like "https://download.group.org/releases/app/{major}.{minor}/app-{major}.{minor}.{micro}.tar.gz" ? | 15:25 |
| bochecha | some of them you have to call the API, then you get a compressed json response which you parse... | 15:25 |
| bochecha | or like pypi : https://pypi.python.org/packages/9a/57/73447be9e7d0512d601e3f0a1fb9d7d1efb941911f49efdfe036d2826507/pyramid-1.9.1.tar.gz | 15:25 |
| tristan | hmmm I guess tracking with that kind of input would be weird though, you want directory listings | 15:25 |
| bochecha | that directory effectively only contains one version | 15:26 |
| bochecha | since the hash of the tarball is in the path of the directory | 15:26 |
| bochecha | so what do you list? :) | 15:26 |
| tristan | yeah exactly | 15:26 |
| bochecha | to be honest, that's why I was thinking of a dedicated source kind | 15:27 |
| bochecha | there are so many ways people put tarballs on servers, trying to figure it all out seems like a lost battle | 15:27 |
| tristan | github-tar, pypi-tar, etc | 15:27 |
| bochecha | so a `pypi` source kind would know how to query pypi to get the latest version when running `bst track` | 15:28 |
| tristan | and it would require some python related host tool to do it | 15:29 |
| bochecha | and that doesn't solve the problem for all tarballs, but at least it's solved for some... and someone else can come up and solve it for rubygems, or npm, or... | 15:29 |
| bochecha | I'm not sure what you mean by "python related host tool" | 15:29 |
| tristan | maybe with pypi you do it directly in python | 15:30 |
| bochecha | (also, I realize I'm taking a lot of your time with this discussion, feel free to leave me rambling alone if you have more pressing things to do :) ) | 15:30 |
| tristan | :) | 15:30 |
| *** tiagogomes has quit IRC | 15:50 | |
| bochecha | can a project use build artifacts from another project? | 15:51 |
| bochecha | I have a project which is a runtime, and then projects for apps which all use that runtime, and when building the apps I'd like bst to just reuse the built artifacts for the runtime | 15:52 |
| bochecha | or do I have to export my runtime to an ostree repository, and then in the apps use an import element to import that runtime? | 15:53 |
| tristan | bochecha, until https://wiki.gnome.org/Projects/BuildStream/Roadmap/RecursivePipeline you need to export/import | 15:57 |
| tristan | actually that is next on my todo list | 15:57 |
| tristan | I have some things I need to finish up first | 15:57 |
| bochecha | \o/ | 15:58 |
| *** jonathanmaw has quit IRC | 16:23 | |
| gitlab-br-bot | push on buildstream@70-third-party-plugin-sharing (by Tristan Maat): 2 commits (last: Add plugin dependencies) https://gitlab.com/BuildStream/buildstream/commit/a699262e4ced29a17043bdfa68b61d258ab6f1fd | 16:48 |
| tlater | tristan: I'd really like to quickly go over how I've implemented this so far, just to see if it's acceptable; it seems like most of the work will be documenting the process of writing plugins. | 16:52 |
| tlater | I suppose that will just have to happen on Monday then :/ | 17:09 |
| *** tlater has quit IRC | 17:12 | |
| tristan | oops :-/ | 17:50 |
| gitlab-br-bot | buildstream: merge request (patch-source->master: WIP: Add patch source plugin) #86 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/86 | 18:14 |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 6 commits (last: Removing artifact cache tests.) https://gitlab.com/BuildStream/buildstream/commit/d1d5873ff50750c092a762ef851ff6ae915c414f | 18:18 |
| gitlab-br-bot | buildstream: merge request (patch-source->master: WIP: Add patch source plugin) #86 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/86 | 18:19 |
| gitlab-br-bot | buildstream: merge request (patch-source->master: WIP: Add patch source plugin) #86 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/86 | 18:34 |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 1 commit (last: Fixed, lets try it) https://gitlab.com/BuildStream/buildstream/commit/6a9e8b455e9e59c7a9182b37523eaf1b413d07b9 | 19:04 |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 1 commit (last: Fixup) https://gitlab.com/BuildStream/buildstream/commit/fddec9f5fe0bd1ae297e46cd8dd91ab94206d9d3 | 19:06 |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 3 commits (last: .coveragerc: Configuring so that reports from different environments can be combined) https://gitlab.com/BuildStream/buildstream/commit/3d6e198dfa0eed92414f949c84128c4c46eb6f1a | 19:58 |
| *** tristan has quit IRC | 20:04 | |
| *** bochecha has quit IRC | 21:02 | |
| *** bochecha has joined #buildstream | 21:31 | |
| *** xjuan has quit IRC | 22:39 | |
| *** tristan has joined #buildstream | 23:33 | |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 1 commit (last: Fixup) https://gitlab.com/BuildStream/buildstream/commit/b03cbf01920fd16cf6d4f4541fc9cb8698309ef4 | 23:41 |
| gitlab-br-bot | push on buildstream@test-coverage-reports (by Tristan Van Berkom): 1 commit (last: .gitlab-ci.yml: Produce combined coverage report) https://gitlab.com/BuildStream/buildstream/commit/0e07c5521531fc4ca0b1346bf28df5ad8a5fb5a7 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!