| gitlab-br-bot | push on buildstream@testing-shell (by Tristan Van Berkom): 2 commits (last: element.py: Changed Element._shell() behaviors.) https://gitlab.com/BuildStream/buildstream/commit/4780f71105e6494cb63a34e551cba1a4e8c75a37 | 00:27 |
|---|---|---|
| gitlab-br-bot | push on buildstream@testing-shell (by Tristan Van Berkom): 1 commit (last: _sandboxbwrap.py: Make terminal control setting conditional on stdin being a tty) https://gitlab.com/BuildStream/buildstream/commit/6e6924fc15b85763d83ed917e74b653f05655ffd | 01:22 |
| gitlab-br-bot | push on buildstream@testing-shell (by Tristan Van Berkom): 2 commits (last: _frontend/main.py: Changes to the bst shell command) https://gitlab.com/BuildStream/buildstream/commit/9270d4214cfb66088dd8cf67123047229a08a541 | 03:07 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: element.py: Changed Element._shell() behaviors.) https://gitlab.com/BuildStream/buildstream/commit/4780f71105e6494cb63a34e551cba1a4e8c75a37 | 03:18 |
| gitlab-br-bot | buildstream: issue #86 ("bst shell --command requires a shell") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/86 | 03:18 |
| *** exalm has joined #buildstream | 04:18 | |
| *** bochecha has joined #buildstream | 08:22 | |
| *** bochecha has quit IRC | 08:27 | |
| *** bochecha_ has joined #buildstream | 08:28 | |
| *** jonathanmaw has joined #buildstream | 08:54 | |
| *** jude has joined #buildstream | 08:59 | |
| *** tlater has joined #buildstream | 09:20 | |
| gitlab-br-bot | push on buildstream@24-better-validation-for-loaded-yaml (by Tristan Maat): 7 commits (last: Add _yaml.validate_node) https://gitlab.com/BuildStream/buildstream/commit/e7612f42a4e53140204adacecfda50788aeca710 | 11:10 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 11:10 |
| gitlab-br-bot | push on buildstream@24-better-validation-for-loaded-yaml (by Tristan Maat): 20 commits (last: _artifactcache/artifactcache.py: Removed remove()) https://gitlab.com/BuildStream/buildstream/commit/e368f29bfc0f1df7ca91dc023cfce69670b78a96 | 11:11 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 11:11 |
| gitlab-br-bot | push on buildstream@70-third-party-plugin-sharing (by Tristan Maat): 2 commits (last: Add plugin dependencies) https://gitlab.com/BuildStream/buildstream/commit/f4ee0efafc4565a5f6dad0d1f24f56bcab5713c6 | 11:41 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 11:57 |
| *** tlater has quit IRC | 12:02 | |
| *** tlater has joined #buildstream | 12:25 | |
| bochecha_ | huh | 12:43 |
| bochecha_ | pushreceive: INFO: Executing ssh -l bochecha -p 22 richmond.toniob.net bst-artifact-receive --verbose /srv/www/daitauha.fr/www/static/ideascube/artifacts | 12:44 |
| bochecha_ | pushreceive: INFO: Receiving repository information | 12:44 |
| bochecha_ | bash: bst-artifact-receive: command not found | 12:44 |
| bochecha_ | that's what I get when I try pushing artifacts :-/ | 12:44 |
| bochecha_ | yet, bst opens a login shell (`ssh -l`) which means the ~/.profile file should be read | 12:45 |
| bochecha_ | and that files adds ~/.local/share/bin (where bst-artifact-receive is installed) to the PATH :-/ | 12:45 |
| bochecha_ | same with setting the PATH in the ~/.bashrc file anyway | 12:46 |
| bochecha_ | oh no! | 12:47 |
| bochecha_ | > If command is specified, it is executed on the remote host instead of a login shell. | 12:47 |
| bochecha_ | ok, so that explains it :-/ | 12:47 |
| gitlab-br-bot | push on buildstream@24-better-validation-for-loaded-yaml (by Tristan Maat): 5 commits (last: project.py: Add project config node validations) https://gitlab.com/BuildStream/buildstream/commit/ae6e7e335457b7377e5bb56f9721067d124001fc | 14:00 |
| gitlab-br-bot | push on buildstream@24-better-validation-for-loaded-yaml (by Tristan Maat): 4 commits (last: context.py: Add user config node validations) https://gitlab.com/BuildStream/buildstream/commit/1661053b50c474af90868ce44f17c3a4754cc79b | 14:24 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 14:31 |
| gitlab-br-bot | buildstream: merge request (24-better-validation-for-loaded-yaml->master: Resolve "Better validation for loaded YAML") #87 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/87 | 14:54 |
| gitlab-br-bot | push on buildstream@70-third-party-plugin-sharing (by Tristan Maat): 17 commits (last: Removing artifact cache tests.) https://gitlab.com/BuildStream/buildstream/commit/d1d5873ff50750c092a762ef851ff6ae915c414f | 15:05 |
| tristan | tlater, do you have time to look over the plugins stuff together before I jump on a bus and get to a proper coffee shop ? | 15:25 |
| tlater | tristan: Yup | 15:25 |
| tlater | Hang on just a second, I'll link you to the current plugins repo... | 15:26 |
| tlater | https://gitlab.com/tlater/buildstream-plugins | 15:27 |
| tlater | With the current interface all a plugin repo needs to do is define entry_points with a specific key | 15:27 |
| tlater | Using plugins from such a repo is a bit less clear | 15:28 |
| tlater | I don't think I like the idea of 'requiring' a plugin, since we already capture that idea with namespaces | 15:28 |
| tristan | I'm looking at the branch now | 15:29 |
| tristan | setuptools and entry points | 15:29 |
| tlater | Simply specifying 'buildstream-plugins;autotools' as the kind seems more sensible. | 15:29 |
| tristan | buildstream's plugins should not need the namespace though | 15:29 |
| tlater | They don't with the current implementation | 15:30 |
| tlater | 'buildstream' could be a special namespace just for consistency, too, even if not required. | 15:30 |
| tristan | well I see your point about not requiring packages explicitly, you will anyway get an error | 15:30 |
| tristan | that a plugin is missing | 15:30 |
| tristan | And perhaps if the error comes with a prefix that works | 15:31 |
| tristan | Also the plugins are individually format revisioned already | 15:31 |
| tristan | So you should be able to require a version of plugin 'foo:bar' | 15:31 |
| tlater | Would that translate to 'kind: buildstream-plugins;autotools:1.0'? | 15:32 |
| tristan | Is there a way we can continue to use pluginbase for these packages too though ? | 15:32 |
| tristan | tlater, no you require them once in your project.conf | 15:32 |
| tlater | Unfortunately there's no way pluginbase can work with this, there is no way to access the original source file. | 15:33 |
| tlater | pluginbase explicitly says that it's incompatible | 15:33 |
| tristan | So can we get setuptools to report a directory ? | 15:33 |
| tristan | And share an install directory with buildstream ? | 15:33 |
| tlater | Not as far as I've seen - especially not since things might be packaged as zipfiles | 15:34 |
| tristan | There must be a way | 15:34 |
| tlater | We could maybe hack it with __file__, but I don't think that would work well, if only because zipfiles can happen. | 15:34 |
| tristan | tlater, there is a reason we use pluginbase | 15:35 |
| tristan | So I'm trying to find out how to properly share data files/directories between packages | 15:35 |
| tlater | Why do we use pluginbase? | 15:36 |
| tristan | because while it might be possible that one can assume the plugin sets loaded for two pipelines in the same python interpretor are the same, for installed plugins; I'd rather not break that | 15:36 |
| tristan | pluginbase is a namespaced loader, so the set of plugins in two loaded Pipelines are completely isolated, intentionally | 15:37 |
| tristan | if for instance with recursive pipelines, I want to explicitly use this package of "foo" for every "foo:bar", I will no longer be able to offer that possibility if we throw away pluginbase | 15:37 |
| tristan | The buildstream package structure is also very delicately constructed to ensure that buildstream plugins are not part of the same package namespace | 15:38 |
| tristan | So we make sure we are always exercising the same codepath for loading plugins | 15:38 |
| tristan | external or internal | 15:39 |
| tlater | tristan: Doesn't setuptools' 'dist' property cover that as well? | 15:39 |
| tristan | Cover what ? | 15:40 |
| tlater | Namespaces - to be a different 'dist' you need to be a completely separate package. | 15:40 |
| tristan | tlater, what if I want 1.0 of foo in pipeline A and 1.2 of foo in pipeline B | 15:41 |
| tristan | in any case, setuptools is something I want to trust less, not more | 15:41 |
| tristan | it's completely inappropriate crap for our use cases, we're barely forcing it against it's will to do what we need | 15:41 |
| tlater | tristan: Hm, alright, I understand. | 15:42 |
| tlater | I'll try and see if there's a way to hack this in, though I'm pretty sure from past experiences that it will break on zipfiles | 15:42 |
| tristan | Then there is another way | 15:42 |
| tristan | To ensure they are not zip files | 15:43 |
| tristan | When installing a plugin package, that package requires buildstream; which means buildstream is there is to inform it already with some metadata that "you must put your plugins *here*, not in some zip file" | 15:43 |
| tristan | I know, setuptools is crap, and I've been pondering what the future will hold | 15:44 |
| tristan | wonder if we can switch to something like meson perhaps | 15:44 |
| tlater | tristan: We can't enforce installation, though, that's up to the package maintainer. Unless we roll our own package installation script that runs on top of setuptools. | 15:45 |
| tlater | Actually, that could work, something like buildstream.setuptools | 15:45 |
| tlater | Which just overrides something very minor to enforce non-zip plugins | 15:46 |
| tristan | I'm not sure what you mean about enforcing installation | 15:50 |
| tlater | tristan: Enforce no zipfiles during installation | 15:50 |
| tristan | tlater, the user should be told that the package was not found and needs to be installed right ? | 15:50 |
| tristan | tlater, we certainly must be able to, for instance anything installed in data_files would break right ? | 15:51 |
| tristan | it has to be readable back from %{prefix}/path/to/resource | 15:51 |
| tlater | tristan: Right, yes, that's necessary anyway. | 15:51 |
| tristan | certainly you cannot install man pages in a zip | 15:51 |
| tlater | tristan: Would it be alright to override the setuptools.setup function to ensure that it contains the no-zipfile flag, and mandate that buildstream plugins use our custom buildstream.setup function? | 15:53 |
| tlater | tristan: I think that will be the closest to a user-friendly way to do this. | 15:53 |
| tristan | Hrm | 15:54 |
| tristan | Well, I guess the API for plugin installation doesnt have to be as golden stable as everything else :-/ | 15:55 |
| tristan | tlater, I'm not seeing the how right now but yeah sure you can, lets make sure we're using pluginsbase and not setuptools for loading plugins, and make sure the project/user facing API for requiring plugins is something we're comfortable with long term | 15:58 |
| tristan | considering that setuptools is something we probably need a real replacement for, we'll hopefully end up yanking it out before too long | 15:59 |
| tlater | tristan: Is the 'kind: <dist>;<plugin>' syntax enough? I think it's pretty sensible, though I'm not sure where versions would move into this. | 16:00 |
| tristan | so then we may have to change the story of how plugins get installed unfortunately; but at least the API for projects to require them will remain the same | 16:00 |
| tristan | I would stick to colon and not semicolon | 16:01 |
| tristan | just like the source aliases | 16:01 |
| tristan | In fact, there may even be an aliasing game that might one day be fun at that level | 16:01 |
| tristan | but anyway | 16:01 |
| tristan | tlater, to answer your last more concretely; I think it's enough to address a plugin from a project element, I'm not *sure* that we dont need anything explicit in project.conf about the plugin package, but in any case if we do; it looks like that can later be added in an API compatible way | 16:09 |
| tristan | so for now, addressing a plugin from a package as <package>:<plugin> being the only thing, is fine | 16:10 |
| *** jude has quit IRC | 16:17 | |
| tristan | tlater, looks like maybe http://setuptools.readthedocs.io/en/latest/pkg_resources.html#resource-extraction might be of help | 16:47 |
| tlater | tristan: Interesting, last I used that function it did not extract files | 16:51 |
| tristan | was the resource_name meaningful ? | 16:52 |
| tlater | tristan: I'll check, I think this is how I made a plugin system in the past | 16:53 |
| tristan | resource_listdir() should tell I think | 16:53 |
| *** exalm has left #buildstream | 16:53 | |
| tlater | This works: pkg_resources.resource_string('elements', 'autotools.py') | 16:56 |
| tlater | The package module actually has to be sensible, but that's not too bad, and extending with '.py' automatically should be fine. | 16:57 |
| tlater | Hmmm | 16:57 |
| tlater | Although now two packages could technically define the same module, which would break things | 16:58 |
| tristan | sigh | 16:59 |
| tristan | can we solve the big problem now ? what can we put off till later | 16:59 |
| tristan | I looked at meson python support and it looks like we would have to change a lot (and the support is not that amazing either) | 17:00 |
| tristan | at least looking at: https://github.com/mesonbuild/meson/tree/master/test%20cases/python3 | 17:00 |
| tristan | tlater, so we can consider that a python package is a valid source of loading packages, in which case the namespace should be the unique package name according to the package manager right ? | 17:01 |
| tlater | tristan: Probably just something in the documentation I haven't found yet - there is no way this function would *require* every package to have different module names | 17:01 |
| tristan | In which case there can never be two | 17:01 |
| tristan | More than one separate package can have the same modulename but that would be fine | 17:02 |
| tristan | In the future, we can add other sources for loading packages | 17:02 |
| tlater | tristan: Yes, but this function doesn't seem to use python packages, but rather python modules, and presumably picks the correct file randomly. | 17:02 |
| tristan | instead of just "your pip managed mess" | 17:02 |
| tristan | So the module needs to be an __init__.py, and the actual plugin has to be in a subdirectory of that separated by a directory that has no __init__.py ? | 17:04 |
| tristan | yuck | 17:04 |
| tristan | maybe setting the name="packagename" in setup() should be enough ? | 17:04 |
| tlater | tristan: That could be it, I named the package buildstream-plugins, it might not like the dash | 17:05 |
| tristan | it wouldnt no | 17:05 |
| tristan | I like dashes, python wouldnt | 17:05 |
| tlater | Hm, no, that isn't it either. Damnit this documentation needs links. | 17:06 |
| *** jude has joined #buildstream | 17:07 | |
| tristan | tlater, I think you need to do this in the shell | 17:08 |
| tristan | maybe you are anyway :) | 17:08 |
| tlater | tristan: I even installed ipython x) | 17:08 |
| tristan | seems to be a common python thing | 17:09 |
| tristan | poking at live apis and seeing what they do, and printing their docstrings | 17:09 |
| * tristan better check the next bus time | 17:10 | |
| tristan | Oh... before I go... | 17:10 |
| tristan | ANY thoughts on the email which is too long to read ? | 17:10 |
| * tlater No, but probably not meant for that question either. I like the cool keys | 17:11 | |
| tlater | ;) | 17:11 |
| tristan | https://mail.gnome.org/archives/buildstream-list/2017-September/msg00000.html | 17:11 |
| tristan | Particularly, I think the S-Expressions are going to be winner | 17:12 |
| tristan | I've used those to automate queries in custom orm-ish setups (EDS does that too), they save a lot of typing | 17:13 |
| tristan | and they are also easily parseable (just additional structured recursive lists of lists) | 17:13 |
| tristan | I personally wish I could take away the quotes in the special keys | 17:14 |
| tristan | that would make it not quite yaml but slightly augmented yaml | 17:14 |
| tristan | and would require a bit of tinkering in the parser | 17:14 |
| tristan | Or; possibly just a preprocessing phase | 17:15 |
| tristan | I think there is precedence, I could of swore I saw some gitlab docs for special keys like `>>:` (without quotes) | 17:15 |
| tristan | that was what inspired that idea of using tokens like that | 17:16 |
| *** jonathanmaw has quit IRC | 17:17 | |
| tristan | Ahhh right that is part of yaml itself | 17:17 |
| * tristan palmface | 17:18 | |
| tristan | https://stackoverflow.com/questions/41063361/what-is-the-double-left-arrow-syntax-in-yaml-called-and-wheres-it-specced | 17:18 |
| tristan | eg | 17:18 |
| tristan | still viable but I wonder what we can do better... | 17:18 |
| tlater | tristan: I found the magic formula \o/ | 17:19 |
| tlater | "package.dist.get_resource_string(None, package.module_name.replace('.', os.sep) + '.py')" | 17:19 |
| tristan | tlater, mhm so... this is basically allowing us to use the pip installation as a dictionary of plugin packages | 17:20 |
| tlater | Yup. This should work for the .yaml templates too, though we should add specialized loading for those files too. | 17:20 |
| tristan | I think that this is decent first step, even without anything explicit in the project talking about plugin packages | 17:20 |
| tristan | Later if we want to evolve on that and treat plugins more strictly (it's interesting to tie third party plugins to stricter revisioning policy) | 17:21 |
| tristan | then we can add something to project.conf, telling buildstream exactly where to load the package from | 17:21 |
| tristan | but in any case, we use pkg_resources as a dictionary and it means we'll be able to load system wide or user level installations | 17:22 |
| tristan | and we still use the same loading mechanics, ok sounds good to me | 17:22 |
| tlater | Anyway, it's almost 6:30 here, I think I'll implement that tomorrow. | 17:23 |
| tristan | yup | 17:24 |
| tristan | good evening ! | 17:24 |
| tlater | o/ | 17:24 |
| *** tlater has quit IRC | 17:25 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Generating man pages) https://gitlab.com/BuildStream/buildstream/commit/7bc2e070182fc45b9e6d630e95d5e90d97fc910e | 17:25 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch testing-shell | 17:25 |
| gitlab-br-bot | buildstream: merge request (interactive-bst-shell-cmd->master: Allow forcing an interactive 'bst shell') #85 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/85 | 17:32 |
| *** tristan has quit IRC | 17:39 | |
| *** tristan has joined #buildstream | 18:18 | |
| *** ChanServ sets mode: +o tristan | 18:18 | |
| *** jude has quit IRC | 18:30 | |
| *** tristan has quit IRC | 20:02 | |
| *** tristan has joined #buildstream | 20:02 | |
| *** ChanServ sets mode: +o tristan | 20:02 | |
| tristan | So here is some fun little code: https://bpaste.net/show/ee645f3d5ecd | 20:03 |
| tristan | A little writeup sexp variable evaluation expression code | 20:03 |
| tristan | condition: (and (opt "debug") (ifeq (opt "logging") "max")) | 20:04 |
| *** tristan has quit IRC | 20:20 | |
| *** bochecha has joined #buildstream | 21:01 | |
| *** bochecha has quit IRC | 21:02 | |
| *** bochecha has joined #buildstream | 21:02 | |
| *** bochecha_ has quit IRC | 21:03 | |
| *** jude has joined #buildstream | 21:03 | |
| *** bochecha has quit IRC | 21:12 | |
| *** bochecha has joined #buildstream | 21:13 | |
| *** bochecha has quit IRC | 21:17 | |
| *** bochecha has joined #buildstream | 21:17 | |
| *** bochecha has quit IRC | 21:22 | |
| *** bochecha has joined #buildstream | 21:23 | |
| *** bochecha has quit IRC | 21:27 | |
| *** bochecha has joined #buildstream | 21:27 | |
| *** bochecha has quit IRC | 21:33 | |
| *** bochecha has joined #buildstream | 22:49 | |
| *** bochecha has quit IRC | 23:02 | |
| *** bochecha has joined #buildstream | 23:03 | |
| *** bochecha has quit IRC | 23:07 | |
| *** bochecha has joined #buildstream | 23:07 | |
| *** bochecha has quit IRC | 23:22 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!