| *** mcatanzaro has quit IRC | 01:10 | |
| *** tristan has joined #buildstream | 05:24 | |
| *** jude has joined #buildstream | 08:02 | |
| *** bethw has joined #buildstream | 08:31 | |
| gitlab-br-bot | push on buildstream@132-loading-external-plugins-works-without-explicit-requirement-in-project-conf (by Jonathan Maw): 20 commits (last: main.py: Fix app initialization) https://gitlab.com/BuildStream/buildstream/commit/4c73b85dbf6e84c710dc55bc756b02c7952ba764 | 09:30 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: WIP: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 09:30 |
| *** ssam2 has joined #buildstream | 09:47 | |
| *** valentind has joined #buildstream | 09:49 | |
| *** tpollard has joined #buildstream | 09:50 | |
| tristan | ssam2, just wanted to let you know that now we have Element.compute_manifest(), which should unblock your https://gitlab.com/BuildStream/buildstream/merge_requests/140 | 09:54 |
| ssam2 | excellent | 09:56 |
| *** ssam2 has quit IRC | 09:57 | |
| *** ssam2 has joined #buildstream | 09:57 | |
| tristan | tlater, if I press the rebase button on your branch, am I going to see if it works with coverage pinned at 4.2 ? | 10:11 |
| tristan | is that a good idea ? | 10:11 |
| *** bethw has quit IRC | 10:17 | |
| *** bethw has joined #buildstream | 10:36 | |
| tlater | tristan: I think so | 10:38 |
| * tlater will try that | 10:38 | |
| tristan | thanks ! | 10:38 |
| tristan | juergbi, so I have been thinking a bit about logging in conjunction with recursive pipelines; maybe you noticed the chatter here in IRC some days back | 10:38 |
| juergbi | about enabling logging early on? | 10:39 |
| tristan | juergbi, I suppose that your branch at least partially changes things in that regard, as it needs that extra pre-step to fully load a pipeline | 10:39 |
| tristan | yeah | 10:39 |
| juergbi | improving logging is still on my todo | 10:39 |
| tristan | It's probably going to help Angelos's project for benchmarking, and will also clean some things up | 10:39 |
| tristan | alright... do you think it's something that can be postponed or does it need to land with the main recursive pipelines branch ? | 10:40 |
| tristan | juergbi, I've been thinking about this a bit... and it would be nice to remove all that "ticker" code in favor of proper logging, but today it struck me... maybe all of our status logging should be going to stderr ? | 10:41 |
| juergbi | i think something in this aspect will have to land with the main branch | 10:41 |
| tristan | Ok | 10:41 |
| gitlab-br-bot | buildstream: Tristan Maat created branch tracking-changes | 10:41 |
| juergbi | but it could be a minimal approach with improvements coming later on | 10:41 |
| tristan | So good to raise this now I think | 10:41 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 10:41 |
| tristan | juergbi, mostly I wanted to just share some ideas in case you were going to address that stuff "anyway" | 10:41 |
| juergbi | yes, ta | 10:42 |
| tristan | juergbi, one of the reasons we have the tickers (obnoxious things really) separate is for `bst show` | 10:42 |
| juergbi | for the curses-like UI part, stderr may not make sense | 10:42 |
| tristan | and that's the reason it goes to stderr while everything else goes to stdout | 10:42 |
| tristan | Ah | 10:43 |
| tristan | So needs some thought maybe - indeed some terminals might not like ANSI control codes happening on stderr ? | 10:43 |
| juergbi | as long as stdout and stderr are connected to the same terminal, i don't expect any real issues | 10:44 |
| tristan | My thoughts were, if it is simpler; do all logging on stderr - and only a few things in stdout | 10:44 |
| juergbi | but it would be a bit odd to use both | 10:44 |
| tristan | i.e. reserving stdout for things which programs / scripts may want to consume | 10:44 |
| juergbi | i'll give it some thought | 10:44 |
| tristan | So I think we have like `bst workspace list` which prints some yaml | 10:44 |
| tristan | that should go to stdout, and of course... the `bst show` content should too | 10:45 |
| juergbi | as we only use the curses-like UI when we're connected to a terminal (where stdout and stderr are anyway the same), it may not be an issue to mix | 10:45 |
| tristan | but now that we have some complex stuff to print ahead of the `bst show` result, we would want that stuff going to stderr | 10:45 |
| tristan | and only deliver the formatted, desired output to stdout | 10:45 |
| juergbi | yes, that definitely makes sense | 10:45 |
| tristan | that approach should also let us safely kill those "tickers", which take up a lot of lines of code | 10:46 |
| jonathanmaw | \o/ jubilation! cmake-test didn't segfault! | 10:46 |
| tristan | only for the sake of loading separately (initially so that `bst show` output would be parseable) | 10:46 |
| tristan | jonathanmaw, :) | 10:46 |
| tristan | jonathanmaw, yeah, its coming together, things are unblocking now that we side step that coverage bug | 10:47 |
| juergbi | tristan: i.e., you'd just inline logging instead of going via tickers? | 10:47 |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 10:47 |
| tristan | juergbi, yes, we'd use the regular messaging stuff for that, enable the logging earlier; it would mean the summary will be printed after some initial loading logs; but that seems reasonable | 10:48 |
| tristan | juergbi, also we'd want to do the loading with the "silent_nested" timed_activity() approach (silencing every message that is not in the 'unconditional' mask) | 10:48 |
| tristan | juergbi, because at that stage, we interrogate caches and get a ton of annoying `git cat-file` host commands that we wouldnt want showing up in the master log | 10:49 |
| juergbi | ok, sounds reasonable. will take a look | 10:49 |
| juergbi | yes, we don't want to print all of that | 10:49 |
| tristan | note that it also means that eventually we can parse the logs directly to make benchmarking results | 10:49 |
| tristan | which will be very cool :) | 10:49 |
| tristan | time the loads etc, maybe with an option to print finer grained timers and format the log lines in a convenient way for a benchmarking program to parse | 10:50 |
| tristan | anyway, that's future stuff; but standardizing on logging opens the door for it :D | 10:50 |
| tlater | jonathanmaw: \o/ | 10:53 |
| tristan | and we haz fedora instructions now: http://buildstream.gitlab.io/buildstream/install.html#fedora ^_^ | 10:57 |
| * tlater frowns at gitlab only running one pipeline at a time | 10:58 | |
| tristan | It all depends on the day and direction of the wind | 10:58 |
| tlater | tristan: You told me I should have a look at this issue a couple of weeks ago, but I somehow never set up a MR for it: https://gitlab.com/BuildStream/buildstream/issues/142 | 11:02 |
| tlater | Iirc I had a quick discussion with ssam2 about what you meant with a 'stack' of provenances | 11:02 |
| tristan | tlater, myeah ... I need a refresher :-/ | 11:03 |
| * tlater isn't sure how to summarize, needs a refresher too | 11:04 | |
| tristan | tlater, so first step is: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_options/optionpool.py#L228 | 11:04 |
| tristan | tlater, see how _yaml.composite() is in that try block ? | 11:05 |
| tristan | tlater, that is a problem | 11:05 |
| tlater | Yup, but if we move it we can sometimes not get provenance | 11:06 |
| tristan | tlater, that basically means that if we get an error from _yaml.composite() which prints provenance; we prepend a second provenance about the self.evaluate() conditional to it | 11:06 |
| tristan | tlater, that is the other half of the problem | 11:06 |
| tristan | tlater, _yaml.composite() has to *guarantee* that it will print provenance in it's LoadError()s | 11:06 |
| tristan | tlater, the "provenance stack" idea... is to basically keep track of provenances while _yaml.composite() recurses | 11:07 |
| tristan | so that there is always a provenance for _yaml.composite() to print | 11:07 |
| tristan | at the entry point of it... there is always a provenance, because the passed node is _always_ a dict (or a "Mapping" more precisely) which has a provenance | 11:08 |
| tlater | Hm, wouldn't this be slightly less accurate than it was before? | 11:09 |
| tristan | No, why ? | 11:10 |
| tlater | Right, I didn't really read into where the parent is getting the provenance from | 11:10 |
| tristan | We are currently printing the provenance of the option conditional statement when "anything" is problematic inside of a resulting composition | 11:11 |
| tristan | this is incorrect | 11:11 |
| *** valentind has quit IRC | 11:17 | |
| gitlab-br-bot | buildstream: merge request (invoking_page_changes->master: Separated :show-nested: into individual calls) #157 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/157 | 11:22 |
| gitlab-br-bot | push on buildstream@sam/canonical-push-urls (by Sam Thursfield): 4 commits (last: Check connectivity to remote cache on `bst push`) https://gitlab.com/BuildStream/buildstream/commit/1b6625f48d55224913f5a97ac1b31ad6ff4de3b0 | 11:24 |
| gitlab-br-bot | buildstream: merge request (sam/canonical-push-urls->master: WIP: Canonical push/pull URLs (no backwards compatibility)) #158 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/158 | 11:25 |
| gitlab-br-bot | buildstream: merge request (sam/canonical-push-urls->master: WIP: Canonical push/pull URLs (no backwards compatibility)) #158 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/158 | 11:26 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 6 commits (last: Issue #113: Split tracking and saving in `bst build`) https://gitlab.com/BuildStream/buildstream/commit/f40aea65a397a56af3c7223fc0845d1704ec9b11 | 11:47 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 11:47 |
| tristan | nexus, so https://gitlab.com/BuildStream/buildstream/merge_requests/157 looks like a start, is that the right approach to documenting the UX ? | 11:49 |
| tristan | nexus, are you planning to fill in some docs related to each command ? or add a separate doc which links to those ? | 11:49 |
| tristan | nexus, also; please WIP prefix that; when I see a merge request without the WIP, it cries out to me: "I am a complete work and I am very ready to be reviewed and land on master !" | 11:50 |
| gitlab-br-bot | buildstream: merge request (invoking_page_changes->master: WIP: Separated :show-nested: into individual calls) #157 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/157 | 11:51 |
| tristan | :) | 11:51 |
| nexus | tristan: I was considering having the docs point to each of those commands as needed. That way it's useful for understanding exactly what you're doing when you run commands | 11:52 |
| tristan | nexus, sounds like a better, human oriented approach I agree | 11:52 |
| tristan | nexus, i.e. start out with "I want to build a project, what do I do ?" and... "When I've finished building the project, what can I do with it ? How can I get the output ?" | 11:53 |
| tristan | And move on to "Now I want to modify code and test the result", and "How can I modify this module and test it against something which depends on it ?" | 11:54 |
| tristan | that kind of thing is sorely needed | 11:54 |
| tristan | then, we'll want to link to this kind of docs from https://wiki.gnome.org/Newcomers/BuildSystemComponentBst | 11:54 |
| tristan | nexus, also; try building GNOME on your machine, and ask questions about anything you want to understand :D | 11:56 |
| nexus | That would make it fairly effective as a starter tutorial. I'd like for it to have enough of an explaination that from start to finish, you can look into what each thing you're doing actually does, so you can follow what's happening. As someone entirely new to this, i got utterly lost in the process | 11:56 |
| nexus | will do | 11:57 |
| * tlater 's branch also succeeds without a segfault now :) | 12:05 | |
| * tlater updates his issue on pycoverage | 12:05 | |
| nexus | \o/ | 12:11 |
| nexus | # Args: | 12:13 |
| nexus | 54 # context (Context): The BuildStream context | 12:13 |
| nexus | wut does this mean? | 12:13 |
| nexus | what is "The BuildStream context"? | 12:13 |
| tristan | nexus, in the source code ? | 12:14 |
| tristan | nexus, do you need to know that ? | 12:14 |
| tristan | I mean, I can tell you; but I dont see how it is at all relevant to the UX | 12:14 |
| nexus | tristan: i'm looking at another ticket that bethw gave me | 12:14 |
| tristan | only needed for writing code | 12:14 |
| tristan | Ah | 12:14 |
| tristan | nexus, Ok so, the Context is the "invocation context" really | 12:15 |
| nexus | invocation context? | 12:15 |
| tristan | it is mostly comprised of settings which are loaded from a collection of A.) The default configuration options.. B.) The user provided config file.. and C.) The command line options | 12:15 |
| tristan | Where C overrides B overrides A | 12:15 |
| tristan | nexus, it is also the central handle where logging is performed, and the frontend listens to it for messages propagated towards the user | 12:16 |
| tristan | so yeah, it's the context in which buildstream is invoked | 12:16 |
| tristan | the invocation context | 12:16 |
| nexus | thanks :) | 12:17 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 14 commits (last: Handle removed files from integration in compose plugin) https://gitlab.com/BuildStream/buildstream/commit/cecbbec7ae5a9e6f9021bb087427c41cdb2bd633 | 12:34 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 12:34 |
| tristan | tlater, many comments made; I'm not sure anymore that this is landing this week. | 12:51 |
| tlater | Alright, I'll start working through them... Want to discuss any of them before you're gone? | 12:54 |
| tristan | tlater, changing the whole philosophy of `bst build --track` to run two separate sessions is my main problem | 12:55 |
| tristan | I dont expect it to be easy | 12:55 |
| tristan | So I'm sure it's going to take you some time | 12:55 |
| tristan | Also, thorough test cases please for the wide variety of possibilities that are now opened with single/multiple --track and single/multiple --track-except options given | 12:56 |
| tlater | Yup, that makes sense. Do you think the `--track` option is alright as its defined here, i.e. when it can be specified multiple times and tracks each element recursively? | 12:57 |
| tristan | Yes | 12:58 |
| tristan | Seems sane, although, it's annoying that you need to specify a target all the time | 12:58 |
| tristan | when probably the --track target is the target itself | 12:59 |
| tristan | but meh, that doesnt matter immensely I guess | 12:59 |
| tristan | I suppose it can easily be improved after with a `--track-all` or `--track-targets` switch which automatically considers all of the build targets given, as recursive track targets | 13:00 |
| *** bethw has quit IRC | 13:03 | |
| *** cs_shadow has quit IRC | 13:10 | |
| tlater | Hm, would it be enough to run the tracking queue in the scheduler first and then call scheduler.run again afterwards with a different plan? | 13:12 |
| tlater | Or does everything *have* to be done in the same invocation of scheduler.run? | 13:12 |
| * tlater doesn't see anything that would keep the scheduler from being used like that | 13:13 | |
| jonathanmaw | tristan: do you have time to review https://gitlab.com/BuildStream/buildstream/merge_requests/125 ? | 13:13 |
| *** bochecha has quit IRC | 13:15 | |
| *** bochecha has joined #buildstream | 13:15 | |
| * tlater realizes that's no different from what he did before | 13:16 | |
| tristan | tlater, all in the same scheduler, that was the point of doing it together in the first place, I spend almost a week making it work very nicely, I hold it dear | 13:17 |
| tristan | i.e. when an element is tracked and fetched, it's ready to be built, building it starts before tracking all elements finish | 13:17 |
| tristan | thats the beauty of these scheduler queues | 13:17 |
| tlater | Oh, right, I didn't realize that | 13:18 |
| tlater | Yeah, I understand then, that's definitely wrong | 13:18 |
| tristan | tlater, a side effect is that we *need* to start taking element state more seriously | 13:18 |
| tristan | as discussed here: https://gitlab.com/BuildStream/buildstream/merge_requests/126#note_47825150 | 13:18 |
| tristan | we already have some suboptimal things happening, and actually workspaced builds already have a bug | 13:19 |
| tristan | well, not really sorry, they only have a bug in the incremental build branch | 13:19 |
| tristan | but; they *do* have a bug whereas, their cache key is recalculated twice, once before and once after the build | 13:19 |
| *** brejoc has joined #buildstream | 13:19 | |
| *** brejoc has left #buildstream | 13:20 | |
| tristan | (although at least currently, the bug doesnt have bad side effects) | 13:20 |
| * tristan was happy to see brejoc appear after so long ! | 13:20 | |
| *** bochecha has quit IRC | 13:20 | |
| tristan | oh well :-/ | 13:20 |
| tristan | jonathanmaw, I took a look and made some comments | 13:20 |
| *** bochecha has joined #buildstream | 13:20 | |
| jonathanmaw | ta tristan | 13:20 |
| tristan | jonathanmaw, can I infer from that branch that using an external plugin must refer to it's "package name" or smth ? | 13:21 |
| tristan | If so, is a "package name" even a thing ? | 13:21 |
| tristan | I.e. `kind: foo:bar` | 13:21 |
| tristan | I'm not sure if I'm reading that right, or if it's only in the guts of the tests and internal APIs which do that | 13:21 |
| tristan | jonathanmaw, that's the part I still feel iffy about | 13:22 |
| ssam2 | i had to do "kind: buildstream-external:x86image" to use the x86image plugin | 13:22 |
| jonathanmaw | tristan: yeah, it would check whether it finds a match for "foo:bar" | 13:22 |
| tristan | So how do package names work ? Is that documented ? | 13:22 |
| jonathanmaw | and like ssam2 said, when referring to external plugins, they're prefixed with the python package | 13:22 |
| tristan | Hopefully they are not something that assumes "all plugins come from pip" right ? | 13:22 |
| tristan | Oh crap | 13:22 |
| tristan | Ok so, if we're going to have a "package name", it should be something we define | 13:23 |
| tristan | there is no such thing as a "python package" as far as I'm aware | 13:23 |
| tristan | only that pip provides the first discovery mechanism of a directory of plugins we can load from | 13:23 |
| tristan | if that implies python wants to have a "package name", it's not really any of our business | 13:23 |
| ssam2 | a "python package" is any directory that contains a __init__.py file, to use Python's own terminology | 13:24 |
| tristan | If we're going to have package names; if those are useful, then we need to be the ones defining them | 13:24 |
| tristan | ssam2, there is no requirement for having an __init__.py in a directory of buildstream plugins | 13:24 |
| tristan | we have one, and it's purely because not having it causes the docs to break | 13:24 |
| *** bochecha has quit IRC | 13:25 | |
| ssam2 | it's needed for Python to consider it "a package". which is needed for you to be able to `import` it | 13:25 |
| tristan | It is not | 13:25 |
| *** bochecha has joined #buildstream | 13:25 | |
| tristan | ssam2, It is absolutely not needed for our loader (pluginsbase) to load plugins | 13:25 |
| ssam2 | that's probably doing some trickery | 13:26 |
| tristan | our definition of a plugin has always been a file, _not_ a directory | 13:26 |
| ssam2 | which is fine, i'm just pointing out that "python package" has a specific meaning to Python itself | 13:26 |
| tristan | Yes, and I'm pointing out that if we're going to use a "name", we have to define and control what that name is | 13:26 |
| ssam2 | sure. we could use the name of the directory that contains the file; that's what Python uses to control the name of "Python packages" | 13:28 |
| *** bochecha has quit IRC | 13:35 | |
| *** bochecha has joined #buildstream | 13:35 | |
| tristan | Sorry I had to run out | 13:41 |
| tristan | ssam2, jonathanmaw, I dont know what we have to do really, or if we need this package name at all, and where it can be defined | 13:42 |
| ssam2 | supplying some kind of package name seems like a good idea | 13:42 |
| tristan | to clarify; the "right way" IMO for plugins to be installed, is for buildstream to install a pkg-config file | 13:42 |
| tristan | and for the pkg-config file to specify a build-time prefix resolved directory via a variable name | 13:42 |
| ssam2 | otherwise, 3 separate individuals could publish repos containing 'vm-image' plugins, and maybe your project depends on other plugins from 2 of those repos | 13:42 |
| ssam2 | which 'vm-image' plugin does buildstream use, in that instance? | 13:43 |
| tristan | which can be introspected by modules which want to install plugins | 13:43 |
| tristan | i.e. "buildstream informs the module where it should install plugins" | 13:43 |
| tristan | that is the sane thing; the pip thing is the ugly workaround we're living with "right now" | 13:43 |
| tristan | becuase yuck, setuptools | 13:43 |
| tristan | Also, I should mention that the *very first test case* I added to buildstream, was to ensure that separate projects have separate plugin namespaces | 13:44 |
| tristan | in anticipation of recursive pipelines | 13:44 |
| ssam2 | that sounds like a good setup. I still think we'd want some kind of namespacing within /usr/share/buildstream/plugins | 13:44 |
| tristan | I.e. its totally possible for two separate projects to pull in a *different* plugin named "foo" | 13:44 |
| tristan | So, maybe the project needs to declare which version of "foo" it wants, and the elements still need only refer to it as "foo" | 13:45 |
| ssam2 | ah, that could make sense | 13:45 |
| tristan | Namespacing is fine, agreed it's useful, and maybe that's what the project could specify | 13:45 |
| tristan | But it needs to be under our control | 13:45 |
| tristan | i.e. what defines the name, etc | 13:45 |
| tristan | maybe /usr/lib/buildstream/plugins has an index | 13:46 |
| tristan | which plugin installation can update with paths to where their plugins are installed, possibly even at what version | 13:46 |
| tristan | That is all stuff we dont want to "solve right now" | 13:46 |
| tristan | but the API is | 13:46 |
| * tristan had to take a phone call, sorry for seeming to "storm out" or anything, was an inopportune moment | 13:47 | |
| tlater | tristan: It's so that pip can find the directory it wants to load the file from - this is only required for pip plugins | 13:49 |
| tristan | I think that two plugins of the same name need not be supported in the *same project* | 13:51 |
| tristan | Also, we want to veer away from parsing/deconstructing things that are represented in yaml, we do it for the aliases yes | 13:51 |
| tristan | So if we wanted that, we could specify that `kind` could be a dictionary | 13:52 |
| tristan | but still, probably not needed | 13:52 |
| tristan | We can be doing this from project.conf | 13:52 |
| jonathanmaw | ok, if I understand this discussion correctly, we want to make as much plugin loading config happen in project.conf as possible, so the value of "kind" is just the element's name | 13:52 |
| jonathanmaw | while current loading specifies plugins from pip with "$dist:$plugin_name", we should make the "$dist" value be defined as a source of plugins in project.conf | 13:53 |
| tristan | Yes that is a good way forward, we probably want to really consider project.conf well | 13:54 |
| tristan | And we also probably want to consider locally loaded plugins; I'm not aware of any projects using that at the moment and it's acceptable to break at this time | 13:54 |
| tristan | In which case; we probably want to remove the plugin-path attributes from the toplevel of project.conf and consider local plugins as a "separate plugin discovery method" | 13:55 |
| tristan | I.e. we probably want to be explicit about everything regarding a plugin, in project.conf | 13:56 |
| tristan | a plugin is assumed if it's core buildstream | 13:56 |
| tristan | if a plugin needs to be discovered, it needs to specify the discovery method, and any further information required by that method | 13:56 |
| tristan | I.e. for a local plugin, it's discovered with a project relative path | 13:56 |
| tristan | for a pip plugin, it's discovered with a pip "package name" | 13:57 |
| tristan | that project declaration probably wants to be consistent and not provide different ways of specifying plugin sources | 13:57 |
| tristan | Further, another question in the project.conf format is redundancy; what happens if I want to load 5 plugins from the same external non-core source ? | 13:58 |
| tristan | So, maybe we need the format for this to group by sources; list the discovery method, any data needed by that discovery method, and a list of element/source names which are to be discovered by that method | 13:59 |
| tristan | currently it looks like we have only: http://buildstream.gitlab.io/buildstream/projectconf.html#plugin-paths | 14:00 |
| tristan | and http://buildstream.gitlab.io/buildstream/projectconf.html#versioning | 14:01 |
| tristan | It's debatable if we want plugin versioning to fall into the scope of `plugins:`, I think we probably should though | 14:01 |
| jonathanmaw | tristan: ok, based off this, I've put together some suggested ways of formatting the data in project.conf: https://pastebin.com/LHscPegy | 14:08 |
| jonathanmaw | I think the third one most closely matches what you want if we're replacing required-versions | 14:09 |
| tristan | I just did a draft too | 14:10 |
| tristan | https://bpaste.net/show/a86c0e268939 | 14:10 |
| *** bethw has joined #buildstream | 14:10 | |
| * tristan looks at jonathanmaw's draft | 14:10 | |
| tristan | jonathanmaw, your last looks very similar to my draft | 14:12 |
| tristan | although a little bit different, my draft differs mostly only inasmuch as it is a little more verbose and explicit | 14:12 |
| tristan | Also, I took the liberty of merging the versioning, and declaring the "core" plugin origin | 14:13 |
| tristan | This makes version requirements of core buildstream plugins appear in the same way as external plugin version requirements | 14:13 |
| tristan | which I think is a good thing | 14:13 |
| tristan | jonathanmaw, I still like mine better :) | 14:14 |
| tristan | heh | 14:14 |
| jonathanmaw | tristan: yeah, my third variation doesn't consider versioning for core elements | 14:15 |
| tristan | There are few reasons though, for instance your draft uses a plugin specific attribute as a key in a dictionary | 14:15 |
| tristan | that is not extremely clear | 14:15 |
| tristan | Also, besides not being clear; it's totally plausible that a given plugin origin requires *more than one* origin specific attribute | 14:15 |
| tristan | which would immediately throw the parser into disarray | 14:16 |
| jonathanmaw | tristan: also, do we want to use the ordering for how these origins are listed in the yaml as a way of specifying loading order? | 14:16 |
| tristan | not loading order no, but perhaps priority | 14:16 |
| *** bochecha has joined #buildstream | 14:17 | |
| tristan | jonathanmaw, I think we need not care about this right away though; currently I believe the practice is to error out if more than one plugin of the same name is reachable by the same project | 14:17 |
| jonathanmaw | tristan: i.e. if the last one also provided a "git" source, it'd choose that one over any of the others? | 14:17 |
| tristan | jonathanmaw, however, it's a good point that in the future, we could use this to prioritize | 14:17 |
| jonathanmaw | tristan: ok | 14:17 |
| jonathanmaw | tristan: It'd give developers a way to prioritise their special sauce git source over buildstream's standard one, but I think since they could accomplish the same by removing it from their list of plugins in core, it's not a very high-priority task to consider. | 14:18 |
| tristan | this of course leaves the base project format version declaration out of the picture | 14:19 |
| tristan | jonathanmaw, note that I wrote CAN in that section where I wrote MUST in the other sections | 14:19 |
| jonathanmaw | yep | 14:19 |
| tristan | jonathanmaw, I dont think one MUST declare every plugin that they use from core | 14:19 |
| tristan | right | 14:19 |
| tristan | So yes its a good point you make | 14:20 |
| tristan | jonathanmaw, that it can be useful, although I dont want that functionality to block the release | 14:20 |
| tristan | i.e. for right now, I believe we raise a LoadError() if there is any name collision, and yes there is a shortcoming when one wants to "override the git source" | 14:21 |
| tristan | but making that possible in the future does not break API | 14:21 |
| tristan | While inspecting warts... I wonder, will variable namespacing become an issue: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/data/projectconfig.yaml#L6 ? | 14:25 |
| tristan | I've been thinking, ever since implementing the strip-commands to break them down into separate macros | 14:25 |
| tristan | What is our strategy for effecting changes to the default project.conf ? | 14:26 |
| tristan | While retaining compatibility with all projects ? | 14:26 |
| tristan | not an easy question | 14:26 |
| tristan | I suppose that it's API, and that anything introduced as global variables in default project.conf *must* always be used in the predicted way, and can never change | 14:27 |
| tristan | Or, can be enhanced, but at least the same names must be used to achieve the same things | 14:27 |
| tristan | Also, during stable iterations of BuildStream, changes to default variable values can *never* be made | 14:27 |
| tristan | Because we've already concluded that artifact cache key is expected to remain stable at least for a single stable release cycle | 14:28 |
| tristan | anyone care to throw some fuel on this fire ? | 14:28 |
| tristan | ;-P | 14:28 |
| tristan | I should not that: the change I made to `strip-binaries` was a "compatible" change (it added new variables, but `strip-commands` still serves the same purpose), however it *did* change artifact keys | 14:30 |
| tristan | I guess moral of that story: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/data/projectconfig.yaml#L6 <-- if there is anything less than ideal that anyone thinks needs to be changed: "NOW is the time" | 14:32 |
| *** tristan has quit IRC | 14:36 | |
| *** tristan has joined #buildstream | 14:42 | |
| tristan | nexus, the documentation link at https://wiki.gnome.org/Projects/BuildStream/ is the only documentation there currently is; and probably will be, I dont mind to break that down into separate links | 15:25 |
| tristan | but what I dont want, is to have various different docs floating around the project | 15:26 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/2695b7dd68c9fc91cad29b963211e7126c0acaa8 | 15:26 |
| *** mcatanzaro has joined #buildstream | 15:26 | |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 15:26 |
| tristan | from years of maintaining FOSS projects, it's been my experience that if you allow various tutorials and disparate wiki pages documenting a project, they will inevitably go out of date | 15:26 |
| tristan | nexus, which is why I would prefer to keep all the docs maintained in one place as much as possible | 15:27 |
| tristan | so, if a part of the docs is a reference manual for authoring projects, and another part of the docs is a user manual for using BuildStream, that is fine; most importantly they are both a part of the buildstream git repo and maintaining them is in lock step with the code | 15:28 |
| *** ssam2 has quit IRC | 15:30 | |
| *** ssam2 has joined #buildstream | 15:32 | |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/dc6c3c03c0a3310651117bdd6ff0c82d548826d5 | 15:54 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 15:54 |
| ssam2 | i'm finding it really slow working with the converted GNOME modulesets | 15:56 |
| ssam2 | like 3 minutes to run ldconfig for each build, another 3 minutes to run update-mime-database ... | 15:56 |
| ssam2 | and many minutes to check out the files for (as an example) gnome-font-viewer | 15:56 |
| ssam2 | top and iotop show pretty low CPU and IO utilization | 15:57 |
| tristan | ssam2, that is odd, I experience long times but not that long | 15:58 |
| tristan | less than a minute for integration, but with decent SSD | 15:58 |
| tristan | the base may have been growing, though | 15:59 |
| ssam2 | this is with a fairly average HDD | 15:59 |
| tristan | ssam2, I've been noticing slowness too, I'm not sure if it's "increased" or not | 16:00 |
| ssam2 | so far it's checked out 2GB in about 6 minutes ... about 5MB/sec | 16:01 |
| ssam2 | i would expect better when copying files from one place in the same partition to another... | 16:02 |
| ssam2 | i think it must be something other than simply a slow disk | 16:02 |
| tristan | ssam2, one thing is the fuse layer is applied only during integration commands | 16:04 |
| tristan | and it's currently not multithreaded | 16:04 |
| tristan | but, I'm not sure just how much overhead that should be adding | 16:04 |
| ssam2 | i'd expect bst to be high up in 'top' if that was the blocker, too | 16:04 |
| tristan | especially that programs which run integration commands are typically not multithreaded | 16:05 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/d803da90550ba00ac444c89e3b30c4c4b8f61745 | 16:09 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 16:09 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/6ecb76625ef700e4a1ce692b769827d5a0313d6e | 16:17 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 16:18 |
| ssam2 | took 29 minutes to checkout gnome-font-viewer.bst in the end! that can only be a bug | 16:50 |
| ssam2 | 3.1G in the checkout dir | 16:50 |
| ssam2 | ok, update-mime-database is slow because fsync | 16:58 |
| ssam2 | if I set PKGSYSTEM_ENABLE_FSYNC=0 in the environment it goes from 30 secs in my chroot to less than 1 second | 16:58 |
| tlater | Does the order in which we track elements matter? | 17:00 |
| tlater | Hmm, it probably does considering things like base-system take a long time to track. | 17:02 |
| ssam2 | well, you could have a small base and then a huge application on top, either | 17:05 |
| ssam2 | if we track from the base dependency upwards, we can start building sooner | 17:05 |
| ssam2 | in general | 17:06 |
| * tlater asks because he considered using a set() for track dependencies for a minute | 17:08 | |
| * tlater will just have to do without the neat set operations | 17:09 | |
| ssam2 | probably better to stick with list; unless it's cheap to resort again after | 17:10 |
| tlater | I don't think it's very easy to resort after, unfortunately | 17:11 |
| ssam2 | update-mime-database still slow when I disable fsync as part of the buildstream build :-( | 17:14 |
| tristan | tlater, the order does matter yes | 17:19 |
| tristan | tlater, you want the first tracked element to be the first one that is ready to build; you cannot know how long it will take to track the "base system" or whatever is at the bottom | 17:20 |
| tristan | but you certainly want to unblock other tasks as soon as possible | 17:21 |
| tristan | tlater, in any case; there is only one order | 17:21 |
| tristan | when track & build are combined, it's one scheduler | 17:21 |
| tristan | and build order should be used | 17:21 |
| tlater | As in, feed the tracking "pipeline" through plan? | 17:22 |
| tlater | So that we get the elements depth-sorted? | 17:22 |
| tristan | there is no tracking pipeline vs build pipeline, it's one run through the scheduler | 17:23 |
| tristan | and should be depth sorted yeah | 17:23 |
| tlater | tristan: Yes, but we need to split it regardless, because some elements won't be 'tracked', only pulled/fetched/built - I'm doing it in one run of the scheduler now though ;) | 17:23 |
| tristan | tlater, previously I suppose this was less complicated; we simply assume --all and then sort it | 17:23 |
| tlater | I just need to make sure things are done in the correct order now. | 17:24 |
| tristan | Aha | 17:24 |
| tristan | tlater, so what changes here is that something needs to inform the TrackQueue whether to skip an element that is not selected for tracking or not | 17:24 |
| tristan | Or, just place elements that are not selected for tracking on the subsequent queue I suppose | 17:25 |
| tlater | Yeah, I went with the latter | 17:25 |
| tristan | yeah; that *seems* better | 17:26 |
| tristan | anyway I wouldnt whine about that detail | 17:26 |
| tlater | Hm, if we mandate tracked elements are part of the build pipeline it is very easy to sort. | 17:26 |
| tristan | it does strike me though, that if we ever place a queue in *advance* of the TrackQueue, this will get more complex, while a direct "feed all elements through" approach would be simpler | 17:26 |
| tlater | I suppose we can always cross that bridge if we get there? It doesn't seem too likely that we'll get another queue before that, does it? | 17:27 |
| tristan | yeah, I guess pull queue can only come after track queue | 17:27 |
| tristan | anyway | 17:28 |
| tristan | because we need to track before we can know the cache key of what we want to check is pullable or not | 17:28 |
| tristan | (with non strict in play, and automated builders, `bst build --track` *will* produce a lot of cache keys that can be pulled and dont need building) | 17:29 |
| tlater | Those don't have to be tracked though, do they? Would a warning if we aren't tracking anything suffice? | 17:30 |
| tlater | `bst build --track` doesn't feel like it should be used to track anything not related to the actual build. | 17:31 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/ec55d8d7af943336516e738a6f1c1fd5a0b44758 | 17:32 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 17:32 |
| *** valentind has joined #buildstream | 17:40 | |
| *** jude has quit IRC | 17:46 | |
| gitlab-br-bot | buildstream: issue #159 (""BuildStream on your host" docs not accurate") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/159 | 17:47 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/d65bb3c91e0c655cb830f80084517a0b7366c408 | 18:05 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:06 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:08 |
| *** jonathanmaw has quit IRC | 18:09 | |
| *** ssam2 has quit IRC | 18:21 | |
| tristan | tlater, was on a phone call... | 18:25 |
| tristan | tlater, what I mean is this... | 18:25 |
| tristan | tlater, if you run `bst build --track` and you get a new revision for a source on an element | 18:25 |
| tristan | tlater, after tracking, the cache key is calculated | 18:25 |
| tristan | tlater, then, that element moves on to the PullQueue | 18:25 |
| tristan | With the new cache key nice and ready... | 18:26 |
| tristan | it's *very* possible that that freshly tracked element *need not be built* | 18:26 |
| tristan | because it can now be pulled instead from an artifact share | 18:26 |
| tristan | (if it already exists) | 18:26 |
| tristan | And the chances of that, are much higher when not in strict mode | 18:26 |
| tlater | Right | 18:26 |
| tlater | Pulling != fetching, sorry, confused them | 18:27 |
| tristan | Anything that is tracked *must also be fetched* | 18:27 |
| tristan | well, unless it can be pulled | 18:27 |
| tristan | the fact that currently tracking fetches as a side effect is an implementation detail of most/all Source implementations | 18:28 |
| tristan | but that should eventually be improved | 18:28 |
| tristan | tlater, the rationale behind this is that: It *should* be very cheap to ask remote linux.git "what is the latest commit sha for master" | 18:28 |
| *** bethw has quit IRC | 18:29 | |
| tristan | and after that in fetch(), it should be much cheaper to download *only the tree needed for that commit* than to fetch the entire history of linux.git | 18:29 |
| tristan | tlater, currently it's implemented cheap and easy, but the design allows for this to be optimized | 18:29 |
| * tlater understands, yeah, it is a bit stupid that we need to download the entirety of base-system - although for that particular case I wonder if there even is a better solution | 18:29 | |
| tristan | for ostree sources I believe it's already more optimized than this | 18:30 |
| tristan | we really only download the refs we want; I *think* | 18:30 |
| tristan | we just happen to be asking for larger payloads with ostree | 18:30 |
| tlater | Wishful thinking on my end then... | 18:31 |
| tristan | well, at least I know that it's happened to me before that we had a "ref" object present, but not the actual file objects backing it | 18:31 |
| tristan | it probably can be better optimized than it is | 18:31 |
| tristan | but we do have checks in place to ensure the backing object is there when checking ostree source consistency | 18:32 |
| tristan | has_ref() or such | 18:32 |
| tlater | There's one more detail I want to quiz you on, but /me will leave you alone for tonight, it must be like 4 am over there at this point | 18:32 |
| tristan | pulling a base system anyway, is akin to installing an entire OS | 18:32 |
| tristan | it's about 3:30am yeah :) | 18:33 |
| *** cs_shadow has joined #buildstream | 19:02 | |
| *** jude has joined #buildstream | 19:12 | |
| *** jude has quit IRC | 19:21 | |
| *** bochecha has quit IRC | 20:52 | |
| *** bochecha has joined #buildstream | 20:52 | |
| *** bochecha has quit IRC | 20:53 | |
| *** bochecha has joined #buildstream | 20:53 | |
| *** bochecha has joined #buildstream | 20:55 | |
| *** bochecha has joined #buildstream | 20:56 | |
| *** bochecha has quit IRC | 20:57 | |
| *** bochecha has joined #buildstream | 20:57 | |
| *** bochecha has joined #buildstream | 20:59 | |
| *** bochecha has joined #buildstream | 20:59 | |
| *** bochecha has joined #buildstream | 21:01 | |
| *** bochecha has joined #buildstream | 21:02 | |
| *** bochecha has joined #buildstream | 21:03 | |
| *** bochecha has joined #buildstream | 21:05 | |
| *** bochecha has joined #buildstream | 21:06 | |
| *** bochecha has joined #buildstream | 21:07 | |
| *** bochecha has joined #buildstream | 21:08 | |
| *** bochecha has quit IRC | 21:09 | |
| *** bochecha has joined #buildstream | 21:11 | |
| *** bochecha has joined #buildstream | 21:12 | |
| *** bochecha has joined #buildstream | 21:12 | |
| *** bochecha has quit IRC | 21:13 | |
| *** bochecha has joined #buildstream | 21:13 | |
| *** bochecha has joined #buildstream | 21:15 | |
| *** bochecha has quit IRC | 21:15 | |
| *** bochecha has joined #buildstream | 21:16 | |
| *** bochecha has joined #buildstream | 21:17 | |
| *** bochecha has joined #buildstream | 21:18 | |
| *** bochecha has quit IRC | 21:19 | |
| *** cs_shadow has quit IRC | 22:10 | |
| *** tristan has quit IRC | 22:44 | |
| *** mcatanzaro has quit IRC | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!