| *** aminbegood has joined #buildstream | 03:29 | |
| *** aminbegood has quit IRC | 03:38 | |
| *** tristan has quit IRC | 03:40 | |
| *** aminbegood has joined #buildstream | 03:50 | |
| *** aminbegood has quit IRC | 03:55 | |
| *** mohan43u has joined #buildstream | 04:58 | |
| *** mohan43u has quit IRC | 05:24 | |
| *** tristan has joined #buildstream | 06:04 | |
| *** ChanServ sets mode: +o tristan | 06:04 | |
| *** benschubert has joined #buildstream | 07:43 | |
| *** thinkl33t has joined #buildstream | 07:52 | |
| *** cphang has joined #buildstream | 07:55 | |
| *** phildawson has joined #buildstream | 07:56 | |
| tristan | hi benschubert, any thoughts on !1937 ? | 08:08 |
|---|---|---|
| benschubert | tristan: hey, currently looking at 1935. do you want 1937 first? | 08:09 |
| *** jude has joined #buildstream | 08:11 | |
| tristan | benschubert, nah, 1937 is a smaller one | 08:20 |
| benschubert | ok! I'll look at it afterwards | 08:20 |
| tristan | landing 1935 will be good to get out of the way :) | 08:21 |
| *** tpollard has joined #buildstream | 08:38 | |
| benschubert | tristan: for 1937, this seems like a breaking change from BST1 no? Would be great to always get those through the ML I think, at least for record tracking. Other than that I am not 100% sure, the 'target' to me was the thing you were pointing to, which was clear. 'Link' speaks less to me, but I can do with it too | 08:38 |
| *** santi has joined #buildstream | 08:38 | |
| tristan | benschubert, :-S | 08:42 |
| tristan | Any other suggestions ? I really liked "inherit", that was clearest to me | 08:43 |
| tristan | Target is something that you smash with whatever you're throwing at it (i.e. replacing the target junction configuration with this one) | 08:43 |
| benschubert | That's one way of seeing target. For me it's more the think you point to to answer. | 08:44 |
| benschubert | I like `inherit` too actually, or `use` maybe if we want something shorte | 08:44 |
| tristan | I'm not picky about length, I do find target confusing | 08:44 |
| tristan | benschubert, I'm in the middle of yet-another-list-post, reviving old mummies to walk among us again | 08:45 |
| benschubert | yeah I think `inherit` would be better then. Might want to ping cs-shadow once he's in though :) | 08:45 |
| benschubert | ouch, which one? | 08:45 |
| tristan | artifact cache configurations | 08:45 |
| tristan | benschubert, see: https://irclogs.baserock.org/buildstream/%23buildstream.2020-05-25.log.html#t2020-05-25T07:58:23 | 08:46 |
| tristan | Anyway | 08:46 |
| tristan | Don't want to distract with that | 08:46 |
| tristan | I was gonna say, I could raise the issue of 'inherit' for 1937 on the list after finishing this mail | 08:46 |
| tristan | it'll be easier to write | 08:46 |
| * tristan tries to catch cs-shadow on list or issue comments since he doesn't seem to be around IRC much these days | 08:47 | |
| tristan | maybe focused on doing things :) | 08:47 |
| benschubert | tristan: I'll ping him but yesterday was a day off here :) | 08:49 |
| *** phildawson_ has joined #buildstream | 09:07 | |
| *** phildawson has quit IRC | 09:07 | |
| WSalmon | benschubert, give me a ping if i can help with https://gitlab.com/BuildStream/buildstream/-/issues/1310 its a pretty massive blocker for anyone trying to work with bst-master, its even hurting my ability to evaluate other MR's | 10:35 |
| benschubert | WSalmon: how can I reproduce? take bst master and build which element? I don't want to have to build everything | 10:42 |
| benschubert | if you have a link to the failed build, please add it also | 10:43 |
| WSalmon | benschubert, components/libffi.bst | 10:43 |
| WSalmon | bst-master + bst-plugins-experimental-master, seem to brake all the FD branches like this, i have eg willsalmon/bst2-buildorder which i linked in the ticket | 10:49 |
| tristan | benschubert, fwiw I've revived the mummies | 10:57 |
| tristan | It's a real can of worms | 10:57 |
| tristan | and thanks for the review ! | 10:58 |
| tristan | I'll be needing more thoughts on "allowing duplicate junctions" (https://mail.gnome.org/archives/buildstream-list/2020-May/msg00017.html), I think we've got a collection of ideas and we'll need to choose a winner | 10:59 |
| *** sudip has joined #buildstream | 10:59 | |
| tristan | I like "internal junctions", "whitelisting conflicting junctions" (would need both with that approach), or the new "allow renaming projects" | 11:00 |
| tpreston | tristan: regarding your reply to WSalmon's email here, I think you are interpreting the problem correctly https://mail.gnome.org/archives/buildstream-list/2020-May/msg00016.html | 11:00 |
| tristan | tpreston, Great :D | 11:01 |
| tristan | It's actually hard to know :) | 11:01 |
| tpreston | but I'm still trying to parse your answer (and the links you provided) - I'm new to BuildStream! | 11:01 |
| tristan | Ahhhh yes, there is a lot of context packed in there | 11:01 |
| tpreston | I'm going to outline a concrete example and then try to see if I can find an answer in it all :p | 11:02 |
| tristan | tpreston, Basically I'm proposing 2 things, one of them was proposed a long time ago and was fairly popular | 11:02 |
| tristan | tpreston, can I try to communicate it to you here actually ? | 11:03 |
| tpreston | go ahead | 11:03 |
| tristan | Ok so, let me first try to explain how we arrived at the idea of a third plugin type | 11:04 |
| tristan | tpreston, as you probably understand, BuildStream's data model is a build graph composed of Elements related by dependency (which in turn "own" Sources) | 11:04 |
| tristan | This is intended to be very simple, so that elements just do permutations of filesystem data | 11:05 |
| tristan | Now it happened on a few occasions, that we saw proposals adding more and more features to the Element class | 11:05 |
| tristan | Notably the testing proposal | 11:05 |
| tristan | tpreston, This was kind of counter design, adding more and more functionality to Element fails to leverage fine grained elements, making everything exponentially more complex | 11:06 |
| tristan | Basically, what I think was driving this was the fact that people think of Elements as if they were .bst files | 11:07 |
| tristan | And people want convenience | 11:07 |
| tpreston | just to clarify, what do you mean an Element "owns" a Source? | 11:07 |
| tristan | tpreston, I mean that the lifecycle of a Source object is tied to an Element basically, the Element creates them | 11:07 |
| tristan | more than that, the Element does not share it with anyone, Elements cannot *refer* to Sources belonging to another Element | 11:08 |
| tpreston | ok | 11:09 |
| tristan | tpreston, Ok so... that's not really the most relevant part of this; the more relevant thing is that people want to achieve a lot with a little (of course) | 11:09 |
| tristan | So approaches to a problem which involve "Lets create a simple element and encourage this configuration of elements" were not very popular | 11:09 |
| tristan | Even though... thats sort of what BuildStream is meant to do | 11:10 |
| tristan | Give you a lot of simple elements which do little things, and allow you to express anything with that | 11:10 |
| tristan | tpreston, it became pretty clear I think, we needed a way for people to express multiple elements, more conveniently, in YAML | 11:10 |
| tristan | So I had a first basic idea which is linked in that email... but juergbi had a better idea, which was to invent a third kind of plugin which would allow customization of how groups of elements were expressed | 11:11 |
| WSalmon | <tristan> more than that, the Element does not share it with anyone, Elements cannot *refer* to Sources belonging to another Element <- but the source dose not know which element asked for it and as far as i can tell the bst can see when a uni sorce is used twice and dose not need to invoke it if its used in mulitple places | 11:11 |
| tristan | WSalmon, that second part is a side effect of how caching works as I understand | 11:12 |
| tristan | Bug number 5 https://gitlab.com/BuildStream/buildstream/-/issues/5 | 11:13 |
| tpreston | tristan: I can't visualise this 'So approaches to a problem which involve "Lets create a simple element and encourage this configuration of elements" were not very popular' | 11:13 |
| tristan | heh, still open, but not hugely bad | 11:13 |
| tpreston | Have you got examples of simple elements and that kind of configuration? | 11:13 |
| tristan | tpreston, ok so, Chandan wanted to implement "testing of elements" | 11:13 |
| tristan | tpreston, for one example (there were others I can recall, not specifically right now) | 11:14 |
| tristan | tpreston, So the thing about testing, like running `make check` for instance... | 11:14 |
| tristan | tpreston, is that I think ultimately, you don't want to block reverse dependency builds on the completion of `make check` | 11:14 |
| tristan | That's quite suboptimal | 11:14 |
| tristan | Instead you want to model your data so that you have another element which tests while reverse dependencies of the element currently under test, are also building | 11:15 |
| tristan | tpreston, right now there are still some issues blocking such an approach from happening, notably sharing of build trees needs to be something a testing element can depend on | 11:16 |
| tristan | tpreston, But as you can see, that would require writing multiple .bst files | 11:16 |
| tristan | and people like the idea of "One module one bst file", especially if you have a larger project | 11:16 |
| tpreston | hang on, what's a "reverse dependency build" and why would it block on `make check` - wouldn't the element just build, get cached and then it can be used? | 11:16 |
| tristan | tpreston, say you build glib and then build GTK+, but you want to run `make check` for glib | 11:17 |
| tristan | GTK+ is a reverse dependency build of glib (because it depends on glib) | 11:17 |
| tristan | tpreston, Now today, if you simply ran `make check` in your autotools or meson element at the end, GTK+ could not start building until glib was exhaustively tested | 11:18 |
| tpreston | ok, so you're saying you want to run `make check` after the build - sometime later? | 11:18 |
| tpreston | ok got that | 11:18 |
| tristan | right, what a waste of time :) | 11:18 |
| tristan | hehe | 11:18 |
| tristan | But you still want something to signify that glib was validated | 11:18 |
| tristan | it need not be the presence of the built artifact, though | 11:18 |
| tpreston | ... hang on | 11:18 |
| tpreston | what if the tests fail | 11:18 |
| tristan | then you have a built glib artifact | 11:19 |
| tristan | and failed test artifact for that same build | 11:19 |
| tristan | At some point of your process, you'll want to stop the train because of that failure of course | 11:19 |
| tpreston | And GTK+ is half-way through building? On a failing glib? | 11:19 |
| tristan | Sure | 11:19 |
| tpreston | Ok so it's like a pipeline | 11:19 |
| tpreston | pre-emptive pipeline I mean | 11:19 |
| tristan | that is exactly what we call it yes :) | 11:19 |
| tpreston | :p | 11:19 |
| tristan | I was trying to avoid introducing another term :) | 11:20 |
| tpreston | haha np, thanks for taking it slow | 11:20 |
| tristan | So, with the idea of a plugin which allows the expression of multiple elements with a single .bst file, we could do a lot | 11:20 |
| tristan | We could have special purpose plugins like this which users can write | 11:20 |
| tristan | Which define implied dependency relationships between the elements declared | 11:21 |
| tristan | and parse their own YAML | 11:21 |
| tristan | tpreston, like little programs of bst elements which build on other existing element plugins | 11:22 |
| tristan | tpreston, with this approach, you might have something like a plugin which lets you define a bunch of dependencies to deploy, some parameters specifying options in what the image should be | 11:22 |
| tristan | tpreston, and it could internally expand to a cluster of elements which includes a compose element and a filesystem image creating element | 11:23 |
| tristan | tpreston, following me so far ? | 11:23 |
| tpreston | I think so. To reword: the idea is to create a more complex plugin type, which allows expressing dependencies. | 11:24 |
| tpreston | Can this not be achieved with the current plugins | 11:24 |
| tpreston | ? | 11:24 |
| tpreston | Or includes? | 11:25 |
| tpollard | I think that's the description of the current deployment script elements | 11:25 |
| tristan | tpreston, I hope it would not be *that* more complex, but yeah basically, it would be a plugin type which "assembles element clusters" | 11:25 |
| tristan | It has the responsibility of parsing a format and integrating with the loader | 11:25 |
| tristan | in such a way that a hand full of elements can be expressed with a single file, with a special purpose | 11:25 |
| tpreston | tristan: parsing a YAML format and intergrating with the bst loader, you mean? | 11:26 |
| tristan | tpollard, right I'm using that as an example; not sure *which* deployment scripts you mean but it's generally the process we use in fdsdk | 11:26 |
| tpollard | formalising script elements into plugins is something I'd prefer for this | 11:27 |
| tristan | tpreston, Ummm, I think I lost the trail here... | 11:27 |
| tristan | tpollard, Right but, what if you also wanted to include the compose step | 11:28 |
| tpreston | tristan: sorry I don't know what you mean by format and loader in this statement "It has the responsibility of parsing a format and integrating with the loader" | 11:29 |
| tristan | tpollard, it's all basically the same domain, wouldn't it be cool if you had one bst file which (A) Implicitly depended on the tools it needs for deployment... (B) Indicated which dependencies needed to be deployed, including split rules, and (C) Expressed things like partition sizes etc | 11:29 |
| * tristan is now juggling two conversations with two staggeringly similar nicks ;-) | 11:29 | |
| tristan | tpreston, sorry I think I'm referring to something too internal | 11:30 |
| benschubert | Or in another direction: having a single bst file that gives you: a library that oyu can consume in BuildStream, a 'deb/rpm' file for it, and a test target to make sure it works, without having to have 3-4 elements there :) | 11:30 |
| tpollard | tristan: yes it would | 11:31 |
| tristan | tpreston, Basically I mean it's a plugin, so it's just a python file... it has Plugin.configure() and that is given a YAML node, just like other plugins | 11:31 |
| tristan | tpreston, Then... it would "do something" which would result in creating elements, which might or might not be related to eachother implicitly by dependency | 11:32 |
| tristan | tpreston, And the output of that plugin would be the instantiation of these elements basically | 11:32 |
| tristan | The BuildStream core would need to define an API contract for correctly interacting with this thing | 11:32 |
| tristan | i.e. being able to depend on it's elements | 11:32 |
| tristan | or some of them in some way | 11:33 |
| tristan | Right, such a thing would allow modularity and reusability of constructs of multiple elements | 11:33 |
| tristan | which would satisfy a wide (unlimited ?) variety of use cases | 11:33 |
| tristan | tpollard, of course my email brings in *another* concept along with the older (hmmm, maybe "composite plugin" I could call that idea ?) | 11:35 |
| tristan | err, tpreston, or both | 11:35 |
| tpreston | :p | 11:35 |
| tristan | Which is the idea that a plugin could be "bound to a project", such that it had an intimate relation with that project | 11:35 |
| tristan | And could implicitly depend on the tooling used to deploy an image, with the certainty that that element exists, and provides what is needed by the scripts used | 11:36 |
| tpreston | tristan: I know you're thinking abstract, but I'm trying to think how this solves our problem of sharing element construction between similar projects. Is it almost like an element factory? | 11:39 |
| tpreston | From my limited perspective, an example problem we have is "There is one way to create an sdcard.img for a Raspberry Pi" and we want to allow projects to just grab the sdcard-image.bst element off-the-shelf, or customise it to some degree (different rootfs-size, format) | 11:41 |
| tpreston | So your proposing some kind of plugin which "generates" this kind of element, for your project? | 11:41 |
| tpreston | I have a prototype which does this using includes: | 11:43 |
| tpreston | general raspberry pi image: https://gitlab.com/celduin/bsps/bst-boardsupport/-/blob/master/include/board/raspberrypi/image.bst | 11:43 |
| tpreston | specific raspberry pi image (for a sample project) https://gitlab.com/celduin/bsps/bst-boardsupport/-/blob/master/sample/rpi4-features/elements/deploy/image.bst | 11:44 |
| tpreston | I guess I don't grok why this is "wrong" yet | 11:45 |
| tristan | templating is another approach which I haven't given much thought to, I feel like that road should have some way to give parameters to the (@) include semantic | 11:46 |
| tpreston | I've done that in the form of YAML variable overrides | 11:46 |
| tristan | tpreston, i.e. as WSalmon already pointed out on the list, you are going to need to presume how this project decided to junction you (i.e. junction name) | 11:47 |
| tristan | Also it only gets you as far as one element at a time | 11:47 |
| tristan | tpreston, I mean, includes were not really made with templating/macros in mind, and I do feel that this road will lead to things which don't report errors super easy to understand by users | 11:48 |
| *** hasebastian has joined #buildstream | 11:49 | |
| tristan | I still find it surprising that the loader was changed to process some of the directives in advance of parsing the dependency dictionaries | 11:50 |
| tristan | I wonder what this implies, I think that an element cannot really rely on it's YAML until it's been fully composited | 11:51 |
| tristan | It may have opened up a whole class of errors we're not seeing yet | 11:52 |
| tpreston | tristan: What *is* the way to express templating? Plugins I guess. | 11:54 |
| tristan | Essentially yes, those are the things being reused | 11:55 |
| tristan | Currently only at a one-element-at-a-time basis | 11:55 |
| tristan | tpreston, this might be interesting: https://docs.buildstream.build/master/arch_data_model.html#element-composition | 11:56 |
| tristan | tpreston, note that the files like this: https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/plugins/elements/autotools.yaml are basically what is "Element defaults" in that picture | 11:57 |
| tristan | And that those default yaml files for elements are not specific to BuildElements | 11:57 |
| tristan | Any element can provide such a "template", which undergoes the regular process of composition | 11:58 |
| tristan | With the element declaration (the .bst file) having the final say | 11:58 |
| WSalmon | yes, plugins are the best way to reuse understanding and reduce the amount that needs to be in a element but the question i am trying to ask is when thats not good enough, when you need to reuse more than what a plugin allows for then includes allow you to share/reuse more than what a plugin allows for | 11:58 |
| tristan | the https://gitlab.com/BuildStream/bst-plugins-experimental/-/blob/master/src/bst_plugins_experimental/elements/x86image.yaml implementation for example, did not have a lot of work go into it | 11:59 |
| tristan | it's not immensely reusable or smart and could be improved a great deal I think | 11:59 |
| tristan | WSalmon, right, and you are also pointing out some of the drawbacks of that, and I can see that "Right now" it may be something you can do | 12:00 |
| tristan | WSalmon, it's not really very bullet proof, so I'm suggesting revisiting the idea of composite plugins which could get you much further | 12:01 |
| tpreston | tristan: how do you envision the composite plugin improving, say x86image.yaml? | 12:03 |
| tristan | tpreston, in the way I described today (possibly too vaguely) :) | 12:04 |
| tristan | tpreston, I mean if you look at a project that uses x86image, you will see that (A) You need to build the tools which x86image requires in the / sysroot... (B) You need to write your compose element separately to decide what your main filesystem will be composed of... (C) You need to roll your initramfs separately with other elements, (D) You need to define the dependency relations of all of these | 12:05 |
| tristan | tpreston, but instead, you could have an element which (A) Already knows the element which provides the tools x86image needs (granted the additional "bound to a project" feature) ... (B) Internally uses a compose element... (C) Internally defines how to deploy your provided /init script or such for the initramfs... (D) Implicitly ties these elements together with dependency relationships | 12:07 |
| tristan | tpreston, So you could have one bst file which is read by a "composite plugin" (look it's already got a name ;-)) with some sane defaults, give it some parameters and declare that whole construct in one go | 12:08 |
| tpreston | (sorry just in a meeting, brb) | 12:13 |
| * tristan has to split at this point... | 12:26 | |
| *** tristan has quit IRC | 12:29 | |
| *** toscalix has joined #buildstream | 12:38 | |
| benschubert | WSalmon: can you point me to where `infodir` is defined? I could not find it. I think it is a bug where we are not reporting an unknown variable correctly | 12:51 |
| benschubert | WSalmon: ah nevermind found it it's in the projects' default in buildstream | 13:02 |
| benschubert | Any reasons to not break the cache keys with https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/buildelement.py#L148 ? Seems like an easy cleanup if not | 13:38 |
| *** santi has quit IRC | 13:41 | |
| *** santi has joined #buildstream | 13:41 | |
| benschubert | WSalmon: ok the problem is wiht 'integration-commands' which are part of the 'public data' and not part of the element config, this part is apparently _not_ expanded. I'll have to write a test for this and fix it | 13:50 |
| benschubert | I wonder if htere are other places where this is not done now :/ | 13:50 |
| *** tristan has joined #buildstream | 13:51 | |
| benschubert | tristan: > Any reasons to not break the cache keys with https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/buildelement.py#L148 ? Seems like an easy cleanup if not | 14:05 |
| WSalmon | benschubert, thanks for looking in to this, i have been in meetings a lot of this after noon | 14:05 |
| *** ChanServ sets mode: +o tristan | 14:06 | |
| tristan | benschubert, juergbi asked me the same thing the other week I believe | 14:06 |
| tristan | or maybe it's the second time ? | 14:06 |
| tristan | anyway yeah, lets clean that up | 14:06 |
| juergbi | yes, that was me | 14:06 |
| juergbi | when we do that we should make sure to change get_unique_key() such that it skips unused command lists | 14:07 |
| juergbi | that way if we add new command groups in the future we won't break anything | 14:07 |
| benschubert | juergbi: yep seems good. Thanks both! | 14:07 |
| *** hasebastian has quit IRC | 14:09 | |
| douglaswinship | newb question: filter elements and compose elements are the only kinds of plugin that respond to domains and split rules, right? (In terms of the basic plugin types, i mean). I know that manual elements can _apply_ split rules to their own artifacts, but I'm talking about recognizing and responding in some way based on the domains of the files in their dependencies. So for instance, I can't | 14:28 |
| douglaswinship | write a script element, and then tell it to perform a certain operation on all the files in a particular domain? | 14:28 |
| tristan | douglaswinship, you can write a script element with the tools you need staged at /, and stage the output of a compose element into the %{install-root}, do your thing on those files and be done | 14:30 |
| douglaswinship | Right, but i do need to go via a compose element first? | 14:31 |
| douglaswinship | That's what i was double-checking. | 14:32 |
| tristan | Theres a bunch of ways you know, and it of course depends on what you want to do | 14:32 |
| tristan | However, it's recommended that you use tools in the sandbox and not modify content directly using python in Element.assemble() | 14:33 |
| tristan | kind of breaks determinism | 14:33 |
| tristan | note that split rules are a first class citizen, they are inherently understood by Element.stage_dependencies() | 14:33 |
| tristan | So any element you write can potentially stage pre-split artifacts | 14:34 |
| douglaswinship | Tbh, the details are going over my head here. I'm still not at all familiar with internals like Element.assemble() and Element.stage_dependencies(). But I think you're confirming what I was trying to ask. The basic plugins like script and manual elements, can't discriminate the files in their dependencies based on domains (at least not directly). But, assuming I have an appropriate use case, I | 14:42 |
| douglaswinship | can write a new plugin that does. | 14:42 |
| tristan | pretty much | 14:43 |
| coldtom | that's the gist douglaswinship | 14:43 |
| tristan | douglaswinship, we try to provide stock elements that are simple, such that you can combine them and achieve mostly whatever you need | 14:43 |
| tristan | With BuildStream 2 and the way CAS and staging works with buildbox, I am under the impression that the overhead of going through compose | script is fairly low | 14:44 |
| tristan | I think real paths are not manipulated on disk with compose except for when running integration commands with bst2... if I'm not mistaken | 14:45 |
| tristan | just the virtual directory API is mostly used to manipulate paths and define the resulting artifact | 14:45 |
| douglaswinship | tristan: yup, it's not a big overhead at all. I was just talking with someone who didn't (initially) realise that you needed even that small overhead, and it made me want to double check i'd understood it correctly. | 14:46 |
| tristan | with bst1 it is a bit heavy | 14:46 |
| tristan | oh I mean overhead in terms of processing and I/O | 14:46 |
| WSalmon | tristan, we have a element, bob.bst, and then we need some domains in alpha.bst and some in beta.bst, douglaswinship is asking if, without a custom plugin, you can avoid having, bob-a.bst and bob-b.bst were bob-*.bst is a compose with filter element, the cost in turms of build is low but there is a cost of humans maintaining elements | 14:46 |
| tristan | I think in bst1, you need to stage the whole thing | 14:46 |
| tristan | and it really goes into a directory | 14:46 |
| tristan | WSalmon, sounds like yet another interesting usage for a hypothetical composite kind of plugin, simple constructs like this could let people build their own compose-and-script thingies :) | 14:47 |
| douglaswinship | coldtom: thanks for the one-line answer btw. That was exactly what I was looking for. :D | 14:50 |
| WSalmon | a script element can not be used here as you cant AFAIK filer on the output of our existing elements not the input | 14:53 |
| WSalmon | * for our existing plugins | 14:54 |
| tpreston | tristan: | 15:20 |
| tpreston | whoops | 15:20 |
| tpreston | one second | 15:21 |
| tpreston | tristan: sorry for the delay, I got caught up in meetings and lunch. I sort of understand the need for a composite plugin but I don't really understand problems with the first list you described about x86image. As you said just above the point is to have many small and simple elements we can compose larger ones from - so a composite element kinds of goes against that. | 15:24 |
| tpreston | WSalmon: maybe you have a clearer view of the problem that this solves? You've spent more time with this than me | 15:24 |
| tpreston | The problem I'm seeing at the minute is sharing A. common element behaviour (eg. building linux), B. common element data (raspberry pi linux kernel all have the same sources) and C. specific element data (raspberry pi 3 model b uses bcm2711_defconfig, some device tree overlay, $foo firmware and $bar config.txt) | 15:28 |
| tpreston | I think the right solution for A is a new plugin (a "linux" plugin) | 15:29 |
| tpreston | Maybe includes *are* the right solution for B? | 15:30 |
| WSalmon | tpreston, i think tristan is trying to say that plugins are the best way to share element "code/internal" and i agree with that, I think tristan is also saying that long turm we can make bst better at that, and i agree there too. but were i have tried to say that for now we can take advantage of includes to solve some other issues, tristan has said that thats not how he envisaged it working. but he is not saying oh this should never work, more | 15:32 |
| WSalmon | that its quite fragile. i may be miss reading the situation | 15:32 |
| WSalmon | i think that for our use case we may need to use includes to achive what we want for now while also reducing our use as upstream gives use more syntactic suger along the way. | 15:33 |
| WSalmon | i dont like to put works in some ones mouth but some times trying to rephrase what you think some one said can highlight differences of understanding | 15:36 |
| tpreston | My take on it is that "plugins are the right way to share /behaviour/" and if the current plugins don't support your complex use-case, then we need more plugins! :D | 15:37 |
| tpreston | But I don't really grok the nature of that complex use-case | 15:37 |
| WSalmon | that gose some of the way, linux for instance falls in that camp but bst puts some constraints on plugins that mean we can only go so far | 15:38 |
| tpreston | What I grok is: libreML (for example), want to add support for Raspberry Pi, and it needs someway to consume - from another BuildStream project - /behaviour/ for that target platform (linux plugin, u boot plugin, sdcard-image plugin?) and /data/ (Raspberry Pi 3 defconfig, etc) | 15:39 |
| WSalmon | a element-template/plugin can not know what project it came from with public API and this can make dependencies a bit tricky | 15:39 |
| WSalmon | also plugins cant say please depend on these and then let my dependencies be expanded after that | 15:40 |
| tpreston | Specifically how it makes dependencies tricky is what I'm trying to understand | 15:41 |
| WSalmon | we agree these features would help reduce the amount that needed templating, but i would still argue that including would be useful for other projects to use elements | 15:42 |
| WSalmon | we agree these features would help reduce the amount that needed templating, but i would still argue that including would be useful for other projects to use "reuse-elements" <- i need a better way to express this | 15:42 |
| WSalmon | * minimise the amount that a upstream project needs to recreate in its own project to use script like elements | 15:43 |
| tpreston | Just "use"? As I understand it, junctions just increase the "bag-o-elements" you have access too. And it's really about constructing an element pipeline from that big ol' bag | 15:43 |
| *** toscalix has quit IRC | 15:43 | |
| tpreston | So increasing "re-usability" is about creating generic "behaviour" (plugins) and consumable data (bst files) | 15:45 |
| WSalmon | tpreston, right but the issue is when the artifact of those elements isnt quite right and you need a new artifact that's instructions are almost identiacal, how do we share the instruction or "element" | 15:45 |
| tpollard | fork or patch the element definition | 15:46 |
| * tpollard runs | 15:46 | |
| tpreston | You make a plugin | 15:46 |
| tpreston | eg. how do we share almost identical autotools instructions? | 15:46 |
| WSalmon | you cant say i want to use those exact instructions "element" but with a slightly diffrent dependency | 15:46 |
| WSalmon | thats what a include lets you do | 15:47 |
| tpreston | So for autotools, the behaviour is specified in the plugin, then we change the dependency list for the exact implementation for this element | 15:47 |
| WSalmon | i want to include the instructions but i need to supply a extra dependency | 15:47 |
| tpreston | (I'm speculating, someone please correct me if I'm wrong) | 15:47 |
| tpreston | WSalmon: Hm, I'm trying to think of an example for your dependency case | 15:49 |
| WSalmon | tpreston, but the bst model so far has been to have a auto tools project and then 100 elements that use that, we could have 100 plugins and then 100 elements that have a 1:1 relation ship, but as plugins cant inherit from each other the "core" auto tools bits would be repleted in every single plugin | 15:49 |
| WSalmon | by saying that this can all be solve with plugins i think tristan is just moving the maintenance task rather than reducing it. | 15:52 |
| benschubert | tpollard: I don't think your answer is so bad. If you have a coherent project that is given to you but you don't like it, adding patches is often an easy way to tweak it, and widely used | 15:52 |
| WSalmon | benschubert, right, but as some one how has had to maintain such patches over the long term i have had my fingers burnt. the also dont solve the issue of script like elements that you want to depend on element in your project | 15:53 |
| WSalmon | Script elements can be used to take existing elements and tweak them, the act of tweaking may want to be defined once and shared via a junction but the element subject to the tweak may come from the project containing the junction. | 15:55 |
| WSalmon | ^ to quote me on the mailing list | 15:55 |
| juergbi | https://gitlab.com/BuildStream/buildstream-docker-images/-/merge_requests/169 is needed for https://gitlab.com/BuildStream/buildstream/-/merge_requests/1918 (field-based node properties), however, the former breaks incremental workspace builds until the latter is merged | 16:01 |
| juergbi | any objections to force merging the former. the pipeline in the latter (already using the new docker images) is green | 16:02 |
| benschubert | juergbi: fine by me | 16:03 |
| juergbi | done | 16:09 |
| *** cs-shadow has joined #buildstream | 16:11 | |
| cs-shadow | juergbi: hi, I didn't manage to write it in time :) but please see https://gitlab.com/BuildStream/buildstream-docker-images/-/merge_requests/169#note_349433056 | 16:14 |
| juergbi | cs-shadow: yep, replied | 16:15 |
| juergbi | as long as we merge !1918 soon enough, there shouldn't be any real issues | 16:15 |
| juergbi | afaict | 16:15 |
| cs-shadow | juergbi: thanks! that makes sense. Just wanted to make sure, seems all good now | 16:15 |
| juergbi | ta | 16:16 |
| *** santi has quit IRC | 17:13 | |
| *** santi has joined #buildstream | 17:13 | |
| tristan | WSalmon, I think your paraphrasing is just about accurate yes. I also think that making BuildStream better at supporting more complex constructs in this way is an already discussed and widely accepted idea (in other words, should someone decide to pick that work up, it would certainly be a welcome contribution) | 17:14 |
| tristan | certainly, if a group of developers were trying to prototype an ideal solution to this sort of thing with buildstream, I'd recommend exploring that avenue | 17:14 |
| *** cs-shadow has quit IRC | 17:15 | |
| tpreston | ok, I think that answers our questions for now | 17:19 |
| tpreston | thanks tristan, really appreciate your input | 17:19 |
| *** cs-shadow has joined #buildstream | 17:20 | |
| *** tpollard has quit IRC | 17:43 | |
| *** santi has quit IRC | 18:07 | |
| *** santi has joined #buildstream | 18:08 | |
| *** santi has quit IRC | 18:11 | |
| *** cs-shadow has quit IRC | 18:41 | |
| *** santi has joined #buildstream | 19:01 | |
| *** cs-shadow has joined #buildstream | 19:52 | |
| *** benschubert has quit IRC | 20:08 | |
| *** jude has quit IRC | 21:12 | |
| *** cs-shadow has quit IRC | 22:01 | |
| *** sudip has quit IRC | 22:56 | |
| *** cphang has quit IRC | 22:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!