| *** jude has joined #buildstream | 07:34 | |
| *** jude has quit IRC | 07:37 | |
| *** tristan has joined #buildstream | 07:51 | |
| *** jude has joined #buildstream | 08:03 | |
| *** tlater has joined #buildstream | 08:14 | |
| *** jonathanmaw has joined #buildstream | 08:47 | |
| *** ssam2 has joined #buildstream | 09:02 | |
| tlater | Should I modify the element API to add workspace support? | 09:09 |
|---|---|---|
| tlater | I can either modify the sources after they've loaded, keeping everything the same, or add another property. | 09:09 |
| jonathanmaw | wat? I replied to the mailing list and it now thinks it was sent from jonathan.maw@codethink.co.uk again | 09:27 |
| jonathanmaw | ah, I was replying to an E-mail adressed to jonathan.maw@codethink.co.uk, so it sent it from the .co.uk address. And since I unsubscribed my .co.uk address, it's having a strop at me. | 09:28 |
| *** tlater has quit IRC | 09:31 | |
| *** tlater has joined #buildstream | 09:32 | |
| tlater | tristan: Should I avoid making workspaces part of the element API? I feel like it would be most clear if it was another property. | 09:39 |
| tristan | tlater, it should certainly not be public API, but the code will probably read better if you delegate through the data model | 09:52 |
| tristan | well, not sure how much of that is actually needed | 09:52 |
| tristan | tlater, at least we decided we are not going to add the functionality of checking out "from the proper upstream" and add a new Source API that everything would have to implement | 09:53 |
| tristan | so I guess for now there is no need ? | 09:53 |
| tristan | tlater, the Element will probably need some changes anyway, though | 09:53 |
| tlater | It would be really convenient not to have to write a separate parser for a few functions though. | 09:54 |
| tristan | separate parser, not sure what you mean | 09:54 |
| tristan | you mean for project state in <project directory>/.bst/ ? | 09:54 |
| tlater | Well, the workspaces have to be loaded somehow | 09:54 |
| tlater | Yeah | 09:54 |
| tristan | I think that needs to be a new file | 09:55 |
| tristan | tlater, That or... it can be handled also by _project.py | 09:55 |
| tristan | but it is clearly separate | 09:55 |
| tristan | tlater, the workspaces is not project data and should not be written into element.bst files at all | 09:55 |
| tristan | it is buildstream state around project data | 09:56 |
| tlater | ok, that makes sense. I suppose functions that need data from there will just have to call a separate load function? | 09:56 |
| tristan | tlater, I think Project.load() should take care of that | 09:57 |
| tristan | in addition to loading project.conf | 09:57 |
| tristan | Or somewhere in Project.__init__() anyway | 09:58 |
| tlater | If this is going to be a loader for all project state, should it create a dict with a "workspaces" key for now? | 09:58 |
| tristan | it's given the project directory | 09:58 |
| tristan | tlater, that sounds sensible, it can have a private project._workspaces dictionary | 09:59 |
| tristan | or whatever data type describes it best | 09:59 |
| tristan | maybe it's a dictionary where the values are sparse arrays of directories ? | 09:59 |
| tlater | You also need the original element to map the directories | 10:00 |
| tlater | So something like | 10:00 |
| tristan | tlater, if it ends up looking too messy (with other things looking into the dict), than having a sensible accessor function is desirable | 10:00 |
| tristan | I am saying sparse arrays because elements can have more than one source | 10:00 |
| tristan | but it could equally be a dict of dicts, where the nested dict is an index -> directory table | 10:01 |
| tlater | You also need the source index | 10:01 |
| tristan | so you could have Project._get_workspace(element, index) which returns an absolute path | 10:02 |
| tlater | Ah, wait, no I understand, yeah, a sparse array is probably best | 10:02 |
| tristan | either works fine yeah | 10:02 |
| tlater | Will we never store any other data? | 10:02 |
| tristan | tlater, we might, but I think it's okay to just use <project directory>/.bst/ as an area for storing state | 10:02 |
| tristan | and then workspaces is in the .bst/workspaces.yaml file | 10:03 |
| tristan | and we can store other stuff in other files in there | 10:03 |
| tristan | note, one brutal way for clearing all workspaces, is to just rm -rf .bst/ | 10:03 |
| tristan | that should always be okay to do | 10:03 |
| tlater | Yeah, I've been debugging like that for lack of a close command ;) | 10:04 |
| tristan | jonathanmaw, ummm, so I now have you subscribed twice | 10:05 |
| tristan | jonathanmaw, and I have two bounces in my inbox | 10:06 |
| tristan | jonathanmaw, and only one in the moderator queue, for .com | 10:06 |
| tristan | dont really understand what's going on :-/ | 10:07 |
| jonathanmaw | I don't understand how that last one is pending review, either :/ | 10:07 |
| tristan | Ummm, I think I know | 10:08 |
| tristan | So I did some emailing with curl a couple years back and learned about email a bit | 10:08 |
| tristan | When I send a message with curl, I need to tell it where it's from | 10:08 |
| tristan | And I need to set the From: message header separately | 10:09 |
| tristan | It can be that these are two separate data, and you are in fact sending from .com, but disguising the message header From: with .co.uk | 10:09 |
| tristan | and that mailman doesnt like it | 10:09 |
| tristan | I will just accept your messages and set a filter and hope for the best | 10:10 |
| tristan | jonathanmaw, I guess there should be some kind of protection like that, it makes sense | 10:12 |
| tristan | jonathanmaw, otherwise I could just send email to any mailing list and munge my From: header with a member's email address | 10:12 |
| * juergbi sees weak cache keys appearing in the artifact cache :) | 11:27 | |
| tristan | nice :) | 11:31 |
| tristan | I think it's going to be a nice feature in combination with `bst track` and flexibly choosing what you update... if there are people who regularly build everything master, with weak build planning mode they can get something much quicker when glib changes | 11:32 |
| tristan | I'm a bit torn on whether or not to share artifacts in weak planning mode, I feel like maybe allow downloading them, but not uploading them ? | 11:33 |
| ssam2 | how enabled-by-default is sharing artifacts going to be in general? | 11:34 |
| tristan | Also there is a distinction that we could easily make, i.e. "Is the strong key of an artifact what it would have been in strong build planning mode anyway ?" | 11:34 |
| ssam2 | if it's limited to people with the right SSH access, it's a fairly unusual thing to be doing in any case | 11:34 |
| tristan | ssam2, you will need to specify a url in order to be able to pull them | 11:34 |
| tristan | ssam2, indeed, and pushing them needs another url | 11:34 |
| ssam2 | so maybe don't *prevent* sharing weak artifacts, but perhaps have it off by default ? | 11:35 |
| tristan | ssam2, I think though it's probably not the right setup for GNOME, or in many cases, to have many peers sharing artifacts, seems more sensible to me to have that done by auto builders | 11:35 |
| juergbi | tristan: i thought the idea was to share artifacts also in weak planning mode | 11:36 |
| ssam2 | yes indeed.. there's nothing to actually enforce that cache keys are what we think they are | 11:36 |
| juergbi | but yes, i expect that to be done by auto builders, not by regular devs | 11:36 |
| ssam2 | jrandom user can push an artifact named 0x12356843 that actually contains some horrible trojan | 11:36 |
| tristan | juergbi, yeah I think so... the thing is when you want to download a weak artifact and a build server is constantly publishing artifacts in fully deterministic build mode, the cache will have your weak artifacts anyway | 11:37 |
| ssam2 | i guess it's possible to mitigate that server-side, but until that code is there, we should just have artifacts pushed by autobuilders that we control | 11:37 |
| tristan | however, it might just take a bit longer to get into the cache | 11:37 |
| juergbi | tristan: not sure what you mean | 11:37 |
| tristan | juergbi, every artifact has both a weak and a strong key | 11:38 |
| juergbi | buildstream will switch to strong artifacts when they are available | 11:38 |
| tristan | juergbi, so if my project and the build server's project is at the same revision... then even if locally I have some workspaces and dont require a strong build mode, the artifact cache still has weak key artifacts which correspond to my sources | 11:38 |
| tristan | so lets say I bst track --deps all... <gnome.bst>... everything... and the build server constantly builds the whole thing | 11:39 |
| tristan | I might have overridden some things with workspaces, but can download the rest via it's weak key | 11:39 |
| tristan | because I'm building in weak planning mode | 11:40 |
| tristan | juergbi, do we understand how it works ? So... if I am building in strong mode... *and* uploading artifacts, whenever I upload an artifact, anyway I'm pushing 2 refs | 11:41 |
| tristan | a strong and a weak one | 11:41 |
| juergbi | yes, actually already implemented (pending testing) | 11:41 |
| tristan | although technically on the server, the data is not duplicated, it's two refs of the same objects | 11:41 |
| juergbi | yes | 11:42 |
| tristan | Right, so when artifacts have been published, via a totally deterministic planning mode (need better vocabulary for build modes)... then the weak key is available for each individual modules exact set of sources, which went into that build | 11:42 |
| tristan | (as well as the strong key) | 11:42 |
| juergbi | yes | 11:43 |
| tristan | So what I meant was that... if we constantly publish only via the deterministic build mode, but we download with weak keys as needed... it's probably around the same as if we allowed uploading the artifacts to a share, which were themselves built in a weak planning mode | 11:43 |
| tristan | not much more benefit, except that *perhaps* artifacts are available a bit sooner | 11:44 |
| tristan | (a bit faster if say, we have a continuous like CI also contributing to the cache, with weakly planned builds) | 11:44 |
| juergbi | it would make a difference for people downloading for deterministic builds | 11:44 |
| tristan | if you are downloading for deterministic builds, then you only care about downloading via the strong key anyway | 11:45 |
| tristan | right ? | 11:45 |
| juergbi | yes but if there was no autobuilder running in deterministic mode, you might not have much luck finding artifacts with the strong key on the remote repo | 11:46 |
| juergbi | depending on frequency of changes of the different components | 11:46 |
| tristan | juergbi, good point that | 11:46 |
| tristan | indeed | 11:46 |
| juergbi | (assuming CI only builds component if component itself has been changed) | 11:46 |
| tristan | however I think that anyway an autobuilder needs to sometimes rebuild in strong mode (perhaps just less frequently) | 11:47 |
| tristan | otherwise it doesnt have a great chance of correcting itself in the case of false positive failure reports | 11:47 |
| juergbi | yes, probably makes sense | 11:48 |
| tristan | jonathanmaw, I just replied your email... for the sake of being thorough; I made some comments which are probably unrealistic to implement in the first iteration, but we should be taking into consideration | 12:23 |
| tristan | jonathanmaw, dont worry too much though, I think it's important to consider the idealistic case, the realistic case, and the plausible migration path from short term realistic to eventual idealistic case | 12:24 |
| tristan | So anyway, just to say, I'm not demanding that it be perfect from day one :) | 12:25 |
| tristan | just trying to consider all the angles | 12:25 |
| jonathanmaw | tristan: not now we have some idea of API versioning :) | 12:25 |
| tristan | yeah, in my RFC today I kind of switched around... there are two approaches | 12:26 |
| tristan | One is, there is an API version for things, and depending on the API version: things are interpreted differently | 12:26 |
| tristan | The other is minimal bound dependencies, which is a bit harder but in some ways more sensible for the user | 12:26 |
| tristan | Which is basically: Every future version needs to understand everything in the previous format... so "foo" cannot one day be interpreted as anything other than the original definition of "foo" | 12:27 |
| tristan | However future versions can add features | 12:27 |
| tristan | And projects can require a minimum format version for an element | 12:27 |
| tristan | I feel like minimal bound dependency is more sensible to be honest | 12:28 |
| tristan | (which is what I'm proposing in the RFC) | 12:28 |
| tristan | alright, I should have finished this ostree-push work by tomorrow | 14:20 |
| tristan | but it's late, I'm going :) | 14:20 |
| tristan | o/ | 14:20 |
| *** tristan has quit IRC | 14:23 | |
| tlater | o/ | 14:23 |
| jonathanmaw | tristan: hrm, _yaml.node_copy seems to be misbehaving. the problem seems to be that it forgot to copy the value into the new dict if it's not either a dict, a list or provenance | 14:28 |
| jjardon[m] | jonathanmaw: I keep getting notifications about messages sent to you to the list; are you not a member? | 14:38 |
| jjardon[m] | jonathanmaw: nm, seems you are now | 14:39 |
| jonathanmaw | jjardon[m]: yep, my webmail client likes to send E-mails as jonathan.maw@codethink.co.uk or jonathan.maw@codethink.com seemingly at random | 14:40 |
| jonathanmaw | I need to re-learn how to use mutt | 14:40 |
| jjardon[m] | :) | 14:40 |
| *** tristan has joined #buildstream | 14:50 | |
| jonathanmaw | tristan: following E-mail conversations, I've taken my dpkg-build branch and am pre-emptively auto-mangling package name and adding a "name" field to dpkg-data which overrides it. | 14:57 |
| *** waltervargas[m] has quit IRC | 15:35 | |
| *** mattiasb has quit IRC | 15:35 | |
| *** tristan has quit IRC | 15:35 | |
| *** juergbi has quit IRC | 15:35 | |
| *** persia has quit IRC | 15:35 | |
| *** jjardon[m] has quit IRC | 15:35 | |
| *** dabukalam has quit IRC | 15:35 | |
| *** tristan has joined #buildstream | 15:36 | |
| *** dabukalam has joined #buildstream | 15:36 | |
| *** jjardon[m] has joined #buildstream | 15:36 | |
| *** persia has joined #buildstream | 15:36 | |
| *** juergbi has joined #buildstream | 15:36 | |
| *** waltervargas[m] has joined #buildstream | 15:36 | |
| *** mattiasb has joined #buildstream | 15:36 | |
| gitlab-br-bot | push on buildstream@workspaces (by Tristan Maat): 3 commits (last: _pipeline.py: Implement workspace yaml logic) https://gitlab.com/BuildStream/buildstream/commit/e3afda36470545644e49216bf85efde61fb37652 | 16:32 |
| *** xjuan has joined #buildstream | 16:44 | |
| *** tiagogomes has quit IRC | 16:50 | |
| *** jonathanmaw has quit IRC | 16:59 | |
| gitlab-br-bot | push on buildstream@workspaces (by Tristan Maat): 1 commit (last: Improve documentation) https://gitlab.com/BuildStream/buildstream/commit/e4c3a4d83a46e5bfeb700a5ae428a4d928a58cbe | 17:02 |
| *** tlater has quit IRC | 17:05 | |
| *** ssam2 has quit IRC | 19:15 | |
| *** xjuan has quit IRC | 22:34 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!