| *** mohan43u has quit IRC | 03:04 | |
| *** mohan43u has joined #buildstream | 03:49 | |
| *** tristan has joined #buildstream | 06:18 | |
| Nexus | tristan: you on yet? | 07:56 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (jjardon/license->master: docs: Add license: Creative Commons Attribution 4.0 International License) #336 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/336 | 08:14 |
| tristan | Nexus, yep, trying to get a release out today... | 08:33 |
| tristan | Nexus, maybe we should take a moment to talk about this workspace thing ? | 08:33 |
| Nexus | not a good time then? :) | 08:33 |
| Nexus | yes, that was what i wanted you for :) | 08:33 |
| tristan | Well, I think you need some help yeah :) | 08:33 |
| tristan | I didnt have time to look deeply into it, but I was concerned there is a conflicting issue with what a source does for opening workspaces | 08:34 |
| tristan | So I just threw that into your court hoping you'd come up with some possible courses of action | 08:34 |
| tristan | Nexus, I know it was not an "easy" think I was asking of you, btw :) | 08:35 |
| Nexus | https://gitlab.com/BuildStream/buildstream/issues/311 | 08:35 |
| Nexus | those are the ideas i came up with | 08:35 |
| tristan | right, I responded to that... | 08:35 |
| tristan | I was hoping that you would come up with a response to my "Workspace special sauce" part | 08:36 |
| Nexus | ah i see, i think i need to understand exactly HOW that gets applied before i can really put in an option for dealing with it | 08:37 |
| Nexus | my knowledge of the caching system is still shakey | 08:37 |
| Nexus | and my workspace knowledge, more so | 08:37 |
| Nexus | my initial thought was to attempt to do the check for a cache, early enough that it doesn't invoke the "special sauce" but i don't know if that is possible | 08:38 |
| tristan | So, I want to be helpful, and at the same time I want to educate you on how to figure it out yourself, which is why I often answer questions, with questions :) | 08:38 |
| tristan | That is one option, but it's a bit less than ideal right ? | 08:39 |
| Nexus | yes, it's a bit of a hack, but without knowing more, it's hard to give a better option | 08:39 |
| tristan | I mean, if you work on webkit, it means you might have commit access to webkit, and you might want to push patches to the upstream git | 08:39 |
| tristan | That doesnt mean you don't want the advantage of workspacing and already cached build tree, though | 08:40 |
| tristan | One thing we should keep in mind though, is when this "special sauce" was introduced | 08:40 |
| tristan | Can we find that out ? | 08:40 |
| Nexus | do know you know where i can start looking for that? | 08:41 |
| tristan | Well, first of all, where is the abstract method ? | 08:41 |
| tristan | Nexus, if the special sauce has to do with sources, ... would it make sense to check say, in source.py ? | 08:42 |
| Nexus | potentially in the init_workspace method | 08:43 |
| tristan | Correct, that is the one :) | 08:44 |
| tristan | So, in the absence of very thorough documentation explaining how people should implement plugins (currently we _only_ have reference material in this regard)... | 08:44 |
| tristan | Nexus, how would we go about finding out how this abstract method is employed ? | 08:44 |
| Nexus | looking into the current implementations of source plugins? | 08:45 |
| tristan | That is one way indeed, that tells us how Source plugins are implementing it, which gives us some valuable clues | 08:46 |
| tristan | Nexus, another thing we can do is check how the core employs the method | 08:46 |
| tristan | The first thing I would do, in the absence of this know how, is start with grep | 08:46 |
| tristan | I.e. `grep -r init_workspace buildstream/` | 08:47 |
| Nexus | in _pipeline? | 08:47 |
| tristan | Will give you an interesting list of results | 08:47 |
| tristan | grep is a programmers best friend, really. | 08:47 |
| paulsherwood | tristan: second-best. git stole my heart :) | 08:49 |
| tristan | paulsherwood, :) | 08:49 |
| tristan | we're getting to git, soon | 08:49 |
| tristan | Nexus, my results are telling me that source.py has also a private _init_workspace function, this is probably where the calling side of things are happening | 08:50 |
| tristan | ah, but pipeline indeed has the bulk of it | 08:50 |
| Nexus | so pipeline calls _init_workspace for each source | 08:51 |
| tristan | Correct | 08:51 |
| Nexus | which seems to, in turn, call relevent bash commands if invoked | 08:52 |
| tristan | Nexus, to expedite, I'm going to jump right to the conclusion here... look at Source.init_workspace() abstract method | 08:52 |
| tristan | This can be overridden by Source implementations, but it does something by default; unlike other methods; it does not raise ImplError() | 08:52 |
| Nexus | tristan: in the 2 cases i'm looking at, if the source implements init_workspace, it doesn't call stage | 08:54 |
| tristan | So by default, it calls Source.stage() | 08:54 |
| tristan | Nexus, ok so, looking at the git and bzr plugin implementations of init_workspace(), we can see that they basically do Source.stage() a little bit differently | 08:56 |
| tristan | Nexus, the conclusion I'm getting at here is, at a highlevel; when we delegate init_workspace() to a Source plugin; we delegate *all the work* to the Source plugin | 08:56 |
| tristan | Which makes this very difficult, if we want to stage a while cached build directory; asking the plugin to init_workspace() doesnt work anymore, right ? | 08:57 |
| * Nexus nods | 08:58 | |
| tristan | Ok well, I have unfortunate news :-/ | 08:59 |
| tristan | I wasnt sure, because it took a while | 08:59 |
| Nexus | ? | 09:00 |
| tristan | Nexus, the next path of thinking I was moving onto was... "Can this be changed ?" | 09:00 |
| tristan | What would we need to check, if we wanted to change init_workspace() to conform more to something we can use ? | 09:01 |
| tristan | Nexus, you may not be familiar with this aspect yet but, we release stable versions on a 6 month basis | 09:01 |
| Nexus | if we can change it without breaking existing functionality? | 09:02 |
| tristan | Nexus, for BuildStream, we use an "even minor point = stable" and "odd minor point = unstable/dev", we only guarantee API stability in stable | 09:02 |
| tristan | This means, whatever API is introduced during unstable, once we reach that stable release (for us it's coming up after this summer GUADEC, for 1.2)... those things by rule "cannot be changed" | 09:03 |
| tristan | Nexus, while it is undesirable to change APIs during unstable cycles, it is completely within our rights to do so | 09:03 |
| tristan | The point is, every stable release can only add new API, but not break API which was present in an older stable release | 09:04 |
| Nexus | which this would do? | 09:04 |
| tristan | Nexus, how would we find out ? | 09:05 |
| Nexus | by trying it? | 09:05 |
| tristan | Nexus, I'm sorry, I'm focusing much more on coaching you at the moment than the issue at hand, but I think it's a good exercise | 09:05 |
| tristan | By trying it | 09:05 |
| tristan | hmmm | 09:05 |
| tristan | Nexus, what would "trying it" entail, I'm not sure I understand ? | 09:06 |
| Nexus | by seeing if you can implement a way that allows the older stable to work whilst still introducing the new functionality | 09:07 |
| tristan | Ok I think you misunderstand my question, no biggy... | 09:08 |
| Nexus | probably | 09:08 |
| tristan | "How would we find out ?" was in response to "which this would do ?" | 09:08 |
| tristan | What I mean is... | 09:09 |
| tristan | "How would we find out if such a change to init_workspace() would break the API in an already stable release ?" | 09:09 |
| Nexus | running a currently stable use case on it? | 09:09 |
| tristan | Ok, that is a way :) | 09:10 |
| tristan | But it's horribly inefficient hehe | 09:10 |
| tristan | Nexus, ask git, would be much easier | 09:10 |
| Nexus | git can do that? | 09:10 |
| tristan | Nexus, there are multiple ways... I'll first suggest the way which was more round-about, but interesting | 09:11 |
| tristan | First, try `git annotate buildstream/source.py` | 09:11 |
| tristan | And search for init_workspace() | 09:11 |
| Nexus | ok | 09:12 |
| Nexus | i got 4 results | 09:12 |
| tristan | well, I sort of meant, looking at output; git will pipe it into your $PAGER, which will be `less` by default | 09:13 |
| Nexus | ah, i piped it into a grep | 09:13 |
| tristan | so you would type `/` to start searching, type init_workspace and then ENTER (as you would do in less or vi) | 09:13 |
| Nexus | kk | 09:14 |
| tristan | right, better to look at the output without the grep here | 09:14 |
| Nexus | got it | 09:14 |
| tristan | get a feel for what annotations does | 09:14 |
| tristan | So now you see a source file, annotated by git | 09:14 |
| Nexus | looks a lot like blame | 09:14 |
| tristan | Now you can see every line number, and every commit sha on the left | 09:14 |
| tristan | yeah | 09:14 |
| tristan | same thing | 09:14 |
| tristan | So you can see init_workspace() method all comes from 90850f2bd | 09:15 |
| Nexus | yep | 09:15 |
| tristan | if you git show that, you can see the pipeline changed to call that instead of Source._stage(), etc | 09:16 |
| tristan | Now, if you type `git tag`, you can see the release tags we have | 09:16 |
| Nexus | yup | 09:16 |
| tristan | Nexus, you can basically use this (much less efficient) technique to check if 90850f2bd comes before, or after the tag 1.0.0 | 09:16 |
| tristan | The more efficient technique I would say, would be to just `git checkout bst-1.0`, which is the "latest stable branch" | 09:17 |
| Nexus | how do i connect the two? | 09:17 |
| tristan | and just check if init_workspace() is there | 09:17 |
| tristan | Well, you can do that in a few ways, one is you can do `git log 1.0.0`, and search for 90850f2bd | 09:18 |
| tristan | and then be disappointed, because you found it :'( | 09:18 |
| Nexus | yes :( | 09:18 |
| tristan | Nexus, right, so now we have the sad and depressing news that it exists in stable, so by rule of law, it should not change | 09:18 |
| Nexus | So we need another way | 09:19 |
| tristan | Strictly speaking yes, in this case, 1.0.x is our *first* stable release cycle - and we very highly suspect that damage is minimal, if existent at all, should we change this particular API | 09:20 |
| tristan | So, a maintainer *might* be convinced to accept a patch to change that at this stage, if it were to be known to break nothing in the wild, and significantly improve the user experience | 09:21 |
| juergbi | you can also use: git branch --contains 90850f2bd bst-1.0 | 09:21 |
| tristan | juergbi, nice :) | 09:21 |
| tristan | I'll try to remember that, it's neat | 09:21 |
| juergbi | or rather, 1.0.0 | 09:21 |
| Nexus | that didn't seem to do anything | 09:21 |
| tristan | 1.0.0 is a tag, though, is that still right ? | 09:21 |
| juergbi | hm, i thought it would, both just a ref, but it doesn't seem like it | 09:22 |
| tristan | looks like it outputs bst-1.0 | 09:22 |
| tristan | but maybe this reports something for scripted use, via exit status | 09:23 |
| juergbi | without the branch argument it lists the branches that contain the commit | 09:23 |
| tristan | Ah | 09:23 |
| tristan | pretty nice feature anyway | 09:24 |
| juergbi | git tag --contains 90850f2bd | 09:24 |
| tristan | Nexus, I am personally tempted to change this; other source plugins known to exist in the wild are very few; and relate to binary stuff which may not make much sense to have special support for workspaces anyway | 09:24 |
| Nexus | ah, thats better | 09:24 |
| tristan | Not sure what juergbi thinks | 09:25 |
| tristan | Nexus, the rationale here being: I see an opportunity for the core to first do "Source.stage() OR stage a cached build tree" and THEN do "Source.init_workspace()" on all sources AFTER the fact | 09:25 |
| tristan | Which I'm not entirely sure makes the most sense, but it's my gut feeling that that would solve this well | 09:26 |
| tristan | There may be other problems with it, though | 09:26 |
| tristan | juergbi, what do you think about this (actually pretty tricky) problem, for https://gitlab.com/BuildStream/buildstream/issues/311 ? | 09:27 |
| juergbi | not sure whether you discussed this already, but don't we first have the more fundamental issue of .git being part of the source/build tree | 09:27 |
| juergbi | i.e., as it is right now, we'd have to cache the whole .git dir | 09:28 |
| tristan | Ewwwwww | 09:28 |
| juergbi | that's actually also an issue for remote execution | 09:28 |
| tristan | nice catch, I think that crossed my mind the other day, and I conveniently swept it under the rug | 09:28 |
| tristan | Actually, I think I was thinking of that in another context, yeah... `bst source-bundle` was what I was thinking of | 09:29 |
| tristan | Hmmm, so this problem grows | 09:29 |
| juergbi | ideally, i would like to see .git only in workspaces and not in regular source/build directories | 09:29 |
| juergbi | however, that breaks builds | 09:29 |
| tristan | Unfortunate, but true | 09:29 |
| juergbi | even buildstream itself is guilty of that | 09:29 |
| tristan | it breaks builds for build scripts which I tend to frown upon :-/ | 09:30 |
| tristan | because of setuptools_scm, indeed | 09:30 |
| juergbi | yes | 09:30 |
| tristan | juergbi, our build scripts, by nature of being setuptools driven, are far from model citizens :D | 09:30 |
| tristan | hehehe | 09:30 |
| tristan | sigh | 09:31 |
| tristan | Ok so, I can see an opportunity for not breaking API at all too, but it means adding more API and conditional support | 09:31 |
| tristan | But this probably needs some more design than "just that" | 09:31 |
| tristan | Because if we were to do this, we would want to also cover some way of staging sans .git/ (or .bzr/ or whatever) into a build directory too | 09:32 |
| tristan | And we'd want source YAML options to turn it off for offensive modules | 09:32 |
| juergbi | yes, something like that | 09:34 |
| * Nexus got a little lost in that convo | 09:39 | |
| tristan | Nexus, when juergbi says "not sure whether you discussed this already, but don't we first have the more fundamental issue of .git being part of the source/build tree" | 09:43 |
| tristan | Nexus, do you see the issue with that ? | 09:43 |
| tristan | Nexus, consider say, that we are caching a build tree of linux.git | 09:43 |
| Nexus | it could be huge? | 09:45 |
| tristan | Correct | 09:45 |
| Nexus | ah | 09:45 |
| Nexus | so we need to exclude the .git dir? | 09:46 |
| tristan | So, it also stands to reason that cloning the whole repository at build time, just to build... is also inefficient | 09:46 |
| tristan | Were we to solve it at the time of staging, the issue is much lessened for cached build trees anyway | 09:47 |
| tristan | so I doubt it's important to special case that for workspaces, if we solve it at staging time | 09:47 |
| tristan | *but*, it brings back the issue that; developers who want a workspace; they want their whole repository to work in | 09:48 |
| *** jonathanmaw has joined #buildstream | 09:49 | |
| tristan | From what I understand, build tree caching is still done without any additional file metadata, i.e. like with artifacts, we dont preserve mtime of files | 09:49 |
| tristan | Instead, for the purpose of incremental builds, we have a heuristic in place which makes everything the same timestamp | 09:50 |
| Nexus | the files are currently just lifted out of the sandbox and cached, nothing is done to them | 09:50 |
| tristan | and then we have a (still not executed ?) plan in place to make sure that "dependencies which have changed" since the last build, are staged with a hard coded *newer* timestamp | 09:51 |
| tristan | So, this leads me to deduce the following: | 09:52 |
| *** tiago has joined #buildstream | 09:52 | |
| tristan | If we enhance our staging in such a way described above, such that by default history is excluded unless explicitly requested... | 09:52 |
| Nexus | Doesn't that lead to the same issue as before? | 09:53 |
| Nexus | if someone is relying on the history being there in stable for example | 09:54 |
| tristan | Then, at the time of opening a workspace against an already cached build tree; we can at that time *overwrite* the revisioned files, with history and all, over the cached build, and ensure they are set to the correct mtime | 09:54 |
| tristan | This would not cause any concern of inconsistent files and state, because if the cache keys match; then the sources must be exactly the same | 09:54 |
| tristan | *or*, we could take into account builds which *modify* sources inline | 09:55 |
| tristan | And do a sort of shadow copy | 09:55 |
| tristan | Where we do: A.) Checkout a git tree to a certain temporary location... B.) Copy over files to the staging area *only if they do not already exist in the staging area* | 09:55 |
| tristan | juergbi, ^^^^^^ sounds relatively sane ? | 09:56 |
| juergbi | in the future we probably also want to be able to use non-identical cached build trees. so we want the real vcs sources to override the cached build tree where there is a mismatch, at least long term | 09:56 |
| tristan | Hmmm, this is for non-workspace related incremental builds | 09:57 |
| juergbi | yes | 09:57 |
| juergbi | not a topic right now | 09:57 |
| juergbi | but we should already align the logic | 09:57 |
| tristan | interesting, but I think with this approach we can fry that cat when it scratches us in the face | 09:57 |
| tristan | well, you're right, we should take it into account in design | 09:58 |
| juergbi | interesting saying | 09:58 |
| juergbi | ;) | 09:58 |
| tristan | I like to be visual with my expressions :) | 09:58 |
| tristan | So now, the final step to this brainstorming I think, is what part of this can be separated | 09:58 |
| juergbi | we can also postpone it if it's too complex for now | 09:58 |
| juergbi | just wanted to raise awareness | 09:59 |
| tlater | juergbi just suggests turning the frier on | 09:59 |
| juergbi | haha | 09:59 |
| tristan | I.e. the optimization of shallow staging solves the issue for both things at once, so I would be interested in an approach where we can first land this workspaces using cached build trees, and then later optimize with shallow staging | 10:00 |
| juergbi | i.e., first cache whole .git? | 10:01 |
| tristan | Yes, specifically because we have this caching of build trees branch ready and waiting for a purpose | 10:02 |
| juergbi | makes sense. it shouldn't break anything | 10:02 |
| juergbi | but it's a reason more to make it optional, at least for the time being | 10:03 |
| tristan | indeed | 10:03 |
| tristan | Maybe API-wise this can be a new Source.stage_full() method, which defaults to existing Source.stage() in the absence of an implementation | 10:04 |
| tristan | This could have some additional options, such as "Overwrite the files in the destination", "Only populate files in the destination if they do not yet exist", "Stage with or without history" | 10:05 |
| tristan | But by default, Source.stage() is used ? ... hmmm, no that will not work for workspaces on cached build directories, cause the default implementation will fail. | 10:06 |
| gitlab-br-bot | buildstream: merge request (jmac/exception-hook->master: Add a handler for otherwise unhandled exceptions in the main BuildStream app) #248 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/248 | 10:09 |
| *** aday has joined #buildstream | 10:13 | |
| gitlab-br-bot | buildstream: merge request (issue-243_dpkg_import_source_plugin->master: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 10:13 |
| gitlab-br-bot | buildstream: issue #313 ("empty /buildstream/build directory when dropping to a shell after a failed build") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/313 | 10:20 |
| jmac | I've not come across the public/private distinction in Python before - is it defined in a PEP? | 10:21 |
| tristan | jmac, I think I saw a discussion where you asked about making private accessors like the others | 10:21 |
| tristan | but failed to reply in that topic | 10:22 |
| jmac | Yes you did | 10:22 |
| tristan | Anyway, single underscore in python is private by convention, importing something to a public module from a private one, makes it public | 10:22 |
| tristan | This cannot be enforced, but it is a general convention; it will for instance be considered by Sphinx when generating documentation | 10:23 |
| *** dominic has joined #buildstream | 10:23 | |
| tristan | The double underscore is special in python | 10:23 |
| jmac | You were asking for a Sandbox._get_config() accessor | 10:23 |
| tristan | jmac, the double underscore actually creates a namespace when viewed from outside of the declaring object, so internally it can be accessed as __config, while externally it can only be abused as _Sandbox__config | 10:24 |
| jmac | OK, not come across any of this before. | 10:25 |
| tristan | jmac, for BuildStream... (while presently privateness is in a messy state, there is a bug I filed last week about that) we only use the double underscore for abstract classes which we expect to be derived | 10:25 |
| tristan | So, that's why we use it heavily in Element and Source, and also in Sandbox | 10:25 |
| tristan | This is just some added protection to free up namespace in concrete implementations | 10:25 |
| tristan | jmac, if we wanted to be more robust in this regard, we would create accompanying Controller classes for the private methods we use inside BuildStream | 10:26 |
| tristan | we just didnt take it *that* far | 10:26 |
| jmac | It's possible Sandbox._get_config will become necessary if we tighten up the scope | 10:26 |
| *** dominic has quit IRC | 10:26 | |
| *** dominic has joined #buildstream | 10:26 | |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 10:27 |
| tristan | jmac, in my review comment I have now asked for one, I failed to reply to that topic (think the discussion is resolved), but instead made a new comment asking for one | 10:27 |
| tristan | sorry for that confusion :) | 10:27 |
| jmac | np | 10:28 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 10:29 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 10:30 |
| Nexus | tristan: So was a conclusion reached about wether or not you want to merge a change to the API? | 10:39 |
| tristan | right, I was just about to get back to you before I go | 10:41 |
| tristan | there was not a solid conclusion reached per se | 10:41 |
| tristan | Nexus, for right now, it would be best that you implement the feature while bypassing init_workspace() completely, as a first step | 10:41 |
| tristan | Nexus, after that, we have to come up with a plan to address it, depending on how complex that plan is; we may merge the initial implementation "without the special init_workspace() sauce" and punt that part as an enhancement | 10:42 |
| tristan | Or, we may block the patch on first fixing that so that it works nicely off the bat | 10:43 |
| tristan | Nexus, make sense ? | 10:43 |
| Nexus | yep | 10:43 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 10:44 |
| gitlab-br-bot | buildstream: merge request (richardmaw/build-status-extensions->master: WIP: Print summary of failed builds) #329 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/329 | 10:46 |
| *** tristan has quit IRC | 10:49 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 10:59 |
| *** ernestask has joined #buildstream | 11:13 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 11:25 |
| *** jennis has quit IRC | 11:31 | |
| *** jennis has joined #buildstream | 11:34 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 11:40 |
| *** tiago has quit IRC | 11:47 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 12:01 |
| gitlab-br-bot | buildstream: merge request (issue-243_dpkg_import_source_plugin->master: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 12:03 |
| jmac | I don't know if tristan is still online; I'm having trouble with the scoping issues on !318 (see the latest comment) | 12:11 |
| Nexus | no he want off | 12:16 |
| jmac | In which case, could anyone else take a look at it? | 12:16 |
| jennis | Has anyone built anything with gnome-build-meta recently? | 12:19 |
| jennis | jjardon[m]? | 12:19 |
| jennis | Getting these errors: https://paste.baserock.org/etaritetig | 12:19 |
| juergbi | jmac: one option could be to put it in something like sandbox/_config.py and import that from Element | 12:20 |
| juergbi | that would still keep it out of the really public API (the plugin facing one) | 12:21 |
| juergbi | or follow the _SandboxConfig approach | 12:21 |
| juergbi | I'm not an expert on best practice with visibility on Python, though. maybe there are better solutions | 12:22 |
| jmac | That's just a name change from _private to _config, then | 12:23 |
| *** tiago has joined #buildstream | 12:23 | |
| jmac | _SandboxConfig sounds like the best option so far | 12:24 |
| jmac | Hence why I asked if there's some PEP document about how visibility should work - I don't know what the rules are for this | 12:24 |
| *** bethw has joined #buildstream | 12:25 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 12:25 |
| juergbi | jmac: yes but Element could import from sandbox._config. importing from sandbox._private seems a bit odd | 12:27 |
| juergbi | there is https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces but i don't think it helps here | 12:27 |
| jjardon[m] | jennis: update your bst | 12:28 |
| juergbi | but _SandboxConfig might be better | 12:28 |
| jjardon[m] | tristan: anything we can do to avoid above situations? ^ | 12:28 |
| jennis | jjardon[m], I have | 12:28 |
| juergbi | jjardon[m]: set format-version in project.conf | 12:28 |
| juergbi | ah, already the case, hm | 12:28 |
| jennis | jjardon[m], ahh dammit don't worry | 12:29 |
| jjardon[m] | tristan: maybe bst can check if it supports a specific version of bst format and fail with a nice "I do not support this" or something like that? | 12:29 |
| juergbi | jjardon[m]: see the last line in the pastebin | 12:29 |
| juergbi | however, i don't know why this wasn't shown in the first invocation | 12:29 |
| juergbi | maybe we check this too late? | 12:30 |
| jjardon[m] | ah nice :) | 12:30 |
| jjardon[m] | you are already on it | 12:30 |
| jjardon[m] | yeah, maybe is better to exit inmediatelly | 12:30 |
| juergbi | looks like a bug | 12:30 |
| Nexus | juergbi: can you merge this? :) Pretty please with a cherry on top https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 12:32 |
| jjardon[m] | juergbi: do you want me to open an issue? | 12:38 |
| juergbi | jjardon[m]: yes, please do | 12:39 |
| gitlab-br-bot | buildstream: merge request (issue-243_dpkg_import_source_plugin->master: Created DPKG Source plugin for Issue #10) #305 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 12:41 |
| gitlab-br-bot | buildstream: issue #314 ("buildstream should exit inmediatelly if it format is not supported") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/314 | 12:44 |
| jennis | So artifact cache's can be declared in project.conf or ~/.config/buildstream.conf... The documentation shows this for caches which we pull from. For a artifact cache that we also intend to push to, can we also declare this in either? | 12:45 |
| juergbi | jennis: yes, you can specify push in both locations | 12:50 |
| jennis | thanks juergbi | 12:52 |
| *** mohan43u has quit IRC | 12:53 | |
| *** mohan43u has joined #buildstream | 12:59 | |
| jjardon[m] | juergbi: any idea if the differences between format versions are documented somewhere? | 13:04 |
| juergbi | jjardon[m]: the format documentation contains notes about features that have been introduced after the initial stable release | 13:08 |
| juergbi | 'available since format version N' | 13:08 |
| jjardon[m] | juergbi: nice, thaks. | 13:09 |
| * jjardon[m] searchs for the format docs | 13:09 | |
| Nexus | ty juergbi \o/ | 13:10 |
| gitlab-br-bot | buildstream: merge request (buildproject->master: Added buildproject doc) #238 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/238 | 13:11 |
| gitlab-br-bot | buildstream: merge request (createProject->master: Adding create project document) #237 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/237 | 13:11 |
| gitlab-br-bot | buildstream: merge request (postbuild->master: Added Postbuild documentation) #236 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/236 | 13:11 |
| gitlab-br-bot | buildstream: merge request (modAndTest->master: Making changes to various documents:) #206 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/206 | 13:11 |
| tlater | Hm, is gnome-modulesets outdated? | 13:11 |
| tlater | Its core-deps/meson.bst still has a "pre-configure-commands" key | 13:12 |
| * Nexus can't remember what's going on with https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 13:12 | |
| gitlab-br-bot | buildstream: merge request (jmac/build-uid-2->master: Sandbox configuration key to specify uid/gid for sandbox) #318 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/318 | 13:16 |
| gitlab-br-bot | buildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 13:23 |
| Nexus | juergbi: are you happy with https://gitlab.com/BuildStream/buildstream/merge_requests/310 ? | 13:27 |
| Nexus | tlater: which thing are you building? | 13:27 |
| * tlater was building core.bst for a rudimentary staging performance check. | 13:28 | |
| juergbi | Nexus: I'm confused. that still sets the element variable but the tests are passing | 13:30 |
| juergbi | ah, it might use the element-name '' fallback | 13:31 |
| juergbi | sounds like better test would be in order | 13:31 |
| juergbi | and obviously fixing the remaining 'element' | 13:31 |
| gitlab-br-bot | buildstream: merge request (jmac/build-uid-2->master: Sandbox configuration key to specify uid/gid for sandbox) #318 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/318 | 13:32 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 13:34 |
| Nexus | juergbi: now i'm confused, because i definitely changed it... | 13:34 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 13:34 |
| Nexus | i've just re-pushed it | 13:34 |
| gitlab-br-bot | buildstream: issue #315 ("bash completion not working properly when adding PATH="${PATH}:~/.local/bin"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/315 | 13:36 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 13:42 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 13:43 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 13:47 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 13:50 |
| gitlab-br-bot | buildstream: merge request (jmac/build-uid-2->master: Sandbox configuration key to specify uid/gid for sandbox) #318 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/318 | 13:54 |
| *** xjuan has joined #buildstream | 14:13 | |
| gitlab-br-bot | buildstream: merge request (jmac/build-uid-2->master: Sandbox configuration key to specify uid/gid for sandbox) #318 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/318 | 14:15 |
| laurence | Nexus, re Allow project relative workspace paths - https://gitlab.com/BuildStream/buildstream/merge_requests/251, looks like it needs reviewing before the WIP is removed | 14:29 |
| laurence | seems to have been over-looked / dropped in the past, but i also can't remember why | 14:29 |
| mohan43u | hi I'm getting socket timeout "Failed to fetch ref 'debian/testing/amd64' from 'origin': Socket I/O timed out" for base/base-system.bst after like 2% downloaded.. any hints? | 14:31 |
| jmac | Right, I think that's all discussions resolved on !318 | 14:32 |
| tlater | mohan43u: That *sounds* like a network issue - might be something else though, perhaps someone else has seen it happen before. | 14:32 |
| Nexus | laurence: same, i'm not sure what happened to it, but i need to sort out the CI fails anyway | 14:35 |
| mohan43u | tlater: every time I retry, the total number of git chunks getting reduced, but timeout happens after after getting 15-20MB of data | 14:38 |
| tlater | Hrrmmm... I don't know enough about Linux networking to debug sockets, sorry mohan43u . I don't think there's a know network issue atm, though. | 14:40 |
| gitlab-br-bot | buildstream: merge request (jmac/exception-hook->master: Add a handler for otherwise unhandled exceptions in the main BuildStream app) #248 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/248 | 14:55 |
| tlater | mohan43u: Looks like you're asking at the wrong time :| I suggest asking a bit earlier in the day, as tristan is more likely to be around then. | 15:05 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 15:11 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 15:19 |
| gitlab-br-bot | buildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 15:35 |
| gitlab-br-bot | buildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 15:36 |
| *** valentind has joined #buildstream | 15:38 | |
| Nexus | juergbi: you there? | 16:06 |
| Nexus | please can someone review https://gitlab.com/BuildStream/buildstream/merge_requests/251 for me? | 16:19 |
| jennis | has anyone successfully set up a remote artifacts cache before? | 16:25 |
| * jennis is having problems | 16:25 | |
| Nexus | i think tlater explained it to me once | 16:26 |
| Nexus | so he might know | 16:26 |
| jmac | Nexus: I'm not very familiar with workspaces, but if both projectdir and workdir are absolute paths (returned by os.path.abspath) then does it makes sense to join them together? | 16:26 |
| *** mohan43u has quit IRC | 16:26 | |
| Nexus | jmac: workdir is a relpath ln622 | 16:27 |
| tlater | Nexus: I've been trying to guide jennis through it, actually, we're running into trouble with getting him authorized on his own cache | 16:27 |
| jmac | Nexus: Yes, but what if workdir doesn't start with projectdir? | 16:27 |
| Nexus | http://i.imgur.com/pLksjqM.png | 16:28 |
| *** mohan43u has joined #buildstream | 16:29 | |
| gitlab-br-bot | buildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 16:31 |
| Nexus | that should fix it | 16:31 |
| gitlab-br-bot | buildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/251 | 16:33 |
| *** striker has joined #buildstream | 16:39 | |
| jmac | Nexus: Now I'm a bit confused as to what this actually does - it looks like it's removing projectdir from the front of workdir and then putting it back on later | 16:44 |
| Nexus | jmac: i'm making it into a relative path, so it may end up being /buildstream/build/../foo/bar | 16:47 |
| Nexus | meaning oyu can move the project and it should still know where to find the workspace | 16:48 |
| jmac | But you turn workdir back into an absolute path before you call _set_workspace | 16:49 |
| tlater | Would there be any objections to including the artifact size in its metadata? | 16:51 |
| * tlater thinks this could speed up artifact cache size management a lot | 16:51 | |
| Nexus | jmac: yeah i see the problem, darn | 16:55 |
| Nexus | ty | 16:55 |
| juergbi | tlater: I don't see an issue with that, however, reading metadata of all artifacts sounds very expensive | 17:00 |
| *** sstriker has joined #buildstream | 17:00 | |
| *** striker has quit IRC | 17:01 | |
| *** sstriker has quit IRC | 17:01 | |
| tlater | juergbi: It's about expiring the correct number of artifacts | 17:01 |
| *** striker has joined #buildstream | 17:01 | |
| tlater | Once we have a list of LRU artifacts, removing 1 doesn't guarantee we have enough space for the new artifact | 17:01 |
| juergbi | ah, you just read it for artifacts you get rid of? | 17:01 |
| tlater | Yep :) | 17:01 |
| juergbi | Sounds fine to me | 17:01 |
| *** sstriker has joined #buildstream | 17:21 | |
| *** striker has quit IRC | 17:22 | |
| *** tiago has quit IRC | 17:33 | |
| *** bethw has quit IRC | 17:40 | |
| *** dominic has quit IRC | 17:45 | |
| *** aday has quit IRC | 17:46 | |
| *** aday has joined #buildstream | 18:00 | |
| *** jonathanmaw has quit IRC | 18:24 | |
| *** aday has quit IRC | 18:32 | |
| *** aday has joined #buildstream | 19:06 | |
| *** aday has quit IRC | 19:27 | |
| *** ernestask has quit IRC | 20:38 | |
| *** sstriker has quit IRC | 21:07 | |
| *** striker has joined #buildstream | 21:07 | |
| *** sstriker has joined #buildstream | 21:12 | |
| *** striker has quit IRC | 21:15 | |
| *** sstriker has quit IRC | 21:32 | |
| *** valentind has quit IRC | 23:25 | |
| *** xjuan has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
