| *** mcatanzaro has joined #buildstream | 00:18 | |
| gitlab-br-bot | buildstream: issue #186 ("Improve timestamps logging") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/186 | 00:45 |
|---|---|---|
| jjardon[m] | Hi, does the bst cache server have any automatic cleanup? | 00:53 |
| jjardon[m] | if yes, is it configurable? | 00:53 |
| jjardon[m] | I can not find any mention about that at https://buildstream.gitlab.io/buildstream/artifacts.html | 00:56 |
| *** mcatanzaro has quit IRC | 01:10 | |
| persia | jjardon[m]: I think the answer to both is "Not yet". | 05:43 |
| *** tristan has joined #buildstream | 08:10 | |
| jjardon[m] | Persia OK, thanks. tristan mind if I fill an issue? | 08:45 |
| tlater | jjardon[m]: There already is one for this | 08:47 |
| tlater | Lemme dig it up | 08:47 |
| tristan | jjardon[m], there is one | 08:48 |
| tlater | jjardon[m]: https://gitlab.com/BuildStream/buildstream/issues/135 and https://gitlab.com/BuildStream/buildstream/issues/136, in case you want to follow | 08:48 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues/135 and https://gitlab.com/BuildStream/buildstream/issues/136 | 08:48 |
| tristan | :) | 08:48 |
| jjardon[m] | Ah! You already have think on everything :) | 08:48 |
| * jjardon[m] subscribes | 08:49 | |
| tristan | Well, knowing is only half the battle :) | 08:49 |
| tristan | but... half a bottle is better than an empty one ? | 08:49 |
| tlater | Depends on what the other half of the bottle contains | 08:50 |
| jjardon[m] | Also, I noticed the cache server is quite CPU intensive (at least in our use case) ; maybe it would be good to document that somewhere? | 08:50 |
| tristan | Is it ? | 08:50 |
| tristan | well | 08:50 |
| tristan | jjardon[m], you could... I'm not sure what would cause that to be honest, it's mostly I/O intensive | 08:51 |
| tristan | jjardon[m], we stream ostree "objects" to the server and it dumps it on disk, and after that renames them into place, the objects are pre-compressed | 08:51 |
| tristan | So, a bit weird that it would be cpu intensive indeed | 08:51 |
| tristan | Oh, perhaps updating the summary | 08:52 |
| tristan | that needs to be done at the end of every transaction | 08:53 |
| tristan | and is probably costly | 08:53 |
| jjardon[m] | Tristan I will open an issue explaining the use case and the current hardware config | 08:53 |
| tristan | Sure, if it's causing you trouble; I dont know that it's fixable | 08:53 |
| tristan | the resolution of that may be to enhance the artifact server setup docs | 08:54 |
| jjardon[m] | Sam and me it would be more memory bounded and we choose a server with a single CPU and 8gb ram; the CPU is almost all the time at 100% | 08:54 |
| tristan | you do a lot of pushes ? | 08:55 |
| jjardon[m] | So a note saying; hey! You need CPU for this would be handy | 08:55 |
| tristan | I guess, from your CI | 08:55 |
| jjardon[m] | Well, we can have several runners at the same time, yes | 08:55 |
| tristan | indeed, I mean; if you have it idling and taking 100% cpu that would indicate a bug in bst-artifact-receive | 08:56 |
| jjardon[m] | But start to be in trouble with only 3-4 running already | 08:56 |
| tristan | but it does to an ostree summary update on every push, that's needed | 08:56 |
| tristan | s/to/do | 08:56 |
| jjardon[m] | No no, its working like crazy; I will put details later | 08:57 |
| tristan | ok :) | 08:57 |
| tlater | tristan: Did we ever have a ./setup.py integration-tests or something of the sort? | 08:59 |
| jjardon[m] | We only open this for now in our side: : https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/35 | 09:00 |
| jjardon[m] | To give you an idea, we fillup almost 200GB of the cache server in around a week. That's why I asked for the autocleanup mechanism as well ) | 09:01 |
| tristan | tlater, nope | 09:09 |
| tristan | jjardon[m], oh jaysus :) | 09:09 |
| tristan | jjardon[m], Have you noticed any curve on that ? | 09:10 |
| tristan | jjardon[m], i.e. what I would hope for (with a number like 200GB in a week), is that you've reached 190GB on day 6 | 09:11 |
| jjardon[m] | curve? | 09:11 |
| tristan | like, grow fast at first and then grow more slowly | 09:11 |
| jjardon[m] | mmm | 09:11 |
| tristan | The efficiency of that curve would be improved, the more reproducible the builds are | 09:12 |
| tristan | jjardon[m], i.e. that's kind of what ostree buys us | 09:12 |
| tristan | as an artifact server | 09:12 |
| jjardon[m] | I only noticed yesterday, I will try to monitor closely from now on | 09:12 |
| tristan | well, it's just a thing of curiousity really :) | 09:12 |
| tristan | we know it should do that; just curious if it was noticeable | 09:13 |
| jjardon[m] | We suffer this issue you fixed yesterday so probably some of the artifacts were incorrectly duplicated | 09:13 |
| tristan | jjardon[m], ah the local source plugin issue ? | 09:13 |
| jjardon[m] | Yeah | 09:13 |
| tristan | hmmm, interestingly; that should not greatly effect the cache size | 09:14 |
| tristan | even though it triggers rebuilds; (but assumption really relies on high percentage of reproducibility of builds) | 09:15 |
| jjardon[m] | Also, would it be possible to put a big fat warning when you have configured a cache server and bst fail to get the artifact? I mean every time that happen | 09:15 |
| tristan | I believe we do | 09:15 |
| tristan | We first do it once at startup (check the initial connectivity), and if we have it, we should be retrying 3 times for each pull iirc | 09:16 |
| tristan | One thing I'm certainly not open to, is adding exclamation points and competing warning message fatness haha | 09:17 |
| jjardon[m] | Is there a warning then if that fails? | 09:17 |
| jjardon[m] | Hahaha | 09:17 |
| jjardon[m] | OK, I will take a closer look next time | 09:17 |
| tristan | yeah, if there is no warning at startup let us know... I'm not 100% sure right now... | 09:18 |
| tristan | I'm sure that if we started up with connectivity and it subsequently fails, it will generate warnings and noise in the logs whenever it fails | 09:18 |
| tristan | if you have `pull` tasks at all, it means it detected connectivity | 09:19 |
| tristan | otherwise we never try them | 09:19 |
| *** valentind has joined #buildstream | 09:22 | |
| jjardon[m] | Right, I will let you know, cheers | 09:22 |
| *** jude has joined #buildstream | 09:31 | |
| gitlab-br-bot | buildstream: merge request (juerg/element-state->master: WIP: element state updates) #215 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/215 | 09:40 |
| gitlab-br-bot | buildstream: merge request (juerg/element-state->master: WIP: element state updates) #215 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/215 | 09:41 |
| tristan | juergbi, So I wanted to pick your brain about this: https://gitlab.com/BuildStream/buildstream/merge_requests/181/diffs#note_53641176 | 09:41 |
| tristan | juergbi, it looks like missed opportunity for API cleanup :-S | 09:42 |
| tristan | but maybe we can sneak something in at this stage, as it's very early | 09:42 |
| gitlab-br-bot | buildstream: merge request (juerg/element-state->master: element state updates) #215 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/215 | 09:42 |
| juergbi | let me take a look | 09:43 |
| tristan | juergbi, few options I can think of... one is; we could add an optional 'detail' keyword param to ElementError and SourceError, allowing us to construct a full error message from that in the log | 09:44 |
| tristan | and just nuke Plugin.error() from existence, that is I suppose the best for the code, but most aggressive break | 09:44 |
| juergbi | self.error was added only because ElementError doesn't support detail? | 09:44 |
| juergbi | or is the ElementError somehow not visible in the log? | 09:45 |
| tristan | It was added a looooooooong time ago | 09:45 |
| juergbi | i mean in this diff | 09:45 |
| tristan | Well, in that diff, jonathan adds self.error() + raise ElementError() | 09:45 |
| tristan | which is... ugh | 09:45 |
| tristan | self.error() does not cause the error, just sends a visible error message | 09:45 |
| juergbi | right, if we say errors are anyway fatal, there isn't really a point in also having .error() | 09:46 |
| tristan | Another less aggressive approach is to still add detail to the ElementError(), and cause .error() to raise it instead | 09:46 |
| tristan | Currently, there is nothing using Plugin.error() prior to this patch | 09:46 |
| juergbi | i don't think i like raising an error from a library method | 09:47 |
| tristan | it's there since we originally carved out the messaging API | 09:47 |
| juergbi | less clear code path | 09:47 |
| juergbi | (in this particular case) | 09:47 |
| juergbi | (it's obviously ok if a library method itself failed) | 09:48 |
| tristan | Mkay, so; i feel like this should also preemptively go into bst 1.0.1 | 09:48 |
| tristan | to reduce the chance that code breaks in 1.0 -> 1.2 transition | 09:48 |
| juergbi | we can still sneak it in, i suppose | 09:49 |
| tristan | it's approximately the same, Plugin.error() would not be returning in 1.2 | 09:50 |
| juergbi | but then you'd get a python error, not a proper buildstream error | 09:50 |
| tristan | Not sure what you mean | 09:51 |
| juergbi | what i don't like is having Plugin.error() as official/recommended way to _raise_ an error from a plugin | 09:51 |
| tristan | We should really have better developer docs, plugin and buildstream hacker facing hehe | 09:51 |
| juergbi | as i like clear code paths | 09:51 |
| tristan | Yes | 09:51 |
| juergbi | so the question would simply be how to best handle this given that 1.0 is already out | 09:52 |
| tristan | exactly | 09:52 |
| tristan | that's mostly the thing yeah | 09:52 |
| juergbi | it's up to you whether we should follow the approach 'not used yet in the wild, so nobody will care if we remove it in 1.0.1' | 09:53 |
| tristan | hehe | 09:53 |
| juergbi | 1.0.1 release should follow soon in that case, though | 09:53 |
| tristan | I know, I dont want it to be a precedent | 09:53 |
| tristan | Indeed | 09:54 |
| juergbi | does python has a way to declare deprecated API in such a way that the caller actually notices? | 09:54 |
| tristan | Only via documentation, and I'm not sure that there is anything builtin to sphinx for this | 09:55 |
| tristan | we would use `.. warning::\n\n Deprecated: ${version}` I guess | 09:55 |
| juergbi | theoretically we could also print a warning to the buildstream log | 09:56 |
| juergbi | and maybe even remove it from the documentation, so nobody gets confused | 09:56 |
| juergbi | a bit odd but would not break anything | 09:56 |
| *** jonathanmaw has joined #buildstream | 09:57 | |
| tristan | Actually, I'm still hunting for a method to remove specific symbols from the docs :-S | 09:57 |
| * tlater figures a custom decorator would work really well for this | 09:57 | |
| tristan | jonathanmaw, fwiw we're now discussing an alternative to having to both `self.error()` + `raise ElementError()` for the sake of being able to abort with a detailed message | 10:00 |
| jonathanmaw | tristan: cool, I was wrestling with that problem at the end of yesterday | 10:01 |
| tristan | yeah I got your added comments | 10:01 |
| tristan | jonathanmaw, I think the answer is going to be an added `detail` keyword argument to SourceError() and ElementError() | 10:01 |
| *** ssam2 has joined #buildstream | 10:03 | |
| tristan | juergbi, I think for this instance I'm going to break it and release early, to be frank, it's not the only breakage; I suppose it can be forgiven for an initial release | 10:05 |
| *** noisecell has joined #buildstream | 10:05 | |
| tristan | juergbi, the other break is the fix from yesterday for local source plugin, this was really actually broken so I can't leave it out, but the change does constitute a cache key change for local sources | 10:06 |
| juergbi | the main point to keep in mind is impact to other users/developers. if that's low enough, it's ok to occasionally break such rules | 10:06 |
| tristan | Yeah, in this case it's insanely low | 10:07 |
| juergbi | (even Linus accepts a rare userspace ABI breakage if it's known not to affect anything) | 10:07 |
| tristan | now that we're in 1.1 territory, we're going to need some since/deprecated annotations for the docs, maybe tlater's decorator suggestion is a great one for ensuring that shows up consistently | 10:08 |
| juergbi | yes, would be good. don't know about best practice in Python for this | 10:10 |
| juergbi | btw: comments/review on element state branch appreciated. i'm not aware of any remaining issues | 10:10 |
| * tristan is asking in #python and #sphinx-doc, but it's not a popular question in that crowd | 10:11 | |
| tristan | juergbi, yeah saw it roll through, will take a look today :D | 10:11 |
| juergbi | ta | 10:11 |
| tristan | hopefully we land ssam2's branch today and you can rebase junctions on that too | 10:11 |
| juergbi | yes, sounds good | 10:14 |
| jjardon[m] | Curious question; will 1.1 be the next version or the next stable will be 1.2 ? | 10:16 |
| *** jude has quit IRC | 10:26 | |
| *** jude has joined #buildstream | 10:26 | |
| tristan | jjardon[m], 1.2 | 10:27 |
| tristan | jjardon[m], master is technically 1.1.x | 10:27 |
| jjardon[m] | Nice, thanks | 10:39 |
| tristan | I dont have any strategy for post-release version bump though, unfortunately | 10:46 |
| tristan | It needs a tag | 10:46 |
| tristan | So, *perhaps* I should tag master at the branchpoint 1.1.0 without a release, and just state that the unstable releases always start with a .1 release | 10:47 |
| tristan | that would make `bst --version` report something 1.1-ish right away | 10:47 |
| *** valentind has quit IRC | 10:58 | |
| *** valentind has joined #buildstream | 10:58 | |
| *** valentind has quit IRC | 10:59 | |
| gitlab-br-bot | buildstream: issue #187 ("Incorrect overall status when failing to save a tracked reference") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/187 | 11:03 |
| tristan | jjardon[m], I can confirm that with master there is such a warning, I am seeing: | 11:09 |
| tristan | [--:--:--] WARNING Failed to fetch remote refs | 11:10 |
| tristan | However, since you raise it... | 11:10 |
| jjardon[m] | yeah | 11:10 |
| jjardon[m] | I meant a warning every time it tries but it didnt manage | 11:10 |
| tristan | ssam2, It would be nice if you could add the artifact URI to the above warning | 11:10 |
| jjardon[m] | I _think_ there is not a warning, I can be wrong | 11:10 |
| tristan | jjardon[m], there will not be | 11:10 |
| jjardon[m] | ok, thats why I didnt see it :) | 11:11 |
| tristan | jjardon[m], As I mentioned above, we dont waste time poking an artifact cache we already know we cannot reach, we establish that once at startup | 11:11 |
| tristan | if it *becomes* unreachable in mid build, then there will be noise | 11:11 |
| jjardon[m] | yeah, but It can happen that you reach it in the beginning but not later | 11:12 |
| jjardon[m] | ah, ok; my use case is cover then | 11:12 |
| tristan | jjardon[m], and in *that* case there should be noise | 11:12 |
| tristan | right | 11:12 |
| *** tiago has quit IRC | 11:24 | |
| *** tiago has joined #buildstream | 11:27 | |
| ssam2 | tristan, yeah, in my branch you get more detailed errors | 11:46 |
| jonathanmaw | tristan: Is there a task related to adding a "detail" field to ElementErrors and SourceErrors, or shall I do that? | 11:50 |
| tristan | jonathanmaw, I'm on it as we speak | 11:51 |
| tristan | jonathanmaw, for your branch, dont care about that detail for now... | 11:51 |
| tristan | jonathanmaw, how are the other changes coming along ? close ? | 11:52 |
| jonathanmaw | tristan: okay, I'll stick to the generic overlap reporting, and raise an error as normal | 11:52 |
| jonathanmaw | tristan: most of them are underway, assuming my approach of adding extra checking in various places to handle ElementErrors coming through seems appropriate to you | 11:53 |
| jonathanmaw | which turned out to just be a matter of checking in Main's handle_failure for the exception (where it seemed to make sense to report it with an error message, and leave it to prompt you again with what to do) | 11:55 |
| tristan | jonathanmaw, for that line, dont worry about it, I'm fixing up ElementError (and SourceError) to include a detail string basically, and flat out removing Plugin.error(), which makes no sense | 11:55 |
| tristan | jonathanmaw, the most important is to reverse the nature of the error of course, i.e. the error is "Ooops I stepped on something that exists" | 11:55 |
| tristan | Not "Oh damn some random thing overwrote me" | 11:55 |
| jonathanmaw | tristan: yep, that's in hand | 11:55 |
| jonathanmaw | I just need to make sure that the tests make sense now | 11:56 |
| tristan | And the other is to filter out the whitelist from warnings and errors equally | 11:56 |
| jonathanmaw | I'm not quite sure what was meant by filtering, but that's probably because I've slept since looking at the code | 11:58 |
| tristan | jonathanmaw, one new config you add turns the warnings into an error, the other new thing you add allows whitelisting the overlaps that are allowed on a per-element basis | 11:59 |
| tristan | jonathanmaw, your previous patch show the entire list of overlaps even if they are whitelisted; unless its treated as an error | 11:59 |
| tristan | jonathanmaw, the whitelist should effect the warning-or-error equally | 12:00 |
| tristan | As least, I think it's much more useful if it does | 12:00 |
| jonathanmaw | ah, okay. I hadn't noticed that. Good catch! | 12:00 |
| tristan | grrrr more bugs | 12:25 |
| tristan | bst --error-lines 50 ... now doesnt print any of the last lines of the log file when the log file contains < 50 lines | 12:26 |
| tristan | Probably ever since changing that to use this mmap thing instead of tail | 12:26 |
| tlater | Eep | 12:34 |
| * tlater was pretty sure he checked that case | 12:34 | |
| tlater | And had a test for it | 12:34 |
| tlater | But can be wrong | 12:35 |
| tristan | grrr | 12:38 |
| tristan | tlater, be20f411 | 12:38 |
| tristan | tlater, funny how the comments are all speaking of this mythical `end+1` that never occurs in the code | 12:38 |
| tristan | and then adding the +1, we actually see the log when we want to print more lines than those which exist | 12:39 |
| tristan | s/then/when | 12:39 |
| tlater | Hm, I think there was a reason why I didn't have that +1 | 12:40 |
| tristan | tlater, window is closing on remembering what it was... | 12:44 |
| tlater | Yeah, nevermind, it's possible it just got lost | 12:44 |
| tlater | I wonder where that unit test went | 12:44 |
| tristan | ok the +1 was needed, amended comments to make it more clear | 12:50 |
| tristan | If we search before the beginning of the file, it will be -1; making 0 the correct choice | 12:50 |
| tristan | If we find a newline, we want to discard it, hence +1 again is correct. | 12:50 |
| tlater | Yes, I do think the +1 is needed | 12:51 |
| * tlater is struggling to get pytest to run two independent test suites | 13:00 | |
| tristan | tlater, it need not be integrated into setup.py really | 13:04 |
| tristan | but, there may be a nice trick to do here... lemme see... | 13:04 |
| tlater | I think there's a plugin that can do it | 13:05 |
| tristan | http://blog.devork.be/2009/12/skipping-slow-test-by-default-in-pytest.html | 13:06 |
| tristan | tlater, something like the above can work | 13:06 |
| tristan | tlater, with that approach, we can make `integration` become a subdir of `tests`, and "mark" them | 13:06 |
| tlater | Right, we would have to mark each integration test though | 13:07 |
| tristan | tlater, then we could simplify our gitlab CI thing to just "run them all" (or a user could pass ./setup.py test --addopts '--slow' for example to run the whole thing) | 13:07 |
| tlater | Or can we do that per-module? | 13:07 |
| tristan | I dont know | 13:07 |
| * tlater has a look at that | 13:08 | |
| tristan | but, marking them wouldnt be horrible | 13:08 |
| tlater | I guess trying to keep the directories apart isn't necessary anymore anyway | 13:08 |
| tristan | tlater, that's just a suggestion; it seems like an elegant enough approach and maybe has the benefit of more people running the tests on their own laptop more often | 13:08 |
| tlater | It's much more elegant than what I was trying | 13:09 |
| tlater | I was attempting to extend pytest=runner | 13:09 |
| tristan | ensuring that we catch these problems that kept popping up, like contamination of the user's caches and such | 13:09 |
| *** xjuan has joined #buildstream | 13:09 | |
| tristan | and dirtying the tree, if a user/developer runs in to that; they should complain immediately haha | 13:10 |
| tlater | Yeah, this should make everything a lot nicer :) | 13:10 |
| tlater | As an aside, do you think we should still attempt any caching at all? | 13:10 |
| tlater | I still dislike how long it takes to fetch the sdk | 13:11 |
| tlater | So for now I keep a not-so-temporary cache dir | 13:11 |
| tlater | But it does mean that tests technically aren't always exactly equal | 13:11 |
| tristan | Is there a reason `bst push` and `bst pull` was added and does not print the summary at the end of the run, like all the other commands ? | 13:27 |
| tristan | ssam2, Also for your branch, note that I currently get "Not configured for pulling artifacts" message, when in fact I am, but connectivity is just not established | 13:31 |
| tristan | ssam2, It would be fine to just append " or unable to reach server" or such to the same message | 13:32 |
| *** adds68 has joined #buildstream | 13:42 | |
| gitlab-br-bot | buildstream: merge request (refactor-error-details->master: Refactor error details) #221 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/221 | 13:44 |
| *** tristan has quit IRC | 13:48 | |
| *** tristan has joined #buildstream | 14:14 | |
| tristan | jonathanmaw, https://gitlab.com/BuildStream/buildstream/merge_requests/221 is close to being merged, that will give you a new `detail` kwarg on ElementError() | 14:29 |
| tristan | jonathanmaw, also, Plugin.error() goes away, so for your branch, just rebase after that lands; and use only the ElementError() in the case that the overlaps should be errors | 14:30 |
| jonathanmaw | tristan: \o/ | 14:30 |
| tristan | eh... the unix integration test takes so much more time | 14:31 |
| adds68 | Sorry this may sound like a dumb question, but if i've built something with buildstream, how do i then push the thing i have built into a local ostree repo? | 15:01 |
| tristan | adds68, bst checkout | 15:02 |
| tristan | and do what you want with it | 15:02 |
| tristan | adds68, --hardlinks will make it fast, just dont open the files checked out for writing | 15:02 |
| adds68 | Then simply move it into the ostree repo? | 15:02 |
| tristan | if thats what you wanna do with the output | 15:03 |
| adds68 | tristan, what does --hardlinks do? | 15:03 |
| tristan | uses hardlinks | 15:04 |
| tristan | for now that's what we have, but has limitations in terms of what you can checkout, i.e. the files checked out will be owned by you | 15:04 |
| * persia is confused by the above, and suspects adds68 may be looking for `bst push` | 15:12 | |
| *** mcatanzaro has joined #buildstream | 15:13 | |
| adds68 | persia, that was where my confusion came from, but as i've heard, ostree isn't usually configured for pushing? | 15:13 |
| *** mcatanzaro has quit IRC | 15:15 | |
| *** mcatanzaro has joined #buildstream | 15:16 | |
| persia | Is a cache server not just an ostre server? | 15:17 |
| persia | I mean ostree + extra bits to deal with things like generating manifests | 15:17 |
| *** xjuan has quit IRC | 15:18 | |
| gitlab-br-bot | buildstream: merge request (refactor-error-details->master: Refactor error details) #221 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/221 | 15:24 |
| ssam2 | persia, `bst push` is for pushing buildstream artifacts to a buildstream artifact cache | 15:26 |
| ssam2 | those can be used by other instances of buildstream | 15:26 |
| ssam2 | nothing more | 15:26 |
| ssam2 | i believe adam is trying to solve a different problem than 'how to get my buildstream artifacts into a buildstream artifact cache' | 15:27 |
| persia | Ah. Interesting distinction. | 15:27 |
| ssam2 | flatpak happens to use ostree, in a way that is totally independent from buildstream | 15:27 |
| persia | Yes, I think adds68 is trying to solve "how to get buildstream artifacts into an ostree repo". | 15:27 |
| adds68 | ssam2, ah yes that makes it a lot clearer now, you're correct in your thinking | 15:28 |
| ssam2 | persia, "ostree repo" is not specific enough to be useful here; as there are multiple incompatible ways you can use an ostree repo | 15:29 |
| persia | adds68: What higher-level problem are you trying to solve? | 15:30 |
| persia | ssam2: Thanks for the clarification: I now understand miuch better why most of what I wrote above was unhelpful. | 15:30 |
| adds68 | persia, i have a local build of the freedesktop-sdk built, which i need to put into a local ostree repo, so i can point a local Flatpak app to the local build for testing | 15:31 |
| ssam2 | i would use `ostree commit` to do that | 15:31 |
| ssam2 | pointing at the files you checked out with `bst checkout` | 15:31 |
| persia | In which case, `bst checkout; ostree commit` is probably the right sequence. | 15:31 |
| adds68 | ssam2, oh! I didn't think of doing it that way around | 15:33 |
| adds68 | I shall give that a go, thanks persia ssam2 | 15:33 |
| tristan | adds68, you might look at the github for flatpak's freedesktop-sdk-base built from yocto for a decent CLI invocation of `ostree commit` | 15:37 |
| tristan | adds68, i.e. directly in the Makefile at the toplevel | 15:38 |
| lantw44 | I am trying to get BuildStream work on FreeBSD. I have created a element 'freebsd-base-system.bst' to import the base system with binary release tarballs download from https://download.freebsd.org, and I can run 'bst shell base/freebsd-base-system.bst' with some local patches to BuildStream now. What is the proper way to add packages to this system? | 15:43 |
| persia | lantw44: How do you mean "packages"? Additional elements? What result are you building? | 15:45 |
| lantw44 | persia: I want to install packages with the package manager provided by the system, but this will require network access. | 15:46 |
| lantw44 | The result is a FreeBSD image without any package. | 15:47 |
| * persia will have to defer to someone else: I would expect either a) the base system has enough that one can build things, and one builds the other things one wants using BuildStream or b) the base system doesn't have enough to build with buildstream, and should be extended before importing. | 15:48 | |
| ssam2 | there's no way to run a package manager inside a buildstream sandbox, deliberately | 15:48 |
| persia | I realise these are not necessarily realistic expectations. | 15:48 |
| ssam2 | we only want to wrap deterministic processes like compilations | 15:49 |
| ssam2 | package managers doing network access don't guarantee repeatable results | 15:49 |
| ssam2 | i'd suggest running the package manager in a chroot or something, then tarring up the result | 15:49 |
| lantw44 | so the only way to do it is to make a prebuilt system image and upload it to some places for other to download? | 15:49 |
| ssam2 | yeah | 15:49 |
| persia | One might be able to force it by creating elements where the "source" is the binary package and the "build" process runs the package manager in non-network mode to install it, but that sort of defeats the point. | 15:50 |
| ssam2 | we are interested in making that not a painful experience, though | 15:50 |
| ssam2 | as an example, https://gitlab.com/BuildStream/debootstrap-ostree/ is being used to generate the base that the gnome-modulesets repo uses | 15:50 |
| ssam2 | i've also proposed a way to import from the docker hub, mainly as that's a nice way around having to set up your own server | 15:51 |
| ssam2 | (https://gitlab.com/BuildStream/bst-external/merge_requests/9) | 15:51 |
| ssam2 | but it's not a goal of buildstream to wrap package managers, because it wouldn't gain anything over what the package manager already provides | 15:51 |
| lantw44 | Yes, so I think I have to make a similar script to make a base system image with more packages | 15:52 |
| lantw44 | Is it possible to have project options set based on the OS ? | 15:54 |
| lantw44 | I hope there is a way to write '- os == "linux"' or '- os == "freebsd"' in modulesets without manually setting this option | 15:55 |
| ssam2 | right now, there isn't | 15:55 |
| ssam2 | the only 'automatic' option is the `arch` type | 15:55 |
| ssam2 | i can see it would be useful in some cases | 15:56 |
| lantw44 | Yes, the jhbuild modulesets have many preset conditions which is useful in <if> tags | 15:57 |
| lantw44 | but it seems that currently the automatically converted BuildStream modulesets don't include these conditions | 15:59 |
| ssam2 | yeah | 16:00 |
| * tlater is running the new integration tests on CI for the first time \o/ | 16:00 | |
| ssam2 | certainly, adding the conditions into the gnome-modulesets project.conf is welcome | 16:00 |
| ssam2 | we can add code to buildstream to automatically fill some of those options later if needed | 16:00 |
| ssam2 | in the meantime you could do something like 'alias bstf="bst -o os freebsd" or something as a workaround | 16:01 |
| lantw44 | currently I have alias bst="sudo env HOME=$HOME BST_FORCE_BACKEND=unix ~/.local/bin/bst" | 16:02 |
| lantw44 | and I will try whether it is possible to have the default values of other options to depend on this 'os' option | 16:03 |
| ssam2 | i don't think it is | 16:03 |
| ssam2 | however, you can set variables in the project.conf based on the 'os' option | 16:03 |
| ssam2 | you just can't then allow overriding those variables on the commandline | 16:04 |
| lantw44 | jhbuild has a '--conditions' option that can be used to override default conditions on the command line | 16:05 |
| persia | From curiosity, what assumptions come with choice of "OS"? Does it need a more complex model? Is it kernel? C library? Compiler suite? | 16:05 |
| lantw44 | Can I use (?) to check variables? | 16:06 |
| lantw44 | persia: jhbuild uses kernel | 16:07 |
| lantw44 | It sets different default conditions for linux, freebsd, macos | 16:08 |
| * persia vaguely wonders if GNOME can be built with bionic | 16:08 | |
| ssam2 | lantw44, (?) only works for options | 16:09 |
| persia | I don't know about the conditionals used in jhbuild, but in another build system, I remember lots of changes being required to deal with differences between glibc and musl. | 16:09 |
| lantw44 | ssam2: so if I use variables instead of options, how can I add dependencies conditionally? | 16:10 |
| ssam2 | good point, not sure you can | 16:12 |
| ssam2 | but the option parsing system doesn't dependencies, at least not yet | 16:12 |
| lantw44 | ssam2: I saw a webkit bug report which said I broke the build on Debian GNU/kFreeBSD because I added some code using FreeBSD libc specific features in a #if OS(FREEBSD) block ... | 16:13 |
| ssam2 | i'm not sure how much complexity it would add if we did support that | 16:13 |
| persia | lantw44: That class of error is precisely why the idea of an "OS" switch worries me. Lots of environments end up mixing kernels, C libraries, and compilers (which triplet is often considered "OS"). | 16:14 |
| lantw44 | and even 'arch' can have different results on different OS | 16:15 |
| tristan | ssam2, "but the option parsing system doesn't dependencies, at least not yet" what does this mean ? | 16:15 |
| lantw44 | FreeBSD uses 'amd64' and Linux uses 'x86_64' | 16:15 |
| * tristan getting confused here, looks like everyone is talking about different things | 16:15 | |
| juergbi | speculative discussion ;) | 16:16 |
| ssam2 | tristan, i mean, you can't say "the default value of option C depends on the value of option A" | 16:16 |
| ssam2 | tristan, or is that in fact possible ? | 16:17 |
| tristan | lantw44, the name of the "machine arch" is what you choose it to be, everyone uses different names, this is known; buildstream's convenience is to automatically set the result from `uname`, but it's only convenience for not specifying one at invocation time. | 16:17 |
| tristan | There is the answer for one specific question | 16:17 |
| tristan | ssam2, no that is not possible, so A.) Yes an element can have conditional dependencies, set by the options... and B.) One option can not be automatically set by the value of another | 16:18 |
| tristan | The question I believe was "so if I use variables instead of options, how can I add dependencies conditionally?" | 16:19 |
| lantw44 | tristan, ok, so I should check for different name in different OS conditions | 16:19 |
| juergbi | (with recursive pipelines you can set subproject options based on main project options, though) | 16:19 |
| tristan | You cannot have conditional fragments via variables, options are for that | 16:19 |
| tristan | lantw44, with the project, you first decide your nomanclature | 16:20 |
| tristan | lantw44, if some host OS's report unames differently, oh well too bad, less convenient; people who have a uname that doesnt match the arch, must just specify `bst --option arch <the arch name desired> ... build` | 16:21 |
| lantw44 | tristan, I am trying to modify gnome-modulesets to support both Linux and FreeBSD, like what we did in jhbuild | 16:21 |
| tristan | That would be *awesome*, I'm not sure what the state of buildstream is in regarding that, it should be possible building as root with unix backend as you seem to be trying | 16:22 |
| tristan | it's not thoroughly tested on BSD but it should be | 16:22 |
| lantw44 | in jhbuild we have many preset 'conditions' (like BuildStream 'options') which can be used to conditionally include a block | 16:22 |
| tristan | Yes I'm aware | 16:23 |
| lantw44 | tristan, yes, I have get it work with some patches to the sandbox part | 16:23 |
| tristan | Nice :) | 16:23 |
| tristan | jjardon[m], do you know about the secret sauce to enable BSD runners on gitlab ? | 16:24 |
| tristan | hehehe | 16:24 |
| lantw44 | FreeBSD doesn't support rbind | 16:24 |
| jjardon[m] | tristan: let me check | 16:24 |
| tristan | juergbi, I just got it | 16:24 |
| tristan | juergbi, good joke :D | 16:24 |
| lantw44 | and I replace it with 'bind' (which is 'mount -t nullfs' on FreeBSD) | 16:25 |
| tlater | Ugh, that was a pain for Solaris/AIX as well | 16:25 |
| * tlater wonders if there's a way to do that in general that isn't messy | 16:26 | |
| tristan | lantw44, seeing as it's about 1:30am here... I think it would be really awesome if you could file a report with your patch; I.e. not necessarily a proper patch but just to let us know what it was you had to hack to get it to work | 16:26 |
| lantw44 | tristan, do you mean a merge request that cannot be merged? :) | 16:27 |
| tristan | lantw44, eh, could be that; could just be inline; I dont mind :) | 16:27 |
| tristan | I think you can inline it in comments with ``` | 16:27 |
| tristan | maybe even ```diff or ```patch will highlight things nicely | 16:28 |
| lantw44 | currently the only thing I tested on FreeBSD is base system importing | 16:28 |
| lantw44 | 'bst build' and 'bst shell' | 16:28 |
| tristan | that covers a lot | 16:28 |
| tristan | bst shell at least covers the parts you're not yet using from bst build (the sandbox) | 16:29 |
| tristan | I suppose we're not really sure about fuse yet | 16:29 |
| lantw44 | There is also some patches to gnome-modulesets to make ostree optional | 16:29 |
| tristan | but I suspect it would work, I think ssam2 fixed something in our local copy of fuse specifically for bsd ? (I dont know, I'm pretty foggy on that detail) | 16:30 |
| lantw44 | FreeBSD has fuse. It is a kernel module that have to be loaded by the user. | 16:30 |
| ssam2 | tristan, i fixed something for PPC64, nothing for BSD | 16:30 |
| tristan | ah | 16:30 |
| lantw44 | 'kldload fuse' and /dev/fuse appears | 16:30 |
| *** tristan has quit IRC | 16:33 | |
| *** tristan has joined #buildstream | 16:35 | |
| tristan | interestingly speculative discussion, also leaves you open to disclosing details unintentionally via side channels; the usual result is an amplification of confusion in the discussion, though | 16:39 |
| tristan | but, it could be interesting to use it as an attack vector, perhaps in debate class | 16:40 |
| persia | Yes. Contrapositives are part of the foundation of stout rhetoric. | 16:44 |
| persia | ("Were I to say $foo, how might I expect you to react?" type questions, allowing easy refutation of the claim that one might have said $foo) | 16:44 |
| tristan | persia, speculative discussion is rather when one person is mid-phrase, but the responder starts answering 3 or more answers at once | 16:46 |
| tristan | only one of them is intended to be committed to the discussion | 16:46 |
| tristan | But if observed through side channels, causes mass confusion | 16:46 |
| persia | Definitely orange, although I'd accept hot pink if forced. | 16:47 |
| tristan | However, could also be turned against the one optimizing their responses by inadvertently disclosing answers one might not have permission to give | 16:47 |
| tristan | For instance, "had the sentence granted me permission to answer with my password, I would have answered ******" | 16:47 |
| *** noisecell has quit IRC | 16:48 | |
| *** noisecell has joined #buildstream | 16:49 | |
| *** noisecell has quit IRC | 16:51 | |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: Multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 17:35 |
| *** jonathanmaw has quit IRC | 17:36 | |
| *** ssam2 has quit IRC | 18:11 | |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: Multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 18:18 |
| gitlab-br-bot | buildstream: merge request (modAndTest->master: Making changes to various documents:) #206 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/206 | 18:19 |
| gitlab-br-bot | buildstream: merge request (modAndTest->master: Making changes to various documents:) #206 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/206 | 18:20 |
| *** jude has quit IRC | 18:20 | |
| gitlab-br-bot | buildstream: merge request (modAndTest->master: Making changes to various documents:) #206 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/206 | 18:23 |
| gitlab-br-bot | buildstream: merge request (master->master: WIP: Documentation improvements) #183 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/183 | 19:28 |
| gitlab-br-bot | buildstream: merge request (show_sources->master: Added a flag to bst show) #222 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/222 | 19:31 |
| *** noisecell has joined #buildstream | 19:37 | |
| *** valentind has joined #buildstream | 19:44 | |
| gitlab-br-bot | buildstream: issue #141 ("Stack trace when aborting fetching of artifact list") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/141 | 19:55 |
| gitlab-br-bot | buildstream: issue #141 ("Stack trace when aborting fetching of artifact list") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/141 | 19:55 |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: Multiple remote cache support) #166 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 19:55 |
| juergbi | \o/ | 19:58 |
| tristan | :) | 20:14 |
| gitlab-br-bot | buildstream: issue #85 ("Support for pulling from multiple artifact caches") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/85 | 20:27 |
| *** luc14n0 has quit IRC | 21:41 | |
| *** luc14n0 has joined #buildstream | 21:41 | |
| *** tristan has quit IRC | 22:10 | |
| *** valentind has quit IRC | 22:30 | |
| *** jennis has quit IRC | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!