| *** Loko has joined #buildstream | 01:44 | |
| *** Loko has left #buildstream | 01:46 | |
| *** Prince781 has joined #buildstream | 04:21 | |
| *** Prince781 has quit IRC | 05:13 | |
| *** tristan has joined #buildstream | 06:29 | |
| *** mohan43u has quit IRC | 07:06 | |
| *** mohan43u has joined #buildstream | 07:09 | |
| *** dominic has joined #buildstream | 07:27 | |
| *** bethw has joined #buildstream | 07:30 | |
| *** jennis has quit IRC | 08:04 | |
| *** mohan43u has quit IRC | 08:05 | |
| *** jennis has joined #buildstream | 08:06 | |
| *** dominic has quit IRC | 08:07 | |
| *** mohan43u has joined #buildstream | 08:09 | |
| *** bochecha has joined #buildstream | 08:17 | |
| bochecha | hi, I'm looking at the freedesktop flatpak sdk 1.8 (the future one, made with BuildStream), and I have a question about it: what's this /usr/lib/x86_64-linux-gnu dir? | 08:23 |
|---|---|---|
| bochecha | was it a decision to use Debian-style multiarch in the runtime or did I do something wrong? | 08:23 |
| bochecha | and is that %{libdir} in the context of this project? or do I access it with %{libdir}/%{gcc_triplet} in element definitions? | 08:24 |
| tristan | I suspect %{libdir} plays a part in this, but from what I recall; gcc upstream does not support any way to force configure the gcc libs into /usr/lib | 08:26 |
| bochecha | there's much more than just the gcc libs in this dir though, it contains most of the runtime | 08:27 |
| tristan | bochecha, I think ssssam[m] knows the most on this as he wrote the last iteration of that runtime bootstrapping, if one can catch his attention... but what I think is pretty safe to say, is that the runtime is going to have libs in /usr/lib, with the exception of gcc stuff | 08:27 |
| tristan | bochecha, ah, if that is the case; it may have more to do with some of valentind's work | 08:28 |
| juergbi | yes, it seems to use Debian-style multiarch | 08:28 |
| tristan | bochecha, I think there was an initiative to allow multiarch; and some things (was it steam ?) required this | 08:28 |
| bochecha | ok | 08:29 |
| * tristan has to do a review of all of this freedesktop-sdk work, and ensure it's inline with what Alex wants, and am swamped with other things this week :-( | 08:30 | |
| bochecha | basically, I found some stuff that can be removed to make the platform quite smaller, and I'm wondering how to access those files in an element yaml | 08:30 |
| tristan | bochecha, https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/sdk/project.conf#L23 | 08:31 |
| tristan | bochecha, saying that "lib" is "lib/%{gcc_triplet}", will mean that %{libdir} is automatically "%{prefix}/%{lib}" | 08:31 |
| bochecha | since the element is quite early in the pipeline, I don't really want to go trial-and-errors on this, I'd rather ask if someone knows what's the right variable ^_^ | 08:32 |
| bochecha | tristan: ah, so %{prefix}/%{lib} then :) | 08:32 |
| bochecha | oh, that's what libdir is defined to? | 08:32 |
| tristan | exactly, %{libdir} is automatically that | 08:32 |
| bochecha | I kind of want a `bst eval` command, which would just expand variables to their value in a given project | 08:32 |
| tristan | bochecha, `bst show --format "%{variables}"` | 08:32 |
| tristan | or %{vars}, forget which one | 08:32 |
| bochecha | grmbl | 08:33 |
| tristan | on an element basis of course, because this can vary per element depending on various overrides | 08:33 |
| bochecha | every single time I want something out of buildstream, I'm told it's already there… why don't you let me have good ideas first?!?! :P | 08:33 |
| tristan | hehehe | 08:33 |
| bochecha | bst show --format "%{vars}" base/python2.bst | 08:34 |
| bochecha | it was vars, not variables :) | 08:34 |
| tristan | yeah, it's in the help output, but we need to improve that, maybe move it to docs | 08:35 |
| tristan | (the formatting of that help output is a disaster when automatically rendered as html or man pages) | 08:35 |
| jennis | mhmm tlater it looks like `ostree refs --delete` doesn't actually free up any disk space | 08:40 |
| jennis | Does anyone know if/how ostree can be used for such a thing? | 08:41 |
| bochecha | jennis: in my experience, it only deletes the refs, you need to also prune the repo | 08:41 |
| bochecha | `ostree --repo=... prune --refs-only | 08:42 |
| bochecha | that should remove the objects that were pointed to by the refs you just removed | 08:42 |
| jennis | Ahh yes bochecha, it definitely seems that way: http://termbin.com/3omv | 08:45 |
| jennis | thank you | 08:45 |
| jennis | bochecha: that's worked thanks. However, what i'm trying to do, for now, is find ostree's equivalent of `rm -rf`, once this is done, i'll look into removing least-recently-used | 08:47 |
| jennis | Do you have any advice on this? | 08:48 |
| tlater | jennis: You've already got all the ingredients for a LRU implementation, no? | 08:48 |
| bochecha | well, you don't `rm -fr` in ostree | 08:48 |
| bochecha | just like you wouldn't in git | 08:48 |
| bochecha | you remove a ref, then gc/prune the repo | 08:49 |
| jennis | ahh makes sense | 08:49 |
| jennis | tlater, yes, but i'd like to get both working so we could potentially implement an option | 08:50 |
| * tlater doesn't think an "rm -rf" is particularly useful, and if it's going to be more difficult to implement, why spend the time? | 08:50 | |
| * juergbi agrees | 08:51 | |
| juergbi | if a minimally better approach takes too long to implement, we could use rm -rf as stop gap, but I don't think that's necessary | 08:52 |
| tlater | jennis: In either case, as a side note, I've just finished implementing "remove" and "list_refs" methods for our ostree helper module: https://gitlab.com/BuildStream/buildstream/blob/2e51376b69ba53648d68ffdd39b6b97712a9c3c4/buildstream/_ostree.py | 08:52 |
| tlater | You can probably reuse that quite nicely to implement cache expiry on a remote host | 08:53 |
| tlater | An "rm -rf" would be as simple as iterating over what list_refs() returns and remove()-ing all of it | 08:53 |
| jennis | right thanks tlater | 08:54 |
| * tlater still has to think about how to do prune more efficiently, but we can sort that out a bit later | 08:54 | |
| tristan | just keep in mind that buildstream doesnt really use branches; one ref is a project-name, element name and cache key; meaning you essentially have only one commit per ostree ref | 08:59 |
| *** dominic has joined #buildstream | 08:59 | |
| *** aday has joined #buildstream | 09:16 | |
| tlater | tristan: About the local artifact cache expiry - for the quota specification I'm considering using this package: https://pypi.python.org/pypi/humanfriendly/4.9 | 09:23 |
| tlater | It seems like a bit of a pain to try and write our own implementation of a unit parser | 09:23 |
| tlater | And that one seems really neat | 09:23 |
| tristan | Eh ? | 09:29 |
| tristan | tlater, I recall this was extremely easy to check with `df` in the flatpak-build-scripts shell scripts | 09:29 |
| tristan | tlater, probably we want to use something pythonic and not call out to df... but... | 09:30 |
| * tristan looks at that thingy | 09:30 | |
| tlater | tristan: does `df` convert 20000 -> 20KB? | 09:30 |
| tristan | A variant of it does | 09:30 |
| tlater | Huh, that's new to me, I need to read df's man at some point | 09:31 |
| tristan | tlater, ah sorry, we use `stat` | 09:32 |
| tlater | Right, I wonder if python can do that too | 09:32 |
| tristan | see: https://github.com/flatpak/flatpak-build-scripts/blob/master/include/build-source.sh#L77 | 09:32 |
| tlater | ty, will try that first then :) | 09:32 |
| tristan | then again, I think what we're doing there is different | 09:32 |
| tristan | tlater, again, we probably want something pythonic and not to call out to the shell, that is preferable | 09:33 |
| tlater | tristan: Yeah, but python does have os.stat | 09:33 |
| tlater | Or something like that | 09:33 |
| tristan | but actually checking the volume should be doable with standard libs, the dep you want to add is about human parsable thingies | 09:33 |
| tlater | Ah, right, yeah | 09:34 |
| tristan | tlater, one thing I think I should make clear... there is no such thing as a "quota for ~/.cache/buildstream" | 09:34 |
| tlater | I'm talking about having something like "cache-size: 20GB" in userconf.yaml | 09:34 |
| tristan | yeah I understand that | 09:34 |
| tristan | tlater, clarification on the above, BuildStream only has configurable directories where sources, artifacts, and builds are performed; and it is *recommended* that artifacts and build directories remain on the same disk | 09:35 |
| tristan | otherwise, things become reaaaaaly slow, but the same is not true for sources | 09:35 |
| dominic | tristan, what is it exactly I'll be adding to --info-fd in bubblewrap to separate these errors? Looking at the code it just currently shows the child PID it gets from os.fork() | 09:46 |
| tristan | right, issue https://gitlab.com/BuildStream/buildstream/issues/286 | 09:47 |
| dominic | yeah, sorry forgot to mention that :P | 09:47 |
| tristan | dominic, you have looked at the upstream report here: https://github.com/projectatomic/bubblewrap/issues/257 ? | 09:47 |
| dominic | yeah | 09:47 |
| tristan | Notice Colin says "right now it just has the child PID, but we could easily extend it to include precise details like this." | 09:48 |
| tristan | Which gives us some leeway to do it how we want, or, maybe we'll have a small back-and-forth depending | 09:48 |
| tristan | dominic, The goal here is to distinguish between bwrap errors, and program errors | 09:49 |
| tristan | One approach might be to encode both, but I think that encoding only the child exit code is reasonable | 09:49 |
| tristan | Ummm, maybe not | 09:49 |
| tristan | I was thinking maybe we could rely on bwrap's exit code to distinguish, but it will *still* report an indistinguishable failure either way | 09:50 |
| tristan | dominic, First, have you deduced the format of the file it creates ? | 09:51 |
| dominic | that's what I originally thought I would be doing before I realised | 09:51 |
| tristan | ini style key values ? | 09:51 |
| tristan | dominic, another thing to consider is; "Is it, or can it be possible, that bwrap ever fails by itself if child process is actually launched ?" | 09:52 |
| tristan | maybe, maybe not | 09:53 |
| dominic | right | 09:53 |
| tristan | dominic, if the format of the info fd thing is like, GKeyFile for instance... | 09:53 |
| tristan | then we can just put the exit code there *if the child exited* | 09:53 |
| tristan | dominic, I would suspect that also, the PID only gets into that file, *if the PID was ever created* | 09:54 |
| tristan | As such, we can parse the result and easily deduce if the child exited with an error | 09:54 |
| tristan | if there was no exit code to be spoken for, then bwrap had a problem | 09:55 |
| dominic | OK, I'll have a look | 09:55 |
| tristan | Seems sensible, as it keeps the info-fd thing describing only information about the child process | 09:55 |
| ssssam[m] | bochecha_: there is a channel dedicated to the freedesktop SDK 1.8, it's #freedesktop-sdk in freenode | 09:55 |
| ssssam[m] | from what i understand, yeah it has debian-style multilib setup | 09:55 |
| ssssam[m] | multiarch even | 09:55 |
| tristan | dominic, you might consider also, by seeing how bwrap handles the exit; encoding the SIGCHLD info instead; i.e. you might only get an exit code if the process exited | 09:56 |
| tristan | dominic, you dont really get an exit code for a SIGSEGV or SIGKILL | 09:57 |
| tristan | dominic, I suppose it makes sense to encode as much as possible about how a process terminated | 09:57 |
| * dominic nods | 09:58 | |
| tristan | then again, if bwrap just blocks on the command and doesnt call waitpid(), it may makes sense to do something simpler and more inline with the bwrap codebase | 09:58 |
| tristan | dominic, if you find that the format of the info-fd is not extensible (i.e. is only a number written to a file, and risks breaking other users if we change it), then please raise a flag immediately | 10:05 |
| tristan | and I'll try to sort out what to do about that | 10:05 |
| dominic | will do, thanks tristan :) | 10:06 |
| bochecha | ssam2: thanks, I joined it and will continue there :) | 10:14 |
| jjardon[m] | Hi, can someone review these two trivial MR's: https://gitlab.com/BuildStream/buildstream/merge_requests/343 and https://gitlab.com/BuildStream/buildstream/merge_requests/344 | 10:31 |
| jmac | jjardon[m]: I've given both a thumbs-up | 10:35 |
| jjardon[m] | thanks! | 10:36 |
| *** bethw has quit IRC | 10:59 | |
| *** aday has quit IRC | 11:30 | |
| *** aday has joined #buildstream | 11:31 | |
| *** tristan has quit IRC | 11:46 | |
| *** tristan has joined #buildstream | 11:53 | |
| tlater | Another weird ostree case... I have deleted a ref, its object and pruned the full cache. | 13:26 |
| tlater | Yet the space doesn't appear to be freed up | 13:26 |
| tlater | The ref does not show up in the listed artifacts anymore | 13:26 |
| tlater | fsck doesn't show anything useful, either :| | 13:26 |
| tlater | Hm, even when manually deleting all other commits and pruning it doesn't free up any further | 13:37 |
| tlater | And I can see objects left in the objects/ dir | 13:37 |
| tlater | All empty directories though... | 13:37 |
| juergbi | tlater: do you still have the files in extract/ or build/? | 13:49 |
| juergbi | (as hardlinks) | 13:49 |
| tlater | juergbi: Would those still be picked up if I used du -hb in the ostree directory? | 13:49 |
| juergbi | no, if you only check the ostree directory | 13:50 |
| tlater | :| | 13:50 |
| juergbi | tlater: with du it should be easy to track down where the space is being used, though | 13:50 |
| juergbi | drill down into subdirectories? | 13:50 |
| tlater | It appears to be taken up entirely by empty directories | 13:50 |
| juergbi | how much space are we talking about? | 13:51 |
| tlater | I'm only working with 200KB of data, so I guess those might affect my measurements | 13:51 |
| tlater | It's ~100KB of directories | 13:51 |
| juergbi | directories could take that much space | 13:51 |
| juergbi | depends on the filesystem | 13:51 |
| juergbi | in any case, should not be an issue | 13:51 |
| tlater | I wonder if we should ignore directories in the disk size measurements | 13:51 |
| tlater | The issue is that it's a bit hard to tell what's just leftover directories and what's actually a file | 13:52 |
| juergbi | can you try du --apparent-size | 13:52 |
| * tlater thinks -b implies that? | 13:52 | |
| juergbi | ah right | 13:52 |
| juergbi | for real world quota, the empty directory disk usage shouldn't be relevant | 13:53 |
| juergbi | so I don't think anything special needs to be done here | 13:54 |
| juergbi | or do you have a specific concern with real world quota? | 13:54 |
| tlater | Not really, no | 13:54 |
| tlater | It just screws with my tests x) | 13:54 |
| tlater | Mayhap I should go bigger with the files I'm testing on | 13:54 |
| tlater | juergbi: Yeah, if I set my file sizes to a few MB everything works fine | 14:04 |
| * tlater is sufficiently annoyed by the inaccuracy of his size measurements | 14:04 | |
| *** ernestask has joined #buildstream | 14:07 | |
| tlater | Hmm, maybe the fact that I'm creating perfectly compressible artifacts has something to do with this | 14:15 |
| tlater | Yeah, that makes a lot more sense | 14:17 |
| * tlater stops writing files that consist only of '0' | 14:17 | |
| *** bethw has joined #buildstream | 14:22 | |
| tristan | tlater, deduplication of identical files ! | 14:48 |
| tristan | doh! | 14:48 |
| tlater | tristan: Oh, heh, I didn't even think that far o\ | 14:57 |
| jmac | I'm trying to set up my own artifact cache server. bst build reports "WARNING Failed to fetch remote refs from ssh://artifacts@selectron.dyn.ducie.codethink.co.uk/artifacts: Could not connect: Connection refused". But I can ssh to artifacts@selectron.dyn.ducie.codethink.co.uk without any problems. | 15:32 |
| jmac | Any idea what I should do next to debug it? | 15:33 |
| juergbi | jmac: hm, maybe add -v to the ssh_commandline in pushreceive.py? | 15:39 |
| jmac | Good plan | 15:41 |
| juergbi | we have a timeout of 3 seconds. probably not the issue | 15:41 |
| juergbi | fyi: it's initialize_push_connection() that fails | 15:43 |
| jmac | Right, I think it's the pull-url returned by ssh that's failing, not the ssh connection | 15:52 |
| jmac | Fixed now, might have a look into improving that error message | 15:56 |
| tlater | Is there a case in which we don't extract() a dependency element's artifact before we build? | 15:59 |
| * tlater thought this always had to happen | 16:00 | |
| juergbi | tlater: I can't think of a case where we don't need it | 16:05 |
| juergbi | anything that is staged needs to be extracted | 16:05 |
| tlater | Hm, odd, a raise Exception() in ostreecache.extract doesn't interrupt my build, even though I do have an element with a dependency in the pipeline | 16:06 |
| * tlater is trying to assert the order in which artifacts are expired | 16:06 | |
| juergbi | tlater: you raise it before the early returns, I assume? | 16:07 |
| tlater | First thing in the method | 16:07 |
| tlater | Hm | 16:07 |
| tlater | Ah, wait | 16:07 |
| tlater | Import elements probably don't call that | 16:08 |
| tlater | o\ | 16:08 |
| juergbi | import elements normally have no dependencies, so no | 16:08 |
| *** noisecell has quit IRC | 16:25 | |
| *** dominic has quit IRC | 16:48 | |
| *** tristan has quit IRC | 17:01 | |
| *** mohan43u has quit IRC | 17:05 | |
| *** mohan43u has joined #buildstream | 17:08 | |
| *** mohan43u has quit IRC | 17:11 | |
| *** mohan43u has joined #buildstream | 17:15 | |
| *** bethw has quit IRC | 17:20 | |
| *** mohan43u has quit IRC | 18:06 | |
| *** mohan43u has joined #buildstream | 18:09 | |
| *** Prince781 has joined #buildstream | 18:32 | |
| *** Prince781 has quit IRC | 19:00 | |
| *** mohan43u has quit IRC | 19:06 | |
| *** mohan43u has joined #buildstream | 19:10 | |
| *** ernestask has quit IRC | 19:52 | |
| *** mohan43u has quit IRC | 20:59 | |
| *** mohan43u has joined #buildstream | 21:02 | |
| *** aday has quit IRC | 21:20 | |
| *** Prince781 has joined #buildstream | 21:48 | |
| *** mohan43u has quit IRC | 21:49 | |
| *** mohan43u has joined #buildstream | 21:53 | |
| *** mohan43u has quit IRC | 21:59 | |
| *** mohan43u has joined #buildstream | 22:02 | |
| *** Prince781 has quit IRC | 22:04 | |
| *** Prince781 has joined #buildstream | 22:42 | |
| *** Prince781 has quit IRC | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!