| *** zoli__ has quit IRC | 00:18 | |
| *** jeak_ has joined #baserock | 02:43 | |
| *** paulw has quit IRC | 02:53 | |
| *** paulsherwood has joined #baserock | 02:53 | |
| *** zoli__ has joined #baserock | 04:31 | |
| *** zoli__ has quit IRC | 04:43 | |
| *** zoli__ has joined #baserock | 04:52 | |
| *** zoli__ has quit IRC | 06:13 | |
| *** straycat has quit IRC | 06:35 | |
| *** straycat has joined #baserock | 06:36 | |
| *** zoli__ has joined #baserock | 06:41 | |
| *** gary_perkins has joined #baserock | 07:16 | |
| *** perryl has joined #baserock | 07:24 | |
| *** gary_perkins has quit IRC | 07:32 | |
| *** mariaderidder has joined #baserock | 07:35 | |
| *** mdunford has joined #baserock | 07:48 | |
| *** SotK has joined #baserock | 07:53 | |
| *** mdunford has quit IRC | 07:56 | |
| *** benbrown_ has joined #baserock | 07:56 | |
| *** mariaderidder has quit IRC | 07:57 | |
| *** benbrown_ has joined #baserock | 07:57 | |
| *** mariaderidder has joined #baserock | 07:57 | |
| *** benbrown_ has quit IRC | 08:06 | |
| *** benbrown_ has joined #baserock | 08:06 | |
| *** tiagogomes_ has quit IRC | 08:19 | |
| *** tiagogomes has joined #baserock | 08:20 | |
| *** Stanto has quit IRC | 08:29 | |
| *** gary_perkins has joined #baserock | 08:29 | |
| tiagogomes | can anyone try to build the openstack system? I am getting a build failure although it is working for franred | 08:30 |
|---|---|---|
| *** Stanto has joined #baserock | 08:32 | |
| *** franred has quit IRC | 08:34 | |
| *** franred has joined #baserock | 08:34 | |
| franred | tiagogomes, do you still having troubles with conntrack-tools package? | 08:37 |
| tiagogomes | franred, yep | 08:37 |
| franred | if this is the case, this package only lives in my branch, no in master definitions | 08:37 |
| *** paulw has joined #baserock | 08:38 | |
| franred | tiagogomes, I can try to remove my cache artifacts and try to rebuild it | 08:39 |
| *** jonathanmaw has joined #baserock | 08:39 | |
| tiagogomes | franred ok, I managed to compile it | 08:44 |
| franred | tiagogomes, it builds for me | 08:44 |
| franred | tiagogomes, what was the problem? | 08:44 |
| tiagogomes | it didn't handle well parallelism in the Makefiles | 08:47 |
| *** kejiahu has joined #baserock | 08:52 | |
| tiagogomes | franred you will need to add `max-jobs: 1` to conntrack-tools | 08:53 |
| *** CTtpollard has quit IRC | 08:53 | |
| *** CTtpollard has joined #baserock | 08:53 | |
| franred | umm, I didn't have to do it | 08:53 |
| franred | tiagogomes, ^^ | 08:53 |
| *** mariaderidder has quit IRC | 08:54 | |
| franred | and I've compiled 3 times | 08:54 |
| tiagogomes | franred that's because the machine that you used to build conntrack-tools has less cores than mine (mine has 8) | 08:54 |
| franred | tiagogomes, probably...and madness... | 08:55 |
| franred | tiagogomes, could you explain why in machines with more cores this would fail? | 08:56 |
| tiagogomes | erm, badly written Makefiles? | 08:57 |
| *** edcragg has joined #baserock | 08:57 | |
| * franred amends 'max-jobs: 1' | 08:57 | |
| *** pdar has joined #baserock | 08:58 | |
| *** mariaderidder has joined #baserock | 09:00 | |
| *** jmacs has joined #baserock | 09:02 | |
| SotK | does anyone have time to have a look at my branch for moving writeexts.py into definitions? | 09:03 |
| SotK | branch is here: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/log/?h=baserock/adamcoldrick/remove-dependencies-v3 | 09:04 |
| SotK | discussion of that version is on the mailing list, starting here: http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-June/013101.html | 09:05 |
| *** Zara has joined #baserock | 09:06 | |
| *** bashrc has joined #baserock | 09:07 | |
| *** rjek has quit IRC | 09:08 | |
| *** rjek has joined #baserock | 09:08 | |
| *** bashrc has quit IRC | 09:08 | |
| *** lachlanmackenzie has joined #baserock | 09:09 | |
| *** ssam2 has joined #baserock | 09:11 | |
| *** ChanServ sets mode: +v ssam2 | 09:11 | |
| *** bashrc has joined #baserock | 09:14 | |
| *** Zara has quit IRC | 09:14 | |
| *** Zara has joined #baserock | 09:15 | |
| *** bashrc has quit IRC | 09:15 | |
| straycat | richard_maw, i have updated https://gerrit.baserock.org/#/c/878/ | 09:17 |
| richard_maw | straycat: aye, that's grand. | 09:17 |
| richard_maw | I'll +1 so you can merge it when its dependents are ready. | 09:18 |
| richard_maw | although… it's not actually dependent on the workspace command removal | 09:18 |
| straycat | well it's based on https://gerrit.baserock.org/#/c/870/2 where i remove all the workspace yarns | 09:19 |
| richard_maw | ah, so there may be common code that you depend on, or in the very least, some merge conflicts | 09:20 |
| straycat | i've got changes in flight to fix yarns for remaining use cases to use definitions repo rather than workspaces, but those will take a little more time | 09:20 |
| richard_maw | gotcha | 09:20 |
| straycat | i could rework that change to be based on current master if you prefer? no one's commented on 870 yet | 09:21 |
| *** gary_perkins has quit IRC | 09:23 | |
| *** paulw has joined #baserock | 09:23 | |
| *** gary_perkins has joined #baserock | 09:23 | |
| *** paulw has quit IRC | 09:23 | |
| richard_maw | I'd generally prefer the work for making the yarns use the new workflow happen before the patches to remove the old workflow | 09:24 |
| richard_maw | as I'd more happily +1 those than the ones to remove the workflow code | 09:24 |
| *** bashrc has joined #baserock | 09:26 | |
| * straycat nods | 09:26 | |
| *** tiagogomes has quit IRC | 09:29 | |
| *** tiagogomes_ has joined #baserock | 09:29 | |
| *** bashrc has quit IRC | 09:30 | |
| *** bashrc has joined #baserock | 09:31 | |
| bashrc | looks like IRC is borked. "dh key too small" | 09:32 |
| jonathanmaw | bashrc -> #codethink on freenode | 09:32 |
| radiofree | please be aware this is a public channel... | 09:33 |
| * rjek facepalms bashrc | 09:33 | |
| richard_maw | rjek: I believe that's classed as assault | 09:33 |
| ssam2 | jmacs: just to double check, is it safe for me to destroy your instance in the public cloud now? | 09:36 |
| jmacs | ssam2: Yes | 09:36 |
| ssam2 | ok, will do | 09:37 |
| straycat | okay functools.wraps is new to me, cool | 09:39 |
| richard_maw | straycat: aye, it's not massively useful for anything but decorator functions to keep the docstrings for the decorated function | 09:40 |
| richard_maw | but for those it is the most useful tool for the job | 09:41 |
| *** bashrc has quit IRC | 09:44 | |
| *** bashrc has joined #baserock | 09:44 | |
| richard_maw | I may have to throw the memoise away though, it's not sufficient for the resolve_ref call, since it needs to cache on some partial value of the result | 09:45 |
| *** CTtpollard has quit IRC | 09:45 | |
| *** CTtpollard has joined #baserock | 09:45 | |
| richard_maw | and monkeypatching the resolve_ref_to_tree call isn't sufficient either, as that is from the CachedRepo object, which gets returned to the CachingRepoCache internally | 09:46 |
| *** tiagogomes_ has quit IRC | 09:47 | |
| *** tiagogomes has joined #baserock | 09:47 | |
| *** mwilliams_ct has left #baserock | 09:48 | |
| ssam2 | all my artifacts built with YBD last night have repo: None and ref: None in their metadata | 09:50 |
| ssam2 | this worked the night before. :( | 09:50 |
| ssam2 | and nothing seems to have changed in ybd.git | 09:50 |
| ssam2 | oh, it's not all of them, just some of them | 09:51 |
| *** franred has quit IRC | 09:51 | |
| ssam2 | oh, in fact it's strata that have this | 09:51 |
| *** franred has joined #baserock | 09:51 | |
| *** sebh has quit IRC | 09:53 | |
| *** sebh has joined #baserock | 09:53 | |
| *** tiagogomes has quit IRC | 09:54 | |
| *** tiagogomes has joined #baserock | 09:54 | |
| jmacs | When I visit http://git.baserock.org/cgi-bin/cgit.cgi/delta/gcc-tarball.git/, it takes ages to generate the list of most recent commits when all I want is the links to clone it at the bottom | 09:59 |
| Kinnison | jmacs: Hmm | 09:59 |
| * Kinnison wonders if we should look to integrate a newer cgit | 09:59 | |
| jmacs | Well, if it were just for me I could put those links at the top instead :) | 10:00 |
| *** mariaderidder has quit IRC | 10:02 | |
| *** mariaderidder has joined #baserock | 10:02 | |
| *** pdar has quit IRC | 10:04 | |
| *** paulw has joined #baserock | 10:06 | |
| *** mwilliams_ct has joined #baserock | 10:18 | |
| *** pdar has joined #baserock | 10:20 | |
| *** bashrc has quit IRC | 10:32 | |
| ssam2 | perryl, tlsa, SotK: seems that with GNU gzip, 'gzip --no-name' stops gzip including the mtime and original filename in the .gz file | 10:32 |
| ssam2 | busybox gzip doesn't support that though. | 10:32 |
| ssam2 | maybe we could add it, it shouldn't be too hard | 10:33 |
| perryl | ssam2: sounds deceptively simple :) | 10:33 |
| richard_maw | gzip from stdin and call gzip with faketime? | 10:34 |
| straycat | richard_maw, i was about to say if that text isn't being used by the tests then i think we should just remove it | 10:38 |
| *** chibuzor_ has joined #baserock | 10:40 | |
| jmacs | Does anyone know where mpfr/gmp/mpc get built for gcc? I can't find them in definitions. | 10:41 |
| jmacs | s/built/referenced | 10:43 |
| *** bashrc has joined #baserock | 10:43 | |
| SotK | iirc they are in the gcc-tarball repo | 10:44 |
| * SotK may be completely wrong though :) | 10:44 | |
| jmacs | I did a git-clone recursive... they're not there | 10:44 |
| tiagogomes | jmacs I think I imported them to the GCC repo and committed. But things may have changed since them... | 10:45 |
| SotK | jmacs: are you looking at the right ref? | 10:45 |
| SotK | they appear to be there at 'baserock/build-essential' | 10:46 |
| jmacs | Apparently not, I was looking in master | 10:47 |
| ssam2 | perryl: I moved the http://wiki.baserock.org/projects/deterministic-builds/build-system-x86_64-non-reproducible/ page to the deterministic-builds subfolder | 10:56 |
| jmacs | I guess because mpfr/mpc/gmp all come from tarballs there isn't much better we can do than that | 10:58 |
| perryl | ssam2: thanks, sorry, been distracted fixing up a shell script i'll soon be adding to that page :) | 11:07 |
| *** mariaderidder has quit IRC | 11:07 | |
| *** mariaderidder has joined #baserock | 11:08 | |
| *** mariaderidder has quit IRC | 11:17 | |
| ssam2 | seems that when YBD creates a tar file, it doesn't sort the files inserted, so the entire artifact is non-deterministic | 11:21 |
| *** mariaderidder has joined #baserock | 11:21 | |
| ssam2 | that should be pretty easy to fix though. i'll look at doing a patch | 11:21 |
| ssam2 | I've set up a continous builder at http://185.98.148.127:8080/ which has now identified that every single chunk is unreproducible :) but I think that's due to the tarfile issue, as we already investigated the contents of all artifacts in a build-system and some of them were identical | 11:23 |
| *** zoli__ has quit IRC | 11:29 | |
| *** mariaderidder has quit IRC | 11:38 | |
| *** paulsher1ood has joined #baserock | 11:41 | |
| * paulsher1ood wonders if his irc is actually working | 11:48 | |
| ssam2 | i can see you | 11:48 |
| *** zoli__ has joined #baserock | 11:49 | |
| paulsher1ood | :) | 11:49 |
| paulsher1ood | i see your new server, ssam2 - how is that working? ie what is powering it? | 11:49 |
| *** pacon has joined #baserock | 11:50 | |
| straycat | richard_maw, to be fair for the most part the yarns i removed are for commands we don't need | 11:51 |
| straycat | probably i should sort out something to cover temporary build branches though | 11:51 |
| richard_maw | straycat: that's fine, I'd just prefer the removal of the yarns before the removal of the commands | 11:52 |
| ssam2 | paulsher1ood: http://185.98.148.127:8080/ ? it's a VM at DataCentred | 11:53 |
| ssam2 | the web UI is a bunch of stuff I added to morph-cache-server, in a branch | 11:53 |
| straycat | richard_maw, i'm confused now, that's exactly what i did? | 11:53 |
| ssam2 | it's in morph-cache-server.git, which might not be the right place, but works for now | 11:53 |
| * straycat is a cat of little brain today :( | 11:53 | |
| paulsher1ood | ssam2: ok - how are you 'scheduling' the build process? | 11:53 |
| ssam2 | paulsher1ood: more simple hacks: https://github.com/ssssam/ybd/tree/sam/continuous-hack/continuous | 11:54 |
| ssam2 | it's basically: while True: run ybd, submit artifacts to cache, move them into a directory | 11:54 |
| paulsher1ood | ssam2: oh, ok. so not gnome continuous, then? | 11:54 |
| ssam2 | heh, no | 11:54 |
| ssam2 | gnome-continuous' build system is written in Javascript, it's a bit unweildly (although it functions quite similar to ybd/morph in some ways) | 11:55 |
| paulsher1ood | does your list http://185.98.148.127:8080/artifacts imply there's another list of Matching builds? | 11:56 |
| ssam2 | that's the only list | 11:57 |
| ssam2 | none of them match right now :( I think that's due to changes in file ordering in the tarfile, though | 11:57 |
| ssam2 | however, to fix that in YBD now, we need need to either monkeypatch shutil.make_archive(), or reimplement it a bit | 11:58 |
| ssam2 | it's a one line patch to the shutil module to fix this, I think, but I imagine you don't want YBD to depend on a patched version of latest Python3 .. | 11:58 |
| paulsher1ood | correct :) | 11:59 |
| straycat | richard_maw, ? | 11:59 |
| paulsher1ood | ssam2: i'd be happy if such a patch were offered upstream of course :) | 12:00 |
| richard_maw | straycat: From the dependency ordering of the series I saw, you had some patches to the yarn suite after you'd removed commands for other bits of the yarn suite. I'd prefer all the patches that alter yarns to come before all patches that removed commands from morph | 12:00 |
| SotK | can you not construct the tarfile using the `tarfile` module, or would that have the same issue? | 12:01 |
| paulsher1ood | (the shutil fix, as well as any improvement to ybd) | 12:01 |
| richard_maw | straycat: if that's what's actually happening, I'm sorry about wasting your time with misinformed complaints, and I'm sorry I didn't make myself clearer earlier so I could've been told I'm wrong earlier. | 12:01 |
| straycat | richard_maw, right, ok | 12:01 |
| ssam2 | SotK: yes, using tarfile and sorting order we add the files fixes the issue | 12:02 |
| richard_maw | it's just more code, and paulsher1ood hates code | 12:02 |
| ssam2 | paulsher1ood: i'm trying to submit a patch to python, but I need to work out how to run the Python testsuite before submitting the patch, and so far I got a random error | 12:02 |
| paulsher1ood | ssam2: ain't it always the way? :) | 12:03 |
| ssam2 | ah! it works. | 12:03 |
| Kinnison | s | 12:11 |
| *** paulsherwood has quit IRC | 12:14 | |
| *** bjdooks has quit IRC | 12:20 | |
| ssam2 | bah, Python's tarfile module also unconditionally puts a timestamp in the .gz header | 12:28 |
| ssam2 | perhaps the easiest thing to do here is monkeypatch time.time and os.listdir ;) | 12:29 |
| Kinnison | Are you trying to make the artifact bit-for-bit rather than the content of the artifact? | 12:31 |
| ssam2 | indeed | 12:32 |
| Kinnison | why? | 12:32 |
| ssam2 | in order to be able to measure whether the content is bit for bit more easily | 12:32 |
| Kinnison | hrf | 12:32 |
| ssam2 | sort of defeats the point otherwise | 12:32 |
| Kinnison | I'd measure by unpacking | 12:32 |
| Kinnison | since what matters is the content, not the transfer encoding | 12:33 |
| ssam2 | I think the best way to achieve success here is to make it as easy as possible to measure success | 12:34 |
| ssam2 | being able to check if two artifacts are the same using `sha1sum` is quite handy | 12:34 |
| * Kinnison didn't think we gzipped artifacts anyway | 12:35 | |
| Kinnison | it was too slow on arm32 | 12:35 |
| ssam2 | ybd seems to do that | 12:35 |
| Kinnison | I still think you should compare the contents | 12:35 |
| Kinnison | because otherwise it's harder to compare ybd's artifacts and morph's artifacts | 12:36 |
| Kinnison | Unless you define a cached-artifact protocol properly | 12:36 |
| ssam2 | the .meta files differ in any case right now. | 12:38 |
| rjek | Include a manifest file with the sha of each file? | 12:39 |
| rjek | No, ignore me. | 12:39 |
| ssam2 | that might make sense, but i'm not going to suggest that yet | 12:39 |
| paulsher1ood | ssam2: i removed elapsed from .meta | 12:40 |
| * paulsher1ood is strongly in favour of whole artifact b4b | 12:40 | |
| *** bwh has joined #baserock | 12:41 | |
| * richard_maw shrugs | 12:42 | |
| paulsher1ood | i know others here don't care so much | 12:42 |
| paulsher1ood | (about that) | 12:43 |
| * Kinnison thinks that expecting the transfer-encoding to be bit-for-bit is daft | 12:43 | |
| richard_maw | from my perspective all that buys is the ability to use the same signatures and checksums for artifacts across every cache server, rather than having to fetch the signature from each server | 12:43 |
| paulsher1ood | it buys more than that, imo, but we've had these discussions already | 12:44 |
| Kinnison | If we only publish transfer-encoding signatures then we are defining the artifact format, cache key shape, etc. Which might be good, but might also be a pain | 12:45 |
| paulsher1ood | Kinnison: you may be right, you usually/often are - but that does not stop me wanting to try for it | 12:45 |
| paulsher1ood | Kinnison: sorry, please explain 'transfer-encoding' | 12:46 |
| *** mariaderidder has joined #baserock | 12:46 | |
| Kinnison | paulsher1ood: the *content* of the artifact is what matters to the final system, yes? | 12:46 |
| paulsher1ood | Kinnison: no, not just that | 12:46 |
| Kinnison | paulsher1ood: that makes the actual artifact file itself merely "transport" | 12:46 |
| paulsher1ood | no, i disagree | 12:46 |
| Kinnison | Okay, what about a tarball of files is more important that the files? | 12:46 |
| paulsher1ood | for example, i am defining cache-key format. the name of the file is important. | 12:47 |
| Kinnison | Okay | 12:47 |
| paulsher1ood | not *more* important, but important | 12:47 |
| Kinnison | So you're defining the full caching structure across tooling? | 12:47 |
| Kinnison | Brave | 12:47 |
| paulsher1ood | for ybd, yes | 12:47 |
| Kinnison | Okay, so you only care about bit-for-bit for ybd? | 12:47 |
| paulsher1ood | i have no hope of causing morph to move in this direction | 12:47 |
| * Kinnison sees | 12:47 | |
| Kinnison | okay | 12:47 |
| Kinnison | then your approach is sound and fine for you | 12:48 |
| * Kinnison misunderstood and thought you were trying to make it possible to demonstrate that given a particular definitions input, ybd and morph produced bit-for-bit the same output | 12:48 | |
| paulsher1ood | for example i tried to discuss cach-key format a long time ago, my ideas were discounted | 12:48 |
| paulsher1ood | there would be some separate value in comparing contents of morph + ybd artifacts. that's a different opportunity imo | 12:49 |
| Kinnison | Okies | 12:49 |
| * Kinnison was approaching the discussion from the wrong end then :) | 12:49 | |
| * Kinnison stands down | 12:49 | |
| richard_maw | paulsher1ood: If it's about how to name the artifacts, I think both morph and ybd are wrong. | 12:49 |
| paulsher1ood | sorry, i appreciate i've not fully explained all aspects of ybd | 12:49 |
| Kinnison | :) | 12:49 |
| paulsher1ood | richard_maw: we can agree to differ on this, i think :) | 12:50 |
| paulsher1ood | they're right for ybd as it stands, in my view | 12:50 |
| paulsher1ood | perhaps the format may be improved in future, of course | 12:51 |
| richard_maw | paulsher1ood: I don't agree in agreeing to differ, if it results in you losing all confidence in it being possible to take morph in the direction it should go. | 12:51 |
| paulsher1ood | sadly, i lost confidence in my ability to help with that for morph last october. | 12:52 |
| straycat | why? | 12:53 |
| paulsher1ood | i think it's all there in my patches, irc discussion, comments on the list. i can make what i want happen faster in ybd, mainly | 12:53 |
| paulsher1ood | hopefully what i'm doing is also benefiting others and morph too | 12:54 |
| straycat | i don't really get this, cause i tried to help with ybd and found that i'd have to do tons of work there too to have it production ready, so regardless of which tool you pick you still have loads to do | 12:55 |
| richard_maw | paulsher1ood: well, currently you're funding Baserock developers, familiar with morph tooling, who are in a better position to make changes to morph than yourself, to make changes to YBD. | 12:55 |
| paulsher1ood | straycat: your view of produciton ready and mine differ. | 12:55 |
| * Kinnison fears that in this instance there's an amount of devil you know vs. devil you don't; and that perhaps there's no right answer. | 12:56 | |
| paulsher1ood | Kinnison: i agree. and it's all just code... the main objective is the solutions, not the code | 12:57 |
| Kinnison | I'd be concerned about steamrollering the developers to reach the objective though :-) | 12:57 |
| paulsher1ood | Kinnison: i'm not doing that. | 12:57 |
| Kinnison | Good. | 12:57 |
| straycat | paulsher1ood, that's confusing, i thought you wrote ybd because you feel morph's code is too complex (which it is) | 12:57 |
| paulsher1ood | that's one problem with it. i'm sure you're aware of others | 12:58 |
| paulsher1ood | anyway, ybd is explained to the best of my current ability at https://github.com/devcurmudgeon/ybd - i'll get round to more documentation when it's ready for 'release' (whatever release means in baserock - we're still wrestling with that i think) | 13:01 |
| paulsher1ood | richard_maw: returning to your comment earlier ' If it's about how to name the artifacts, I think both morph and ybd are wrong.' - do you have a suggestion to improve either or both? | 13:09 |
| richard_maw | paulsher1ood: yes, sha256 digest only, no names in identifiers, being a subdirectory in the cache directory, with artifacts and metadata existing as files inside | 13:10 |
| paulsher1ood | interesting... | 13:10 |
| richard_maw | though tbh I'd prefer if we could move to ostree, rather than maintaining our own code for the cache, since then we'd have access to a first-class propagation protocol | 13:11 |
| paulsher1ood | but as someone who actually debugs caching, and as a user who wants some idea of what's already there, i need the name of the component in the cache-key | 13:12 |
| richard_maw | it's simple to have a tool inspect the metadata and tell you what is inside there | 13:12 |
| paulsher1ood | richard_maw: sorry, i've thought about this a lot, i disagree. | 13:13 |
| paulsher1ood | what do you mean by 'being a subdirectory in the cache directory' ? | 13:13 |
| paulsher1ood | are you suggresting a cache artifact is a directory, rather than a file? | 13:14 |
| richard_maw | your metadata would be $cachedir/1234abcd…ef/metadata.json, your artifact would be $cachedir/1234abcd…ef/artifacts/gcc-bins/tree.tar | 13:15 |
| paulsher1ood | richard_maw: i did look at ostree, and it seems very interesting. but too much extra stuff for my use-cases | 13:15 |
| paulsher1ood | richard_maw: again, i disagree i'm afraid. | 13:15 |
| straycat | i don't think anyone actually wants to work with objects in the cache as files, the problem paulsher1ood now has is that having written so much implementation he already knows way more than the average user does. all i want to know as a user is 1) show me the build log for artifact foo? 2) is foo present in the cache? | 13:16 |
| straycat | (so the artifact name isn't all that important from a user perspective) | 13:17 |
| straycat | *filename | 13:17 |
| paulsher1ood | straycat: you're falling into one of the many traps of being a software developer.... | 13:17 |
| richard_maw | paulsher1ood: and you're not? | 13:18 |
| paulsher1ood | i do, all the time. | 13:18 |
| Kinnison | naah, paulsher1ood isn't a software developer | 13:18 |
| * Kinnison hides | 13:18 | |
| paulsher1ood | but not this one.. | 13:18 |
| paulsher1ood | the specific trap i'm referring to is assuming that you know what a 'user' wants... | 13:18 |
| paulsher1ood | as my my friend Nick Healey said.. 'often developers think that users are just like themselves, only less intelligent' | 13:19 |
| * robtaylor nods | 13:19 | |
| * Kinnison assumes users are clever bastards who always want to thwart me | 13:19 | |
| Kinnison | this is why I advocate "everything in git" :) | 13:20 |
| Kinnison | (and "git in everything" for good measure) | 13:20 |
| * SotK wonders what use-case a user would have for wanting to mess around in the cache | 13:20 | |
| * robtaylor reccomends the 'design of everyday things' and 'emotional design' by Donald A. Norman | 13:21 | |
| Zara | if by 'artifacts' we mean any of the stuff listed in /src/cache/artifacts, then I work with them quite a bit and filenames are handy (when I want to check behaviour at the build or install steps, or grab the whole artifact to extract (eg: for cross-bootstrap). Might be misinterpreting this. | 13:21 |
| straycat | SotK, 1) build log 2) why am i building foo? | 13:21 |
| richard_maw | SotK: It's all gone wrong and they hope to get their files out? | 13:21 |
| straycat | robtaylor, design of everyday things was good :) | 13:21 |
| paulsher1ood | Zara: try ybd :-) | 13:21 |
| * straycat has yet to read emotional design | 13:21 | |
| paulsher1ood | straycat: let me put it another way... and i hope no-one here will take offence at this... | 13:22 |
| Kinnison | IMO, if you have to go into a cache dir (which ought to be an implementation detail of the software you're using) the developer is "doing it wrong" | 13:22 |
| * Kinnison never has to go into ~/.cache/GoogleChrome (except to rm -rf the whole lot) | 13:22 | |
| robtaylor | straycat: emotional design is his 'ok, so everyone's been doing what i said for 20 years. Now this is the stuff I think i got wrong' | 13:22 |
| paulsher1ood | i think i've been the most passionate and long-term *user* of morph. | 13:22 |
| SotK | Kinnison: +1 | 13:23 |
| richard_maw | paulsher1ood: 1 think I dislike about ybd's format, is that I find it *harder* to read, because the names of the artifacts don't line up, and they get lost in the long hex string. That may be because I read in blocks rather than lines though. | 13:23 |
| robtaylor | i think when it comes to dev tools, it tends to be a good idea to not try to hide things or do too much magic | 13:24 |
| paulsher1ood | richard_maw: ah, i hadn't thougt of that. i would gently say that ibelieve lots of other people read right-to-left, though, and can grok the name easiest at the left | 13:24 |
| Kinnison | if I'm reading a single line, and I'm not reading hebrew, then I read RtL. If I'm reading an "ls" output, I tend to read in blocks | 13:24 |
| robtaylor | cos your audience are developers and they will get frustrated when things don't wotk and will want to try and work around the problem ;) | 13:25 |
| paulsher1ood | left to right :) | 13:25 |
| Kinnison | yes, LtR not RtL | 13:25 |
| robtaylor | compare using Make and shell to using VS.. | 13:25 |
| * Kinnison is having difficulty braining today | 13:25 | |
| richard_maw | paulsher1ood: it's not the direction that's the problem, it's the alignment. If the names were padded appropriately, I could read it in the other direction too | 13:25 |
| straycat | paulsher1ood, have you tried using a version of ls that doesn't format its output into columns? :) | 13:25 |
| paulsher1ood | richard_maw: now that's a great idea. can/should we impose a maxlength on names? | 13:26 |
| paulsher1ood | straycat: sorry, i don't understand? | 13:26 |
| richard_maw | yes, it is exactly 64 characters | 13:26 |
| * richard_maw ducks | 13:26 | |
| paulsher1ood | richard_maw: i mean names of chunks/strata/systems/ | 13:26 |
| straycat | paulsher1ood, ls -l without padding is not much fun for similar reasons | 13:26 |
| straycat | anyhow... bit of a digression i guess >.> | 13:27 |
| * Kinnison also recommends the book robtaylor mentioned. I've only skimmed it, but it had a lot of good points to make | 13:27 | |
| richard_maw | paulsher1ood: it ought to be none of our business what the user has decided to call their definitions. After all, who are we to judge what the user wants to call their definitions? | 13:28 |
| Zara | Kinnison: agree that it shouldn't be necessary, to go into a cache dir, not sure what alternative there is atm (as far as cross-bootstrap goes, anyway) | 13:28 |
| Zara | ignore extra first comma :P | 13:28 |
| Kinnison | Zara: I am confused as to why you need to go into the cache dir for cross-bootstrap | 13:28 |
| * Kinnison never had to | 13:28 | |
| Kinnison | Or at least, I don't remember ever having to | 13:29 |
| paulsher1ood | richard_maw: so in that case, we can't cause alignment except by being at the end. | 13:29 |
| richard_maw | or not being there at all, and not needing alignment | 13:30 |
| * SotK only goes in the cache dir to remove things as far as he can remember (except when hacking on the caching stuff) | 13:30 | |
| Zara | how else do you get the tarball of the system to extract? afaik it's in /src/cache | 13:30 |
| Kinnison | I thought it put it in . | 13:30 |
| paulsher1ood | richard_maw: fwiw i've had some user feedback ybd preferring ybd's cache format | 13:30 |
| paulsher1ood | anyway, bikeshedding :)( | 13:30 |
| richard_maw | paulsher1ood: oh, and the arbitrary naming thing is also a reason to not put it in the flie names in the cache, the user could perfectly easily put something that's not encodable in the file system | 13:31 |
| Kinnison | Zara: Oh horror it does refer to the cache | 13:31 |
| paulsher1ood | richard_maw: if they do, it won't work. | 13:31 |
| * Kinnison ewwws | 13:31 | |
| Kinnison | though you can treat it as an opaque filename | 13:32 |
| Kinnison | it is, however, horrendous | 13:32 |
| richard_maw | paulsher1ood: yes, and it's our fault for assuming the user wouldn't want to do that | 13:32 |
| paulsher1ood | richard_maw: i'm ok with that. | 13:32 |
| * rjek names something foo///\0bar: | 13:32 | |
| paulsher1ood | i stand by my assumption. | 13:32 |
| richard_maw | rjek: it's the / that would choke both morph and ybd on Linux | 13:33 |
| rjek | Backspace in filenames can be exciting, too | 13:33 |
| richard_maw | hah! | 13:33 |
| Zara | Kinnison: yeah, the cross bootstrap guide says 'once morph has finished building, it will tell you where the tarball is' (it has said that since the first version), so I've been going with that. | 13:33 |
| paulsher1ood | richard_maw: i'd have this kind of stuff rejected in the schema, myself | 13:33 |
| richard_maw | I'd consider not being able to handle everything the user could possibly put in the name: field as a bug. | 13:34 |
| paulsher1ood | richard_maw: again, we fundamentally disagee | 13:34 |
| nowster | ybd gives you the filename, not the path. morph gives you the full path. | 13:35 |
| paulsher1ood | that's like saying 'the customer is always right' which is clearly wrong | 13:35 |
| straycat | no it's not | 13:35 |
| straycat | it's just not imposing artificial constraints | 13:35 |
| paulsher1ood | well, i'm disagreeing a lot here. hence ybd | 13:36 |
| * paulsher1ood goes to breakfast | 13:36 | |
| *** flatmush has quit IRC | 13:55 | |
| *** flatmush has joined #baserock | 13:56 | |
| *** fay_ has quit IRC | 14:05 | |
| straycat | I'm very sorry but I won't be able to complete the changes listed at https://gerrit.baserock.org/#/q/owner:%22Richard+Ipsum%22+status:open would someone here like to take ownership of them or shall I abandon the changes? | 14:07 |
| richard_maw | straycat: I'll accept ownership of all but the distbuild one. I'd prefer someone with more knowledge of distbuild take that one. | 14:10 |
| straycat | richard_maw, Thanks :) | 14:10 |
| straycat | SotK, ssam2, perryl, Would any of you be willing to take ownership of https://gerrit.baserock.org/#/c/666/ ? | 14:11 |
| perryl | i may be able to take a look at it next week, though i'm not sure how much time it'll need on it? | 14:13 |
| *** pacon has quit IRC | 14:13 | |
| straycat | perryl, It should be for the most part ready, there's 2 comments where I realised the options I'd hardcoded should be able to be set via morph.conf | 14:13 |
| perryl | ahh, that seems alright; yeah, i'll take ownership of that one :) | 14:15 |
| straycat | perryl, awesome thanks :) | 14:15 |
| straycat | I've really enjoyed being apart of this upstream thank you for teaching me so much | 14:16 |
| *** straycat has left #baserock | 14:16 | |
| * richard_maw wonders how he's supposed to adopt a change | 14:18 | |
| *** fay_ has joined #baserock | 14:19 | |
| * persia catches up on backscroll, and wonders why OSTree isn't interesting for ybd. Didn't it provide a massive speedup for morph, both in terms of cache object download times and in terms of system assembly times? | 15:35 | |
| SotK | it made deployment and system construction much quicker | 15:35 |
| SotK | I'm not sure how much testing was done with downloading of cache | 15:35 |
| persia | I had the impression the objects themselves were smaller somehow, but I am unsurprised to be wrong. | 15:36 |
| ssam2 | OStree deduplicates at the file level, so an OSTree artifact cache is likely to take up much less disk space | 15:37 |
| ssam2 | it can also serve objects zipped, which our current cache server doesn't | 15:37 |
| *** chibuzor_ has quit IRC | 16:28 | |
| paulsher1ood | ssam2: i think there's some github-sponsored git server functionality for this, which i'd like to compare vs ostree... i need to track it down | 16:39 |
| paulsher1ood | persia: ^^ | 16:39 |
| persia | paulsher1ood: Are you proposing storing cache objects in git? | 16:40 |
| paulsher1ood | persia: when i originally tried to integrate ostree into baserock, i got stuck - it was xmas holidays iirc, and no-one was around to answer my cries for help | 16:40 |
| paulsher1ood | persia: probably. not sure yet. depends on performance, and nubmer of dependencies introduced, and reliability etc | 16:40 |
| persia | Since then, I believe SotK did do that integration. | 16:40 |
| paulsher1ood | ack | 16:40 |
| persia | I believe that caching with git is a very bad idea: it is everything git is least good at doing. | 16:41 |
| paulsher1ood | persia: in other news drnic has been experimenting http://54.82.85.58:8080/pipelines/demo-ybd | 16:41 |
| persia | There are ways to use git backing stores to cache that may be sensible, but I fail to see the benefit over a traditional filesystem. | 16:41 |
| persia | More users is always a good thing. | 16:41 |
| paulsher1ood | persia: noted. i'm not jumpiong to any conclusions | 16:41 |
| paulsher1ood | persia: one benefit may be gittorrent all the things? :-) | 16:42 |
| persia | bittorrent all the things is easier to implement | 16:42 |
| paulsher1ood | but i'm speculating | 16:42 |
| persia | Plus, bittorrent is really bad at dealing with lots of files many of which are only occasionally interesting, or sets of files that change regularly. | 16:43 |
| paulsher1ood | fair point. i'm just a dilettante :-) | 16:43 |
| persia | git is a lovely object store. I really like it. I just don't think it is the best class of object store for this sort of thing. | 16:43 |
| persia | A flat HTTP namespace, backed by a traditional filesystem is a fine solution. | 16:44 |
| persia | For more complicated needs (e.g. requirements for artifact archival), it may be better to back the *same* http interface with something like Swift or Ceph. | 16:44 |
| persia | But the build tooling doesn't need to care, as long as the API for both pushing and polling objects is unchanged. | 16:45 |
| *** mariaderidder has quit IRC | 16:45 | |
| persia | Which reminds me: that API ought be documented, and it ought most definitely include some means of storing data, other than the odd backend tricks that are used today. | 16:45 |
| paulsher1ood | yup. i need help :) | 16:46 |
| perryl | i've written a shell script that, when ran, aims to automate the process of setting off consecutive YBD build of build-system-x86_64.morph, extracting the built artifact shasums to files, sanitising the shasum files and then obtaining a diff to show which artifacts are non-reproducible. at least, that's the aim; my shell-fu is rather lacking | 16:47 |
| perryl | if anyone has a chance to look at http://wiki.baserock.org/projects/projects/deterministic-builds/get-non-deterministic-builds/ and possibly let me know where i've gone wrong i would be incredibly appreciative! | 16:47 |
| paulsher1ood | perryl: why shell? | 16:48 |
| persia | perryl: You may find || and && useful as shell constructs. | 16:48 |
| perryl | paulsher1ood: it was based on the terminal commands i used to obtain the diffs manually. perhaps not the neatest way | 16:48 |
| persia | e.g. `[ -d foo ] && bar` | 16:48 |
| persia | That said, when testing conditions, it is probably best to test them backwards. | 16:49 |
| persia | So: | 16:49 |
| persia | [ ! -d "$definitions_dir" ] && exit (1) | 16:49 |
| persia | git pull origin head | 16:49 |
| richard_maw | perryl: your variable assignment are incorrect, assuming wiki.baserock.org hasn't fiddled the text | 16:49 |
| persia | etc. | 16:49 |
| persia | There is also an "else" construct you may find useful. | 16:50 |
| richard_maw | perryl: so `build1 = build1` should be `build1=build1` | 16:50 |
| richard_maw | perryl: `rm -r "$artifact_dir/*" | 16:50 |
| richard_maw | ` won't glob properly, you need to run `rm -r "$artifact_dir/"*` | 16:50 |
| richard_maw | note that the * is outside the quotes | 16:50 |
| persia | it isn't safe to assume that just because someone has a directory named "foo", that this is a .git archive. Best to test for foo/.git if one expects foo/ to be a directory in which one can run git commands. | 16:51 |
| richard_maw | perryl: you don't need to quote most of your calls to mkdir, in `if [ ! -d '/home/build' ]`, but it won't hurt | 16:51 |
| persia | `mkdir -p ...` is a safe way to 1) make any missing subdirectories and 2) do nothing if the files already exist | 16:51 |
| * persia would use `mkdir -p /home/build/ccache /home/build/gits /home/build/tmp/deployments` | 16:52 | |
| perryl | richard_maw, persia: noted, i'll make those changes and re-test, thanks! | 16:53 |
| richard_maw | `find "$artifact_dir" -type f -print0 | xargs -0 shasum > $build1.shasum`: if you're piping to xargs you ought to use -r, so if you find no files it doesn't choke | 16:53 |
| persia | perryl: I doubt either of us is close to done yet : you may want to wait :) | 16:53 |
| richard_maw | you can just do `find "$artifact_dir" -type f -exec sha1sum '{}' + > $build1.shasum` | 16:53 |
| paulsher1ood | perryl: i forgot to ask the key question... does it work? is ybd generating b4b reproducible systems? :-) | 16:53 |
| persia | For many cases, "-exec {} +" avoids the need to use xargs with find. | 16:54 |
| persia | There is a limit to command lengths (4095 chars, I think), so for very long find lists, this doesn't work. | 16:54 |
| richard_maw | oh, and `$build1.shasum` ought to be `"${build1}.shasum"`, so quote the interpolation, and use ${} expansion to make it more obvious that you're subtituting a value and putting .shasum at the end, rather than trying to interpolate a variable called build1.shasum | 16:54 |
| * paulsher1ood is frankly jawdropped by the level of deep fu on this channel sometimes | 16:55 | |
| perryl | paulsher1ood: it's managed to get a list of everything in build-systems (manually) that isn't b4b here http://wiki.baserock.org/projects/deterministic-builds/build-system-x86_64-non-reproducible/ , which should be formatted to be more readable...it seems the majority of files that aren't bit for bit are .pyc, .pyo, .gzip and binary files with timestamps | 16:55 |
| richard_maw | `grep -e stage1 -e stage2 -e build-log -e meta` if you put a -F in there it'll grep faster, since all your strings are fixed strings, rather than regular expressions, but competent greps will manage to optimise this themselves | 16:55 |
| persia | For open file safety, I would recommend `mv "$artifact_dir" "artifact_dir-tmp" && mkdir artifact_dir` over the current mkdir/mv pairing. | 16:56 |
| richard_maw | `while read j`, use `read -r j` in case one of your file names has a \ in it | 16:56 |
| richard_maw | perryl: you could do with a function for the blocks starting "Creating temporary duplicates" and ending "Sorting alphabetically by component" | 16:58 |
| persia | I don't like % for sed. Where / is inappropriate, I tend to use |, but that might just be due to me reading % as variable-related from other languages. | 16:58 |
| richard_maw | and I've finished | 16:58 |
| persia | useless use of cat: avoid `cat foo | bar`, instead `bar < foo` | 16:58 |
| perryl | persia: specific cat instance or all of them? | 16:59 |
| persia | perryl: The one I noticed was under "Contracting filenames ...", but anywhere you have `cat foo | bar`, you are running more processes than you need, for no advantage. | 16:59 |
| perryl | noted! i'll change accordingly | 17:00 |
| *** bashrc has quit IRC | 17:01 | |
| persia | Annoyingly, for while constructs, it is the "done" that takes the input file, so it may be clearer to uselessly use cat there: one needs to optimise both for readability and performance. | 17:01 |
| persia | (being as `while read -r j;do echo $j; done < diff.compare` is hard to read. | 17:02 |
| *** jonathanmaw has quit IRC | 17:02 | |
| persia | Actually, the "cleaning up the diff file" section has a race condition. | 17:02 |
| persia | You assume that the cat has finished before the first mv call in the loop, which only happens for systems where the available buffers are much larger than the file size. | 17:03 |
| persia | It isn't safe to overwrite diff.compare until after the done. | 17:03 |
| * persia is done | 17:03 | |
| perryl | persia, richard_maw: thank you both very much for your help!! i'll implement the changes, test it and edit, thanks! | 17:05 |
| persia | On the larger issue: .pyc and .pyo files shoudn't need to be bit-for-bit, but mostly because they should never actually be present on a deployment image (they are supposed to be runtime-compiled local cache objects built on the target, so building pre-deploy means they are misoptimised) | 17:06 |
| paulsher1ood | perryl: happy to accept the tweaked script upstream, if you want to offer it in, say, a scripts subdirectory? | 17:07 |
| persia | gzip can be made reproducible, but many gzip files are tarballs, which themselves need to be made reproducible. | 17:07 |
| * persia forgets the arguments offhand, but can look them up if discovery is difficult | 17:07 | |
| perryl | paulsher1ood: sure! i'll send a pull request once it's polished! | 17:07 |
| paulsher1ood | tvm | 17:07 |
| ssam2 | persia: what about systems with read-only /usr, though ? | 17:08 |
| ssam2 | I've not looked into whether we can configure Python system-wide to put all the .pyc files somewhere that is writable | 17:09 |
| ssam2 | if so, great | 17:09 |
| persia | ssam2: I am not aware of any consensus within the python community of how to prepopulate read-only filesystems containing python code. | 17:09 |
| ssam2 | oh :( | 17:09 |
| persia | That doesn't mean it doesn't exist, just that I'm not aware. | 17:09 |
| persia | I'd recommend trolling PEPs before proposing anything, but it is something that is of wider interest than just Baserock, and belongs in a PEP. | 17:10 |
| persia | I do feel fairly strongly that we shouldn't care much about bit-for-bit reproducible .pyc files until/unless we know of that consensus, and are implementing it. | 17:10 |
| ssam2 | indeed. I'd like to just rm them all at build time | 17:13 |
| paulsher1ood | ssam2: we could add that as a default command? | 17:13 |
| paulsher1ood | ssam2: also, do you want to move that function into utils, or shall i merge as is? | 17:14 |
| ssam2 | if you're not bothered, merge as-is | 17:18 |
| ssam2 | paulsher1ood: we could easily remove the .pyc and .pyo files, the worry is that if /usr was readonly, Python would then recompile every .py file every time it loaded it | 17:19 |
| ssam2 | I don't know how slow that is, but I presume the .pyc files are cached for a reason | 17:20 |
| paulsher1ood | ssam2: merged, thank you :-) | 17:22 |
| ssam2 | I hope http://185.98.148.127:8080/artifacts shows some matching builds tomorrow | 17:22 |
| ssam2 | should do! | 17:22 |
| persia | python compilation is similar to other compilation in time: pyc caching can add from 10% to 95% performance, depending on the code (and the degree to which compilation is complicated). | 17:24 |
| persia | Solving the read-only /usr problem is something that ought be done, but at a wider scale. | 17:24 |
| persia | One approach would be to take an image, pre-cache everything on the target hardware, and then use that as the basis of a read-only image. | 17:24 |
| persia | But I don't know of any way to pre-optimise everything, and I believe that python sometimes recompiles hotspots for even more optimisation. | 17:25 |
| *** ssam2 has quit IRC | 17:39 | |
| persia | ssam2: PEP 3147 seems to cover similar ground. PEP 304 was directly addressing this issue, but was withdrawn. | 17:39 |
| persia | Bother. Timing error :( | 17:39 |
| *** gary_perkins has quit IRC | 18:04 | |
| *** lachlanmackenzie has quit IRC | 18:29 | |
| *** edcragg has quit IRC | 19:02 | |
| *** zoli__ has quit IRC | 19:34 | |
| *** zoli__ has joined #baserock | 20:17 | |
| *** edcragg has joined #baserock | 21:06 | |
| *** zoli__ has quit IRC | 21:30 | |
| *** zoli__ has joined #baserock | 21:33 | |
| *** zoli__ has quit IRC | 22:01 | |
| *** edcragg has quit IRC | 22:25 | |
| *** zoli__ has joined #baserock | 22:36 | |
| *** edcragg has joined #baserock | 22:54 | |
| *** zoli__ has quit IRC | 23:24 | |
| *** edcragg has quit IRC | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!