IRC logs for #baserock for Thursday, 2015-06-18

*** zoli__ has quit IRC00:18
*** jeak_ has joined #baserock02:43
*** paulw has quit IRC02:53
*** paulsherwood has joined #baserock02:53
*** zoli__ has joined #baserock04:31
*** zoli__ has quit IRC04:43
*** zoli__ has joined #baserock04:52
*** zoli__ has quit IRC06:13
*** straycat has quit IRC06:35
*** straycat has joined #baserock06:36
*** zoli__ has joined #baserock06:41
*** gary_perkins has joined #baserock07:16
*** perryl has joined #baserock07:24
*** gary_perkins has quit IRC07:32
*** mariaderidder has joined #baserock07:35
*** mdunford has joined #baserock07:48
*** SotK has joined #baserock07:53
*** mdunford has quit IRC07:56
*** benbrown_ has joined #baserock07:56
*** mariaderidder has quit IRC07:57
*** benbrown_ has joined #baserock07:57
*** mariaderidder has joined #baserock07:57
*** benbrown_ has quit IRC08:06
*** benbrown_ has joined #baserock08:06
*** tiagogomes_ has quit IRC08:19
*** tiagogomes has joined #baserock08:20
*** Stanto has quit IRC08:29
*** gary_perkins has joined #baserock08:29
tiagogomescan anyone try to build the openstack system? I am getting a build failure although it is working for franred08:30
*** Stanto has joined #baserock08:32
*** franred has quit IRC08:34
*** franred has joined #baserock08:34
franredtiagogomes, do you still having troubles with conntrack-tools package?08:37
tiagogomesfranred, yep08:37
franredif this is the case, this package only lives in my branch, no in master definitions08:37
*** paulw has joined #baserock08:38
franredtiagogomes, I can try to remove my cache artifacts and try to rebuild it08:39
*** jonathanmaw has joined #baserock08:39
tiagogomesfranred ok, I managed to compile it08:44
franredtiagogomes, it builds for me08:44
franredtiagogomes, what was the problem?08:44
tiagogomesit didn't handle well parallelism in the Makefiles08:47
*** kejiahu has joined #baserock08:52
tiagogomesfranred you will need to add `max-jobs: 1` to conntrack-tools08:53
*** CTtpollard has quit IRC08:53
*** CTtpollard has joined #baserock08:53
franredumm, I didn't have to do it08:53
franredtiagogomes, ^^08:53
*** mariaderidder has quit IRC08:54
franredand I've compiled 3 times08:54
tiagogomesfranred that's because the machine that you used to build conntrack-tools has less cores than mine (mine has 8)08:54
franredtiagogomes, probably...and madness...08:55
franredtiagogomes, could you explain why in machines with more cores this would fail?08:56
tiagogomeserm, badly written Makefiles?08:57
*** edcragg has joined #baserock08:57
* franred amends 'max-jobs: 1'08:57
*** pdar has joined #baserock08:58
*** mariaderidder has joined #baserock09:00
*** jmacs has joined #baserock09:02
SotKdoes anyone have time to have a look at my branch for moving writeexts.py into definitions?09:03
SotKbranch is here: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/log/?h=baserock/adamcoldrick/remove-dependencies-v309:04
SotKdiscussion of that version is on the mailing list, starting here: http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-June/013101.html09:05
*** Zara has joined #baserock09:06
*** bashrc has joined #baserock09:07
*** rjek has quit IRC09:08
*** rjek has joined #baserock09:08
*** bashrc has quit IRC09:08
*** lachlanmackenzie has joined #baserock09:09
*** ssam2 has joined #baserock09:11
*** ChanServ sets mode: +v ssam209:11
*** bashrc has joined #baserock09:14
*** Zara has quit IRC09:14
*** Zara has joined #baserock09:15
*** bashrc has quit IRC09:15
straycatrichard_maw, i have updated https://gerrit.baserock.org/#/c/878/09:17
richard_mawstraycat: aye, that's grand.09:17
richard_mawI'll +1 so you can merge it when its dependents are ready.09:18
richard_mawalthough… it's not actually dependent on the workspace command removal09:18
straycatwell it's based on https://gerrit.baserock.org/#/c/870/2 where i remove all the workspace yarns09:19
richard_mawah, so there may be common code that you depend on, or in the very least, some merge conflicts09:20
straycati've got changes in flight to fix yarns for remaining use cases to use definitions repo rather than workspaces, but those will take a little more time09:20
richard_mawgotcha09:20
straycati could rework that change to be based on current master if you prefer? no one's commented on 870 yet09:21
*** gary_perkins has quit IRC09:23
*** paulw has joined #baserock09:23
*** gary_perkins has joined #baserock09:23
*** paulw has quit IRC09:23
richard_mawI'd generally prefer the work for making the yarns use the new workflow happen before the patches to remove the old workflow09:24
richard_mawas I'd more happily +1 those than the ones to remove the workflow code09:24
*** bashrc has joined #baserock09:26
* straycat nods09:26
*** tiagogomes has quit IRC09:29
*** tiagogomes_ has joined #baserock09:29
*** bashrc has quit IRC09:30
*** bashrc has joined #baserock09:31
bashrclooks like IRC is borked. "dh key too small"09:32
jonathanmawbashrc -> #codethink on freenode09:32
radiofreeplease be aware this is a public channel...09:33
* rjek facepalms bashrc09:33
richard_mawrjek: I believe that's classed as assault09:33
ssam2jmacs: just to double check, is it safe for me to destroy your instance in the public cloud now?09:36
jmacsssam2: Yes09:36
ssam2ok, will do09:37
straycatokay functools.wraps is new to me, cool09:39
richard_mawstraycat: aye, it's not massively useful for anything but decorator functions to keep the docstrings for the decorated function09:40
richard_mawbut for those it is the most useful tool for the job09:41
*** bashrc has quit IRC09:44
*** bashrc has joined #baserock09:44
richard_mawI may have to throw the memoise away though, it's not sufficient for the resolve_ref call, since it needs to cache on some partial value of the result09:45
*** CTtpollard has quit IRC09:45
*** CTtpollard has joined #baserock09:45
richard_mawand monkeypatching the resolve_ref_to_tree call isn't sufficient either, as that is from the CachedRepo object, which gets returned to the CachingRepoCache internally09:46
*** tiagogomes_ has quit IRC09:47
*** tiagogomes has joined #baserock09:47
*** mwilliams_ct has left #baserock09:48
ssam2all my artifacts built with YBD last night have repo: None and ref: None in their metadata09:50
ssam2this worked the night before. :(09:50
ssam2and nothing seems to have changed in ybd.git09:50
ssam2oh, it's not all of them, just some of them09:51
*** franred has quit IRC09:51
ssam2oh, in fact it's strata that have this09:51
*** franred has joined #baserock09:51
*** sebh has quit IRC09:53
*** sebh has joined #baserock09:53
*** tiagogomes has quit IRC09:54
*** tiagogomes has joined #baserock09:54
jmacsWhen I visit http://git.baserock.org/cgi-bin/cgit.cgi/delta/gcc-tarball.git/, it takes ages to generate the list of most recent commits when all I want is the links to clone it at the bottom09:59
Kinnisonjmacs: Hmm09:59
* Kinnison wonders if we should look to integrate a newer cgit09:59
jmacsWell, if it were just for me I could put those links at the top instead :)10:00
*** mariaderidder has quit IRC10:02
*** mariaderidder has joined #baserock10:02
*** pdar has quit IRC10:04
*** paulw has joined #baserock10:06
*** mwilliams_ct has joined #baserock10:18
*** pdar has joined #baserock10:20
*** bashrc has quit IRC10:32
ssam2perryl, tlsa, SotK: seems that with GNU gzip, 'gzip --no-name' stops gzip including the mtime and original filename in the .gz file10:32
ssam2busybox gzip doesn't support that though.10:32
ssam2maybe we could add it, it shouldn't be too hard10:33
perrylssam2: sounds deceptively simple :)10:33
richard_mawgzip from stdin and call gzip with faketime?10:34
straycatrichard_maw, i was about to say if that text isn't being used by the tests then i think we should just remove it10:38
*** chibuzor_ has joined #baserock10:40
jmacsDoes anyone know where mpfr/gmp/mpc get built for gcc? I can't find them in definitions.10:41
jmacss/built/referenced10:43
*** bashrc has joined #baserock10:43
SotKiirc they are in the gcc-tarball repo10:44
* SotK may be completely wrong though :)10:44
jmacsI did a git-clone recursive... they're not there10:44
tiagogomesjmacs I think I imported them to the GCC repo and committed. But things may have changed since them...10:45
SotKjmacs: are you looking at the right ref?10:45
SotKthey appear to be there at 'baserock/build-essential'10:46
jmacsApparently not, I was looking in master10:47
ssam2perryl: I moved the http://wiki.baserock.org/projects/deterministic-builds/build-system-x86_64-non-reproducible/ page to the deterministic-builds subfolder10:56
jmacsI guess because mpfr/mpc/gmp all come from tarballs there isn't much better we can do than that10:58
perrylssam2: thanks, sorry, been distracted fixing up a shell script i'll soon be adding to that page :)11:07
*** mariaderidder has quit IRC11:07
*** mariaderidder has joined #baserock11:08
*** mariaderidder has quit IRC11:17
ssam2seems that when YBD creates a tar file, it doesn't sort the files inserted, so the entire artifact is non-deterministic11:21
*** mariaderidder has joined #baserock11:21
ssam2that should be pretty easy to fix though. i'll look at doing a patch11:21
ssam2I've set up a continous builder at http://185.98.148.127:8080/ which has now identified that every single chunk is unreproducible :) but I think that's due to the tarfile issue, as we already investigated the contents of all artifacts in a build-system and some of them were identical11:23
*** zoli__ has quit IRC11:29
*** mariaderidder has quit IRC11:38
*** paulsher1ood has joined #baserock11:41
* paulsher1ood wonders if his irc is actually working11:48
ssam2i can see you11:48
*** zoli__ has joined #baserock11:49
paulsher1ood:)11:49
paulsher1oodi see your new server, ssam2 - how is that working? ie what is powering it?11:49
*** pacon has joined #baserock11:50
straycatrichard_maw, to be fair for the most part the yarns i removed are for commands we don't need11:51
straycatprobably i should sort out something to cover temporary build branches though11:51
richard_mawstraycat: that's fine, I'd just prefer the removal of the yarns before the removal of the commands11:52
ssam2paulsher1ood: http://185.98.148.127:8080/ ? it's a VM at DataCentred11:53
ssam2the web UI is a bunch of stuff I added to morph-cache-server, in a branch11:53
straycatrichard_maw, i'm confused now, that's exactly what i did?11:53
ssam2it's in morph-cache-server.git, which might not be the right place, but works for now11:53
* straycat is a cat of little brain today :(11:53
paulsher1oodssam2: ok - how are you 'scheduling' the build process?11:53
ssam2paulsher1ood: more simple hacks: https://github.com/ssssam/ybd/tree/sam/continuous-hack/continuous11:54
ssam2it's basically: while True: run ybd, submit artifacts to cache, move them into a directory11:54
paulsher1oodssam2: oh, ok. so not gnome continuous, then?11:54
ssam2heh, no11:54
ssam2gnome-continuous' build system is written in Javascript, it's a bit unweildly  (although it functions quite similar to ybd/morph in some ways)11:55
paulsher1ooddoes your list http://185.98.148.127:8080/artifacts  imply there's another list of Matching builds?11:56
ssam2that's the only list11:57
ssam2none of them match right now :( I think that's due to changes in file ordering in the tarfile, though11:57
ssam2however, to fix that in YBD now, we need need to either monkeypatch shutil.make_archive(), or reimplement it a bit11:58
ssam2it's a one line patch to the shutil module to fix this, I think, but I imagine you don't want YBD to depend on a patched version of latest Python3 ..11:58
paulsher1oodcorrect :)11:59
straycatrichard_maw, ?11:59
paulsher1oodssam2: i'd be happy if such a patch were offered upstream of course :)12:00
richard_mawstraycat: From the dependency ordering of the series I saw, you had some patches to the yarn suite after you'd removed commands for other bits of the yarn suite. I'd prefer all the patches that alter yarns to come before all patches that removed commands from morph12:00
SotKcan you not construct the tarfile using the `tarfile` module, or would that have the same issue?12:01
paulsher1ood(the shutil fix, as well as any improvement to ybd)12:01
richard_mawstraycat: if that's what's actually happening, I'm sorry about wasting your time with misinformed complaints, and I'm sorry I didn't make myself clearer earlier so I could've been told I'm wrong earlier.12:01
straycatrichard_maw, right, ok12:01
ssam2SotK: yes, using tarfile and sorting order we add the files fixes the issue12:02
richard_mawit's just more code, and paulsher1ood hates code12:02
ssam2paulsher1ood: i'm trying to submit a patch to python, but I need to work out how to run the Python testsuite before submitting the patch, and so far I got a random error12:02
paulsher1oodssam2: ain't it always the way? :)12:03
ssam2ah! it works.12:03
Kinnisons12:11
*** paulsherwood has quit IRC12:14
*** bjdooks has quit IRC12:20
ssam2bah, Python's tarfile module also unconditionally puts a timestamp in the .gz header12:28
ssam2perhaps the easiest thing to do here is monkeypatch time.time and os.listdir ;)12:29
KinnisonAre you trying to make the artifact bit-for-bit rather than the content of the artifact?12:31
ssam2indeed12:32
Kinnisonwhy?12:32
ssam2in order to be able to measure whether the content is bit for bit more easily12:32
Kinnisonhrf12:32
ssam2sort of defeats the point otherwise12:32
KinnisonI'd measure by unpacking12:32
Kinnisonsince what matters is the content, not the transfer encoding12:33
ssam2I think the best way to achieve success here is to make it as easy as possible to measure success12:34
ssam2being able to check if two artifacts are the same using `sha1sum` is quite handy12:34
* Kinnison didn't think we gzipped artifacts anyway12:35
Kinnisonit was too slow on arm3212:35
ssam2ybd seems to do that12:35
KinnisonI still think you should compare the contents12:35
Kinnisonbecause otherwise it's harder to compare ybd's artifacts and morph's artifacts12:36
KinnisonUnless you define a cached-artifact protocol properly12:36
ssam2the .meta files differ in any case right now.12:38
rjekInclude a manifest file with the sha of each file?12:39
rjekNo, ignore me.12:39
ssam2that might make sense, but i'm not going to suggest that yet12:39
paulsher1oodssam2: i removed elapsed from .meta12:40
* paulsher1ood is strongly in favour of whole artifact b4b12:40
*** bwh has joined #baserock12:41
* richard_maw shrugs12:42
paulsher1oodi know others here don't care so much12:42
paulsher1ood(about that)12:43
* Kinnison thinks that expecting the transfer-encoding to be bit-for-bit is daft12:43
richard_mawfrom my perspective all that buys is the ability to use the same signatures and checksums for artifacts across every cache server, rather than having to fetch the signature from each server12:43
paulsher1oodit buys more than that, imo, but we've had these discussions already12:44
KinnisonIf we only publish transfer-encoding signatures then we are defining the artifact format, cache key shape, etc.  Which might be good, but might also be a pain12:45
paulsher1oodKinnison: you may be right, you usually/often are - but that does not stop me wanting to try for it12:45
paulsher1oodKinnison: sorry, please explain 'transfer-encoding'12:46
*** mariaderidder has joined #baserock12:46
Kinnisonpaulsher1ood: the *content* of the artifact is what matters to the final system, yes?12:46
paulsher1oodKinnison: no, not just that12:46
Kinnisonpaulsher1ood: that makes the actual artifact file itself merely "transport"12:46
paulsher1oodno, i disagree12:46
KinnisonOkay, what about a tarball of files is more important that the files?12:46
paulsher1oodfor example, i am defining cache-key format. the name of the file is important.12:47
KinnisonOkay12:47
paulsher1oodnot *more* important, but important12:47
KinnisonSo you're defining the full caching structure across tooling?12:47
KinnisonBrave12:47
paulsher1oodfor ybd, yes12:47
KinnisonOkay, so you only care about bit-for-bit for ybd?12:47
paulsher1oodi have no hope of causing morph to move in this direction12:47
* Kinnison sees12:47
Kinnisonokay12:47
Kinnisonthen your approach is sound and fine for you12:48
* Kinnison misunderstood and thought you were trying to make it possible to demonstrate that given a particular definitions input, ybd and morph produced bit-for-bit the same output12:48
paulsher1oodfor example i tried to discuss cach-key format a long time ago, my ideas were discounted12:48
paulsher1oodthere would be some separate value in comparing contents of morph + ybd artifacts. that's a different opportunity imo12:49
KinnisonOkies12:49
* Kinnison was approaching the discussion from the wrong end then :)12:49
* Kinnison stands down12:49
richard_mawpaulsher1ood: If it's about how to name the artifacts, I think both morph and ybd are wrong.12:49
paulsher1oodsorry, i appreciate i've not fully explained all aspects of ybd12:49
Kinnison:)12:49
paulsher1oodrichard_maw: we can agree to differ on this, i think :)12:50
paulsher1oodthey're right for ybd as it stands, in my view12:50
paulsher1oodperhaps the format may be improved in future, of course12:51
richard_mawpaulsher1ood: I don't agree in agreeing to differ, if it results in you losing all confidence in it being possible to take morph in the direction it should go.12:51
paulsher1oodsadly, i lost confidence in my ability to help with that for morph last october.12:52
straycatwhy?12:53
paulsher1oodi think it's all there in my patches, irc discussion, comments on the list. i can make what i want happen faster in ybd, mainly12:53
paulsher1oodhopefully what i'm doing is also benefiting others and morph too12:54
straycati don't really get this, cause i tried to help with ybd and found that i'd have to do tons of work there too to have it production ready, so regardless of which tool you pick you still have loads to do12:55
richard_mawpaulsher1ood: well, currently you're funding Baserock developers, familiar with morph tooling, who are in a better position to make changes to morph than yourself, to make changes to YBD.12:55
paulsher1oodstraycat: your view of produciton ready and mine  differ.12:55
* Kinnison fears that in this instance there's an amount of devil you know vs. devil you don't; and that perhaps there's no right answer.12:56
paulsher1oodKinnison: i agree. and it's all just code... the main objective is the solutions, not the code12:57
KinnisonI'd be concerned about steamrollering the developers to reach the objective though :-)12:57
paulsher1oodKinnison: i'm not doing that.12:57
KinnisonGood.12:57
straycatpaulsher1ood, that's confusing, i thought you wrote ybd because you feel morph's code is too complex (which it is)12:57
paulsher1oodthat's one problem with it. i'm sure you're aware of others12:58
paulsher1oodanyway, ybd is explained to the best of my current ability at https://github.com/devcurmudgeon/ybd - i'll get round to more documentation when it's ready for 'release' (whatever release means in baserock - we're still wrestling with that i think)13:01
paulsher1oodrichard_maw: returning to your comment earlier ' If it's about how to name the artifacts, I think both morph and ybd are wrong.' - do you have a suggestion to improve either or both?13:09
richard_mawpaulsher1ood: yes, sha256 digest only, no names in identifiers, being a subdirectory in the cache directory, with artifacts and metadata existing as files inside13:10
paulsher1oodinteresting...13:10
richard_mawthough tbh I'd prefer if we could move to ostree, rather than maintaining our own code for the cache, since then we'd have access to a first-class propagation protocol13:11
paulsher1oodbut as someone who actually debugs caching, and as a user who wants some idea of what's already there, i need the name of the component in the cache-key13:12
richard_mawit's simple to have a tool inspect the metadata and tell you what is inside there13:12
paulsher1oodrichard_maw: sorry, i've thought about this a lot, i disagree.13:13
paulsher1oodwhat do you mean by 'being a subdirectory in the cache directory' ?13:13
paulsher1oodare you suggresting a cache artifact is a directory, rather than a file?13:14
richard_mawyour metadata would be $cachedir/1234abcd…ef/metadata.json, your artifact would be $cachedir/1234abcd…ef/artifacts/gcc-bins/tree.tar13:15
paulsher1oodrichard_maw: i did look at ostree, and it seems very interesting. but too much extra stuff for my use-cases13:15
paulsher1oodrichard_maw: again, i disagree i'm afraid.13:15
straycati don't think anyone actually wants to work with objects in the cache as files, the problem paulsher1ood now has is that having written so much implementation he already knows way more than the average user does. all i want to know as a user is 1) show me the build log for artifact foo? 2) is foo present in the cache?13:16
straycat(so the artifact name isn't all that important from a user perspective)13:17
straycat*filename13:17
paulsher1oodstraycat: you're falling into one of the many traps of being a software developer....13:17
richard_mawpaulsher1ood: and you're not?13:18
paulsher1oodi do, all the time.13:18
Kinnisonnaah, paulsher1ood isn't a software developer13:18
* Kinnison hides13:18
paulsher1oodbut not this one..13:18
paulsher1oodthe specific trap i'm referring to is assuming that you know what a 'user' wants...13:18
paulsher1oodas my my friend Nick Healey said.. 'often developers think that users are just like themselves, only less intelligent'13:19
* robtaylor nods13:19
* Kinnison assumes users are clever bastards who always want to thwart me13:19
Kinnisonthis is why I advocate "everything in git" :)13:20
Kinnison(and "git in everything" for good measure)13:20
* SotK wonders what use-case a user would have for wanting to mess around in the cache13:20
* robtaylor reccomends the 'design of everyday things' and 'emotional design' by Donald A. Norman13:21
Zaraif by 'artifacts' we mean any of the stuff listed in /src/cache/artifacts, then I work with them quite a bit and filenames are handy (when I want to check behaviour at the build or install steps, or grab the whole artifact to extract (eg: for cross-bootstrap). Might be misinterpreting this.13:21
straycatSotK, 1) build log 2) why am i building foo?13:21
richard_mawSotK: It's all gone wrong and they hope to get their files out?13:21
straycatrobtaylor, design of everyday things was good :)13:21
paulsher1oodZara: try ybd :-)13:21
* straycat has yet to read emotional design13:21
paulsher1oodstraycat: let me put it another way... and i hope no-one here will take offence at this...13:22
KinnisonIMO, if you have to go into a cache dir (which ought to be an implementation detail of the software you're using) the developer is "doing it wrong"13:22
* Kinnison never has to go into ~/.cache/GoogleChrome (except to rm -rf the whole lot)13:22
robtaylorstraycat: emotional design is his 'ok, so everyone's been doing what i said for 20 years. Now this is the stuff I think i got wrong'13:22
paulsher1oodi think i've been the most passionate and long-term *user* of morph.13:22
SotKKinnison: +113:23
richard_mawpaulsher1ood: 1 think I dislike about ybd's format, is that I find it *harder* to read, because the names of the artifacts don't line up, and they get lost in the long hex string. That may be because I read in blocks rather than lines though.13:23
robtaylori think when it comes to dev tools, it tends to be a good idea to not try to hide things or do too much magic13:24
paulsher1oodrichard_maw: ah, i hadn't thougt of that. i would gently say that ibelieve lots of other people read right-to-left, though, and can grok the name easiest at the left13:24
Kinnisonif I'm reading a single line, and I'm not reading hebrew, then I read RtL.  If I'm reading an "ls" output, I tend to read in blocks13:24
robtaylorcos your audience are developers and they will get frustrated when things don't wotk and will want to try and work around the problem ;)13:25
paulsher1oodleft to right :)13:25
Kinnisonyes, LtR not RtL13:25
robtaylorcompare using Make and shell to using VS..13:25
* Kinnison is having difficulty braining today13:25
richard_mawpaulsher1ood: it's not the direction that's the problem, it's the alignment. If the names were padded appropriately, I could read it in the other direction too13:25
straycatpaulsher1ood, have you tried using a version of ls that doesn't format its output into columns? :)13:25
paulsher1oodrichard_maw: now that's a great idea. can/should we impose a maxlength on names?13:26
paulsher1oodstraycat: sorry, i don't understand?13:26
richard_mawyes, it is exactly 64 characters13:26
* richard_maw ducks13:26
paulsher1oodrichard_maw: i mean names of chunks/strata/systems/13:26
straycatpaulsher1ood, ls -l without padding is not much fun for similar reasons13:26
straycatanyhow... bit of a digression i guess >.>13:27
* Kinnison also recommends the book robtaylor mentioned. I've only skimmed it, but it had a lot of good points to make13:27
richard_mawpaulsher1ood: it ought to be none of our business what the user has decided to call their definitions. After all, who are we to judge what the user wants to call their definitions?13:28
ZaraKinnison: agree that it shouldn't be necessary, to go into a cache dir, not sure what alternative there is atm (as far as cross-bootstrap goes, anyway)13:28
Zaraignore extra first comma :P13:28
KinnisonZara: I am confused as to why you need to go into the cache dir for cross-bootstrap13:28
* Kinnison never had to13:28
KinnisonOr at least, I don't remember ever having to13:29
paulsher1oodrichard_maw: so in that case, we can't cause alignment except by being at the end.13:29
richard_mawor not being there at all, and not needing alignment13:30
* SotK only goes in the cache dir to remove things as far as he can remember (except when hacking on the caching stuff)13:30
Zarahow else do you get the tarball of the system to extract? afaik it's in /src/cache13:30
KinnisonI thought it put it in .13:30
paulsher1oodrichard_maw: fwiw i've had some user feedback ybd preferring ybd's cache format13:30
paulsher1oodanyway, bikeshedding :)(13:30
richard_mawpaulsher1ood: oh, and the arbitrary naming thing is also a reason to not put it in the flie names in the cache, the user could perfectly easily put something that's not encodable in the file system13:31
KinnisonZara: Oh horror it does refer to the cache13:31
paulsher1oodrichard_maw: if they do, it won't work.13:31
* Kinnison ewwws13:31
Kinnisonthough you can treat it as an opaque filename13:32
Kinnisonit is, however, horrendous13:32
richard_mawpaulsher1ood: yes, and it's our fault for assuming the user wouldn't want to do that13:32
paulsher1oodrichard_maw: i'm ok with that.13:32
* rjek names something foo///\0bar:13:32
paulsher1oodi stand by my assumption.13:32
richard_mawrjek: it's the / that would choke both morph and ybd on Linux13:33
rjekBackspace in filenames can be exciting, too13:33
richard_mawhah!13:33
ZaraKinnison: yeah, the cross bootstrap guide says 'once morph has finished building, it will tell you where the tarball is' (it has said that since the first version), so I've been going with that.13:33
paulsher1oodrichard_maw: i'd have this kind of stuff rejected in the schema, myself13:33
richard_mawI'd consider not being able to handle everything the user could possibly put in the name: field as a bug.13:34
paulsher1oodrichard_maw: again, we fundamentally disagee13:34
nowsterybd gives you the filename, not the path. morph gives you the full path.13:35
paulsher1oodthat's like saying 'the customer is always right' which is clearly wrong13:35
straycatno it's not13:35
straycatit's just not imposing artificial constraints13:35
paulsher1oodwell, i'm disagreeing a lot here. hence ybd13:36
* paulsher1ood goes to breakfast13:36
*** flatmush has quit IRC13:55
*** flatmush has joined #baserock13:56
*** fay_ has quit IRC14:05
straycatI'm very sorry but I won't be able to complete the changes listed at https://gerrit.baserock.org/#/q/owner:%22Richard+Ipsum%22+status:open would someone here like to take ownership of them or shall I abandon the changes?14:07
richard_mawstraycat: I'll accept ownership of all but the distbuild one. I'd prefer someone with more knowledge of distbuild take that one.14:10
straycatrichard_maw, Thanks :)14:10
straycatSotK, ssam2, perryl, Would any of you be willing to take ownership of https://gerrit.baserock.org/#/c/666/ ?14:11
perryli may be able to take a look at it next week, though i'm not sure how much time it'll need on it?14:13
*** pacon has quit IRC14:13
straycatperryl, It should be for the most part ready, there's 2 comments where I realised the options I'd hardcoded should be able to be set via morph.conf14:13
perrylahh, that seems alright; yeah, i'll take ownership of that one :)14:15
straycatperryl, awesome thanks :)14:15
straycatI've really enjoyed being apart of this upstream thank you for teaching me so much14:16
*** straycat has left #baserock14:16
* richard_maw wonders how he's supposed to adopt a change14:18
*** fay_ has joined #baserock14:19
* persia catches up on backscroll, and wonders why OSTree isn't interesting for ybd. Didn't it provide a massive speedup for morph, both in terms of cache object download times and in terms of system assembly times?15:35
SotKit made deployment and system construction much quicker15:35
SotKI'm not sure how much testing was done with downloading of cache15:35
persiaI had the impression the objects themselves were smaller somehow, but I am unsurprised to be wrong.15:36
ssam2OStree deduplicates at the file level, so an OSTree artifact cache is likely to take up much less disk space15:37
ssam2it can also serve objects zipped, which our current cache server doesn't15:37
*** chibuzor_ has quit IRC16:28
paulsher1oodssam2: i think there's some github-sponsored git server functionality for this, which i'd like to compare vs ostree... i need to track it down16:39
paulsher1oodpersia: ^^16:39
persiapaulsher1ood: Are you proposing storing cache objects in git?16:40
paulsher1oodpersia: when i originally tried to integrate ostree into baserock, i got stuck - it was xmas holidays iirc, and no-one was around to answer my cries for help16:40
paulsher1oodpersia: probably. not sure yet. depends on performance, and nubmer of dependencies introduced, and reliability etc16:40
persiaSince then, I believe SotK did do that integration.16:40
paulsher1oodack16:40
persiaI believe that caching with git is a very bad idea: it is everything git is least good at doing.16:41
paulsher1oodpersia: in other news drnic has been experimenting http://54.82.85.58:8080/pipelines/demo-ybd16:41
persiaThere are ways to use git backing stores to cache that may be sensible, but I fail to see the benefit over a traditional filesystem.16:41
persiaMore users is always a good thing.16:41
paulsher1oodpersia: noted. i'm not jumpiong to any conclusions16:41
paulsher1oodpersia: one benefit may be gittorrent all the things? :-)16:42
persiabittorrent all the things is easier to implement16:42
paulsher1oodbut i'm speculating16:42
persiaPlus, bittorrent is really bad at dealing with lots of files many of which are only occasionally interesting, or sets of files that change regularly.16:43
paulsher1oodfair point. i'm just a dilettante :-)16:43
persiagit is a lovely object store.  I really like it.  I just don't think it is the best class of object store for this sort of thing.16:43
persiaA flat HTTP namespace, backed by a traditional filesystem is a fine solution.16:44
persiaFor more complicated needs (e.g. requirements for artifact archival), it may be better to back the *same* http interface with something like Swift or Ceph.16:44
persiaBut the build tooling doesn't need to care, as long as the API for both pushing and polling objects is unchanged.16:45
*** mariaderidder has quit IRC16:45
persiaWhich reminds me: that API ought be documented, and it ought most definitely include some means of storing data, other than the odd backend tricks that are used today.16:45
paulsher1oodyup. i need help :)16:46
perryli've written a shell script that, when ran, aims to automate the process of setting off consecutive YBD build of build-system-x86_64.morph, extracting the built artifact shasums to files, sanitising the shasum files and then obtaining a diff to show which artifacts are non-reproducible. at least, that's the aim; my shell-fu is rather lacking16:47
perrylif anyone has a chance to look at http://wiki.baserock.org/projects/projects/deterministic-builds/get-non-deterministic-builds/ and possibly let me know where i've gone wrong i would be incredibly appreciative!16:47
paulsher1oodperryl: why shell?16:48
persiaperryl: You may find || and && useful as shell constructs.16:48
perrylpaulsher1ood: it was based on the terminal commands i used to obtain the diffs manually. perhaps not the neatest way16:48
persiae.g. `[ -d foo ] && bar`16:48
persiaThat said, when testing conditions, it is probably best to test them backwards.16:49
persiaSo:16:49
persia[ ! -d "$definitions_dir" ] && exit (1)16:49
persiagit pull origin head16:49
richard_mawperryl: your variable assignment are incorrect, assuming wiki.baserock.org hasn't fiddled the text16:49
persiaetc.16:49
persiaThere is also an "else" construct you may find useful.16:50
richard_mawperryl: so `build1 = build1` should be `build1=build1`16:50
richard_mawperryl: `rm -r "$artifact_dir/*"16:50
richard_maw` won't glob properly, you need to run `rm -r "$artifact_dir/"*`16:50
richard_mawnote that the * is outside the quotes16:50
persiait isn't safe to assume that just because someone has a directory named "foo", that this is a .git archive.  Best to test for foo/.git if one expects foo/ to be a directory in which one can run git commands.16:51
richard_mawperryl: you don't need to quote most of your calls to mkdir, in `if [ ! -d '/home/build' ]`, but it won't hurt16:51
persia`mkdir -p ...` is a safe way to 1) make any missing subdirectories and 2) do nothing if the files already exist16:51
* persia would use `mkdir -p /home/build/ccache /home/build/gits /home/build/tmp/deployments`16:52
perrylrichard_maw, persia: noted, i'll make those changes and re-test, thanks!16:53
richard_maw`find "$artifact_dir" -type f -print0 | xargs -0 shasum > $build1.shasum`: if you're piping to xargs you ought to use -r, so if you find no files it doesn't choke16:53
persiaperryl: I doubt either of us is close to done yet : you may want to wait :)16:53
richard_mawyou can just do `find "$artifact_dir" -type f -exec sha1sum '{}' + > $build1.shasum`16:53
paulsher1oodperryl: i forgot to ask the key question... does it work? is ybd generating b4b reproducible systems? :-)16:53
persiaFor many cases, "-exec {} +" avoids the need to use xargs with find.16:54
persiaThere is a limit to command lengths (4095 chars, I think), so for very long find lists, this doesn't work.16:54
richard_mawoh, and `$build1.shasum` ought to be `"${build1}.shasum"`, so quote the interpolation, and use ${} expansion to make it more obvious that you're subtituting a value and putting .shasum at the end, rather than trying to interpolate a variable called build1.shasum16:54
* paulsher1ood is frankly jawdropped by the level of deep fu on this channel sometimes16:55
perrylpaulsher1ood: it's managed to get a list of everything in build-systems (manually) that isn't b4b here http://wiki.baserock.org/projects/deterministic-builds/build-system-x86_64-non-reproducible/ , which should be formatted to be more readable...it seems the majority of files that aren't bit for bit are .pyc, .pyo, .gzip and binary files with timestamps16:55
richard_maw`grep -e stage1 -e stage2 -e build-log -e meta` if you put a -F in there it'll grep faster, since all your strings are fixed strings, rather than regular expressions, but competent greps will manage to optimise this themselves16:55
persiaFor open file safety, I would recommend `mv "$artifact_dir" "artifact_dir-tmp" && mkdir artifact_dir` over the current mkdir/mv pairing.16:56
richard_maw`while read j`, use `read -r j` in case one of your file names has a \ in it16:56
richard_mawperryl: you could do with a function for the blocks starting "Creating temporary duplicates" and ending "Sorting alphabetically by component"16:58
persiaI don't like % for sed.  Where / is inappropriate, I tend to use  |, but that might just be due to me reading % as variable-related from other languages.16:58
richard_mawand I've finished16:58
persiauseless use of cat: avoid `cat foo | bar`, instead `bar < foo`16:58
perrylpersia: specific cat instance or all of them?16:59
persiaperryl: The one I noticed was under "Contracting filenames ...", but anywhere you have `cat foo | bar`, you are running more processes than you need, for no advantage.16:59
perrylnoted! i'll change accordingly17:00
*** bashrc has quit IRC17:01
persiaAnnoyingly, for while constructs, it is the "done" that takes the input file, so it may be clearer to uselessly use cat there: one needs to optimise both for readability and performance.17:01
persia(being as `while read -r j;do echo $j; done < diff.compare` is hard to read.17:02
*** jonathanmaw has quit IRC17:02
persiaActually, the "cleaning up the diff file" section has a race condition.17:02
persiaYou assume that the cat has finished before the first mv call in the loop, which only happens for systems where the available buffers are much larger than the file size.17:03
persiaIt isn't safe to overwrite diff.compare until after the done.17:03
* persia is done17:03
perrylpersia, richard_maw: thank you both very much for your help!! i'll implement the changes, test it and edit, thanks!17:05
persiaOn the larger issue: .pyc and .pyo files shoudn't need to be bit-for-bit, but mostly because they should never actually be present on a deployment image (they are supposed to be runtime-compiled local cache objects built on the target, so building pre-deploy means they are misoptimised)17:06
paulsher1oodperryl: happy to accept the tweaked script upstream, if you want to offer it in, say, a scripts subdirectory?17:07
persiagzip can be made reproducible, but many gzip files are tarballs, which themselves need to be made reproducible.17:07
* persia forgets the arguments offhand, but can look them up if discovery is difficult17:07
perrylpaulsher1ood: sure! i'll send a pull request once it's polished!17:07
paulsher1oodtvm17:07
ssam2persia: what about systems with read-only /usr, though ?17:08
ssam2I've not looked into whether we can configure Python system-wide to put all the .pyc files somewhere that is writable17:09
ssam2if so, great17:09
persiassam2: I am not aware of any consensus within the python community of how to prepopulate read-only filesystems containing python code.17:09
ssam2oh :(17:09
persiaThat doesn't mean it doesn't exist, just that I'm not aware.17:09
persiaI'd recommend trolling PEPs before proposing anything, but it is something that is of wider interest than just Baserock, and belongs in a PEP.17:10
persiaI do feel fairly strongly that we shouldn't care much about bit-for-bit reproducible .pyc files until/unless we know of that consensus, and are implementing it.17:10
ssam2indeed. I'd like to just rm them all at build time17:13
paulsher1oodssam2: we could add that as a default command?17:13
paulsher1oodssam2: also, do you want to move that function into utils, or shall i merge as is?17:14
ssam2if you're not bothered, merge as-is17:18
ssam2paulsher1ood: we could easily remove the .pyc and .pyo files, the worry is that if /usr was readonly, Python would then recompile every .py file every time it loaded it17:19
ssam2I don't know how slow that is, but I presume the .pyc files are cached for a reason17:20
paulsher1oodssam2: merged, thank you :-)17:22
ssam2I hope http://185.98.148.127:8080/artifacts shows some matching builds tomorrow17:22
ssam2should do!17:22
persiapython compilation is similar to other compilation in time: pyc caching can add from 10% to 95% performance, depending on the code (and the degree to which compilation is complicated).17:24
persiaSolving the read-only /usr problem is something that ought be done, but at a wider scale.17:24
persiaOne approach would be to take an image, pre-cache everything on the target hardware, and then use that as the basis of a read-only image.17:24
persiaBut I don't know of any way to pre-optimise everything, and I believe that python sometimes recompiles hotspots for even more optimisation.17:25
*** ssam2 has quit IRC17:39
persiassam2: PEP 3147 seems to cover similar ground.  PEP 304 was directly addressing this issue, but was withdrawn.17:39
persiaBother.  Timing error :(17:39
*** gary_perkins has quit IRC18:04
*** lachlanmackenzie has quit IRC18:29
*** edcragg has quit IRC19:02
*** zoli__ has quit IRC19:34
*** zoli__ has joined #baserock20:17
*** edcragg has joined #baserock21:06
*** zoli__ has quit IRC21:30
*** zoli__ has joined #baserock21:33
*** zoli__ has quit IRC22:01
*** edcragg has quit IRC22:25
*** zoli__ has joined #baserock22:36
*** edcragg has joined #baserock22:54
*** zoli__ has quit IRC23:24
*** edcragg has quit IRC23:31

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!