IRC logs for #baserock for Friday, 2016-07-08

*** gtristan has joined #baserock02:06
*** gtristan has quit IRC02:15
*** gtristan has joined #baserock02:15
*** gtristan has quit IRC02:44
*** gtristan has joined #baserock03:16
*** locallycompact has joined #baserock03:54
*** locallycompact has quit IRC04:18
*** gtristan has quit IRC06:27
*** paulwaters_ has joined #baserock06:54
*** paulwaters_ has quit IRC06:55
*** toscalix has joined #baserock07:29
*** gtristan has joined #baserock07:40
paulsherwoodjjardon: no need to have separate build + deploy phases... just specify the clusters, and they will cause builds before deploy07:40
paulsherwoodin the current situation (with no cached artifacts on runners) the 'build' process (which is really download + combine) is happening twice07:41
*** rdale has joined #baserock07:53
*** jonathanmaw has joined #baserock08:19
*** pedroalvarez has quit IRC08:44
*** pedroalvarez has joined #baserock08:46
*** ChanServ sets mode: +v pedroalvarez08:46
*** tiagogomes has joined #baserock09:09
jjardonpaulsherwood: sure, but I still would like to separate them to see if the pipeline actually fails in the build step or in the deploy one: also,  I think we can keep the cache and the artifacts between stages with the cache: and depends: parameters in our .gitlab.ci.yml; I will try later09:30
paulsherwoodok. IIUC the deploy is failing now because of the tar error09:31
paulsherwoodhttps://gitlab.com/devcurmudgeon/definitions/builds/225137109:32
*** locallycompact has joined #baserock09:32
paulsherwoodwhich i think is something to do with docker, rather than ybd... https://github.com/meteorhacks/meteord/issues/6409:34
richard_mawah, that'll be the non-POSIX file-system semantics from using overlayfs09:34
richard_mawor any other unionfs for that matter09:35
richard_mawthat error happens if there's a different file mid-extract, as tar writes the files, then goes back and fixes up the permissions, but records what inode the file had beforehand, so it can detect if something else moved the files in the meantime09:36
richard_mawthe listed files are directories where it's added new files, which means overlayfs has to create a new directory with the new set of files in09:37
richard_mawthere might be a way to stop that happening, I think it's dependent on the order of the entries in the tar file09:48
*** jonathanmaw has quit IRC09:55
*** jonathanmaw has joined #baserock09:55
paulsherwoodrichard_maw: thanks for the suggestion. iirc ybd is re-using some stuff from morph for creating its tar archives...10:08
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/utils.py#L31010:08
paulsherwoodi wonder if just reverse(sort...) would test your suggestion? but also would it break something else?10:09
* paulsherwood cannot remember why deterministic order in tar was required10:09
richard_mawpaulsherwood: deterministic order was for your project to produce bit-for-bit identical output binaries10:10
richard_mawand changing the sorted to reversed wouldn't do it10:10
paulsherwoodah, yes ok10:11
richard_mawthe problem is you're doing a top-down traversal, when you need a bottom-up one10:11
paulsherwoodok thanks10:13
richard_mawrewrite it to use os.walk rather than recursively calling the add function10:14
pedroalvarezjjardon: https://gerrit.baserock.org/#/c/220210:38
jjardonpedroalvarez: I was just testing a patch for that: https://gitlab.com/baserock/definitions/pipelines/367142910:40
jjardon:)10:40
pedroalvarezheh10:42
pedroalvarezbtw, autogen runs twice10:42
jjardonhow so?10:43
pedroalvarezin pre-configure commands, and in the default configure commands10:44
jjardonand thats why we need bootstrap-commands :) I will fix my patches10:46
*** gtristan has quit IRC11:05
*** gtristan has joined #baserock11:33
paulsherwoodsomeone has asked if it's possible to treat components *without* splitting them at all...12:12
paulsherwoodthe reason being, for some big things, the process of wading through an creating the metadata is slow...12:13
paulsherwoodand then the process of assembling systems is again slow, because it's running through the metadata, rather than just taking the whole set of files (which in this case is what's required)12:14
richard_mawhow is that implemented?, do you have a tarball of the build which you extract based on whether the files match the pattern?12:16
locallycompactAs an aside, what is the splitting logic specifically because I'd quite like to try it in rust outside the context of chunks etc12:20
paulsherwoodeach component artifact is a single tarball, containing a meta file which describes the splits12:20
locallycompacti.e Directory + (Regexy Thing) = Something12:21
richard_mawlocallycompact: in morph it was that artifacts were processed in-order, each artifact could list regular expressions matching the files, and once those files go into the artifact they get removed from the install directory and it moves onto the next artifact's regular expressions12:22
richard_mawpaulsherwood: ah, we didn't see any slowness in morph because it stored a tarball per artifact, so extraction didn't need to look at regexes12:22
paulsherwoodack12:23
locallycompactrichard_maw, What happens when the rules for an artifact overlap12:23
locallycompacti.e, -bin -devel *.so or something12:24
richard_mawlocallycompact: the ordering of the artifacts means that the first one that matches gets it. This means you can have a final -misc which has .* to get everything else12:24
* paulsherwood looks at http://sprunge.us/ifhF and wonders if ybd is still doing the wrong thing12:24
richard_mawpaulsherwood: it stores the matched filenames rather than the patterns?12:25
locallycompactrichard_maw, isn't it the case that both -bin and -devel might both want a file, or is it implied that -devel => -bin12:26
paulsherwoodassuming that's been split using default rules, i would have expected stuff in -bin, -devel12:26
paulsherwoodrichard_maw: yes12:26
richard_mawpaulsherwood: I'm not surprised it's slow then, tarballs aren't random accessible12:26
richard_maware you listing the files to extract on the tar command-line or something?12:27
paulsherwoodrichard_maw: ah, no. the tarball is fully unpacked. ybd assembles using the stuff from foo.unpacked12:27
richard_mawok, so you're iterating through the components list and copying them out then12:29
pedroalvarezpaulsherwood: regarding the default split rules, none of them include the tools/ prefix, so that metadata file looks ok12:29
pedroalvarez(everything in -misc)12:30
paulsherwoodah, ok12:30
paulsherwoodrichard_maw: correct12:30
richard_mawnot sure how it's that slow then12:31
richard_mawhow do you copy into the staging area?12:31
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/sandbox.py#L8312:33
paulsherwoodbut that's only half of the slowness... the other half is wading through the component generating the .meta list. i guess we could abandon that and only do it at install time, but there was something comforting about having the full manifest in /baserock/foo.meta12:35
pedroalvarezit's nice to have the full manifest to find from where a file comes from12:35
pedroalvarezs/from//12:36
paulsherwood:)12:36
richard_mawpaulsherwood: I don't see how you're copying a filtered bit, does get_cache filter the files?12:38
paulsherwoodoh, sorry. when creating a system, it's https://github.com/devcurmudgeon/ybd/blob/master/ybd/splitting.py#L2712:41
richard_mawpaulsherwood: part of it could be that you're creating temporary lists all the time, part of it could be that you're building up this list before doing any copying, both of which causes memory pressure which can slow things, plus generating the split stratum metadata on demand also slows things down12:50
paulsherwoodack12:51
richard_mawstring encoding is potentially non-trivial work too, and if you're trying to make the parent directory for each file before you copy it, you're at least doubling the amount of shared effort for every file in the same directory12:53
richard_mawbasically, even if you are just copying from an existing list, there's extra work involved on top compared to already having it on-disk somewhere12:54
richard_mawand putting it on-disk somewhere is work you can cache and share between builds12:54
richard_mawwhich is why morph kept the artifacts unpacked on disk rather than filtering at install time12:54
paulsherwoodthe artifacts are unpacked on disk. most of the splitting was written by others. i still don't really understand it, i'm afraid.12:55
paulsherwoodi'll stop whining, now :)12:55
*** gtristan has quit IRC14:41
*** franred has quit IRC14:44
*** franred has joined #baserock14:46
*** franred has quit IRC15:22
*** jonathanmaw has quit IRC16:46
*** tiagogomes has quit IRC17:03
*** rdale has quit IRC17:26
*** toscalix has quit IRC17:40
*** locallycompact has quit IRC17:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!