| *** gtristan has joined #baserock | 02:06 | |
| *** gtristan has quit IRC | 02:15 | |
| *** gtristan has joined #baserock | 02:15 | |
| *** gtristan has quit IRC | 02:44 | |
| *** gtristan has joined #baserock | 03:16 | |
| *** locallycompact has joined #baserock | 03:54 | |
| *** locallycompact has quit IRC | 04:18 | |
| *** gtristan has quit IRC | 06:27 | |
| *** paulwaters_ has joined #baserock | 06:54 | |
| *** paulwaters_ has quit IRC | 06:55 | |
| *** toscalix has joined #baserock | 07:29 | |
| *** gtristan has joined #baserock | 07:40 | |
| paulsherwood | jjardon: no need to have separate build + deploy phases... just specify the clusters, and they will cause builds before deploy | 07:40 |
|---|---|---|
| paulsherwood | in the current situation (with no cached artifacts on runners) the 'build' process (which is really download + combine) is happening twice | 07:41 |
| *** rdale has joined #baserock | 07:53 | |
| *** jonathanmaw has joined #baserock | 08:19 | |
| *** pedroalvarez has quit IRC | 08:44 | |
| *** pedroalvarez has joined #baserock | 08:46 | |
| *** ChanServ sets mode: +v pedroalvarez | 08:46 | |
| *** tiagogomes has joined #baserock | 09:09 | |
| jjardon | paulsherwood: sure, but I still would like to separate them to see if the pipeline actually fails in the build step or in the deploy one: also, I think we can keep the cache and the artifacts between stages with the cache: and depends: parameters in our .gitlab.ci.yml; I will try later | 09:30 |
| paulsherwood | ok. IIUC the deploy is failing now because of the tar error | 09:31 |
| paulsherwood | https://gitlab.com/devcurmudgeon/definitions/builds/2251371 | 09:32 |
| *** locallycompact has joined #baserock | 09:32 | |
| paulsherwood | which i think is something to do with docker, rather than ybd... https://github.com/meteorhacks/meteord/issues/64 | 09:34 |
| richard_maw | ah, that'll be the non-POSIX file-system semantics from using overlayfs | 09:34 |
| richard_maw | or any other unionfs for that matter | 09:35 |
| richard_maw | that error happens if there's a different file mid-extract, as tar writes the files, then goes back and fixes up the permissions, but records what inode the file had beforehand, so it can detect if something else moved the files in the meantime | 09:36 |
| richard_maw | the listed files are directories where it's added new files, which means overlayfs has to create a new directory with the new set of files in | 09:37 |
| richard_maw | there might be a way to stop that happening, I think it's dependent on the order of the entries in the tar file | 09:48 |
| *** jonathanmaw has quit IRC | 09:55 | |
| *** jonathanmaw has joined #baserock | 09:55 | |
| paulsherwood | richard_maw: thanks for the suggestion. iirc ybd is re-using some stuff from morph for creating its tar archives... | 10:08 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/ybd/utils.py#L310 | 10:08 |
| paulsherwood | i wonder if just reverse(sort...) would test your suggestion? but also would it break something else? | 10:09 |
| * paulsherwood cannot remember why deterministic order in tar was required | 10:09 | |
| richard_maw | paulsherwood: deterministic order was for your project to produce bit-for-bit identical output binaries | 10:10 |
| richard_maw | and changing the sorted to reversed wouldn't do it | 10:10 |
| paulsherwood | ah, yes ok | 10:11 |
| richard_maw | the problem is you're doing a top-down traversal, when you need a bottom-up one | 10:11 |
| paulsherwood | ok thanks | 10:13 |
| richard_maw | rewrite it to use os.walk rather than recursively calling the add function | 10:14 |
| pedroalvarez | jjardon: https://gerrit.baserock.org/#/c/2202 | 10:38 |
| jjardon | pedroalvarez: I was just testing a patch for that: https://gitlab.com/baserock/definitions/pipelines/3671429 | 10:40 |
| jjardon | :) | 10:40 |
| pedroalvarez | heh | 10:42 |
| pedroalvarez | btw, autogen runs twice | 10:42 |
| jjardon | how so? | 10:43 |
| pedroalvarez | in pre-configure commands, and in the default configure commands | 10:44 |
| jjardon | and thats why we need bootstrap-commands :) I will fix my patches | 10:46 |
| *** gtristan has quit IRC | 11:05 | |
| *** gtristan has joined #baserock | 11:33 | |
| paulsherwood | someone has asked if it's possible to treat components *without* splitting them at all... | 12:12 |
| paulsherwood | the reason being, for some big things, the process of wading through an creating the metadata is slow... | 12:13 |
| paulsherwood | and then the process of assembling systems is again slow, because it's running through the metadata, rather than just taking the whole set of files (which in this case is what's required) | 12:14 |
| richard_maw | how is that implemented?, do you have a tarball of the build which you extract based on whether the files match the pattern? | 12:16 |
| locallycompact | As an aside, what is the splitting logic specifically because I'd quite like to try it in rust outside the context of chunks etc | 12:20 |
| paulsherwood | each component artifact is a single tarball, containing a meta file which describes the splits | 12:20 |
| locallycompact | i.e Directory + (Regexy Thing) = Something | 12:21 |
| richard_maw | locallycompact: in morph it was that artifacts were processed in-order, each artifact could list regular expressions matching the files, and once those files go into the artifact they get removed from the install directory and it moves onto the next artifact's regular expressions | 12:22 |
| richard_maw | paulsherwood: ah, we didn't see any slowness in morph because it stored a tarball per artifact, so extraction didn't need to look at regexes | 12:22 |
| paulsherwood | ack | 12:23 |
| locallycompact | richard_maw, What happens when the rules for an artifact overlap | 12:23 |
| locallycompact | i.e, -bin -devel *.so or something | 12:24 |
| richard_maw | locallycompact: the ordering of the artifacts means that the first one that matches gets it. This means you can have a final -misc which has .* to get everything else | 12:24 |
| * paulsherwood looks at http://sprunge.us/ifhF and wonders if ybd is still doing the wrong thing | 12:24 | |
| richard_maw | paulsherwood: it stores the matched filenames rather than the patterns? | 12:25 |
| locallycompact | richard_maw, isn't it the case that both -bin and -devel might both want a file, or is it implied that -devel => -bin | 12:26 |
| paulsherwood | assuming that's been split using default rules, i would have expected stuff in -bin, -devel | 12:26 |
| paulsherwood | richard_maw: yes | 12:26 |
| richard_maw | paulsherwood: I'm not surprised it's slow then, tarballs aren't random accessible | 12:26 |
| richard_maw | are you listing the files to extract on the tar command-line or something? | 12:27 |
| paulsherwood | richard_maw: ah, no. the tarball is fully unpacked. ybd assembles using the stuff from foo.unpacked | 12:27 |
| richard_maw | ok, so you're iterating through the components list and copying them out then | 12:29 |
| pedroalvarez | paulsherwood: regarding the default split rules, none of them include the tools/ prefix, so that metadata file looks ok | 12:29 |
| pedroalvarez | (everything in -misc) | 12:30 |
| paulsherwood | ah, ok | 12:30 |
| paulsherwood | richard_maw: correct | 12:30 |
| richard_maw | not sure how it's that slow then | 12:31 |
| richard_maw | how do you copy into the staging area? | 12:31 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/ybd/sandbox.py#L83 | 12:33 |
| paulsherwood | but that's only half of the slowness... the other half is wading through the component generating the .meta list. i guess we could abandon that and only do it at install time, but there was something comforting about having the full manifest in /baserock/foo.meta | 12:35 |
| pedroalvarez | it's nice to have the full manifest to find from where a file comes from | 12:35 |
| pedroalvarez | s/from// | 12:36 |
| paulsherwood | :) | 12:36 |
| richard_maw | paulsherwood: I don't see how you're copying a filtered bit, does get_cache filter the files? | 12:38 |
| paulsherwood | oh, sorry. when creating a system, it's https://github.com/devcurmudgeon/ybd/blob/master/ybd/splitting.py#L27 | 12:41 |
| richard_maw | paulsherwood: part of it could be that you're creating temporary lists all the time, part of it could be that you're building up this list before doing any copying, both of which causes memory pressure which can slow things, plus generating the split stratum metadata on demand also slows things down | 12:50 |
| paulsherwood | ack | 12:51 |
| richard_maw | string encoding is potentially non-trivial work too, and if you're trying to make the parent directory for each file before you copy it, you're at least doubling the amount of shared effort for every file in the same directory | 12:53 |
| richard_maw | basically, even if you are just copying from an existing list, there's extra work involved on top compared to already having it on-disk somewhere | 12:54 |
| richard_maw | and putting it on-disk somewhere is work you can cache and share between builds | 12:54 |
| richard_maw | which is why morph kept the artifacts unpacked on disk rather than filtering at install time | 12:54 |
| paulsherwood | the artifacts are unpacked on disk. most of the splitting was written by others. i still don't really understand it, i'm afraid. | 12:55 |
| paulsherwood | i'll stop whining, now :) | 12:55 |
| *** gtristan has quit IRC | 14:41 | |
| *** franred has quit IRC | 14:44 | |
| *** franred has joined #baserock | 14:46 | |
| *** franred has quit IRC | 15:22 | |
| *** jonathanmaw has quit IRC | 16:46 | |
| *** tiagogomes has quit IRC | 17:03 | |
| *** rdale has quit IRC | 17:26 | |
| *** toscalix has quit IRC | 17:40 | |
| *** locallycompact has quit IRC | 17:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!