| paulsherwood | so iiuc from gtristan's comments, this is a definitions issue? or did something break in ybd? or was it already a bug in ybd masked by soemthing else? | 06:58 |
|---|---|---|
| *** jude has joined #baserock | 07:05 | |
| *** rdale has joined #baserock | 07:10 | |
| jjardon | paulsherwood: seems something has broke in ybd (https://gitlab.com/baserock/definitions/pipelines); I'm bisecting at the moment | 07:14 |
| jjardon | maybe not related, but we need this: https://gitlab.com/baserock/definitions/merge_requests/34 can someone review, please? | 07:28 |
| paulsherwood | tvm | 07:28 |
| paulsherwood | +1 for 34 | 07:29 |
| *** ctbruce has joined #baserock | 07:33 | |
| jjardon | thanks, that fixed the error I posted before, lets see if we can reproduce the original problem now | 07:39 |
| *** toscalix has joined #baserock | 07:42 | |
| jjardon | this pipeline will rebuild everything with current ybd master: https://gitlab.com/baserock/definitions/pipelines/7898831 | 07:50 |
| *** jonathanmaw has joined #baserock | 08:25 | |
| *** ssam2 has joined #baserock | 09:08 | |
| *** ChanServ sets mode: +v ssam2 | 09:08 | |
| *** gtristan has joined #baserock | 09:29 | |
| ironfoot | maybe this error only happens when executing ybd with multiple instances and that's why we haven't seen it in the CI? | 09:37 |
| ironfoot | well, I'm assuming that paulsherwood was running ybd with multiple instances yesterday | 09:38 |
| ironfoot | Or maybe ybd randomises the build order also for a single instance? I really don't know the internals of ybd :/ | 09:39 |
| ironfoot | just trying to throw here random ideas of why is this happening now | 09:39 |
| gtristan | ironfoot, I believe ybd randomizes build order (and consequently staging order) in every scenario, because it's intended that it can run on separate machines while contributing to the same artifact cache | 09:52 |
| gtristan | I'm not sure how the CI could have ever passed except for by random coincidence, in the case that CI runs without an existing artifact cache | 09:53 |
| gtristan | in the case that an artifact cache already exists, enough builds will result in a complete build | 09:53 |
| gtristan | ironfoot, there is another part of the puzzle which is my "fault" for fixing | 09:54 |
| gtristan | which is how ybd was (and is now) handling symlinks | 09:54 |
| gtristan | First: YBD was removing an existing empty directory to replace it with a symlink in the case of staging a symlink to a directory | 09:54 |
| gtristan | Next: YBD I believe for a brief period was recursively removing a symlink directory target to replace it with a symlink | 09:55 |
| gtristan | Finally: I changed this to issue a warning and *not* stage a symlink to a directory in the case that a non-empty directory exists | 09:55 |
| gtristan | ironfoot, it could be that some artifacts in a shared cache somewhere were created before that change, but I believe there was a cache key revision bump since then | 09:56 |
| ironfoot | buildstream us soon, please! | 09:57 |
| gtristan | it's there man ! | 09:57 |
| gtristan | ironfoot, file bugs for what is blocking your transition | 09:57 |
| gtristan | ironfoot, otherwise, I think a next step would be to try the build on arm hosts | 09:58 |
| gtristan | but as long as it's linux based arm hosts, I dont see why it would not work (but admit to not having really tested there) | 09:58 |
| ironfoot | i don't think nothing is blocking me from moving tbh | 10:00 |
| ironfoot | I guess my only worry would be git.baserock.org, which atm is built using baserock, and upgradeable | 10:00 |
| gtristan | So the upgrade story | 10:01 |
| ironfoot | but maybe that host should become a non-baserock host in the future, maybe debian? | 10:01 |
| gtristan | And btrfs | 10:01 |
| gtristan | ironfoot, I think for baserock purposes, as a reference linux build, it certainly makes sense to dogfood it | 10:01 |
| ironfoot | yep, there was a time where we wanted all our infra in baserock | 10:02 |
| gtristan | I think ultimately, we probably want it to be an ostree based reference build | 10:02 |
| ironfoot | then we didin't, or maybe, it wasn't going to be easy to maintain, so we didn't | 10:02 |
| ssam2 | dogfooding only makes sense if we have the resources to improve the dogfood, which we don't at present | 10:03 |
| gtristan | right, well... maybe you want to take a step back, use debian, introduce a sane branching scheme in baserock first | 10:03 |
| gtristan | and then run latest-stable on infra | 10:03 |
| ironfoot | in the case we want to, or have resources for dogfooding, btrfs story is not needed, only upgrades | 10:03 |
| ssam2 | for me it's more about our ability to respond to security updates | 10:03 |
| gtristan | Also, what ssam2 says is true of course :) | 10:03 |
| gtristan | ironfoot, right, I would think the upgrade story should ultimately be ostree based and not tied to fs | 10:04 |
| gtristan | then it's only a matter of deploying a built system image to an ostree repo, which computers (infra or other) can use to update | 10:04 |
| gtristan | but that has some blockers I think, quite some detail to explore to ensure the filesystem structure, config files, tmp files, etc, all fit with the ostree atomic update paradigm | 10:05 |
| ironfoot | configuration files are in /etc, persistent data in /home | 10:07 |
| ironfoot | well, /home is another disk | 10:08 |
| ironfoot | logs in /var | 10:08 |
| ironfoot | I think git.baserock.org would be allright using ostree as is | 10:09 |
| ssam2 | +100 to swapping btrfs for ostree, but i think it'll be the first time we've done that so would need a chunk of tim,e | 10:10 |
| ssam2 | -, | 10:10 |
| ironfoot | time, and effort | 10:10 |
| ssam2 | IIRC `ostree admin` even provides a tool to do 3-way merge of /etc/ so we can ditch our hacky baserock-system-config-sync script | 10:10 |
| gtristan | ironfoot, right, saying that is one thing, but it has to be tried I think | 10:10 |
| gtristan | that can be a little, or a lot of effort, before trying it out really it's hard to tell | 10:11 |
| ironfoot | ssam2: we didn't do any hacky software :P | 10:11 |
| *** locallycompact has joined #baserock | 12:36 | |
| jjardon | gtristan: we need to fix this first: https://gitlab.com/baserock/definitions/issues/9 | 13:02 |
| gtristan | jjardon, So can we build on a machine that has xattrs support in the filesys, please ? | 13:09 |
| gtristan | I dont think that is really in need of a "fix" per se, it's just a strange host that doesnt have extended file attribute support | 13:09 |
| gtristan | (pretty standard fare on all linux machines, I'm quite startled that the gitlab runner would not have that) | 13:10 |
| jjardon | runners use coreos , which maybe disable that feature for wherever reason: I tried to change it to another distro the other day but runners stop working and I didnt have time to debug the problem | 13:12 |
| jjardon | I will try again today when I get home | 13:12 |
| gtristan | sure, I dont think we can easily work around that | 13:13 |
| gtristan | xattrs are sort of a hard requirement (at least for the time being) for ostree | 13:14 |
| *** paulw has joined #baserock | 14:10 | |
| jjardon | https://gitlab.com/baserock/definitions/builds/15169037 | 14:12 |
| jjardon | \o/ | 14:12 |
| jjardon | gtristan: ^ | 14:13 |
| * jjardon prepares a branch to build all the current baserock CI with BuildStream :) | 14:16 | |
| *** jude has quit IRC | 14:34 | |
| jjardon | gtristan probably the main blocker for baserock is the cache server | 14:44 |
| gtristan | That one still says: GLib.GError: g-io-error-quark: Unable to set xattr: Operation not supported (15) | 14:45 |
| gtristan | :-/ | 14:45 |
| gtristan | but we caught a "BUG" :) | 14:45 |
| gtristan | so that should be the _ostree.py module catching that specific exception and generating a more human readable recommendation to use a system with xattr support (rather than a BUG and a stack trace) | 14:47 |
| * gtristan also agree artifact sharing is kindof really important | 14:49 | |
| paulsherwood | couldn't you trivially use the existing kbas? | 14:49 |
| gtristan | Not really no | 14:50 |
| paulsherwood | why not? | 14:50 |
| gtristan | Not more trivially than using a hosted ostree repo, anyway | 14:50 |
| gtristan | What is more difficult, transforming ostree artifacts to and from tarballs in order to interface with kbas _and_ do the step of uploading and downloading ? Or just implement the uploading/downloading and use a standard ostree repo ? | 14:51 |
| paulsherwood | whatever :) | 14:52 |
| gtristan | yeah, it should be quite easy... just... have to... get back to writing code ! | 14:55 |
| * paulsherwood notices https://www.scaleway.com now has armv8 servers | 15:03 | |
| *** gtristan has quit IRC | 15:03 | |
| paulsherwood | jjardon: did your bisect establish what broke? | 15:15 |
| *** gtristan has joined #baserock | 15:31 | |
| *** paulw has joined #baserock | 16:09 | |
| *** ctbruce has quit IRC | 16:16 | |
| *** toscalix has quit IRC | 16:33 | |
| *** paulw has quit IRC | 16:35 | |
| *** jonathanmaw has quit IRC | 17:02 | |
| *** locallycompact has quit IRC | 17:13 | |
| *** ctbruce has joined #baserock | 17:34 | |
| *** ssam2 has quit IRC | 18:15 | |
| *** ctbruce has quit IRC | 18:55 | |
| *** jude has joined #baserock | 19:01 | |
| *** locallycompact has joined #baserock | 20:03 | |
| *** jude has quit IRC | 20:04 | |
| *** locallycompact has quit IRC | 21:45 | |
| jjardon | paulsherwood didn't manage to reproduce yet; I will try tomorrow again with several ybd instances | 22:10 |
| *** benbrown_ has quit IRC | 22:15 | |
| *** paulsherwood has quit IRC | 22:15 | |
| *** paulsherwood has joined #baserock | 22:16 | |
| *** benbrown_ has joined #baserock | 22:21 | |
| *** anahuelamo has quit IRC | 23:02 | |
| *** chrispolin has quit IRC | 23:02 | |
| *** gary_perkins has quit IRC | 23:03 | |
| *** chrispolin has joined #baserock | 23:10 | |
| *** anahuelamo has joined #baserock | 23:12 | |
| *** gary_perkins has joined #baserock | 23:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!