IRC logs for #baserock for Thursday, 2017-04-27

paulsherwoodso iiuc from gtristan's comments, this is a definitions issue? or did something break in ybd? or was it already a bug in ybd masked by soemthing else?06:58
*** jude has joined #baserock07:05
*** rdale has joined #baserock07:10
jjardonpaulsherwood: seems something has broke in ybd (https://gitlab.com/baserock/definitions/pipelines); I'm bisecting at the moment07:14
jjardonmaybe not related, but we need this: https://gitlab.com/baserock/definitions/merge_requests/34 can someone review, please?07:28
paulsherwoodtvm07:28
paulsherwood+1 for 3407:29
*** ctbruce has joined #baserock07:33
jjardonthanks, that fixed the error I posted before, lets see if we can reproduce the original problem now07:39
*** toscalix has joined #baserock07:42
jjardonthis pipeline will rebuild everything with current ybd master: https://gitlab.com/baserock/definitions/pipelines/789883107:50
*** jonathanmaw has joined #baserock08:25
*** ssam2 has joined #baserock09:08
*** ChanServ sets mode: +v ssam209:08
*** gtristan has joined #baserock09:29
ironfootmaybe this error only happens when executing ybd with multiple instances and that's why we haven't seen it in the CI?09:37
ironfootwell, I'm assuming that paulsherwood was running ybd with multiple instances yesterday09:38
ironfootOr maybe ybd randomises the build order also for a single instance? I really don't know the internals of ybd :/09:39
ironfootjust trying to throw here random ideas of why is this happening now09:39
gtristanironfoot, I believe ybd randomizes build order (and consequently staging order) in every scenario, because it's intended that it can run on separate machines while contributing to the same artifact cache09:52
gtristanI'm not sure how the CI could have ever passed except for by random coincidence, in the case that CI runs without an existing artifact cache09:53
gtristanin the case that an artifact cache already exists, enough builds will result in a complete build09:53
gtristanironfoot, there is another part of the puzzle which is my "fault" for fixing09:54
gtristanwhich is how ybd was (and is now) handling symlinks09:54
gtristanFirst: YBD was removing an existing empty directory to replace it with a symlink in the case of staging a symlink to a directory09:54
gtristanNext: YBD I believe for a brief period was recursively removing a symlink directory target to replace it with a symlink09:55
gtristanFinally: I changed this to issue a warning and *not* stage a symlink to a directory in the case that a non-empty directory exists09:55
gtristanironfoot, it could be that some artifacts in a shared cache somewhere were created before that change, but I believe there was a cache key revision bump since then09:56
ironfootbuildstream us soon, please!09:57
gtristanit's there man !09:57
gtristanironfoot, file bugs for what is blocking your transition09:57
gtristanironfoot, otherwise, I think a next step would be to try the build on arm hosts09:58
gtristanbut as long as it's linux based arm hosts, I dont see why it would not work (but admit to not having really tested there)09:58
ironfooti don't think nothing is blocking me from moving tbh10:00
ironfootI guess my only worry would be git.baserock.org, which atm is built using baserock, and upgradeable10:00
gtristanSo the upgrade story10:01
ironfootbut maybe that host should become a non-baserock host in the future, maybe debian?10:01
gtristanAnd btrfs10:01
gtristanironfoot, I think for baserock purposes, as a reference linux build, it certainly makes sense to dogfood it10:01
ironfootyep, there was a time where we wanted all our infra in baserock10:02
gtristanI think ultimately, we probably want it to be an ostree based reference build10:02
ironfootthen we didin't, or maybe, it wasn't going to be easy to maintain, so we didn't10:02
ssam2dogfooding only makes sense if we have the resources to improve the dogfood, which we don't at present10:03
gtristanright, well... maybe you want to take a step back, use debian, introduce a sane branching scheme in baserock first10:03
gtristanand then run latest-stable on infra10:03
ironfootin the case we want to, or have resources for dogfooding, btrfs story is not needed, only upgrades10:03
ssam2for me it's more about our ability to respond to security updates10:03
gtristanAlso, what ssam2 says is true of course :)10:03
gtristanironfoot, right, I would think the upgrade story should ultimately be ostree based and not tied to fs10:04
gtristanthen it's only a matter of deploying a built system image to an ostree repo, which computers (infra or other) can use to update10:04
gtristanbut that has some blockers I think, quite some detail to explore to ensure the filesystem structure, config files, tmp files, etc, all fit with the ostree atomic update paradigm10:05
ironfootconfiguration files are in /etc, persistent data in /home10:07
ironfootwell, /home is another disk10:08
ironfootlogs in /var10:08
ironfootI think git.baserock.org would be allright using ostree as is10:09
ssam2+100 to swapping btrfs for ostree, but i think it'll be the first time we've done that so would need a chunk of tim,e10:10
ssam2-,10:10
ironfoottime, and effort10:10
ssam2IIRC `ostree admin` even provides a tool to do 3-way merge of /etc/ so we can ditch our hacky baserock-system-config-sync script10:10
gtristanironfoot, right, saying that is one thing, but it has to be tried I think10:10
gtristanthat can be a little, or a lot of effort, before trying it out really it's hard to tell10:11
ironfootssam2: we didn't do any hacky software :P10:11
*** locallycompact has joined #baserock12:36
jjardongtristan: we need to fix this first: https://gitlab.com/baserock/definitions/issues/913:02
gtristanjjardon, So can we build on a machine that has xattrs support in the filesys, please ?13:09
gtristanI dont think that is really in need of a "fix" per se, it's just a strange host that doesnt have extended file attribute support13:09
gtristan(pretty standard fare on all linux machines, I'm quite startled that the gitlab runner would not have that)13:10
jjardonrunners use coreos , which maybe disable that feature for wherever reason: I tried to change it to another distro the other day but runners stop working and I didnt have time to debug the problem13:12
jjardonI will try again today when I get home13:12
gtristansure, I dont think we can easily work around that13:13
gtristanxattrs are sort of a hard requirement (at least for the time being) for ostree13:14
*** paulw has joined #baserock14:10
jjardonhttps://gitlab.com/baserock/definitions/builds/1516903714:12
jjardon\o/14:12
jjardongtristan: ^14:13
* jjardon prepares a branch to build all the current baserock CI with BuildStream :)14:16
*** jude has quit IRC14:34
jjardongtristan probably the main blocker for baserock is the cache server14:44
gtristanThat one still says: GLib.GError: g-io-error-quark: Unable to set xattr: Operation not supported (15)14:45
gtristan:-/14:45
gtristanbut we caught a "BUG" :)14:45
gtristanso that should be the _ostree.py module catching that specific exception and generating a more human readable recommendation to use a system with xattr support (rather than a BUG and a stack trace)14:47
* gtristan also agree artifact sharing is kindof really important14:49
paulsherwoodcouldn't you trivially use the existing kbas?14:49
gtristanNot really no14:50
paulsherwoodwhy not?14:50
gtristanNot more trivially than using a hosted ostree repo, anyway14:50
gtristanWhat is more difficult, transforming ostree artifacts to and from tarballs in order to interface with kbas _and_ do the step of uploading and downloading ? Or just implement the uploading/downloading and use a standard ostree repo ?14:51
paulsherwoodwhatever :)14:52
gtristanyeah, it should be quite easy... just... have to... get back to writing code !14:55
* paulsherwood notices https://www.scaleway.com now has armv8 servers15:03
*** gtristan has quit IRC15:03
paulsherwoodjjardon: did your bisect establish what broke?15:15
*** gtristan has joined #baserock15:31
*** paulw has joined #baserock16:09
*** ctbruce has quit IRC16:16
*** toscalix has quit IRC16:33
*** paulw has quit IRC16:35
*** jonathanmaw has quit IRC17:02
*** locallycompact has quit IRC17:13
*** ctbruce has joined #baserock17:34
*** ssam2 has quit IRC18:15
*** ctbruce has quit IRC18:55
*** jude has joined #baserock19:01
*** locallycompact has joined #baserock20:03
*** jude has quit IRC20:04
*** locallycompact has quit IRC21:45
jjardonpaulsherwood didn't manage to reproduce yet; I will try tomorrow again with several ybd instances22:10
*** benbrown_ has quit IRC22:15
*** paulsherwood has quit IRC22:15
*** paulsherwood has joined #baserock22:16
*** benbrown_ has joined #baserock22:21
*** anahuelamo has quit IRC23:02
*** chrispolin has quit IRC23:02
*** gary_perkins has quit IRC23:03
*** chrispolin has joined #baserock23:10
*** anahuelamo has joined #baserock23:12
*** gary_perkins has joined #baserock23:13

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!