| *** rdale has quit IRC | 00:18 | |
| *** rdale has joined #baserock | 00:21 | |
| *** rdale has quit IRC | 01:32 | |
| *** rdale has joined #baserock | 01:34 | |
| paulsherwood | Kinnison: not afaik? | 01:42 |
|---|---|---|
| *** rdale has quit IRC | 02:34 | |
| *** zoli__ has joined #baserock | 04:01 | |
| *** zoli__ has quit IRC | 04:20 | |
| *** zoli__ has joined #baserock | 04:22 | |
| *** zoli__ has quit IRC | 04:52 | |
| *** petefoth has quit IRC | 05:11 | |
| *** zoli__ has joined #baserock | 05:11 | |
| *** petefoth has joined #baserock | 06:25 | |
| *** mike has joined #baserock | 06:58 | |
| *** mike is now known as Guest21191 | 06:58 | |
| *** a1exhughe5 has joined #baserock | 07:05 | |
| *** Guest21191 has quit IRC | 07:12 | |
| *** Guest21191 has joined #baserock | 07:22 | |
| *** Albert_ has joined #baserock | 07:30 | |
| *** petefoth has left #baserock | 07:45 | |
| radiofree | jjardon: i believe the rawdisk images install the bootloader by running extlinux on the *host*, so you'll need to build an image with the image you've built | 07:46 |
| *** petefoth has joined #baserock | 07:48 | |
| * SotK finds that the build-mode is in the stratum, not the chunk morph | 07:54 | |
| SotK | having data for a chunk split over two files is really annoying | 07:54 |
| SotK | also, this means that for any given chunk artifact, artifact.source.morphology['build-mode'] == 'staging' | 07:55 |
| paulsherwood | SotK: see ybd for simplest logic i could get to | 08:02 |
| *** bashrc_ has joined #baserock | 08:05 | |
| SotK | paulsherwood: whereabouts does ybd handle build-mode? | 08:07 |
| *** rdale has joined #baserock | 08:08 | |
| *** jonathanmaw has joined #baserock | 08:08 | |
| *** mwilliams_ct has joined #baserock | 08:09 | |
| paulsherwood | sandbox.py, assemlby.py | 08:17 |
| jjardon | radiofree: yep, I've done that as well | 08:24 |
| *** edcragg has joined #baserock | 08:30 | |
| SotK | paulsherwood: That seems to just try to get 'build-mode' but default to 'staging' if its not there, I can't see where it sets build-mode for each component though? | 08:31 |
| jonathanmaw | I'm getting this error when building tar: http://paste.baserock.org/buxeharice | 08:33 |
| jonathanmaw | it looks like tar's pre-configure commands try to init submodules, but the path to the git repo isn't in the chroot. | 08:33 |
| jonathanmaw | which shouldn't be necessary, since morph has already checked out the submodules | 08:34 |
| jonathanmaw | bootstrap has options for specifying where the submodules' sourcedirs are, so I think I'll use those. | 08:37 |
| *** gary_perkins has joined #baserock | 08:37 | |
| radiofree | jjardon: if it boots then i guess it works! | 08:38 |
| *** ssam2 has joined #baserock | 08:38 | |
| *** ChanServ sets mode: +v ssam2 | 08:38 | |
| jonathanmaw | I am quite puzzled how tar ever built. | 08:52 |
| jonathanmaw | especially since tar's bootstrap thinks that it still needs to fiddle with submodules. | 08:53 |
| jonathanmaw | even when --gnulib-srcdir is defined. | 08:53 |
| pedroalvarez | jonathanmaw: A build log, if that helps: http://sprunge.us/eDcI | 08:54 |
| jonathanmaw | pedroalvarez: which ref is used for that? | 08:55 |
| jonathanmaw | i.e. which ref of tar? | 08:56 |
| franred | good morning, there are a patch-series which would be nice if someone can have a look today https://gerrit.baserock.org/#/q/status:open+project:baserock/baserock/definitions+branch:master+topic:baserock/franred/fix-some-openstack-service-bugs | 08:57 |
| pedroalvarez | jonathanmaw: 9a58d148c26c220cb1b163c71e7a51a2e41f6b37 | 08:58 |
| pedroalvarez | I migh be confusing some metadata fields, let me check | 08:59 |
| jonathanmaw | that looks like the same ref I was using | 08:59 |
| ssam2 | franred: look fine to me | 08:59 |
| franred | ssam2, thank you!! | 08:59 |
| pedroalvarez | jonathanmaw: the error sounds like /src/cache/gits/git___git_baserock_org_delta_tar is corrupted? | 09:00 |
| jonathanmaw | pedroalvarez: nope, it looks fine. the problem is that it's looking for that dir from inside a chroot | 09:00 |
| ssam2 | `git fsck` in there will tell you for sure | 09:00 |
| pedroalvarez | jonathanmaw: wow | 09:01 |
| jonathanmaw | git fsck in git___git_baserock_org_delta_tar seems fine | 09:01 |
| Kinnison | SotK: What are you looking at build-mode for? | 09:02 |
| Kinnison | SotK: ybd assumes 'staging' build mode unless told otherwise | 09:02 |
| SotK | I'm looping through a set of chunks (with the intention of finding the upstream URL for each one) and wanted to ignore bootstrap chunks. However, using chunk.source.morphology['build-mode'] always gave me 'staging' because the build-mode is defined in the stratum rather than the chunk. | 09:04 |
| Kinnison | aah | 09:05 |
| Kinnison | yes | 09:05 |
| Kinnison | that's (IMO) a failing of morph's internal model | 09:05 |
| SotK | I agree | 09:05 |
| Kinnison | Once we have definitions *fully* specified | 09:05 |
| Kinnison | at that point we can try and work out a more idealised model for morph to use internally | 09:05 |
| Kinnison | Until that point, we're throwing darts in the dark, in a room filled with acid balloons | 09:06 |
| petefoth | Kinnison: isn't that a bit dangerous? ;) | 09:07 |
| Kinnison | petefoth: best not do it then | 09:07 |
| petefoth | :) | 09:07 |
| SotK | Kinnison: do you mean we fully specify the current layout, or decide upon a final layout? | 09:08 |
| Kinnison | current | 09:08 |
| jonathanmaw | whee! the solution was to remove .gitmodules in pre-configure-commands | 09:08 |
| Kinnison | Until we know what we have, how can we possibly know what we want? | 09:08 |
| *** Krin has joined #baserock | 09:08 | |
| *** a1exhughe5 has quit IRC | 09:08 | |
| *** a1exhughe5 has joined #baserock | 09:09 | |
| SotK | aha, I can actually see that happening sometime then :) | 09:09 |
| ssam2 | it's started here already, just needs everyone to help tie the details down: http://wiki.baserock.org/definitions/current/ | 09:12 |
| ssam2 | there are quite a lot of details :( | 09:12 |
| *** lachlanmackenzie has joined #baserock | 09:15 | |
| *** flatmush has quit IRC | 09:15 | |
| Kinnison | Aye | 09:16 |
| jonathanmaw | systemd has decided it needs libmount. | 09:17 |
| jonathanmaw | brief googling suggests that util-linux should be providing this | 09:19 |
| jjardon | jonathanmaw: I think that's in util-linux | 09:19 |
| *** flatmush has joined #baserock | 09:20 | |
| ssam2 | I have some patches that it'd be really handy if people could review them! | 09:25 |
| ssam2 | https://gerrit.baserock.org/#/c/317/ and https://gerrit.baserock.org/#/c/339/ are both trivial and both have +1 already | 09:25 |
| ssam2 | https://gerrit.baserock.org/#/c/274/ and the two related changes I hope are easy to understand and have a clear benefit | 09:26 |
| paulsherwood | SotK: do you believe that something else is required, beyond that logic? | 09:26 |
| ssam2 | and https://gerrit.baserock.org/#/c/235/2 and 2 related changes, which fixes a bug in distbuild builds aren't actually always cancelled when expected | 09:27 |
| SotK | paulsherwood: Unless ybd puts build-mode into the chunk morphologies, I couldn't see how they would ever be built as anything except 'staging', but I could be missing something. | 09:28 |
| *** ssam2 has quit IRC | 09:44 | |
| *** ssam2 has joined #baserock | 09:44 | |
| *** ChanServ sets mode: +v ssam2 | 09:44 | |
| Kinnison | SotK: believe me, we build stuff in bootstrap mode :) | 09:53 |
| jonathanmaw | hrm, the reason systemd can't find libmount is that for some reason making util-linux depend on linux-pam and shadow has caused it to list its version as UNKNOWN..0 | 09:54 |
| SotK | Kinnison: I believe you :) | 09:54 |
| Kinnison | SotK: it somehow magically happens in definitions.py | 09:55 |
| Kinnison | SotK: which is a tangle I have yet to grok | 09:55 |
| SotK | aha, I figured it must be in there somewhere but couldn't work out where | 09:59 |
| Kinnison | I think, basically, the stratum stuff pushes down into the dict which the chunk then fills out | 10:00 |
| Kinnison | in theory you might be able to put build-system et al in the stratum, but I'm not certain | 10:01 |
| *** persia_ has quit IRC | 10:06 | |
| * paulsherwood takes some perverse pleasure in thr thought that kinnison remains baffled by definitions.py :/ | 10:07 | |
| *** persia_ has joined #baserock | 10:07 | |
| *** persia_ has joined #baserock | 10:07 | |
| Albert_ | I'm looking at a problem with Genivi, specifically AudioManager. Its gets so far then falls down starting CAmNodeStateCommunicatorCAPI. If anyone has any experience with AudioManager that might be of help, or any ideas to thrown in, to speed things along, we would be most grateful. | 10:11 |
| pedroalvarez | Albert_: is this building audiomanager? using it? | 10:15 |
| pedroalvarez | Albert_: in both cases, some logs would help the readers to understand what is going on | 10:16 |
| Albert_ | This is a runtime error. | 10:20 |
| Albert_ | So I am going through the execution and getting to where I said. | 10:20 |
| Albert_ | I'll provide more info where I can | 10:21 |
| Albert_ | r | 10:28 |
| jonathanmaw | hrm, I can't replicate util-linux giving an UNKNOWN version when chrooting into a failed staging dir. | 10:33 |
| *** Krin has quit IRC | 10:37 | |
| tiagogomes_ | pedroalvarez, this happened http://paste.baserock.org/isuhexetoh | 10:41 |
| tiagogomes_ | I hope that f. (https://github.com/nvbn/thefuck) adds support for gerrit soon | 10:42 |
| pedroalvarez | tiagogomes_: git push origin HEAD:refs/for/master/your-topic (IIRC) | 10:43 |
| pedroalvarez | tiagogomes_: and that's what you tired after that | 10:43 |
| * pedroalvarez continues reading | 10:43 | |
| tiagogomes_ | yep | 10:43 |
| tiagogomes_ | And the topic branch name used to be optional | 10:45 |
| *** Krin has joined #baserock | 10:45 | |
| pedroalvarez | tiagogomes_: oh | 10:46 |
| pedroalvarez | try with `git push --no-thin <whatever else you need>` | 10:46 |
| pedroalvarez | Sam has mentioned this several times I think | 10:47 |
| tiagogomes_ | pedroalvarez, uff, that worked, thanks | 10:49 |
| pedroalvarez | I found the answer here: http://stackoverflow.com/a/20329880 | 10:50 |
| *** edcragg has quit IRC | 10:52 | |
| jonathanmaw | well, that's weird. during the build, it claims that "Not a git repository (or any of the parent directories): .git" | 10:53 |
| jonathanmaw | blew away my cached gits and tried again, that worked. | 10:55 |
| jonathanmaw | ¬_¬ | 10:55 |
| Albert_ | pedroalvarez, I have pasted the relevant bit of code from AudioManager where we get the failure. If you're familiar with this code then I hope you might have seen this before and have a magical solution :) Otherwise I'll thank you for looking and continue plugging on. | 10:58 |
| Albert_ | http://paste.baserock.org/bebemawade | 10:58 |
| Albert_ | In the meantime I'm trying to get to grips with gdb code dumps | 10:59 |
| Albert_ | s/code/core | 11:02 |
| jonathanmaw | \o/ systemd building! | 11:03 |
| Albert_ | Whoop! | 11:03 |
| pedroalvarez | jonathanmaw: I'm curious about what are you trying to achieve. Sound like you are moving some things to core. | 11:04 |
| ssam2 | PSA: with master of Morph, it's now possible to specify 'upgrade-type' and 'upgrade-location' in a cluster, so you can use the same cluster for initial deployment and live updates | 11:04 |
| jonathanmaw | pedroalvarez: getting systemd to use pam so that it starts user sessions | 11:04 |
| jonathanmaw | because gdp-hmi-launcher only writes debugging output to journald | 11:05 |
| jonathanmaw | since I have a journal, and it doesn't have anything from user units, this seems to be one of the things that needs systemd started properly | 11:05 |
| *** edcragg has joined #baserock | 11:08 | |
| pedroalvarez | jonathanmaw: thanks for clarifying :) makes sense to me to fix that | 11:10 |
| tiagogomes_ | Can I configure gerrit to not compare differences side by side, and showing line diffs instead? | 11:18 |
| pedroalvarez | tiagogomes_: I think so, never tried though | 11:19 |
| pedroalvarez | tiagogomes_: https://gerrit.baserock.org/#/settings/preferences | 11:19 |
| tiagogomes_ | I wonder why that is not on the Diff Preferences overlay window | 11:20 |
| * pedroalvarez was wondering the same | 11:20 | |
| pedroalvarez | but I also think that the side-by-side diff is always more useful | 11:21 |
| Kinnison | ssam2: All my thoughts keep returning to "just pickle it" | 11:54 |
| Kinnison | ssam2: If the major slowness is yaml.{dump,load} | 11:54 |
| ssam2 | regarding https://gerrit.baserock.org/#/c/343/ ? | 11:54 |
| Kinnison | aye | 11:55 |
| ssam2 | seems fair. Let's see what SotK thinks when he returns | 11:55 |
| Kinnison | It'll need encoding somehow | 11:55 |
| Kinnison | so someone should do a test | 11:55 |
| Kinnison | grab the data structure which is currently transferred by yaml | 11:55 |
| Kinnison | and compare yaml.dump+yaml.load vs. pickle+encode+decode+depickle | 11:55 |
| Kinnison | otherwise it's not fair | 11:56 |
| ssam2 | agreed. what's your view on merging https://gerrit.baserock.org/#/c/343/ and related changes in the meantime? it's maddening using distbuild without that branch! | 11:56 |
| * SotK doesn't have time to test the speed differences | 12:13 | |
| SotK | If possible I'd like to merge that branch and look at other ways in which we could improve it after the fact | 12:14 |
| jonathanmaw | *angry noises* systemd still isn't starting with users. | 12:15 |
| pedroalvarez | :( | 12:23 |
| Kinnison | ssam2: I'm happy for it to be merged in the meantime given that it does improve matters and we have no time right now :) | 12:24 |
| jonathanmaw | pedroalvarez: problem identified: shadow's autogen.sh sneaks a --without-pam in | 12:30 |
| *** fay_ has quit IRC | 12:31 | |
| Kinnison | jonathanmaw: without moving pam down under shadow, will that even work? | 12:31 |
| SotK | Kinnison: thanks, I'll send a patch to fix the typos and the like soon | 12:32 |
| SotK | s/patch/new version/ | 12:32 |
| pedroalvarez | jonathanmaw: git will blame me, I guess. | 12:32 |
| jonathanmaw | pedroalvarez, I doubt it. If you wanted to make changes to how to build, you'd do it in the morph files, not in autogen.sh | 12:33 |
| SotK | Does anyone know why this would fail? http://paste.baserock.org/ugifupezoz | 12:41 |
| *** zoli__ has quit IRC | 12:42 | |
| *** fay_ has joined #baserock | 12:51 | |
| *** Krin has quit IRC | 12:52 | |
| Albert_ | Further to my query re Audiomanager, I have a stack trace. I'm going to go through it now, but if you have a moment (jonathanmaw or pedroalvarez or anyone) maybe you'll spot something I wouldn't (having had no previous involvement). | 12:53 |
| Albert_ | http://paste.baserock.org/hevarinisi | 12:54 |
| *** Krin has joined #baserock | 12:54 | |
| Kinnison | Albert_: looks like a logical error in audiomanager somehow | 12:59 |
| Albert_ | really? | 13:03 |
| Kinnison | given the assert | 13:03 |
| Albert_ | sadly it's really far down in the call stack, so it's going to be hard to pin down | 13:06 |
| Albert_ | Mind you it's still in CAM | 13:07 |
| Albert_ | Actually, that is reported by audiomanager's output, but not visible in gdb. So we're back where we started in a way. | 13:10 |
| SotK | hmm, the gawk clone succeeded this time | 13:19 |
| pdar | A question regarding the baserock openstack, where will glance store images? | 13:29 |
| ssam2 | SotK: i've not seen that before, or not for a long time at least! | 13:31 |
| ssam2 | are you still stuck? | 13:31 |
| franred | pdar, /var/lib/glance/images? | 13:31 |
| SotK | ssam2: it seemed to work when I tried it again, I assumed I'd done something dumb | 13:32 |
| pedroalvarez | pdar: yes, there, but sometimes the images don't appear there | 13:32 |
| SotK | when have you seen it before? | 13:32 |
| ssam2 | just at random I think | 13:32 |
| pedroalvarez | pdar: I think that depending how you use glance they will appear there or not | 13:32 |
| ssam2 | actually, I don't think I've ever seen 'ERROR Command failed: git config --unset remote.origin.mirror' before | 13:32 |
| pdar | Ahh, ok i thought it might be there but when I looked there the cirros64 image was not there | 13:32 |
| pedroalvarez | pdar: I *think* that images uploaded using the url of the image, won't appear there | 13:34 |
| pedroalvarez | I havn't done any research though | 13:34 |
| pdar | ahh, ok, thats how i got it | 13:34 |
| pdar | so it would follow | 13:35 |
| pdar | Perhaps a silly question: If I want to deploy an image to my baserock opestack, will I have to make the baserock openstack-server image larger than the image I want to import? | 13:35 |
| pedroalvarez | pdar: yes, you have to | 13:37 |
| pedroalvarez | you can try `btrfs filesystem resize max /` on the openstack host | 13:37 |
| pedroalvarez | (you might have some space free on the disk not used by btrfs) | 13:38 |
| pdar | ooh, that may be what I was working up to asking | 13:38 |
| pedroalvarez | ssam2: anoyiingly, mounting a external disk using latest baserock, on a VM (x86) worked | 13:39 |
| ssam2 | bah | 13:39 |
| pdar | boom! that worked great thanks pedroalvarez and franred | 13:40 |
| SotK | did we not fix the GENIVI systems? | 13:46 |
| ssam2 | in what way? | 13:46 |
| SotK | referring to https://gerrit.baserock.org/#/c/404/ | 13:46 |
| SotK | they wouldn't build a while back, but I thought someone fixed that | 13:47 |
| radiofree | jjardon: we actually do release genivi systems | 13:47 |
| radiofree | what's broken about them? | 13:47 |
| radiofree | i think jonathanmaw is building them? | 13:47 |
| radiofree | can you have gerrit notify you when a new patch series is sent? | 13:48 |
| jjardon | ok, great, lets add them to the ci then? | 13:49 |
| SotK | radiofree: I think you have to add a project to your "Watched Projects" and configure it there | 13:49 |
| pedroalvarez | err | 13:49 |
| pedroalvarez | CI is going to explode | 13:49 |
| radiofree | is it too hot? | 13:50 |
| radiofree | SotK: thanks | 13:50 |
| pedroalvarez | but yeah, we can do that | 13:50 |
| radiofree | i do miss being able to flick through patches on a list | 13:50 |
| pedroalvarez | radiofree: you can do the same on gerrit :) | 13:50 |
| ssam2 | it's possible to set up 'dashboards' in gerrit, i know nothing about them right now but I think it lets you create custom views per-project | 13:51 |
| ssam2 | might need an admin to set them up though | 13:51 |
| radiofree | pedroalvarez: i don't constantly have gerrit open though | 13:51 |
| ssam2 | i have it pinned as a tab | 13:52 |
| * pedroalvarez plugs a SATA disk to a VM, and VM kernel panics when booting | 14:12 | |
| pedroalvarez | and this kids, is what happens when you start debugging an unrelated error | 14:13 |
| pedroalvarez | And this is the kernel panic: http://paste.baserock.org/raw/ocaruyijud | 14:23 |
| pedroalvarez | oh | 14:23 |
| pedroalvarez | is trying to boot from the disk that I've just attached | 14:24 |
| pedroalvarez | sorry for the noise | 14:24 |
| * pedroalvarez moves on | 14:24 | |
| Kinnison | :) | 14:26 |
| Zara | hi, I'm confused-- I'm trying to make a new system, but when I run morph build, I get 'ERROR: Couldn't find morphology: systems/openbmc-test-build-system.morph | 14:28 |
| Kinnison | you may need to add+commit brand-new morphologies | 14:29 |
| Kinnison | IIRC | 14:29 |
| bashrc_ | in the past I think I saw a similar error, where the morph clearly existed | 14:29 |
| bashrc_ | and yes I think the solution was to commit latest changes | 14:29 |
| Zara | hm, I think I have committed them-- at least, git status doesn't show it as a staged or untracked file | 14:30 |
| Zara | but it's there in 'definitions/systems/' | 14:31 |
| Kinnison | and the filename is correct and the 'name:' field inside matches? | 14:31 |
| ssam2 | what's your current working directory? | 14:31 |
| Zara | ssam2: /src/workspace/apr20/baserock/baserock/definitions | 14:33 |
| Zara | Kinnison: yes, I checked and double-checked and triple-checked just in case | 14:33 |
| Kinnison | :) | 14:33 |
| Kinnison | No idea then :( | 14:33 |
| jjardon | Zara: can you pastebin the contents of that commit? | 14:35 |
| Zara | jjardon: do you mean from git log? (if so, here: http://paste.baserock.org/oyojahidum ) | 14:37 |
| radiofree | ssam2: having the same problem as you with new systemd | 14:38 |
| radiofree | [ *] A start job is running for dev-disk...b850a07f.device (12s / 1min 30s) | 14:38 |
| jjardon | Zara: I meant 'git show' or 'git diff master' :) | 14:38 |
| pedroalvarez | radiofree: I was starting to have a look at it | 14:39 |
| Zara | heh, ah, I've found what's going on-- it's working from an older commit. | 14:39 |
| Zara | I made a new branch, and committed there, so I guess it confused morph. | 14:39 |
| Zara | so I can do a morph checkout from my github and work from that, and that ought to fix it. | 14:40 |
| Zara | grr, this has caught me out before | 14:40 |
| pedroalvarez | Is anybody against killing workspaces? | 14:41 |
| pdar | why were they necessary? | 14:41 |
| radiofree | i don't even get a recovery console though | 14:41 |
| radiofree | just hangs there | 14:41 |
| * radiofree boots into factory to manually edit the fstab | 14:42 | |
| straycat | pdar, there's a concept of a system branch and having all repos in your workspace set to that system branch, morph can push local changes to the trove, for building/distbuilding | 14:42 |
| straycat | getting rid of system branches is a pretty big change though | 14:43 |
| radiofree | hmm actually it's not /src it's hanging on :\ | 14:43 |
| Zara | is there a quick command to tell morph to work from the branch currently checked out in git? | 14:44 |
| straycat | the various improvements to morph deploy possibly just saved me 30 minutes | 14:44 |
| straycat | Zara, don't set local-changes=ignore in your morph.conf ? | 14:45 |
| pedroalvarez | radiofree: isn't it? | 14:46 |
| Zara | straycat: ah, it's probably just set to whatever the default is on the wiki | 14:46 |
| Zara | straycat: though hm, there's nothing about local-changes in my morph.conf | 14:47 |
| Zara | so I'm guessing that default is set elsewhere? | 14:48 |
| pedroalvarez | Zara: I sometmes do a random change in a random file in definitions.git to workaround this bug | 14:48 |
| radiofree | [ 16.032032] systemd[117]: /lib/systemd/system-generators/systemd-gpt-auto-generator failed with error code 1. | 14:48 |
| radiofree | what does that mean? | 14:48 |
| SotK | Zara: the default is include | 14:48 |
| Kinnison | radiofree: gpt is a partitioning scheme | 14:48 |
| Kinnison | radiofree: I imagine it's meant to generate units for gpt partitions | 14:49 |
| straycat | Zara, i thought the default was to include local changes, you can set local-changes=include to be sure | 14:49 |
| SotK | making a new branch is what caused the problem, morph builds the ref defined in a config file in your system branch | 14:49 |
| radiofree | hmm.. well this new image doesn't boot at all now | 14:49 |
| radiofree | [ TIME ] Timed out waiting for device dev-di...\x2d9b01\x2d9370b850a07f.device. | 14:49 |
| radiofree | [DEPEND] Dependency failed for /root. | 14:49 |
| radiofree | http://fpaste.org/213863/27816142/ | 14:50 |
| ssam2 | radiofree: can you roll back to the old version? for me, the old version still worked if I selected that one in the U-Boot boot menu | 14:50 |
| radiofree | ssam2: yep, old version works fine | 14:50 |
| radiofree | i think i'll just add systemd on top of the system and redeploy | 14:50 |
| radiofree | (old systemd) | 14:50 |
| *** Krin has quit IRC | 14:51 | |
| radiofree | my deploy cluster was wrong | 14:53 |
| radiofree | http://fpaste.org/213867/62798214/ | 14:53 |
| * radiofree wonders how this worked at all | 14:53 | |
| paulsherwood | baserock mentioned in the news... http://www.visteon.com/media/newsroom/2015/150421_story1.html | 14:53 |
| ssam2 | radiofree: argh, nasty | 14:53 |
| ssam2 | radiofree: i found the other day that I had a Baserock rootfs, or at least part of one, installed to the ext2 boot partition | 14:54 |
| *** Krin has joined #baserock | 14:54 | |
| pedroalvarez | paulsherwood: great! | 14:55 |
| robtaylo1 | \o/ | 14:55 |
| radiofree | i'm not sure *how* it worked though, it wrote the extlinux.conf correctly | 14:59 |
| ssam2 | does it read BOOT_DEVICE from /baserock/deployment.meta if it can? | 15:00 |
| radiofree | i was under the impression it would use the deployment details from the deployed upgrade | 15:00 |
| pedroalvarez | extlinunx.conf doesn't look fine on this jetston I'm debugging | 15:01 |
| radiofree | but yes it probably ended up falling back to BOOT_DEVICE on factory | 15:01 |
| radiofree | which is an amazing fail-safe mechanism i totally intended to implement | 15:01 |
| pedroalvarez | meh, it does looks ok | 15:01 |
| pedroalvarez | radiofree: so, there shouldnt be any "systems" folders on the BOOT_DEVICE, right?: | 15:04 |
| radiofree | no it created them fine | 15:05 |
| pedroalvarez | I moved that folder, and then everything fails | 15:05 |
| radiofree | it gets as far as booting and mounting the rootfs | 15:05 |
| radiofree | then http://fpaste.org/213863/27816142/ happens | 15:05 |
| radiofree | i still think it's a systemd problem, redeploying now though | 15:05 |
| radiofree | i'm just amazed it worked at all | 15:06 |
| pedroalvarez | I think it was trying to mount btrfs subvolumes from a ext4 partition | 15:06 |
| pedroalvarez | ext2, or whatever | 15:06 |
| radiofree | i don't think so? | 15:06 |
| radiofree | my kernel command line was /dev/mmcblk0p2 | 15:07 |
| radiofree | ROOT_DEVICE was set correctly, and for some reason BOOT_DEVICE ended up being correct | 15:07 |
| radiofree | i don't think BOOT_DEVICE has any influence over the system? other than to instruct system-version-manager where to put the kernel/where to edit extlinux.conf | 15:07 |
| radiofree | which it did... | 15:07 |
| pedroalvarez | radiofree: I removed the "systems" folder from BOOT_DEVICE and now this is really broken | 15:07 |
| radiofree | yes because it won't be able to read the kernel | 15:08 |
| radiofree | BOOT_DEVICE systems folder is just a copy of the btrfs structure, for simplicities sake | 15:08 |
| radiofree | if you remove that then u-boot will try and read a non-existent kernel/device tree | 15:08 |
| pedroalvarez | :) | 15:09 |
| radiofree | deployed again without the typo, let's see what happens | 15:09 |
| radiofree | nope :( | 15:10 |
| radiofree | rebuilding systemd now | 15:22 |
| pedroalvarez | radiofree: so, have you reverted the systemd upgrade? | 15:25 |
| radiofree | i'm not going to rebuild from foundation, i just added a new "systemdrev" stratum with the old sha | 15:28 |
| pedroalvarez | well, yes, my question wasn't really well phrased | 15:30 |
| radiofree | it's building now, takes about 20 minutes | 15:30 |
| ssam2 | might be useful to build it locally in the node and 'git bisect' | 15:31 |
| *** paulw has quit IRC | 15:32 | |
| ssam2 | i'm pretty sure that SHA1 a88abde72169ddc2df77df3fa5bed30725022253 (v219) will work | 15:32 |
| Kinnison | Hmm, after the openstack work was merged there was a lot of name duplication introduced ;( | 15:39 |
| Kinnison | paulsherwood: Are you *sure* erlang doesn't get built for you when you use ybd to build build-system-x86_64 ? | 15:40 |
| Kinnison | If so, then there's a non-determinism issue in ybd | 15:41 |
| SotK | Kinnison: I was wondering about that issue when looking at definitions.py this morning | 15:42 |
| SotK | my wild guess was what you're implying, in that the first chunk into the definitions dict varied each time | 15:43 |
| Kinnison | mmm | 15:43 |
| SotK | I didn't test or investigate properly though | 15:43 |
| Kinnison | I think it might be dependent on the order in which things come off the filesystem | 15:43 |
| Kinnison | but that would really make me sad | 15:43 |
| SotK | yeah, it does os.walk() on the cwd | 15:44 |
| * SotK doesn't know if that can come out in a random order or not | 15:45 | |
| Kinnison | it likely comes out in dirent order | 15:46 |
| Kinnison | which is filesystem dependent | 15:46 |
| Zara | Does anyone know why I'd get an error saying 'field build commands not allowed in morphology'? | 15:46 |
| SotK | does it tell you which morphology? | 15:46 |
| Zara | yeah, it's one I've just made so I've probably done something weird, just wondering if anyone knows what that error generally means | 15:47 |
| Kinnison | is the morphology in question a chunk? | 15:48 |
| *** Krin has quit IRC | 15:48 | |
| ssam2 | 'build commands' or 'build-commands' ? | 15:48 |
| SotK | oh, the field is 'build-commands' not 'build commands' | 15:48 |
| SotK | snap | 15:48 |
| Zara | ahhhh, hahaha | 15:48 |
| Zara | yes, that will be it | 15:48 |
| Zara | thanks | 15:48 |
| Zara | ^^' | 15:48 |
| * Kinnison thinks error messages which quote part of the input should quote their quotes | 15:48 | |
| persia | +1 | 15:48 |
| Kinnison | An error of the form: "field 'build commands' not allowed in morphology" would be clearer | 15:49 |
| Zara | yeah, I thought it was saying it was some kind of special morphology that couldn't have any buid commands | 15:49 |
| Zara | and I didn't know why it would be special | 15:49 |
| ssam2 | interesting new distbuild bug: progress messages taking almost 5 minutes to get from the controller to the initiator | 15:49 |
| ssam2 | is it possible that this is due to port forwarding being done by 'firehol'? | 15:50 |
| Kinnison | Though it puts paid to a statement on the definitions page on the wiki which says: | 15:50 |
| Kinnison | "Note that currently, unknown keys in definitions are silently ignored. | 15:50 |
| Kinnison | " | 15:50 |
| Kinnison | clearly they're not | 15:50 |
| radiofree | oh, building systemd failed :( | 15:50 |
| radiofree | gcc: error: /lib/libacl.so: No such file or directory | 15:50 |
| SotK | yeah, unknown keys cause that error | 15:50 |
| Kinnison | ssam2: seems very unlikely | 15:50 |
| ssam2 | the message is logged as sent by the InitiatorConnection at 15:31 in the controller log, and received at 15:35 in the initiator log | 15:51 |
| ssam2 | never seen this before | 15:51 |
| Kinnison | were other messages passed around? | 15:51 |
| ssam2 | in the meantime? no | 15:51 |
| Kinnison | hmm | 15:51 |
| Kinnison | is the initiator setting the tcp flags to keep the socket alive? | 15:52 |
| Kinnison | if not, address translation and connection tracking could cause confusing delays | 15:52 |
| Kinnison | Hmm, the default keepalive is 2 hours | 15:52 |
| Kinnison | so NAT is unlikely to be causing it | 15:53 |
| Kinnison | (I really should read the manpage before asking silly questions) | 15:53 |
| *** a1exhughe5 has quit IRC | 15:53 | |
| ssam2 | if the connection dropped the build would stop | 15:53 |
| pedroalvarez | radiofree: I had that error when building master of systemd, which one are you trying to build? | 15:53 |
| ssam2 | I wonder if this is distbuild.sockbuf in action | 15:54 |
| Kinnison | if there's buffers which aren't being flushed then it's possible I guess | 15:54 |
| ssam2 | it doesn't seem to have any 'flush buffer after this number of seconds' timeout | 15:54 |
| ssam2 | and I did remove a huge message that did used to get sent at the start of each build, a while back | 15:54 |
| * ssam2 looks forward to sending even more last-minute patches :( | 15:55 | |
| radiofree | pedroalvarez: a88abde72169ddc2df77df3fa5bed30725022253 | 15:55 |
| radiofree | i.e the old version of systemd | 15:56 |
| Kinnison | ssam2: speaking of last minute patches, anything you need reviews on still? | 15:56 |
| radiofree | i have strata/systemd.morph, which build-depends on foundation | 15:56 |
| radiofree | and contains a systemd chunck with that ref | 15:56 |
| Kinnison | Grrr | 15:56 |
| Kinnison | Please stop having multiple definitions of the same name | 15:56 |
| Kinnison | it makes me very sad | 15:56 |
| Kinnison | because I have to kill a kitten every time I see one, and I like kittens | 15:56 |
| pedroalvarez | radiofree: this error might be because now systemd is being built with more dependencies | 15:56 |
| pedroalvarez | ei, the whole foundation stratum | 15:57 |
| * radiofree weeps uncontrollably | 15:58 | |
| pedroalvarez | meh, sometimes puting a chunk on the top is not that easy | 15:58 |
| straycat | hrm | 15:58 |
| ssam2 | Kinnison: indeed, the biggest one is Adam's OStree patch series | 15:59 |
| ssam2 | Kinnison: that is awaiting a few fixes from Adam, but bar a couple of issues, is testable as-is | 15:59 |
| ssam2 | looking at distbuild.sockbuf, it doesn't seem to have anything obvious that would cause a delay. It writes as soon as it receives data | 16:00 |
| pedroalvarez | radiofree: --disable-acl might help | 16:00 |
| * SotK is running the test suite before rebasing and sending a new version for ostree | 16:00 | |
| Kinnison | I'm not in a position to run much, so I am kinda limited to stuff I can review from a readability PoV only | 16:00 |
| radiofree | pedroalvarez: thanks, i'll give that a go | 16:02 |
| ssam2 | Kinnison: that's useful too :) | 16:03 |
| * Kinnison kicks off another build and gets himself into a position to review | 16:04 | |
| jjardon | Hi, compiling a ARMv5 tarball in a ARMv7 hardware (running native-bootstrap script), I get a compilation error about a missing "__gmpn_invert_limb" . There is a comment in the morph file to workaround this, but it doesnt seems to work here. Reading the conversation here: | 16:12 |
| jjardon | http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-January/010884.html was the bug filed upstream at some point? | 16:12 |
| jjardon | tiagogomes_: ^ | 16:12 |
| tiagogomes_ | jjardon, IIRC I tried to create an account on gcc bugzilla to report the bug as requested by Sam, but without success | 16:15 |
| ssam2 | oh, max_buffer is 16384 on the controller->initiator socket ... that may be why distbuild messages are getting held up! | 16:19 |
| ssam2 | I'm not sure about lowering that value though, would it cause big messages to send slower? | 16:20 |
| * ssam2 is largely ignorant about TCP sockets | 16:20 | |
| Kinnison | By the time you're at TCP you're into other stuff entirely | 16:20 |
| Kinnison | and TCP isn't likely to cause these delays unless you're corking and not uncorking | 16:21 |
| ssam2 | what does 'corking and not uncorking' mean? | 16:21 |
| Kinnison | In TCP you can cork the socket which stops it transmitting to the underlying network | 16:22 |
| rjek | ssam2: corking means "don't send anything yet, I'm about to give you more data, so you can send it all in one packet" | 16:22 |
| Kinnison | then you write() as many times as you like | 16:22 |
| rjek | Uncorking means "OK, send it now" | 16:22 |
| Kinnison | and then you uncork and it sends it as efficiently as it can | 16:22 |
| Kinnison | typically only used by seriously high-performance stuff | 16:22 |
| ssam2 | I don't see a cork() function anywhere | 16:22 |
| rjek | ie, if you're making several write() calls, you cork the socket to stop it popping. | 16:22 |
| rjek | It's an ioctl IIRC | 16:22 |
| ssam2 | none of those in Morph either, so I guess it's not that | 16:23 |
| Kinnison | or a sockopt | 16:23 |
| rjek | Oh no, get/setsockopt | 16:23 |
| rjek | TCP_CORK | 16:23 |
| ssam2 | only one, which is SO_REUSEADDR | 16:23 |
| rjek | See man 7 tcp | 16:23 |
| rjek | (which covers TCP-specifics) | 16:24 |
| rjek | (as sockets are protocol-generic.) | 16:24 |
| ssam2 | thanks | 16:25 |
| ssam2 | maybe I need to set TCP_NODELAY on the sockets? | 16:26 |
| ssam2 | but that seems like it might cause performance issues.. | 16:26 |
| rjek | You can also avoid small writes | 16:26 |
| Kinnison | It's distinctly more likely that the delay is *inside* the python code somewhere | 16:26 |
| Kinnison | i.e. that you've a buffer not being flushed | 16:26 |
| rjek | The Nagle algorithm disabled by TCP_NODELAY is designed to protect against naive software that writes to sockets a byte at a time | 16:26 |
| Kinnison | Unless you're doing something *VERY* odd, you'll never manage to get a 5 minute delay on a tcp socket itself | 16:27 |
| rjek | Plus the Nagle algorithm's delay is going to be less than cork's 200ms | 16:27 |
| rjek | (otherwise they'd be no point in corking) | 16:27 |
| ssam2 | Kinnison: ah, good to know | 16:28 |
| ssam2 | I'm still combing through state machine transitions in a log file | 16:28 |
| ssam2 | but it does seem that the sockbuf object starts writing to the socket as soon as it receives data and the socket is writable | 16:28 |
| rjek | 5 minute delay on TCP? I'm not sure that's even possible: there's the back-off during packet loss, but I think the maximum RTT for TCP before it gives up is <5m | 16:29 |
| Kinnison | rjek: You could have fun with fragments and the keepalive limits | 16:29 |
| rjek | (Which is why TCP needs replacing before we colonise Mars) | 16:29 |
| radiofree | pedroalvarez: thanks, that worked | 16:34 |
| radiofree | old systemd works fine | 16:34 |
| pedroalvarez | i was goin to try latest master, but i dont have cache | 16:35 |
| pedroalvarez | radiofree: are you planning to continue debugging this? | 16:38 |
| Kinnison | rjek: maybe QUIC will help for mars | 16:39 |
| * Kinnison gdl&rs | 16:39 | |
| radiofree | pedroalvarez: absolutely not | 16:40 |
| radiofree | pedroalvarez: in fact i've not wasted so much time on it i can't go about doing what i wanted to do (upgrading jetson to 4.0) | 16:40 |
| pedroalvarez | radiofree: ok, I will try to debug this then | 16:43 |
| radiofree | pedroalvarez: i'll probably have some time to do it thursday/friday, if it can wait | 16:43 |
| radiofree | we could just cherry-pick that networking commit of richards on-top of the 219 release? | 16:44 |
| pedroalvarez | radiofree: yeah, but I think there were more things needed for openstack | 16:45 |
| pedroalvarez | I mean, not only that commit | 16:45 |
| *** Guest21191 has quit IRC | 16:55 | |
| radiofree | bummer | 16:57 |
| pedroalvarez | 1. A word describing the misfortune of something or someone. | 16:59 |
| pedroalvarez | I think there are 400ish commits from the current sha1 to 219 | 16:59 |
| pedroalvarez | that means 10ish git bisect iterations | 17:00 |
| pedroalvarez | around 5 hours | 17:00 |
| pedroalvarez | yeah, bummer | 17:01 |
| jjardon | http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/foundation.morph#n84: exactly 314 | 17:01 |
| *** bashrc_ has quit IRC | 17:01 | |
| pedroalvarez | oh! useful data in the unpetrify ref! | 17:02 |
| rjek | who'd a thunk it | 17:02 |
| jjardon | pedroalvarez: :) | 17:02 |
| jjardon | Id try current master, the systemd we are currently using is quite old already (mid March) and in the middle of the development cycle, so probably quite unstable | 17:04 |
| pedroalvarez | jjardon: yeah, I wanted to try that first | 17:04 |
| ssam2 | ok idea, but that might introduce new bugs again | 17:04 |
| ssam2 | do we feel lucky? :) | 17:05 |
| *** franred has quit IRC | 17:05 | |
| radiofree | i suppose i could try that now | 17:06 |
| radiofree | just point my systemd stratum at master then? | 17:06 |
| jjardon | come one!, What could possibly go wrong? :) | 17:06 |
| pedroalvarez | oh! :) | 17:06 |
| ssam2 | nothing more than already has | 17:06 |
| ssam2 | ha! just kidding! | 17:06 |
| * pedroalvarez will wait for that test | 17:06 | |
| radiofree | about 30 minutes | 17:06 |
| radiofree | build + deploy | 17:07 |
| *** franred has joined #baserock | 17:07 | |
| ssam2 | It's pretty clear that this 'delayed messages' distbuild bug is definitely that the socket in the controller to the initiator doesn't become writable for 3 minutes after we are ready to write to it | 17:07 |
| jjardon | looking at the history, systemd release are usually every 2 months; last one was 2 months ago so Id expect current master to be quite stable | 17:07 |
| radiofree | master of systemd! how exciting! | 17:07 |
| pedroalvarez | the should be releasing 220 soon, no? | 17:10 |
| Kinnison | ssam2: that's surprising | 17:10 |
| Kinnison | ssam2: is the socket in the set for testing for writability all the time, or only after it has stuff put in it? | 17:10 |
| ssam2 | only after it has stuff put in it | 17:10 |
| ssam2 | but that is logged 3 minutes before it becomes writable | 17:10 |
| jjardon | pedroalvarez: I think so, yes. We can ask to be sure though | 17:10 |
| jonathanmaw | uh, I accidentally sent an E-mail to baserock-internal instead of baserock-dev by accident | 17:12 |
| jonathanmaw | more annoyingly, I don't have a "sent" folder in my mail client, so I can't easily resend it to baserock-dev | 17:13 |
| *** jonathanmaw has quit IRC | 17:15 | |
| ssam2 | 'any reasonably healthy socket will return as writable - it just means outbound network buffer space is available' | 17:16 |
| ssam2 | I wonder if that is a shared buffer -- in that case, it may be huge artifact graphs being sent to the workers that are delaying this message | 17:16 |
| *** lachlanmackenzie has quit IRC | 17:16 | |
| Kinnison | No, it's per-socket | 17:18 |
| Kinnison | unless something very odd is going on | 17:18 |
| ssam2 | radiofree: if you're still about, could you paste the morph file you used to build systemd on top of all the other strata? | 17:23 |
| ssam2 | i have a feeling i'm going to need to do the same | 17:23 |
| Kinnison | ssam2: good luck with that socket issue | 17:23 |
| * Kinnison has to head off | 17:23 | |
| radiofree | ssam2: yep hold on | 17:25 |
| ssam2 | updating to the artifact serialisation speedups branch *has* made the problem go away... so while I'd like to know what is actually going on, I think i'll just hope for the best on this one, for the time being | 17:26 |
| *** edcragg has quit IRC | 17:27 | |
| radiofree | ssam2: strata/systemdrev.morph http://fpaste.org/213952/14296371/ | 17:27 |
| * SotK wishes the test suite would hurry up and finish running | 17:27 | |
| radiofree | ssam2: strata/foundation/systemdrev.morph http://fpaste.org/213953/37233142/ | 17:27 |
| ssam2 | cheers | 17:27 |
| radiofree | deploying now! | 17:28 |
| radiofree | ssam2: a88abde72169ddc2df77df3fa5bed30725022253 works fo'sure | 17:28 |
| radiofree | master works \o/ | 17:33 |
| pedroalvarez | Yay | 17:33 |
| pedroalvarez | 5 hours saved! | 17:34 |
| radiofree | heh | 17:34 |
| * radiofree heads off | 17:35 | |
| pedroalvarez | Thanks radiofree | 17:35 |
| * SotK starts typing `git push` and remembers he didn't pull before he rebased :( | 17:35 | |
| *** Albert_ has quit IRC | 17:39 | |
| *** gary_perkins has quit IRC | 17:44 | |
| *** zoli__ has joined #baserock | 18:04 | |
| *** ssam2 has quit IRC | 18:32 | |
| *** ssam2 has joined #baserock | 18:35 | |
| *** ChanServ sets mode: +v ssam2 | 18:35 | |
| *** Albert has joined #baserock | 18:37 | |
| *** Albert has quit IRC | 18:53 | |
| *** rdale has quit IRC | 19:09 | |
| * SotK decides commands which take REPO REF FILENAME should work anywhere | 20:32 | |
| *** zoli__ has quit IRC | 21:32 | |
| *** ssam2 has quit IRC | 22:14 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!