IRC logs for #baserock for Tuesday, 2015-04-21

*** rdale has quit IRC00:18
*** rdale has joined #baserock00:21
*** rdale has quit IRC01:32
*** rdale has joined #baserock01:34
paulsherwoodKinnison: not afaik?01:42
*** rdale has quit IRC02:34
*** zoli__ has joined #baserock04:01
*** zoli__ has quit IRC04:20
*** zoli__ has joined #baserock04:22
*** zoli__ has quit IRC04:52
*** petefoth has quit IRC05:11
*** zoli__ has joined #baserock05:11
*** petefoth has joined #baserock06:25
*** mike has joined #baserock06:58
*** mike is now known as Guest2119106:58
*** a1exhughe5 has joined #baserock07:05
*** Guest21191 has quit IRC07:12
*** Guest21191 has joined #baserock07:22
*** Albert_ has joined #baserock07:30
*** petefoth has left #baserock07:45
radiofreejjardon: i believe the rawdisk images install the bootloader by running extlinux on the *host*, so you'll need to build an image with the image you've built07:46
*** petefoth has joined #baserock07:48
* SotK finds that the build-mode is in the stratum, not the chunk morph07:54
SotKhaving data for a chunk split over two files is really annoying07:54
SotKalso, this means that for any given chunk artifact, artifact.source.morphology['build-mode'] == 'staging'07:55
paulsherwoodSotK: see ybd for simplest logic i could get to08:02
*** bashrc_ has joined #baserock08:05
SotKpaulsherwood: whereabouts does ybd handle build-mode?08:07
*** rdale has joined #baserock08:08
*** jonathanmaw has joined #baserock08:08
*** mwilliams_ct has joined #baserock08:09, assemlby.py08:17
jjardonradiofree: yep, I've done that as well08:24
*** edcragg has joined #baserock08:30
SotKpaulsherwood: That seems to just try to get 'build-mode' but default to 'staging' if its not there, I can't see where it sets build-mode for each component though?08:31
jonathanmawI'm getting this error when building tar:
jonathanmawit looks like tar's pre-configure commands try to init submodules, but the path to the git repo isn't in the chroot.08:33
jonathanmawwhich shouldn't be necessary, since morph has already checked out the submodules08:34
jonathanmawbootstrap has options for specifying where the submodules' sourcedirs are, so I think I'll use those.08:37
*** gary_perkins has joined #baserock08:37
radiofreejjardon: if it boots then i guess it works!08:38
*** ssam2 has joined #baserock08:38
*** ChanServ sets mode: +v ssam208:38
jonathanmawI am quite puzzled how tar ever built.08:52
jonathanmawespecially since tar's bootstrap thinks that it still needs to fiddle with submodules.08:53
jonathanmaweven when --gnulib-srcdir is defined.08:53
pedroalvarezjonathanmaw: A build log, if that helps:
jonathanmawpedroalvarez: which ref is used for that?08:55
jonathanmawi.e. which ref of tar?08:56
franredgood morning, there are a patch-series which would be nice if someone can have a look today
pedroalvarezjonathanmaw: 9a58d148c26c220cb1b163c71e7a51a2e41f6b3708:58
pedroalvarezI migh be confusing some metadata fields, let me check08:59
jonathanmawthat looks like the same ref I was using08:59
ssam2franred: look fine to me08:59
franredssam2, thank you!!08:59
pedroalvarezjonathanmaw: the error sounds like /src/cache/gits/git___git_baserock_org_delta_tar is corrupted?09:00
jonathanmawpedroalvarez: nope, it looks fine. the problem is that it's looking for that dir from inside a chroot09:00
ssam2`git fsck` in there will tell you for sure09:00
pedroalvarezjonathanmaw: wow09:01
jonathanmawgit fsck in git___git_baserock_org_delta_tar seems fine09:01
KinnisonSotK: What are you looking at build-mode for?09:02
KinnisonSotK: ybd assumes 'staging' build mode unless told otherwise09:02
SotKI'm looping through a set of chunks (with the intention of finding the upstream URL for each one) and wanted to ignore bootstrap chunks. However, using chunk.source.morphology['build-mode'] always gave me 'staging' because the build-mode is defined in the stratum rather than the chunk.09:04
Kinnisonthat's (IMO) a failing of morph's internal model09:05
SotKI agree09:05
KinnisonOnce we have definitions *fully* specified09:05
Kinnisonat that point we can try and work out a more idealised model for morph to use internally09:05
KinnisonUntil that point, we're throwing darts in the dark, in a room filled with acid balloons09:06
petefothKinnison: isn't that a bit dangerous? ;)09:07
Kinnisonpetefoth: best not do it then09:07
SotKKinnison: do you mean we fully specify the current layout, or decide upon a final layout?09:08
jonathanmawwhee! the solution was to remove .gitmodules in pre-configure-commands09:08
KinnisonUntil we know what we have, how can we possibly know what we want?09:08
*** Krin has joined #baserock09:08
*** a1exhughe5 has quit IRC09:08
*** a1exhughe5 has joined #baserock09:09
SotKaha, I can actually see that happening sometime then :)09:09
ssam2it's started here already, just needs everyone to help tie the details down:
ssam2there are quite a lot of details :(09:12
*** lachlanmackenzie has joined #baserock09:15
*** flatmush has quit IRC09:15
jonathanmawsystemd has decided it needs libmount.09:17
jonathanmawbrief googling suggests that util-linux should be providing this09:19
jjardonjonathanmaw: I think that's in util-linux09:19
*** flatmush has joined #baserock09:20
ssam2I have some patches that it'd be really handy if people could review them!09:25
ssam2 and are both trivial and both have +1 already09:25
ssam2 and the two related changes I hope are easy to understand and have a clear benefit09:26
paulsherwoodSotK: do you believe that something else is required, beyond that logic?09:26
ssam2and and 2 related changes, which fixes a bug in distbuild builds aren't actually always cancelled when expected09:27
SotKpaulsherwood: Unless ybd puts build-mode into the chunk morphologies, I couldn't see how they would ever be built as anything except 'staging', but I could be missing something.09:28
*** ssam2 has quit IRC09:44
*** ssam2 has joined #baserock09:44
*** ChanServ sets mode: +v ssam209:44
KinnisonSotK: believe me, we build stuff in bootstrap mode :)09:53
jonathanmawhrm, the reason systemd can't find libmount is that for some reason making util-linux depend on linux-pam and shadow has caused it to list its version as UNKNOWN..009:54
SotKKinnison: I believe you :)09:54
KinnisonSotK: it somehow magically happens in definitions.py09:55
KinnisonSotK: which is a tangle I have yet to grok09:55
SotKaha, I figured it must be in there somewhere but couldn't work out where09:59
KinnisonI think, basically, the stratum stuff pushes down into the dict which the chunk then fills out10:00
Kinnisonin theory you might be able to put build-system et al in the stratum, but I'm not certain10:01
*** persia_ has quit IRC10:06
* paulsherwood takes some perverse pleasure in thr thought that kinnison remains baffled by :/10:07
*** persia_ has joined #baserock10:07
*** persia_ has joined #baserock10:07
Albert_I'm looking at a problem with Genivi, specifically AudioManager. Its gets so far then falls down starting CAmNodeStateCommunicatorCAPI. If anyone has any experience with AudioManager that might be of help, or any ideas to thrown in, to speed things along, we would be most grateful.10:11
pedroalvarezAlbert_: is this building audiomanager? using it?10:15
pedroalvarezAlbert_: in both cases, some logs would help the readers to understand what is going on10:16
Albert_This is a runtime error.10:20
Albert_So I am going through the execution and getting to where I said.10:20
Albert_I'll provide more info where I can10:21
jonathanmawhrm, I can't replicate util-linux giving an UNKNOWN version when chrooting into a failed staging dir.10:33
*** Krin has quit IRC10:37
tiagogomes_pedroalvarez, this happened
tiagogomes_I hope that f. ( adds support for gerrit soon10:42
pedroalvareztiagogomes_: git push origin HEAD:refs/for/master/your-topic (IIRC)10:43
pedroalvareztiagogomes_: and that's what you tired after that10:43
* pedroalvarez continues reading10:43
tiagogomes_And the topic branch name used to be optional10:45
*** Krin has joined #baserock10:45
pedroalvareztiagogomes_: oh10:46
pedroalvareztry with `git push --no-thin  <whatever else you need>`10:46
pedroalvarezSam has mentioned this several times I think10:47
tiagogomes_pedroalvarez, uff, that worked, thanks10:49
pedroalvarezI found the answer here:
*** edcragg has quit IRC10:52
jonathanmawwell, that's weird. during the build, it claims that "Not a git repository (or any of the parent directories): .git"10:53
jonathanmawblew away my cached gits and tried again, that worked.10:55
Albert_pedroalvarez, I have pasted the relevant bit of code from AudioManager where we get the failure. If you're familiar with this code then I hope you might have seen this before and have a magical solution :) Otherwise I'll thank you for looking and continue plugging on.10:58
Albert_In the meantime I'm trying to get to grips with gdb code dumps10:59
jonathanmaw\o/ systemd building!11:03
pedroalvarezjonathanmaw: I'm curious about what are you trying to achieve. Sound like you are moving some things to core.11:04
ssam2PSA: with master of Morph, it's now possible to specify 'upgrade-type' and 'upgrade-location' in a cluster, so you can use the same cluster for initial deployment and live updates11:04
jonathanmawpedroalvarez: getting systemd to use pam so that it starts user sessions11:04
jonathanmawbecause gdp-hmi-launcher only writes debugging output to journald11:05
jonathanmawsince I have a journal, and it doesn't have anything from user units, this seems to be one of the things that needs systemd started properly11:05
*** edcragg has joined #baserock11:08
pedroalvarezjonathanmaw: thanks for clarifying :) makes sense to me to fix that11:10
tiagogomes_Can I configure gerrit to not compare differences side by side, and showing line diffs instead?11:18
pedroalvareztiagogomes_: I think so, never tried though11:19
tiagogomes_I wonder why that is not on the Diff Preferences overlay window11:20
* pedroalvarez was wondering the same11:20
pedroalvarezbut I also think that the side-by-side diff is always more useful11:21
Kinnisonssam2: All my thoughts keep returning to "just pickle it"11:54
Kinnisonssam2: If the major slowness is yaml.{dump,load}11:54
ssam2regarding ?11:54
ssam2seems fair. Let's see what SotK thinks when he returns11:55
KinnisonIt'll need encoding somehow11:55
Kinnisonso someone should do a test11:55
Kinnisongrab the data structure which is currently transferred by yaml11:55
Kinnisonand compare yaml.dump+yaml.load vs. pickle+encode+decode+depickle11:55
Kinnisonotherwise it's not fair11:56
ssam2agreed. what's your view on merging and related changes in the meantime? it's maddening using distbuild without that branch!11:56
* SotK doesn't have time to test the speed differences12:13
SotKIf possible I'd like to merge that branch and look at other ways in which we could improve it after the fact12:14
jonathanmaw*angry noises* systemd still isn't starting with users.12:15
Kinnisonssam2: I'm happy for it to be merged in the meantime given that it does improve matters and we have no time right now :)12:24
jonathanmawpedroalvarez: problem identified: shadow's sneaks a --without-pam in12:30
*** fay_ has quit IRC12:31
Kinnisonjonathanmaw: without moving pam down under shadow, will that even work?12:31
SotKKinnison: thanks, I'll send a patch to fix the typos and the like soon12:32
SotKs/patch/new version/12:32
pedroalvarezjonathanmaw: git will blame me, I guess.12:32
jonathanmawpedroalvarez, I doubt it. If you wanted to make changes to how to build, you'd do it in the morph files, not in autogen.sh12:33
SotKDoes anyone know why this would fail?
*** zoli__ has quit IRC12:42
*** fay_ has joined #baserock12:51
*** Krin has quit IRC12:52
Albert_Further to my query re Audiomanager, I have a stack trace. I'm going to go through it now, but if you have a moment (jonathanmaw or pedroalvarez or anyone) maybe you'll spot something I wouldn't (having had no previous involvement).12:53
*** Krin has joined #baserock12:54
KinnisonAlbert_: looks like a logical error in audiomanager somehow12:59
Kinnisongiven the assert13:03
Albert_sadly it's really far down in the call stack, so it's going to be hard to pin down13:06
Albert_Mind you it's still in CAM13:07
Albert_Actually, that is reported by audiomanager's output, but not visible in gdb. So we're back where we started in a way.13:10
SotKhmm, the gawk clone succeeded this time13:19
pdarA question regarding the baserock openstack, where will glance store images?13:29
ssam2SotK: i've not seen that before, or not for a long time at least!13:31
ssam2are you still stuck?13:31
franredpdar, /var/lib/glance/images?13:31
SotKssam2: it seemed to work when I tried it again, I assumed I'd done something dumb13:32
pedroalvarezpdar: yes, there, but sometimes the images don't appear there13:32
SotKwhen have you seen it before?13:32
ssam2just at random I think13:32
pedroalvarezpdar: I think that depending how you use glance they will appear there or not13:32
ssam2actually, I don't think I've ever seen 'ERROR Command failed: git config --unset remote.origin.mirror' before13:32
pdarAhh, ok i thought it might be there but when I looked there the cirros64 image was not there13:32
pedroalvarezpdar: I *think* that images uploaded using the url of the image, won't appear there13:34
pedroalvarezI havn't done any research though13:34
pdarahh, ok, thats how i got it13:34
pdarso it would follow13:35
pdarPerhaps a silly question: If I want to deploy an image to my baserock opestack, will I have to make the baserock openstack-server image larger than the image I want to import?13:35
pedroalvarezpdar: yes, you have to13:37
pedroalvarezyou can try `btrfs filesystem resize max /` on the openstack host13:37
pedroalvarez(you might have some space free on the disk not used by btrfs)13:38
pdarooh, that may be what I was working up to asking13:38
pedroalvarezssam2: anoyiingly, mounting a external disk using latest baserock, on a VM (x86) worked13:39
pdarboom! that worked great thanks pedroalvarez and franred13:40
SotKdid we not fix the GENIVI systems?13:46
ssam2in what way?13:46
SotKreferring to
SotKthey wouldn't build a while back, but I thought someone fixed that13:47
radiofreejjardon: we actually do release genivi systems13:47
radiofreewhat's broken about them?13:47
radiofreei think jonathanmaw is building them?13:47
radiofreecan you have gerrit notify you when a new patch series is sent?13:48
jjardonok, great, lets add them to the ci then?13:49
SotKradiofree: I think you have to add a project to your "Watched Projects" and configure it there13:49
pedroalvarezCI is going to explode13:49
radiofreeis it too hot?13:50
radiofreeSotK: thanks13:50
pedroalvarezbut yeah, we can do that13:50
radiofreei do miss being able to flick through patches on a list13:50
pedroalvarezradiofree: you can do the same on gerrit :)13:50
ssam2it's possible to set up 'dashboards' in gerrit, i know nothing about them right now but I think it lets you create custom views per-project13:51
ssam2might need an admin to set them up though13:51
radiofreepedroalvarez: i don't constantly have gerrit open though13:51
ssam2i have it pinned as a tab13:52
* pedroalvarez plugs a SATA disk to a VM, and VM kernel panics when booting14:12
pedroalvarezand this kids, is what happens when you start debugging an unrelated error14:13
pedroalvarezAnd this is the kernel panic:
pedroalvarezis trying to boot from the disk that I've just attached14:24
pedroalvarezsorry for the noise14:24
* pedroalvarez moves on14:24
Zarahi, I'm confused-- I'm trying to make a new system, but when I run morph build, I get 'ERROR: Couldn't find morphology: systems/openbmc-test-build-system.morph14:28
Kinnisonyou may need to add+commit brand-new morphologies14:29
bashrc_in the past I think I saw a similar error, where the morph clearly existed14:29
bashrc_and yes I think the solution was to commit latest changes14:29
Zarahm, I think I have committed them-- at least, git status doesn't show it as a staged or untracked file14:30
Zarabut it's there in 'definitions/systems/'14:31
Kinnisonand the filename is correct and the 'name:' field inside matches?14:31
ssam2what's your current working directory?14:31
Zarassam2: /src/workspace/apr20/baserock/baserock/definitions14:33
ZaraKinnison: yes, I checked and double-checked and triple-checked just in case14:33
KinnisonNo idea then :(14:33
jjardonZara: can you pastebin the contents of that commit?14:35
Zarajjardon: do you mean from git log? (if so, here: )14:37
radiofreessam2: having the same problem as you with new systemd14:38
radiofree[     *] A start job is running for dev-disk...b850a07f.device (12s / 1min 30s)14:38
jjardonZara: I meant 'git show' or 'git diff master' :)14:38
pedroalvarezradiofree: I was starting to have a look at it14:39
Zaraheh, ah, I've found what's going on-- it's working from an older commit.14:39
ZaraI made a new branch, and committed there, so I guess it confused morph.14:39
Zaraso I can do a morph checkout from my github and work from that, and that ought to fix it.14:40
Zaragrr, this has caught me out before14:40
pedroalvarezIs anybody against killing workspaces?14:41
pdarwhy were they necessary?14:41
radiofreei don't even get a recovery console though14:41
radiofreejust hangs there14:41
* radiofree boots into factory to manually edit the fstab14:42
straycatpdar, there's a concept of a system branch and having all repos in your workspace set to that system branch, morph can push local changes to the trove, for building/distbuilding14:42
straycatgetting rid of system branches is a pretty big change though14:43
radiofreehmm actually it's not /src it's hanging on :\14:43
Zarais there a quick command to tell morph to work from the branch currently checked out in git?14:44
straycatthe various improvements to morph deploy possibly just saved me 30 minutes14:44
straycatZara, don't set local-changes=ignore in your morph.conf ?14:45
pedroalvarezradiofree: isn't it?14:46
Zarastraycat: ah, it's probably just set to whatever the default is on the wiki14:46
Zarastraycat: though hm, there's nothing about local-changes in my morph.conf14:47
Zaraso I'm guessing that default is set elsewhere?14:48
pedroalvarezZara: I sometmes do a random change in a random file in definitions.git to workaround this bug14:48
radiofree[   16.032032] systemd[117]: /lib/systemd/system-generators/systemd-gpt-auto-generator failed with error code 1.14:48
radiofreewhat does that mean?14:48
SotKZara: the default is include14:48
Kinnisonradiofree: gpt is a partitioning scheme14:48
Kinnisonradiofree: I imagine it's meant to generate units for gpt partitions14:49
straycatZara, i thought the default was to include local changes, you can set local-changes=include to be sure14:49
SotKmaking a new branch is what caused the problem, morph builds the ref defined in a config file in your system branch14:49
radiofreehmm.. well this new image doesn't boot at all now14:49
radiofree[ TIME ] Timed out waiting for device dev-di...\x2d9b01\x2d9370b850a07f.device.14:49
radiofree[DEPEND] Dependency failed for /root.14:49
ssam2radiofree: can you roll back to the old version? for me, the old version still worked if I selected that one in the U-Boot boot menu14:50
radiofreessam2: yep, old version works fine14:50
radiofreei think i'll just add systemd on top of the system and redeploy14:50
radiofree(old systemd)14:50
*** Krin has quit IRC14:51
radiofreemy deploy cluster was wrong14:53
* radiofree wonders how this worked at all14:53
paulsherwoodbaserock mentioned in the news...
ssam2radiofree: argh, nasty14:53
ssam2radiofree: i found the other day that I had a Baserock rootfs, or at least part of one, installed to the ext2 boot partition14:54
*** Krin has joined #baserock14:54
pedroalvarezpaulsherwood: great!14:55
radiofree i'm not sure *how* it worked though, it wrote the extlinux.conf correctly14:59
ssam2does it read BOOT_DEVICE from /baserock/deployment.meta if it can?15:00
radiofreei was under the impression it would use the deployment details from the deployed upgrade15:00
pedroalvarezextlinunx.conf doesn't look fine on this jetston I'm debugging15:01
radiofreebut yes it probably ended up falling back to BOOT_DEVICE on factory15:01
radiofreewhich is an amazing fail-safe mechanism i totally intended to implement15:01
pedroalvarezmeh, it does looks ok15:01
pedroalvarezradiofree: so, there shouldnt be any "systems" folders on the BOOT_DEVICE, right?:15:04
radiofreeno it created them fine15:05
pedroalvarezI moved that folder, and then everything fails15:05
radiofreeit gets as far as booting and mounting the rootfs15:05
radiofreethen happens15:05
radiofreei still think it's a systemd problem, redeploying now though15:05
radiofreei'm just amazed it worked at all15:06
pedroalvarezI think it was trying to mount btrfs subvolumes from a ext4 partition15:06
pedroalvarezext2, or whatever15:06
radiofreei don't think so?15:06
radiofreemy kernel command line was /dev/mmcblk0p215:07
radiofreeROOT_DEVICE was set correctly, and for some reason BOOT_DEVICE ended up being correct15:07
radiofreei don't think BOOT_DEVICE has any influence over the system? other than to instruct system-version-manager where to put the kernel/where to edit extlinux.conf15:07
radiofreewhich it did...15:07
pedroalvarezradiofree: I removed the "systems" folder from BOOT_DEVICE and now this is really broken15:07
radiofreeyes because it won't be able to read the kernel15:08
radiofreeBOOT_DEVICE systems folder is just a copy of the btrfs structure, for simplicities sake15:08
radiofreeif you remove that then u-boot will try and read a non-existent kernel/device tree15:08
radiofreedeployed again without the typo, let's see what happens15:09
radiofreenope :(15:10
radiofreerebuilding systemd now15:22
pedroalvarezradiofree: so, have you reverted the systemd upgrade?15:25
radiofreei'm not going to rebuild from foundation, i just added a new "systemdrev" stratum with the old sha15:28
pedroalvarezwell, yes, my question wasn't really well phrased15:30
radiofreeit's building now, takes about 20 minutes15:30
ssam2might be useful to build it locally in the node and 'git bisect'15:31
*** paulw has quit IRC15:32
ssam2i'm pretty sure that SHA1 a88abde72169ddc2df77df3fa5bed30725022253 (v219) will work15:32
KinnisonHmm, after the openstack work was merged there was a lot of name duplication introduced ;(15:39
Kinnisonpaulsherwood: Are you *sure* erlang doesn't get built for you when you use ybd to build build-system-x86_64 ?15:40
KinnisonIf so, then there's a non-determinism issue in ybd15:41
SotKKinnison: I was wondering about that issue when looking at this morning15:42
SotKmy wild guess was what you're implying, in that the first chunk into the definitions dict varied each time15:43
SotKI didn't test or investigate properly though15:43
KinnisonI think it might be dependent on the order in which things come off the filesystem15:43
Kinnisonbut that would really make me sad15:43
SotKyeah, it does os.walk() on the cwd15:44
* SotK doesn't know if that can come out in a random order or not15:45
Kinnisonit likely comes out in dirent order15:46
Kinnisonwhich is filesystem dependent15:46
ZaraDoes anyone know why I'd get an error saying 'field build commands not allowed in morphology'?15:46
SotKdoes it tell you which morphology?15:46
Zarayeah, it's one I've just made so I've probably done something weird, just wondering if anyone knows what that error generally means15:47
Kinnisonis the morphology in question a chunk?15:48
*** Krin has quit IRC15:48
ssam2'build commands' or 'build-commands' ?15:48
SotKoh, the field is 'build-commands' not 'build commands'15:48
Zaraahhhh, hahaha15:48
Zarayes, that will be it15:48
* Kinnison thinks error messages which quote part of the input should quote their quotes15:48
KinnisonAn error of the form: "field 'build commands' not allowed in morphology" would be clearer15:49
Zarayeah, I thought it was saying it was some kind of special morphology that couldn't have any buid commands15:49
Zaraand I didn't know why it would be special15:49
ssam2interesting new distbuild bug: progress messages taking almost 5 minutes to get from the controller to the initiator15:49
ssam2is it possible that this is due to port forwarding being done by 'firehol'?15:50
KinnisonThough it puts paid to a statement on the definitions page on the wiki which says:15:50
Kinnison"Note that currently, unknown keys in definitions are silently ignored.15:50
Kinnisonclearly they're not15:50
radiofreeoh, building systemd failed :(15:50
radiofreegcc: error: /lib/ No such file or directory15:50
SotKyeah, unknown keys cause that error15:50
Kinnisonssam2: seems very unlikely15:50
ssam2the message is logged as sent by the InitiatorConnection at 15:31 in the controller log, and received at 15:35 in the initiator log15:51
ssam2never seen this before15:51
Kinnisonwere other messages passed around?15:51
ssam2in the meantime? no15:51
Kinnisonis the initiator setting the tcp flags to keep the socket alive?15:52
Kinnisonif not, address translation and connection tracking could cause confusing delays15:52
KinnisonHmm, the default keepalive is 2 hours15:52
Kinnisonso NAT is unlikely to be causing it15:53
Kinnison(I really should read the manpage before asking silly questions)15:53
*** a1exhughe5 has quit IRC15:53
ssam2if the connection dropped the build would stop15:53
pedroalvarezradiofree: I had that error when building master of systemd, which one are you trying to build?15:53
ssam2I wonder if this is distbuild.sockbuf in action15:54
Kinnisonif there's buffers which aren't being flushed then it's possible I guess15:54
ssam2it doesn't seem to have any 'flush buffer after this number of seconds' timeout15:54
ssam2and I did remove a huge message that did used to get sent at the start of each build, a while back15:54
* ssam2 looks forward to sending even more last-minute patches :(15:55
radiofreepedroalvarez:  a88abde72169ddc2df77df3fa5bed3072502225315:55
radiofreei.e the old  version of systemd15:56
Kinnisonssam2: speaking of last minute patches, anything you need reviews on still?15:56
radiofreei have strata/systemd.morph, which build-depends on foundation15:56
radiofreeand contains a systemd chunck with that ref15:56
KinnisonPlease stop having multiple definitions of the same name15:56
Kinnisonit makes me very sad15:56
Kinnisonbecause I have to kill a kitten every time I see one, and I like kittens15:56
pedroalvarezradiofree: this error might be because now systemd is being built with more dependencies15:56
pedroalvarezei, the whole foundation stratum15:57
* radiofree weeps uncontrollably 15:58
pedroalvarezmeh, sometimes puting a chunk on the top is not that easy15:58
ssam2Kinnison: indeed, the biggest one is Adam's OStree patch series15:59
ssam2Kinnison: that is awaiting a few fixes from Adam, but bar a couple of issues, is testable as-is15:59
ssam2looking at distbuild.sockbuf, it doesn't seem to have anything obvious that would cause a delay. It writes as soon as it receives data16:00
pedroalvarezradiofree: --disable-acl might help16:00
* SotK is running the test suite before rebasing and sending a new version for ostree16:00
KinnisonI'm not in a position to run much, so I am kinda limited to stuff I can review from a readability PoV only16:00
radiofreepedroalvarez: thanks, i'll give that a go16:02
ssam2Kinnison: that's useful too :)16:03
* Kinnison kicks off another build and gets himself into a position to review16:04
jjardonHi, compiling a ARMv5 tarball in a ARMv7 hardware (running native-bootstrap script), I get a compilation error about a missing "__gmpn_invert_limb" . There is a comment in the morph file to workaround this, but it doesnt seems to work here. Reading the conversation here:16:12
jjardon was the bug filed upstream at some point?16:12
jjardontiagogomes_: ^16:12
tiagogomes_jjardon, IIRC I tried to create an account on gcc bugzilla to report the bug as requested by Sam, but without success16:15
ssam2oh, max_buffer is 16384 on the controller->initiator socket ... that may be why distbuild messages are getting held up!16:19
ssam2I'm not sure about lowering that value though, would it cause big messages to send slower?16:20
* ssam2 is largely ignorant about TCP sockets16:20
KinnisonBy the time you're at TCP you're into other stuff entirely16:20
Kinnisonand TCP isn't likely to cause these delays unless you're corking and not uncorking16:21
ssam2what does 'corking and not uncorking' mean?16:21
KinnisonIn TCP you can cork the socket which stops it transmitting to the underlying network16:22
rjekssam2: corking means "don't send anything yet, I'm about to give you more data, so you can send it all in one packet"16:22
Kinnisonthen you write() as many times as you like16:22
rjekUncorking means "OK, send it now"16:22
Kinnisonand then you uncork and it sends it as efficiently as it can16:22
Kinnisontypically only used by seriously high-performance stuff16:22
ssam2I don't see a cork() function anywhere16:22
rjekie, if you're making several write() calls, you cork the socket to stop it popping.16:22
rjekIt's an ioctl IIRC16:22
ssam2none of those in Morph either, so I guess it's not that16:23
Kinnisonor a sockopt16:23
rjekOh no, get/setsockopt16:23
ssam2only one, which is SO_REUSEADDR16:23
rjekSee man 7 tcp16:23
rjek(which covers TCP-specifics)16:24
rjek(as sockets are protocol-generic.)16:24
ssam2maybe I need to set TCP_NODELAY on the sockets?16:26
ssam2but that seems like it might cause performance issues..16:26
rjekYou can also avoid small writes16:26
KinnisonIt's distinctly more likely that the delay is *inside* the python code somewhere16:26
Kinnisoni.e. that you've a buffer not being flushed16:26
rjekThe Nagle algorithm disabled by TCP_NODELAY is designed to protect against naive software that writes to sockets a byte at a time16:26
KinnisonUnless you're doing something *VERY* odd, you'll never manage to get a 5 minute delay on a tcp socket itself16:27
rjekPlus the Nagle algorithm's delay is going to be less than cork's 200ms16:27
rjek(otherwise they'd be no point in corking)16:27
ssam2Kinnison: ah, good to know16:28
ssam2I'm still combing through state machine transitions in a log file16:28
ssam2but it does seem that the sockbuf object starts writing to the socket as soon as it receives data and the socket is writable16:28
rjek5 minute delay on TCP?  I'm not sure that's even possible: there's the back-off during packet loss, but I think the maximum RTT for TCP before it gives up is <5m16:29
Kinnisonrjek: You could have fun with fragments and the keepalive limits16:29
rjek(Which is why TCP needs replacing before we colonise Mars)16:29
radiofreepedroalvarez: thanks, that worked16:34
radiofreeold systemd works fine16:34
pedroalvarezi was  goin to try latest master, but i dont have cache16:35
pedroalvarezradiofree: are you planning to continue debugging this?16:38
Kinnisonrjek: maybe QUIC will help for mars16:39
* Kinnison gdl&rs16:39
radiofreepedroalvarez: absolutely not16:40
radiofreepedroalvarez: in fact i've not wasted so much time on it i can't go about doing what i wanted to do (upgrading jetson to 4.0)16:40
pedroalvarezradiofree: ok, I will try to debug this then16:43
radiofreepedroalvarez: i'll probably have some time to do it thursday/friday, if it can wait16:43
radiofreewe could just cherry-pick that networking commit of richards on-top of the 219 release?16:44
pedroalvarezradiofree: yeah, but I think there were more things needed for openstack16:45
pedroalvarezI mean, not only that commit16:45
*** Guest21191 has quit IRC16:55
pedroalvarez1. A word describing the misfortune of something or someone.16:59
pedroalvarezI think there are 400ish commits from the current sha1 to 21916:59
pedroalvarezthat means 10ish git bisect iterations17:00
pedroalvarezaround 5 hours17:00
pedroalvarezyeah, bummer17:01
jjardon exactly 31417:01
*** bashrc_ has quit IRC17:01
pedroalvarezoh! useful data in  the unpetrify ref!17:02
rjekwho'd a thunk it17:02
jjardonpedroalvarez: :)17:02
jjardonId try current master, the systemd we are currently using is quite old already (mid March) and in the middle of the development cycle, so probably quite unstable17:04
pedroalvarezjjardon: yeah, I wanted to try that first17:04
ssam2ok idea, but that might introduce new bugs again17:04
ssam2do we feel lucky? :)17:05
*** franred has quit IRC17:05
radiofreei suppose i could try that now17:06
radiofreejust point my systemd stratum at master then?17:06
jjardoncome one!, What could possibly go wrong? :)17:06
pedroalvarezoh! :)17:06
ssam2nothing more than already has17:06
ssam2ha! just kidding!17:06
* pedroalvarez will wait for that test17:06
radiofreeabout 30 minutes17:06
radiofreebuild + deploy17:07
*** franred has joined #baserock17:07
ssam2It's pretty clear that this 'delayed messages' distbuild bug is definitely that the socket in the controller to the initiator doesn't become writable for 3 minutes after we are ready to write to it17:07
jjardonlooking at the history, systemd release are usually every 2 months; last one was 2 months ago so Id expect current master to be quite stable17:07
radiofreemaster of systemd! how exciting!17:07
pedroalvarezthe should be releasing 220 soon, no?17:10
Kinnisonssam2: that's surprising17:10
Kinnisonssam2: is the socket in the set for testing for writability all the time, or only after it has stuff put in it?17:10
ssam2only after it has stuff put in it17:10
ssam2but that is logged 3 minutes before it becomes writable17:10
jjardonpedroalvarez: I think so, yes. We can ask to be sure though17:10
jonathanmawuh, I accidentally sent an E-mail to baserock-internal instead of baserock-dev by accident17:12
jonathanmawmore annoyingly, I don't have a "sent" folder in my mail client, so I can't easily resend it to baserock-dev17:13
*** jonathanmaw has quit IRC17:15
ssam2'any reasonably healthy socket will return as writable - it just means outbound network buffer space is available'17:16
ssam2I wonder if that is a shared buffer -- in that case, it may be huge artifact graphs being sent to the workers that are delaying this message17:16
*** lachlanmackenzie has quit IRC17:16
KinnisonNo, it's per-socket17:18
Kinnisonunless something very odd is going on17:18
ssam2radiofree: if you're still about, could you paste the morph file you used to build systemd on top of all the other strata?17:23
ssam2i have a feeling i'm going to need to do the same17:23
Kinnisonssam2: good luck with that socket issue17:23
* Kinnison has to head off17:23
radiofreessam2: yep hold on17:25
ssam2updating to the artifact serialisation speedups branch *has* made the problem go away... so while I'd like to know what is actually going on, I think i'll just hope for the best on this one, for the time being17:26
*** edcragg has quit IRC17:27
radiofreessam2: strata/systemdrev.morph
* SotK wishes the test suite would hurry up and finish running17:27
radiofreessam2: strata/foundation/systemdrev.morph
radiofreedeploying now!17:28
radiofreessam2: a88abde72169ddc2df77df3fa5bed30725022253 works fo'sure17:28
radiofreemaster works \o/17:33
pedroalvarez5 hours saved!17:34
* radiofree heads off17:35
pedroalvarezThanks radiofree17:35
* SotK starts typing `git push` and remembers he didn't pull before he rebased :(17:35
*** Albert_ has quit IRC17:39
*** gary_perkins has quit IRC17:44
*** zoli__ has joined #baserock18:04
*** ssam2 has quit IRC18:32
*** ssam2 has joined #baserock18:35
*** ChanServ sets mode: +v ssam218:35
*** Albert has joined #baserock18:37
*** Albert has quit IRC18:53
*** rdale has quit IRC19:09
* SotK decides commands which take REPO REF FILENAME should work anywhere20:32
*** zoli__ has quit IRC21:32
*** ssam2 has quit IRC22:14

Generated by 2.14.0 by Marius Gedminas - find it at!