IRC logs for #baserock for Monday, 2015-09-14

*** edcragg has quit IRC01:52
*** flatmush has quit IRC01:52
*** fay_ has quit IRC01:52
*** nowster has quit IRC01:52
*** petefoth has joined #baserock01:52
*** edcragg has joined #baserock01:52
*** nowster has joined #baserock01:53
*** gary_perkins has quit IRC01:53
*** fay_ has joined #baserock01:53
*** flatmush has joined #baserock01:54
*** gary_perkins has joined #baserock01:54
*** zoli__ has joined #baserock05:48
*** paulw has joined #baserock06:10
petefothMorning all06:26
*** paulw has quit IRC06:52
*** paulw has joined #baserock06:54
*** zoli__ has quit IRC07:39
*** toscalix__ has joined #baserock07:42
paulsherwoodmorning petefoth07:54
paulsherwoodKinnison, pdar - fwiw i've found in testing that 4 instances of ybd on aws c4.8xlarge is as fast as or slightly faster than 5 on the m4.10xlarge08:01
paulsherwoodhttps://raw.githubusercontent.com/devcurmudgeon/build-logs/master/aws-c4.8x-large-4x14.log08:01
paulsherwoodhttps://raw.githubusercontent.com/devcurmudgeon/build-logs/master/aws.ci-again.log08:01
paulsherwoodand the c4 is 30% cheaper: $1.763 vs $2.5208:03
*** bashrc_ has joined #baserock08:03
Kinnisonpaulsherwood: interesting, thank you08:06
*** paulw has quit IRC08:13
*** paulw has joined #baserock08:14
*** persia__ is now known as persia08:21
*** persia has joined #baserock08:21
*** tiagogomes_ has joined #baserock08:21
*** paulw has quit IRC08:22
*** paulw has joined #baserock08:25
*** mariaderidder has joined #baserock08:26
*** zoli__ has joined #baserock08:27
*** ssam2 has joined #baserock08:44
*** ChanServ sets mode: +v ssam208:44
paulsherwoodalso ybd master now should clear tmp on start09:21
paulsherwood(reliably009:21
KinnisonUseful to know09:25
*** mariaderidder has quit IRC10:01
*** mariaderidder has joined #baserock10:16
tiagogomes_meh `./check --full` is failing on master10:43
KinnisonErk10:55
tiagogomes_https://gerrit.baserock.org/#/c/1105/10:57
*** Guest73814 has quit IRC11:09
*** persia_ has joined #baserock11:10
*** franred has joined #baserock11:46
*** zoli__ has quit IRC11:52
*** zoli__ has joined #baserock12:08
*** paulw has quit IRC12:16
*** paulw has joined #baserock12:25
*** zoli__ has quit IRC12:26
*** jjardon has quit IRC12:53
*** jjardon has joined #baserock12:54
*** franred has quit IRC13:16
*** persia_ has quit IRC13:53
*** persia_ has joined #baserock13:54
*** paulw has quit IRC14:02
*** paulw has joined #baserock14:03
*** zoli__ has joined #baserock14:06
*** zoli__ has quit IRC14:21
*** paulw has quit IRC14:36
*** flatmush has quit IRC15:45
*** flatmush has joined #baserock15:46
*** bashrc_ has quit IRC15:50
*** bashrc_ has joined #baserock15:51
*** flatmush has quit IRC15:54
*** flatmush has joined #baserock15:55
*** bashrc_ has quit IRC15:55
*** fay_ has quit IRC15:57
*** petefoth has quit IRC15:57
*** ssam2 has quit IRC15:57
*** mariaderidder_ has joined #baserock15:58
*** edcragg has quit IRC15:58
*** flatmush has quit IRC15:58
*** tiagogomes_ has quit IRC15:58
*** nowster has quit IRC15:58
*** edcragg has joined #baserock15:58
*** petefoth has joined #baserock15:58
*** ssam2 has joined #baserock15:58
*** ChanServ sets mode: +v ssam215:58
*** fay_ has joined #baserock15:58
*** gary_perkins has quit IRC15:58
*** nowster has joined #baserock15:58
*** tiagogomes_ has joined #baserock15:58
*** flatmush has joined #baserock15:58
*** mariaderidder has quit IRC15:59
*** gary_perkins has joined #baserock15:59
*** bashrc_ has joined #baserock16:01
paulsherwoodssam2: thanks for the packaging of ybd! i'm certainly not against it.16:23
ssam2ok, cool16:23
paulsherwoodcould/should it include dependencies?16:23
ssam2this is https://github.com/devcurmudgeon/ybd/pull/12316:24
ssam2paulsherwood: the setup.py file? hmm16:24
ssam2i thought ybd didn't have any :-)16:24
ssam2i guess there's sandboxlib16:24
paulsherwoodhttp://wiki.baserock.org/ybd/ shows pyyaml sandboxlib jsonschema requests16:24
paulsherwoodjsonschema is optional16:24
paulsherwoodrequests is required for kbas16:24
ssam2oh yeah16:24
ssam2we could add requirements.txt16:25
paulsherwoodis that how deps are specified for pythong packages?16:25
ssam2for the required dependencies. then the optional dependencies go in setup.cfg16:26
ssam2python packaging is a wilderness. but that's how PBR recommends doing it16:26
paulsherwoodssam2: i'd be tempted to ignore the optional for now16:26
ssam2http://docs.openstack.org/developer/pbr/#requirements16:26
ssam2so it's just sandboxlib. easy enough to add16:26
ssam2possibly only useful if you plan to release YBD on PyPI16:26
paulsherwoodthe jsonschema stuff depends on us implementing schemas for definitions16:26
paulsherwoodpyyaml seems to be missing on some python default implementations, same with requests?16:27
paulsherwood(they're both in baserock by default, i know)16:27
ssam2actually yes, neither of those are built-in to Python, so we should list them16:27
paulsherwoodi assume this is only for python packages...?16:29
paulsherwoodhow would one automagically install git, autotools, wget etc?16:29
radiofreei hate apps that try and call apt or yum16:30
* paulsherwood too16:30
ssam2using Baserock, perhaps16:30
paulsherwood:)16:30
radiofreeyou can't use baserock-chroot-manager on baserock16:31
pedroalvarezIf the app fails suggesting what to do to solve it, would be enough16:31
paulsherwoodack16:31
ssam2radiofree: that's annoying!16:34
ssam2paulsherwood: i updated the pull request with requirements.txt file16:36
paulsherwoodssam2: what's the process of pushing this to pypi?16:39
* paulsherwood wonders if he should do a 'release' prior to merging this, for safety16:39
ssam2paulsherwood: see https://github.com/CodethinkLabs/sandboxlib/blob/master/HACKING.rst16:40
ssam2that said, it seems like PBR isn't picking up the version number from the tags in YBD.git16:41
ssam2oh, that might be because it only looks at 'annotated' tags16:41
ssam2seems that a 15-38 tag causes PBR to call the package 'ybd-15.post38', though16:42
ssam2would you consider switching to 15.38 instead of 15-38 for numbers?16:42
ssam2otherwise, you could manually set a version number in setup.cfg, but it seems like that would be more faff in the long run16:43
*** mariaderidder_ has quit IRC16:44
paulsherwoodyup, happy to switch16:45
paulsherwoodplease hold16:45
paulsherwoodssam2: https://github.com/devcurmudgeon/ybd/releases/tag/13.3816:48
ssam213.38? I don't understand this numbering scheme!16:51
paulsherwoodffs!16:51
paulsherwoodshould be 15.3816:51
pedroalvarez:)16:51
* paulsherwood hopes no-one will mind if i delete it16:52
ssam2there must be a 5 minute rule or something.16:52
paulsherwooddone16:52
paulsherwoodhttps://github.com/devcurmudgeon/ybd/releases/tag/15.3816:53
paulsherwoodyy.ww16:53
richard_mawpaulsherwood: did you know that when you fork off a subprocess, that subprocess also holds a reference to your lock, hence why when your original process closes the file descriptor it locked, when it exits the `with open` scope, the process doing the cleanup still has the shared lock, hence your cleanup is protected from another simultaneous cleanup attempt16:53
paulsherwoodrichard_maw: i think i did? but since you're asking, maybe i'm missing something?16:54
paulsherwoodrichard_maw: iiuc my merged version includes your recommendation, and should be safe as you describe above16:55
richard_mawI know, I had a look at the version you eventually merged.16:55
paulsherwoodi hope you are happy with it... did i miss something?16:56
richard_mawI would have done it by returning the shared-locked file descriptor, but I'm trying to work out whether you accidentally or purposefully produced a more interesting solution.16:56
paulsherwoodlol16:56
paulsherwoodaccident probably16:56
richard_maw16:56
paulsherwoodi was aware that the shared lock would be held by both processes while in the cleanup function, though :)16:57
*** bashrc_ has quit IRC16:57
richard_mawyou end up closing the reference to the lock in your main process after you exit the cleanup function, since your reference to the lock is kept in the file which you close16:57
richard_mawwhich I suspected may be accidental, but since you're keeping the reference in the child process, it still works16:58
richard_mawthere's 1 race left I can see, the failure mode for which is just that you need to retry starting a build, but it can happen when you go a long time without cleaning up tempdirs, and a process is holding the exclusive lock and calling os.listdir while another is started17:01
richard_mawthe second process could try to take the exclusive lock, fail, and then also fail to take the shared lock, since the other process is still trying to os.listdir17:01
paulsherwoodack17:02
richard_mawwhich is why in my e-mail I recommended a blocking lock with a timeout17:02
paulsherwoodok i'll look at that17:02
paulsherwoodthanks for the continued feedback :)17:03
* paulsherwood has to dash17:03
* richard_maw does too17:03
richard_mawa blocking lock will do most of the time, as would a retry loop, but the former is bad if it's gotten totally stuck in os.listdir (I've seen this happen in btrfs) and the latter can end up missing the window where it's allowed to take the lock (unlikely) and is slower than blocking when the other process behaves17:06
richard_mawunfortunately the only way to do a blocking lock with a timeout involves signals, and despite the process overhead involved in doing it by shelling out to http://man7.org/linux/man-pages/man1/flock.1.html , I still prefer it, as it doesn't mess with your own process' state so badly17:08
*** toscalix__ has quit IRC17:12
*** toscalix__ has joined #baserock17:13
*** ssam2 has quit IRC17:20
paulsherwoodhttp://stackoverflow.com/questions/5255220/fcntl-flock-how-to-implement-a-timeout ?19:35
*** toscalix__ has quit IRC19:37
*** zoli__ has joined #baserock19:40
*** zoli___ has joined #baserock19:48
*** zoli__ has quit IRC19:48
*** zoli__ has joined #baserock20:13
*** zoli___ has quit IRC20:13
*** zoli__ has quit IRC20:46
*** zoli__ has joined #baserock20:47
paulsherwoodradiofree: incidentally, with swap, on scaleway qtwebkit built ok in 02:53:4921:07
radiofreepaulsherwood: it builds, with swap, on a jets21:08
radiofreeOn.. In about an hour21:08
radiofreeOn the mustang it can probably build without swap in less time than thst21:08
persiaHow much RAM is on a mustang?21:09
radiofreeBut apparently no one here wants a fast machine populating arm cache21:09
radiofreepersia: 16G21:09
persiaI'd like one :)21:09
paulsherwoodradiofree: i'd like one too21:10
paulsherwoodi guess we need mustangs in the cloud... queue https://www.youtube.com/watch?v=qMM3MgS4yxc21:12
* paulsherwood doesn't know why he suddenly thought of the osmonds screaming21:16
radiofreepaulsherwood: you already one21:17
jjardonHi, Is there a generic armv7 image available somewhere?21:22
paulsherwoodyes21:25
*** zoli__ has quit IRC21:25
* paulsherwood hunts21:25
*** zoli__ has joined #baserock21:25
paulsherwoodjjardon: what's your use-case?21:25
paulsherwoodhttp://212.47.232.3/artifacts/genivi*21:31
paulsherwoodhttp://212.47.232.3/artifacts/build-system*21:32
jjardonId like to try to add support for a new board21:32
jjardonCheers21:32
paulsherwoodjjardon: note these are ybd images21:33
paulsherwoodjjardon: wget http://212.47.232.3/get/build-system-armv7lhf-jetson.cee219f7d667c5d65d86b57c92878ae43453cc173f824918d3a8b97182b8b189 for example21:34
* paulsherwood has not tested them, also21:34
paulsherwoodactually, i may be misleading you here.. these are system artifacts created by build, not deployment21:35
jjardonpaulsherwood: I think that's good enough, thanks. By any chance you dont have a Build system chroot , isnt it?21:43
paulsherwoodjjardon: fraid not, i'm not using chroots at all21:44
jjardonpaulsherwood: that's ok. btw, any opinion about https://gerrit.baserock.org/#/c/1098/ ? it would be good to have a agreement about that kind of updates21:48
paulsherwoodjjardon: i would be delighted to see it, just not sure how to address richard ipsum's valid point21:50
paulsherwoodi tihnk by default we should adopt each new major kernel release as soon as it comes out21:52
paulsherwoodin other news... https://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJjbxMSppiMcg/pubchart?oid=1593295908&format=interactive21:56
paulsherwoodso i think our assumption (in morph and ybd) that max-jobs = 1.5*cores is not actually the best number for AWS at least21:57
paulsherwoodincreasing max-jobs past 20 actually SLOWS DOWN the build!21:57
paulsherwood(cxlarge has 36 cores, mxlarge has 40 cores)21:57
paulsherwoodand cx machine is 10% faster than mx, as well as being 30% cheaper per hour21:58
paulsherwoods/faster/faster for builds/21:58
paulsherwoodto be fair, other components may vary a bit - qtwebkit did get slightly faster for -j > 2022:05
* radiofree looks at backscroll... saves it for tomorrow22:05
* paulsherwood should write a blog post22:07
*** zoli__ has quit IRC22:09
*** zoli___ has joined #baserock22:10
jjardonpaulsherwood: yeah; as I said in the comments I did the opposite before (update all the BSPs, warning is not tested), but others didnt like it, with valid points as well. I only want to merge my patch :)22:11
paulsherwoodjjardon: i would prefer the opposite, tbh22:11
paulsherwoodie keep things in step, even if we don't test all the things22:12
jjardonpaulsherwood: Id prefer keep updating everything, but only merge when someone test it; if time passes and the changes are not merged, remove the BSP as it has not being maintained (or move it to a "unsupported" directory or something)22:14
paulsherwoodjjardon: did you manage to get that artifact? i'm seeing some errors on the server side, timeout for example22:15
paulsherwoodjjardon: makes sense22:15
jjardonnope, didn't receive the board yet, only asking to prepare when the moment arrive :)22:16
paulsherwoodok22:16
paulsherwoodwell, someone seems to have downloaded it, after a couple of tries :)22:17
paulsherwoodand now the world and their friends are probing it22:18
paulsherwood104.148.44.191 - - [14/Sep/2015 18:16:09] "GET http://www.sogou.com/index.html HTTP/1.1" 404 78222:18
paulsherwood104.148.44.191 - - [14/Sep/2015 18:16:10] "CONNECT www.paypal.com:443 HTTP/1.1" 404 76122:18
bjdookspaulsherwood: the sad fact of public facing services22:25

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!