IRC logs for #baserock for Tuesday, 2015-09-29

*** Walkerdine has joined #baserock01:17
*** gtristan has quit IRC02:45
*** gtristan has joined #baserock03:16
*** zoli__ has joined #baserock03:24
*** zoli__ has quit IRC03:50
gtristanOk this ybd warning message is just killing me: WARNING: rogue builds in a chroot sandbox may overwrite your system03:54
gtristanfollowing the rabbit hole... turns out it's when ybd encounters a sandboxlib using traditional chroot where linux-user-chroot is not found03:55
* gtristan wonders... it 'may' overwrite my system... yet it's not an error and doesnt halt the build... you just hit CNTL-C repeatedly after that... hoping no damage was done03:56
gtristanhmmm, well, it's looking good04:10
gtristanso, building with ybd, stage2-make fails to find gmp.h while compiling guile.c... causing subsequent builds to fail to find make04:51
gtristansound familiar ?04:52
gtristanrichard_maw, I suspect reverting caf277de258967d361f07d636fa6805652462a9f will fix this...04:53
gtristanhmmm, this is from 2012... looks like I'm the only one left who never updated their base system since then05:03
gtristananyway, excuse random remarks... trying to get through my first ybd build, with an old ubuntu (12.04) install05:04
* gtristan thinks it should still work though...05:05
* gtristan wipes slate clean... installs baserock build system image in kvm... if ybd builds anywhere, it will build in that vm06:15
*** zoli__ has joined #baserock06:51
*** paulwaters_ has joined #baserock06:54
*** zoli__ has quit IRC06:55
*** gtristan has quit IRC07:01
*** fay_ has joined #baserock07:12
*** Walkerdine has quit IRC07:33
paulsherwoodi guess i need to fix that 'rogue builds' message. any rogue software could overwrite someone's system, but we don't say it all the time07:34
*** gtristan has joined #baserock07:43
gtristanpaulsherwood, hi... just got back from a late lunch and reading your comments now...07:55
paulsherwoodgtristan: hi!07:55
paulsherwoodbasically i'm just surprised how broken things seemed to be when you tried07:56
paulsherwoodi've fixed the warning message...07:56
gtristanyeah, I have been extremely lazy about upgrading my own laptop, I avoid that normally but should upgrade07:56
gtristannote, I cannot install linux-user-chroot here, tried building it manually... fails building on an outdated (unversioned!?) seccomp library07:58
paulsherwoodgtristan: ok, well for getting started, chroot should be fine. that's my default setup anyway07:58
gtristanI have master of both baserock definitions and ybd07:58
*** tiagogomes_ has joined #baserock07:59
paulsherwoodok. i may have broken some thingds over the last week in ybd... safest would be 15.3907:59
gtristanone thing is... I ran initially as not root (maybe causes permissions issues ?)... and then re-ran it as sudo07:59
gtristanand... once stage2-make failed to build, the build 'completed' mysteriously with no output... it would seem ybd also leaves the tree in a tagged state and doesnt try to rebuild failed builds08:00
gtristanso not sure if I need to remove that cache08:01
paulsherwoodgtristan: i think that's a bug. almost certainly introduced since 15.3908:01
gtristanok, well, I should try that tag08:01
paulsherwoodi suggest you clear the cache, reset ybd to 15.39. sorry for the confusion08:01
paulsherwoodi was hoping to tidy some stuff over the weekend but injured myself instead08:02
gtristanhonestly, I ran the baserock vm thinking it would be a good reference, but then got stumped with trying to sort out it's internet connection (playing with routing tables, nats and stuff... is /really/ my weak point)08:02
paulsherwoodmine too. :)08:02
gtristanit's all gibberish, goes in one ear and out the other08:02
gtristanso, I was about to resort to using a virtualbox of a recent ubuntu distro and trying there08:03
gtristanI think I may try both in parallel08:03
gtristanpaulsherwood, did you see the commit from 2012 ?08:03
paulsherwoodi saw what you wrote, but didn't understand the meaning08:04
gtristanI find it suspicious, as gmp was listed 'directly' before make in the old bootstrap json file08:05
gtristanthe commit removes it, stating it was only a dependency for gcc08:05
pedroalvarezbut that is really really old08:05
gtristanand so is my OS ;-)08:05
pedroalvarezheh :)08:05
gtristanshould not be any correlation I think, as it's all from source08:05
gtristanjust suspicious08:05
paulsherwoodwell, if you could try again with clean cache and 15.39, i hope all will be hunky dory08:05
gtristanI will try both08:06
radiofreeMaybe you should upgrade your distro08:06
gtristanradiofree, yeah, been saying that about 3 times now ;-)08:06
gtristanguess every 2 years it's about time08:07
*** jonathanmaw has joined #baserock08:09
gtristanpaulsherwood, just to double-check, is it important to *be root*, or is 'sudo' sufficient ?08:09
tiagogomes_IIRC you need to change the network adapter virtual driver on VBox for having Internet on Baserock08:10
*** mariaderidder has joined #baserock08:17
wdutchI think that CIAT pipeline config should look something like this , firehose config would exist separately08:19
KinnisonWhere would the slave configs come from?08:20
gtristanso, is there a nifty way to 'parent' a baserock environment to another ? perhaps avoiding the trip to ?08:20
Kinnisongtristan: not trivially, though you *should* be able to transfer the git cache from one system to another, and there are ways to transfer artifacts around too08:20
Kinnisongtristan: everything is kinda oriented (for now) around the idea that you'd have a local git service anyway08:21
Kinnisongtristan: the git server (trove) is designed to parent to another, to build a tree08:21
wdutchKinnison: I guess they would have to exist before that is processed. I don't think slaves should be defined in conjunction with the pipelines08:21
Kinnisonwdutch: I'm happy for it to be nearby, but it's important to consider all the inputs at once08:21
Kinnisonwdutch: Also I'm leery of the pipelines specifying the build slaves08:21
Kinnisonwdutch: I'd rather they didn't08:21
Kinnisonwdutch: otherwise elasticity becomes a pain08:22
*** paulwaters_ has quit IRC08:22
wdutchso just speicify architecture? It might get a bit tricky when flavours and things get involved08:22
* wdutch was thinking of openstack08:23
KinnisonI think builders are builders08:23
Kinnisonthe important thing here is that we don't want to have to rewrite configurations to spawn more build slaves08:24
KinnisonI'm fine if we specify a kind of build slave appropriate in the config, but it should be exactly that -- a kind08:24
*** paulwaters_ has joined #baserock08:24
wdutchwe shall call it slavour: slave flavour08:25
* Kinnison isn't trying to make your life difficult, just trying to ensure that you don't make our future lives difficult08:25
wdutchI think it's a valid point08:25
* wdutch wonders what the collective noun for slaves is08:26
wdutcha coffle it seems08:26
wdutchthat's a bit obscure08:26
pedroalvarezwe could call them "workers"08:31
wdutchthat doesn't really abstract away the notion though08:32
pedroalvarezI'm trying to figure out, how to get, for a given firehose change, the build number of the integration, build, deploy, and testing steps08:33
wdutchone could track the sha of the candidate ref of definitions08:34
pedroalvarezthat's what I was thinking. It is possible to add steps that run in the master08:35
wdutchyou'd have to add it to bottlerock, buildbot master doesn't execute code unless reconfigured08:35
pedroalvarezhm.. I haven't looked at bottlerock yet08:35
wdutchit's just python and bottle to expose buildbot to the webz08:36
pedroalvarezMy current idea was to add an extra step in buildbot, that writes the info I need in a DB/file on the master08:37
pedroalvarez(sha, column, buildnumber)08:37
wdutchsounds sensible08:39
wdutchI should probably store the clusters and their dependencies in the database08:39
wdutchI'll have to learn to use sqlachemy08:39
*** toscalix__ has joined #baserock08:40
pedroalvarezonce I have the relation of sha1 -> buildnumber, we can do whatever we want with the UI08:40
*** mariaderidder has quit IRC08:46
*** mariaderidder has joined #baserock08:46
*** rdale has joined #baserock08:47
*** franred has joined #baserock09:00
*** ssam2 has joined #baserock09:03
*** ChanServ sets mode: +v ssam209:03
paulsherwoodgtristan: i don't know the answer to that, sorry. anyone? do we need to be root for ybd/sandboxlib/morph, or is sudo sufficient?09:07
KinnisonIn general I'd recommend sudo -H $app09:08
Kinnisonotherwise you get root-owned stuff in your ~09:08
gtristanalright, no problem09:09
gtristanpaulsherwood, I take it you generally become root09:10
paulsherwoodwell, actually i normally sudo -i, and then run ybd09:10
paulsherwood(having had Kinnison explain to me what sudi -i is)09:10
gtristanI doubt it will make a difference to the build09:11
ssam2the 'root' requirement is discussed a bit here:
pedroalvarezwdutch: I guess the best way to fiddle with is to create my own instance, not touching's one09:40
wdutchpedroalvarez: yes tis probably for the best, I'm hoping to get rid of that file at some point though09:41
wdutchmoving things like urls and name to yaml and python to a separate module ciatlib which is in trove09:42
paulsherwoodseems like no builds have happened today... is upstream havign a day off?10:16
pedroalvareznothing since Mon 28 Sep 2015 17:13:2610:18
Kinnisonanything in the bottlerock logs?10:19
* paulsherwood has deja vu again10:21
* richard_maw doesn't see any recent systemd changes, but we should be seeing firehose changes10:22
rjekagain?  is that déjà déjà vu vu?10:23
richard_mawlast thing in the bottlerock logs too, it's running but hasn't seen any triggers10:24
Kinnisonthere's definitely changes hitting cu010-trove in the past few hours10:24
richard_mawwdutch: what did you do to change the url from
richard_mawsince that happened roughly at the time when it stopped sending changes10:25
wdutchchanged orchenv-master/master/master.cfg line 12310:26
richard_mawhm, unlikely to be the cause10:27
* richard_maw checks netstat to see if is even listening10:28
richard_mawdefinitely thinks it's listening on
Kinnisonrichard_maw: I'm trying a curl from the trove10:31
Kinnisonit's not responding10:31
KinnisonThe connect is timing out10:32
Kinnisonwhich is fun10:32
* Kinnison tries some tcpdumping10:32
KinnisonOkay that time it connected but the post is blocking10:33
richard_mawit's blocking in recvfrom, but there's 2 sockets open10:34
richard_mawI'll double check, but it looks like while it's got the right socket open, it's blocking on the wrong one10:34
Kinnisonsingle-threaded bottle10:34
Kinnisonsomeone opened a connection10:34
KinnisonWe've seen this kind of cockup before we switched the morph artifact cache10:35
Kinnisonto a proper web frontend10:35
*** brlogger` has joined #baserock10:37
*** brlogger has quit IRC10:38
richard_mawlooks like it's blocking on a connection from
Kinnisonkill it and restart it10:39
Kinnisonso it at least gets on with things10:39
richard_mawrestarted it, let's see what happens10:42
* richard_maw wonders what the IP address of the arm build slave is10:42
* richard_maw finds and is amused by the quaint list of supported operating systems for import10:47
richard_mawit looks like Amazon has eaten its old tooling, which trusted you to do the right thing10:48
wdutchtoscalix__: I'm here10:49
richard_mawunfortunately there appears to have been enough churn in the tooling that a lot of old guides no longer work10:49
toscalix__wdutch: sorry10:49
toscalix__wdutch: thanks for the paragraph for the wiki10:49
richard_maw"Make sure that you have at least 250 MB of available disk space for installing drivers and other software on any VM you want to import into an Amazon EC2 AMI running Microsoft Windows or Linux." O_O10:50
richard_mawit looks like as part of the import process they install packages10:50
richard_mawso unless the old ec2-bundle-image tooling still works, it may not currently be possible to import a Linux VM that isn't a Debian or RHEL derivative10:53
ssam2we could hack something useful that pretends to be 'rpm', perhaps. Pretty ugly though11:07
paulsherwoodrichard_maw, Kinnison - did you see my comments yesterday about not setting instances: 5 on a single arm builder?11:11
paulsherwoodit's causing a herd of 5, each with max-jobs: 111:12
*** franred has quit IRC11:53
*** gtristan has quit IRC12:13
*** paulwaters_ has quit IRC12:18
* richard_maw had not seen that comment12:25
paulsherwoodbest for a moonshot cartridge is just default. instances will be automatically 1, max-jobs automatically 812:26
* paulsherwood *thinks* it's best, at least12:26
* Kinnison imagines 2 with 4 will be better for most situations12:27
paulsherwoodeasy to try - just set instances: 212:29
richard_mawso number of builders to try is number of cores / 4 rounded down?12:29
paulsherwoodi was assuming we'd have mulyiple cartridges in working at once12:29
* richard_maw is looking for a metric for splitting up a single node12:30
paulsherwoodbased on my testing of full builds, i believe one instance is faster for full builds unless the node has significantly more than 10 cores12:31
paulsherwoodbut i'm interpolating and extrapolating. mileage may vary based on actual loads12:32
* richard_maw is just thinking that 8 cores is a more common configuration than 10, and 8 cores sounds meaty enough to handle two parallel builds.12:33
paulsherwoodgiven most of our builds at this point will be systemd, we could test easily over a few iterations?12:33
paulsherwoodyes it can handle the parallel builds, but the compiles will take twice as long?12:34
paulsherwoodit's easy to try this, though :)12:34
*** paulwaters_ has joined #baserock12:35
richard_mawpaulsherwood: it's a difficult metric to guess for, you need to know the average parallelism of each of the chunks to make reliable estimates, which entirely depends on how the chunk is configured12:36
paulsherwoodto be clear i established in my testing it's definitely better to have parallel instances on the big AWS machines, becuase max-jobs > 10 seems like a waste of time. i didn't look deeply at the performance on these 8 core machines so far12:36
paulsherwoodrichard_maw: yup. but we can know after a few runs12:36
richard_mawtbh, I think having multiple builds going on at once is better, partly just to test that it's handling multiple builds better, and to keep it looking busy12:38
* richard_maw puts banging his head against AWS on hold to see what he can do to make parallelism on ARM better12:39
*** gtristan has joined #baserock12:55
*** rdale has quit IRC13:05
*** rdale has joined #baserock13:12
richard_mawthere, should get 2 builders on ARM with max-jobs 4, rather than 5 with max-jobs 113:21
* richard_maw tries to work out whether he can find the old version of the AWS tools that try to be less helpful, hence are more useful when you're doing something they don't expect.13:25
Kinnisondo the eucalyptus tools work with ec2 ?13:29
richard_mawdon't know13:29
* richard_maw has another thing to research13:29
richard_mawit says they at least did13:30
richard_mawhm, possibly not, given it said the AWS APIs were built on top of Eucalyptus, so potentially their own tooling uses their own APIs instead13:31
richard_mawooh, euca2ools is python13:32
richard_mawo_O that's a lot of binaries installed13:33
richard_maw200 odd13:34
richard_mawbut it does appear to have equivalent tools that I need13:36
*** rdale has quit IRC13:38
Kinnisonrichard_maw: also might be of use13:39
Kinnisonrichard_maw: esp. given it's for FreeBSD which definitely isn't ubuntu or redhat13:39
richard_mawKinnison: no… but at least initially it was bootstrapped by taking a Windows instance and installing FreeBSD on its disk and snapshotting that. I'm trying to work out whether subsequent builds work by taking a FreeBSD image and doing the same dance, or whether he's just building in the cloud out of convenience.13:47
KinnisonI think he said that it's no longer necessary, the Windows bit was only to get HVM back when it wasn't the standard option13:49
KinnisonUnless I misread it13:49
richard_mawoh he's saying that's no longer necessary to be able to register images, but AFAICT he's still producing the images by taking an existing image and trashing the disk with a new install13:50
richard_mawbundle/import is apparently a separate step to registration13:50
*** ssam2 has quit IRC13:53
richard_mawhm, he's building his systems in the cloud by spawning a VM with extra volumes attached and snapshotting those volumes, then registering those volumes in images13:59
* paulsherwood tried something similar at scaleway14:00
paulsherwood(failed, but should be possible)14:00
richard_mawso it's an approach which ought to work without needing to already be in the Amazon cloud14:02
richard_mawwe should jut be able to import a volume and register an image with it, provided with the right manifest14:02
*** ssam2 has joined #baserock14:06
*** ChanServ sets mode: +v ssam214:06
* richard_maw wonders what an S3 bucket name is14:17
paulsherwoodit's the id for a storage location14:18
richard_mawis that like availability zones?14:18
* paulsherwood doesn't know, sorry14:19
richard_mawhm, appears to be some arbitrary label for grouping14:20
* richard_maw uses the pre-existing ciat-bucket14:21
Kinnisons3 buckets are labels for groupings of storage which can have policy and access control applied to them14:25
richard_maw'tis uploadin'14:26
richard_mawuploading a bunch of 10M chunks, half of which are likely to be empty14:27
* richard_maw guesses this is why the BSD guy did his builds in the cloud14:28
Kinnisonin part, almost certainly14:28
richard_mawyou never get bored in this job eh, there's always some new technology you've got to work out htf to use14:30
tiredcatuntil the technology figures out how to use us that is14:36
*** toscalix__ has quit IRC14:45
*** toscalix__ has joined #baserock14:47
*** franred has joined #baserock14:56
*** tiredcat has quit IRC15:00
paulsherwoodis the pipelining working now? arm build hasn't run for a while iiuc15:00
richard_mawlast systemd merge was 13:32 < sd-bot1> [systemd|michich] Merge pull request #1403 from dvdhrm/prioq-comment
SotKlooks to me like the last builds were at 10:4715:03
richard_mawit's not impossible that we just happen to have hit a poor synchronisation between the two troves involved in fetches before it turns up in CIAT15:04
richard_mawbut it's rather unlikely15:04
Kinnisoncheck bottlerock again?15:04
Kinnisonthough the firehose triggers are running15:05
Kinnisonso it's unlikely to be that15:05
richard_mawit's not stuck like it used to be15:05
paulsherwoodno x86 build at that time either?15:05
richard_mawwe don't get build triggers immediately as upstream pushes changes, since troves have to poll for changes15:06
wdutchI can see a problem15:06
wdutchI'm not sure what terminated15:07
richard_mawouch, is that from buildbot?15:07
wdutchbottlerock looks to be running15:07
* wdutch will just restart everything15:08
richard_mawwdutch: can we do that without buildbot losing state?15:08
wdutchI think interupted builds restart15:08
wdutchbut I could be wrong15:08
wdutchany sendchanges between two nodes will probably be lost15:09
wdutchbut I don't see how we could have any15:09
wdutchseems odd that bottlerock has a much lower pid that buildbot15:10
wdutchor does that not imply order once it's been running this long?15:11
richard_mawwdutch: correct15:11
richard_mawpids get reused all the time15:11
richard_mawthough there's a sysctl to increase the maximum pid number15:12
wdutchI knew that :) I didn't know if they eventually just cycled or what15:12
richard_mawthough generally you don't want to touch it, as if you have a wider range of pids, it can be harder to find a free one15:12
wdutchkill all the things!15:12
pedroalvarezI was wondering 10 minutes ago what was going on...15:23
pedroalvareznow I know15:23
pedroalvarezDev version of the UI:
wdutchnow we know15:23
wdutchsorry pedro15:23
pedroalvarezwdutch: no worries :)15:23
pedroalvarezkind of :)15:24
pedroalvarezI've been learning js15:25
pedroalvareztook me a while to do something useful15:25
pedroalvarezand of course, is code I'm not proud of15:25
*** doffm_ is now known as doffm15:25
radiofreepedroalvarez: i know it's a WIP, but there's a couple of issues with firefox15:26
pedroalvarezradiofree: please, go ahead with them15:26
pedroalvarezI hope nobody tries it in their phones :)15:26
paulsherwoodgreat idea!15:27
* paulsherwood does15:27
pedroalvarezthanks pedro..15:28
SotKit needs to not have the vertical centering on mobile, but other than that looks fine IMO15:28
paulsherwoodthe boxes appear in a column on safari/iphone15:29
paulsherwoodas opposed to a row by default15:29
radiofreepedroalvarez: ok, 1, the title text goes onto a new line at my resolution :
radiofreepedroalvarez: 2, the scrollbar on the right? It scrolls the header
radiofreepedroalvarez: 3, clicking something that requires you to scroll down (e.g arm build) results in a brand new scrollbar!
*** tiredcat has joined #baserock15:37
wdutchI didn't think that was possible any more :/15:38
*** tiredcat has quit IRC15:40
*** tiredcat has joined #baserock15:43
*** ssam2 has quit IRC15:49
*** ssam2 has joined #baserock15:51
*** ChanServ sets mode: +v ssam215:51
* richard_maw has a baserock-test AMI now16:07
pedroalvarezrichard_maw: ohh!16:10
tlsapedroalvarez: the css from my branch should have fixed radiofree's 2nd and 3rd things16:10
pedroalvarezradiofree: haha!16:11
pedroalvareztlsa: you are a star16:12
pedroalvarezI don't know what I've done with that merge..16:15
richard_mawthe hypervisor doesn't think it's crashed, but since I didn't upload a system with cloud-init, I doubt we'll actually be able to do anything with it16:15
* richard_maw wonders if there's a way to get a console16:15
KinnisonYou might be able to get the current full-content of the serial port output16:16
Kinnisonbut that's it16:16
Kinnisonit's possible that it DHCP'd okay16:16
pedroalvarezradiofree: try now?16:16
richard_mawKinnison: it says it has a public IP16:16
Kinnisonso if it has an SSH server open etc, then if the web UI shows you an IP then you could try SSHing to it16:16
radiofreepedroalvarez: marvellous!16:16
radiofreehowever clicking on the "arm build" forces the title to wrap (since there's now a scrollbar)16:17
pedroalvarezI think I'll make  it smaller16:17
radiofreemaybe get rid of "with baserock"?16:18
Kinnisonrichard_maw: any luck?16:18
pedroalvarezradiofree: done16:19
richard_mawKinnison: ssh connection timed out16:19
radiofreepedroalvarez: nice :D16:19
Kinnisonwhat security domain did you stand the server up in?16:19
richard_mawsecurity group is launch-wizard-2, it's got inbound SSH from anywhere and outbound everything16:20
*** toscalix__ is now known as toscalix16:20
SotKpedroalvarez: looks good!16:20
Kinnisonrichard_maw: hmm16:20
toscalixpedroalvarez: I suggest you deploy the changes you have done so far first thing tomorrow morning....not late this afternoon16:21
* toscalix advices: deployments early in the morning if possible16:22
*** tiagogomes_ has quit IRC16:28
richard_mawhmm, potentially we'd need to depoly with console=ttyS0 to get any logs16:30
richard_mawthough it could be a driver problem with xen16:31
pedroalvareztoscalix: I'll take your advice16:31
pedroalvarezSotK: don't look at the code :)16:31
*** toscalix has quit IRC16:32
*** toscalix has joined #baserock16:32
pedroalvarezare ther plans to add an extra Column in buildbot to publish after testing?16:33
KinnisonI'd hope there'll be a job to publish the changeset at minimum to a candidate review system16:34
toscalixwdutch: we need that, if you think you canoot make it, please let me know and I will find somebody to help16:35
toscalixI understand you have a lot in your plate already16:36
*** mariaderidder has quit IRC16:37
*** franred has quit IRC16:40
*** gtristan has quit IRC16:40
*** ssam2 has quit IRC16:50
*** paulwaters_ has quit IRC16:51
*** jonathanmaw has quit IRC16:54
*** toscalix has quit IRC16:58
*** toscalix has joined #baserock17:01
pedroalvarezit would be something simple, to move the file to the rigth place of a file server17:03
pedroalvarezI need to read the backlog, I missed a lot of things and I don't know where we are now17:03
wdutchyes a file could be moved somewhere by orchestration17:17
wdutchI do wonder if that's not a job for ciat-testing17:17
pedroalvarezNope, is something different17:26
SotKcan it not just push its change to Gerrit?17:26
pedroalvarezAfter testing we might want to send a patch to Gerrit, or publish the images, or send an email17:27
pedroalvarezMaybe all of them17:27
wdutchokay I'll set that up tomorrow17:27
SotKpedroalvarez: Also, too late, I already did! It's no worse than what I gave you to begin with :)17:39
pedroalvarezHeh, it's ok for now I think17:41
*** Stanto has quit IRC17:54
*** Stanto has joined #baserock17:59
paulsherwoodybd can and does already 'publish' if it knows about a kbas server. currently it only publishes chunks, and it needs a proper security scheme.18:03
paulsherwoodrjek: did you manage to take a look at that?18:03
*** toscalix has quit IRC18:53
rjekpaulsherwood: I looked, but not close enough sadly.  I have a plan this evening.22:22
*** paulsherwood has quit IRC23:33
*** paulsherwood has joined #baserock23:33
*** De|ta has quit IRC23:34
*** De|ta has joined #baserock23:34

Generated by 2.15.3 by Marius Gedminas - find it at!