IRC logs for #baserock for Tuesday, 2016-09-20

*** _longines has quit IRC01:24
*** _longines has joined #baserock01:24
*** gtristan has joined #baserock05:15
*** toscalix has joined #baserock07:37
*** rdale has joined #baserock07:43
*** CTtpollard has quit IRC08:26
*** CTtpollard has joined #baserock08:29
*** toscalix_ has joined #baserock08:33
*** toscalix has quit IRC08:34
*** locallycompact has joined #baserock08:58
*** CTtpollard has quit IRC09:01
*** CTtpollard has joined #baserock09:03
*** rdale has quit IRC09:04
locallycompactThis is going to be way more work than I expected09:09
rjekI find that is usually the case for any enterprise.09:10
*** rdale has joined #baserock09:21
*** toscalix has joined #baserock09:31
*** toscalix_ has quit IRC09:35
*** JPohlmann has quit IRC10:00
*** JPohlmann has joined #baserock10:00
*** gtristan has quit IRC10:38
*** ctbruce has joined #baserock10:57
*** gtristan has joined #baserock11:35
*** tardyp has joined #baserock12:05
tardyphello, I wanted to look at CIAT, and I can see that it is currently down http://ciat.baserock.org/12:06
pedroalvareztardyp: yeah, it has been down for quite a long time12:06
tardypthe buildbot is also not responding. Are you still using  it?12:06
pedroalvareztardyp: what do you mean by "the buildbot is not responding", I thought all of it was shut down12:08
tardypwell on http://wiki.baserock.org/ciat/12:08
tardypthere is a link to buildbot12:08
tardyphttp://ciat.baserock.org:8010/12:09
tardypand this page timesout12:09
pedroalvarezah, yes, what I expected12:09
tardypso CIAT is cancelled project?12:10
tardypwhat is baserock using then for CI?12:10
pedroalvarezCIAT was aiming to be more than a simple CI. It was aiming to auto-integrate new versions of upstream components, build them, deploy them and test them12:12
tardypthat what I understood12:12
tardypand your experience is that it was too difficult?12:13
pedroalvarezwe have other simple CIs in place, but it's not doing the same as CIAT was doing12:14
pedroalvarezwell, yes difficult, but doable12:14
tardypbut you still cancelled the project?12:15
pedroalvarezI guess we didn't have enough people interested on it12:16
pedroalvarezand keeping it running were it was, was expensive12:16
*** jonathanmaw has joined #baserock12:18
tardypok make sense.12:21
tardypso you dont use buildbot anymore, and now you use concourse, right12:24
tardypfull disclosure: I am th buildbot maintainer12:24
pedroalvareztardyp: well, we have a concourse instance (http://concourse.baserock.org/), but we have also "a loop in a shell script that builds latest master" (https://mason-x86-64.baserock.org/) , also some pipelines in Gitlab infra (https://gitlab.com/baserock/definitions/pipelines)13:19
pedroalvarezwe also tried to move to Zuul ..13:19
paulsherwoodtardyp: full disclosure, i'm the guy who thought concourse was sexier :013:21
paulsherwoodtardyp: i was suprised at the time by the reactions we got, though... i had hoped that folks in genivi/agl would be more interested13:21
paulsherwoodsince we expressly chose buildbot (in part) because it was already in use at yoctoproject13:22
paulsherwoodsomehow ciat as interpreted as 'baserock specific' and agl went/stuck with jenkins, genivi went with go.cd13:24
paulsherwoods/ciat as/ciat was/13:26
*** wdutch has joined #baserock13:26
locallycompactupdatesssss, makes actual graphs now https://gitlab.com/baserock/defslib13:32
* paulsherwood would love to see a readme/getting-started/video13:34
locallycompactsure sec13:35
locallycompacthave to stop coding anyway have python jitters13:35
paulsherwood:)13:35
paulsherwooddo we have to keep callign things 'stratum'?13:35
locallycompactwe do as long as it's in the spec13:36
tiagogomesstratum2?13:36
paulsherwoodoh, and you should apply a license and copyright13:36
tardypis there a link on genivi's go.cd instance?13:36
wdutchI guess the CIAT wiki page should be updated13:36
paulsherwoodtardyp: http://go.genivi.org/go/auth/login13:36
locallycompactAs long as we are running untyped the spec is king of what things are13:36
paulsherwoodtardyp: login instructions are on that page (!!!)13:36
tardyppaulsherwood: yep, I saw that13:38
paulsherwoodtardyp: can you share what you're interested in?13:39
tardypI am trying to understand the state of CI in the genivi ecosystem13:39
paulsherwoodtardyp: ok, best channel for that would be #automotive13:40
tardypsorry for the noise :)13:40
paulsherwoodthe users are there. also gunnarx is online there from time to time, and he's the guy who championed go.cd13:40
paulsherwoodtardyp: it's not noise, you're welcome here13:40
paulsherwoodwe still haven't figured out the perfect ci, i think13:41
paulsherwoodconcourse is pretty, and cool in lots of ways, but we had to massage some things13:43
paulsherwoodand they trust docker/containers way too much imo13:43
tardypI find it indeed pretty, but not quite obvious to look at13:43
tardypI am a little lost with the big svg home page13:44
tardypAlso I am not sure how you would make a precommit CI with this13:44
tardypi.e testing of PR/gerrit reviews13:44
paulsherwoodtardyp: that's one of the things we couldn't figure out either13:45
tardypthats a thing that I have a lot of experience on android systems13:45
paulsherwoodack13:46
paulsherwoodgitlab ci and travis seem to have it covered pretty well, without the extra overhead of gerrit13:46
* paulsherwood is not a gerrit fan.. but others here like it13:46
*** gtristan has quit IRC13:51
locallycompactpaulsherwood, https://gitlab.com/baserock/defslib13:55
pedroalvarezlocallycompact: s/README/README.md/ may help with the visualisation in Gitlab14:06
* locallycompact tries14:07
locallycompactyeha that looks better14:07
locallycompactbetter stlil14:09
locallycompacthttps://gitlab.com/baserock/defslib14:09
*** gtristan has joined #baserock14:17
paulsherwoodlocallycompact: so no plan for defslib.parse('./definitions') ?14:19
locallycompactas in the directory?14:20
paulsherwoodyup14:20
locallycompactI don't think it makes sense14:21
paulsherwoodand you're hardcoding to system/chunk/stratum14:21
locallycompactI'm writing to the spec14:21
paulsherwoodstill, you're hardcoding14:22
locallycompactNo, it's implemented against the spec14:22
paulsherwoodif i change the spec/schema, all of your code would have to change too?14:22
leemingno, cos he is coding against a versioned spec?14:22
locallycompactif you change the spec you change the types, I'd expect the code to change14:23
paulsherwoodhis function names feature words from the spec14:23
rjekWhy doesn't that make sense?14:23
* paulsherwood has been trying to undo the daft names we came up with for over three years... and people keep hardcoding them into more stuff14:25
locallycompactthe function names are a substitute for the fact that I can't implicitly deserialize, a la https://github.com/locallycompact/tremor/blob/master/src/lib.rs14:25
rjekIf implementing a spec that uses certain nouns, naming functions and variables with those nouns makes sense to me14:26
leemingagreed14:26
* paulsherwood thinks, not for the first time, that sw engineers should be forced to uses words of three syllables or less14:26
leemingnot sure if i follow the reasoning for that?14:27
leemingif a spec exists and defines names... you should follow convention in the code.. no?14:27
rdaleusing nouns with latin plurals should be avoided14:27
leemingbut that is more of an issue with the spec though? not implementation14:27
locallycompactI did also add an arbitrary parse_morph(filename), which will deserialize based on kind, if you'd prefer to use that14:28
leemingFTFY : parse_buildy_mc_build_tool(filename)14:30
leeming:)14:30
locallycompactI'm also absolutely fine with further demolishing the spec to get rid of name:, kind: and anything that isn't a local list of things with build-depends between them, in fact I think that's a good idea. Then we get arbitrary nesting and the like for quite little effort14:31
paulsherwood+114:32
locallycompactThe problem is people are waiting on this V9 in order to integrate things and I still don't know how to wedge it in to ybd14:32
leemingwell, the big issue of moulding bits of code, vs having a hard think from ground level14:33
paulsherwoodlocallycompact: if you can get to an optimum version of the spec (output) based on some migration of current and past definitions (input), i'd be happy to help you wedge it14:33
leemingmaintaining backwards compatibility is always an issue14:34
* paulsherwood wonders whether to bother refuting that :)14:35
paulsherwoods/refuting/arguing/14:35
paulsherwoodleeming: have you read http://wiki.baserock.org/back-and-forth/ btw?14:36
leemingpaulsherwood, I think that confirms what I indented to say, regardless of my english 'skills'14:37
leemingwhen i said issue, I mean, it is something to consider14:39
leemingnot that it is a PITA, if that was the way it was read? :\14:39
rdalei think the approach on the 'back and forth' wiki page is incomplete as it doesn't mention migrations14:40
leemingwell.. it is a PITA, but needs consideration... yeah... words.. I will stop trying to rephrase14:40
locallycompactbackwards is always behind us no matter which way we turn14:41
paulsherwoodrdale: agreed. folks would be welcome to improve the text14:41
jmacsBy "aiming for forward compatibility" do you mean working with specs you don't know about yet?14:41
paulsherwoodleeming: for one thing, it's only something to consider once people realise that it is. i wrote that page to try to tease out some of the things that we were clearly not realising14:42
paulsherwoodjmacs: 'Forward compatibility: user's old version still works with the new service/infrastructure/data'14:43
paulsherwoodbut other definitions are possible of course :)14:43
jmacsWhich implies that the old version was designed to take into account new data formats which did not exist when it was written14:44
paulsherwoodthat may or may not be possible, as you are clearly flagging. but (for example) a tool could at least *try* to parse some future data format, and do whatever it can, rather thna just saying 'your definitions are too new for me'14:47
jmacsSo you're saying that we should not deliberately stop forward compatibility. That sounds sensible, at least for non-safety-critical things14:49
* SotK thinks "your definitions are too new for me" is more helpful than everything breaking when you try to build definitions with a previously required field no longer present14:49
* paulsherwood will continue to disagree with SotK and others on this14:50
* locallycompact wonders what locallycompact thinks about this problem14:50
leemingwait.. you disagree with helpful error messages? :\14:50
leemingor was there some hidden context there14:50
paulsherwoodi disagree with error/exit when i could have warn/continue14:50
rjekerror please14:51
rjek-Werror14:51
paulsherwoodunless i specify that i want error/exit14:51
rjekOtherwise the warning will scroll off my screen unnoticed14:51
leemingError != warning14:51
paulsherwoodi'm the user. every time sw folks make software not work, on purpose, i sigh inside14:51
leemingif it is an error.. people exit with a message14:51
leemingpeople / please14:51
jmacsNo, you deliberately made it an error. It is a warning.14:52
leemingif it is a warning and my terminal isn't spammed, then I can act on it..14:52
paulsherwoodas i said 'do  whatever it can14:52
locallycompactwhat if "what it can" is calculating bad cache keys and uploading blank artifacts14:53
rjek"I have no idea how to interpret these instructions, so I will guess" is never the right thing for software to do, IMO14:53
paulsherwoodrjek: bonus points for making it easy for users to decipher warnings, and not flooding with too many of them14:53
SotKrjek: +114:53
leeminglocallycompact, I think I've actually had this issue before14:53
* richard_maw prefers failing early to failing late or not even knowing that you've failed14:53
leemingagree. early fail is much better14:53
leemingrjek, also bad security/reproducibility14:53
rjek+114:53
paulsherwoodjust because you guys agree, doesn't make you correct :)14:54
leemingnot a dig here... but... ybd floods the stdout with non errors/warnings14:54
rjekFailing an hour in because of a trivial typo or something that could have been errors at instantly is a ballache14:54
leemingybd has loads of warnings.. many are ignored due to them flying off the screen14:55
paulsherwoodleeming: agreed14:55
leemingthus why I was trying to add in extra logging for ybd14:55
leeming(as in to file)14:55
jmacsThis isn't the first project to have this issue. Warnings, with specific warnings or all warnings made errors with -Werror, is the usual solution.14:55
jmacsAs rjek said.14:56
leemingELI5, -Werror?14:56
jmacsMakes warnings act as errors.14:56
leemingok14:56
richard_mawgcc style also has -Werror=foo if there's a specific class of warnings you consider to be errors14:58
richard_mawas in there's -Werror=vla if you need your program to be portable to compilers that don't support vlas14:58
paulsherwoodfwiw, the main reason ybd 'floods the stdout with non errors/warnings' is to record that our actual definitions have inconsistencies in them. if ybd errored on them by default, it wouldn't build any previous definitions, which from a user perspective would be not useful. it *can* error and exit, but that's not the default15:03
paulsherwoodsome of the inconsistencies are not even accepted as problems by others here (eg name of file != name of chunk). i expect that if we'd started from locallycompact's perspective of a completely tight schema, we'd have avoided these issues, but we didn't realise that 'back in the day'15:06
paulsherwoodthis project is (and has been) a learning experience, if nothing else :)15:06
locallycompactmmm15:07
locallycompactwhen is the five year learning experience in haskell? ^_^15:07
leemingafter the year of linux on the desktop15:07
rjekLinux will never lead on the desktop because nobody uses desktops anymore.  It's already won on phones and cars though, and Google now sells as many Chromebooks as Apple does expensive fisher price toys for adults :)15:08
paulsherwoodwon on cars? perhaps in some parallel universe that's true15:09
jmacsWhoever wins that, we lose15:10
* rjek nods sadly15:11
paulsherwoodtardyp: continuing that build discussion from #automotive here, since i think most #automotive folks don't care that much about build systems15:23
tardypah15:23
rjekThey *should*15:23
paulsherwoodhttps://ninja-build.org/manual.html15:23
paulsherwoodmaybe they do, i've been wrong before :)15:23
tardypits very good for incremental builds for huge builds like android were everything is in one Makefile15:24
tardypfor a componentified build like baserock/yocto it is different15:25
paulsherwoodack15:25
paulsherwoodbut why are they sticking with one Makefile?15:25
tardypthere are huge pros and cons for the two approach15:25
tardypone build simplify the message for the build15:26
* paulsherwood doesn't understand15:26
tardypyou just have all the source code here, and you can change which ever source code 2 minutes after its reflashfed on your device15:26
tardypthen you repo upload your change and that is done15:27
tardypI'm not sure about baserock, but with yocto, the developer workflow is very complex15:27
tardypyocto is for integration, and does not help for development15:28
paulsherwoodack15:28
tardypwhile the android system helps both teams15:28
tardypat the price of getting rid of the componentification15:28
tardypand letting the dev do -I ../../../kernel in there userland15:28
paulsherwood:)15:29
tardypI've seen that much more than you can imagine15:29
paulsherwoodtardyp: does it require all the source locally?15:29
tardypit does15:29
tardyp80GB per work environment minimum15:30
locallycompactidd15:30
tardypincluding the generated binaries15:30
paulsherwoodtardyp: so to show where we've got to with the baserock workflow15:33
paulsherwoodi just made a change direct in gitlab https://gitlab.com/devcurmudgeon/definitions/commit/fd163b1784491a41d97476f56b36c9c49250c00315:34
paulsherwoodit's just bumping the kernel to whatever is master from upstream15:34
paulsherwooda build is running https://gitlab.com/devcurmudgeon/definitions/builds/424604215:35
paulsherwoodit's checking if any of the things needed for the whole system are already built (and if so is downloading them from a separate cache)15:35
tardypthats one of the big advantage of the component system15:36
paulsherwoodeventually (i hope) it will realize we've not built this version of the kernel before, and will clone and build it15:36
tardypand for the developer workflow? I want to make a patch to that kernel. how would I do it?15:37
paulsherwoodeither update the repo: to point to your local checkout of linux or whichever git server you put the patch on15:38
tardypok15:39
tardypgitlab hosted does not help the cache part to be very fast though15:39
paulsherwoodwe use upstream: by default as an alias for a server hosting the foss repos15:40
paulsherwoodtrue. that's one of the issues we also have with concourse... both want the runners to be isolated, no state from previous builds15:40
paulsherwoodwhich makes sense, except when we need to download 1GB of pre-built content for each run :)15:41
tardypor you need to store your cache very near from your workers15:41
paulsherwoodyup. for private workers or on developer machine, we can keep artifacts locally too15:42
tardypthose days, I am working with hyper.sh startup15:47
tardypthey have very cheap container aas which can be useful for such CI15:47
paulsherwoodoh, interesting15:47
tardypnotably you could attach a docker volume pointing to prebuilt artifacts to your worker instances15:48
leemingseems sensible15:49
leemingthe gitlab creating new instances each time, is a little... tiresome15:49
tardypthey have an API which is derived from docker15:51
tardypso if gitlab is already supporting docker, should be able to quickly support hyper15:51
paulsherwoodrequirements vary ... for ybd releases i prefer just a c4.8xlarge AWS machine... it's fastest wallclock time15:51
tardypI have made the buildbot port in a few days15:51
paulsherwoodcool15:52
mwilliams_ctpaulsherwood: oh, speaking of fast instances, you might want to take a look at https://aws.amazon.com/about-aws/whats-new/2016/05/now-available-x1-instances-the-largest-amazon-ec2-memory-optimized-instance-with-2-tb-of-memory/. x1.32xlarge should be faster again (though I'm not convinced it makes economic sense)15:53
richard_mawmwilliams_ct: sheesh, you'd spend a bundle with the CPU time spend loading stuff into that memory!15:54
mwilliams_ctrichard_maw: but those numbers are so big and shiny!15:54
* paulsherwood is actually tempted to spin one up15:54
mwilliams_ctpaulsherwood: record the build if you do!15:55
paulsherwoodof course15:55
tardypwhat is the average build time for a full baserock build?15:55
CTtpollardwow15:55
rjekTIL: 1,952MB is 2TB15:58
mwilliams_ctrjek: nope, reread15:58
mwilliams_ct"Instance Memory (GiB)"15:58
paulsherwoodtardyp: it very much depends on which system is built, and what it's run on16:12
tardypobviously, but for your use case of ybd releaes on thos c48x machines16:13
tardypwhat is the order of idea?16:13
* paulsherwood is looking for specific examples16:13
tardypbehind that, is hyper per sec billing interresting for your load16:14
paulsherwoodhttps://raw.githubusercontent.com/devcurmudgeon/build-logs/master/aws-c4.8x-large-4x9.log16:14
paulsherwoodpossibly interesting, yes16:14
paulsherwoodso that log is old, back when our default ci included just openstack, genivi and a couple of other systems16:15
paulsherwoodit took (as you can see) around 90 minutes. that's from scratch, but without pulling all of the git repos16:16
paulsherwoodsince then we've added a full gnome system, which has increased the wallclock time dramatically (since WebKitGTK is a monster, and other stuff is slow also)16:17
pedroalvarezwebkitgtk is a double sized monster, given that we buld 2 different versions of it16:18
paulsherwoodtardyp: while logging various things, i concluded that it didn't make sense to parallelise an indivdiual compile beyond ~ 8-10 cores https://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJjbxMSppiMcg/edit#gid=24568023616:20
paulsherwoodso that ybd build is actually running up to four separate builders at once when it can16:21
tardypinterresting. Our android infra uses 72 core xeons :)16:21
tardypor.. no thats 40 core, with 72GB RAM16:22
paulsherwoodas one thread with max-jobs=40 ?16:22
paulsherwoodsorry -j=4016:22
tardypthat is also one of the interresting fact about single makefile, you can use -j40 and have the kernel and the java stuff built at the same time16:22
tardypthe yocto build I did was also using those 40cpu quite well by doing several recipes at the same time16:23
paulsherwoodack16:23
tardypI use -j40 *and* parrallel build = 4016:23
tardypI always wonder how it didn't blow up as it is supposed to be actually 40x40 thread worst case16:24
paulsherwoodok, i hazard a guess your wallclock time might be faster with -j1016:24
paulsherwoodbut i may be wrong, of course.16:24
tardypmaybe. I did not try16:24
tardypI did not do the big study on build time yet16:25
* paulsherwood has stared at a lot of builds16:25
paulsherwoodhttps://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJjbxMSppiMcg/edit#gid=133680405816:25
tardypat the moment, yocto was just PoC level for my team16:25
tardypstill no permission :)16:25
paulsherwoodhttp://wiki.baserock.org/build-times/16:25
* paulsherwood personally doesn't like yocto very much... this is no secret16:26
tardypI'd guess that16:27
paulsherwood:)16:27
paulsherwoodsorry, i screwed up the permission... fixed now16:28
leemingwait, you don't like yocto?16:30
richard_mawleeming: Quick! Smiley face!16:30
paulsherwoodlol16:30
leeming:)16:30
* paulsherwood has been called out in public for speaking about it16:30
* paulsherwood will be quiet now16:31
leemingwell the yocto book says good things about us ;) - https://scontent.xx.fbcdn.net/v/t35.0-12/14438982_1308509195836742_355956326_o.jpg?oh=d45e16d67444f96b53ccd890ae67c295&oe=57E335DD16:31
* leeming isn't responsible for the fuzz16:31
tardyppaulsherwood: so this is build time for a single component16:34
tardypI would have expected gcc to scale better though16:35
tardyp link is often a big part of that time, and its not //16:35
paulsherwoodtardyp: yup16:37
*** fay_ has quit IRC16:37
richard_mawgcc has c++ in it these days, which slows things down dramatically, and it has to build the compiler 3 times for its own internal bootstrapping16:38
leemingah, wondered why gcc was so slow16:38
leemingrelative to others16:38
jmacsI'd suggest you are hitting the limit of I/O on those machines16:41
*** ctbruce has quit IRC16:49
*** anahuelamo_ has joined #baserock16:59
*** anahuelamo has quit IRC16:59
*** ctbruce has joined #baserock17:03
*** jonathanmaw has quit IRC17:05
*** toscalix has quit IRC18:13
*** ctbruce has quit IRC19:48
*** rdale has quit IRC20:08
*** locallycompact has quit IRC20:22

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!