IRC logs for #baserock for Friday, 2017-02-10

*** gtristan has joined #baserock07:05
*** fay has joined #baserock08:14
*** fay is now known as Guest9412308:15
*** ctbruce has joined #baserock08:27
*** paulwaters_ has joined #baserock08:31
*** rdale has joined #baserock08:56
*** tiagogomes has joined #baserock08:56
*** noisecell has joined #baserock09:02
*** toscalix has joined #baserock09:10
*** jonathanmaw has joined #baserock09:22
paulsherwoodelsewhere i'm seeing folks fiddling with -j again09:25
paulsherwoodback in the day, morph had an algorithm to set max-jobs: ncores * 1.509:25
paulsherwoodbecause a couple of experienced engineers had found, in their experience, that it gave fastest results09:26
paulsherwoodi tested that assumption for ybd, because i was watching *lots* of morph builds and ybd builds, and total elapsted time was an important metric for me09:26
gtristanSounds perfectly sensible, but continue :)09:26
paulsherwooddisclaimer: there may be some gaps in my testing method, since i am a dilettante at this... i only cared about optimising wallclock time, for a given kind of workload (ybd ci.morph) on the fastest kit i could find09:27
paulsherwoodso, i noticed that much time was spent building big things - qtwebkit, gcc for example09:29
paulsherwoodi assumed that these projects were as parallelisable as possible - their upstreams, just like me, must care to make their builds as fast as poss09:29
paulsherwoodusing ybd i was able to time creeation of qtwebkit and gcc artifacts in isolation, on huge AWS machines09:30
paulsherwoodi could try max-jobs: 1,2,4,8,10,16,24,32 etc09:30
gtristanhere we're talking about one build in isolation and configuring -j <jobs>09:31
paulsherwoodthe graphs are https://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJjbxMSppiMcg/edit#gid=24568023609:31
* gtristan just noting09:31
paulsherwoodso basically that says to me, at least, that for these well established jobs, i don't get a huge benefit beyond 10 cores09:32
gtristanHow many cores did that machine have btw ?09:32
gtristanmx4.10xlarge means 10 cores ?09:32
paulsherwoodhttps://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJjbxMSppiMcg/edit#gid=133680405809:32
paulsherwoodthat's two different kinds of AWS machines09:33
paulsherwoodmx is 'memory optimised'09:33
paulsherwoodcx is 'compute optimised'09:33
paulsherwoodhttps://aws.amazon.com/ec2/instance-types/09:33
paulsherwoodso, then i used that info, and tried to optimise for elapsed time on ci.morph as a whole...09:34
paulsherwoodAWS cx4.8xlarge gives me better price/performance than mx, so i focused on that09:35
gtristanI dont get how many cores... from that graph... maximum is 8 cores ?09:35
gtristanoh below09:36
gtristandoes vCPU mean number of cores ?09:36
paulsherwoodyup i think so09:37
paulsherwoodand actually my graph calls it cores, but really i mean the number after -j09:37
gtristanyes I get that, I'm wondering about the machine constraints :)09:37
paulsherwoodunderstood. this can be affected by memory, io etc09:38
gtristanAssuming unlimited cores, or cores > max-jobs... we reach the average limit of 'number of object files to generate in a given directory'09:38
paulsherwoodbut anyway... given that broadly i saw much less gain in elapsed time once you get beyond 10 or so cores, i fiddled with instances of ybd vs max-jobs, and compared results09:39
gtristangcc has one specific directory that has (I would think) vastly parallelizable objects09:39
paulsherwoodthat's not what my results show, iiuc09:39
paulsherwoodit's bound to be an exponential curve at best anyway09:39
gtristanSure, so far they show to me that "there is no noticeable performance gain by reducing the number of max-jobs"09:40
gtristanbut I'm missing the next part of your experiment :)09:40
paulsherwood"there is no noticeable performance gain by INCREASING the number of max-jobs" either. and those extra cores cost money09:40
gtristanThat.09:41
* paulsherwood pays for his AWS instances, which helps to focus the mind09:41
paulsherwoodso i trade off the cost of the machine time, versus the cost of my own time09:41
gtristanpaulsherwood, that's a very particular case indeed, you are saying that spawning a task will magically cost more money on some fancy service which detects that a process is launched and hands it more resources ?09:42
paulsherwoodand fastest wallclock time i could get for ci.morph (builds, not git clones) was achieved on cx4.8, 4 instances, 8 cores each09:42
paulsherwoodno, not quite. i have a lot of builds to do, on a constrained machine, and want to get them done in fastest possible time09:43
paulsherwoodthat's not so uncommon09:43
gtristanOn my machine (most machines ?), spawning a task which is in a wait state does not cost more energy09:43
paulsherwoodyou are considering your personal machine. i'm considering shared/cloud/ci09:44
gtristanI am considering a machine with it's own resource limitations of course09:45
paulsherwoodif one build has 32 cores on a 32 core machine, even if most of them are in wait state, other builds are affected, afaict09:45
paulsherwoodymmv09:45
gtristanpaulsherwood, right now I suspect you may be onto something, but it's not clear to me what that is exactly09:47
paulsherwoodheh09:48
paulsherwoodonce you figure it out, i'll agree that's what i meant all along09:48
paulsherwoodgtristan: my results are empirical. fastest wallclock for the whole load, for the fastest/biggest kit i could afford09:49
gtristanHave we proven that spawning processes in wait state in advance of actually processing them, incurs a non negligible overhead ?09:49
paulsherwoodi didn't dig into that level of detail - these tests were quite time-consuming and expensive as it is :)09:49
paulsherwoodi did collect the logs, though09:49
gtristanthe tests should be checking "what does it cost us more to spawn additional tasks in advance"09:49
paulsherwoodfrom my pov, the wallclock time was the metric i wanted to optimise, period09:50
paulsherwoodand as someone paying for both engineer time and computer/cloud time, that's the one that i continue to care about09:50
gtristanAnd you found that "reducing number of spawned jobs does not cause the build to take longer to complete", correct ?09:50
gtristanBut are worried that for some reason, a build which spawns more tasks in advance may be more costly09:51
paulsherwoodi wasn't testing that. i was just interested in getting to fastest builds, by varying instances vs max-jobs, after selecting highest perf/price aws09:51
gtristanI dont see how that follows, but that's the metric we want09:52
paulsherwoodiirc wallclock ci.morph for instances: 4, max-jobs: 9 was slower than instances: 4, max-jobs: 8 on a 32 core machine09:52
paulsherwoodand instances:5 max-jobs anything was slower still09:53
paulsherwood(but i'd have to check the logs, and i'm unable to spend more time on this now, sorry)09:53
* paulsherwood notes that ybd already got to be way faster than morph, by being able to support instances: >1, and the knowledge that max-jobs: cores was not normally slower than max-jobs: cores*1.509:55
gtristanAn interesting thing to check would be comparing "max-jobs = cores / instances" on a 32core machine... vs the same configuration on a measly 8core laptop09:57
gtristan(or 8thread)09:57
gtristanI am certain that compiling webkit with only 4 threads while my other instance is crunching configure scripts, is wasting 3 threads most of the time09:58
paulsherwoodgtristan: it's relatively easy for you to test this, i think :)09:59
gtristan(that is quite frustrating :))09:59
gtristansure, we'll do some benchmarking then, after we have conversions :)09:59
* paulsherwood did experiment with instances: 2, max-jobs: 8 on 8-core, but can't remember the outcome09:59
* gtristan likes benchmarks09:59
paulsherwoodhttps://github.com/devcurmudgeon/build-logs and http://wiki.baserock.org/build-times/10:01
*** CTtpollard has quit IRC10:03
*** CTtpollard has joined #baserock10:06
*** locallycompact has joined #baserock10:08
*** ssam2 has joined #baserock10:11
*** ChanServ sets mode: +v ssam210:11
*** jude_ has joined #baserock10:14
*** ssam2 has quit IRC10:14
*** ssam2 has joined #baserock10:16
*** ChanServ sets mode: +v ssam210:16
*** tiagogomes has quit IRC10:23
*** locallycompact has quit IRC10:30
jonathanmawHas something gone wrong with ybd's CI? the cache_keys stage has been failing with an ugly stack trace even after I reverted my newest changes.10:30
jonathanmawhttps://gitlab.com/jonathanmaw/ybd/builds/10240954 for example10:30
pedroalvarezjonathanmaw: yes10:31
paulsherwoodjonathanmaw: there was a problem yesterday, https://gitlab.com/baserock/ybd/commit/0af8892a267e9d7f4534c25c1d27feab7ddb956310:31
jonathanmawah, ok.10:32
*** cornel has quit IRC10:46
pedroalvarezI wish gitlab wasn't stuck in "Checking ability to merge automatically…"10:57
pedroalvarezi don't know if i will have to rebase, and re-trigger the CI10:57
pedroalvarezor not10:57
pedroalvarezlots of results in google regarding this, none of them useful11:01
*** locallycompact has joined #baserock11:01
*** tiagogomes has joined #baserock11:06
*** CTtpollard has quit IRC11:12
*** CTtpollard has joined #baserock11:12
jjardonpedroalvarez: yeah, seems a problem since the last upgrade12:25
jjardonCan someone else approve pedro's MR so we can merge it? https://gitlab.com/baserock/definitions/merge_requests/3012:27
paulsherwoodjjardon: done13:04
pedroalvarezthanks13:05
jjardonthansks for the updates pedroalvarez !13:36
*** gtristan has quit IRC14:32
*** jonathanmaw has quit IRC16:35
*** toscalix has quit IRC16:37
*** jonathanmaw has joined #baserock16:51
*** toscalix has joined #baserock16:54
*** ctbruce has quit IRC17:01
*** jonathanmaw has quit IRC17:06
*** Guest94123 has quit IRC17:16
*** noisecell has quit IRC17:18
*** jonathanmaw has joined #baserock17:18
*** jonathanmaw_ has joined #baserock17:19
*** toscalix has quit IRC17:21
*** toscalix has joined #baserock17:24
*** jonathanmaw_ has quit IRC17:48
*** jonathanmaw has quit IRC17:48
*** locallycompact has quit IRC17:53
*** locallycompact has joined #baserock18:01
*** CTtpollard has quit IRC18:04
*** locallycompact has quit IRC18:24
*** toscalix has quit IRC18:28
*** toscalix has joined #baserock18:28
*** toscalix has quit IRC19:00
*** tiagogomes_ has joined #baserock19:04
*** tiagogomes has quit IRC19:04
*** gtristan has joined #baserock19:09
*** paulwaters_ has joined #baserock19:19
*** ssam2 has quit IRC19:24
*** tiagogomes_ has quit IRC20:33
*** gtristan has quit IRC21:03
*** toscalix has joined #baserock21:34
*** toscalix has quit IRC22:45

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!