| *** dabukalam has quit IRC | 00:27 | |
| *** pedroalvarez has quit IRC | 00:27 | |
| *** bfletcher has quit IRC | 00:28 | |
| *** pedroalvarez has joined #baserock | 00:29 | |
| *** ChanServ sets mode: +v pedroalvarez | 00:29 | |
| *** dabukalam has joined #baserock | 00:35 | |
| *** bfletcher has joined #baserock | 00:42 | |
| *** doffm has quit IRC | 02:15 | |
| *** doffm has joined #baserock | 02:15 | |
| *** ratmice___ has quit IRC | 02:27 | |
| *** ratmice___ has joined #baserock | 02:28 | |
| *** zoli__ has joined #baserock | 05:13 | |
| *** zoli__ has quit IRC | 06:19 | |
| *** zoli__ has joined #baserock | 06:41 | |
| *** franred has joined #baserock | 06:51 | |
| *** petefoth has quit IRC | 07:23 | |
| *** petefoth has joined #baserock | 07:36 | |
| *** mariaderidder has joined #baserock | 07:49 | |
| *** mdunford has joined #baserock | 07:57 | |
| *** gary_perkins has joined #baserock | 08:03 | |
| *** bashrc_ has joined #baserock | 08:06 | |
| *** tiagogomes_ has joined #baserock | 08:07 | |
| *** mike has joined #baserock | 08:18 | |
| *** mike is now known as Guest46605 | 08:18 | |
| *** ssam2 has joined #baserock | 08:38 | |
| *** ChanServ sets mode: +v ssam2 | 08:38 | |
| *** jonathanmaw has joined #baserock | 08:46 | |
| *** sambishop has joined #baserock | 08:59 | |
| *** lachlanmackenzie has joined #baserock | 09:19 | |
| *** paulw has quit IRC | 09:41 | |
| *** sambishop has quit IRC | 09:50 | |
| *** paulw has joined #baserock | 10:00 | |
| *** sambishop has joined #baserock | 10:03 | |
| paulsherwood | is it true that morph-cache-server is now in morph? | 10:18 |
|---|---|---|
| perryl | yes | 10:18 |
| perryl | the separate git repo seems to be marked obsolete on GBO | 10:18 |
| perryl | s/separate/morph-cache-server/ | 10:19 |
| paulsherwood | that invalidates most of the email i've been writing, then.... | 10:19 |
| paulsherwood | ok... i'll re-phrase :) | 10:19 |
| perryl | AFAICT everything from mcs.git has just been moved to a directory inside morph.git as-is, i could be wrong though | 10:20 |
| ssam2 | it would be trivial to move it back out again | 10:27 |
| ssam2 | as it is 2 files | 10:28 |
| *** sambishop has quit IRC | 10:35 | |
| *** sambishop has joined #baserock | 10:36 | |
| *** sam__ has joined #baserock | 10:40 | |
| *** sambishop has quit IRC | 10:41 | |
| *** zoli__ has quit IRC | 10:48 | |
| *** petefoth has quit IRC | 10:50 | |
| *** petefoth has joined #baserock | 10:55 | |
| *** zoli__ has joined #baserock | 11:15 | |
| *** zoli___ has joined #baserock | 11:35 | |
| *** zoli__ has quit IRC | 11:39 | |
| *** tiagogomes_ has quit IRC | 11:44 | |
| *** tiagogomes has joined #baserock | 11:44 | |
| *** franred has quit IRC | 11:48 | |
| *** franred has joined #baserock | 12:08 | |
| *** sam__ has quit IRC | 12:39 | |
| *** sambishop has joined #baserock | 12:53 | |
| *** sambishop has quit IRC | 13:44 | |
| paulsherwood | 15-07-01 15:03:31 [1/273/273] [stage1-binutils] Elapsed time 00:02:38 | 14:06 |
| paulsherwood | on thunderX... | 14:06 |
| persia | heh | 14:07 |
| persia | How many parallel jobs? | 14:07 |
| rjek | And how long does it take on other machines? | 14:08 |
| paulsherwood | 1 minute on macbook. this is max-jobs=1 | 14:09 |
| nowster | paulsherwood: I take it that's ybd? | 14:09 |
| persia | paulsherwood: Try max-jobs=96 | 14:09 |
| * rjek finds that Macbooks have surprisingly good storage I/O | 14:09 | |
| rjek | How much RAM does each core have on ThunderX? | 14:10 |
| rjek | (I'm assuming it's all shared, but what is ram_size/cores?) | 14:10 |
| persia | 64G/48 cores | 14:11 |
| persia | At least for the low-end servers. Some folk have bigger servers. I've heard of 256G/96cores, but haven't actually touched such a unit. | 14:11 |
| rjek | Not much RAM per core for doing lots of builds, then | 14:11 |
| rjek | And how is RAM arranged? ie, how many banks and thus how much access must be serialised/ | 14:12 |
| rjek | ? | 14:12 |
| persia | Hmm. Good point. max-jobs=24 is a better arrangement. | 14:12 |
| * rjek is thinking that those sort of CPUs might be great for doing processing that fits entirely in cache | 14:12 | |
| * persia doesn't know enough about paulsherwood's specific server. | 14:12 | |
| rjek | persia: Do a build with each of max-jobs=${seq 1 96) and report back with a histogram >:) | 14:13 |
| rjek | $( | 14:13 |
| persia | The results of that would be interesting, actually. | 14:14 |
| *** sambishop has joined #baserock | 14:15 | |
| persia | I don't think we've played much with Baserock on systems with more than 8 cores, so we haven't really been able to measure the right core/memory ratio for building. | 14:18 |
| *** petefoth has quit IRC | 14:19 | |
| paulsherwood | persia: no difference with 96 afaict | 14:21 |
| paulsherwood | nowster: yes | 14:21 |
| persia | Interesting. I wonder what is taking the time. | 14:21 |
| ssam2 | ./configure | 14:21 |
| Kinnison | I strongly doubt that most workloads will parallelise that greatly | 14:21 |
| Kinnison | you'd do better having multiple less parallel workloads | 14:21 |
| persia | Hrm? Doesn't max-jobs=96 mean that it will kick off as many builds within a strata as it can (up to 96)? | 14:22 |
| persia | Or, actually, if we're only doing stage1-binutils, that's not something that can parallelise well. | 14:23 |
| *** mariaderidder has quit IRC | 14:23 | |
| ssam2 | persia: In Morph, --max-jobs N is used to set MAKEFLAGS to -j N | 14:28 |
| ssam2 | that's all | 14:28 |
| Kinnison | indeed | 14:29 |
| paulsherwood | persia: it doesn't parallelise itself (yet) | 14:33 |
| persia | :( | 14:33 |
| Kinnison | That's what distbuild was for, for morph | 14:34 |
| paulsherwood | persia: i need to iron out how to clean up dead tmp directories is all... that proved to be a surprisingly complex topic on here a few days ago | 14:34 |
| paulsherwood | actually, i could let it go without that, and see 'does not clean up' as a tolerable bug | 14:35 |
| *** mariaderidder has joined #baserock | 14:36 | |
| *** JPohlmann has joined #baserock | 14:41 | |
| *** JPohlmann has joined #baserock | 14:41 | |
| *** Zara_ has joined #baserock | 14:42 | |
| *** radiofree_ has joined #baserock | 14:43 | |
| *** juergbi` has joined #baserock | 14:44 | |
| *** bashrc_ has quit IRC | 14:45 | |
| *** Zara has quit IRC | 14:46 | |
| *** radiofree has quit IRC | 14:46 | |
| *** juergbi has quit IRC | 14:46 | |
| *** JPohlman1 has quit IRC | 14:46 | |
| *** tiagogomes has quit IRC | 14:57 | |
| *** bashrc_ has joined #baserock | 14:58 | |
| *** tiagogomes has joined #baserock | 15:10 | |
| * persia digs through backscroll to develop context for the tmp directories issue | 15:14 | |
| persia | Ah, I was actually part of that discussion. Ignoring the digression on how to make temporary directories secure, the usual way is either to use lockfiles (as Kinnison suggested) or for the controlling program to track the temporary directories, trap on common signals, and attempt to remove directories before exiting. | 15:19 |
| paulsherwood | in this case the controlling program doesn't want to remove directories on exiting | 15:24 |
| paulsherwood | and/or there is no 'controlling program' | 15:24 |
| paulsherwood | i think my use case is: n instances of ybd run. some fail. the temp directories contain useful debris from the crash | 15:24 |
| * richard_maw would like a tmpdir-daemon which you request a tempdir from, and if you close the fd the daemon cleans up the tempdir, so it's done automatically when your process exits | 15:25 | |
| paulsherwood | but eventually this fills up disk, so i need some periodic cleanup to notice that some tmp dirs have been untouched and therefore are probably useless | 15:25 |
| persia | richard_maw: +1 | 15:25 |
| richard_maw | persia: you'd also need it to clean up any tempdirs that are on-disk in its managed area which it doesn't recognise, so it can cleanup on system crash | 15:26 |
| paulsherwood | richard_maw: maybe offer that as a systemd function? :) | 15:26 |
| richard_maw | paulsherwood: I got the inspiration from the systemd-logind inhibitor functionality | 15:27 |
| persia | paulsherwood: I'd recommend adding a stamp file to each tmpdir when it is created, with the PID of the running integration tool. When the reaper runs, it can check the creation time of the stamp file, and whether the PID in question is in use, and use that as the basis for a decision. | 15:27 |
| richard_maw | alternatively you can rely on your OS' built-in tempdir management | 15:27 |
| Kinnison | PID reuse is a real race condition people don't take account of :( | 15:28 |
| richard_maw | systemd will clean up tempfiles older than a specified period | 15:28 |
| richard_maw | but only if systemd is managing the directory used | 15:28 |
| persia | Kinnison: Good point. Maybe PID+program start time? | 15:31 |
| richard_maw | that was what kdbus was doing to mitigate the problem | 15:32 |
| richard_maw | as expected it got a bunch of hate, but is the best approach currently, though now I think people are working on some kind of pidfd | 15:32 |
| *** mdunford has quit IRC | 15:46 | |
| *** paulw has quit IRC | 15:59 | |
| *** franred has quit IRC | 16:01 | |
| *** Guest46605 has quit IRC | 16:01 | |
| *** mdunford has joined #baserock | 16:02 | |
| *** Guest46605 has joined #baserock | 16:17 | |
| richard_maw | :¬/ you wouldn't be able to pass a fd of the thing you created to the daemon and have it able to remove it, since only linkat has AT_EMPTY_PATH, so you'd either need to pass a dirfd and a basename over the socket, or have the daemon be the one to make the tempdir, and deal with having to check whether you're allowed to make the tempdir on behalf of the client process | 16:18 |
| *** zoli___ has quit IRC | 16:19 | |
| *** sambishop has quit IRC | 16:26 | |
| *** edcragg has joined #baserock | 16:32 | |
| *** mariaderidder has quit IRC | 16:32 | |
| *** jonathanmaw has quit IRC | 16:32 | |
| *** Guest46605 has quit IRC | 16:35 | |
| edcragg | this baserock system is telling me there's no space left on /tmp, which is clearly not true :( | 16:45 |
| persia | Are you using morph? If so, did you create a morph.conf? | 16:46 |
| persia | The error message for not doing so can be confusing. | 16:47 |
| edcragg | it's not morph, just running some python | 16:47 |
| persia | Then the answer to the question is entirely dependent on the system. Are you running a reference system? If so, which? How was it deployed? | 16:47 |
| edcragg | jetson, the usual flashing method | 16:48 |
| *** paulw has joined #baserock | 16:48 | |
| edcragg | it was the current release around a week or so ago | 16:48 |
| richard_maw | edcragg: when does it claim to not have any space in /tmp, and how are you proving it should have enough space? | 16:48 |
| edcragg | richard_maw: http://paste.baserock.org/oziwijavec | 16:49 |
| edcragg | it's also the same if i unmount /tmp and leave it on the rootfs | 16:50 |
| paulsherwood | win 58 | 16:51 |
| paulsherwood | gah! | 16:51 |
| richard_maw | edcragg: it looks like you're running a deploy of some form, you're right that the tempdir isn't too small, but you've got /dev/sda mounted into a tempdir in /tmp | 16:51 |
| richard_maw | so it's your / that is too small | 16:51 |
| richard_maw | s/\/dev\/sda/\/dev\/mmcblk0p2/ | 16:52 |
| edcragg | i'm still confused... / has plenty of space too | 16:54 |
| edcragg | yes, i'm running a deployment extension directly with python | 16:54 |
| richard_maw | how big is the thing you want to deploy onto it? | 16:57 |
| edcragg | it's around 1.4 GB | 16:59 |
| paulsherwood | edcragg: interesting - you may be the first person to run directly with python :) | 17:00 |
| richard_maw | paulsherwood: incorrect, I've done it before. | 17:00 |
| paulsherwood | ah, i should have known! :) | 17:00 |
| richard_maw | edcragg: what does `btrfs filesystem df` say? | 17:00 |
| * paulsherwood is the first to run ybd on thunderx though :-) | 17:01 | |
| edcragg | richard_maw: http://paste.baserock.org/abeludavak | 17:02 |
| *** bashrc_ has quit IRC | 17:03 | |
| richard_maw | hmm, should still be plenty available | 17:03 |
| richard_maw | wait… is this an upgrade or a raw disk deployment? | 17:04 |
| edcragg | raw disk | 17:04 |
| richard_maw | since if it's the latter, then you may need to change the DISK_SIZE variable | 17:04 |
| edcragg | change it to what? it's set to 7G | 17:04 |
| edcragg | it has been working for some time, just suddenly stopped this afternoon | 17:05 |
| richard_maw | hmm, it's not the bug I was thinking of then | 17:06 |
| richard_maw | s/bug/failure/ | 17:06 |
| edcragg | it's a mystery to me... | 17:09 |
| *** gary_perkins has quit IRC | 17:10 | |
| edcragg | thanks for your help, richard_maw | 17:21 |
| *** ssam2 has quit IRC | 17:34 | |
| *** edcragg has quit IRC | 17:47 | |
| *** zoli__ has joined #baserock | 17:59 | |
| *** zoli__ has quit IRC | 18:04 | |
| *** lachlanmackenzie has quit IRC | 18:41 | |
| *** petefoth has joined #baserock | 19:03 | |
| *** paulw has quit IRC | 20:55 | |
| *** radiofree_ is now known as radiofree | 22:45 | |
| *** bfletcher has quit IRC | 23:51 | |
| *** bfletcher has joined #baserock | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!