IRC logs for #baserock for Thursday, 2015-09-10

*** zoli__ has joined #baserock01:33
*** zoli__ has quit IRC03:03
*** zoli__ has joined #baserock03:04
*** zoli__ has quit IRC04:37
*** zoli__ has joined #baserock04:38
*** zoli__ has quit IRC04:52
*** zoli__ has joined #baserock04:52
*** zoli__ has quit IRC05:33
*** paulw has joined #baserock06:16
*** Walkerdine has quit IRC06:51
*** zoli__ has joined #baserock07:22
*** zoli__ has quit IRC07:36
*** zoli__ has joined #baserock07:36
*** toscalix has joined #baserock07:55
*** bashrc has joined #baserock08:00
*** zoli__ has quit IRC08:11
*** rdale has joined #baserock08:21
*** mariaderidder has joined #baserock08:24
*** radiofree has quit IRC08:27
*** paulw has quit IRC08:29
*** radiofree has joined #baserock08:29
*** paulw has joined #baserock08:30
*** toscalix has quit IRC08:40
*** toscalix__ has joined #baserock08:52
*** radiofree has quit IRC08:54
*** radiofree has joined #baserock08:56
*** zoli__ has joined #baserock08:56
*** zoli__ has quit IRC09:01
*** lmackenzie75 has joined #baserock09:16
*** cyndis has quit IRC10:30
*** paulw has quit IRC11:03
*** zoli__ has joined #baserock11:03
*** zoli__ has quit IRC11:08
*** zoli__ has joined #baserock11:22
*** zoli___ has joined #baserock11:28
*** zoli__ has quit IRC11:31
*** zoli___ has quit IRC11:38
*** zoli__ has joined #baserock11:47
paulsherwoodyay!11:49
* rjek wonders what paulsherwood is celebrating.11:50
pedroalvarez<Walkerdine> Now I'm finally building an image for real this time11:50
pedroalvarezbut he left11:50
rjekaha11:50
*** paulw has joined #baserock12:35
*** toscalix__ is now known as toscalix12:36
*** cyndis has joined #baserock12:37
*** gary_perkins has quit IRC12:43
*** gary_perkins_ has joined #baserock12:43
*** gary_perkins_ has quit IRC12:55
*** gary_perkins has joined #baserock12:56
*** gary_perkins has quit IRC12:59
*** cyndis has quit IRC13:00
*** tiagogomes_ has quit IRC13:01
*** gary_perkins has joined #baserock13:02
*** zoli__ has quit IRC13:03
*** gary_perkins has quit IRC13:06
*** gary_perkins has joined #baserock13:08
*** gary_perkins has quit IRC13:11
*** gary_perkins has joined #baserock13:14
*** tiagogomes_ has joined #baserock13:15
*** gary_perkins has quit IRC13:17
*** paulw has quit IRC13:30
*** paulw has joined #baserock13:31
*** cyndis has joined #baserock13:32
*** toscalix has quit IRC13:35
*** toscalix__ has joined #baserock13:35
*** Walkerdine has joined #baserock13:41
*** paulw has quit IRC13:47
*** paulw has joined #baserock13:48
*** Walkerdine__ has joined #baserock13:49
*** Walkerdine has quit IRC13:52
*** zoli__ has joined #baserock14:27
*** zoli___ has joined #baserock14:33
*** zoli__ has quit IRC14:33
*** Walkerdine__ has quit IRC14:52
*** Walkerdine__ has joined #baserock14:57
*** zoli__ has joined #baserock14:59
*** zoli___ has quit IRC14:59
*** tiagogomes_ has quit IRC15:05
*** fay_ has quit IRC15:05
*** fay_ has joined #baserock15:06
*** gary_perkins has joined #baserock15:10
*** Walkerdine has joined #baserock15:10
WalkerdineYeah finally. I decided to just use a usb hub to attach the ssd15:11
radiofreethat's not exactly going to be very performant15:12
WalkerdineWhy not?15:15
rjekUSB's slow compared to SATA15:16
rjek480Mbps, polled IO verses 3Gbps interrupt-driven15:16
WalkerdineDoes it matter if I'm using usb 3.0 or no15:16
*** tiagogomes_ has joined #baserock15:17
radiofreei don't think usb 3 works with that kernel (on a jetson)15:18
WalkerdineOh well I could never get it working by just plugging it into the sata port15:20
rjekWell, slow is better than not working15:21
*** tiagogomes_ has quit IRC15:21
*** zoli__ has quit IRC15:22
*** zoli___ has joined #baserock15:22
radiofreeWalkerdine: did you power the drive?15:24
radiofreelike so https://richardstechnotes.files.wordpress.com/2014/07/tk1_disk.jpg15:25
WalkerdineYeah I powered the drive. I spent over a week trying to get that drive to work on the sata port15:26
WalkerdineI've tried with 2 different SSDs 2 different Jetsons a couple different versions of baserock15:26
WalkerdineI reinstalled the original software on the jetson and the drives worked fine15:27
rjekWhen I googled for this, there appears to be some manufacture variance related to the SATA port on those boards15:31
WalkerdineIt connected once but I got errors when I tried to mount it. I rebooted and I never got it working again15:31
wdutchpaulsherwood: any idea what's going wrong here? http://52.19.1.31:8010/builders/trigger_builders/builds/20/steps/shell_1/logs/stdio (you have to scroll to the bottom for the ybd output)16:04
paulsherwoodwdutch: is its trove working properly?16:08
paulsherwoodCloning into bare repository '/src/gits/git___cu010_trove_codethink_com_delta_binutils_redhat.tmp'...16:08
paulsherwoodconnected.16:08
paulsherwoodHTTP request sent, awaiting response... 404 Not Found16:08
paulsherwood2015-09-10 15:58:10 ERROR 404: Not Found.16:08
paulsherwoodtar: git___cu010_trove_codethink_com_delta_binutils_redhat.tar: Cannot open: No such file or directory16:08
paulsherwoodtar: Error is not recoverable: exiting now16:08
paulsherwood(for example)16:08
wdutchit's a bit slow16:09
Kinnisonthat's just a name mismatch16:09
Kinnisonpaulsherwood: purely a name mismatch, just slows down fetches16:10
paulsherwoodi think i've not tested the parallelisation with no gits16:10
* Kinnison could correct that with a bit of poking but it only slows initial clone16:10
paulsherwoodfor your use-case are you expecting to download all gits every time or could they be pre-populated?16:11
KinnisonWe're expecting the git cache to be populated in the general case16:11
KinnisonIt'd hurt immensely to download every time16:11
KinnisonIMO16:11
paulsherwoodso in this case the fastest hack would be to remove any .tmp directories in base/gits, and set git-server: to something that doesn't work16:15
paulsherwoodthat will force all repos to be cloned at the start, rather than during the builds16:15
paulsherwoodand thereafter things should work tidily16:15
Kinnisonpaulsherwood: that sounds icky16:16
paulsherwoodi can improve it, certainly. and will do so16:16
Kinnisonso you think that not having all the gits up-front will cause issues?16:16
richard_mawI'm stumped as to why the build is failing, AFAICT the last thing attempted was a link command, which should not have anything to do with its build environment, and we've not made any changes beneath it from firehose16:16
Kinnison/src/tmp/tmpv_m1YT/stage1-gcc.build/missing: line 81: makeinfo: command not found16:17
paulsherwoodKinnison: i'm unsure. as i said, i've not tested the parallel case without gits16:17
KinnisonWARNING: 'makeinfo' is missing on your system.16:17
Kinnison         You should only need it if you modified a '.texi' file, or16:17
Kinnison         any other file indirectly affecting the aspect of the manual.16:17
Kinnison         You might want to install the Texinfo package:16:17
Kinnison         <http://www.gnu.org/software/texinfo/>16:17
Kinnison         The spurious makeinfo call might also be the consequence of16:17
Kinnison         using a buggy 'make' (AIX, DU, IRIX), in which case you might16:17
Kinnison         want to install GNU make:16:17
Kinnison         <http://www.gnu.org/software/make/>16:17
Kinnisonmake[4]: *** [../../../gmp/doc/gmp.info] Error 12716:17
KinnisonI think that's more likely the issue16:17
Kinnisonwhich's odd16:17
richard_mawit's a warning an the associated message says "you'll be fine as long as you don't modify .texi files"16:18
Kinnisonand then proceeds to error 12716:18
richard_mawhmm16:18
KinnisonI think tis a case of parallel make confusing the error message16:18
* Kinnison installs texinfo onto the AWS box16:19
Kinnisonrichard_maw: can you try causing builders to trigger again?16:19
wdutchI can16:19
Kinnisonwdutch: ta16:20
wdutchoff it goes16:20
* Kinnison wonders if we can persuade it to run as though under a pty16:21
Kinnisonso we get line-buffered logging rather than 4k or whatever it is16:21
*** paulw has quit IRC16:22
paulsherwoodis it working, wdutch ?16:26
wdutchit died :(16:26
wdutchhttp://52.19.1.31:8010/builders/trigger_builders/builds/23/steps/shell_1/logs/stdio16:26
KinnisonENOSPC16:27
KinnisonI wonder what's using up all the room on that 100G partition I set aside for it16:27
* Kinnison decides to blow away all the old artifacts etc16:27
paulsherwoodKinnison: blow away /src/tmp/* first16:27
Kinnisonpaulsherwood: I'm clearing it down in general16:27
paulsherwoodok16:28
Kinnisonpaulsherwood: since we *know* it's going to build from scratch (since it's currently dying on stage1-gcc)16:28
* Kinnison would rather clear it down16:28
* Kinnison is wondering what the right approach to clearing out this stuff will be longer-term16:28
paulsherwoodit leaves tmp in case folks need to explore the debris.16:28
* paulsherwood is wondering too16:29
Kinnisonrichard_maw: If you have access to the relevant scripts, could you have it clear down /src/tmp before running builds?16:29
richard_mawerm, maybe16:29
* Kinnison is currently clearing out /src/tmp16:29
paulsherwoodit would be easy enough to blat all tmp on start, assuming no other ybds are active16:29
KinnisonWe're limiting ourselves to one top-level builder per slave16:30
*** mariaderidder has quit IRC16:30
paulsherwoodack16:30
paulsherwoodso that would be the best bet then, have it blat tmp first of all16:30
* Kinnison nods16:30
paulsherwoodi can add that with a config option16:30
* Kinnison also clears out all the cached gits from the wrong trove :-)16:30
* Kinnison removes an old backup of /src too16:31
Kinnisonsuch cruft accumulated in one week16:31
paulsherwoodwhat's the best way to check if a program foo is running, in a python program?16:31
Kinnisonthere, 91G spare16:31
Kinnisonrichard_maw: If you can cause a builder run again, we can see if it gets further16:31
Kinnisonpaulsherwood: Hmm16:32
Kinnisonpaulsherwood: well you could use pgrep16:32
richard_mawpaulsherwood: that's a deceptively difficult question to solve correctly16:32
jmacspaulsherwood: There are libraries, but nothing built in that I know of. Either iterating over /proc with os.listdir or calling pgrep using subprocess16:32
* paulsherwood finds psutil16:33
richard_mawthat will tell you if a process with the name you think your program should have is running16:33
paulsherwoodack16:33
Kinnisonrichard_maw: now the ybd stuff seems to be expecting .co.uk16:33
paulsherwoodwhich is not necessarily safe :)16:33
Kinnisonrichard_maw: buh16:33
Kinnisonrichard_maw: where is the ybd config for the builder, do you  know?16:34
richard_mawKinnison: in pdar's homedir I think16:34
paulsherwoodKinnison: in order of priority . ybd/ ybd/config16:34
Kinnisonpdar: what is the trove configured to?16:35
paulsherwood(and only ybd/config actually contains a ybd.conf file by default)16:35
Kinnisonpdar: it should be cu010-trove.codethink.com16:35
*** zoli___ has quit IRC16:35
*** zoli__ has joined #baserock16:35
pdaroh, i changed it to .co.uk at about 17:1516:36
pdarI think i misunderstood what was going on16:36
Kinnisonpdar: doh16:36
Kinnisonpdar: No, the *trove* was misconfigured, not ybd :-)16:36
pdarahh I'll change it back then16:37
* Kinnison was perhaps not clear enough16:37
*** Walkerdine has quit IRC16:37
paulsherwoodnote it's probably more predictable to create a new ybd/ybd.conf and leave ybd/config/ybd.conf as default (so git pulls won't get worried)16:37
richard_mawpaulsherwood: pretty much the only reliable options on linux are to have been the one to spawn it, at which point you can look up its status PID safely since it can't re-use it until you reap it, or for you to stuff the PID into a cgroup that nothing else is allowed to change, or to defer this logic to something else. This is one of the reasons why systemd exists.16:38
paulsherwoodrichard_maw: ack. so i'd better not go down this rabbithole16:38
richard_mawI'm lacking context, what is the problem you want to solve?16:38
paulsherwoodworking out whether it's safe to blat tmp dir on starting ybd16:39
Kinnisonpaulsherwood: that's what lockfiles are for16:39
Kinnisonpaulsherwood: and there's lots and lots of good research and implementations for lockfiles16:39
richard_mawor take a lock on the directory itself16:39
* paulsherwood will have to google, then16:40
paulsherwoodall of my experience of lock files in former lives was cleaning up messes when locks got left around by accident16:41
richard_mawpaulsherwood: that's why you use posix lock file descriptors, rather than lock files16:41
richard_mawpaulsherwood: the lifetime of the lock is maximally bound by the process, so when your process dies, so does the lock16:42
richard_mawpaulsherwood: and if you crash *hard* you don't leave locks around16:42
richard_mawI think Tiago did locking for tempdirs in morph16:42
* richard_maw saw patches16:42
Kinnisonwell stage1-gcc built that time16:43
*** Walkerdine__ has quit IRC16:43
Kinnisonso I think we stand a chance \o/16:43
paulsherwoodrichard_maw: was this recent?16:45
richard_mawfairly16:45
richard_mawpaulsherwood: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/morphlib/stagingarea.py#n63 for making sure the staging area can't be removed by gc while in use, and http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/morphlib/plugins/gc_plugin.py#n94 for removing it16:45
paulsherwoodrichard_maw: tvm!16:45
richard_mawpaulsherwood: you don't have to go to staging area granularity, just open /src/tmp instead16:45
paulsherwoodagreed16:46
* Kinnison thinks we care enough about the result that he'll leave this builder on overnight16:48
richard_mawpaulsherwood: you need to know when you want a shared lock and when you want an exclusive lock, and when you want to do a blocking lock, or a non-blocking lock16:48
richard_mawwhen using it you'll want a shared lock, so multiple YBD instances can use it16:49
richard_mawand when removing it you want to take an exclusive lock16:49
paulsherwoodack16:49
richard_mawso only the thing removing it can use it16:49
* richard_maw departs16:50
* paulsherwood too16:50
richard_mawthough locks are interesting so I may be around to talk about them later16:50
* paulsherwood too16:50
*** zoli__ has quit IRC16:55
*** bashrc has quit IRC17:02
richard_mawOk, locking! It's better to be explicit and have processes take a lock, than leave it implied that "ybd.py" processes lock removal of the tempdir.17:39
*** Walkerdine has joined #baserock17:40
richard_mawYou could make a lock file, but that causes interesting problems if you might have multiple processes needing the directory17:40
richard_mawand problems of processes needing to clean up the lock file correctly after they're done17:41
richard_mawand with abnormal termination you can't get rid of them17:41
richard_mawso you can use flock() (fcntl.flock() in python) on an open file descriptor to take an advisory lock on a file17:42
richard_mawthis is advisory locking, since nothing prevents you going behind the program's back and removing the directory anyway17:42
richard_mawyou need programs that may decide to clean up the directory to ask if they are allowed to17:43
richard_maw(there is mandatory locking in Linux, but it's a bad idea)17:43
richard_maw(one reason why Windows updates are such a pain is that mandatory locking is the norm, so to apply updates you have to shut everything down)17:44
richard_mawby making locks file descriptor based, you can guarantee that it is eventually released17:44
richard_mawas if your process dies, you release the lock17:44
richard_mawin normal cases you would take an exclusive lock when writing and a shared lock when reading (hence why some literature calls them read locks and write locks)17:45
richard_mawmorph makes the staging areas gc safe by taking an exclusive lock when creating the staging area, leaving the file descriptor open while it is in use, then removing it and closing the file descriptor when it is no longer needed17:47
richard_mawthis relies on the linux behaviour of letting you remove a file when you have it open17:47
richard_mawwhen morph wants to gc staging areas it can know that one is in use by attempting to take an exclusive, non-blocking lock17:48
richard_mawat which point it can move on to the next staging area to remove17:48
richard_mawif the lock fails, or it can proceed to clean up the directory17:49
richard_mawif you ran another gc during this time they could not confuse each other by both trying to clean up the staging area17:49
richard_mawsince you don't care about this fine grained locking, you would, on start, attempt to open the directory for reading, and if this fails, skip trying to remove it17:51
richard_mawthen you'd take an exclusive lock to clean up its contents17:51
richard_mawdo this non-blocking if you want to make ybd report an error, or carry on without trying to remove it17:52
richard_mawif taking the exclusive lock fails you should take a read lock, so nothing else tries to remove it while you are using it17:52
richard_mawif you took the exclusive lock, then you can remove all its contents, and convert your exclusive lock to a shared lock17:53
richard_mawyou should also take a read lock when you first create the directory17:56
richard_mawthe point of all this locking is that when you have a process cleaning it up, you have an exclusive lock, at which point you can take a blocking read lock, and wait for it to be tidied up17:57
*** lmackenzie75 has quit IRC17:57
richard_mawor it can be in use, at which point you can decide whether you want to wait for everything to finish so you can clean it up (blocking exclusive lock), proceed without cleaning it up, or terminate (non-blocking exclusive lock)17:58
richard_mawI'm willing to do code review for your locking code, since I was the one to suggest it to tiago and I reviewed his code for morph18:00
paulsherwoodok thanks - i'll get to it later today18:17
*** zoli__ has joined #baserock18:45
*** zoli___ has joined #baserock19:00
*** zoli__ has quit IRC19:00
*** zoli___ has quit IRC20:07
*** zoli__ has joined #baserock20:07
*** Walkerdine has quit IRC20:09
*** toscalix__ has quit IRC20:41
*** toscalix_ has joined #baserock20:41
*** Walkerdine has joined #baserock21:09
*** zoli__ has quit IRC21:23
*** Walkerdine has quit IRC21:35
*** Walkerdine has joined #baserock21:41
*** zoli__ has joined #baserock22:04
*** Walkerdine has quit IRC22:45
paulsherwoodrichard_maw: https://github.com/devcurmudgeon/ybd/compare/clean-tmp-with-locks seems to work22:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!