IRC logs for #baserock for Sunday, 2015-06-21

*** jjardon has joined #baserock00:01
*** Stanto has quit IRC00:04
*** Stanto has joined #baserock00:07
*** petefoth has joined #baserock00:07
*** kejiahu has joined #baserock00:28
*** tlsa has joined #baserock00:30
*** jjardon_ has joined #baserock00:33
*** pacon has quit IRC00:33
*** jjardon has quit IRC00:36
*** tlsa_ has quit IRC00:36
*** sam_ has quit IRC00:36
*** kejiahu_ has quit IRC00:36
*** sambishop has joined #baserock00:36
*** jmacs_ has joined #baserock00:45
*** jjardon_ is now known as jjardon00:46
*** jmacs has quit IRC00:55
*** Guest19028 has quit IRC01:18
*** Guest19028 has joined #baserock01:18
*** Guest19028 is now known as persia_01:18
*** persia__ has quit IRC01:19
*** persia__ has joined #baserock01:19
*** persia__ is now known as persia01:19
*** edcragg has quit IRC01:24
jeak_?07:17
*** edcragg has joined #baserock08:20
*** Kinnison_ is now known as Kinnison10:03
*** pacon has joined #baserock10:47
*** pacon has quit IRC11:49
*** pacon has joined #baserock11:50
*** pacon has quit IRC12:34
*** jmacs has joined #baserock13:01
*** jmacs_ has quit IRC13:08
*** petefotheringham has quit IRC13:20
*** petefotheringham has joined #baserock13:21
*** petefotheringham has quit IRC13:49
*** petefotheringham has joined #baserock13:49
paulsher1oodhi jeak_ ?15:12
paulsher1ooddoes anyone know if there's an established pattern for cleaning up tmp files for re-entrant programs that may fail leaving tmp files behind?15:36
paulsher1oodeg if i were able to run n instances of ybd on one machine, and some fail, leaving their staging areas... next time i run a ybd i'd like it to clean up dead ones, but not the tmp files being used by live instances of ybd15:38
KinnisonOne option is to inject a lockfile into a tempdir15:45
Kinnisonif you can't lock it, you can't use/clean it up15:45
Kinnisontypically, tmp files are not cleaned up if a program dies, so that investigation can happen15:46
juergbipaulsher1ood: depending on how they're used, you might be able to get away with unlinking them right after creation and just keeping the file descriptors open15:51
juergbi(or O_TMPFILE if you can depend on recent kernel)15:51
juergbimay not be an option when the temp files are to be used by external processes where you can't just pass the fd15:51
juergbiusing systemd and tmpfiles.d you can configure automatic cleanup after N hours or days of last modification. default for /tmp is 10d15:54
persiaCan't depend on recent kernel, if we're trying to make it work for arbitrary unices (enterprise linux distros, OS X, etc.)16:32
juergbiunlinking right after file creation is not much worse in practice. in the off chance that you hit the race condition, you'll leave one empty temp file around16:40
juergbii suspect the bigger limitation is interaction with external processes16:41
rjekAlso if the data to be written to them is sensitive, make sure you fstat the open fd and count the number of hardlinks it has and confirm it is zero before writing to it.16:42
juergbirjek: isn't it typically sufficient to use mkstemp?16:43
rjekI assumed that wasn't an option for some reason due to all the discussion about doing it :)16:44
juergbiah, i thought this was just about cleanup, which mkstemp doesn't directly solve16:44
rjekIndeed not16:45
rjekBut writing your own mkstemp that does unlink (opening O_EXCL, unlink, fstat to confirm zero links) probably would.16:46
juergbii would simply used unlink after mkstemp, if i can't use O_TMPFILE16:46
juergbi*use16:46
persiaThe issue there is that someone else with filesystem accesss could potentially link the file between those two calls.16:48
juergbishould only be possible for process of the same user, right? and protecting against that is somewhat futile with regular unix user-based access to processes16:49
persiaIf you are unconcerned with the ability to capture the data, that's fine.  If you are concerned (e.g. you are storing plaintext of something typically enciphered), you need to do the zero check after unlinking to confirm there is no consistent log.16:49
persiaWhether it needs to be the same user depends on umask.  002 is surprisingly popular.16:50
juergbishouldn't mkstemp's 0600 permission help here16:50
juergbialso, hardlink only works on the same filesystem. to avoid persistence, you will therefore create a tempfile on tmpfs and thus, the hardlink itself can't cause persistent storage16:51
juergbiit might still be accessed by other processes running as the same user but that's true with or without the zero link check16:53
persiaThe permission value is libc dependent.16:54
persiaBut mkostemp(foo, bar, O_EXCL) might be a reduction of steps in the formula above.16:55
juergbipersia: well, i think we can ignore ancient glibc versions (2.06 and earlier). mac also claims to handle it correctly and recent POSIX mandates it17:10
persiaAh, I wasn't aware of the recent POSIX mandate.17:11
juergbisince 200817:12
persiamusl seems to have used 600 since mkstemp and mkostemp were first supported.17:12
persiaFreeBSD's libc also.17:13
juergbithe issue in glibc was in 199717:14
persiaMaybe I'm being overparanoid, or showing my age when I suggest we can't trust libc impementation for 0600 :)17:14
juergbithere could well be some broken libc still out there, however, i wouldn't work around such OS bugs in my software17:15
persiaI'm only concerned about recent systems running.  I suspect it is safe.17:16
persiaI can't confirm uclibc is safe, but in practice, uclibc is an unlikely libc for a development environment.17:16
persiabionic also does the right thing, completing my list of platforms to check.17:18
juergbiuclibc is safe as well17:18
juergbi    return __gen_tempname (template, __GT_FILE, 0, S_IRUSR | S_IWUSR);17:18
persiaOh, cool.  I was just looking for the string 0600, rather than doing semantic analysis.17:19
persiaI don't find any mention of permissions for _mktemp_s(), but there are other things that would need changing to run on a non-POSIX OS, and the documentation of the PCL is limited, at best.17:22
juergbiyes, no idea about Windows in this regard out of the top of my head17:25
persiaYeah, it wasn't really on my list of potential platforms, except I got carried away :)17:25
*** robtaylo1 is now known as robtaylor18:44
paulsher1oodKinnison: my plan is to leave the files for invesigation. but then on re-running ybd, it should clean first22:03
* paulsher1ood reads the backscroll, and sadly fails to understand most of if22:06
*** wdutch has quit IRC22:17
*** wdutch has joined #baserock22:30
*** paulw has quit IRC23:43
*** paulw has joined #baserock23:43

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!