IRC logs for #baserock for Tuesday, 2016-10-25

*** gtristan has joined #baserock04:06
*** tiagogomes_ has joined #baserock06:42
*** tiagogomes has quit IRC06:42
*** paulw has joined #baserock07:24
*** ctbruce has joined #baserock07:26
*** ctbruce has joined #baserock07:27
*** paulw has quit IRC07:49
*** toscalix__ has joined #baserock07:55
*** toscalix__ is now known as toscalix07:59
*** rdale has joined #baserock08:23
*** locallycompact has joined #baserock08:34
*** jude_ has joined #baserock08:46
gtristanBasement !09:11
paulsher1ood?09:11
paulsher1oodyou are proposing to rename the project?09:12
paulsher1ood:)09:12
gtristanhahaha09:12
gtristanit just hit me while composing mail09:13
gtristan"The project where theoretical ideas are perused to no end without materializing into something concrete and useable" : Basement !09:13
* gtristan ducks incoming chunks of rock09:14
leemingreminiscing about discussions with a certain someone perhaps?09:14
paulsher1oodgtristan: are there some specific ideas that you need to shoe in concrete?09:15
gtristanSigh, I should finish the email first09:15
* paulsher1ood hasn't had enough time for ml responses lately09:15
paulsher1oodiiuc, from some public and some private conversations, it seems that you are wanting to move to a more oo implementation for tooling, either as a derivative of ybd or new base... whereas locallycompact is pushing a more functional programming approach?09:17
* paulsher1ood can only comment that from his personal perspective, fp seems too hard for most programmers...09:18
gtristanpaulsher1ood, I want full extensibility of build pipelines, allowing people to implement their own custom elements - that is the extent of what needs to be OO09:18
gtristanpaulsher1ood, I think fp has it's uses, and you probably find yourself using fp approaches at times, within OO code bases without realizing it09:19
paulsher1oodgtristan: that seems like a worthwhile objective. certainly there are usecases where extensibility would be useful09:20
paulsher1oodeg if someone wants to add testing/analysis at any stage09:20
gtristanpaulsher1ood, check out the new pdf09:21
gtristanpaulsher1ood, dont even bother reading it, jump to page 2 or 3 where pretty pictures will tell you what I'm after09:21
gtristanbut by all means, read it if you have time :D09:21
paulsher1oodgtristan: is this PDF public?09:22
gtristanpaulsher1ood, it's on the private list for now09:23
gtristanattached to email sent out yesterday09:23
paulsher1oodgtristan: any reason we can't invite community members to chime in? or are you trying to avoid the rocks? :)09:23
* tiagogomes_ reminds that morph was extensible through the use of plugins.09:23
* paulsher1ood separately wonders whether a google doc, or wiki page, would be more collaborative than a pdf09:24
gtristanpaulsher1ood, I will reiterate on it once more and do so, but first I wanted to limit comments to those where were going to work on the project09:24
paulsher1oodack09:24
gtristans/where/who09:25
tiagogomes_But regarding the build tool, I would much prefer that existing build tools were improved, instead of repeating the cycle.09:27
paulsher1oodtiagogomes_: i have a slightly different perspective. i'm happy to deprecate all current code, so long as there is a smooth path for users, which leads them to improved experience09:31
tiagogomes_new tool, new bugs09:32
paulsher1oodtrue of course09:33
paulsher1oodtiagogomes_: https://gitlab.com/baserock/ybd/merge_requests/254 - did you take into account multiinstance when you made this change?09:34
paulsher1oodtiagogomes_: the current layout ensures that if two instances try to create the same artifact, precisely one succeeds09:37
gtristantiagogomes_, fwiw I originally agreed with you, and the code will certainly be written by taking all the knowledge gained from ybd09:40
pedroalvareznot morph, morph is bad09:41
paulsher1oodlol09:41
paulsher1oodybd took as much as it could from morph :)09:41
pedroalvarezoh09:41
pedroalvarezybd is bad too!09:41
paulsher1oodall code is bad09:41
pedroalvarezI say it again, this is a very complicated problem09:42
paulsher1oodyup09:42
pedroalvarezand every time we think about it, we put more requirements on top09:42
paulsher1oodactually it's a set of very complicated problems09:42
anahuelamo_tiagogomes_, I tried https://gitlab.com/baserock/ybd/commit/2a7963a3960c07c9e4caa1e7a4c0b69b800bd452 to build the wip for the /usr merge and it worked, is there any reason you didn't try to merge that in ybd?09:43
* gtristan out for food09:45
*** anahuelamo_ is now known as anahuelamo09:45
paulsher1oodpedroalvarez: would you rather we ignore the requirements?09:46
leemingthe best approach is to have the requirements before starting :) If the previous code is written well enough, sure there is time spent re-implementing, but re-factoring should be so much easier09:47
pedroalvareznope, I wasn't suggesting that.09:48
* leeming is jumping in and out of channels, so may have missed something09:48
pedroalvarezactually, maybe we need a set of solutions for these set of very complicated problems?09:49
pedroalvarezs/these/this/09:49
tiagogomes_paulsher1ood I did not. Looking at the code, it could fail as there are now two consecutive runs of shutil.move(). I'll have a look at it.09:49
*** gtristan has quit IRC09:50
tiagogomes_What is multi-instance? Parallel instances of ybd or the use of threads inside ybd?09:50
paulsher1oodybd forks itself, but also n users could run ybd on the same machinre09:54
tiagogomes_why would they want that?09:55
paulsher1oodleeming: in the real world, we learn about requirements as we go. defining requirements up front is often (but not always) counter-productive09:56
tiagogomes_anahuelamo, I didn't create a PR because honestly I don't know how artifact split is supposed to work. So I am not sure if my fix is the right approach.09:56
paulsher1oodtiagogomes_: the forking is for build performance - some builds are parallelizable09:56
paulsher1oodand it would not be unreasonable, on (say) a many-core machine, for there to be more than one user, sharing artifacts09:59
paulsher1oodit would be unreasonable to insist that only one instance of ybd can run at a time, imo09:59
paulsher1oodi think nobody is now clear on how splitting is supposed to work. the morph implementation was never really used in anger iiuc, and the original ybd implementation exposed some corner-cases/requirements10:01
paulsher1oodhence gtristan's mails about this recently10:02
tiagogomes_In case of a multi-tenancy machine, I'd expect the share of artifacts to be mediated with a server. But I understand if you don't want to impose that requirement.10:05
tiagogomes_I thought tristan email was more about the relation of artifact splitting and s-i commands.10:07
paulsher1oodtiagogomes_: you mean 'impose that limitation' i think. but there are also the case where an organisation buys a humungous 'build server'10:09
paulsher1oodtiagogomes_: that the relation between splitting and si is still open to discussion confirms my suggestion that 'nobody is now clear on how splitting is supposed to work', i think?10:10
tiagogomes_paulsher1ood true :)10:11
tiagogomes_The semantics of artifact splitting should have be written somewhere, so that I could determine if I had to fix ybd or definitions such that all the systems included build-essential-minimal.10:13
tiagogomes_On the branch pointed by Ana, I chose to fix ybd as I thought it is how morph behaved.10:14
pedroalvarezthe semantics are written in morph codebase10:15
tiagogomes_pedroalvarez, are you volunteering to map hard to grasp code to documentation ;)10:16
pedroalvarezI implemented the SI commands10:17
pedroalvarezI am the creator10:18
tiagogomes_basegod10:18
*** gtristan has joined #baserock10:40
locallycompactpaulsher1ood, no that's absolutely not the major difference. The major difference is gtristan wants to talk about declaring pipelines and I'm not interesting in declaring pipelines. I declare systems and I talk about systems10:58
* tiagogomes_ doesn't follow11:10
*** locallycompact has quit IRC11:38
*** toscalix has quit IRC11:40
*** toscalix has joined #baserock11:56
*** locallycompact has joined #baserock12:08
tiagogomes_paulsher1ood, can you provide some insight about the following line of code in ybd: `app.config.get("TMPDIR", "/tmp")` ? How does TMPDIR get set?12:31
tiagogomes_That is in sandbox.py12:32
mwilliam1_cttiagogomes_: doesnt that get the values from the config files?12:32
*** mwilliam1_ct is now known as mwilliams_ct12:32
tiagogomes_Reading the code, it looks possible to do it through the config files. Although it is odd to have an uppercase config key.12:36
tiagogomes_But I wonder if the correct instruction would be `os.environ.get("TMPDIR", "/tmp")`12:37
*** gtristan has quit IRC13:02
* paulsher1ood is as confused as tiagogomes_ 13:31
paulsher1oodit seems to have been originally introduced in 66cf5ba013:36
paulsher1oodand then all subsequent fiddlings with the code must have assumed it was correct13:36
paulsher1oodthe actual value that ybd cares about is config['tmp'] which is set in the loop at https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L19313:40
paulsher1ood(for normal tmp file creation)13:41
paulsher1oodtiagogomes_: i think you're right that it should be `os.environ.get("TMPDIR", "/tmp")`13:42
paulsher1oodor just config['tmp'] since os.environ['TMPDIR'] has already been set that earlier in sandbox.py13:43
tiagogomes_I think setting to config['tmp'] would be problematic.13:47
paulsher1oodwell, as i say, os.environ['TMPDIR'] has already been set to that13:48
tiagogomes_The sandboxes are created in  config['tmp'], and inside the sandoxes only config['tmp']/sandboxfoo/chunk.inst,  config['tmp']/sandboxfoo/chunk.build and config['tmp']/sandboxfoo/tmp should be writable for building chunks13:48
tiagogomes_If you use config['tmp'] there, every directory in the sandbox dir will be writable13:49
paulsher1oodin other news 'Downloads:100204'13:49
paulsher1oodtiagogomes_: in that case, can we just drop the line, and have writable_paths = [dn['checkout'], dn['install']] ?13:50
tiagogomes_No. When building a chunk, they need a writable /tmp directory.13:51
paulsher1oodok. so this has been working, but only because most host systems have a writeable /tmp13:54
paulsher1oodif someone were to set TMPDIR: as a conf variable that woudl work also, for a non-standard system.13:55
paulsher1oodso oddly enough, i think the current code is reasonable?13:55
paulsher1oodbut maybe there needs to be # TMPDIR entry in ybd.conf13:56
tiagogomes_tbh I think it is reasonable to assume the presence of /tmp, either a directory, mounting point or symlink to the canonical place (e.g. /run/tmp)14:03
tiagogomes_Or maybe not, if the build tool is to run on non-Linux systems.14:04
pedroalvarezstop thinking about building in linux14:07
pedroalvarezin windows, I mean14:07
tiagogomes_I was about to say it would be easy with the Microsoft partnership with Canonical, but I doubt the they can translate the kernel syscalls needed for bubblewrap14:09
*** toscalix has quit IRC14:10
*** toscalix has joined #baserock14:22
tiagogomes_I wonder if using ccache on the build essential components was tested.14:53
*** ctbruce has quit IRC14:54
*** ctbruce has joined #baserock14:55
tiagogomes_Looks like it will not work, as it is not setting CCACHE_BASEDIR. But morph also doesn't set it15:11
paulsher1oodtiagogomes_: iirc, gtristan had ccache working in his aboriginal variant15:18
*** ctbruce has quit IRC15:59
*** franred has quit IRC16:41
*** jude_ has quit IRC16:56
*** locallycompact has quit IRC17:29
*** toscalix has quit IRC17:38
*** jude_ has joined #baserock18:04
*** jude_ has quit IRC18:08
*** jude_ has joined #baserock18:08
*** gtristan has joined #baserock18:47
*** gtristan has quit IRC19:16
*** jude_ has quit IRC19:44
*** jude_ has joined #baserock19:52
*** jude_ has quit IRC22:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!