IRC logs for #baserock for Tuesday, 2016-05-10

*** richard_maw has quit IRC02:45
*** richard_maw has joined #baserock02:45
*** ChanServ sets mode: +v richard_maw02:45
*** richard_maw has quit IRC02:50
*** richard_maw has joined #baserock02:51
*** ChanServ sets mode: +v richard_maw02:51
*** locallycompact has joined #baserock06:25
*** ctbruce has joined #baserock07:31
*** mwilliams_ct has joined #baserock07:33
*** toscalix has joined #baserock07:42
*** CTtpollard has quit IRC08:27
*** franred has quit IRC08:33
*** tiagogomes has joined #baserock08:41
*** jonathanmaw has joined #baserock08:46
anahuelamopaulsherwood, it works for me too. I can build the system, deploy it and boot the image. But depends on which system I use the files in /usr/src/MPC remain in the system or not08:52
*** CTtpollard has joined #baserock08:53
*** toscalix has quit IRC08:56
*** toscalix_ has joined #baserock08:56
paulsherwoodanahuelamo: both systems worked for me08:57
paulsherwoodie both systems don't have  /usr/src/MPC08:57
anahuelamopaulsherwood, Ok, I'll test again, I'm not sure what is happening then08:58
*** ssam2 has joined #baserock09:00
*** ChanServ sets mode: +v ssam209:00
*** CTtpollard has quit IRC09:05
*** CTtpollard has joined #baserock09:05
*** edcragg has joined #baserock09:10
paulsherwoodssam2: i'm occasionally tripping over http://paste.baserock.org/wozuzomusu09:27
paulsherwoodany idea what that traceback signifies, please?09:27
ssam2possibly that the output of the command that ran contains non-ascii characters09:28
ssam2is this with python 2 or python 3 ?09:28
paulsherwoodpython 209:28
ssam2right09:29
ssam2probably a bug in sandboxlib code needs09:29
ssam2-code needs09:29
ssam2i don't have time to dig in further right now, sorry09:29
paulsherwoodwhat do you mean by needs?09:30
paulsherwoodthe command is 'if which depmod; then (cd /lib/modules && for version in *; do depmod -a "$version"; done) fi ' which seems to be ascii to me :)09:30
ssam2i was going to write that it needs to handle non-ascii characters better09:30
ssam2no, I mean the output of the command09:30
paulsherwoodah, ok09:30
ssam2that's a guess, but it seems most likely09:30
richard_mawpython in general needs to handle non-ascii characters better, and python3 gets it wrong for file paths09:31
* richard_maw grumbles about string types09:31
paulsherwoodhence locallycompact is thinkinng of starting in rust. trouble is rust is mostly incomprehensible :-)09:31
ssam2hmm, maybe it's something else09:32
ssam2http://python.6.x6.nabble.com/Can-t-use-subprocess-Popen-after-os-chroot-why-td1417932.html09:32
locallycompactpaulsherwood, what part of rust are you finding troubling? :)09:32
ssam2is there a way to get the full traceback, rather than just the exception ?09:32
ssam2hmm, that is the full traceback09:32
edcraggpaulsherwood: incomprehensible if you don't understand it, clearly ;)09:33
richard_mawpaulsherwood: If you're going to rewrite everything from scratch, then Rust might be good for the bits you need to reinvent, but I would strongly recommend working out exactly what functionality the tool needs, and satisfy that with as many existing tools as are feasible.09:33
ssam2ok, seems sandboxlib itself is eating the traceback09:34
locallycompactrichard_maw, that's sort of missing the point of this exercise, I'll be happy to explain after I head into work.09:34
richard_mawpaulsherwood: My take from both morph and ybd is that the problem is inherently complex, and the best way to manage that complexity is to break it down, so while there is more total complexity, the abstractions mean you can avoid needing to know the details.09:34
jmacsedcragg: That's pretty much the definition of incomprehensible09:34
* richard_maw raises eyebrow at locallycompact's comment09:34
rjekRust fails my "vaguely understandable at a glance to somebody who has programmed in other languages" test09:35
locallycompactc fails that test for me09:35
locallycompactrust is kind of just obvious09:35
locallycompactto me09:35
locallycompactanyway, back in a bit09:36
jmacsAny language you've already programmed in looks obvious09:36
*** toscalix_ is now known as toscalix09:36
paulsherwoodonly while you remember how to program in it :)09:37
paulsherwoodrichard_maw: i'm not going to write everything from scratch... i didn't even do that for ybd :)09:38
paulsherwoodi agree this is complex, though09:39
*** locallycompact has quit IRC09:40
richard_mawpaulsherwood: a proposed Rust rewrite would require writing much more code, YBD could pinch code from morph to handle bits.09:40
paulsherwoodybd already did pinch code from morph09:41
richard_mawI know, I just said it did.09:41
paulsherwoodah, sorry, i misunderstood09:41
* richard_maw also wasn't unambiguous about his meaning, so appologieses there09:41
paulsherwoodi originally suggested improving on aboriginal, or xdg-app-builder... locallycompact, being young, bright eyed and of purist ideals, suggested starting in rust instead :)09:42
* paulsherwood has learned that sometimes the young can achieve miracles...09:43
rdalehas rust got a good web framework?09:44
mwilliams_ctyou once told me that the job of the old is to demand the impossible of the young, and the job of the young is to do it because they don't realise it's impossible09:44
paulsherwoodmwilliams_ct: yup. well remembered, young man :-)09:45
mwilliams_ctpaulsherwood: dementia won't set in for a while yet :)09:45
paulsherwoodmaybe not for you :-)09:45
rjekDementia in whom?09:45
mwilliams_ctit's a good line though, I've been careful to ignore everything older people have told me ever since09:45
rdalecheers09:45
mwilliams_ct:)09:46
rjektbh, most people are older than mwilliams_ct09:47
rdalei prefer to collaborate on things that no one single person could pull off by themselves - including the strengths of the older and the younger people09:48
mwilliams_ctrdale: to be quite clear, if I ever ignored anything you told me I'd be a fool. I was very much kidding!09:48
paulsherwoodback on topic...09:49
rdaleno problem - knowing who to ignore is always an important skill though09:49
paulsherwoodwhat provides 'which' in baserock systems, please?09:50
edcraggbusybox?09:50
richard_mawit used to be busybox09:50
ssam2try this in a br system: grep 'which' /baserock/*.meta09:51
paulsherwoodok, thanks. i'm trying to regularise ybd's system artifacts so it includes -runtime and -devel09:51
paulsherwoodwhen i do that, it seems 'which' has disappeared09:51
*** locallycompact has joined #baserock10:11
paulsherwoodah, so busybox is specified in build-essential-minimal...10:26
locallycompactso the stage0 rust builds and installs fine, then I try to use it to compile the next one and I get this10:27
paulsherwoodrichard_maw: so the idea is that if we want to create (say) a -runtime system, we'd add *-minimal + *-runtime ?10:27
locallycompact /tools/bin/rustc: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /tools/bin/../lib/librustc_llvm-db5a760f.so)10:27
* paulsherwood also wonders if a sensible *default* behaviour for ybd would be to create -runtime systems (smaller) rather than -devel systems10:29
jjardonlocallycompact: does rust depend on llvm ?10:29
richard_mawjjardon: yes10:29
richard_mawpaulsherwood: yes10:30
paulsherwoodok great10:30
paulsherwoodrichard_maw: i think i finally understand splitting! :-)10:30
jjardonlocallycompact: only checking but, are you including the strata/llvm-common dep in your rust stratum?10:30
jjardonpaulsherwood: havinf -runtime and -devel system is a _major_ achievement, congrats10:31
locallycompactjjardon, no, I thought it was using the submodule10:31
locallycompactsince stage0 works10:32
* richard_maw wishes that had happened before paulsherwood declared it to be too complicated and unnecessary10:32
*** richard_maw has left #baserock10:32
* paulsherwood notices richard_maw seems to have left the building10:34
paulsherwoodfwiw i 'declared' it to be too complicated. i don't remember ever saying it was unnecessary10:34
*** cosm has joined #baserock10:34
paulsherwoodjjardon: i've pushed it to master. let's hope i haven't broken anything10:37
* jjardon starts a gazilion of builds10:40
paulsherwoodjjardon: you'll get -runtime by default now10:41
jjardoncool10:41
paulsherwoodhah! ybd ci actually works... i changed the system cache-keys and broke the cache-key test :-)11:22
paulsherwoodcan anyone give me a clue on http://paste.baserock.org/tixarigere please?11:33
*** ssam2 has quit IRC11:46
locallycompactjjardon, that didn't help any11:53
paulsherwoodwin 3511:57
paulsherwoodbah11:57
*** ssam2 has joined #baserock11:58
*** ChanServ sets mode: +v ssam211:58
rdalepaulsherwood: i'm not undstanding the discussion about -runtime and -devel - they are default splitting rules and ybd will already have them if you want to specify them in a system12:09
paulsherwoodrdale: it wasn't working - afaict there was no logic to apply the default rules for systems, so it only worked for systems (eg minimal) where the splits were specified12:12
paulsherwoodmaybe i broke it... but you did say you hadn't tested on anything other than minimal :)12:13
rdaleyes, you need to specify 'foo-runtime' or 'foo-devel' in a system otherwise you get everything - that is what it is supposed to do12:13
paulsherwoodack - so i've now made it that ybd does runtime systems by default12:14
paulsherwoodand that can be changed without editing the definitions themselves12:14
SotKI thought we wanted things to be fully deterministic based just on the definitions?12:14
rdalei don't think that is a fix, it is an enhancement - it worked as defined by the behaviour of morph doing the same thing12:15
paulsherwoodSotK: good point. but at the moment there's nothing in definitions saying which default rules apply, iiuc?12:16
paulsherwoodrdale: you may be right, but afaict morph's base-system defaults to stripping, and ybd's didn't12:17
paulsherwoods/stripping/splitting/12:17
paulsherwoodi got lost in a maze on this... the splitting code was using foo['strata'] info which i believed i had expunged from ybd from the start12:18
SotKI've always assumed that morph's implementation is the de-facto default behaviour of definitions (at least for behaviour from before ybd existed)12:20
rdalethe morph splitting didn't work properly when i tried it, and so i was basing the ybd spec on the intention behind the code - but i never noticed anyone say that if no splits were specified you just got '-runtime'12:20
SotKthat may not be written down anywhere though12:20
paulsherwoodack12:20
paulsherwoodSotK: my compromise for the moment is that -runtime systems generate a different cache-key from -devel systems12:21
rdalehow do you define -runtime system and -devel systems without specifying any splits in the system definition?12:22
paulsherwoodby specifying them as a default-splits config12:22
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/config/ybd.conf#L7212:23
rdaleis that a new ybd only thing?12:23
paulsherwoodyes12:23
paulsherwoodi was getting pressed to reduce system artifact sizes12:23
rdalewhat is the difference betwen minimal and runtime then?12:24
paulsherwoodi don't know. most strata don't have a minimal12:24
paulsherwoodi guess i can try it to find out :)12:24
rdalei don't think you can have a default splitting rule to do that as it is stratum specify what can be removed to make it 'minimal'12:25
rdale^specify^specific12:25
paulsherwoodack12:26
paulsherwoodhttp://paste.baserock.org/texuyasire12:27
paulsherwooddoesn't work :)12:27
paulsherwoodhowever just specifying runtime doesn't work either... since build-essential-runtime doesn't include busybox or fhs-dirs12:28
rdalewell that is why we have 'build-essential-minimal' then12:29
*** tiagogomes has quit IRC12:29
paulsherwoodseems so12:29
*** franred has joined #baserock12:43
*** tiagogomes has joined #baserock12:44
*** edcragg has quit IRC13:01
*** edcragg has joined #baserock13:03
*** edcragg has quit IRC13:14
*** edcragg has joined #baserock13:14
ssam2might be easier if you can choose which artifacts to *exclude* from a system13:15
locallycompactI'm very stuck with this glibc thing, is this a problem encountered by build-essential?13:16
*** tiagogomes has quit IRC13:17
ssam2at a random guess, looks like a libc version mismatch13:17
ssam2not sure how that would happen, i've never looked at how Rust bootstraps itself13:17
locallycompactis there something I can do to diagnose? All I'm doing is this twice: once with bootstrap on and then once off :CFG_ENABLE_LOCAL_RUST=1 ./configure --prefix=$PREFIX, make, make install13:21
locallycompactand it gives that GLIBCXX thing the second time13:21
*** tiagogomes has joined #baserock13:30
ssam2I think this issue is specific to Rust, i've never seen it before13:34
ssam2i don't really know how to go about debugging it13:35
ssam2I guess comparing the symbols and their versions in /tools/bin/../lib/librustc_llvm-db5a760f.so and /usr/lib64/libstdc++.so.6 will tell you if it is indeed a version mismatch13:35
ssam2does Rust bootstrap itself by downloading a prebuilt binary? if so, maybe it expects a different version of GCC & GLIBC to what we have in the Baserock reference systems?13:36
ssam2although you'd think if it downloads a prebuilt binary it would be statically linked13:36
ssam2but you'd think the same about our stage1 and stage2 chunks, and actually they aren't and are completely fragile to glibc version mismatches on the host system  :(13:36
locallycompactit's not downloading a prebuilt, I turned that off13:37
locallycompactit's building one from the host tools with bootstrap on13:38
locallycompactand prefix: /tools13:38
locallycompactand then one depending on that13:38
locallycompactI'm running this on ubuntu if that matters13:39
ssam2hmm, hang on13:40
ssam2is it building in a chroot or is this in bootstrap mode?13:40
ssam2must be chroot I guess or /tools would be /src/tmpxa7864/tools13:40
locallycompactIt's in bootstrap mode13:41
locallycompactI dunno anything about a chroot13:41
ssam2ah, right13:41
ssam2how can it be building stuff in /tools then ??13:41
ssam2is it actually putting stuff in your host system in the /tools directory?13:42
locallycompactI thought that was $DESTDIR/tools13:42
locallycompactor something13:42
locallycompactthis is the stratum https://github.com/locallycompact/definitions/blob/lc/rust/strata/rust.morph13:43
ssam2$DESTDIR/tools is where stuff gets copied when its build, yeah13:43
ssam2which should definitely not be /tools in your host system!!!13:43
locallycompactthere's def not a /tools in my host systems13:44
ssam2ok, good. It seemed like there was based on the error you posted earlier13:44
locallycompactthat's the second one13:44
ssam2i see stage1-rust is build in 'staging' mode13:45
ssam2so that makes sense13:45
ssam2ok, so maybe the 1st rust is linking against your host libstdc++ instead of the Baserock one13:45
ssam2you probably have to pass various paths into the stage0 configure script to make it use one in a nonstandard location13:46
locallycompactah13:46
ssam2the Baserock libstdc++ will be installed in the staging area of the stage0 chunk, and the stage0 chunks .build dir will be the cwd13:46
ssam2so something like LDPATH='../lib/'13:47
ssam2that's a guess, i've not looked at how Rust's configure script works13:47
locallycompacthmm, there's something in here that says13:52
locallycompactopt llvm-static-stdcpp 0 "statically link to libstdc++ for LLVM"13:52
*** gtristan has quit IRC13:57
ssam2with gcc you can pass --with-build-sysroot to the configure script13:57
ssam2which means "link against libraries from here"13:57
ssam2can't find anything similar in Rust13:57
locallycompactnop13:57
ssam2is there any docs on how to cross-compile rust?13:58
ssam2or you could build stage0 not in bootstrap mode?13:58
ssam2the reason for bootstrap mode is basically so that you can use the compiler & shell from the host system13:59
ssam2but i don't think you actually need that here?13:59
locallycompacthow do I get the compiler in then?13:59
locallycompactIt normally calls out to the internet for its stage014:00
locallycompactthat turns off with CFG_ENABLE_LOCAL_RUST14:00
locallycompactthen I need a host compiler14:00
*** paulwaters_ has joined #baserock14:00
*** fay__ has joined #baserock14:03
*** fay_ has quit IRC14:03
ssam2what sort of compiler?14:04
ssam2a host Rust compiler?14:04
ssam2right, in that case you do need bootstrap mode14:04
ssam2and then you have to figure out how to get the stage0 Rust to link against a sysroot14:05
locallycompactright14:05
*** astrophys has joined #baserock14:17
*** gtristan has joined #baserock14:23
*** fay_ has joined #baserock14:36
*** fay__ has quit IRC14:38
locallycompactI can't find anything about this, other than the configure seems to advocate clang in generla15:08
jjardonAFAIK, we do not have clang in baserock15:11
jjardonneither go, paulsherwood15:13
ssam2it's possible nobody has tried to do this yet15:17
ssam2I think building with sysroots is generally only something embedded developers want to do15:18
paulsherwoodhttp://git.baserock.org/cgit/baserock/baserock/definitions.git/commit/?h=baserock/ps/docker&id=2047757886509d4b2530a6f1d0c32be4a6758ca015:19
paulsherwoodjjardon: ^^15:20
ssam2an alternative approach would be download the thing it wants to download, commit that to a repo somewhere, and install it into the chroot15:20
paulsherwoodeek :)15:21
paulsherwoodif folks want go, i'm happy to update it and submit15:25
jjardonpaulsherwood: I want everything :) if you dont have time, submit with WIP in the tittle, so at least we know a15:28
jjardon someone already work in it15:28
locallycompactssam2, I may try that just for good vibes atm15:44
paulsherwoodbenbrown_: are you around?15:49
*** ctbruce has quit IRC15:54
*** astrophys has quit IRC15:55
benbrown_paulsherwood: hello15:57
locallycompactmozilla #rust-internals has recommended I link against this thing https://www.musl-libc.org/16:05
locallycompactand presumably that's static by default16:06
rjekstatic :(16:07
ssam2static is good when bootstrapping16:07
ssam2I'm not sure using musl instead of glibc will solve the problem you posted earlier16:08
ssam2because that was a link error against libstdc++, which is part of GCC16:08
*** fay_ has quit IRC16:08
locallycompactThe guy said16:08
ssam2actually, if you could get the stage0 compiler to link itself statically then you would probably be fine, regardless of what exactly it linked against16:08
locallycompactregarding stage0 compiler:16:09
locallycompact<mbrubeck> locallycompact: For glibc, you need to build on a system with a glibc that's the same or older than the system you'll run on.16:09
locallycompact<mbrubeck> locallycompact: This is why the official binaries are build on CentOS 5, which is from 200716:09
locallycompactbut that seems to be the case for me anyway16:09
locallycompactldd (Ubuntu GLIBC 2.21-0ubuntu4) 2.2116:09
locallycompact^host16:09
locallycompactand baserock is 2.22?16:09
ssam2that seems to imply that Rust can't build against a sysroot :-)16:09
ssam2and they work around that by requiring the actual host machine has a really old glibc that they can link dynamically against, without breaking things for people with newer glibc16:10
ssam2breaking things for people with older glibc, I mean16:10
rjekThat sounds like a bit of a hack16:11
locallycompactwhy is my error version `GLIBCXX_3.4.21' not found, what version number is that16:11
ssam2the error you posted earlier was libstdc++, not glibc16:11
*** gtristan has quit IRC16:11
ssam2at least, that's what the error message appeared to say16:11
ssam2I've no idea how to interpret that version string16:11
locallycompactright16:12
locallycompactI guess I'm confused as to what the things are still16:12
paulsherwoodbenbrown_: i was wondering, given you've been working on lorry-controller etc...16:13
ssam2is https://github.com/japaric/rust-cross useful ?16:14
paulsherwoodwould you mind reading up on what Christoph has been discovering, and fix things up based on his work?16:14
ssam2the Baserock bootstrap is effectively a cross-compile, even if it's x86_64 to x86_6416:14
locallycompactthat looks like compiling crates16:14
benbrown_paulsherwood: Can do :)16:16
paulsherwoodtvm16:16
paulsherwoodwe also need some reviewers for your gitlab work16:16
ssam2"If you want to set up your Rust toolchain as a cross compiler, you have come to the right place!"16:17
benbrown_paulsherwood: Aye, just cheekily asked ssam2 if he could go through and re+1 everything16:17
paulsherwood:-)16:17
ssam2I see, and in the "How do I compile a fully statically linked Rust binaries?" section, it suggests using 'linux-musl' OS instead of 'linux-gnu'16:20
ssam2which is slightly weird, but I guess that's why your man suggested using musl16:20
ssam2so you could try building musl on the host, building stage0 rust on the host statically linked against musl, then building stage1 rust inside Baserock16:22
ssam2the stage0 rust should work anywhere because it's statically linked16:22
locallycompactI will try that, ace, tyvm16:22
*** gtristan has joined #baserock16:25
benbrown_ssam2: tyvm!16:27
paulsherwoodbenbrown_: is there any easy way to demonstrate that you've tested your lc stuff?16:28
locallycompactactually I don't think this will work16:29
locallycompactthis is still for cargo calls : cargo build --target x86_64-unknown-linux-musl16:29
locallycompactThe actual compiler just says "build a musl-enabled rust"16:30
locallycompacthttps://doc.rust-lang.org/book/advanced-linking.html#linux16:30
locallycompact ./configure --target=x86_64-unknown-linux-musl --musl-root=$PREFIX --prefix=$PREFIX16:30
locallycompactI don't think that's the same thing?16:30
locallycompactwe'll see anyway16:30
benbrown_paulsherwood: I've been ansiblising lorry-controller installation on a Ubuntu VM with GitLab16:31
ssam2i presume both cargo and the ./configure script are ultimately using the rust compiler16:31
benbrown_Almost done16:31
ssam2so if it's possible to do it with cargo, it should hopefully be possible to do it in the compiler bootstrap, even if the code isn't there right now16:32
ssam2see also the "Cross compiling with rustc" section of that doc16:32
*** toscalix has quit IRC16:42
*** mwilliams_ct has quit IRC16:45
*** franred has quit IRC16:58
*** jonathanmaw has quit IRC17:02
*** edcragg has quit IRC17:11
*** ssam2 has quit IRC17:19
*** locallycompact has quit IRC17:26
*** anahuelamo has quit IRC17:38
paulsherwoodbenbrown_: cool!18:03
*** locallycompact has joined #baserock19:10
*** gtristan has quit IRC19:32
*** cosm has quit IRC19:34
*** cosm has joined #baserock19:49
paulsherwood"Docker's build environment itself is a Docker container, so the first step is to install Docker on your system.20:23
paulsherwoodwtf20:23
paulsherwoodoh, joy... https://github.com/docker/docker/blob/master/Dockerfile20:48
locallycompactI'm shocked -> -_-20:50
paulsherwoodme too20:57
*** gtristan has joined #baserock21:33
*** gtristan has quit IRC22:35
*** edcragg has joined #baserock23:24
*** cosm has quit IRC23:28

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!