IRC logs for #baserock for Friday, 2016-01-08

*** gtristan has quit IRC03:58
*** gtristan has joined #baserock04:46
*** franred has joined #baserock08:19
*** bruce_ has joined #baserock08:25
*** bashrc has joined #baserock09:01
*** gtristan has quit IRC09:01
*** jonathanmaw has joined #baserock09:23
*** rdale has joined #baserock09:30
*** tiagogomes_ has joined #baserock09:42
*** ssam2 has joined #baserock10:05
*** ChanServ sets mode: +v ssam210:05
*** mwilliams_ct has joined #baserock10:05
*** gtristan has joined #baserock10:06
perrylpedroalvarez: regarding the baserock docker image, how easy would it be to add dependencies to it, or for me to create one myself?10:06
*** gtristan has joined #baserock10:06
pedroalvarezI tried yesterday to add sandboxlib but failed..10:08
pedroalvarezperryl: create one should be easy. You only need a baserock chroot ( that you can build using morph/ybd) and then  use `docker import`10:09
pedroalvarezsome info about that in here: http://wiki.baserock.org/guides/docker/10:09
perrylargh, that doesn't sound promising! but i'll give it a go, see how i go on :) thanks!10:11
*** Lachlan1975 has joined #baserock10:11
*** edcragg has joined #baserock10:12
pedroalvarezit should sound promising :/10:13
pedroalvarezlet me know if you are stuck :)10:13
perrylwill do :)10:13
radiofreeyikes 16-01-08 14:14:09 [474/474/477] [TOTAL] Elapsed time 14:14:0910:15
radiofreefor a gnome image10:15
rjekBuilding on a RPi? :)10:15
pedroalvarezrjek: ermm... on a Pi  more than 8h just for gcc..10:16
rjek:)10:16
* rjek remembers when building a kernel on his main machine was an overnight job.10:17
rjekProtip: Don't run Linux on a 200MHz ARM as your main machine.10:17
radiofreethis was on a mustang10:17
rjekRelated, then10:18
*** richard_1aw is now known as richard_maw10:25
*** ChanServ sets mode: +v richard_maw10:26
nowsterpedroalvarez: pi or pi2?10:42
nowsterEdgerouter was about 3-4 hours for gcc10:43
rdalex3 stages10:45
paulsherwood2 16-01-07 02:46:50 [956/956/962] [TOTAL] Elapsed time 02:46:43 (for ci.morph, mostly from scratch on AWS 36 core machine)10:52
paulsherwoodgnome sysem has more than doubled the wallclock time10:52
paulsherwood(previously i was seeing 80 mins or so)10:52
pedroalvareznowster: Pi11:07
gtristanhmmm, any pointers on how to install pyyaml, sandboxlib and jsonschema python libraries without pip ? preferably without requiring a python built with ssl support ?11:14
* gtristan was always annoyed about python modules acting so special, and not providing any 'make install' rules at the very least11:15
radiofreeis pip the thing that allows "python setup.py install" to happen?11:16
ssam2gtristan: in a git checkout, just run 'python ./setup.py; sudo python ./setup.py install'11:17
gtristanNot sure, no it provides 'pip install'11:17
ssam2that should work for any sane Python module11:17
gtristanssam2, thanks11:17
gtristanhmmm, does PyYAML require libyaml ?11:20
* gtristan see: libyaml is not found or a compiler error: forcing --without-libyaml11:20
gtristanat 'python setup.py install' time... I guess if it installed, it just fell back to PyYAML without libyaml11:21
franredgtristan, that is what we do in definitions with pyyaml :)11:22
franredhttp://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/python2-core/pyyaml.morph11:23
ssam2wow, that must be slow11:24
ssam2we should probably add libyaml and parse YAML with a C library rather than Python code!11:24
gtristanssam2, eh11:25
gtristanssam2, would shave off ~0.0001% of the build time, considering you parse definitions once and then run a build which potentially lasts hours :)11:25
* tiagogomes_ doesn't think it slow, after checking the time required to parse definitions11:26
gtristanbut yeah, to be by the book :)11:26
ssam2yeah, i guess so11:28
ssam2don't ignore the impact of 'time to compute build graph' in a build tool though, for a long time in the history of Morph it took several minutes just to start building11:29
ssam2you can imagine how great it was waiting 3 minutes to find you had a syntax error or something11:29
gtristanssam2, after working with baserock for a while, 3 minutes to find out there was a mistake somewhere sounds like the least of my worries11:30
gtristaneven a great turn-around time :)11:30
gtristanoh shit, I screwed up that configure time in build-essential... sigh, guess I'll get to test this tomorrow ;-P11:31
gtristans/time/line11:31
tiagogomes_ssam2 that's because morph was loading all morphs, and writing them back to disk. It should be pretty fast now11:40
ssam2there were various reasons! i'm talking about years ago11:48
ssam2like when Morph updated *every* git repo involved in the build each time you ran it11:49
gtristaneek, after manually installing setuptools, I find that sandboxlib and jsonschema want to download stuff :-/11:52
* gtristan wonders if it's worth going to the trouble of building openssl11:52
ssam2what do they want to download?11:53
ssam2you mean setuptools is trying to fetch their dependencies?11:53
gtristanyeah11:54
gtristanjsonschema wants vcversioner11:54
gtristansandboxlib wants pbr11:54
ssam2ah, yes it will need pbr11:54
* gtristan adds those and hopes to not discover too many others11:54
* gtristan thinks if he takes the jsonschema version from definition it will not require that other thing11:57
gtristanweird, pbr wants 'pip'11:59
gtristanssam2, is that what python-distutils does ? 'python setup.py install' ?11:59
pedroalvarezgtristan: mainly12:00
pedroalvarezgtristan: see DEFAULTS12:00
pedroalvarezalso, pbr is in strata/python2-core.morph12:01
gtristanpedroalvarez, I see12:02
gtristanyeah, I took the pbr referred to from that file12:02
gtristanexact SHA112:02
* gtristan gets this:12:02
gtristanrunning install12:02
gtristan/home/INSTALL/bin/python: No module named pip12:02
gtristanerror: ['/home/INSTALL/bin/python', u'-m', u'pip.__init__', u'install', u'pip'] returned 112:02
gtristanoh yeah12:03
gtristanhttp://git.baserock.org/cgi-bin/cgit.cgi/delta/pbr.git/tree/requirements.txt?id=0.10.712:04
gtristanhow is that possible ? do we have pip in definitions ?12:06
ssam2i think it's now bundled with Python12:06
ssam2it used to be separate12:06
ssam2unfortunate that pbr has tied things in knots. It exists to simplify the whole setuptools/distutils/distribute/setup.py mess :/12:06
gtristanssam2, the really annoying thing is... I'm sure all this stuff is for, is to end up doing a cp ${script} -> ${prefix}/lib/${pythonversion}/modules... or similar12:09
gtristancould be a few lines of makefile :-/12:09
ssam2in this case, yeah12:10
gtristanok... install pip, so it can sit there without https support and do nothing but satisfy a dependency...12:10
ssam2if you prefer, add a makefile to sandboxlib12:11
gtristansandboxlib installed fine12:13
gtristanit's pbr12:13
* gtristan downloading pip from github12:13
gtristanwhich *wants* to do some magick 'get-pip.py' as the regular install path, which wont work from my current vm12:14
gtristanbut should have a setup.py as well I presume12:14
ssam2yeah12:15
ssam2sandboxlib only depends on pbr for its setup.py file12:15
ssam2pbr is a tool to make writing setup.py a bit less crazy (you write a setup.cfg instead, so the package info is data rather than code)12:15
gtristannice12:18
gtristanthat worked12:18
gtristanis 'requests' supposed to be builtin to python ?12:24
franredgtristan, no, AFAIK12:25
* gtristan finds https://pypi.python.org/pypi/requests12:25
* gtristan files minor bug12:25
gtristanumm, I'm not sure, does it's existence here: https://pypi.python.org/pypi/requests... mean that it is indeed builtin to python ?12:29
gtristanclicking the download link oddly brings me to a page for downloading python12:29
gtristanoh would you look at that12:30
gtristanthe page says if you want to install, use pip install12:30
gtristanhahaha12:30
* gtristan finds it is actually maintained here: https://github.com/kennethreitz/requests.git12:30
gtristanHave a clue what could be causing this: http://paste.baserock.org/vomacasese12:53
gtristanthats from running: ../ybd/ybd.py strata/build-essential.morph armv5l12:54
gtristanfwiw, on this box; python -c "import sandboxlib"12:54
gtristansays nothing, and echo $? reports 0 exist status12:55
gtristanas opposed to python -c "import ridethepony"; which reports an error and complains about it's inability to ride ponies12:55
pedroalvarezthat's odd12:56
gtristanyeah :-/12:56
* gtristan thinks he installed everything alright12:56
gtristanI admit I did not pass the --prefix argument12:56
gtristanbut since python is installed under /home/INSTALL... python setup.py install seems to have been correctly shoving everything under there too12:57
ssam2it does that, yeah13:05
ssam2one thing it can do better than a Makefile :-)13:06
gtristanssam2, pkg-config macros (and other m4 bits) help with that, but yes it's much the same thing :)13:07
gtristanok I see, sort of13:21
gtristanI dont have sandboxlib in /home/INSTALL/lib/python2.7/site-packages13:21
gtristanand there is some warning when building/installing it13:21
gtristanfor some reason though; python -c "import sandboxlib" incorrectly seems to succeed13:22
paulsherwoodlc - please raise the ybd problem on #baserock13:22
paulsherwoodoops, wrong window13:22
SotKgtristan: which directory are you running `python -c "import sandboxlib"` in?13:23
gtristanSotK, that makes perfect sense !13:24
gtristanI hadnt thought of that :)13:24
gtristanindeed, it only works from the sandboxlib directory, hadn't noticed that13:24
SotKdo you have to actually install sandboxlib for what you're trying to do or can you just add the git checkout to your PYTHONPATH when running ybd?13:25
gtristanI only care about running ybd right now13:26
gtristaneverything is done completely wrong, the point is just to try it out13:26
* gtristan tries PYTHONPATH trick13:26
gtristanworks !13:27
gtristanSotK, thanks :)13:27
SotKgood :)13:28
gtristanquestion; I'm in an armv5l VM, but is that a valid arch name for definitions ?13:30
*** rdale_ct has joined #baserock13:30
gtristanor should it rather be armv5lhf ?13:30
jjardongtristan: for the future, probably it would help if you create a ybd stratum with all the dependencies ybd needs13:31
jjardongtristan: nope, I dont think any armv5 supports hard float13:32
*** rdale_ct_ has joined #baserock13:32
SotKgtristan: armv5l is correct I think13:32
gtristanok thanks13:33
*** rdale has quit IRC13:34
gtristanjjardon, we'll see where we're going, honestly I think one of 2 things; A.) this experiment works out, and the aboriginal bootstrap image contains exactly enough tooling to get the build started and nothing more... B.) YBD anyway documents it's dependencies, and we try not to presume anything about the system which is building (like if its baserock or debian or what)13:35
*** rdale_ct has quit IRC13:35
*** rdale has joined #baserock13:36
gtristanjjardon, one thing we've learned, is that the idea that one can assume that what is installed on the machine is correctly what is needed to build, just because it is another baserock build image, is a false presumption13:36
gtristanso if you can build one set of definitions on one baserock system which was built with a drastically different set of definitions, you can also do it from a debian system or other13:38
ssam2I don't think we ever expected that13:38
ssam2we assumed that if it was a specific release of systems/build-system-xxx.morph then you could assume that it contains the right stuff13:38
*** rdale_ct has joined #baserock13:38
*** rdale_ct_ has quit IRC13:39
gtristanssam2, sure, this doesnt exactly work in the current state either13:40
gtristani.e. host tooling such as syslinux/btrfs, in any case, clearly that loop needs to be closed13:40
*** rdale has quit IRC13:42
gtristanwell, ybd clearly runs... and binutils is building, but... with the chroot sandboxing and all, it is not using distcc13:42
gtristaneither I will "build the whole thing" at turtle speed, or rip apart definitions and ybd, trying a hack13:43
gtristanor more likely, I will do both :)13:43
jjardonok to lorry qtdeclarative-testsuites? https://gerrit.baserock.org/#/c/1728/13:43
*** rdale has joined #baserock13:43
*** rdale_ct has quit IRC13:46
ssam2gtristan: once upon a time Morph supported building with distcc, inside a chroot13:52
ssam2I forgot how that worked or if the code still exists13:52
gtristanoh I'm sure I can rig it up :)13:53
gtristanthing is, I will also gut build-essential for this13:53
gtristanat least some of it, at least gcc13:54
gtristanand will have to expose the outer gcc to the inner build env13:54
*** mwilliams_ct has left #baserock13:56
SotKdoes that not have reproducibility implications? (or am I misunderstanding?)13:56
gtristanSotK, not really, you can still choose what compiler you want to use, except you use a cross compiler on the host instead of the compiler built on the target13:59
gtristanSotK, so essentially, it could end up that "building the bootstrap image & compiler" for a given arch, becomes a separate activity from "building everything else on that arch"14:00
gtristanof course, there is always some chance that you have dusty ports, and they effect the bits traveling over distcc ;-)14:02
*** tiagogomes_ has quit IRC14:02
radiofreepedroalvarez: do you think it's safe to make the assumption that if someone has defined a /boot than that's where the bootloader config should go14:02
*** nowster has quit IRC14:02
gtristanSotK, actually there are dozens of ways to leverage that cross setup, one of which I've been thinking, is to start a VM instance for the build of each chunk (or have some VMs on standby), which could allow parallelism of orthogonal builds - while running YBD on the host side to launch the builds14:05
*** edcragg has quit IRC14:05
gtristanbut first, I'll try to just get through a build hehe14:06
*** edcragg has joined #baserock14:07
SotKI see, I guess I misunderstood you a bit then, I haven't thought about the cross-bootstrap stuff for about a year :)14:08
gtristanah, first stumbling block, ybd doesnt like that I have everything in a non-standard location (I built all the additional tooling into /home/INSTALL prefix)14:08
gtristanok, 11pm, but made progress :)14:08
pedroalvarezradiofree: hm..  not sure :/14:12
pedroalvarezbut is not a really ugly workaround, if not a real solution14:13
pedroalvarezcan't decide about what;s the best solution for all of this..14:14
paulsherwoodgtristan: set base: to that dir?14:14
pedroalvarezradiofree: then, upgrades will use whatever has been definied in /boot ? that would be good too14:14
pedroalvarez(just a thought)14:15
*** tiagogomes_ has joined #baserock14:16
radiofreepedroalvarez: as long as your set BOOT_PARTITION system-version-manager will use the correct partition14:17
radiofree(i.e /boot)14:17
radiofreei have to install the btrfs layout on the /boot parition though + write extlinux.conf there14:17
radiofreeassuming "if /boot, do it there" seems to be the simplest way14:18
pedroalvarezI'd like to avoid the BOOT_PARTITION magic... "if /boot, write the magic BOOT_PARTITION variable somewhere"14:19
pedroalvarezbut I agree this is another issue14:19
radiofreehmm14:19
pedroalvarezit looks weird that you are specifing a partitioning layout, and **also** seting ROOT_DEVICE and BOOT_PARTITION (if those are the names) in the cluster14:20
radiofreestill need BOOT_PARTITION14:20
radiofreeyou might have no partition layout and flash the image to /dev/sda (or similar) and BOOT_PARTITION might be on the internal emmc14:21
radiofreee.g BOOT_PARTITION /dev/mmcblk0p1 ROOT_PARTITION /dev/sda14:21
radiofreethat's exactly what i do with my jetson ( boot on emmc, / on ssd)14:22
* radiofree got sick of running out of space for upgrades14:23
gtristanpaulsherwood, not sure I have enough in /home/INSTALL, right now my PATH is /home/INSTALL/bin:$PATH, similar for LD_LIBRARY_PATH, PKG_CONFIG_PATH and ACLOCAL_FLAGS... but it's far from insurmountable14:23
pedroalvarezand in that case, I don't think makes sense to deploy the image using partitioning..14:23
gtristanpaulsherwood, thanks for pointing me to 'base', it will surely lead me in the right direction :)14:23
radiofreepedroalvarez: no it doesn't, and you wouldn't14:23
radiofreei see what you mean14:24
pedroalvarezgood :)14:24
radiofreethis is beyond the scope of this work though, since you'd have to modify x86 systems as well (i.e ones that don't use partitioning)14:25
pedroalvarezin those cases that you need a flashing script, the flashing script will take care of generating the runes needed for upgrades (in this case the BOOT_PARTITION var)14:25
radiofreemaybe..14:25
radiofreewell i suppose you could just hack system-version-manager to check /boot *or* / of the btfs partition14:25
pedroalvarezradiofree: sure! I just never found anyone who wanted to talk about this :)14:25
radiofreeok, so if you have /boot partitioned, you don't need BOOT_PARTITION, and relegate BOOT_PARTITION being set (since system-version-manager will need to know about it eventually) to the weird cases like mine14:27
radiofreecan you define fstab entires in the cluster file?14:27
pedroalvarezyup, although I wonder if in your case isn't possible to guess where is /boot14:28
pedroalvarezyes14:28
radiofreeyou could also achieve BOOT_PARTITION that way... (i.e mount it to /boot)14:28
pedroalvarezoh, so in your case you dont have /boot mounted at all, I see14:28
radiofreeno, /boot is just some folder on /14:28
radiofreesystem-version-manager mounts BOOT_PARTITION to /tmp/foo/ and works there14:29
radiofreeok, i'll get this done, and then take a look at deprecating BOOT_PARTITION14:31
radiofree(though i'm not sure doing FSTAB_....: /dev/mmcblk0p1 loads of foo is better...)14:32
pedroalvarezforget about that for now, until we think about the right solution14:37
*** gtristan has quit IRC14:48
*** gtristan has joined #baserock14:58
*** nowster has joined #baserock15:00
radiofreeis there something i have to do to stop ybd trying to redownload the cache every time?16:08
radiofreeevery time i use it on a cluster/system if goes through "try downloading foo.cache"16:08
radiofreeeven though i've already build it16:08
radiofrees/build/built16:09
radiofreei'll restart the system...16:09
pedroalvarezjust use morph16:09
radiofreewell ybd was already on here and working, i cba to use morph16:10
*** paulw has joined #baserock16:11
paulsherwoodradiofree: could you paste some of that output?16:11
paulsherwood(ideally from the top)16:12
*** paulw has quit IRC16:12
radiofreepaulsherwood: http://fpaste.org/308668/14522696/16:13
radiofreei'm running this in a chroot btw16:13
radiofree`python /src/ybd/ybd systems/build-system-armv7lhf-jetson.morph armv7lhf`16:14
paulsherwood16-01-08 00:00:12 [0/1/1] [stage1-binutils] Cache_key is stage1-binutils.efd9f69dc47162aabb5fc19eaafe97f2a83dc1e8910e7fafe6904ebfc66ef54a16:14
paulsherwoodthat means it hasn't even found the first artifact locally16:15
radiofreei see16:15
paulsherwoodyou say you've built this before?16:15
radiofreeyep16:15
paulsherwoodwhere is your artifact directory?16:16
radiofree/root/ybd/artifacts16:16
radiofreeit seems to be populating it now16:16
radiofree(it looks relatively empty)16:16
radiofreenot sure what went wrong....16:17
paulsherwoodyes, but it's *repopulating*, which seems odd if you already built.16:17
paulsherwoodi wonder if you've changed/added a conf file?16:17
radiofreei don't think so16:17
paulsherwoodgit status on the ybd directory?16:18
paulsherwood(and on the definitions)?16:18
radiofreemodified:   ybd/config/ybd.conf16:19
radiofreehowever that was 2 days ago16:19
radiofreedefinitions is modified, i added somethings to the jetson-bsp16:20
radiofreehmm interestingly now something failed to build >.<16:21
radiofree(the thing i added to the bsp)16:21
paulsherwoodwell that's progress, i guess :?16:22
radiofreeyes, at least it isn't attempting to download everything again16:22
radiofree"autom4te: need GNU m4 1.4 or later: /usr/bin/m4" hmm16:23
radiofreewell i don't care about that for now anyway, i can continue16:24
paulsherwoodit should never download what is already there16:24
paulsherwoodack16:24
pedroalvarezmaybe is ybd auto-removing things?16:24
pedroalvarez(just a thought)16:24
paulsherwoodonly if it's short of space... and it would say so16:25
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/app.py#L20016:26
radiofreecould be short of space?16:26
radiofreei only have 9G16:26
paulsherwoodyes, it will start culling oldest artifacts16:26
paulsherwoodyou could set min-gigabytes to less than 10 if your target is small16:26
*** bruce_ has quit IRC16:31
*** tiagogomes_ has quit IRC16:32
*** franred has quit IRC16:32
*** jonathanmaw has quit IRC16:32
*** bashrc has quit IRC16:32
*** CTtpollard has quit IRC16:32
*** faybrocklebank has quit IRC16:32
*** nowster has quit IRC16:32
*** Lachlan1975 has quit IRC16:32
*** ssam2 has quit IRC16:32
*** edcragg has quit IRC16:32
*** faybrocklebank has joined #baserock16:32
*** Lachlan1975 has joined #baserock16:32
*** ssam2 has joined #baserock16:32
*** ChanServ sets mode: +v ssam216:32
*** tiagogomes__ has joined #baserock16:32
*** CTtpollard has joined #baserock16:32
*** edcragg has joined #baserock16:32
*** nowster has joined #baserock16:32
*** jonathanmaw has joined #baserock16:32
*** bashrc has joined #baserock16:33
*** CTtpollard has quit IRC16:37
*** franred has joined #baserock17:21
*** jonathanmaw has quit IRC17:23
*** franred has quit IRC17:23
*** benbrown1 has joined #baserock17:29
*** benbrown_ has quit IRC17:30
*** nowster has quit IRC17:56
*** Lachlan1975 has quit IRC17:57
*** bashrc has quit IRC17:58
*** benbrown1 is now known as benbrown_18:01
*** tiagogomes__ has quit IRC18:18
*** edcragg has quit IRC18:24
*** ssam2 has quit IRC18:57
*** rdale has quit IRC20:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!