IRC logs for #baserock for Friday, 2015-10-09

jjardongtristan: that looks very promising :)00:01
*** zoli_ has joined #baserock00:27
*** zoli_ has quit IRC00:32
gtristanjjardon, :)01:43
* gtristan back in 15min01:43
*** gtristan has quit IRC01:43
*** gtristan has joined #baserock01:58
*** zoli_ has joined #baserock02:16
*** zoli_ has quit IRC02:20
gtristan <-- 500 internal server error ?02:43
*** zoli_ has joined #baserock04:05
*** zoli_ has quit IRC04:09
*** zoli_ has joined #baserock05:21
paulsherwooderk... g.b.o is completely down05:24
*** zoli_ has quit IRC05:25
paulsherwoodor rather, the cgit is down... gitano is still functioning05:25
gtristanno worry, I got through with
gtristanthere must be a better/automated way... but I crawled through those upstreams picking off the commit IDs for the morphs05:29
*** paulwaters_ has joined #baserock06:31
gtristanmkay... looks like we're almost there... just another [1/141/347] to [re]build, due to systemd.morph missing out on --enable-compat-libs, now required by new xserver06:48
gtristanand I thought I almost had a build06:48
* gtristan snap06:48
*** ssam2 has joined #baserock07:02
*** ChanServ sets mode: +v ssam207:02's cgit page was giving a 500 error, i've restarted lighttpd-git.service07:08
ssam2which has fixed the issue. no idea what the problem was07:08
gtristanwe noticed :)07:09
*** zoli_ has joined #baserock07:09
pedroalvarezssam2: that explains why there were 2 restarts of the service07:13
ssam2we really need to upgrade that thing!07:14
*** zoli_ has quit IRC07:14
pedroalvarezI restarted it too and tried to figure out what happened07:14
ssam2cgit doesn't seem to log anything anywhere07:14
ssam2well, g.b.o in general07:14
ssam2i was thinking we should add trove to the ci.morph cluster in order to remove some of the friction in upgrading it07:15
ssam2it doesn't take very long to build the trove bits when the rest of the system is in the cache, but still ..07:15
pedroalvarezssam2: I have to say that the friction now is also none07:15
pedroalvarezI've upgraded it recently07:16
ssam2ahhh, nice07:16
pedroalvarez4th of September last time07:16
ssam2we should keep a log somewhere ...07:16
ssam2i thought maybe the git log of infrastructure.git would be enough to know, but it's not really07:16
pedroalvarezHmm, true.. I upgraded it from master of definitions.git07:17
pedroalvarezI shall do that next time, and put a07:18
pedroalvarezNote in the commit message07:18
*** gtristan has quit IRC07:29
*** gtristan has joined #baserock07:29
jjardongtristan: I think I've sent a patch to the xserver so if you upgrade it you should not need enable compat libs in systemd07:36
gtristanjjardon, umm, I believe I have master07:36
gtristanor maybe not, most of xlibs I have master07:37
gtristanin anycase, it's building already07:37
gtristan[12/91/347] now, and both xserver & systemd successfully out of the way07:37
* gtristan has ~20 commits he'd like to submit once this thing builds...07:38
gtristanjjardon, I have exactly
gtristanjjardon, looking here, it looks like it should have passed07:43
gtristanbut somehow it did not, in anycase, this time it does find libsystemd-daemon07:43
jjardongtristan: btw, I'd be interested in you r thoughs about using baserock as a development tool, and, if you have, ideas to make the workflow better. I personally think is a bit painful atm07:43
* gtristan did not touch systemd version but left at v225... dont want to screw with that and have to rebuild the kernel07:43
gtristanits incredibly painful07:44
jjardongtristan: :)07:44
gtristanjjardon, 1.) ybd should ideally be able to grok the lorries I think, or there should be some story for adding dependencies locally and testing the build *without* needing to patch upstream baserock directly... 2.) It would be great to have a 'buildone' command to only build a single artifact and update it in the target image directly07:45
gtristanthe alternative of checking out sources and building, make, make install directly in a VM is enough in some cases07:46
gtristanbut when it becomes interesting is embedded, your target most probably does not have build tools installed, and it probably doesnt have SSD and quad i7s07:46
gtristanwould be great to reduce the build cycle to deploy and test on a real target hardware07:47
gtristans/reduce/reduce time needed07:47
* gtristan is sure to accumulate more ideas hehe...07:48
ssam2it's possible to do that already with Morph, but it's a bit of a faff and not really used07:48
ssam2building just one component can be done by adding a new stratum on top of all the others that contains that component07:48
gtristanI know that the whole thing is build-server oriented, but it seems to me it would be awesome if the whole thing could be a 'checkout' with local state07:48
gtristani.e. to view the entire build system, repos, etc, as if it were a single 'git'07:49
ssam2yeah, that would be cool07:49
ssam2again, the bits are there to do that in Morph, would just be 20 lines to write a 'checkout everything for this build' plugin07:50
gtristanssam2, and then I'd like to see with one command... what are all my deltas ahead of upstream ?07:50
gtristanthings like that07:50
gtristan 'what bugs have I closed locally but are still open upstream, which of my patchsets address this'07:51
ssam2gtristan: yeah, that would be awesome07:51
ssam2`morph diff` command has the basis of that already... it shows you which component 'ref' fields have changed between two versions of definitions07:52
* gtristan has been dreaming of distributed (really integrated with VCS) bug tracking for years07:52
ssam2could be extended to show the actual diff between those refs07:52
jjardongtristan: agree, can you open a new story (I'm in my phone ATM, sorry) with "reduce the build cycle to deploy" so we can put our thoughs there?07:52
jjardonFor the other thing too :)07:52
gtristanjjardon, sure... it's always good to have *something* to do with 61 builds remaining before a test (wink)07:57
*** gary_perkins has joined #baserock08:05
*** bashrc_ has joined #baserock08:09
*** toscalix__ has joined #baserock08:09
*** toscalix has quit IRC08:09
*** 20WAB8TAT has joined #baserock08:13
Kinnisongtristan: I believe Lars Wirzenius (liw) has been experimenting with distributed bug/issue tracking08:14
gtristanKinnison, I've seen a couple projects which do bug tracking and use distributed version control as backends, but I think that misses the point08:15
Kinnisongtristan: He's still very early in his experiments, but is what he was playing with08:15
Kinnisongtristan: And I'm sure he'd welcome input on "missing the point" since he's kinda paused distix until he has spare brain again08:15
gtristanKinnison, I would love to visit a sort of cgit page... where I can view a bug, and see which branches it's still open on - what stable branches it's been backported to, etc08:15
gtristanand I think it needs some though as to, how maintainers review branches to merge, instead of just patch sets, one should have a connection between the bugs which the branch "claims to close" and which patches are attached to that closing of a bug08:16
gtristanKinnison, nice to hear someone is working on that :)08:17
* gtristan looks at url08:17
Kinnisongtristan: aah, I'd love to do that from my terminal, rather than a browser :-)08:17
gtristan(or even *thinking* of working on that)08:17
gtristanyes, even in a terminal, of course as a distributed thing, it would all be a part of the state of the repository08:18
gtristanthe main issue I think is collecting bug reports from the wild, and how that would integrate08:18
gtristanKinnison, ideally, it could also remove the need to A.) always show your commit IDs in the bug report when closing a bug, and B.) Always mention the bug reference in the commit message08:19
gtristanwhich... anyway would bring everything closer together and reduce error08:20
KinnisonIt'd require some interesting variant tracking though08:21
Kinnisonsince different commits in different lines may eliminate the same bug in different ways08:21
Kinnisonalso tracking how a bug was introduced can be hard08:21
Kinnisonrjek gave some thought to this for tracking security issues, he might have input08:21
rjekI hate all bug trackers and the people who garden them, basically.08:21
gtristanjjardon, filed!/story/59 ...08:25
gtristanjjardon, and!/story/6008:28
* gtristan wonders if he gets email notifications when people comment08:29
* gtristan wouldnt mind receiving notifications for addition of any-and-all bugs, just to help get a feel for what's going on08:29
gtristanyay... less than 30 to build :)08:31
Kinnisongtristan: happy hangul birthday btw08:31
gtristanyeah that's today.. everyone is off work around here :D08:31
gtristanhehe, there is a beer festival in Gangnam I mean to go meet people at08:31
gtristanafter this build completes and I manage to upload some patches !08:32
ssam2gtristan: email notifications isn't implemented in storyboard at the moment. you can bug zara and sotk to do it, though :-)08:33
Zaragtristan: the email notifications thing is on my todo list for storyboard master. Baserock currently runs an old version so Baserock would need to update it first, to get the changes from master. And since storyboard upstream currently means me and SotK, it could take a while.08:34
Zarawe hang out on #storyboard on freenode, and I'm hoping to do a storyboard session at the OpenStack summit, so hopefully we can attract more developers there.08:35
*** mariaderidder has joined #baserock08:36
gtristanZara, I find myself adding [placeholder] tasks on baserock's storyboard btw08:36
ZaraI'm also going to look into reviewing on other openstack projects. the issue is with getting to know the codebases well enough to review usefully, though the same could be said in reverse.08:36
gtristanas often; when reporting something, it's not yet clear what needs to be done to address the issue08:36
* pedroalvarez remembers he never downloaded the storyboard snapshot08:36
pedroalvarezglance image-download is awesome:08:37
pedroalvarez  --file <FILE>  Local file to save downloaded image data to. If this is not08:37
pedroalvarez                 specified the image data will be written to stdout.08:37
pedroalvarezwhy on earth would I want a diskimage in stdout!!?08:37
Kinnisonmmm pipey08:37
gtristanto stream it into gzip or such08:37
pedroalvarezbut the default..?08:38
20WAB8TATpedroalvarez, `glance image-download --file image | gzip > myimage.csv.gz`08:38
20WAB8TAT20WAB8TAT wonders who 20WAB8TAT is08:38
pedroalvarezthanks 20WAB8TAT08:38
*** 20WAB8TAT is now known as tiagogomes08:38
* tiagogomes is annoyed that that last glance client seems to not be able to create an image from an URL anymore08:40
Zaragtristan: btw, if you have issues with storyboard or the webclient itself, it's best leaving stories here:!/project/457 or here:!/project/456 so we have a more permanent record. or better yet seeing if you can help us hack around; a lot of the time we know about the bugs, we just have a lot to work on and there's only so much we can do at once! The people who wrote mos08:40
Zarawebclient gerrit is here, if anyone would like to help on the reviewing side:,n,z08:41
pedroalvarezZara: "The people who wrote mos...." ?08:41
Zarat of the code originally quit in fury some months ago over a lack08:41
Zara              of resources, so it's also hard for us to get answers about the current implementation except by looking at the code.08:41
Zarathere you go :P08:41
ZaraI just want to explain why it takes a while, so you know it's not a case of us not listening!08:42
Zarastoryboard gerrit here:,n,z08:42
gtristanok, so: echo "Bug report" | Zara --file-report... is not a viable option... I understand :D08:42
gtristanZara, do you know if it's still an issue upstream ?08:43
gtristanI mean... ability to file a bug without adding any task ?08:43
Zara:) I have a lot of backscroll and things to focus on, so I can note that you care about it but it might get lost.08:43
ZaraI don't think it is, no.08:43
gtristanok, baserock seems to force a task, that's what I was saying...08:44
Zaraoh, sorry, I think it is an issue, misread08:44
gtristanif you like... I could file it08:44
* gtristan guesses he'll find out while trying to file one on openstack ;-)08:44
ZaraYeah, I think it's not implemented yet, sorry. I don't think I've tried to do that myself! :)08:45
Zaraif you get an irritating 401, that's what I'm currently looking into, just so you know, haha08:45
*** jonathanmaw has joined #baserock08:46
Zara(the session in japan is going to be effectively 'we have loads of bugs, we have very few people; what should we prioritise?' I was going to ask for stuff on this on the infra mailing list, but if baserock also have priorities, let me know.)08:47
Zarameanwhile, I'll try to keep the task placeholders in mind.08:47
gtristanZara, filed as:!/story/200037508:48
gtristan[74/91/347]... and trucking... chug chug, huff puff...08:49
Zaragreat, thanks. :) please just understand that SotK updating baserock's storyboard will mean development on storyboard master will slow down, though we're doing our best! :)08:54
paulsherwoodrichard_maw: you were right! on moonshot 2 instances max-jobs 8 beats one...08:54
paulsherwoodbut amazingly 3 beats 2, and 4 just about beats 3 !!!08:54
SotKpedroalvarez: did you manage to obtain that snapshot last night? :)08:55
pedroalvarezSotK: nope, sorry, I had some last minute meetings and forgot about it08:56
pedroalvarezI didn't have enough space in the disk anyway.. :/08:57
SotKah, no worries :)08:58
pedroalvarezdownloading now ;)08:58
*** edcragg has joined #baserock09:00
*** toscalix__ has quit IRC09:13
*** fay_ has joined #baserock09:21
*** wschaller has joined #baserock09:24
*** lachlanmackenzie has quit IRC09:34
gtristanjjardon, unfortunately... new X server is having a rough time starting up09:41
* gtristan hangs head in disappointment... after all that time building09:41
*** toscalix__ has joined #baserock09:42
gtristanI'm sure that probably either a less recent xserver, or some tweaking of the config will solve it, and then gnome-shell wont hang anymore09:42
gtristanbut dont have the patience on friday night for another 3 hour build :-/09:42
*** toscalix__ has quit IRC09:49
*** toscalix__ has joined #baserock09:49
* SotK looks at the stories gtristan filed earlier09:56
SotKgtristan: it should be possible to build without lorrying just by changing the "repo" field for the relevant component in your stratum09:57
SotKany URL which works with `git clone` should work there I believe09:57
SotKHaving the repo lorried is only a requirement to merge changes into master of definitions09:57
gtristanthat is true, and it's what I was doing in fact09:59
* pedroalvarez is playing with a rpi209:59
pedroalvarezI'll try to put baserock on it :)09:59
gtristanI suppose it means; it's impossible to send your patch set to definitions without having a lorry commit accepted, and it's impossible to see what the lorry patch does without having the new definitions either10:00
SotKwhat do you mean "see what the lorry patch does"?10:01
SotKyou can run lorry locally in a devel VM to check the patch works as expected10:01
gtristanwell, if you are on the system maintainer side, you receive a lorry patch... want to see what justifies it right ?10:01
SotKah, I see what you mean10:02
SotKI agree it would be good to come up with a better solution to that than asking "What is this for" in here every time10:02
gtristanSotK, right, and "in here" presumes that nobody else is hosting a baserock environment :)10:05
Zarayeah, that used to be a problem for me, too. I'd also find I had to repeat my reasons a lot when adding lorries. I tended to put a note in gerrit but it wasn't ideal.10:05
*** paulwaters__ has joined #baserock10:07
*** paulwaters_ has quit IRC10:10
*** paulwaters_ has joined #baserock10:13
gtristanjjardon, I'm going to try xserver-1.17-branch anyway... that's 63 builds, I'll let it run over dinner and let you know how it went10:15
* gtristan thinks it will pass and then we'll see a gnome-shell finally10:15
gtristanbut let's see, if not, we're missing some additional component, or config10:16
* gtristan better get food10:16
*** gtristan has quit IRC10:17
*** paulwaters__ has quit IRC10:17
pedroalvarezthis rpi kernel is taking almost an hour to compile..10:21
pedroalvarezI think it would be good to disable some things that were only enabled for openstack10:24
jonathanmawIs there a running kbas server anywhere I can use? I'd prefer not to have to rebuild an armv7lhf from scratch10:27
*** gtristan has joined #baserock10:43
*** toscalix__ is now known as toscalix10:46
pedroalvarezjonathanmaw: nope10:46
pedroalvarezWell, at least is not part of baserock infra10:47
Kinnisonkbas is entirely insecure and ybd logs the password (so beware of posting unadulterated logs)10:47
pedroalvarezWe do have a morph cache artifact full of armv7 artifacts10:47
pedroalvarezArtifact server I mean10:49
*** lachlanmackenzie has joined #baserock10:50
pedroalvarezI wonder if anyone around has any experience with rpi2 to point me to all the steps they think are needed to put baserock in there10:55
pedroalvarezedcragg: I've heard you did something with rpi110:57
* pedroalvarez finds a script from kali that creates the kali OS and creates also the image11:00
pedroalvarezit might be useful to understand what's needed11:00
*** paulwaters_ has quit IRC11:06
*** paulwaters_ has joined #baserock11:10
*** cosm has joined #baserock11:25
pedroalvarezhm.. these instructions clone the entire linux-firmware repo in /lib/firmware11:26
pedroalvarezI'm trying to work out what files are needed11:26
*** zoli__ has joined #baserock11:36
*** zoli__ has quit IRC11:41
paulsherwoodnow the only thing is to identify which definitions that set was built from11:48
* jonathanmaw tries to figure out how to configure ybd to use that server11:51
paulsherwoodjonathanmaw: 18242ab38f8514eb36852b29e2dd676f605ae523 of definitions11:52
paulsherwoodjonathanmaw: set kbas-url:
jonathanmawpaulsherwood: that's newer than the GDP branch, by the looks of it.11:53
paulsherwoodjonathanmaw: you wanted a rootfs, i thought?11:53
pedroalvarezjonathanmaw: it may be newer, but if we want to put it  in mainline, we have to rebase the work somehow11:54
jonathanmawpaulsherwood: I'm currently making sure that I can build the GDP in YBD. Once that's done, I can look at merging with master, then appropriate fixes to the GDP.11:54
paulsherwoodjonathanmaw: were you seeing a kbas server to push to, rather than pull from?11:55
jonathanmawpaulsherwood: pull from, initially, in the hopes I can shorten the build time.11:56
paulsherwoodcan you do the work on x86 first?11:58
paulsherwoodalternatively i can kick off build of the gdp on a set of scaleway machines to hit that kbas server11:59
jonathanmawarmv7lhf is currently the most comprehensively-tested, but I can leave that running in the background for now.11:59
jonathanmawnow to see if I can remember my credentials for my x86 builder...12:00
paulsherwoodjonathanmaw: if you mail me your public key i can add you to some scaleway machines12:00
jonathanmawpaulsherwood: biff12:02
*** brlogger has joined #baserock12:42
pedroalvarezhere we go :)12:42
*** toscalix has quit IRC13:02
*** petefoth has quit IRC13:14
*** tiagogomes has quit IRC13:14
*** gary_perkins has quit IRC13:14
*** wschaller has quit IRC13:14
*** edcragg has quit IRC13:14
*** mariaderidder has quit IRC13:14
*** bashrc_ has quit IRC13:14
*** CTtpollard has quit IRC13:14
*** nowster has quit IRC13:14
*** jonathanmaw has quit IRC13:14
*** edcragg has joined #baserock13:14
*** tiagogomes has joined #baserock13:14
*** CTtpollard has joined #baserock13:14
*** wschaller has joined #baserock13:14
*** bashrc_ has joined #baserock13:14
*** mariaderidder has joined #baserock13:14
*** nowster has joined #baserock13:14
*** jonathanmaw has joined #baserock13:14
*** petefoth has joined #baserock13:14
*** gary_perkins has joined #baserock13:15
*** toscalix has joined #baserock13:29
*** paulwaters_ has quit IRC13:54
richard_mawhmm, webapp.db doesn't contain a jobs table…14:08
richard_mawcurrently the cleanup is failing because it doesn't have a jobs table, so there are no jobs listed to clean up14:09
richard_mawssam2: how did you rotate the webapp.db?14:09
richard_mawssam2: yeah…'s lorry-controller is not working, the minions are failing to request jobs, I think because the jobs table is missing14:15
Kinnisonrichard_maw: eep14:16
richard_maw…it will initialize the tables if when the StateDB is accessed the file doesn't exist, but there is a race between opening and using, so if the DB was rotated at the wrong time then it won't be initializde14:17
ssam2richard_maw: I just rename it14:17
ssam2sometimes I have to restart the process after, I guess I forgot this time14:18
ssam2restarting it will fix it14:18
* richard_maw isn't so sure about that unless you also remove the DB14:22
ssam2oh, yeah.14:24
richard_mawas for why it was created in a poor state, I'm guessing it's a race between the file being removed and it being used14:25
Kinnisonthat's worrisome since the webapp is supposed to not be racy at all for the db14:26
richard_mawfor general use it's race safe from locking when using it, but there's a TOCCTOU bug where it checks whether the file exists, stores that result boolean, then connects to the db, and will conditionally initialize it based on the boolean14:27
richard_mawso if you're unlucky enough to remove the file after it's been checked, but during use, then it will create it empty, and fail to notice it needs to initialize it14:28
KinnisonThen again, if the old job report cleanup was working, we'd not have to keep rming it, no?14:33
richard_mawcorrect, but rotating the DB like this isn't a great solution, so we need to fix or replace in-place cleanup surely14:35
richard_mawhrm, the sqlite3 module doesn't let you pass a file descriptor (other than hackily with /proc/self/fd/%d), so we can't do an open to race-freely determine whether it needs initializing. We could uncionditionally run some create statements that will create the table if we don't have it, though that's more overhead14:38
KinnisonI'd rather we just stop rotating it14:39
KinnisonIf we can fix the issue with the size14:39
*** fay_ has quit IRC14:54
richard_mawssam2: did you take action on lorry-controller, or should I?15:00
richard_mawhrm, the C api to sqlite3 requires the file path, though _it_ has flags which would let us open/create it race-free, however they didn't make it to the python level API15:15
KinnisonIf we only ever rotate the database by stopping the webapp first, then we're safe, yes?15:16
*** franred has joined #baserock15:18
Kinnisonin that case I'm less bothered about trying to code in fixes to people doing bad things, and more interested in the cleanups.  Did the l-c on gbo have the cleanup stuff in it, and if so why wasn't it working usefully?15:19
ssam2richard_maw: i've not, go ahead and restart if you're logged in anyway15:20
ssam2it seems to have the cleanup service there (I didn't realise that before), so not sure why it doesn't work15:21
richard_mawand we can't with the DB broken15:21
* richard_maw cleans up15:21
*** wschaller has quit IRC15:25
*** cosm has quit IRC15:29
*** wschaller has joined #baserock15:41
*** persia has quit IRC15:46
*** persia has joined #baserock15:47
pedroalvarezoh well, I think I have something I can try to deploy into this sdcard for the raspberry15:49
richard_mawKinnison: the default period for what's considered too old a job is 1 year, which would explain why it's not cleaned any up15:53
Kinnisonrichard_maw: aah15:53
Kinnisonrichard_maw: Frankly a week is probably enough15:54
Kinnisonrichard_maw: but perhaps set it to a day, and we'll see how it looks on Monday?15:54
richard_mawI was going to go with 2d, but 1d is probably more responsive15:54
richard_mawok, set to 1d now15:58
richard_mawthe logic for cleanup is tested by yarns, so it ought to work, just we had inappropriate config15:58
KinnisonSeems plausible15:59
*** CTtpollard has quit IRC16:00
richard_mawwe might want smarter logic, like keeping the logs for the most recent success, and the start and end of a chain of failures16:00
KinnisonHmm, for now I'll settle for just not needing to rotate the db :_)16:01
ssam2richard_maw: so simple ! are you fixing the default or just the value in ?16:03
ssam2the former would be better I think, so other troves don't have the same problem16:03
richard_mawtrying the value on g.b.o first16:03
*** franred has quit IRC16:15
*** ssam2 has quit IRC16:15
pedroalvarezedcragg: I see you have added a boolean to use partitioning or not :)16:16
*** jonathanmaw has quit IRC16:16
pedroalvarezI guess with that we are good to not break the flashing script until we create another one16:16
*** tiagogomes has quit IRC16:18
pedroalvarezhm.. no, this image is not going to boot :/16:30
pedroalvarezI need to put somewhere where is the rootfs16:31
pedroalvarezinstead of "root=/dev/mmcblk0p2"16:32
pedroalvarezit might be a matter of adding extra info for btrfs16:33
pedroalvarezfootfstype and footflags16:33
pedroalvarezanyway, I'm going to try it to see how it crashes16:35
* SotK wishes pedroalvarez luck16:38
pedroalvarezSotK: thanks16:39
pedroalvarezI'm going to need it :)16:39
pedroalvarezFirst of all, the /boot partition is empty. My partitioning file may be wroing... or the code broken. Will investigate16:39
pedroalvarezI may need something like a serial cable to be able to do this,, I wonder if that exists16:40
pedroalvarezyay!!! kernel panic!!!16:47
* pedroalvarez dances16:47
pedroalvarezso I have to work out why the boot partition is not been populated, and then work out how to make the kernel find the rootfs16:48
*** mariaderidder has quit IRC16:53
pedroalvarezhm.. modifying the cmdline.txt file in the boot partition didn't work16:57
*** bashrc_ has quit IRC17:00
*** wschaller has quit IRC17:00
pedroalvarezI kind of think that if I use an ext3 partition it will work17:01
pedroalvarezbut I'll leave that for another moment17:01
Zaraheh, hope it works out. 'night.17:01
pedroalvareznight Zara!17:02
*** edcragg has quit IRC17:16
*** gary_perkins has quit IRC17:18
*** gtristan has quit IRC17:39
*** toscalix has quit IRC17:46
* pedroalvarez discovers BOOT_PRINTK_DELAY18:39

Generated by 2.15.3 by Marius Gedminas - find it at!