| Walkerdine | I can't seem to get my sata drive to connect to the jetson | 01:29 |
|---|---|---|
| Walkerdine | It shows up on my computer just fine but it comes up with errors when I hook it up to the jetson | 01:30 |
| Walkerdine | I just have endless problems ha | 01:39 |
| Walkerdine | I'm getting abunch of COMRESETS when I hook my sata drive up | 01:41 |
| *** zoli__ has joined #baserock | 07:20 | |
| *** fay_ has joined #baserock | 07:52 | |
| *** toscalix__ has joined #baserock | 08:01 | |
| *** toscalix__ is now known as toscalix | 08:02 | |
| rjek | Walkerdine: I've seen that with faulty SATA leads; have you tried using the same cable you used to connect it to your PC? | 08:11 |
| *** jonathanmaw has joined #baserock | 08:12 | |
| *** rdale has joined #baserock | 08:20 | |
| *** petefoth_ has joined #baserock | 09:01 | |
| *** petefoth has quit IRC | 09:03 | |
| *** petefoth_ is now known as petefoth | 09:03 | |
| pedroalvarez | <paulsherwood> infrastructure/extensions/simple-network.configure requires cliapp, so ybd can't deploy it | 09:56 |
| pedroalvarez | paulsherwood: this might be because it's a fork and it's not up to date | 09:57 |
| paulsherwood | pedroalvarez: yup. i was only trying it since definitions seems to have no 'subsystem' examples | 09:57 |
| *** petefoth_ has joined #baserock | 09:59 | |
| SotK | paulsherwood: definitions/clusters/installer-build-system-x86_64.morph has subsystems | 09:59 |
| *** petefoth has quit IRC | 09:59 | |
| *** petefoth has joined #baserock | 10:02 | |
| *** petefoth_ has quit IRC | 10:03 | |
| paulsherwood | it does? i thought i came up empty on grep | 10:04 |
| *** edcragg has joined #baserock | 10:04 | |
| * paulsherwood wonders what he must've been smoking | 10:05 | |
| rjek | Washing powder? | 10:07 |
| Kinnison | toenails | 10:07 |
| pedroalvarez | :) | 10:11 |
| pedroalvarez | I believe clusters/release.morph also has subsystems | 10:11 |
| * pedroalvarez remembers we have added xfce to ci.morph | 10:12 | |
| tiagogomes_ | the openstack clusters also have subsystems | 10:12 |
| pdar | paulsherwood: Heya, was following your "... ybd on aws" instructions at w.b.o/ybd and got stuck. Ever seen this before? http://paste.baserock.org/mevucuyuku | 10:15 |
| richard_maw | ouch, fun! could be one of your dependencies bumped their version requirement | 10:18 |
| richard_maw | at which point you might be able to get around it by being more specific about the version you need | 10:18 |
| richard_maw | (which tbh I'd recommend anyway) | 10:18 |
| paulsherwood | pdar: what is your pip version? | 10:19 |
| *** rdale has quit IRC | 10:19 | |
| paulsherwood | also, which ami are you using? | 10:19 |
| Kinnison | paulsherwood: we're using the same AMI you said you'd picked | 10:20 |
| Kinnison | paulsherwood: in the same size instance you said you'd used | 10:20 |
| paulsherwood | :) | 10:20 |
| *** rdale has joined #baserock | 10:20 | |
| pdar | hmm, the pip version is 7.1.2 | 10:20 |
| SotK | is `python` python2 or python3 here? | 10:21 |
| richard_maw | pdar: ΓΈ_O that means something you depend on is *hard* depending on an older version | 10:21 |
| paulsherwood | pdar: do the pip installs separately, work out which is misbehaving | 10:22 |
| Kinnison | a commandline tool which tracebacks to indicate issues is not a good tool IMO | 10:22 |
| * paulsherwood didn't choose python for baserock tooling | 10:22 | |
| * paulsherwood didn't choose pip as the default installer for python tools :) | 10:23 | |
| pdar | hmm, ok, I tried to use pip install to degrade pip to the 6.1.1 version but pip played up with the same error | 10:23 |
| * SotK vaguely recalls seeing something like that when he was using pip with python3 in the past | 10:23 | |
| richard_maw | paulsherwood: that may not necessarily generate the same result, as it may be attempting to install the whole set based on the sum of the constraints, while each individually would be satisfiable | 10:23 |
| paulsherwood | richard_maw: ack | 10:24 |
| paulsherwood | it would clearly be preferable to have a baserock ami, but i found that too complicatated to attempt | 10:24 |
| Kinnison | I'm working on that | 10:24 |
| paulsherwood | :) | 10:24 |
| Kinnison | I *believe* I can get something up | 10:24 |
| pedroalvarez | :D | 10:24 |
| richard_maw | it's become way easier since I last looked at what was required | 10:24 |
| Kinnison | just as soon as pdar can get a build going on this EC2 instance | 10:24 |
| paulsherwood | belief is very important | 10:24 |
| * pedroalvarez whispers: "do not build, you only need to deploy nowadays" | 10:26 | |
| paulsherwood | ? | 10:26 |
| pedroalvarez | mason + cache.baserock.org :) | 10:26 |
| paulsherwood | ybd's cache is not compatible with that, sadly | 10:27 |
| paulsherwood | and ybd needs atomicity and some other things for caching | 10:27 |
| paulsherwood | on my aws machine... | 10:28 |
| paulsherwood | Python 2.7.9 | 10:28 |
| paulsherwood | pip 6.0.8 | 10:28 |
| Kinnison | we seem to have pip 7 installed in /usr/local | 10:28 |
| Kinnison | I could probably blat that out | 10:29 |
| paulsherwood | did pdar install that, or was it already there? | 10:29 |
| * paulsherwood will re-run and fix the instructions at some point, or pdar could if he would like to | 10:29 | |
| Kinnison | pdar: did you upgrade pip? | 10:30 |
| pdar | I did not upgrade pip, just used the `get-pip.py` to get pip | 10:33 |
| * paulsherwood notices that in his shell history on that machine he actualy did yum install pip, not what is written on the wiki | 10:33 | |
| pdar | I think | 10:33 |
| Kinnison | paulsherwood: tsk | 10:33 |
| Kinnison | pdar: blat the pip you installed with that, and use 'yum install pip' | 10:34 |
| Kinnison | pdar: I can do that if you want | 10:34 |
| paulsherwood | Kinnison: we all make mistakes | 10:34 |
| Kinnison | paulsherwood: aye | 10:34 |
| paulsherwood | i shut down a power station once | 10:34 |
| Kinnison | paulsherwood: in a previous job, public mistakes led to the doughnut hat | 10:34 |
| * Kinnison dreads to think how hard it'd be to distribute doughnuts to all of #baserock | 10:35 | |
| paulsherwood | :) | 10:35 |
| pdar | Kinnison: how should I effectively blat the current pip? | 10:35 |
| * richard_maw is a fan of `find /usr/local/foo -delete` | 10:36 | |
| Kinnison | umm | 10:36 |
| Kinnison | that might be slightly heavyweight | 10:36 |
| paulsherwood | python -m pip uninstall ? | 10:36 |
| Kinnison | oooh | 10:37 |
| * Kinnison manages it | 10:37 | |
| Kinnison | with repeated 'python -m pip uninstall pip' | 10:37 |
| Kinnison | oooh yay pip | 10:38 |
| Kinnison | it removed the one in /usr | 10:38 |
| Kinnison | as well as the one in /usr/local | 10:38 |
| * richard_maw slow claps | 10:39 | |
| * Kinnison beats the install over the head with some yum --force | 10:39 | |
| * rjek mutters about pip | 10:39 | |
| Kinnison | pdar: okay, pip 6.1.1 is installed | 10:39 |
| Kinnison | pip is worse than cabal | 10:39 |
| pdar | yay! thanks Kinnison | 10:39 |
| richard_maw | Kinnison: well, cabal is at least functional | 10:39 |
| * richard_maw ducks | 10:39 | |
| Kinnison | richard_maw: Right, you owe me lunch | 10:40 |
| richard_maw | I was off for a burrito now-ish, you're welcome to join me, or I can settle up later. | 10:41 |
| Kinnison | heh | 10:41 |
| Kinnison | another day | 10:41 |
| pdar | thanks for helping paulsherwood Kinnison richard_maw | 10:55 |
| Kinnison | pdar: you're very welcome | 10:56 |
| * paulsherwood apologises for the dodgy instructions | 10:57 | |
| Kinnison | paulsherwood: have you corrected them? | 10:57 |
| *** zoli__ has quit IRC | 10:58 | |
| pdar | Ive updated them now | 10:59 |
| pdar | If I run a build with ybd should it utilise all the resources of the monster machine without any special input? | 11:00 |
| Kinnison | paulsherwood: ^^^ | 11:01 |
| radiofree | It should yes | 11:01 |
| radiofree | I think it'll tell you at the start what it's going to pass to -j | 11:02 |
| pdar | ahh yes, so it does. ta radiofree | 11:02 |
| * Kinnison watches CPU utilisation jump to nearly 10% | 11:03 | |
| paulsherwood | no. | 11:03 |
| Kinnison | and network IO jumps way higher | 11:03 |
| paulsherwood | you need to create ybd.conf in its home directory (or modify config/modify.conf) and add: instances: 5 | 11:04 |
| paulsherwood | pdar: kill it | 11:04 |
| paulsherwood | you probably also want to set base: '/src' (assuming you're doing the work on a separate volume) | 11:04 |
| pdar | it has been vanquished | 11:05 |
| pdar | I set the `base: /src` thing | 11:05 |
| Kinnison | we have a /src volume yes | 11:05 |
| paulsherwood | and you'll want to rm -fr /root/.cache/ybd | 11:05 |
| paulsherwood | which is its default base | 11:05 |
| radiofree | paulsherwood: I didn't have to do that on the mustang | 11:06 |
| paulsherwood | radiofree: did you have a /src/ partition? | 11:06 |
| radiofree | Yep | 11:06 |
| radiofree | No I mean set the number of instances | 11:06 |
| paulsherwood | radiofree: no, fair enough. how many cores does that have? | 11:07 |
| radiofree | 8 | 11:07 |
| paulsherwood | well, probably no advantage to adding instaces | 11:07 |
| pdar | how many instances should i give for, is it 40 cores? | 11:07 |
| paulsherwood | but for the biggest aws machines, 5 instances is best | 11:07 |
| pdar | cool, 5 it is | 11:08 |
| paulsherwood | s/best/fastest to complete a full build of ci.morph/ | 11:08 |
| radiofree | ah right, so instances will create 5 instances of ybd | 11:09 |
| radiofree | is it smart enough to split the cores between them? | 11:09 |
| radiofree | e.g -j10 for each or something | 11:10 |
| paulsherwood | yup | 11:10 |
| radiofree | neat | 11:10 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/app.py#L126 | 11:11 |
| paulsherwood | i think there's room for more optimisation though | 11:11 |
| paulsherwood | anyone looking at the output will cringe to see it start building loads of the same stuff five times in parallel | 11:12 |
| *** zoli__ has joined #baserock | 11:34 | |
| * richard_maw is ready to test firehose in the chunky AWS instance, as soon as he has details for the test trove | 12:39 | |
| *** zoli__ has quit IRC | 13:04 | |
| paulsherwood | i'm trying to 'improve' ybd, but my 'improvement' is making glibc fail with 'gcc: fatal error: environment variable 'STAGE2_SYSROOT' not defined' | 13:08 |
| paulsherwood | i'm guessing this means it's got the wrong gcc somehow? | 13:08 |
| richard_maw | depends where in the build it is | 13:09 |
| richard_maw | if it's part of the bootstrap then you're failing to set an environment variable | 13:09 |
| richard_maw | otherwise, yes, it's using a bootstrap GCC instead of a proper self-hosted one | 13:09 |
| paulsherwood | i'm building build-system. i believe there's only one [glibc] ? | 13:09 |
| richard_maw | pardon? | 13:10 |
| richard_maw | oh, only building one glibc | 13:10 |
| paulsherwood | yup, as opposed to stage*glibc | 13:10 |
| richard_maw | in which case, yes, it's using the wrong gcc. I'd guess it's picked the wrong gcc out of the cache, or you're installing dependencies you shouldn't over the top of ones you should | 13:12 |
| *** zoli__ has joined #baserock | 13:13 | |
| *** Walkerdine_ has joined #baserock | 13:19 | |
| Walkerdine_ | rjek: I plugged it into my pc and it worked fine | 13:21 |
| Walkerdine_ | Maybe my jetson doesn't like the sata cable? Idk | 13:22 |
| paulsherwood | richard_maw: tvm, i'll probe further | 13:23 |
| rjek | Walkerdine_: Using the same cable? | 13:24 |
| Walkerdine_ | yeah | 13:24 |
| Walkerdine_ | rjek: Should I try with a different cable on the jetson or because its the same one it should work? | 13:29 |
| rjek | Walkerdine_: What's the precise error you're seeing from the Jetson with the SSD connected? | 13:30 |
| Walkerdine_ | Its saying that the link is slow, please wait | 13:30 |
| Walkerdine_ | and failed to IDENTIFY' | 13:30 |
| Walkerdine_ | rjek: Do you think using a different SSD would help? | 13:43 |
| rjek | Walkerdine_: I can't really suggest anything without knowing what the actual error is | 13:43 |
| rjek | It could be that the is a bug in either/both of the kernel driver or SSD firmware, could be browning out on the supplied power rails, etc | 13:44 |
| Walkerdine_ | The only error I know of is that its saying that the link is slow to respond | 13:46 |
| Walkerdine_ | ata1: link is slow to respond, please be patient (ready=0) | 13:47 |
| Kinnison | is the drive well powered? | 13:47 |
| * rjek is thinking that it may be browning out, then | 13:47 | |
| rjek | How is the drive being powered? | 13:48 |
| Walkerdine_ | I happened to use the same power source when I plugged it into my computer | 13:48 |
| rjek | (Does the Jetson have a SATA power connector, and does it provide both 5V and 12V rails?) | 13:48 |
| * wdutch wonders if SimpleHTTPServer will be okay for the CIAT fileserver | 13:48 | |
| rjek | Walkerdine_: Is any of this helpful? https://devtalk.nvidia.com/default/topic/830349/jetson-tk1-and-sata-drive-issue/ | 13:48 |
| Walkerdine_ | which is the jetson's stat power | 13:48 |
| rjek | Walkerdine_: Do you have another Jetson to try? That page suggests it may be build variance. | 13:49 |
| rjek | (ie, some Jensons have marginal SATA) | 13:49 |
| Walkerdine_ | I do not but I might have to get one | 13:50 |
| Walkerdine_ | I don't see those exact messages but very similar ones. The two things I'm seeing suggested is that its either a kernel issue or the hardware is bad. I'm guessing the hardware is bad | 13:54 |
| richard_maw | Kinnison: I take it that the cu010-trove.codethink.com trove is configured to allow pushes to br6/foo branches, given we're in the br6 groups | 14:03 |
| Kinnison | it might be cu010-trove/br6/ | 14:03 |
| Kinnison | given the trove-prefix concept | 14:03 |
| Kinnison | it's only a temporary place while we prototype this stuff | 14:03 |
| * richard_maw had conflated the trove-prefix and the project prefix | 14:04 | |
| richard_maw | I guess the most reliable way to find out is push something, given I'm not in a group that would allow inspecting the gitano config. | 14:04 |
| Kinnison | well, try creating a project :-) | 14:05 |
| * richard_maw thought the way to do that was create the group structure | 14:06 | |
| Kinnison | rpeo | 14:08 |
| Kinnison | repo | 14:08 |
| Kinnison | I meant repo | 14:08 |
| * Kinnison dies of: inability to type in public | 14:08 | |
| richard_maw | Kinnison: I'm more interested in being able to push refs to existing projects, which appears to work. | 14:10 |
| Kinnison | okies | 14:10 |
| richard_maw | Kinnison, wdutch, pdar: $ git ls-remote ssh://git@cu010-trove.codethink.com/baserock/baserock/definitions.git | grep br6 | 14:42 |
| richard_maw | 793d48720ed057370a2b939bfdb163133d577c10refs/heads/cu010-trove/br6/firehose-test-1 | 14:42 |
| Kinnison | nice | 14:42 |
| Kinnison | though you don't *need* the br6 in that refname :-) | 14:43 |
| * richard_maw shrugs | 14:43 | |
| Kinnison | richard_maw: once our ops chappy has opened the firewall, people should be able to use git:// for that too :-) | 14:43 |
| Kinnison | Apparently I'm the first person to ask for an "everything from everywhere" firewall rule :-) | 14:44 |
| richard_maw | I get the impression I was expected to change the morph config to change what baserock:baserock/definitions resolved to, rather than putting ssh://git@cu010-trove.codethink.com/baserock/baserock/definitions.git in the repo field for the firehose config | 14:47 |
| Kinnison | probably | 14:47 |
| Kinnison | set trove-id to cu010-trove | 14:47 |
| Kinnison | and trove-hostname to cu010-trove.codethink.com | 14:47 |
| Kinnison | and that should do most of it | 14:47 |
| richard_maw | eh, it worked after I changed how it determined the path to the definitions repository | 14:48 |
| Kinnison | hehe | 14:48 |
| Walkerdine_ | I'm really hoping I can contribute something once I get this all set up | 15:02 |
| Kinnison | Walkerdine_: even if you contribute only some notes on how to get started, it'll be helpful | 15:02 |
| Kinnison | Walkerdine_: every contribution is gratefully received | 15:02 |
| Walkerdine_ | Whether it be baserock or genivi | 15:02 |
| *** rdale has quit IRC | 15:03 | |
| Kinnison | okay, so the rule change didn't make stuff public | 15:05 |
| Kinnison | gimme a sec | 15:05 |
| Kinnison | Bingo http://cu010-trove.codethink.com/cgi-bin/cgit.cgi/cu010-trove/br6/orchestration.git/ | 15:08 |
| paulsherwood | wdutch: what fileserver? | 15:11 |
| paulsherwood | if you mean cache server, no it won't imo | 15:12 |
| paulsherwood | (ie serving artifacts from builds, tests etc) | 15:12 |
| wdutch | paulsherwood: yes that's what I meant | 15:13 |
| paulsherwood | wdutch: Kinnison, richard_maw or others may disagree, but i'm pretty sure it needs to deal with atomicity of inbound content submissions, plus scale to potentially lots (100s?) of simultaneous requests for potentially large artifacts in a non-blocking way | 15:15 |
| paulsherwood | my starter-for-ten is based on bottle: https://github.com/devcurmudgeon/ybd/blob/master/kbas.py | 15:15 |
| paulsherwood | it handles atomic, and i believe it can easily be configured to scale (using cherrypy or gevent) but that will require testing | 15:16 |
| paulsherwood | also it already showed up a weakness in ybd's sandbox.install which i'm looking at | 15:17 |
| rjek | This strikes me as something that could be a CGI and just scale easily | 15:17 |
| paulsherwood | rjek: you might be right, i'm in unknown waters here | 15:18 |
| paulsherwood | rjek: however, i have a use-case for ybd where jrandom developer would like to turn his/her machine into a temporary cache server | 15:19 |
| rjek | Nod | 15:19 |
| paulsherwood | and another where a group of ybd users cause their machines to act together sharing caches | 15:19 |
| paulsherwood | so triggering some python seemed easiest to me for my first attempt | 15:20 |
| * paulsherwood wonders how pdar's build got on | 15:20 | |
| richard_maw | paulsherwood: have it announce its cache server over mdns! | 15:22 |
| * richard_maw ducks | 15:22 | |
| Kinnison | richard_maw: you can duck, but you can't hide, your avahi is advertising your presence | 15:23 |
| Walkerdine_ | I'm happy I work right next to a microcenter so I can get another jetson without hassle | 15:23 |
| richard_maw | hahahaha | 15:23 |
| paulsherwood | richard_maw: what is mdns? | 15:30 |
| richard_maw | multicast dns, name resolving without a central server | 15:32 |
| richard_maw | you can also advertise services through it | 15:33 |
| paulsherwood | richard_maw: is this a realistic possibility, or a friday wind-up? :) | 15:36 |
| * paulsherwood genuinely doesn't kow | 15:36 | |
| paulsherwood | know | 15:36 |
| richard_maw | mostly a wind-up, hence ducking after suggesting it | 15:37 |
| richard_maw | it's used a bit in nodejs communities | 15:37 |
| paulsherwood | richard_maw: 'mostly' suggests there might be something in this idea? | 15:37 |
| paulsherwood | it sounds a bit like the spartacus protocol idea, for real? | 15:38 |
| * paulsherwood notes that the 'herd of ybd' idea was ridiculous... until it worked :) | 15:39 | |
| Kinnison | paulsherwood: long ago I mooted using mdns or similar to advertise for distributed build stuff | 15:39 |
| paulsherwood | Kinnison: ok. what happened to the mooting? | 15:39 |
| Kinnison | paulsherwood: it's still ridiculous, it's just that daft things sometimes have useful behavioursin the absence of better things | 15:39 |
| Kinnison | paulsherwood: time | 15:39 |
| Kinnison | paulsherwood: I was in Korea at the time, if that helps you date it | 15:40 |
| paulsherwood | ok, shall we re-moot it? | 15:40 |
| Kinnison | Not right now, no | 15:41 |
| richard_maw | you pretty much need it to be a trusted network to safely use it, which limits its usefulness | 15:42 |
| Kinnison | indeed | 15:42 |
| paulsherwood | ok | 15:44 |
| richard_maw | if you had a bunch of build boxes you could pre-configure to advertise on a physical ethernet port then you could make them plug and play with each other, but you wouldn't want to advertise even on your home or office network really | 15:44 |
| paulsherwood | k | 15:46 |
| richard_maw | which is one of the reasons why Windows 10's offering of software updates via a similar mechanism is scary | 15:46 |
| richard_maw | though if the updates are appropriately signed, it's less of a risk | 15:47 |
| paulsherwood | which brings us (potentially) to signing artifacts etc | 15:48 |
| richard_maw | potentially, though that would only buy you sharing of pre-signed artifacts, you'd have to securely build your web of trust between your builders if you wanted to do that | 15:48 |
| richard_maw | which just moves the authentication problem out of band | 15:49 |
| paulsherwood | ack | 15:49 |
| Walkerdine_ | speaking of artifacts once I finally get my first build working can I just copy them to use for another jetson? | 15:50 |
| paulsherwood | yup | 15:50 |
| Walkerdine_ | Awesome | 15:50 |
| pdar | paulsherwood: heya, the build worked fine in the end, built a build-system in 1hr40mins. speedy! | 15:52 |
| Kinnison | that's surprisingly long compared with what paul suggested it might take | 15:52 |
| paulsherwood | which system? | 15:52 |
| pdar | build one i think | 15:52 |
| paulsherwood | oh, build-system. yup, that seems slow | 15:52 |
| pdar | ill check | 15:52 |
| paulsherwood | pdar: can you confirm it's a mx.10xlarge machine? | 15:53 |
| paulsherwood | oh, also that includes probably up to 30 mins loading gits | 15:54 |
| pdar | yep, twas the build system, it did spend a bunch of time getting gits | 15:55 |
| Kinnison | aah yeah, getting the gits will slow it down | 15:57 |
| paulsherwood | i did think about parallelising that, too, but decided it's a once-off load for most use-cases | 15:59 |
| *** CTtpollard has quit IRC | 16:25 | |
| *** jonathanmaw has quit IRC | 16:25 | |
| *** franred has quit IRC | 16:43 | |
| *** zoli__ has quit IRC | 16:44 | |
| *** zoli__ has joined #baserock | 16:44 | |
| *** Walkerdine_ has quit IRC | 16:48 | |
| jjardon | SotK: hi, can you abandon https://gerrit.baserock.org/#/c/251/ and https://gerrit.baserock.org/#/c/252/ now that xfce systems got fixed? :) | 16:58 |
| *** toscalix has quit IRC | 17:05 | |
| *** mdunford has quit IRC | 17:25 | |
| paulsherwood | jjardon: for 'fixed' do you mean they work now? | 17:28 |
| jjardon | paulsherwood: in a vm yes, in real hardware not yet because there is a problem with the logind configuration that doesn't allow the Intel driver access the hardware. Didn't have time to look deeper yet | 17:36 |
| paulsherwood | well, that's still great news :) | 17:36 |
| pedroalvarez | jjardon: :) | 17:38 |
| pedroalvarez | oh, you have improved brpaste! :D | 17:39 |
| jjardon | pedroalvarez: only a little; next step python3 by default in baserock systems ;) | 17:42 |
| jjardon | (Will send that as a RFC when is ready) | 17:42 |
| *** edcragg has quit IRC | 17:45 | |
| *** Walkerdine_ has joined #baserock | 18:37 | |
| *** bfletcher has quit IRC | 20:16 | |
| *** zoli__ has quit IRC | 20:16 | |
| *** Walkerdine_ has quit IRC | 20:51 | |
| *** bfletcher has joined #baserock | 21:03 | |
| *** Walkerdine_ has joined #baserock | 23:09 | |
| *** Walkerdine_ has quit IRC | 23:35 | |
| *** Walkerdine_ has joined #baserock | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!