IRC logs for #cip for Monday, 2019-04-01

« Sunday, 2019-03-31 Index Tuesday, 2019-04-02 »
szlinpatersonc: ping03:30
szlinpatersonc: could you grant me the permission in patchwork?03:30
*** robert__ has joined #cip06:58
*** toscalix has joined #cip07:49
patersoncszlin: Hello. What is your username on patchwork?08:11
patersoncAlso, do you know Pavels?08:11
szlinpatersonc: "szlin"08:16
szlinpatersonc: I'll ask him.08:17
patersoncThanks08:25
patersoncszlin: Do you know Pavel's email address?08:38
toscalixpatersonc: "pavel+cip@denx.de is preffered email address for mailing lists. (FYI - Mailman does not accept the “+”, so added with pavel@denx.de to cip-members - Neal)"08:47
toscalixextracted from the last TSC meeting minutes08:47
toscalix<pavel@ucw.cz> seems to work too08:49
*** tpollard has joined #cip08:58
patersoncThanks09:10
*** robert__ has quit IRC10:41
*** robert__ has joined #cip12:22
mungaip[m]rynofinn: Are you around?12:43
rynofinnyes, hello12:43
mungaip[m]rynofinn: Hello, could you set up the logs for the kernelci backend, and I'll run a test job12:44
rynofinnok, one moment12:44
mungaip[m]Thanks12:44
rynofinnok, I am following the kernelci_backend logs now12:46
mungaip[m]Great, just a moment12:47
mungaip[m]I've just started a job that should work https://lava.ciplatform.org/scheduler/job/100812:49
rynofinnI don't see any new logs.  Are we expecting it to fail when the job completes?12:50
rynofinnhere we go: [ WARNING/ Thread-28] Token not authorized for IP address 192.168.170.6412:51
rynofinn[ WARNING/MainThread] 403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=2&status_string=complete (192.168.170.64) 1.87ms12:51
mungaip[m]Okay, good I just realised that was using the wrong token12:51
mungaip[m]I'll try again with the one you sent most recently12:51
rynofinncool12:51
rynofinnToken not authorized for IP address 192.168.168.3412:52
rynofinn403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=4&status_string=canceled (192.168.168.34) 1.84ms12:52
rynofinnsome success logs going by now12:53
mungaip[m]But still nothing on the webpage https://kernelci.ciplatform.org/test/12:53
rynofinnI presume those are not interesting12:53
mungaip[m]No they're helpful12:54
mungaip[m]I am fully running another job though12:54
rynofinnok, I'll copy here12:54
rynofinnfirst another error:12:55
rynofinnWARNING/  Thread-1] Provided token (<models.token.Token object at 0x7fe89864a5d0>) is not associated with lab 'lab-cip-renesas' or is not valid12:55
rynofinn[ WARNING/MainThread] 403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=2&status_string=complete (192.168.168.34) 5.64ms12:55
mungaip[m]Was that just now?12:56
rynofinnyes12:56
rynofinnI just sent you a fuller section of the log by email12:56
rynofinnso you can see the order12:56
mungaip[m]Thanks I'm just looking through it now12:56
rynofinntoo bad there are no timestamps in there12:57
mungaip[m]Do you know who's IP address this is: 192.168.168.34?12:59
mungaip[m]Actually scratch that, it doesn't matter13:00
mungaip[m][ WARNING/  Thread-1] Provided token (<models.token.Token object at13:00
mungaip[m]0x7fe89864a5d0>) is not associated with lab 'lab-cip-renesas' or is not13:00
mungaip[m]valid13:00
mungaip[m]That's the log that's of most interest13:01
mungaip[m]rynofinn: can you rerun './kci list_tokens --host cip' and send the log via email?13:03
rynofinnyes, one moment13:03
rynofinnok, email sent13:05
mungaip[m]Thanks13:05
mungaip[m]rynofinn: let me run one more job, and could you send me the kernelci backend log? Either here or email13:12
rynofinnok, just a moment.  I have to log back in13:13
rynofinnI get timed out for inactivity13:13
mungaip[m]ok thanks13:13
rynofinnok, following now13:13
mungaip[m]The job's just running now https://lava.ciplatform.org/scheduler/job/101113:15
rynofinnhmm, I don't see errors.  did it work?13:17
mungaip[m]The job finished13:18
mungaip[m]Are there any POST logs from it though?13:18
rynofinnyes, just this:13:18
rynofinn WARNING/ Thread-22] Token not authorized for IP address 192.168.168.3413:19
rynofinn[ WARNING/MainThread] 403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=2&status_string=complete (192.168.168.34) 2.06ms13:19
mungaip[m]Do you know who has that IP address?13:20
rynofinnhahah, stack overflow is having an april fools day joke.13:21
mungaip[m]Haha yeah I saw that earlier today!13:22
rynofinnwell that IP doesn't belong to any of the docker containers according to this command: docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' kernelci-docker_celery_113:24
rynofinnthe docker host is 192.168.171.71, so I think it is an aws component13:24
rynofinnI'll log in to the aws console and see13:25
rynofinnI don't see that address used anywhere in docker or aws13:31
rynofinnwould it be ok to restart the kernelci host and see if the network settles differently?13:32
mungaip[m]Sure, we're not really using it atm ;)13:32
rynofinngreat, I'll let you know when it is back up13:32
mungaip[m]Great, and then let's try another test job13:33
rynofinnmungaip[m]: ok, kernelci is updated, rebooted.  And I'm tailing the backend log.  Ready for your test13:53
rynofinnI didn't upgrade the kernelci version by the way.  Just applied OS updates13:53
mungaip[m]Ok great, the test is running: https://lava.ciplatform.org/scheduler/job/101213:54
rynofinn[ WARNING/  Thread-5] Token not authorized for IP address 192.168.170.6413:55
rynofinn[ WARNING/MainThread] 403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=2&status_string=complete (192.168.170.64) 2.40ms13:55
mungaip[m]Right, that was what I was expecting last time!13:56
mungaip[m]Any idea who has that address? I imagine it should be an aws component13:57
rynofinnyeah, I can't identify the component.  But AWS has defined that subnet as part of the us-west-2a availability zone13:58
rynofinnthe traffic for that instance is going through a load balancer, so I think that must be it13:59
mungaip[m]Hmm, I wonder why the request is being directed to it in the first place, surely it'll just redirect the request to the relevant docker container14:02
rynofinnyes, I expected all the IPs in the backend log to be managed by docker14:03
rynofinnall the IPs listed for the containers look like this:14:04
rynofinn[root@ip-192-168-171-71 ~]# docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' kernelci-docker_backend_114:05
rynofinn172.19.0.214:05
rynofinnthe routing table for the docker host looks like this:14:06
rynofinn[root@ip-192-168-171-71 ~]# ip r14:07
rynofinndefault via 192.168.171.1 dev eth014:07
rynofinn172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.114:07
rynofinn172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.114:07
rynofinn172.19.0.0/16 dev br-e53965b59f42 proto kernel scope link src 172.19.0.114:07
rynofinn192.168.171.0/24 dev eth0 proto kernel scope link src 192.168.171.7114:07
mungaip[m]What version of kernelci-docker are you running? ie commit SHA14:08
rynofinn[root@ip-192-168-171-71 kernelci-docker]# git log -114:09
rynofinncommit 97ed451878be3326e38486a94e3c9fd639fd2f2814:09
rynofinnMerge: b9db39c 22a95aa14:09
rynofinnAuthor: Luc <luc.juggery@gmail.com>14:09
rynofinnDate:   Fri Sep 7 21:57:21 2018 +020014:09
rynofinnThe version running for AGL is dated Oct 15, 2018.  Not much newer14:10
mungaip[m]Hmm, I just wanted to have a look to see if there were any obvious fixes in later versions14:13
mungaip[m]There don't seem to be any commits on October 15 2018 in https://github.com/lucj/kernelci-docker/commits/master14:14
rynofinnoh, AGL is based on this remote: originhttps://github.com/baylibre/kernelci-docker (fetch)14:15
mungaip[m]Okay, maybe there's something on the for-agl branch that might help?14:18
rynofinnI can set that up and we can give it a try.  Should I do that now?14:19
mungaip[m]Sure14:20
mungaip[m]I'll continue to look through to see if there's anything obvious that we missed14:20
rynofinnok, bringing cip kernelci down to pull from the baylibre remote instead of lucj...14:21
mungaip[m]Okay thanks14:28
rynofinnok, https://kernelci.ciplatform.org/ is back up.  Now I need to do token surgery14:31
rynofinnhmm, it looks like it still has all the tokens14:33
rynofinnI'll start tailing the logs so you can test...14:33
rynofinnok, I'm tailing the backend logs now14:34
mungaip[m]Just submitted another job: https://lava.ciplatform.org/scheduler/job/101314:34
rynofinn[ WARNING/  Thread-3] Token not authorized for IP address 192.168.170.6414:35
rynofinn[ WARNING/MainThread] 403 POST /callback/lava/test?lab_name=lab-cip-renesas&status=2&status_string=complete (192.168.170.64) 2.03ms14:35
mungaip[m]Hmm :/ I couldn't see any commits that would have obviously changed the outcome14:40
rynofinnI'm going to have to switch to another task.  I'll be around, but distracted for the next few hours.  I sent an email with current status.14:44
mungaip[m]rynofinn: Thanks for the email update, I'll ping Kevin as I think he'll have a much better idea of what to do14:44
rynofinnsounds good14:44
*** toscalix has quit IRC17:07
*** robert__ has quit IRC17:27
*** robert__ has joined #cip21:40
*** robert__ has quit IRC21:54
« Sunday, 2019-03-31 Index Tuesday, 2019-04-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!