| *** noisecell has joined #baserock | 07:42 | |
| *** gtristan has joined #baserock | 07:58 | |
| *** toscalix has joined #baserock | 08:34 | |
| *** gtristan has quit IRC | 09:01 | |
| *** gtristan has joined #baserock | 09:44 | |
| jjardon | noisecell: did you noticed the cache server not working when you were doing some work recently? | 13:40 |
|---|---|---|
| flatmush | https://gitlab.com/baserock/definitions/pipelines | 13:44 |
| flatmush | The pipelines are all between 8 and 10 hours, so I'm guessing caching has not been working for a while | 13:44 |
| noisecell | jjardon, I didn't, they were flatmush's pushes | 13:45 |
| jjardon | noisecell: I meant some weeks ago: last commit in master is from you | 13:47 |
| noisecell | jjardon, no, I didn't notice it. | 13:48 |
| noisecell | jjardon, did I change anything that broke the cache? could updating bst to 1.1.3 cause to break the cache? | 13:49 |
| noisecell | I think it is the only change that could break that in my patch series | 13:49 |
| flatmush | looks like it broke around the 1.1.3 update | 13:49 |
| jjardon | _maybe_ | 13:50 |
| noisecell | jjardon, in any case 1.0.1 had the issue of the cache key calculation also definitions wasn't working for 1.0.1, if you remember | 13:50 |
| jjardon | noisecell: the fact is the commit before merging your branch was fine: https://gitlab.com/baserock/definitions/commits/master -> https://gitlab.com/baserock/definitions/commit/eea23f6a325956d17ae35eaeefc67a5f3e5f6a85/pipelines?ref=master (24 min) vs 8h now | 13:58 |
| jjardon | noisecell: can you try and revert to the upgrade to 1.1.3 and see if that's the problem? | 13:58 |
| noisecell | jjardon, you want me to send a branch pointing to -> https://gitlab.com/baserock/definitions/commit/45bb922afeaae26ef5d4e0ee329c678485063c45 ?? | 14:01 |
| jjardon | noisecell: somehow seems some of your changes make the cache to stop working: I think revert only the commit to upgrade bst would be enough but you can try and revert the whole thing if you want | 14:02 |
| jjardon | or do both: simply push the branches and see what happen in the CI | 14:03 |
| noisecell | jjardon, ok, I will send both branches | 14:03 |
| noisecell | jjardon, https://gitlab.com/baserock/definitions/pipelines/26485381 and https://gitlab.com/baserock/definitions/pipelines/26485225 | 14:08 |
| jjardon | thanks, let's see | 14:08 |
| jjardon | seems reverting the 1.1.3 upgrade did the trick: https://gitlab.com/baserock/definitions/-/jobs/84316695 | 14:12 |
| jjardon | noisecell: flatmush I guess you have two options: revert or investigate why is failing to work with 1.1.3 | 14:13 |
| noisecell | jjardon, can you check the cache? | 14:13 |
| noisecell | I mean did someone got pushed? | 14:13 |
| noisecell | because that only shows that it is getting the cache | 14:14 |
| jjardon | yeah, that the only thing we need | 14:14 |
| noisecell | as downloading the artifacts | 14:14 |
| jjardon | before is not getting anything | 14:14 |
| jjardon | before was not getting anything | 14:14 |
| noisecell | jjardon, no, we need to see if anything got pushed as well | 14:14 |
| noisecell | jjardon, can you see any artifacts pushed from master when my changes got into master? | 14:15 |
| noisecell | where we can check the cache? | 14:15 |
| jjardon | noisecell: why anything would be pushed if it's getting everyinthg from the cache> | 14:15 |
| jjardon | ? | 14:15 |
| noisecell | where is the cache? it is local? | 14:15 |
| noisecell | are the runners not separated? | 14:15 |
| jjardon | bst cache is in a server in DigitalOcean | 14:16 |
| noisecell | so the cache comes from a server? | 14:16 |
| jjardon | yes | 14:16 |
| noisecell | then, we need to see if there are artifacts in that cache | 14:16 |
| * jjardon confused | 14:16 | |
| jjardon | why would need to do that? we already proof that downgrading bst the runner manage to download the cache, right? | 14:17 |
| noisecell | jjardon, I want to check if there are any artifact in DigitalOcean server from my 1.1.3 master patches | 14:17 |
| noisecell | jjardon, to check that up is not broken | 14:17 |
| noisecell | you have only tested that download the cache is not broken not that upload the cache is doing what is should do | 14:17 |
| jjardon | ah | 14:17 |
| jjardon | noisecell: do you have a ssh public key? I can give you access | 14:19 |
| jjardon | noisecell: can you try to ssh@46.101.48.48 ? | 14:23 |
| noisecell | jjardon, permission denied ... I use ssh as user is that right? | 14:24 |
| jjardon | noisecell: no sorry! | 14:25 |
| jjardon | use root | 14:25 |
| noisecell | jjardon, do you know where this cache artifacts could be stored? | 14:26 |
| jjardon | mmm, let me see | 14:29 |
| noisecell | jjardon, seems empty to me...but Im not sure what Im looking at | 14:30 |
| * jjardon confused (again) | 14:34 | |
| jjardon | https://gitlab.com/baserock/infrastructure/blob/master/baserock_ostree/etc/systemd/system/ostree-update-summary-cache.service says it should be in /srv | 14:34 |
| noisecell | that folder is empty | 14:35 |
| noisecell | s/folder/directory/ | 14:35 |
| noisecell | jjardon, that service is not in the system | 14:38 |
| noisecell | and therefore it hasn't run ever | 14:38 |
| jjardon | noisecell: yeah, I think we are looking to the incorrect server /o\ | 14:43 |
| noisecell | heh :) | 14:43 |
| jjardon | this is the one with the gitlab cache, not bst cache XD | 14:43 |
| jjardon | ostree.baserock.org is the one! | 14:44 |
| jjardon | I do not have access to that one though; maybe ironfoot ? | 14:45 |
| noisecell | jjardon, you can navigate through the URL | 14:45 |
| noisecell | jjardon, https://ostree.baserock.org/cache/refs/heads/baserock/ -- if you look at core-python3/ for example hasn't been updated since 2017-Dec-04 | 14:46 |
| noisecell | so it seems not to be pushing the cache up | 14:47 |
| jjardon | noisecell: yup | 14:49 |
| jjardon | still strange there is no errors in the build logs | 14:49 |
| jjardon | let me recheck | 14:49 |
| noisecell | with 1.1.3 --- I think it was working for 1.0.1 | 14:49 |
| jjardon | aaaaaaaaaaaaaaaaa | 14:51 |
| jjardon | I think | 14:51 |
| jjardon | the problem can be | 14:51 |
| jjardon | that we need to update the bsst version on the srever as well | 14:51 |
| noisecell | oh! | 14:52 |
| jjardon | maybe some incompatibility | 14:52 |
| noisecell | ok | 14:52 |
| noisecell | jjardon, do you know how to do that? | 14:52 |
| jjardon | yeah, but I do not have access to that server :( | 14:52 |
| noisecell | ironfoot, ^^ ? :/ | 14:53 |
| jjardon | (It's as simply as installing buildstream in your machine: pip3 install . | 14:53 |
| jjardon | noisecell: https://gitlab.com/baserock/infrastructure/blob/master/baserock_ostree/image-config.yml#L48 | 14:54 |
| jjardon | with ansible everything is autodocumented :) | 14:54 |
| noisecell | jjardon, yeah, that's nice :) | 14:55 |
| noisecell | but... is that running from master? | 14:55 |
| noisecell | or from 1.0.1? | 14:56 |
| jjardon | flatmush: so recommendation for you: revert the upgrade to 1.1.3 for now | 14:56 |
| noisecell | I can not see the version | 14:56 |
| jjardon | seems it is https://gitlab.com/baserock/infrastructure/blob/master/baserock_ostree/image-config.yml#L43 : bf8fc373d7711861129ab841a74ecf32b3d8b2dd | 14:56 |
| ironfoot | hi all, haven't read those pings yet. Give me a few mins :) | 14:57 |
| jjardon | but better double check what is actually in the server | 14:57 |
| flatmush | jjardon: will do | 14:57 |
| ironfoot | ah, upgrading ostree? | 14:57 |
| ironfoot | i have access to that server if that's what we want | 14:57 |
| jjardon | ironfoot: no, buildstream | 14:57 |
| ironfoot | yes, sorry, upgrading bst in ostree | 14:57 |
| jjardon | (maybe you need a ostree update as well) | 14:57 |
| ironfoot | I'll look into it | 14:58 |
| ironfoot | to what version? 1.1.3? | 14:58 |
| noisecell | ironfoot, yes | 14:58 |
| noisecell | I assume at some point we will need to think about setting up the CAS cache | 14:58 |
| jjardon | yeah, we are working in the ansible for freedesktop-sdk, so we can share it when done | 14:59 |
| noisecell | jjardon, thank you | 14:59 |
| *** gtristan has quit IRC | 15:08 | |
| ironfoot | upgrading it, ansible script seems to be doing well. Where can we test the result? | 15:32 |
| *** gtristan has joined #baserock | 15:35 | |
| gitlab-br-bot | infrastructure: merge request (pedro/ostre-upgrade-bst->master: baserock_ostree: Upgrade Buildstream to version 1.1.3) #29 changed state ("opened"): https://gitlab.com/baserock/infrastructure/merge_requests/29 | 15:38 |
| ironfoot | Right, upgraded. I'd like to know if it works before merging anything | 15:40 |
| ironfoot | i guess we can see if https://gitlab.com/baserock/definitions/-/jobs/84317312 uploads anything | 15:42 |
| ironfoot | [00:00:01][a7c9a680][build:gnu-toolchain/stage1-binutils.bst] SUCCESS Caching artifact | 15:45 |
| jjardon | ironfoot: AFAIK that means locally cached, not pushed | 16:00 |
| jjardon | gtristan: ^ ? | 16:01 |
| ironfoot | yeah, probably given that it takes 0s | 16:05 |
| ironfoot | indeed, the keyword is "Pushing" the artifact | 16:17 |
| ironfoot | I can't see that happening | 16:17 |
| *** toscalix has quit IRC | 17:15 | |
| gtristan | if there is no push queue showing in the status area, only build & fetch, then BuildStream did not successfully connect to (or authenticate ?) the remote server at launch time | 17:49 |
| gtristan | at the very beginning of the log after the pipeline status (all the cache keys)... there should be an attempt to connect if things are configured to | 17:50 |
| *** gtristan has quit IRC | 21:18 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!