IRC logs for #baserock for Sunday, 2017-01-29

*** ctbruce has quit IRC02:32
*** paulwaters_ has joined #baserock08:25
paulsherwoodnow i find that 'pip install -U pip' on AWS AMI actually breaks pip :/11:19
paulsherwoodalso http://paste.baserock.org/vaxohobupu shows master ybd on master definitions ci.morph appears to be broken11:24
paulsherwoodbuild also broken for master definitions with last ybd tag (16.42)11:32
paulsherwoodbuild *not* broken for last definitions tag (GENIVI-M1.0) with last ybd tag...11:55
paulsherwood(but deploy is broken... AttributeError: 'MultiFS' object has no attribute 'addfs')11:58
jjardonpaulsherwood: master of definitions build fine in the CI; any idea how can we reproduce those problems?12:56
jjardonthe MultiFS error in the latest ybf tag; thats probably because you are pulling current fs python package in your requirements.txt, wich have different api now: thats why we are using requirement.freeze.txt now, so we can reproduce build with the exac dependencies12:58
* jjardon fires a GTK+ build12:59
jjardonabout deployment: this would help to detect that earlier ... https://gitlab.com/baserock/ybd/merge_requests/26213:04
jjardonpaulsherwood: mmm, strange, gtk+2 built fine in the CI: https://gitlab.com/baserock/definitions/builds/946192713:39
paulsherwoodjjardon: does the gitlab ci build everything in ci.morph?18:14
paulsherwoodjjardon: scarily, the gtk+ you are building has a different cache-key18:28
paulsherwoodwhich suggests we have some overlapping dependencies in ci.morph18:28
paulsherwoodor some other weirdness18:28
jjardonpaulsherwood: it builds the same gtk+2;18:29
jjardonand it builds more systems, I think18:29
paulsherwoodjjardon: different. you have gtk+.c660e2a005273921bc9044f5b4337931591d8c4988caf7eae10158dc9d4fe23418:30
paulsherwoodjjardon: my build has gtk+.fcfcb178b0ee81aca5b0ce9962e896dc7ca7b81b13d3a6bf5c219484ef04f44918:30
jjardonI meant the same if you ook to definitions18:30
jjardonlook*18:30
jjardonIts exactly the same sha and build instructions to build gtk+-218:31
paulsherwoodyes. i'm saying that there must be two different possible paths to dependencies for it18:31
paulsherwood(or a bug in ybd, of course)18:31
jjardonok, let me build cluster/ci.morph in the CI and see if we can reproduce18:34
paulsherwoodwell, i can reproduce your cache-key by building just the gnome system18:35
paulsherwoodlet's go back to the beginnign18:40
paulsherwoodfor some reason, an upgrade to pip was added to install_dependencies.sh ...18:41
paulsherwoodwhen i run that on AWS, it breaks pip http://paste.fedoraproject.org/540015/15237148/18:41
jjardona modern version of pip is needed, or fs python package will not build18:43
jjardon(that's the problem you were having the other day)18:44
paulsherwoodright18:44
jjardonone solution is to not install python-pip at all, and use get-pip.py as before; but then we can break systems that already have python-pip installed18:45
paulsherwoodjjardon: were there actual examples of that breakage?18:52
jjardonso, we can revert bfb264cf872d7962a99db0db6fcd0501c139532c, but then we will not support system with a very old version of python-pip already installed with the system package manager (like debian stable)18:53
paulsherwoodmy problem is that what's been done since the 16.42 tag has definitely broken the install for me18:53
jjardonyeah, because what I said before: we install a very old version of python-pip and then it doesnt get upgrade, so the python package fs fails to install18:53
jjardonpaulsherwood: can you check if this branch makes things work again in aws? https://gitlab.com/baserock/ybd/merge_requests/29219:00
paulsherwoodwell, i guess the problem is we're running on different environments. the changes introduced since 16.42 work for the ones you use, but break both of mine.19:00
paulsherwoodjjardon: that would work, because it effectively puts install_dependencies.sh back to 16.42 - but presumably that breaks things for other setups?19:02
jjardonit would break for setups that already have pip but its not new enough, yes19:03
jjardonit will not break the ci or anything we are currently using, so I think lets go ahead with the revert so at least we do not break yours :) (please do a quick test first)19:05
jjardonpaulsherwood: ^19:05
* paulsherwood is thinking19:05
jjardonpaulsherwood: https://gitlab.com/baserock/definitions/builds/9469106 no problems building gtk+.fcfcb178b0ee81aca5b0ce9962e896dc7ca7b81b13d3a6bf5c219484ef04f44919:07
paulsherwoodgreat. so it looks like all my problems are ybd setup issues19:10
paulsherwoodjjardon: so is python:2.7-slim ubuntu, or debian?19:10
jjardonpaulsherwood: debian: https://github.com/docker-library/python/blob/master/2.7/slim/Dockerfile#L119:12
jjardonbut It installs latest pip https://github.com/docker-library/python/blob/master/2.7/slim/Dockerfile#L2219:13
jjardonpaulsherwood: we can also make the ci test debian / fedora / ubuntu, so we will detect this next time19:17
jjardonI will submit a MR19:17
paulsherwoodjjardon: i'm reluctant to revert if you're confident this works for the other distros19:19
paulsherwoodthe aws ami is clearly 'unusual'19:19
jjardonAs its now, debian stable is currently broken (because it install an old python-pip but then it doesnt upgrade it), so I'm ok with the revert19:21
jjardonthen we will add checks for several distros and we can be confident about future changes19:22
paulsherwoodjjardon: you mean that the pip -U doesn't work for debian stable?19:22
jjardoncorrect19:22
paulsherwoodare you confident that the revert won't break the ci?19:24
paulsherwoodjjardon: the revert *doesn't* fix my situation, sadly19:30
paulsherwoodhttp://paste.baserock.org/tuhavaheya19:31
jjardonyes: https://gitlab.com/baserock/ybd/merge_requests/292 can not be merged if the CI doesn't succeed19:31
jjardonpaulsherwood: oh! thats a new one! :)19:32
paulsherwoodindeed :)19:32
* paulsherwood is expert at tripping over new ways to break software19:33
paulsherwoodjjardon: i suggest cancel that MR for now19:34
paulsherwoodweirder still - the upgraded pip seems to be available for non-root, missing for root19:39
jjardonpaulsherwood: debian stable failing to upgrade pip: https://gitlab.com/baserock/ybd/builds/947058419:40
paulsherwoodack19:42
paulsherwoodi've cancelled that mr. given the odd behaviour of aws ami i'm going to switch to ubuntu from now20:09
*** lachlan has joined #baserock20:56
jjardonpaulsherwood: any idea why this is happening : https://gitlab.com/baserock/ybd/builds/9481013 (only happen when I try to build in fedora:25 image !)22:01
jjardonhttps://gitlab.com/baserock/ybd/pipelines/609244622:02
*** lachlan has quit IRC22:04
paulsherwoodno idea, but https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/139422:20
paulsherwoodok i have a better idea22:28
paulsherwoodjjardon: please retry with https://gitlab.com/baserock/ybd/tree/ps-fresh-tmpdir22:32
jjardonpaulsherwood: thanks, lets see: https://gitlab.com/baserock/ybd/pipelines/609295622:36
jjardonpaulsherwood: can you review https://gitlab.com/baserock/ybd/merge_requests/293 and https://gitlab.com/baserock/ybd/merge_requests/294 when you have a second?22:37
paulsherwoodi already approved them :)22:38
jjardonthanks! :) also https://gitlab.com/baserock/ybd/merge_requests/262 ?22:38
paulsherwoodi can't rebase that, for some reason22:40
jjardonI've done it manually22:42
jjardonI think it's a problem with the new gitlab version; rebase through the UI used to work fine22:43
paulsherwoodjjardon: https://gitlab.com/baserock/ybd/builds/9482675 looks happier :)22:44
paulsherwooduntil it fell over :/22:45
jjardonpaulsherwood: nice! seems puting a lot of distros in the ci already found a bug!22:45
jjardonpaulsherwood: seems fedora doesnt comes with the find utility by default; I will send a fix for that22:46
paulsherwoodcool :)22:47
* paulsherwood hits the sack23:04

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!