IRC logs for #buildstream for Tuesday, 2018-01-09

persialuc14n0: setup.py has most of them: there are some optional ones (like you might want to have bzr installed when using the bzr source plugin): some have been added to docs, but I don't know if anyone did an exhausive search of everything that might be called.01:57
luc14n0persia: yeah, I see some install_requires in setup.py. Only the strictly needed for running bst is a good start for now. The optional ones can be added over time.02:06
persiaThere's some guides for Debian and Fedora in doc/source/install.rst : I don't see anything obvious there that isn't in setup.py02:07
persiaIf you find something else that is needed (beyond the source plugin-specific things), I suspect a bug would be appreciated.02:07
luc14n0To make the package build, all I needed was bubblewrap, python 3, pygobject GI, setuptools(_scm), pytest-runner, ostree GI. Nothing beyond of what was pointed out in the docs.02:12
luc14n0I see openSUSE is missing pytest-datafiles, pytest-env and pytest-pep8 packages for the tests.02:17
*** mcatanzaro has quit IRC04:00
*** juergbi has quit IRC07:05
*** laurenceurhegyi has quit IRC07:05
*** tristan has quit IRC07:05
*** saxa has quit IRC07:05
*** ptomato[m] has quit IRC07:05
*** waltervargas[m] has quit IRC07:05
*** ernestask[m] has quit IRC07:05
*** m_22[m] has quit IRC07:05
*** pro[m] has quit IRC07:05
*** mrmcq2u[m] has quit IRC07:05
*** cgmcintyre[m] has quit IRC07:05
*** mattiasb has quit IRC07:05
*** kailueke[m] has quit IRC07:05
*** csoriano has quit IRC07:05
*** ironfoot has quit IRC07:05
*** jjardon[m] has quit IRC07:05
*** bochecha has quit IRC07:05
*** luc14n0 has quit IRC07:05
*** tlater has quit IRC07:05
*** jude has quit IRC07:05
*** tiago has quit IRC07:05
*** gitlab-br-bot has quit IRC07:05
*** lantw44 has quit IRC07:05
*** luc14n0 has joined #buildstream07:06
*** tlater has joined #buildstream07:06
*** jude has joined #buildstream07:06
*** tiago has joined #buildstream07:06
*** ernestask[m] has joined #buildstream07:06
*** saxa has joined #buildstream07:06
*** ptomato[m] has joined #buildstream07:06
*** waltervargas[m] has joined #buildstream07:06
*** m_22[m] has joined #buildstream07:06
*** pro[m] has joined #buildstream07:06
*** mrmcq2u[m] has joined #buildstream07:06
*** cgmcintyre[m] has joined #buildstream07:06
*** mattiasb has joined #buildstream07:06
*** kailueke[m] has joined #buildstream07:06
*** gitlab-br-bot has joined #buildstream07:06
*** bochecha has joined #buildstream07:06
*** lantw44 has joined #buildstream07:06
*** csoriano has joined #buildstream07:06
*** jjardon[m] has joined #buildstream07:06
*** ironfoot has joined #buildstream07:06
*** irc.gimp.ca sets mode: +o ironfoot07:06
*** juergbi has joined #buildstream07:11
*** laurenceurhegyi has joined #buildstream07:12
*** tristan has joined #buildstream07:16
*** laurenceurhegyi is now known as ltu07:36
*** tristan has quit IRC08:11
*** nexus has joined #buildstream08:35
*** tristan has joined #buildstream08:40
tlaterHm, I'm getting an error running our pip plugin09:03
tlaterI get 'no such option `--prefix`'09:04
tlaterWhich, indeed, it doesn't seem to have09:05
tlaterThis is pip version 9.0.109:05
tlaterWhich is from 2016, so no regression there09:06
tlaterOh, nevermind09:06
tlaterLooks like we just have an '=' where there should be nothing?09:07
tlaterAh, our docker image has a very old version of pip09:11
*** noisecell has joined #buildstream09:12
tlaterNo, it's actually the flatpak sdk09:14
tlaterUgh, can't fix that09:14
tlatertristan: ^ The gnome sdk that I'm using for the integration tests now appears to have an ancient pip, which isn't supported by our pip plugin... I guess our options are adding support for it or wrestling upstream?09:19
tlaterAlternatively the non-sdk runtime probably has a more up-to-date version, otherwise this would never have worked, but using that adds more IO again09:20
*** jonathanmaw has joined #buildstream09:25
tristanhehehe09:30
tristantlater, the pip plugin is based on looking at modern builtin rpm macros and dpkg helper scripts09:31
tristanit basically does what mainstream distros are doing today to install python libs/progs as packages on the host09:32
tristanoh no, that is setuptools, sorry09:32
tristantlater, pip is a tricky thing, it's possible that we should have never added that to buildstream proper and left it in bst-external :-/09:33
tristanI think sander added that09:33
* tlater thinks bst-external wasn't about at that time yet09:33
tristanyeah, maybe should have been moved09:33
tristanbut, we do have a test case for it09:34
tlaterYup, I'm migrating that right now, hence I ran into that09:34
tristantlater, maybe if I read your mind.... I will find out that you upgraded the sdk in the test cases09:34
tlaterYes, exactly09:34
tristanso, what sdk is that ?09:35
tristanfrom flatpak ?09:35
tlaterIt's the gnome-sdk we used to use09:35
tlaterThat we merged with the "normal" runtime09:35
tlaterI just removed the "normal" runtime09:35
tristanright, and upgraded, has a broken pip ? (I wonder why the hell it ever had pip to begin with... but eh...)09:35
tlaterYes, it seems that the sdk has pip 6.x and the "normal" runtime has something more modern09:36
tristanI dont understand what "merged" means at all in this context09:36
tristandid we change the ref or not ?09:36
tlaterNo, I think we used to stage them on top of each other?09:36
tristanIn master we are doing something entirely incorrect, yes.09:37
tlaterI.e. "merge" = whatever you thought was necessary to make this work originally09:37
tlaterBut turned out to be wrong09:37
tristanright, this is not taking two runtimes, squashing them together somehow, and getting a result09:37
tristantlater, it's worth noting that the pip test is the one which required the newer ref09:37
tristani.e. not all the sdk refs were pointing to the same version09:38
tristanwe added pip, and it required a new ref09:38
tlaterAh09:38
tristanmake them all use the new ref equally, should work09:38
tlaterCool, ta, let's see what that translates to09:39
tlaterI'm pretty sure we only used the new ref in the other runtime09:39
tristanAs a side note; we dont currently have a documentation strategy (let alone any preflighting of the runtime) for advertizing the requirements plugins have on their runtimes09:40
tristanWould be nice for example, to have the autotools plugin bail out before anything is run, complaining that the provided runtime does not provide a satisfactory version of automake and autoconf and friends09:41
tlaterAh, that would be very useful for the image plugin, too09:41
tristan(after staging the runtime of course, but before assemble()09:41
tristan)09:41
tristantlater, I'm pretty damn sure the pip test uses a different ref; watching the CI when the cache is not downloadable, it ostree fetches again at pip09:43
tristanmore I couldnt say, but it's certainly different09:43
* tlater double checks09:43
tlaterOh, right09:43
tlaterIt even has a different branch09:43
tlaterWonder what happens if I plug that in here09:44
tristanpip = fa0dbd1b1eee9ec89518c1938c89803e0c54a12cd7ce892082433ad56b8a6f9b09:44
tristanautotools = 0d9d255d56b08aeaaffb1c820eef85266eb730cb5667e50681185ccf5cd7c88209:44
tlaterYeah09:44
tristantlater, make everything 1.609:44
tristanit'll all keep working09:44
* tlater is already running the tests again :)09:45
tristanof course, if the test cases are still "expecting bit for bit identical output", they will still break09:45
tristandue to new compiler and host stuff09:45
* tlater has changed that already09:45
tristan:)09:45
* tristan looking very forward to this test revolution :D09:45
tlaterIt even uses pytest.parametrize!09:46
tristantlater, so curiously, what did you do with tests/testutils ?09:46
tristanin order to share it with integration tests ?09:46
tlaterMost of those utils just work perfectly fine, had to change almost nothing09:47
tlaterWhat I changed is how the config works, so that individual bits can be modified09:47
tristantlater, I mean, how do you include them ?09:47
tristanif they are floating in some random dir ?09:47
tlaterNah, they just get imported from tests.testutils09:47
tristandidnt move it all to a new location  ?09:47
tristanah I see09:47
tristanOk no worry then, I wont add to your plate :D09:48
tlaterHeh, I think there will be some discussion on where best to keep certain things once I'm done with the bulk work09:48
tlaterBut for now "it works"09:48
tristanyeah we can land the refactor first09:48
tristanthen as a separate activity, I want to put that module into buildstream09:48
tristan(but notice that it will be tricky mostly because the "Repo" implementations for sources should not be there but rather added by buildstream's test suite)09:50
tristanHmm, anyway will have to think more on that part; maybe it would imply weird dependencies if we did that, but will have to figure a way to share that code with external plugin repos09:51
tlaterI think the repo stuff is actually quite useful in non-source tests too, and external plugins might want to implement the repo class for their tests.09:52
*** ssam2 has joined #buildstream09:59
tristantlater, interesting - what I meant was to export the Repo class and mechanics, but not the concrete classes10:02
tristantlater, more than that; it would be interesting to be able to run buildstream's frontend test battery on a different list of Source / Repo implementations (in an external repo)10:03
tristanNow, the more I think about an "I dont care about the ref" workflow, the more I think about mcatanzaro's late night suggestion about caching source refs separately10:09
tristanA bit of a disturbing thought10:09
ssam2disturbing why ?10:10
tristanIf we're going to allow any form of refless workflow, though; we are gonna need caching of local state10:10
tristanTo be honest, one of the things I like more about using buildstream, is that it forces me to be aware that "a build is not a vague mess of symbolic branches"10:11
tristanI'm not allowed to get away with that naive line of thinking10:11
ssam2me too10:11
tristanI always have to know, that I am using exactly this commit10:11
ssam2i can see an argument for not having automated modifications to the project files, though10:12
tristanTo add caching of stuff behind the scenes, to cater to a more relaxed workflow is a bit disturbing10:12
ssam2how hard would it be to optionally have refs in a separate 'manifest' file ?10:12
ssam2not necessarily 'behind the scenes', it could be in the project dir10:12
tristanIt will essentially be the same except easier to manually nuke10:13
tristani.e. buildstream still modifies the project, which I think is correct10:13
tristanonly the format of the project becomes disconnected, which is a downside10:13
tristanI'm rather more fond of hiding it in `.bst/` along with the workspaces10:14
tristan(at least, I think I'm more fond of that right now...)10:14
tristanreason I'm thinking of this now, is looking at the newcommers wiki page; I can see that the workflow will quickly break down when using track without --track-save10:15
tristan(bst build --track that is)10:15
tristanNot only will it fail for opening a workspace without ever storing a ref, the worst part is what happens when you want to launch a shell and test what you built ?10:16
tristanYou have to know what it is that you built10:16
tristanMaybe that's more what this is about ? "what is it that I just built"10:16
*** benbrown has joined #buildstream10:22
ssam2hmm, yeah that's a big deal10:22
ssam2so we can't just modify the wiki page10:22
ssam2i mean, shell could have a 'use the last thing i built' option ... but its kind of reinforcing bad habits10:23
gitlab-br-botbuildstream: merge request (jjardon/fuse->master: doc/source/install.rst: BuildStream depends on 'fuse' (for fusermount) and libfuse) #214 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/21410:23
ssam2as i understood it the main argument against the current `bst track` workflow was that it gets too confusing for newcomers10:24
ssam2which I can kind of see; if you have 'bst track' stuff mixed in with changes coming from upstream via 'git pull' *and* stuff you did yourself in a text editor ... it's a mess to figure out how to resolve all that (especially once you mix in git's byzantine user interface)10:25
tristanNah, it a bit unwieldy yeah10:25
tristanmostly because of git pull10:25
tristanssam2, using a manifest which the user can manipulate and store at will is starting to sound more and more attractive10:27
tristanI think we'll have to sleep on it a few times before the best solution presents itself10:27
*** paulsherwood has joined #buildstream10:28
tristanjuergbi, soooo, related to the above, I wonder; what happens when you build a subproject which lacks refs ?11:03
tristanjuergbi, and - sorry to cause your brain to explode :)11:04
juergbideps in subprojects are handled the same way as in the main project11:04
juergbiactually, that can't really work, hm11:04
tristanrefs11:04
juergbiit handles things the same way but updating refs in files can't work via git checkouts, of course11:05
tristaninvalid instruction11:06
tristan:)11:06
tristansigh, ok well, the good news I guess is it seems we've identified the fire11:06
tristanjuergbi, fwiw my line of thinking was that with such a side manifest as ssam2 suggests, we might also work around that issue11:09
juergbiyes, could make sense11:11
tlaterHuh, looks like pytest actually has fixtures to enable capturing stdout/err, we don't need to hack around with internals12:18
tristantlater, it does have fixtures yes, if you can figure out how to wire those in behind our own fixtures I'd be happy to change it13:00
tristantlater, I just couldnt find docs on how to use the fixture without ya know, using it "as a fixture"13:00
tlaterYeah, I looked at that, it doesn't seem like we're using pytest the way it's meant to be used13:01
tristanno no, dont worry, we're certainly not doing that :D13:01
tristanwe're horrible individuals.13:01
tlater;P13:02
* tlater gave up on the fixtures since it requires too much odd wiring13:03
tristanthe fixtures as I recall are too rigid for few reasons13:03
tristanthere is no way to turn capturing *on*, only off13:03
tristanwhich is weird13:03
tlaterOh, apparently there's a way to turn them off13:03
tristanor capture output of several runs of bst separately13:03
tlaterIt's just not documented13:03
tristanturn off, not on13:03
tristani.e. what we want, is to "capture this run"13:03
tristanseveral times13:03
tristanover the course of a test13:03
tristanseparately13:03
tlaterHm, that would just be invoking the .readouterr() of the object you get repeatedly, I think, since it flushes what it has collected so far13:04
tlaterAnd you can turn it off tempoarily with a "with"13:04
tlaterEither way, using it involves actually using pytest fixtures13:05
tlaterWhich we kind of don't x)13:05
tristantlater, if we can have cli inherit from the fixture we can do it, I think13:05
tristanfixtures can inherit that way13:06
tlaterHm, I might look at that13:06
tlaterI'm a  bit stuck because of the input redirection, I'd appreciate being able to turn things off13:06
tristanstill, not sure it will work13:06
tristanI tried looking at the API and seemed unsuitable13:06
tristantlater, why would you want to turn it off ?13:07
tristanohhhh13:07
tlaterIt turns out that the io redirection issue I had a few weeks ago is triggered by pytest's own capturing13:07
tristanbecause of integration tests and desire to see it13:07
tlaterYeah, I might need to swap between different modes on/off13:08
tristanI have no idea what that is, always assume I have no idea :)13:08
tlaterIt's a bit hard to tell you exactly what I'm running into in a sentence or two x)13:08
tristanbut if it's needed, we can probably do it without using the "proper pytest API" anyway13:09
tlaterYeah, I tried looking for it13:09
tlaterBut it seems that pytest stores things in an object associated to a test function13:09
tlaterThat you can only access through fixture API13:09
tristanwell, we are invoking the capturing stuff right13:09
tlaterAnd some of that stuff handles the capturing stuff13:09
tristanso, we can always just decide not to, right ?13:09
tlaterYes, but I need to access the capturing that pytest itself does13:09
tristanummm, that's another story I think13:10
tristanI dont know if that'll happen or is possible :-S13:10
*** xjuan has joined #buildstream13:10
tlaterIt's possible with their fixtures ;P13:10
tlaterOr a CLI flag13:10
tlaterI might need to add the no-capture CLI flag for some tests13:10
tlaterThe problem in a nutshell is that subprocess.Popen calls stdout.fileno()13:11
tristanok so... at this point I'm in the dark, if I knew very specifically what was going wrong...13:11
tlaterBut the capture objects by pytest don't have a .fileno()13:11
tlaterWe fixed that previously with fdcapture in our capture hack13:12
tlaterBut it looks like pytest swaps between them in some cases13:12
tlaterSo now their capturing is interfering with my tests13:12
tristan...because of a new version of pytest ?13:12
tlaterNot sure, I think because new dev machine13:13
tlaterHm, I guess I could confirm by running this with the various cli flags13:14
tristanHmmm, sounds like you have to figure out how to properly run a test in pytest where the python being tested expects stdout.fileno()13:14
tristanand that seems orthogonal to our using the capture stuff, but maybe not13:14
tlaterTurns out, the answer to that generally is "Don't test Popen"13:14
tristanMhmmmm, sounds like the stock answer for anything python13:15
tristan"Dont do that"13:15
tlaterWell, either way, with a bit of tricking pytest it seems possible, so I'll just keep mucking with it for now13:15
*** tristan has quit IRC13:37
*** tristan has joined #buildstream13:52
ltuas a reminder to anyone who's here for the monthly team meeting, it's about to kick off in 4 minutes over on #buildstream-meetings13:56
* tristan was just looking at the emails to try to remember if there was an s or not :)13:57
benbrown.wg 2414:08
* benbrown skulks away14:09
*** mcatanzaro has joined #buildstream14:12
*** luc14n0 has quit IRC14:14
*** luc14n0 has joined #buildstream14:14
*** xjuan has quit IRC14:35
tlaterMaybe we should create a buildstream-meeting channel whose topic is simply 'No, go to #buildstream-meetings'?14:57
*** ernestask has joined #buildstream14:59
tristanheh, did you go there ?15:00
tristantlater, in fact we can actually get ChanServ to do a redirect15:01
tristanhave to just register both channels15:01
tlaterProbably not the worst idea in the world ;P15:01
tlaterOn the other hand, maybe too much effort15:01
tristanyeah15:02
tristanand xjuan is gone15:02
tristanHe did exactly that yesterday, so he's probably the one person on the planet who doesnt have to waste time researching how to do it15:02
tlaterAlright, so after plenty of debugging, this is the only thing I can come up with to solve IO redirection: https://gitlab.com/BuildStream/buildstream/commit/adb9121bdd33074bcc48131fd9c885e57f6cc00415:29
* tlater thinks Popen only checks stdin.fileno(), and not stdout/stderr.fileno()15:29
tlaterHence forcing stdin (which we don't use anyway) to be a temporary file when we test a buildstream command seems fine15:30
*** mcatanzaro has quit IRC15:30
tlaterNot the cleanest solution though... tristan, if you're there, how much do you mind that?15:30
*** mcatanzaro has joined #buildstream15:30
tristantlater, try os.devnull15:34
tlaterAh, right, I was looking for that15:34
tlaterta15:34
jjardon[m]Hi, I'm trying to add a aarch64 buildstream image at https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/15 but Im getting an error about docker not running; Im using stable-dind docker image so not sure what could be the problem? I'd expect docker service to be stated at boot?16:23
ssam2boot of what?16:23
ssam2this is running in a container, so it doesn't boot16:23
ssam2but the .gitlab-ci.yml file should start the docker daemon in the before_script16:24
ssam2maybe that's broken and there's a race16:24
ssam2docker-in-docker is a bit of a hack16:24
ssam2from the log, looks as if the docker daemon is crashing16:24
ssam2ime="2018-01-08T19:08:45Z" level=info msg="changing OOM score to -500" module=containerd16:24
ssam2containerd: write /proc/23/oom_score_adj: permission denied16:24
ssam2time="2018-01-08T19:08:45.560162484Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd16:24
ssam2presumably that works on the x86_64 runners, but not on the aarch64 one16:25
ssam2not sure why without digging a lot further though16:25
ssam2or, it might be possible to tell docker not to try and adjust its oom score16:25
ssam2there's a --oom-score-adjust option, but it doesn't say in --help whether or not you can specify 'don't adjust it' as an option16:27
ssam2maybe passing 0 would work16:27
*** xjuan has joined #buildstream16:46
*** noisecell has quit IRC17:08
jjardon[m]ssam2: ok, yeah seems something specific of the aarch64 image as is should be exactly the same as the x86_64 one. Thanks for the hints, I will try with that17:17
*** luc14n0 has quit IRC17:46
*** luc14n0 has joined #buildstream17:46
*** jude has quit IRC17:50
gitlab-br-botbuildstream: merge request (sam/local-source-cachekey-fix->master: utils.py: Sort the results of list_relative_paths()) #216 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/21617:50
*** ssam2 has quit IRC18:02
*** jonathanmaw has quit IRC18:03
*** valentind has joined #buildstream18:05
*** xjuan has quit IRC20:12
*** valentind has quit IRC20:14
*** valentind has joined #buildstream20:51
*** mcatanzaro has quit IRC21:59
*** ernestask has quit IRC22:25
*** tristan has quit IRC22:47
*** valentind has quit IRC22:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!