persia | luc14n0: setup.py has most of them: there are some optional ones (like you might want to have bzr installed when using the bzr source plugin): some have been added to docs, but I don't know if anyone did an exhausive search of everything that might be called. | 01:57 |
---|---|---|
luc14n0 | persia: yeah, I see some install_requires in setup.py. Only the strictly needed for running bst is a good start for now. The optional ones can be added over time. | 02:06 |
persia | There's some guides for Debian and Fedora in doc/source/install.rst : I don't see anything obvious there that isn't in setup.py | 02:07 |
persia | If you find something else that is needed (beyond the source plugin-specific things), I suspect a bug would be appreciated. | 02:07 |
luc14n0 | To make the package build, all I needed was bubblewrap, python 3, pygobject GI, setuptools(_scm), pytest-runner, ostree GI. Nothing beyond of what was pointed out in the docs. | 02:12 |
luc14n0 | I see openSUSE is missing pytest-datafiles, pytest-env and pytest-pep8 packages for the tests. | 02:17 |
*** mcatanzaro has quit IRC | 04:00 | |
*** juergbi has quit IRC | 07:05 | |
*** laurenceurhegyi has quit IRC | 07:05 | |
*** tristan has quit IRC | 07:05 | |
*** saxa has quit IRC | 07:05 | |
*** ptomato[m] has quit IRC | 07:05 | |
*** waltervargas[m] has quit IRC | 07:05 | |
*** ernestask[m] has quit IRC | 07:05 | |
*** m_22[m] has quit IRC | 07:05 | |
*** pro[m] has quit IRC | 07:05 | |
*** mrmcq2u[m] has quit IRC | 07:05 | |
*** cgmcintyre[m] has quit IRC | 07:05 | |
*** mattiasb has quit IRC | 07:05 | |
*** kailueke[m] has quit IRC | 07:05 | |
*** csoriano has quit IRC | 07:05 | |
*** ironfoot has quit IRC | 07:05 | |
*** jjardon[m] has quit IRC | 07:05 | |
*** bochecha has quit IRC | 07:05 | |
*** luc14n0 has quit IRC | 07:05 | |
*** tlater has quit IRC | 07:05 | |
*** jude has quit IRC | 07:05 | |
*** tiago has quit IRC | 07:05 | |
*** gitlab-br-bot has quit IRC | 07:05 | |
*** lantw44 has quit IRC | 07:05 | |
*** luc14n0 has joined #buildstream | 07:06 | |
*** tlater has joined #buildstream | 07:06 | |
*** jude has joined #buildstream | 07:06 | |
*** tiago has joined #buildstream | 07:06 | |
*** ernestask[m] has joined #buildstream | 07:06 | |
*** saxa has joined #buildstream | 07:06 | |
*** ptomato[m] has joined #buildstream | 07:06 | |
*** waltervargas[m] has joined #buildstream | 07:06 | |
*** m_22[m] has joined #buildstream | 07:06 | |
*** pro[m] has joined #buildstream | 07:06 | |
*** mrmcq2u[m] has joined #buildstream | 07:06 | |
*** cgmcintyre[m] has joined #buildstream | 07:06 | |
*** mattiasb has joined #buildstream | 07:06 | |
*** kailueke[m] has joined #buildstream | 07:06 | |
*** gitlab-br-bot has joined #buildstream | 07:06 | |
*** bochecha has joined #buildstream | 07:06 | |
*** lantw44 has joined #buildstream | 07:06 | |
*** csoriano has joined #buildstream | 07:06 | |
*** jjardon[m] has joined #buildstream | 07:06 | |
*** ironfoot has joined #buildstream | 07:06 | |
*** irc.gimp.ca sets mode: +o ironfoot | 07:06 | |
*** juergbi has joined #buildstream | 07:11 | |
*** laurenceurhegyi has joined #buildstream | 07:12 | |
*** tristan has joined #buildstream | 07:16 | |
*** laurenceurhegyi is now known as ltu | 07:36 | |
*** tristan has quit IRC | 08:11 | |
*** nexus has joined #buildstream | 08:35 | |
*** tristan has joined #buildstream | 08:40 | |
tlater | Hm, I'm getting an error running our pip plugin | 09:03 |
tlater | I get 'no such option `--prefix`' | 09:04 |
tlater | Which, indeed, it doesn't seem to have | 09:05 |
tlater | This is pip version 9.0.1 | 09:05 |
tlater | Which is from 2016, so no regression there | 09:06 |
tlater | Oh, nevermind | 09:06 |
tlater | Looks like we just have an '=' where there should be nothing? | 09:07 |
tlater | Ah, our docker image has a very old version of pip | 09:11 |
*** noisecell has joined #buildstream | 09:12 | |
tlater | No, it's actually the flatpak sdk | 09:14 |
tlater | Ugh, can't fix that | 09:14 |
tlater | tristan: ^ The gnome sdk that I'm using for the integration tests now appears to have an ancient pip, which isn't supported by our pip plugin... I guess our options are adding support for it or wrestling upstream? | 09:19 |
tlater | Alternatively the non-sdk runtime probably has a more up-to-date version, otherwise this would never have worked, but using that adds more IO again | 09:20 |
*** jonathanmaw has joined #buildstream | 09:25 | |
tristan | hehehe | 09:30 |
tristan | tlater, the pip plugin is based on looking at modern builtin rpm macros and dpkg helper scripts | 09:31 |
tristan | it basically does what mainstream distros are doing today to install python libs/progs as packages on the host | 09:32 |
tristan | oh no, that is setuptools, sorry | 09:32 |
tristan | tlater, pip is a tricky thing, it's possible that we should have never added that to buildstream proper and left it in bst-external :-/ | 09:33 |
tristan | I think sander added that | 09:33 |
* tlater thinks bst-external wasn't about at that time yet | 09:33 | |
tristan | yeah, maybe should have been moved | 09:33 |
tristan | but, we do have a test case for it | 09:34 |
tlater | Yup, I'm migrating that right now, hence I ran into that | 09:34 |
tristan | tlater, maybe if I read your mind.... I will find out that you upgraded the sdk in the test cases | 09:34 |
tlater | Yes, exactly | 09:34 |
tristan | so, what sdk is that ? | 09:35 |
tristan | from flatpak ? | 09:35 |
tlater | It's the gnome-sdk we used to use | 09:35 |
tlater | That we merged with the "normal" runtime | 09:35 |
tlater | I just removed the "normal" runtime | 09:35 |
tristan | right, and upgraded, has a broken pip ? (I wonder why the hell it ever had pip to begin with... but eh...) | 09:35 |
tlater | Yes, it seems that the sdk has pip 6.x and the "normal" runtime has something more modern | 09:36 |
tristan | I dont understand what "merged" means at all in this context | 09:36 |
tristan | did we change the ref or not ? | 09:36 |
tlater | No, I think we used to stage them on top of each other? | 09:36 |
tristan | In master we are doing something entirely incorrect, yes. | 09:37 |
tlater | I.e. "merge" = whatever you thought was necessary to make this work originally | 09:37 |
tlater | But turned out to be wrong | 09:37 |
tristan | right, this is not taking two runtimes, squashing them together somehow, and getting a result | 09:37 |
tristan | tlater, it's worth noting that the pip test is the one which required the newer ref | 09:37 |
tristan | i.e. not all the sdk refs were pointing to the same version | 09:38 |
tristan | we added pip, and it required a new ref | 09:38 |
tlater | Ah | 09:38 |
tristan | make them all use the new ref equally, should work | 09:38 |
tlater | Cool, ta, let's see what that translates to | 09:39 |
tlater | I'm pretty sure we only used the new ref in the other runtime | 09:39 |
tristan | As a side note; we dont currently have a documentation strategy (let alone any preflighting of the runtime) for advertizing the requirements plugins have on their runtimes | 09:40 |
tristan | Would be nice for example, to have the autotools plugin bail out before anything is run, complaining that the provided runtime does not provide a satisfactory version of automake and autoconf and friends | 09:41 |
tlater | Ah, that would be very useful for the image plugin, too | 09:41 |
tristan | (after staging the runtime of course, but before assemble() | 09:41 |
tristan | ) | 09:41 |
tristan | tlater, I'm pretty damn sure the pip test uses a different ref; watching the CI when the cache is not downloadable, it ostree fetches again at pip | 09:43 |
tristan | more I couldnt say, but it's certainly different | 09:43 |
* tlater double checks | 09:43 | |
tlater | Oh, right | 09:43 |
tlater | It even has a different branch | 09:43 |
tlater | Wonder what happens if I plug that in here | 09:44 |
tristan | pip = fa0dbd1b1eee9ec89518c1938c89803e0c54a12cd7ce892082433ad56b8a6f9b | 09:44 |
tristan | autotools = 0d9d255d56b08aeaaffb1c820eef85266eb730cb5667e50681185ccf5cd7c882 | 09:44 |
tlater | Yeah | 09:44 |
tristan | tlater, make everything 1.6 | 09:44 |
tristan | it'll all keep working | 09:44 |
* tlater is already running the tests again :) | 09:45 | |
tristan | of course, if the test cases are still "expecting bit for bit identical output", they will still break | 09:45 |
tristan | due to new compiler and host stuff | 09:45 |
* tlater has changed that already | 09:45 | |
tristan | :) | 09:45 |
* tristan looking very forward to this test revolution :D | 09:45 | |
tlater | It even uses pytest.parametrize! | 09:46 |
tristan | tlater, so curiously, what did you do with tests/testutils ? | 09:46 |
tristan | in order to share it with integration tests ? | 09:46 |
tlater | Most of those utils just work perfectly fine, had to change almost nothing | 09:47 |
tlater | What I changed is how the config works, so that individual bits can be modified | 09:47 |
tristan | tlater, I mean, how do you include them ? | 09:47 |
tristan | if they are floating in some random dir ? | 09:47 |
tlater | Nah, they just get imported from tests.testutils | 09:47 |
tristan | didnt move it all to a new location ? | 09:47 |
tristan | ah I see | 09:47 |
tristan | Ok no worry then, I wont add to your plate :D | 09:48 |
tlater | Heh, I think there will be some discussion on where best to keep certain things once I'm done with the bulk work | 09:48 |
tlater | But for now "it works" | 09:48 |
tristan | yeah we can land the refactor first | 09:48 |
tristan | then as a separate activity, I want to put that module into buildstream | 09:48 |
tristan | (but notice that it will be tricky mostly because the "Repo" implementations for sources should not be there but rather added by buildstream's test suite) | 09:50 |
tristan | Hmm, anyway will have to think more on that part; maybe it would imply weird dependencies if we did that, but will have to figure a way to share that code with external plugin repos | 09:51 |
tlater | I think the repo stuff is actually quite useful in non-source tests too, and external plugins might want to implement the repo class for their tests. | 09:52 |
*** ssam2 has joined #buildstream | 09:59 | |
tristan | tlater, interesting - what I meant was to export the Repo class and mechanics, but not the concrete classes | 10:02 |
tristan | tlater, more than that; it would be interesting to be able to run buildstream's frontend test battery on a different list of Source / Repo implementations (in an external repo) | 10:03 |
tristan | Now, the more I think about an "I dont care about the ref" workflow, the more I think about mcatanzaro's late night suggestion about caching source refs separately | 10:09 |
tristan | A bit of a disturbing thought | 10:09 |
ssam2 | disturbing why ? | 10:10 |
tristan | If we're going to allow any form of refless workflow, though; we are gonna need caching of local state | 10:10 |
tristan | To be honest, one of the things I like more about using buildstream, is that it forces me to be aware that "a build is not a vague mess of symbolic branches" | 10:11 |
tristan | I'm not allowed to get away with that naive line of thinking | 10:11 |
ssam2 | me too | 10:11 |
tristan | I always have to know, that I am using exactly this commit | 10:11 |
ssam2 | i can see an argument for not having automated modifications to the project files, though | 10:12 |
tristan | To add caching of stuff behind the scenes, to cater to a more relaxed workflow is a bit disturbing | 10:12 |
ssam2 | how hard would it be to optionally have refs in a separate 'manifest' file ? | 10:12 |
ssam2 | not necessarily 'behind the scenes', it could be in the project dir | 10:12 |
tristan | It will essentially be the same except easier to manually nuke | 10:13 |
tristan | i.e. buildstream still modifies the project, which I think is correct | 10:13 |
tristan | only the format of the project becomes disconnected, which is a downside | 10:13 |
tristan | I'm rather more fond of hiding it in `.bst/` along with the workspaces | 10:14 |
tristan | (at least, I think I'm more fond of that right now...) | 10:14 |
tristan | reason I'm thinking of this now, is looking at the newcommers wiki page; I can see that the workflow will quickly break down when using track without --track-save | 10:15 |
tristan | (bst build --track that is) | 10:15 |
tristan | Not only will it fail for opening a workspace without ever storing a ref, the worst part is what happens when you want to launch a shell and test what you built ? | 10:16 |
tristan | You have to know what it is that you built | 10:16 |
tristan | Maybe that's more what this is about ? "what is it that I just built" | 10:16 |
*** benbrown has joined #buildstream | 10:22 | |
ssam2 | hmm, yeah that's a big deal | 10:22 |
ssam2 | so we can't just modify the wiki page | 10:22 |
ssam2 | i mean, shell could have a 'use the last thing i built' option ... but its kind of reinforcing bad habits | 10:23 |
gitlab-br-bot | buildstream: merge request (jjardon/fuse->master: doc/source/install.rst: BuildStream depends on 'fuse' (for fusermount) and libfuse) #214 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/214 | 10:23 |
ssam2 | as i understood it the main argument against the current `bst track` workflow was that it gets too confusing for newcomers | 10:24 |
ssam2 | which I can kind of see; if you have 'bst track' stuff mixed in with changes coming from upstream via 'git pull' *and* stuff you did yourself in a text editor ... it's a mess to figure out how to resolve all that (especially once you mix in git's byzantine user interface) | 10:25 |
tristan | Nah, it a bit unwieldy yeah | 10:25 |
tristan | mostly because of git pull | 10:25 |
tristan | ssam2, using a manifest which the user can manipulate and store at will is starting to sound more and more attractive | 10:27 |
tristan | I think we'll have to sleep on it a few times before the best solution presents itself | 10:27 |
*** paulsherwood has joined #buildstream | 10:28 | |
tristan | juergbi, soooo, related to the above, I wonder; what happens when you build a subproject which lacks refs ? | 11:03 |
tristan | juergbi, and - sorry to cause your brain to explode :) | 11:04 |
juergbi | deps in subprojects are handled the same way as in the main project | 11:04 |
juergbi | actually, that can't really work, hm | 11:04 |
tristan | refs | 11:04 |
juergbi | it handles things the same way but updating refs in files can't work via git checkouts, of course | 11:05 |
tristan | invalid instruction | 11:06 |
tristan | :) | 11:06 |
tristan | sigh, ok well, the good news I guess is it seems we've identified the fire | 11:06 |
tristan | juergbi, fwiw my line of thinking was that with such a side manifest as ssam2 suggests, we might also work around that issue | 11:09 |
juergbi | yes, could make sense | 11:11 |
tlater | Huh, looks like pytest actually has fixtures to enable capturing stdout/err, we don't need to hack around with internals | 12:18 |
tristan | tlater, it does have fixtures yes, if you can figure out how to wire those in behind our own fixtures I'd be happy to change it | 13:00 |
tristan | tlater, I just couldnt find docs on how to use the fixture without ya know, using it "as a fixture" | 13:00 |
tlater | Yeah, I looked at that, it doesn't seem like we're using pytest the way it's meant to be used | 13:01 |
tristan | no no, dont worry, we're certainly not doing that :D | 13:01 |
tristan | we're horrible individuals. | 13:01 |
tlater | ;P | 13:02 |
* tlater gave up on the fixtures since it requires too much odd wiring | 13:03 | |
tristan | the fixtures as I recall are too rigid for few reasons | 13:03 |
tristan | there is no way to turn capturing *on*, only off | 13:03 |
tristan | which is weird | 13:03 |
tlater | Oh, apparently there's a way to turn them off | 13:03 |
tristan | or capture output of several runs of bst separately | 13:03 |
tlater | It's just not documented | 13:03 |
tristan | turn off, not on | 13:03 |
tristan | i.e. what we want, is to "capture this run" | 13:03 |
tristan | several times | 13:03 |
tristan | over the course of a test | 13:03 |
tristan | separately | 13:03 |
tlater | Hm, that would just be invoking the .readouterr() of the object you get repeatedly, I think, since it flushes what it has collected so far | 13:04 |
tlater | And you can turn it off tempoarily with a "with" | 13:04 |
tlater | Either way, using it involves actually using pytest fixtures | 13:05 |
tlater | Which we kind of don't x) | 13:05 |
tristan | tlater, if we can have cli inherit from the fixture we can do it, I think | 13:05 |
tristan | fixtures can inherit that way | 13:06 |
tlater | Hm, I might look at that | 13:06 |
tlater | I'm a bit stuck because of the input redirection, I'd appreciate being able to turn things off | 13:06 |
tristan | still, not sure it will work | 13:06 |
tristan | I tried looking at the API and seemed unsuitable | 13:06 |
tristan | tlater, why would you want to turn it off ? | 13:07 |
tristan | ohhhh | 13:07 |
tlater | It turns out that the io redirection issue I had a few weeks ago is triggered by pytest's own capturing | 13:07 |
tristan | because of integration tests and desire to see it | 13:07 |
tlater | Yeah, I might need to swap between different modes on/off | 13:08 |
tristan | I have no idea what that is, always assume I have no idea :) | 13:08 |
tlater | It's a bit hard to tell you exactly what I'm running into in a sentence or two x) | 13:08 |
tristan | but if it's needed, we can probably do it without using the "proper pytest API" anyway | 13:09 |
tlater | Yeah, I tried looking for it | 13:09 |
tlater | But it seems that pytest stores things in an object associated to a test function | 13:09 |
tlater | That you can only access through fixture API | 13:09 |
tristan | well, we are invoking the capturing stuff right | 13:09 |
tlater | And some of that stuff handles the capturing stuff | 13:09 |
tristan | so, we can always just decide not to, right ? | 13:09 |
tlater | Yes, but I need to access the capturing that pytest itself does | 13:09 |
tristan | ummm, that's another story I think | 13:10 |
tristan | I dont know if that'll happen or is possible :-S | 13:10 |
*** xjuan has joined #buildstream | 13:10 | |
tlater | It's possible with their fixtures ;P | 13:10 |
tlater | Or a CLI flag | 13:10 |
tlater | I might need to add the no-capture CLI flag for some tests | 13:10 |
tlater | The problem in a nutshell is that subprocess.Popen calls stdout.fileno() | 13:11 |
tristan | ok so... at this point I'm in the dark, if I knew very specifically what was going wrong... | 13:11 |
tlater | But the capture objects by pytest don't have a .fileno() | 13:11 |
tlater | We fixed that previously with fdcapture in our capture hack | 13:12 |
tlater | But it looks like pytest swaps between them in some cases | 13:12 |
tlater | So now their capturing is interfering with my tests | 13:12 |
tristan | ...because of a new version of pytest ? | 13:12 |
tlater | Not sure, I think because new dev machine | 13:13 |
tlater | Hm, I guess I could confirm by running this with the various cli flags | 13:14 |
tristan | Hmmm, sounds like you have to figure out how to properly run a test in pytest where the python being tested expects stdout.fileno() | 13:14 |
tristan | and that seems orthogonal to our using the capture stuff, but maybe not | 13:14 |
tlater | Turns out, the answer to that generally is "Don't test Popen" | 13:14 |
tristan | Mhmmmm, sounds like the stock answer for anything python | 13:15 |
tristan | "Dont do that" | 13:15 |
tlater | Well, either way, with a bit of tricking pytest it seems possible, so I'll just keep mucking with it for now | 13:15 |
*** tristan has quit IRC | 13:37 | |
*** tristan has joined #buildstream | 13:52 | |
ltu | as a reminder to anyone who's here for the monthly team meeting, it's about to kick off in 4 minutes over on #buildstream-meetings | 13:56 |
* tristan was just looking at the emails to try to remember if there was an s or not :) | 13:57 | |
benbrown | .wg 24 | 14:08 |
* benbrown skulks away | 14:09 | |
*** mcatanzaro has joined #buildstream | 14:12 | |
*** luc14n0 has quit IRC | 14:14 | |
*** luc14n0 has joined #buildstream | 14:14 | |
*** xjuan has quit IRC | 14:35 | |
tlater | Maybe we should create a buildstream-meeting channel whose topic is simply 'No, go to #buildstream-meetings'? | 14:57 |
*** ernestask has joined #buildstream | 14:59 | |
tristan | heh, did you go there ? | 15:00 |
tristan | tlater, in fact we can actually get ChanServ to do a redirect | 15:01 |
tristan | have to just register both channels | 15:01 |
tlater | Probably not the worst idea in the world ;P | 15:01 |
tlater | On the other hand, maybe too much effort | 15:01 |
tristan | yeah | 15:02 |
tristan | and xjuan is gone | 15:02 |
tristan | He did exactly that yesterday, so he's probably the one person on the planet who doesnt have to waste time researching how to do it | 15:02 |
tlater | Alright, so after plenty of debugging, this is the only thing I can come up with to solve IO redirection: https://gitlab.com/BuildStream/buildstream/commit/adb9121bdd33074bcc48131fd9c885e57f6cc004 | 15:29 |
* tlater thinks Popen only checks stdin.fileno(), and not stdout/stderr.fileno() | 15:29 | |
tlater | Hence forcing stdin (which we don't use anyway) to be a temporary file when we test a buildstream command seems fine | 15:30 |
*** mcatanzaro has quit IRC | 15:30 | |
tlater | Not the cleanest solution though... tristan, if you're there, how much do you mind that? | 15:30 |
*** mcatanzaro has joined #buildstream | 15:30 | |
tristan | tlater, try os.devnull | 15:34 |
tlater | Ah, right, I was looking for that | 15:34 |
tlater | ta | 15:34 |
jjardon[m] | Hi, I'm trying to add a aarch64 buildstream image at https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/15 but Im getting an error about docker not running; Im using stable-dind docker image so not sure what could be the problem? I'd expect docker service to be stated at boot? | 16:23 |
ssam2 | boot of what? | 16:23 |
ssam2 | this is running in a container, so it doesn't boot | 16:23 |
ssam2 | but the .gitlab-ci.yml file should start the docker daemon in the before_script | 16:24 |
ssam2 | maybe that's broken and there's a race | 16:24 |
ssam2 | docker-in-docker is a bit of a hack | 16:24 |
ssam2 | from the log, looks as if the docker daemon is crashing | 16:24 |
ssam2 | ime="2018-01-08T19:08:45Z" level=info msg="changing OOM score to -500" module=containerd | 16:24 |
ssam2 | containerd: write /proc/23/oom_score_adj: permission denied | 16:24 |
ssam2 | time="2018-01-08T19:08:45.560162484Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd | 16:24 |
ssam2 | presumably that works on the x86_64 runners, but not on the aarch64 one | 16:25 |
ssam2 | not sure why without digging a lot further though | 16:25 |
ssam2 | or, it might be possible to tell docker not to try and adjust its oom score | 16:25 |
ssam2 | there's a --oom-score-adjust option, but it doesn't say in --help whether or not you can specify 'don't adjust it' as an option | 16:27 |
ssam2 | maybe passing 0 would work | 16:27 |
*** xjuan has joined #buildstream | 16:46 | |
*** noisecell has quit IRC | 17:08 | |
jjardon[m] | ssam2: ok, yeah seems something specific of the aarch64 image as is should be exactly the same as the x86_64 one. Thanks for the hints, I will try with that | 17:17 |
*** luc14n0 has quit IRC | 17:46 | |
*** luc14n0 has joined #buildstream | 17:46 | |
*** jude has quit IRC | 17:50 | |
gitlab-br-bot | buildstream: merge request (sam/local-source-cachekey-fix->master: utils.py: Sort the results of list_relative_paths()) #216 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/216 | 17:50 |
*** ssam2 has quit IRC | 18:02 | |
*** jonathanmaw has quit IRC | 18:03 | |
*** valentind has joined #buildstream | 18:05 | |
*** xjuan has quit IRC | 20:12 | |
*** valentind has quit IRC | 20:14 | |
*** valentind has joined #buildstream | 20:51 | |
*** mcatanzaro has quit IRC | 21:59 | |
*** ernestask has quit IRC | 22:25 | |
*** tristan has quit IRC | 22:47 | |
*** valentind has quit IRC | 22:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!