IRC logs for #buildstream for Saturday, 2018-08-04

gitlab-br-botbuildstream: merge request (edbaunton/executable-remote-source->master: remote.py: Add support for marking downloaded files executable) #581 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/58100:37
gitlab-br-botbuildstream: issue #551 ("Artifacts are being pushed just after being pulled from the CAS cache") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/55101:30
gitlab-br-botbuildstream: merge request (edbaunton/executable-remote-source->master: remote.py: Add support for marking downloaded files executable) #581 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/58102:00
*** ErrantEgo has joined #buildstream03:05
*** leopi has joined #buildstream08:18
*** tristan has joined #buildstream08:48
*** tristan_ has joined #buildstream08:48
gitlab-br-botbuildstream: merge request (edbaunton/doc-typo->master: element.py (docs): dashes not underscores for build and install root) #577 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/57708:55
*** bochecha has joined #buildstream08:59
tristan_bochecha, just wanted to say thanks for looking into packaging BuildStream in fedora !09:49
bochechatristan_: no problem09:49
bochechaI'm having trouble with grpcio at the moment, though09:49
tristan_ah, try to ping juergbi during the week about that; as I understand it there is some code generation, and committed results of code generation in order to easy the `pip install` case09:50
tristan_but I'm not familiar with all the details09:50
tristan_s/easy/ease09:50
bochechaoh, the problem is not in BuildStream09:52
bochechagrpcio isn't packaged at all in Fedora, so I need to package that first09:52
bochechaand building grpcio fails on Rawhide09:52
bochechafor some reason it doesn't find Python.h :-/09:52
tristan_ah I see10:01
tristan_Python.h is probably from a cpython dev package ?10:02
bochechayes10:03
bochechawhich I have installed in the build root10:03
bochechathe package builds fine on F28, but not on Rawhide10:03
bochechathe only difference I can see is that Rawhide has python 3.7, which F28 has 3.610:04
bochechabut that shouldn't cause any problem…10:04
gitlab-br-botbuildstream: issue #331 ("Project configuration required is too much") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/33110:06
gitlab-br-botbuildstream: merge request (edbaunton/doc-typo->master: element.py (docs): dashes not underscores for build and install root) #577 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/57710:08
gitlab-br-botbuildstream: merge request (tiagogomes/issue-520->master: Fix race condition when calculating disk usage) #600 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/60010:10
*** leopi has quit IRC11:04
*** leopi has joined #buildstream11:08
*** holodoc has joined #buildstream11:15
gitlab-br-botbuildstream: issue #552 ("Add documentation for the element states during a build") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/55212:04
gitlab-br-botbuildstream: merge request (adamjones/systemd-cas->master: Include systemd file examples for google-cas cache) #524 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/52412:07
*** lantw44 has joined #buildstream12:21
gitlab-br-botbuildstream: merge request (jjardon/out-of-tree->master: WIP: buildstream/plugins/elements/autotools.yaml: Make builds out of tree by default) #175 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/17512:28
*** mat has joined #buildstream12:33
*** leopi has quit IRC13:31
*** nurupo has joined #buildstream13:35
*** leopi has joined #buildstream13:41
*** NyanCat has joined #buildstream14:15
bochechaTraceback (most recent call last):14:41
bochecha  File "/builddir/build/BUILD/buildstream-1.1.4/doc/bst2html.py", line 38, in <module>14:41
bochecha    from buildstream import _yaml14:41
bochechaModuleNotFoundError: No module named 'buildstream'14:41
bochechaI'm getting that when trying to build the docs in the rpm package14:42
bochechaseems the build script requires buildstream to be installed…14:42
tristan_bochecha, the docs dont build without buildstream being installed, indeed14:42
bochechathat's not great :-/14:42
bochechait's pretty much a requirement for distro packages (or the docs could be pre-built in the tarball I guess)14:43
bochechaI mean, I can do `PYTHONPATH=.. make all` for now in the package, that works14:43
bochechawould be better if the Makefile did it, though? :P14:44
tristan_I agree it's not great, it hasnt been a requirement thus far so havent been correcting it14:44
tristan_bochecha, an option is to use a python venv14:44
tristan_I think that would work fine14:44
bochechanot going to use a venv in a rpm package :)14:44
tristan_I mean, as a part of the process of building the docs for an rpm package ?14:44
tristan_not sure how straight forward that would be, though14:45
tristan_oh, PYTHONPATH works ?14:45
bochechalike I said, setting PYTHONPATH=.. works14:45
tristan_that seems not too bad14:45
bochechait's just much simpler :)14:45
tristan_yeah14:45
bochechawell, I think it works14:45
bochechahaven't finished a full build yet14:45
bochechabut at least it managed to import the buildstream module14:45
tristan_it has to "work", not sure if it will14:46
tristan_but if the deps are all in the build environment I guess it should14:46
tristan_note: it has to work because we run buildstream (the bst2html thing you mentioned) in order to generate some of the colorized output which goes into the examples14:46
tristan_but I guess it will work14:47
bochechawell, the good thing is that it means if I manage to build the docs, then I'm sure I have all the dependencies :)14:48
bochecha(as long as all of buildstream is documented, of course)14:49
tristan_if you pass setup.py you should have everything you need yes; however... you need additional things for docs building, than you need for BuildStream14:50
tristan_i.e. if you don't have ostree and python-gi, then BuildStream will work fine14:50
bochechasetup.py doesn't check for fuse, ostree or gi14:51
bochechaalso, PYTHONPATH doesn't work14:51
tristan_it will correctly abort early, and tell you that you need to install ostree, if your project uses ostree14:51
bochechathe Makefile sets PYTHONPATH already14:51
bochechaso it overrides what I had set :(14:51
tristan_or, if it doesnt, that's a bug (since ostree dep disappeared from core recently, maybe we need to fix it)14:51
tristan_Also, you probably need git for docs building14:52
tristan_cause some of the examples will use the git plugin which requires host git14:52
tristan_bochecha, I'm perfectly amenable to including prebuilt docs in the release tarballs, but I'm not sure right now what is the easier fix (extending setuptools's sdist command can also be complicated)14:54
tristan_it would be ideally better, if building docs worked from git-or-tarball, of course14:54
*** leopi has quit IRC15:02
bochechatristan_: yeah, docs should be able to build from git or tarball; and they shouldn't be built for the installed buildstream, but for the one in the source tree15:21
bochechaI think I have a patch, testing it right now15:21
bochechatristan_: got it working I think15:35
bochechaModuleNotFoundError: No module named 'google'15:45
bochechaI… huh… is there really a "google" python module? o_O15:45
bochechaah, that seems to be protobuf maybe15:46
tristan_yeah if we can get the docs to build without install that'd be idea16:01
tristan_l16:01
bochechaalmost got them :)16:08
bochechaI'm still fishing for missing modules16:08
bochechaarpy isn't in Fedora, so I have to package that first16:08
tristan_arpy is optional and only used for treating debian packages as if they were tarballs, iirc16:15
bochechathe docs fail to build without it16:15
tristan_still good to have, but don't know if the BuildStream package itself should depend directly on it16:15
tristan_ah, probably it has an example which uses it, then16:15
bochechaplus, I'd rather package bst with all its deps, so that it's not limited artificially16:16
tristan_of course16:16
bochechahow would you see the packaging?16:16
bochechaeverything in one "buildstream" package?16:16
bochechaor split the plugins into multiple buildstream-plugins-* packages?16:16
tristan_From what it installed from the buildstream repo, I think should go into one package16:17
bochecha(I'd prefer everything in one, it's easier to maintain and less confusing for users)16:17
tristan_it really is core plugins, and there is a distinction16:17
bochechayeah, I'm thinking of doing a second bst-external package16:17
bochechaor maybe calling it buildstream-plugins-external or something16:17
tristan_right, I was thinking of bst-external, that one contains stuff that is not really ready for prime time (API is not considered stable there for now)16:18
bochechaand have the main buildstream package Recommends or Suggests it16:18
tristan_so, that is a bit ambiguous16:18
tristan_right now we suggest that people who use bst-external and need those things in a consistent/stable state, use it as a git submodule of their buildstream project repo, and use the plugins as local plugins16:19
tristan_but external plugin loading needs to be hardened in general I think16:19
bochechaok, so maybe I shouldn't package that then, if that's the upstream recommended way to use them16:20
tristan_I think it might be better not to, it could result in misguided expectations16:20
bochechaagain… less work for me :D16:21
bochechait builds!16:22
bochechashould I send the MR against master? will that get it into the next release?16:23
tristan_nice !16:23
tristan_bochecha, master always first; but yeah we will have to backport it to bst-1.2 branch16:24
bochechait applies with fuzz16:24
bochechawell16:24
bochechaI made it on master, but it applies with fuzz on 1.1.4 (which is what I packaged)16:25
tristan_bochecha, one thing though... our CI builds docs from an installed package16:25
tristan_you probably want to change that in the .gitlab-ci.yml, to ensure we never regress it16:25
bochechaindeed16:25
gitlab-br-botbuildstream: merge request (bochecha/build-docs->master: doc: Build the docs without Buildstream installed) #603 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/60316:32
bochechatristan_: ---^16:32
bochecharunning the tests in the package will be harder, there are quite a few modules which don't exist in Fedora, or in the wrong versions :(16:35
bochechaI don't understand why the CI failed :-/16:36
tristan_https://gitlab.com/BuildStream/buildstream/pipelines/27181796 ?16:36
tristan_did it fail ?16:36
bochechait did: https://gitlab.com/BuildStream/buildstream/pipelines/2718173516:37
bochechaoh, maybe that's the first one16:37
bochechaI forced push very quickly, so maybe the CI couldn't find the previous git commit after I forced-push :)16:37
tristan_yeah, that happens16:38
tristan_oops16:41
tristan_aha16:41
tristan_yeah it's gonna be tougher than this16:41
bochechahow come?16:41
bochechawhy did it need to run bst? o_O16:41
tristan_https://gitlab.com/BuildStream/buildstream/-/jobs/86843089 is because of bst2html I suppose...16:41
bochechabut in my package bst2html is run16:42
bochechaand the bst command doesn't exist there either16:42
tristan_bst2html.py:run_command() needs to import the cli and run it directly16:42
tristan_I guess16:42
tristan_weird16:42
bochechawhy wasn't that run in my package then? :-/16:43
tristan_weird that it would work; the bst command itself is generated by setup.py16:43
bochechaah, --force16:43
bochechathe CI sets BST_FORCE_SESSION_REBUILD16:43
bochechaI don't16:43
bochechathat's the difference16:43
bochechaso in the package, I end up not rebuilding those "sessions"16:44
bochechawhat are sessions by the way?16:44
tristan_aha interesting16:44
tristan_bochecha, http://buildstream.gitlab.io/buildstream/tutorial/first-project.html#build-the-element16:45
tristan_for example16:45
bochechathe page doesn't seem to talk about sessions :-/16:45
tristan_bochecha, the examples in http://buildstream.gitlab.io/buildstream/main_using.html, use the output of a bst session to generate the docs16:45
bochechaoooooh16:45
bochechait's a shell session!16:45
tristan_yeah, we force the colors, and then convert that to html for the docs :)16:46
bochechaah, and there are some stored in git/tarball16:47
bochechahmmm16:47
bochechaso basically, the docs can't be fully built without bst being installed :(16:47
tristan_bochecha, as a hint... tests/testutils/runcli.py does something way above and beyond what we need to do in docs/bst2html.py16:47
bochechado you have to change those stored sessions every time someone makes a change which impacts the output?16:47
tristan_bochecha, not entirely... i.e. we render what the command was, the shell is not really real16:47
tristan_we only care about capturing the stdout/stderr together in the same stream16:48
tristan_we certainly want the CI to rebuild them every time, because any change we make in the UI, is automatically reflected and changed in the docs, which are automatically deployed from CI16:48
bochechasure, but not in the tarball/git?16:49
tristan_so the shell prompt is totally faked by bst2html.py, it just needs to invoke _frontend/cli.py and capture the output16:49
tristan_bochecha, the reason why we commit the session files to the repo... is so that a developer who wants to make a change to the docs can do it without having to wait to build everything16:50
bochechayeah, but are they kept up to date when developers change things?16:51
tristan_I suspect people who want to make some touch ups will want to build the docs locally, but not wait for the example sessions to build16:51
bochechaif not, then in the package I have to force-rebuid them like in the CI16:51
tristan_Well, that is a bit ambiguous, ideally indeed they should be force rebuilt for a package; but that will require that building the package has internet access, which is errrm, not playing fair16:52
tristan_so, another approach would be to say, that like man pages, we should update them with every release16:52
tristan_in order to avoid the network requirement for a package build, which I suspect is not supposed to be allowed to connect to the internet16:53
* tristan_ not really sure how the packaging stuff works in practice, but presumes that distros use stable / sandboxed stuff like OBS16:53
tristan_bochecha, at least for the bst2html.py stuff, we could have an easier approach of adding an `if bla bla == __main__` in _frontend/cli.py, and run that directly from bst2html.py16:56
tristan_(that is more straight forward than trying to replicate what we do for pytest)16:56
bochecharight, the build will not have network access (Fedora build system enforces that)16:56
bochecha(it builds in a chroot, populated exclusively of packages that are in Fedora, and network is blocked)16:57
tristan_so in that case, I'll make sure they are updated along with releases16:57
tristan_so at least from a release tag, you'll have up to date sessions16:57
bochecha👍16:57
* tristan_ will have to add that to some docs about how we do releases16:58
bochechahuh16:59
bochechaso tests/testutils/runcli.py eventually does buildstream._frontend.cli.main(…)16:59
bochechabut there's no main() in buildstream/_frontend/cli.py ?17:00
bochechaoh17:01
bochechaclick.BaseCommand.main = override_main17:01
bochechait's a click thing, ok17:02
tristan_bochecha, yeah forget testutils, lets add a clause and shebang to cli.py for this purpose17:04
tristan_http://click.pocoo.org/5/17:05
tristan_shows that you should be able to just run cli() with no args, and click will take care of it17:05
bochechaI got it to build17:07
bochechafrom buildstream._frontend import cli as bst_cli17:07
bochecha    args = ['--colors', '--config', config_file, '--directory', directory] + shlex.split(command)17:07
bochecha    bst_cli.main(args=args, prog_name=bst_cli.name)17:07
bochechathe problem is capturing the output17:07
tristan_exactly :)17:07
bochechaso…17:08
tristan_but if you add `if __name__ == '__main__': cli()` at the bottom of cli.py17:08
bochechawant me to add the if __name__ stuff to cli.py, then subprocess.Popen(['python3', '../buildstream/_frontend/cli.py', … ?17:08
tristan_then you can have bst2html.py just run `python2 path/to/_frontend/cli.py` and it should work17:09
tristan_bochecha, yeah, with a comment on that clause that it's used for generating docs :)17:09
bochechapython3 :P17:09
tristan_ooooops17:09
tristan_typo17:09
tristan_haha17:09
tristan_funny I dint land on 4 or 517:09
bochechalet's see how it goes17:13
bochecha /home/mathieu/Projects/buildstream/doc/source/tutorial/first-project.rst:32:Include file '/home/mathieu/Projects/buildstream/doc/examples/first-project/project.conf' not found or reading it failed17:14
*** leopi has joined #buildstream17:14
bochechathat file exists, but somehow it gets removed when I run `BST_FORCE_SESSION_REBUILD=1 make all` :-/17:15
bochechahahahaha17:19
bochechanow the sessions contain:17:19
bochecha../buildstream/_frontend/cli.py17:19
bochechapython3: can't open file '../buildstream/_frontend/cli.py': [Errno 2] No such file or directory17:19
bochechabecause of course, the cwd is changed for those Popen17:19
bochechatristan_: I don't think we can make this work with the `if __name__` trick17:22
bochecha+Traceback (most recent call last):17:22
bochecha+  File "/home/user/Projects/buildstream/doc/../buildstream/_frontend/cli.py", line 5, in <module>17:22
bochecha+    from .. import _yaml17:23
bochecha+ValueError: attempted relative import beyond top-level package17:23
csorianois it possible to build gtk master?17:52
csorianoit says:17:52
csoriano  /home/csoriano/.cache/buildstream/build/freedesktop-sdk-kck9o52f/sdk/project.conf [line 8 column 14]: Specified path 'elements' must not lead outside of the project directory17:52
csorianowhen doing bst build --track-all core-deps/gtk+.bst17:52
bochechacsoriano: what exactly are you doing?17:57
csorianobst build --track-all core-deps/gtk+.bst17:57
bochechayour current directory, etc…17:58
bochechathe log file seems to indicate you're in the freedesktop-sdk17:58
bochechabut the core-deps/gtk.bst filename suggests you're in gnome-build-meta17:58
csorianocurrent directory..? Didn't know that matters17:58
csorianoI followed https://wiki.gnome.org/Newcomers/BuildSystemComponent17:58
bochechawell of course it does, bst finds the project.conf in your current dir :)17:58
bochecha(unless you explicitly specify it, I think there's an option)17:59
csorianoI'm in the gnome-build-meta source directory17:59
bochechawhy is it talking about freedesktop-sdk then… :-/18:00
csorianono idea...18:00
tristan_gah18:03
* tristan_ palmface18:04
tristan_csoriano, this is because of https://gitlab.com/BuildStream/buildstream/issues/195, I've been waiting on freedesktop-sdk to fix it before plugging up the problem in BuildStream18:05
tristan_jjardon, said I should fix it first, and they will adapt18:05
tristan_but now, disaster strikes /o\18:06
csorianopity...18:06
csorianoseems it's marked as a priotity though, I guess it will be fixed eventually :)18:06
tristan_5171cb0ebc9f5d372820e33463b4192fbbc6ec6418:06
csorianofor now I set up a flatpak manifest for what I want to do, seems it works18:06
tristan_csoriano, for right now, until I guess next week when freedesktop is fixed, the above commit is before the breakage18:07
tristan_(i.e. that commit in the BuildStream repo)18:07
csorianooh cool, will try later on, thanks!18:07
tristan_I'll send some emails to the list with a release on monday, but fyi will be recommending bst-1.2 branch18:08
tristan_bochecha, aha, I didnt realize python couldnt discover it's being launched from a subpackage18:09
tristan_bochecha, I guess a solution for this would have to live in a private file in the toplevel ?18:10
csorianotristan_: maybe you want to put that info in that guide?18:10
tristan_if __name__ trick hmmm, not sure where, maybe even in toplevel __init__.py ?18:10
tristan_csoriano, yes, that is a hot topic as well, not having a website makes things a bit tricky18:11
csorianoI mean there's the wiki18:11
tristan_but at least the install guide in the docs needs to say something18:11
csorianothat should enough, isn't it?18:11
tristan_and I guess the wiki (I keep forgetting that anyone ever looks at the wiki...)18:11
csorianothe install guide heps you isntall18:12
csorianobut nothing more18:12
csorianothen the wiki makes you to actually build18:12
csorianoright?18:12
tristan_yeah, but currently the instructions from git leads one to believe that master branch is a decent choice18:12
tristan_csoriano, ohh, you mean the newcommers wiki page18:13
tristan_yes there too18:13
tristan_when I think wiki page, I think https://wiki.gnome.org/Projects/BuildStream18:13
csorianoyeah, the guide where it says how to build GNOME stack18:14
tristan_https://gitlab.com/BuildStream/buildstream/issues/528 <-- discussion about communicating the intended release tag18:15
tristan_ideally, we should not have to update too many things with every version bump but refer to a download page or install page which says the right thing18:15
tristan_but indeed, that wiki page deserves an update in the short term18:16
*** leopi has quit IRC18:16
csorianooh I see lot of things moving18:16
csorianoI guess I'll wait a bit to settle down18:17
jjardontristan_: I said you will not break freedesktop-sdk, gnome-build-meta CI, not you will not break any other project / use case18:23
jjardontristan_: we already live in the limit using random versions from git (in the CI); as I said several times if you release tags more often it would be easier for us upstream document what exact version recommend to use for our users18:25
bochechatristan_: I think I got it working with cli.main(…)18:25
* jjardon updates https://wiki.gnome.org/Newcomers/BuildSystemComponent18:27
jjardoncsoriano: ^18:27
tristan_jjardon, it doesnt matter at this point what was said and how it was understood, the result is that now fixing freedesktop to not do this ../ stuff became a huge priority18:28
tristan_I overlooked that it would break `bst build` cases for GNOME on the latest of bst-1.2 branch18:29
tristan_that's my fault18:29
csorianojjardon: thanks \m/. Are you sure you pressed save though? The content didn't change18:31
jjardoncsoriano: I'm still editing18:31
csorianooh ok18:32
jjardoncsoriano: updated!18:33
jjardontristan_: you might want to tag that commit18:35
bochechathose docs are a bit long to build :P18:40
tristan_jjardon, having CI use a sha or specific tag is alright, I'm not about to release 1.1.5 on monday and recommend that users use a sha or older tag19:11
tristan_jjardon, the only way out of the mess, is to fix freedesktop-sdk, or revert the patch until freedesktop-sdk is fixed19:11
jjardontristan_: ok, I'd say revert the patch in buildstream is the quickest solution then19:12
gitlab-br-botbuildstream: merge request (bochecha/build-docs->master: doc: Build the docs without Buildstream installed) #603 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/60319:16
tristan_I can do that, it's still a blocker to 1.2 though19:17
bochechaif anyone is on Fedora and wants to try something new: https://copr.fedorainfracloud.org/coprs/bochecha/buildstream/19:34
bochechaI won't push the packages in Fedora proper for now, for 2 reasons19:34
bochecha1. I'd like to test them a bit first :)19:34
bochecha2. I can't use those packages yet, because in freedesktop-sdk we require a more recent version of buildstream than the latest release19:35
bochechabut they should work19:35
jjardonamazing! thanks bochecha19:41
*** leopi has joined #buildstream19:48
gitlab-br-botbuildstream: merge request (bochecha/fedora-packages->master: WIP: doc: Mention the Fedora packages) #604 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/60420:08
tristan_oh20:18
tristan_bochecha, no joy on launching bst with an `if __name__ bla bla` hack ?20:19
tristan_bochecha, problem is, we need stdout/stderr, in sequence, not separate20:19
bochechahuh? stderr is completely ignored isn't it?20:19
tristan_most of the UI goes through stderr, output from bst show and the like is parsable and scriptable, that goes to stdout20:20
bochechaoh no!20:20
bochechaPopen(…, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)20:20
bochechaI misread that20:20
bochechaI read stderr=subprocess.PIPE20:20
tristan_yeah :-S20:20
bochechaah yes, in this case I broke the docs :)20:20
tristan_it's also important that it's serialized, soooo... :-S20:21
bochechaI wonder if it works if I use the same StringIO for both :x20:21
tristan_That might !20:21
bochechawait, how did it even work?20:23
bochechaout.decode() when out is a StringIO?20:23
bochechahuh20:23
*** jackmcbarn0 has joined #buildstream20:25
*** tristan_ has quit IRC20:40
*** leopi has quit IRC20:42
*** leopi has joined #buildstream20:45
gitlab-br-botbuildstream: merge request (bochecha/build-docs->master: doc: Build the docs without Buildstream installed) #603 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/60321:02
*** leopi has quit IRC21:44
*** noisecell has joined #buildstream23:16
*** noisecell has quit IRC23:17

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!