IRC logs for #buildstream for Monday, 2017-04-10

*** tristan has quit IRC05:46
*** tristan has joined #buildstream06:00
*** ChanServ sets mode: +o tristan06:00
*** jonathanmaw has joined #buildstream08:45
jonathanmawmorning tristan, looks like my overnight/weekend build failed, while building gnome-system-image.bst, due to running out of disk space08:46
tristanAha !08:46
tristanYeah that'll happen especially at that stage :-/08:46
tristanWhat could really be handy, is a `dd` tool which handles sparse files well08:47
jonathanmawtristan: yep, my brother's lamented its absence many times.08:47
tristanthat image splicing will balloon your disk space because it will create the filesys images separately and then create a huge partitioned image to fit it inside, and then splice those images into one big one08:48
tristanyeah08:48
jonathanmawso I'm going to need more than 4G free space08:48
tristanjonathanmaw, still, it's good news that that's the only reason it failed :D08:48
tristanyocto has sparse copy which requires python http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/wic/filemap.py#n53308:50
tristanI suppose something like that can be used08:51
tristanjonathanmaw, also, if you have failed builds, they will not be automatically purged by BuildStream08:52
tristantake a look at ~/.cache/buildstream/build08:52
tristananything in there is delete worthy08:52
*** ssam2 has joined #buildstream09:01
jonathanmawtristan: I'm building with 13G free space, and I get a backtrace https://pastebin.com/bAiiF3Qx09:06
jonathanmawthough the build seems to continue trying09:07
jonathanmawit's managed to use all my disk space, so it looks like it's not handling running out of disk space gracefully, rather than failing for other reasons, though.09:07
jonathanmawtristan: I think there's places that store temporary files other than build09:14
jonathanmawsince I had 13G before, and after a couple of failed builds, I have 8.2G after wiping .cache/buildstream/build09:14
tristanHmmm09:18
tristanjonathanmaw, do you have today's master ?09:19
jonathanmawno09:19
jonathanmawpulling now09:19
tristanthat should help with the stack traces, although it looks like perhaps the _ostree.py commit() code is failing to catch an exception09:20
tristanlets see if master fixes it, because that second part of the trace should be handled now, fixed over the weekend09:20
jonathanmaw\o/ build succeeded09:33
tristanYay !09:33
jonathanmawafter I found 50G to clear, so I don't think I can confirm that the ugly stack trace problem was solved09:33
tristanAlright, I did clean up some stack traces in the frontend over the weekend so I hope it's gone09:34
tristanif not, we'll fix it later09:34
jonathanmawI'm sure I'll run out of disk space again09:34
tristanhah09:36
tristanjonathanmaw, Ok so... tarball source !09:37
tristan:D09:37
tristanHave you had time to take a look at how the Source object works and what there is to implement ?09:37
jonathanmawnot yet09:38
jonathanmawcurrently in a meeting for another project, so I can get started in an hour at the latest09:38
tristanSure, I may be stepping out for lunch/dinner soon09:39
tristanbut I'd like to run down the functionality of how the sources work with you so that it's well understood09:39
jonathanmaw\o/ got GNOME graphical interface running in qemu. too slow to run, unfortunately, I should look at setting it up with virt-manager.09:56
tristanjonathanmaw, yeah I use virtualbox normally09:56
* tristan has a script that takes the image and generates a .vdi09:56
tristanjonathanmaw, here: https://pastebin.com/MW3WBpV709:58
tristanmay be helpful, but you may have your own VM wizardry at your disposal :D09:58
*** tristan has quit IRC10:03
ironfootpaulsher1ood: thanks for the reviews. I though python 3.5 was needed10:13
ironfootjonathanmaw: do you know the answer to that? ^10:14
ironfootbest one to ask is tristan, will wait for him :)10:14
jonathanmawI managed to build using python 3.4.210:16
jonathanmawthe only problem that occurred was a circular include loop, that tristan fixed.10:17
paulsher1oodironfoot: from soon-to-bepublished blog post 'BuildStream now requires python 3.4 instead of 3.5, so this should hopefully be repeatable on most stable distros, e.g. debian jessie ships 3.4'10:17
jonathanmawI think you still need to add jessie-backports to get ostree, but I don't think I've had to rely on anything built from source.10:18
ironfootgah, I'm late :)10:21
ironfootalthough some people migh like to use this inside of docker, idk10:21
ironfootmight*10:22
paulsher1oodironfoot: i want to try inside docker, and also vagrant10:23
jjardon[m]ironfoot: we will definitely use it in docker, but in a server, not locally11:00
ironfootI've read a bit about gitlab registry. That might be useful for using it in a server11:01
ironfootgitlab CI will update the docker image in the registry, and server would use the latest version available.11:05
ironfootgitlab CI job to update docker image could run only in tags (releases)11:05
*** albfan[m] has joined #buildstream11:23
albfan[m]Hi there11:23
albfan[m]I think is worth to mention this https://wiki.gnome.org/Initiatives/ContinuousIntegrationForApps11:24
albfan[m]we want to bypass jhbuild, and flatpak is not ready for our purposes, can buildstream be used inside jenkins in some way?11:25
ssam2hi albfan!11:25
ssam2buildstream can be used from Jenkins like any other build tool, but depends what your needs are11:26
ssam2is the plan to build the apps themselves using BuildStream ?11:26
ssam2if so, the goal is to get feature parity with flatpak-builder, but i don't think things are quite there yet11:26
ssam2I think it's the next goal though ?11:26
albfan[m]Roughly reuse a flatpak runtime to build app on top of it and run test suite (to collect test reports)11:31
albfan[m]avoiding dependency build hell11:31
ssam2definitely sounds like something BuildStream could do. tristan is the main guy to speak with, he's not here right now though11:32
albfan[m]At this time, we are waiting for docker+jhbuild to end gnome-todo and deps compilation to see results of a dummy test suite11:32
albfan[m]madness11:32
albfan[m]ssam2: great, if you see the links all is hosted in github by now, so any change can be done by PR or code review there11:33
albfan[m]https://github.com/waltervargas/gnome-todo/blob/master/Dockerfile#L611:34
albfan[m]https://github.com/waltervargas/gnome-todo/blob/master/Jenkinsfile#L1411:34
ssam2right11:34
ssam2to be honest, Docker probably isn't needed if you use BuildStream11:34
ssam2because BuildStream creates its own container to run the build11:34
ssam2so the host system is totally separate11:35
albfan[m]we have gnome-todo on top of debian, but can be fedora if you want or ubuntu11:35
ssam2it does require Python 3.4 and OSTree on the host though11:35
albfan[m]better of11:35
ssam2we have folk running it on Debian Jessie, using OSTree and Bubblewrap from jessie-backports11:35
albfan[m]Here is our jenkins https://github.com/waltervargas/gnome-jenkins11:36
ssam2https://gitlab.com/BuildStream/buildstream-tests/tree/master is an example of building GEdit on top of the GNOME Flatpak SDK11:36
albfan[m]I have been looking at that. Great I will give it a try11:37
* albfan[m] sent a long message: albfan[m]_2017-04-10_11:40:13.txt - https://matrix.org/_matrix/media/v1/download/matrix.org/KxrUxGMtooaTutGuOtjOzgpA11:40
ssam2I think demo from https://blogs.gnome.org/tvb/2017/02/06/introducing-buildstream-2/ will be pretty close to what you need11:41
ssam2`build-stream build gedit.bst` runs the build11:41
ssam2takes a long time the first time because it has to pull the whole GNOME SDK to a local OSTree repo11:41
ssam2but if that OSTree repo is shared between builds then subsequent builds should go quick11:42
ssam2not sure about how `make check` would work off-hand11:44
ssam2I imagine it's something we want to support11:44
ssam2but it's a slightly strange case because in most cases the test suite expects to be run from the build tree11:44
ssam2i think we'll need to wait til tristan returns to figure that out11:46
ssam2in the meantime, just testing that the app executes is useful11:46
ssam2hmm, actually no I think this is already supported11:46
ssam2`build-stream shell --scope build` seems like it might make the build tree available..11:47
ssam2that might just make the build-dependencies available rather than the actual build tree11:47
*** tristan has joined #buildstream11:50
albfan[m]about that no problem. We can go to installed tests https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests and execute them with https://git.gnome.org/browse/gnome-desktop-testing11:51
ssam2ah, that's good news11:51
ssam2tristan, albfan[m] is looking at implementing https://wiki.gnome.org/Initiatives/ContinuousIntegrationForApps using BuildStream to build the GNOME apps against the Flatpak SDK11:52
albfan[m]but cool to hear something about expose build tree anyway. This week will be really interesting11:52
* tristan reads backlog in the logger12:00
tristanhi albfan[m] :)12:01
albfan[m]Hi tristan, nice blog entry indeed!12:01
albfan[m]can wait to try it. afk now12:01
tristanalbfan[m], new blog just hit the streets minutes ago btw :)12:03
tristanhttps://blogs.gnome.org/tvb/2017/04/10/buildstream-progress-and-booting-images/12:03
tristanalbfan[m], ok anyway will hope to discuss soon any help I can provide :)12:04
tristanfrom our perspective, buildstream provides a pipeline for running abstract plugins which perform mutations on filesystem data12:05
tristanHowever, it's quite possible to write plugins which simply make an assertion that installed-tests pass12:05
tristanas usual, there are many ways to skin cats :)12:05
albfan[m]Great, we are looking for something that just work (although awnful) I will make some tries and see how far I get. Then I will ask how to proceed12:10
albfan[m]plugins, plugins, plugins... everything is just a plugin away!12:11
*** juergbi has quit IRC12:15
*** juergbi has joined #buildstream12:16
paulsher1oodtristan: s/into a single artifact, that with some additional/into a single artifact, with some additional/12:30
jonathanmawtristan: is there a "tar" output format like there's an sda.img?12:38
jonathanmawor do I understand correctly that if I just want to look at the output, I can use bst checkout to look at the contents of the stack?12:39
tristanjonathanmaw, I added checkout in a sort of hurry, it may need some changes; checkout will not stage dependencies12:40
tristanjonathanmaw, however if you want for a quick check, use `bst shell`12:40
tristanbst shell --deps runtime foo.bst12:41
tristanfor a stack12:41
jonathanmawand that'll dump me into a shell inside that stack? cool!12:41
tristanjonathanmaw, yep :)12:42
tristanit'll wipe the directory clean though once you exit the shell12:43
jonathanmawI take it I can do `bst build <stack>` too?12:43
tristanyes12:43
jonathanmawright, that gives me a path to testing how tar sources are working.12:43
tristanjonathanmaw, ideally, we should have regression tests for that... see the tests/sources directory12:44
tristanwe've (I've) been falling behind on test coverage, though :'(12:45
jonathanmawaha12:45
tristanSome of it just cannot be tested easily12:45
tristanAnd some of it just doesnt fall into the context of a `make check` like semantic (i.e. I dont want `./setup.py test` to require downloading external resources)12:46
tristanjonathanmaw, Ok so just for a rundown; there is configure(), preflight(), fetch(), track() and stage() basically to implement12:47
tristanalot of this (I hope) is quite obvious, what they should be doing12:47
tristanOh, since recently I also added another12:47
tristan(not that recently actually)12:47
tristanright, set_ref() and get_ref()12:48
jonathanmawtristan: Basically every method that source.py raises an ImplError in?12:48
tristanand get_consistency()12:49
tristanYeah :)12:49
tristanjonathanmaw, for tarballs, we will want track() to resolve the sha256sum of the tarball and return that12:49
tristanSo that users need not spend time with copy/paste of checksums12:50
tristanEventually, we may want something more fancy for that, but for starters, that will be good12:50
jonathanmawok12:52
jonathanmawlooking at plugin.py, do I also need to implement get_unique_key()?12:52
tristanjonathanmaw, yes :)12:53
tristanThat is the basis for cache key calculation of a Source12:53
tristanjonathanmaw, see how git.py does it12:53
jjardon[m]Hi, Im trying to use bst to build definitions but Im getting a quite strange error when trying to build the converted gnome system: https://gitlab.com/baserock/definitions/builds/14040020 (that file is actually there): any idea?12:54
tristanjonathanmaw, note that for a source, we want to use the *unresolved* uri (not resolved by any alias)12:54
tristanjonathanmaw, i.e. this means you can start mirroring your sources in one location and it wont cause all cache keys to change12:54
tristanjjardon[m], is that a re-converted thing, following the defs2bst migration path docs ?12:55
tristanor a naked build of 'build-gnome' branch in buildstream-tests directly ?12:55
tristanjonathanmaw, to understand the power of track, try: bst track foundation/systemd.bst (for example)12:56
tristanjonathanmaw, and then do 'git diff -a' in the buildstream-tests repo (assuming you have the build-gnome branch checked out)12:56
tristanit's still a little rough around the edges (we need to convert errors to warnings in there in case of bad branches)12:57
tristanit also follows the same semantics with a --deps argument (by default now, track will only track one element, unless --deps is specified)12:58
jjardon[m]tristan: converted12:58
tristanjjardon[m], Ok, did you reuse the same project.conf ?12:59
jjardon[m]Its only doing this atm: https://gitlab.com/baserock/definitions/blob/jjardon/bst2/.gitlab-ci.yml#L1912:59
tristanAh, Ok I think I need to pull that change into the gnu-toolchain branch12:59
tristanjjardon[m], Ok I see, I know why it's happening, I think... hold on...13:00
tristanoh that's strange13:00
tristanjjardon[m], ok so, first of all, you are calling `bst build elements/gnome/gnome-system-x86_64-content.bst`13:02
tristanwhich is intuitive, and I've actually been thinking of supporting13:02
tristanbut is currently not supported13:03
tristanjjardon[m], what you want is in fact bst build gnome/gnome-system-x86_64-content.bst13:03
tristanjjardon[m], because I added in project.conf, an option to say "all the elements go into a <subdir>"13:03
tristanevery element is referred to, relative to the elements directory13:03
tristanin the gnu-runtime project.conf there, it's 'elements'13:04
tristanI also dont think that the file `/builds/baserock/definitions/buildstream-tests/./bluetooth.bst` should exist13:05
tristanbecause you ran defs2bst.py with --output into the elements/ subdir (which is what you want anyway)13:05
tristanAlthough, I'm confused as to why it got that far at all13:05
tristanit should have failed before that13:05
tristanjjardon[m], Also one thing I'm not 100% sure of, it may very well be that you need to run defs2bst more than once13:06
tristanjjardon[m], i.e. once on the system morph, and once more on say (for that system), gnome/strata/gnome.morph13:06
tristanjjardon[m], this depends on how ybd generates the <target>.yml we use13:07
tristanI am guessing here, that someone forgot to include bluetooth in the system.morph for the gnome system ?13:07
jjardon[m]maybe, let me check13:09
tristannope13:09
tristanI just checked, it's there13:09
tristanjjardon[m], first try removing the leading `elements/` on your command line13:09
tristanof bst build13:09
tristanAlso note that cd is not really necessary13:10
tristanYou can do: bst -C buildstream-test build gnome/gnome-system-x86_64-content.bst13:10
tristanjjardon[m], note that indeed, in your 'find' output: buildstream-tests/elements/bluetooth.bst13:12
tristanNot buildstream-tests/bluetooth.bst13:12
jonathanmawtristan: `bst track foundation/systemd.bst` is printing an ugly stack trace for me. the error message seems to be "pkg_resources.DistributionNotFound: BuildStream==0.1"13:14
tristanreally ?13:14
tristanwat ?!13:14
tristanjonathanmaw, how can anything else work, then ?13:14
jonathanmawhere's the trace https://pastebin.com/sjJqs1a513:14
jonathanmawaha, I've managed to funt gnome-system-image somehow, too13:15
tristanjonathanmaw, somehow you uninstalled buildstream or smth13:15
jonathanmawall of buildstream seems to have been funted. maybe it was when I did `git clean -dfx` on the buildstream dir13:15
tristanjonathanmaw, that would do it yes13:15
tristanjonathanmaw, `pip install --user -e .` should fix13:16
tristanjonathanmaw, it *might* think it's already installed (due to cruft in ~/.local), if it does, then `pip uninstall buildstream` first13:16
jonathanmawtristan: `pip3 install --user -e .` seems to be the proper invocation in my case.13:16
tristanyep13:16
jonathanmawtrack succeded13:17
tristanjonathanmaw, thats the developer way, because then you never need to do anything whenever you modify a line of code (bst points to your checkout directory)13:17
tristanjonathanmaw, so `git diff -a` will show that bst found the latest commit sha for branch v232 :)13:17
jonathanmawtristan: ok, so when working on source, I push it, and then `bst track <foo>` will fetch those changes and work out what the new ref is?13:20
jonathanmawor is there a workflow which involves local clones of the source that gets tracked?13:21
jonathanmawor when I have local source code do I use a "local" kind element?13:22
tristanjonathanmaw, a local source is a source which uses files from the project directory directly13:22
jonathanmawok, so if I have a local check out of source code, I need to push the changes to have the ref change.13:23
tristanjonathanmaw, the track feature is so that you can have a project which lists branch names for a bunch of sources, and then bst track can update them to 'latest-of-branch' automatically13:23
tristanNope :)13:23
tristanHeh, this seems tricky13:23
tristanjonathanmaw, By "A local checkout of source code" I think you mean a "workspace" (https://wiki.gnome.org/Projects/BuildStream/Roadmap/Workspaces)13:24
tristanThose are not implemented yet, just put that idea aside for now13:24
jonathanmawah, ok.13:24
tristanjonathanmaw, a "local" source (I.e. the "local" source plugin)... is demonstrated by, for example in the 'build-gnome' branch, see the elements/initramfs/initramfs-scripts.bst element13:25
tristanjonathanmaw, I preferred to keep the initramfs init/shutdown scripts "in tree" instead of in a remote repository13:26
jonathanmawtristan: Tarball Source in the roadmap says to use the host wget and the host tar to do the work. Is there any reason not to use the built-in python libraries instead?13:26
tristanyou will find them directly in files/initramfs-scripts/13:26
tristanjonathanmaw, strictly speaking no there is not13:27
tristanif you want to use built-in python stuff that is part of the standard lib, thats fine13:27
tristanjonathanmaw, at initial fetch time, if the tarball is not already downloaded, use plugin.tempdir() context manager for a temp directory to download to13:30
tristanand move the tarball in place after a download completes successfully13:30
tristan(which is essentially self.tempdir())13:30
jonathanmawAny thoughts on how to download tar files from places that use authentication?13:32
jonathanmawAre we expecting to have to download tars from a place that uses basic auth? If so, how would we store username and password info?13:35
jonathanmawgiven we don't want to commit that to git, I don't think we'd write it into the elements.13:36
tristanjonathanmaw, I am hoping that any such kind of auth can be done out of band, honestly13:37
tristani.e. best would be something like scp13:37
tristanand exchange of public keys13:37
jonathanmawok, so I can skip auth with downloading tar files via http.13:38
tristanjonathanmaw, in the worst case, we can allow plugins to interrogate user configuration13:38
tristanbut for right now yes, lets start with a tarball source that "works" first :)13:38
tristanjonathanmaw, note (not entirely related) that the ostree source allows loading of a public key in a project relative directory13:40
tristanbut this is used only for certification of authenticity13:40
tristannot authentication of the downloader13:40
tristan(i.e. signature verification)13:40
jjardon[m]tristan: mmm, seems the new build call didnt work: https://gitlab.com/baserock/definitions/builds/1404228613:41
jjardon[m]let me try with what you suggest last13:41
tristanjjardon[m], you should try this on a machine first before throwing this at gitlab runners no ?13:42
* tristan is signing out for the night...13:43
tristanjjardon[m], I have the impressions *something* is up with the elements/ subdir here13:43
jjardon[m]tristan: I though it would be easier to share build logs and also to have a pristine env every run; I will try locally as well13:45
jonathanmawtristan: bonus! When using python's urllib, you can specify file:/// URIs, which removes the need to be connected to the internet to test it :)13:47
*** tristan has quit IRC13:49
jjardon[m]mmm, no luck: https://gitlab.com/baserock/definitions/builds/14045623#down-build-trace14:01
jonathanmawIs there a way to run the tests for only one test file (e.g. tests/sources/git.py)?14:05
jonathanmawediting python_files in setup.cfg to only point to the file I want doesn't seem to be enough14:06
jonathanmawOk, for tar, I think the source format is going to have the fields kind, url, and sha256. I can't make use of a "track" field because unlike git repos, an URL points to the entire thing.14:25
jonathanmawso bst track <element> is going to update the sha256 field to be whatever the file's sha256sum is.14:26
*** mattiasb has joined #buildstream14:30
ironfootjjardon[m]: that .bst file is under elements/gnome not under gnome/14:37
jjardon[m]ironfoot: yup, tristan told me to change it, I was using "cd buildstream-tests && bst build elements/gnome/gnome-system-x86_64-content.bst" ; but seems that is not supported14:38
ironfootbst -C buildstream-tests build elements/gnome/gnome-system-x86_64-content.bst14:38
ironfootinstead of bst -C buildstream-tests build gnome/gnome-system-x86_64-content.bst14:39
jonathanmawhrm, not sure how set_ref and get_ref are meant to work if I don't have a field named 'ref'. I'm going to repurpose ref as the sha256sum.15:00
jonathanmawand I should probably re-add "track", though I'm only expecting the value to be some variation on true or false.15:02
jjardon[m]ironfoot: nah, same problem as the beginning: https://gitlab.com/baserock/definitions/builds/1405770815:38
ironfootjjardon[m]: well, that's a different error "Could not find file at /builds/baserock/definitions/buildstream-tests/./bluetooth.bst"15:48
ironfootit looks like the generated .bst files are a bit broken15:56
jjardon[m]ironfoot Its the same error when I first ask at 13:54 : https://gitlab.com/baserock/definitions/builds/14040020 :) I will try to compare them with the ones in the -test repo15:59
ironfootjjardon[m]: oh true, I only saw your second and third build logs.16:01
ironfootsomething is up with defs2bst, it might be good to have a look at what has been generated16:02
ironfootBTW, interesting pipeline that one16:02
jjardon[m]my intention is to migrate all the baserock definitions that are in the CI atm and then use the output of that migration to build; so we can compare betwen ybd and buildstream (and also check the migration scripts work)16:25
ironfoot++16:26
jonathanmawhooray for me! I have a substantial amount of code in a git repo somewhere https://gitlab.com/jonathanmaw/buildstream/tree/jonathan/tar-source (BIG NOTE: Nowhere near finished, I'm just feeling proud)16:37
ironfoot:)16:39
ironfootjjardon[m]: this one works: http://paste.baserock.org/kiredusuza16:42
ironfootbasically, the defs2bst is not a very polished program :)16:43
jjardon[m]\o/16:44
* jjardon[m] tries16:44
ironfootmain change: --output option16:44
ironfootand also the `bst` call changed to remove "elements"16:45
* jonathanmaw has an appointment with some kimchi. Cheerio!16:45
*** jonathanmaw has left #buildstream16:45
jjardon[m]ironfoot: should https://gitlab.com/BuildStream/defs2bst/blob/master/README.rst be updated or its actually a bug?16:48
ironfoothm.. given that readme, it's a bug16:49
ironfootjjardon[m]: that error is new, but looks familiar (https://gitlab.com/baserock/definitions/builds/14065419)16:53
ironfootI remember seeing it when running `bst` in docker some weeks ago (I didn't see it when running bst in docker today and yesterday)16:54
jjardon[m]mmm, at least seems we have progress :) I will look around17:02
jjardon[m]ironfoot: thanks for the help!17:02
jjardon[m]seems maybe we are hitting this? https://github.com/docker/docker/issues/107017:31
*** ssam2 has quit IRC18:40
jjardon[m]ironfoot: tristan I've opened https://gitlab.com/baserock/definitions/issues/919:01

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!