IRC logs for #buildstream for Thursday, 2017-12-07

*** mcatanzaro has quit IRC00:00
*** tristan has joined #buildstream01:37
hergertmeare there any plans to make something like `mkosi'?01:44
tristanhergertme, there is so far only the x86image plugin http://buildstream.gitlab.io/bst-external/bst_external.elements.x86image.html#module-bst_external.elements.x86image01:47
tristanwhich draws some inspiration from wic (yocto's thing, similar to mic)01:48
hergertmetristan, cool, that looks interesting03:55
*** jude has joined #buildstream08:48
*** bethw has joined #buildstream08:57
*** adds68_ has quit IRC09:21
*** adds68 has joined #buildstream09:26
*** adds68 has quit IRC09:28
*** adds68 has joined #buildstream09:28
*** bochecha has quit IRC09:43
*** bochecha has joined #buildstream09:43
*** bochecha_ has joined #buildstream09:44
*** bochecha has quit IRC09:47
*** bochecha_ is now known as bochecha09:47
gitlab-br-botbuildstream: merge request (workspaces-yml-version-1->master: WIP: Add support for version 1 workspaces.yml format) #176 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/17609:55
*** ssam2 has joined #buildstream10:09
gitlab-br-botbuildstream: merge request (unused-vars-refactor->master: Refactor, removing unused variables) #177 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/17710:56
ssam2i'm really overjoyed that the multiple cache support tests pass locally but fail in CI12:11
ssam2oh, haha maybe it's my uncommited changes that are to blame :-)12:11
gitlab-br-botbuildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/16612:11
*** bochecha has quit IRC12:17
*** bochecha has joined #buildstream12:19
*** tristan has quit IRC12:32
*** bochecha has quit IRC12:43
*** bochecha has joined #buildstream12:44
*** bochecha has quit IRC12:48
*** bochecha has joined #buildstream12:49
bochechais there a way to tell BuildStream to ignore some files in a source tree?14:03
bochechaI'm using the following to build an app from its git tree:14:04
bochechasources:14:04
bochecha- kind: local14:04
bochecha  path: .14:04
bochechabut then if I just open a source file in vim, that creates a .swp file, bst sees the new file and thinks I need to rebuild the app14:04
ssam2using the workspaces feature ?14:05
ssam2currently not, I think ...14:05
tlaterssam2: Would have the same result, because vim would create a file in the workspace, which buildstream would then find.14:05
ssam2i think i wasn't clear :-)14:06
ssam2i'm guessing bochecha is using workspaces, and saying if so, there's no way to tell buildstream to ignore certain files when deciding whether a rebuild is needed14:06
ssam2a specific workaround would be to configure vim not to put its swap files in the current directory14:07
ssam2i have vim configured to put things in ~/tmp, because i've had other issues in the past from it dumping files in places i didn't want it to14:07
ssam2https://stackoverflow.com/questions/1636297/how-to-change-the-folder-path-for-swp-files-in-vim#21026618 for example14:08
ssam2for a general fix, we have been debating involving source plugins in workspaces14:09
ssam2which would be an optimisation only14:09
gitlab-br-botbuildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/11914:09
gitlab-br-botbuildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/11914:09
ssam2but if we did this, and we delegated the decision on whether to rebuild to Git, and required that the user `git add` any new files that they cared about ... then we could reuse .gitignore for this14:09
ssam2assuming your workspace is for a project stored in Git, which i'm 90% sure it will be14:09
ssam2but anyway, i think short answer is: no14:10
gitlab-br-botbuildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/11914:12
bochechassam2: I'm not using workspaces, no14:13
ssam2ah, right yeah I didn't read you properly14:14
ssam2same answer, though14:15
bochechaI haven't had time to look into workspaces yet though, maybe I should14:15
ssam2yes, i think the "right" way to do this is workspaces14:15
ssam2in practice, i think you don't gain anything at present14:15
bochechaI wish there as a .bstignore, with a similar syntax to .gitignore :P14:15
ssam2but there is ongoing work on making incremental builds work from a workspace14:15
ssam2so in future there will be a benefit to switching :-)14:15
ssam2it's pretty simple too; you just do `bst workspace open ./my-element.bst ./workspace-dir`14:16
ssam2all the app code ends up in workspace-dir and you can hack on it there14:16
ssam2and assuming it's a git repo, it'll be a proper git clone14:16
tlaterssam2: You would have to adjust the bst element to load from a local git repo though, would that be as simple as setting the url to a local directory?14:16
persiaMy worry about the potential for .bstignore is that folk might want to put things in it that change often, but end up being important to build (as is often done with git).  Files in .gitignore just end up getting rebuilt on clone (or people get fussy), but files in ,.bstignore would break the link between "this is changed" and "this is not changed".14:16
ssam2tlater, ??14:17
* tlater assumes bochecha is building from a local project14:17
ssam2ah I see what you mean14:17
bochechassam2: and I can git push in that repo, the remote will be the right one? (i.e not the one inside the bst cache)14:17
ssam2good question :-)14:17
ssam2that *should* be the case, but not sure if it *is* yet14:17
tlaterssam2: Currently workspaces are detached from their versioning system14:18
ssam2ah, ok14:18
bochechatlater: it's not a local repo, it's pushed to the net somewhere, but I clone it, inside there's the bst stuff, one of which has the `path: .` sources14:18
ssam2so actually, it's probably not worth switching to workspaces yet14:18
bochechamy goal was to keep it easy: clone the repo for the app, then build your modifications14:18
ssam2this is definitely the use case that the feature is intended to support, though14:18
tlaterYeah, it's probably better done with a workspace; that way you don't need a .bst file in the repo14:19
ssam2actually, i just opened a workspace and got a working Git clone14:19
bochechaso I'd have one repo with the app code, and one with the bst files?14:19
ssam2with the correct remote URL14:19
tlaterbochecha: That's how buildstream is intended to be used, I think14:19
ssam2bochecha, ah, i guess again it depends on your case14:19
ssam2i was picturing a 'lots of projects' integration workflow14:20
ssam2if this is rather doing packaging for your project... then local source is probably correct14:20
ssam2which brings us back full circle ...14:20
ssam2maybe self-referential workspaces would be OK though14:20
ssam2this is something i never quite figured out with flatpak-builder14:20
bochechaI don't mind separating the bst stuff to its own repo, I just didn't think about it before14:21
ssam2do you import the app from its "upstream" repo, or as a local source14:21
bochechassam2: with flatpak-builder, from the upstream repo14:21
ssam2ok; so perhaps best practice for bst in the flatpak usecase should be the same14:21
bochechawhich is a huge pain, because `flatpak-builder --from-git...` will clone the repo to get the .json file, then start building that .json file which will clone the upstream repo again :)14:21
bochecha(the same repo for gnome apps shipping their manifest in their source tree)14:22
ssam2yeah, I ended up doing an ugly sed hack in an app's CI system to replace the upstream URL with the local directory14:22
bochecha:)14:22
* tlater wonders if a .bstignore file would make sense with local sources14:24
* ssam2 testing if self-referential workspaces 14:24
ssam2...works14:24
tlater;P14:24
bochechatlater: even with non-local sources, inside a workspace the same problem happens14:24
bochechassam2: what's a self-referential workspace?14:24
bochecha(sorry, gotta afk)14:24
ssam2the idea that you have application and buildstream project in the same repo14:25
ssam2and you open a workspace on that repo14:25
ssam2no reason it shouldn't work really14:25
tlaterYeah, this may be worth considering, despite persia's protest - it may be a huge pain for developers14:25
ssam2in workspaces we could really integrate with the VCS though14:25
ssam2and avoid the need to have a .bstignore and a .gitignore that duplicate each other14:25
persiaVCS integration is the preferable way to go, I think.14:26
tlaterssam2: Yeah, but that would require a `git add`, which makes workspaces a lot less useful for testing14:26
ssam2"a lot" ?14:26
tlaterPerhaps we could parse the workspace's .gitignore instead?14:26
persiaThe use case of bst and code in the same repo is the case where it matters, although I remain unconvinced this is best practice for anything.14:26
ssam2tlater, possibly ... note that we have been discussing VCS integration with workspaces for other reasons14:26
persiaTo my mind, there is vast difference between "this code does things", and "this system consumes that code".14:27
tlaterpersia: I think it matters for workspaces in general, not just this odd case14:27
ssam2tlater, notably as a performance optimisation to avoid having to hash the entire source dir (which may be enourmous) each time we need to know the cache key14:27
ssam2persia, the use case I can see is packaging an app in multiple ways14:27
persiatlater: Why?  What file would belong in .bstignore in the separate-repos case?14:27
ssam2persia, the common use case of flatpak-builder is building an app, where you keep the flatpak-builder .json file in the same repo as the app14:27
tlaterpersia: If you check out a workspace some files don't belong in the workspace, but are created by editors etc.14:28
ssam2persia, so if we aim to replace flatpak-builder, we should really support that :-)14:28
tlaterYou probably don't want to rebuild every time14:28
persiassam2: I think doing that in mainline is the wrong way to do it, as it requires more effort to consume downstream if different defaults are chosen.14:28
persiaI do appreciate that many flatpak people disagree with me :)14:28
ssam2i'm not sure the term "mainline" or "downstream" makes much sense when talking about making simple flatpak apps14:29
persiatlater: If I have an editor open in a workspace, and it updates a e.g. swap file, that likely indicates I edited a file, yes?  In which case I want to rebuild, don't I?14:29
persiassam2: Right.  I think "making a flatpak app" is inherently antisocial.  I think people should make software and make systems (or components).14:30
persiaBut this isn't necessarily the right forum for that debate, and I do appreciate that others feel differently.14:30
ssam2yes14:30
tlaterI suppose it might be rare enough not to matter in practice. I certainly dislike the idea of a .bstignore file.14:31
bochechatlater: why does that require a `git add`?14:31
persiaWell, for this use case, I think it's really important.  Whenever we mix contexts, there is benefit to having some means of hinting about our context, so that we can e.g. edit the .bst file without that causing buildstream to think the source was changed.14:32
tlaterbochecha: If we implemented VCS integration for workspaces you would have to `git add` every time you make a change to rebuild14:32
bochechatlater: but why?14:32
bochechabst could just ignore the files from .gitignore, and take everything else, no?14:32
persiabochecha: Because bst would ask git "what changed", so git would have to know to answer.14:32
tlaterbochecha: That's my idea :)14:32
tlaterbochecha: On the other hand, we also want to minimize the number of source files we have to check - atm we hash the entire source tree, which may take forever14:33
tlaterFor that you *would* need `git add`14:33
bochechathat seems annoying :-/14:33
persiatlater: One potential issue with the `git add` issue: is buildstream building from the current git working area or from the filesystem?14:33
bochecharight now I don't need to `git add` when I create a new source file in my project14:33
persias/add`issue/add` model/14:34
tlaterpersia: If it is a workspace, I assume yes?14:34
persiaIndeed, the answer to nearly every well-formed question involving "or" is yes.  Any idea which?14:34
tlaterpersia: From the filesystem, sorry14:35
persiaThat significantly reduces the utility of using `git add`, as buildstream will buikld from the modified content whether the user remembers to type that or not.14:36
ssam2besides .gitignore, one motivation of using Git to discover what has changed is to avoid hashing every file in the source dir every time we build14:37
ssam2but if we're going to automatically include uncommited files, we still need to scan14:37
persiaThe speed improvement would be welcome.  The trick is making sure that this doesn't break things.14:37
ssam2also, incremental build support complicates this further, as that involves knowing what is build output and what is source14:38
ssam2(actually, it might not, my point is just that there's a bunch of considerations here)14:38
persiaOOh, that is very complicated in the case where the user might edit files.14:38
persiaEspecially with something like `make`, where a user can hand-edit a generated file and run a partial rebuild with the hand-edited generated file changing from output to source as a result of the edit.14:38
* tlater wonders if there is a better heuristic that doesn't require VCS14:43
*** mcatanzaro has joined #buildstream15:00
tlaterssam2: Can the docker source plugin create an image from a local Dockerfile?15:11
ssam2no15:12
ssam2and i don't think we want that15:13
ssam2dockerfile should die in a fire and be replaced with buildstream15:13
tlaterHeh, maybe, but it *is* very useful for import elements15:15
gitlab-br-botbuildstream: merge request (flakes-refactor->master: Remove unused imports) #178 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/17815:16
ssam2yes, having it for import elements is crucial for killing dockerfiles :-)15:16
tlaterIt also means I can't actually test my docker image before I push it15:16
tlaterHm15:16
ssam2why not ?15:16
ssam2you can run `docker build` to build a docker file15:16
ssam2oh, i guess then you need to set up a registry15:17
ssam2which is not so much fun15:17
tlaterYup15:17
ssam2`docker export`15:17
ssam2then import it as a tarball15:17
tlaterThat works? Huh15:17
tlaterTa :)15:17
gitlab-br-botbuildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/16615:32
gitlab-br-botbuildstream: merge request (sam/multiple-caches->master: Multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/16615:33
ssam2juergbi, multiple caches finally ready for review15:34
juergbi\o/ great15:34
nexus\o/15:34
bochechais the project name automatically exposed as a %{...} variable which can be used in elements? if so, what's the name of the variable?15:48
ssam2i don't think it is15:48
bochechait would be useful :)15:49
ssam2what for, out of interest ?15:50
ssam2should be fairly simple to do15:50
bochechaas an install-commands, I want the configuration file to end up in /etc/%{project-name}/15:52
bochechait's not too terrible to have to type the project name explicitly, but it would be even nicer if I could use a variable15:53
ssam2you could also have a 'project-name' variable in your project.conf15:53
ssam2variables:15:53
bochechasure15:53
ssam2  project-name: foo15:53
bochechawhich is what I'm doing right now :)15:53
ssam2ah right15:54
bochechabut since the name is defined a few lines above in the same file... :-P15:54
ssam2yeah... i guess it's probably 1 one line patch to the buildstream core, so i wouldn't block a patch to fix it15:56
bochechaI'll try and send a MR after I'm done with what I'm doing15:57
ssam2we don't have an existing utility function in the core to save to a temporary file, then rename in place when fully written, right ?16:10
ssam2thinking we should have something like https://gitlab.com/baserock/morph/blob/master/morphlib/savefile.py16:10
ssam2i need something like that for https://gitlab.com/BuildStream/bst-external/merge_requests/9 (docker source plugin), but seems silly not to share it16:11
bochechaso I'm looking at the workspaces and having separate repos for the code and the bst stuff... opening a workspace does get me a full git clone which is neat, but it checks out a detached head :-/16:12
bochechaalso, having those separated repo, what would be the workflow for having a CI which runs `bst build` of the app whenever someone sends a merge request to the code repo... the Ci would have to clone the repo with the bst recipes, then have it track the submitted branch, then build the app (which clones again the source code repo from which the CI started)16:15
bochechathat seems so much more cumbersome than having the bst recipes in the source code repo, where the CI can just `bst build app.bst` (with app.bst using the `path: .` source)16:15
tlaterbochecha: I think the CI could use the local path as a workspace16:18
tlaterIIRC workspaces don't have to be empty, you can use any directory with the correct magical invocation16:18
*** noah has joined #buildstream16:18
bochechasure, but the CI definitions are in the source code repo16:18
ssam2i do see your point on it being more cumbersome16:18
bochechasince that's the one people are working on16:18
bochechathe workspace is created from the source code repo, but in the bst recipes repo16:19
ssam2maybe there's some way we could extend the 'local' source to ignore files16:19
*** noah has quit IRC16:19
*** noah has joined #buildstream16:21
tlaterbochecha: You could clone the bst files repo in the source code repo, that way you don't have to move CI definitions16:21
tlaterIt's a little more cumbersome, but I think having to clone the bst files in the dev repo is to be expected?16:21
bochechatlater: I didn't expect it16:22
bochechaI thought workspaces meant I wouldn't clone the source code repo manually any more :)16:22
tlaterbochecha: You can :)16:22
bochechaI'd just clone the bst recipes repo, do `bst workspace open app.bst` and I'd have the source code repo cloned for me16:22
tlaterbst workspace open --no-checkout -f16:22
tlaterYou *can* still do it, but cloning it with the workspace command is more convenient in development16:23
bochechayes16:23
tlaterFor CI obviously you'd use the obscure flags to ensure that you don't have to clone through bst16:23
bochechawhich gets me back to my question: why would I clone the bst recipes repo from the source code repo?16:23
tlaterAh, fair enough16:23
* tlater misread that16:24
bochechabasically, so far the only workflow that makes sense to me is having the bst recipes in the source code repo, because then everything (both dev and CI) is simple and elegant16:24
bochecha(except those pesky .swp files of course :P)16:24
ssam2perhaps we could prototype this with a special kind of 'local' source plugin16:25
ssam2maybe 'git-local'16:25
ssam2which would be like 'local', except require it to be git, and honour .gitignore16:25
tlaterssam2: That seems a bit overly specific, but alright for a prototype16:26
* tlater observes some gcc breakage using alpine linux16:27
tlaterssam2: Any chance this might happen because I'm using `docker export`?16:27
ssam2what's the breakage ?16:28
bochechassam2: interesting idea16:28
bochechaI'll try and make a git-local plugin then :)16:28
tlaterssam2: <long path>/ld: cannot find Scrt1.o: No such file or directory16:28
*** noah has quit IRC16:28
ssam2bochecha, cool :-)16:28
ssam2tlater, when doing what ?16:28
tlaterssam2: Running ./configure on nasm16:29
ssam2and does that work inside a container started with `docker run `?16:29
tlaterIt's the very first, basic thing gcc tries to compile to see if gcc works16:29
* tlater hasn't tried, but assumed so because it's installed through a package manager.16:29
* tlater attempts16:29
*** noah has joined #buildstream16:30
tlaterAh, nvm, it doesn't work16:31
* tlater checks alpine linux' notes on gcc16:31
ssam2packaging there is very minimal, you're probably just missing some package16:32
gitlab-br-botbuildstream: merge request (sam/savefile-utility->master: utils.py: Add SaveFile utility class) #179 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/17916:39
* tlater needed g++ -.-16:50
bochechaor instead of a git-local plugin, how about making the git plugin aware of local sources?16:51
bochechaas in, if the url is not a remote one (https://, git://, ...) but a local folder, then that does what we were discussing for a git-local plugin?16:51
tlaterbochecha: I think technically that's already possible if you specify a local url with file://16:51
bochechatlater: right, but then it will still clone the local url, and uncommitted changes won't be taken into account for the build16:52
*** noah has quit IRC16:54
*** mcatanzaro has quit IRC16:55
*** noah has joined #buildstream16:56
*** mcatanzaro has joined #buildstream16:57
*** noah has quit IRC16:58
*** noah has joined #buildstream17:00
ssam2file:/// urls also can't be relative17:03
ssam2so would need some special casing17:03
ssam2i can see advantages and disadvantages to both ways17:03
bochechait's possible, but it's a pain17:03
bochechaI'll look at a git-local plugin, it seems safer as a first draft anyway17:04
bochechashould I add that to bst-external?17:04
ssam2makes sense, yeah17:05
*** bethw has quit IRC17:29
*** ssam2 has quit IRC17:33
*** jude has quit IRC17:38
*** tristan has joined #buildstream17:45
*** tpollard has quit IRC17:46
*** noah has quit IRC17:46
bochechaso I made a new source plugin in bst-external, I ran `python3 setup.py install --user` for bst-external, then my project can't use the source plugin:17:52
bochechaError loading pipeline: No Source type registered for kind 'git-local'17:52
bochechaI did add the new source plugin entry point to bst-external/setup.py17:53
*** noah has joined #buildstream18:00
*** noah has quit IRC18:39
*** noah has joined #buildstream18:45
*** noah has quit IRC18:53
*** valentind has joined #buildstream19:43
bochechaI don't understand how to load external plugins20:19
*** mcatanzaro has quit IRC20:22
bochechadigging into the code with pdb, I find this: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_plugincontext.py#L13320:24
bochechastepping into it, it tries to import the buildstream.plugins.git_local module20:25
bochechawhich of course doesn't exist, it should be bst_external.sources.git_local20:27
bochechaI'm probably missing something here, because I don't see how external plugins can ever be loaded :-/20:27
*** valentind has quit IRC23:19
*** mcatanzaro has joined #buildstream23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!