IRC logs for #buildstream for Wednesday, 2019-06-12

*** jswagner has quit IRC03:46
*** jswagner has joined #buildstream03:47
*** jswagner has joined #buildstream03:50
*** jswagner has joined #buildstream03:53
*** jswagner has joined #buildstream03:57
*** jswagner has joined #buildstream03:59
*** jswagner has joined #buildstream04:00
*** jswagner has joined #buildstream04:02
*** jswagner has joined #buildstream04:04
*** tristan has quit IRC05:42
*** tristan has joined #buildstream05:52
*** bochecha has joined #buildstream07:50
juergbibenschubert: do you know what's wrong in my setup if I get this:08:27
juergbiTypeError: C function buildstream._yaml.node_keys has wrong signature (expected PyObject *(PyObject *, int __pyx_skip_dispatch), got PyObject *(struct __pyx_obj_11buildstream_5_yaml_Node *, int __pyx_skip_dispatch))08:27
juergbiI have rerun pip install but I haven't recreated the venv (yet)08:28
juergbiis there a bug in automatic rebuilding?08:28
benschubertjuergbi: argh, apparently a file depending on _yaml.pxd didn't notice _yaml.pxd got changed. Can you delete src/buildstream/_*.c* ?08:28
benschubertyep08:28
benschubertI can't find the cause of this08:28
juergbithat helped, thanks08:29
juergbimight there be an easy way to regenerate all .c files whenever any .pxd changes. wouldn't be optimal but better than rebuilding too little08:31
benschubertagreed, I've been looking into it, the dependency is apaprently there, but the system doesn't realize it changed08:34
*** raoul has joined #buildstream08:46
jennisjuergbi, benschubert, we found that `git clean -xfd` before the pip install also worked08:47
benschubert^ don;t do this if you have an IDE configuration in the directory08:49
jennisyeah, I should have said this can be dangerous08:49
aevriThis worked for me:  git clean *.c -fx08:50
*** tristan has quit IRC08:50
juergbigit clean -dfX is something I sometimes use in C projects08:51
juergbionly deletes gitignored files08:51
juergbias long as gitignore only includes generated files, this works really well08:51
benschubertbut people with vscode/pycharm will have a local configuration for the IDE in the directory, so it will wipe it too :)08:52
juergbibenschubert: is your vscode config covered by .gitignore?08:52
aevri`.tox` dir is also precious :)08:52
juergbitrue, .tox is not so cheap to rebuild08:52
benschubertjuergbi: I have a global gitignore for all those kind of folders08:53
juergbiah, git clean probably uses that as well08:53
benschubertsadly yep :)08:53
gitlab-br-botaevri opened (was WIP) MR !1391 (aevri/supersuper->master: Refactor 'super(cls, self)' -> 'super()') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139109:00
*** tristan has joined #buildstream09:03
*** phildawson_ has joined #buildstream09:03
gitlab-br-botBenjaminSchubert approved MR !1391 (aevri/supersuper->master: Refactor 'super(cls, self)' -> 'super()') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139109:05
gitlab-br-botmarge-bot123 merged MR !1391 (aevri/supersuper->master: Refactor 'super(cls, self)' -> 'super()') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139109:09
aevriwhoosh :)09:10
benschubertQuestion: I have a git repository containing multiple directories with each having a project I want to build. Is there currently any way to only re-build something if the subdirectory has changed and not everytime the main git source changes?09:14
*** jonathanmaw has joined #buildstream09:21
tpollardjennis: just looking at the artifact subcommand MR you pointed out, I think a lot of it can probably be simplified now with the mapping of the element to it's artifact class member09:34
jennisoh that sounds like good news09:36
jennisWe'd still need to load ArtifactElement's for specified refs though too, right?09:37
tpollardI'd have to dive into it more, and think about AaaP in this scenario too09:41
*** lachlan has joined #buildstream09:41
tpollardstoring metadata in the proto, and expanding on what is serialised into it at creation gives some nice scope for fleshing out Artifact referencing09:46
gitlab-br-botaevri opened (was WIP) MR !1382 (aevri/win32_temptext->master: _artifact: use _TempTextBuffer instead of tmp file) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/138209:49
*** samkirkham has joined #buildstream09:52
*** ChanServ sets mode: +o tristan09:54
tristanbenschubert, I think I understood the question09:54
tristanbenschubert, What you mean is this is a git repository of source code, and you are referring to that git repository in an element, but only use a subset of that element... something like this correct ?09:55
tristanbenschubert, The answer is unfortunately no - but there was some discussion on build avoidance on the source and artifact level which would be a way forward to reduce this09:55
tristanand avoid a rebuild09:55
tristanbenschubert, However that approach I'm thinking of might not work if for instance, you need to actually stage the whole tree09:57
tristanbenschubert, i.e. the idea is essentially... if everything that was staged has not changed (or matches a key), then we can reuse the output - even if the refs have technically changed - this would also avoid rebuilds on say, superficial git merge commits and the like09:58
tristanbenschubert, for the case of "building a subdir" (like we do for example when we build spidermonkey only, but need to build it from the mozilla source tree), then we'd have the source stage only the subdir we need to build - some caching magic would allow us to identify that we've already built something with exactly the same input (even if surrounding directories have changed)10:00
benschuberttristan: yep you got it :) I only need the subtree :)10:01
tristanYup, that would be a great feature and would help at multiple levels :)10:03
tristanFrom what Sander and I had discussed before... it would also address another point... for instance let's say that the code *has* changed, but it produces the same output10:04
tristanThen we could alias that output to the new key, and at least avoid builds of reverse dependencies10:04
tristanI mean basically, the same thing that we could do with SourceCache, we could also do with ArtifactCache10:04
paulsherwoodbenschubert: why are all of thes projects in the same repo?10:20
* paulsherwood is concerned that the projects might overlap, so that a change in one subdir might actually require rebuild of something in another10:20
*** tpollard has quit IRC10:23
*** tpollard has joined #buildstream10:23
*** lachlan has quit IRC10:27
samkirkhamIs there a way to set the uid/gid in the integration sandbox (not the build sandbox) of an element?10:32
benschubertpaulsherwood: no idea, and I don't have control over that10:32
benschubertbut I've been guaranteed they don't overlap :D10:33
tristansamkirkham, nice observation10:36
tristansamkirkham, currently the uid/gid configuration is bound to the element, not specific sandbox invocations10:36
tristansamkirkham, so I suppose it would be a bug that currently we consider those to be the same (i.e. if we wanted to run build commands as a regular user with uid/gid 1000, it would still be wrong that we are running integration commands not as root)10:37
samkirkhamHmm, ok. I'm having problems with some integration commands I want to run. I think it's related to this. atm I can't see a reliable way to test these commands either because the same issue crops up (wrong uid in a shell for testing)10:39
samkirkhamIt may be better for me to just run these commands on first boot, but I'm non sure what the canonical way to do that is. I'd rather not add *too* much complexity.10:40
tristansamkirkham, an option would be to `bst checkout` the tree and try running bwrap on it directly ?10:40
tristanfor testing / debugging10:40
tristansamkirkham, What kind of commands are these btw ?10:41
samkirkhamssh-keygen so I can have an openssh server service running via systemd10:41
tristanKnow that BuildStream currently does not preserve uid/gid bits or any fs attributes beyond 0755/0644, no setuid, nothing10:41
samkirkhamYeah, I'm expecting to have to set some attribute bits when I start encountering problems.10:42
samkirkhamNot ideal but I've been told this already.10:42
tristansec...10:43
tristanhttps://gitlab.gnome.org/GNOME/gnome-build-meta/blob/tristan/gnome-boot/elements/vm/desktop-vm-image-x86_64.bst#L3110:43
tristanAttributes set in the sandbox directly before wrapping up an image with mkfs works10:43
tristansamkirkham, but iirc, you know that because we discussed this on #freedesktop-sdk ?10:44
tristansamkirkham, So another thing... I mean... you technically *can*, but it's really not intended that you should be able to run ssh-keygen in the build sandbox at all10:44
tristansamkirkham, i.e... like we first tried to build systems without having any /dev/urandom at build time, but that just doesnt work10:45
samkirkhamI do. My plan was to use a different tool for creating the FS because of some weirdness on my particular embedded platform10:45
tristanbut generating a key that is intentionally random means that your builds are intentionally non-reproducible10:45
samkirkhamYeah maybe ssh-keygen is better run on first boot anyway.10:45
samkirkhamAlthough again I'm not sure what the recommended way to do that is here.10:47
tristanHmm, systemd may have a facility for this ?10:47
*** lachlan has joined #buildstream10:47
tristanone-off initialization things ?10:48
tristanit does other stuff, like generating users - probably one can set something up to run once10:48
samkirkhamI suspect it does. Just wasn't sure if there was a bst-canon way to do it. Seems like not so I can just see what systemd has to offer =) Thanks.10:49
tristansamkirkham, just seems logical to me... also using a .service file would allow you to set other service files to depend on that thing having already been done once10:49
tpollardyou'll probably want systemd-firstboot.service11:07
*** bochecha_ has joined #buildstream11:15
*** bochecha has quit IRC11:18
*** bochecha_ is now known as bochecha11:18
*** bochecha has quit IRC11:42
*** bochecha has joined #buildstream11:43
*** lachlan has quit IRC12:07
*** tristan has quit IRC12:12
gitlab-br-botaevri opened (was WIP) MR !1393 (aevri/psutil_volume->master: utils._get_volume_size: use shutil.disk_usage) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139312:28
*** phil has joined #buildstream12:42
*** CTtpollard has joined #buildstream12:43
*** CTtpollard has quit IRC12:43
*** CTtpollard has joined #buildstream12:43
*** phildawson_ has quit IRC12:44
*** tpollard has quit IRC12:44
gitlab-br-botaevri opened (was WIP) MR !1390 (aevri/pylint_argsdiff->master: enable PyLint "arguments-differ" error, and fixup complaints) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139013:06
* tlater[m] can't seem to run the integration tests because of an import failure for some of the magic cython functions13:08
tlater[m]Did I miss a PSA about refreshing a cache or somesuch?13:09
aevriIs it this? TypeError: C function buildstream._yaml.node_keys has wrong signature (expected PyObject *(PyObject *, int __pyx_skip_dispatch), got PyObject *(struct __pyx_obj_11buildstream_5_yaml_Node *, int __pyx_skip_dispatch))13:09
aevriIf so `git clean *.c -fx` and re-doing the pip install should do the trick.13:10
tlater[m]aevri: Not quite. It's `extract_depends_from_node` in `loader.py:31`13:10
tlater[m]Trying `git clean` in case it's some pycache screwup anyway...13:10
aevriCan you paste the error line? Maybe it's some new fun :)13:11
tlater[m]aevri: Is pip install required now?13:11
aevriOh, if you haven't done that since the cython changes then sort of yes13:11
tlater[m]I thought tox was smarter than that13:11
aevriOh tox should be fine afaik, you will need to `tox -r` at least once I think though, to get cython in the first place13:12
tlater[m]Gotcha, thanks :)13:13
* tlater[m] has indeed not run the test suite in a while13:13
*** lachlan has joined #buildstream13:15
tlater[m]aevri: And `tox -r` did indeed resolve the problem. I need to sit down and grog tox at some point.13:15
aevriAh great, yeah me too :)13:16
*** phil has quit IRC14:07
*** phil has joined #buildstream14:07
*** rdale has quit IRC14:43
*** rdale has joined #buildstream14:44
tlater[m]So I'm trying to get freedesktop-sdk to build as part of the BuildStream examples docs - this works, for the most part, but it requires bst-external, which - because of some recent change - doesn't work with master at all anymore.14:54
tlater[m]Instead I should probably be using bst-plugins-experimental anyway14:54
tlater[m]The only problem with this is that freedesktop-sdk depends on specifically bst-external, and the two packages don't share a package name.14:54
tlater[m]Why did we change the package name? Is there some way to reconcile this for when any projects requiring those dependencies will inevitably need to convert?14:55
tlater[m]Should I start a freedesktop-sdk-buildstream-master branch?14:55
tlater[m]Would there be any problem with simply renaming the bst-plugins-experimental package so that pip dependencies still resolve correctly?14:57
*** CTtpollard is now known as tpollard14:59
* tlater[m] also wonders if git-tag should perhaps be merged into git as part of all these plugin changes15:01
tlater[m]That would resolve a lot of freedesktop-sdk's burden15:01
*** tristan has joined #buildstream15:04
tpollardtlater[m]: does it depend of bst-external specifically, I'm sure I built some stuff with plugins-experimental recently?15:07
tlater[m]tpollard: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/project.conf#L28415:09
tlater[m]And alas, experimental won't work with that, since this makes it a non-drop-in replacement: https://gitlab.com/BuildStream/bst-plugins-experimental/blob/master/setup.py#L3215:10
tlater[m]If it worked for you (which it did for me), this is likely because there were no breaking changes for the particular plugins you were using15:10
tlater[m]But now I get ```[synthetic node]: Value of 'submodules' is not of the expected type 'Mapping'```15:11
tlater[m]Which is resolved in a recent commit on experimental15:11
tlater[m]But breaks every external plugin15:11
tpollardYep :/15:14
gitlab-br-botmarge-bot123 merged MR !1393 (aevri/psutil_volume->master: utils._get_volume_size: use shutil.disk_usage) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139315:14
benschuberttlater[m]: we do build freedesktop as part of the CI every night, can you use the same branch?15:24
tlater[m]benschubert: Which branch is that?15:26
benschuberttlater[m]:  freedesktop-sdk-18.08.31.1-2-g5c639b8d553016ea6c28ea160a572da2e189745e15:28
benschubertyou can see that in .overnight-tests in .gitlab-ci.yml15:28
* tlater[m] can't see freedesktop-sdk in .gitab-ci.yml15:28
benschuberthttps://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml#L22615:29
tlater[m]benschubert: of buildstream-plugins-experimental?15:29
benschuberttlater[m]: buildstream15:29
benschubertthe full link is ^ :)15:29
tlater[m]Or some other repository?15:30
tlater[m]Ah, gotcha15:30
tlater[m]Hrm, I suppose with any luck that will be merged once 2.0 lands, so this workaround will drop when the docs are actually in use.15:31
* tlater[m] is still not sure if the migration story is set straight, though15:33
benschubertprobably not, but buildstream 1.4 might help a bit :)15:35
gitlab-br-botjuergbi opened MR !1395 (juerg/cas-dead-code->master: cascache.py: Remove unused methods commit() and _commit_directory()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139515:48
gitlab-br-botBenjaminSchubert approved MR !1395 (juerg/cas-dead-code->master: cascache.py: Remove unused methods commit() and _commit_directory()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139515:50
*** lachlan has quit IRC15:56
*** bochecha has quit IRC16:05
*** lachlan has joined #buildstream16:07
*** phil has quit IRC16:32
*** phil has joined #buildstream16:32
gitlab-br-botmarge-bot123 merged MR !1395 (juerg/cas-dead-code->master: cascache.py: Remove unused methods commit() and _commit_directory()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139516:46
*** tpollard has quit IRC16:50
*** samkirkham has quit IRC17:00
*** adds68 has quit IRC17:00
*** ikerperez has quit IRC17:01
*** adds68 has joined #buildstream17:10
*** paulsherwood has quit IRC17:14
*** benbrown has quit IRC17:15
*** adds68 has quit IRC17:16
*** paulsherwood has joined #buildstream17:17
*** rdale has quit IRC17:18
*** jennis has quit IRC17:20
*** WSalmon has quit IRC17:20
*** bethw has quit IRC17:21
*** johnward has quit IRC17:21
*** valentind has quit IRC17:21
*** laurence has quit IRC17:21
*** valentind has joined #buildstream17:23
*** laurence has joined #buildstream17:23
*** valentind has quit IRC17:30
*** laurence has quit IRC17:30
*** paulsherwood has quit IRC17:30
*** phil has quit IRC17:32
*** bochecha has joined #buildstream17:42
*** bochecha has quit IRC17:43
*** lachlan has quit IRC17:45
*** lachlan has joined #buildstream17:53
gitlab-br-botBenjaminSchubert opened MR !1397 (bschubert/create-node-on-set-for-mapping->master: _yaml: Create Node when setting a dictionary) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139717:55
*** lachlan has quit IRC18:00
*** jonathanmaw has quit IRC18:04
*** lachlan has joined #buildstream18:17
*** lachlan has quit IRC18:33
*** bochecha has joined #buildstream19:21
*** toscalix has joined #buildstream19:57
gitlab-br-botBenjaminSchubert closed MR !1397 (bschubert/create-node-on-set-for-mapping->master: _yaml: Create Node when setting a dictionary) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/139721:23
*** paulsherwood has joined #buildstream21:33
*** bethw has joined #buildstream21:45
*** jennis has joined #buildstream21:46
*** valentind has joined #buildstream21:46
*** WSalmon has joined #buildstream21:47
*** laurence has joined #buildstream21:47
*** johnward has joined #buildstream21:47
*** toscalix has quit IRC22:00
*** bochecha has quit IRC22:06

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!