| *** tristan has joined #buildstream | 00:29 | |
| *** xjuan has quit IRC | 00:45 | |
| *** xjuan has joined #buildstream | 01:00 | |
| *** nimish has joined #buildstream | 01:00 | |
| *** xjuan has quit IRC | 01:09 | |
| *** nimish has quit IRC | 03:23 | |
| *** tristan has quit IRC | 06:20 | |
| *** alatiera has joined #buildstream | 07:20 | |
| *** toscalix has joined #buildstream | 08:52 | |
| *** jonathanmaw has joined #buildstream | 09:46 | |
| *** raoul has joined #buildstream | 09:58 | |
| valentind | While building Freedesktop SDK with BuildStream and BuildGrid, I got: "Element plugin 'script' does not support virtual directories." | 10:03 |
|---|---|---|
| valentind | Is that an oversight? | 10:03 |
| valentind | jmac, probably you know about that ^ | 10:06 |
| jmac | Hmm! I've built freedesktop sdk with remote execution recently, so that's odd | 10:07 |
| valentind | Unless it was something merged recently. | 10:09 |
| valentind | I am using a branch. | 10:09 |
| jmac | Script needs a real directory to run, so it's not an oversight; it cannot support virtual directories locally. | 10:10 |
| jmac | What's the working branch for freedesktop-sdk? master does not seem to have moved since August. | 10:12 |
| valentind | jmac, I get also that: https://paste.gnome.org/po8vhxayd | 10:12 |
| valentind | jmac, it is 18.08 | 10:12 |
| jmac | The second error is completely unknown to me, sorry | 10:12 |
| juergbi | jmac: why does script need a real directory? | 10:13 |
| valentind | jmac, I see, it is not an error, for the virtual directory but a warning. | 10:13 |
| valentind | Before it says: "WARNING [script:remote-failure]: Artifact base/locales.bst is configured to use remote execution but element plugin does not support it." | 10:13 |
| valentind | juergbi, I suppose it is more complicated because of the layout. | 10:14 |
| juergbi | I can't think of a reason why virtual directories couldn't be used with a custom layout | 10:15 |
| jmac | juergbi: I don't see how else you could run commands | 10:16 |
| juergbi | the actual script commands still run in the sandbox | 10:17 |
| jmac | Yes. Nothing that requires a sandbox can use virtual directories, can it? | 10:18 |
| jmac | Oh, virtual vs CAS, I suppose | 10:18 |
| juergbi | sandbox.run() is also used by build elements | 10:19 |
| juergbi | and can use a local or a remote sandbox | 10:19 |
| juergbi | don't see a significant difference between build elements and script elements in this regard | 10:19 |
| juergbi | the main special part of script elements is the layout functionality (+ support for read-write rootfs) | 10:19 |
| jmac | Yes, they can use virtual directories then, they just have to be file-based ones | 10:19 |
| valentind | jmac, by the way, when compiling freedesktop-sdk, I had this issue with BuildGrid. Did not you? https://gitlab.com/BuildGrid/buildgrid/issues/150 | 10:19 |
| jmac | valentind: Nope, not seen that either | 10:20 |
| juergbi | jmac: why file-based? | 10:20 |
| jmac | Sorry, you're both now hitting me with too many questions, I'll have to come back to this later | 10:21 |
| juergbi | ok, no problem | 10:22 |
| valentind | jmac, juergbi, I will create an issue for virtual directories in script. That will be more like a placeholder not to forget. Please feel free to edit it. | 10:24 |
| gitlab-br-bot | valentindavid opened issue #850 (script plugin should support virtual directories) on buildstream https://gitlab.com/BuildStream/buildstream/issues/850 | 10:27 |
| valentind | Yes. It is #850 | 10:27 |
| *** lachlan has joined #buildstream | 10:31 | |
| *** tintou has left #buildstream | 10:45 | |
| jmac | Right, ten minutes between meetings | 10:48 |
| jmac | juergbi: I don't see how you could run a sandbox locally with a CAS-based file system; you'd need to either export it all to a normal filesystem then repack it, or have the CAS FUSE running locally | 10:49 |
| *** ahmed89 has joined #buildstream | 10:51 | |
| *** slaf has quit IRC | 10:52 | |
| *** slaf has joined #buildstream | 10:54 | |
| *** slaf has joined #buildstream | 10:54 | |
| *** slaf has joined #buildstream | 10:54 | |
| *** slaf has joined #buildstream | 10:54 | |
| *** slaf has joined #buildstream | 10:55 | |
| valentind | jmac, there is something I need to understand here. Maybe the documentation of BST_VIRTUAL_DIRECTORY is incomplete. It says it should be false if Sandbox.get_directory is called. I do not see where in ScriptElement or script where Sandbox.get_directory is called. So in my opinion BST_VIRTUAL_DIRECTORY should be true. | 10:58 |
| jmac | Sorry, been called into another meeting now :( | 10:59 |
| *** phildawson_ has quit IRC | 11:00 | |
| *** phildawson_ has joined #buildstream | 11:01 | |
| *** raoul_ has joined #buildstream | 11:01 | |
| *** raoul has quit IRC | 11:02 | |
| *** raoul_ has quit IRC | 11:10 | |
| *** phildawson_ has quit IRC | 11:35 | |
| adds68 | hey jonathanmaw, everything has been resolved for the plugin here: https://gitlab.com/BuildStream/bst-external/merge_requests/60, when you get 5 minutes could you review please? | 11:36 |
| *** phildawson_ has joined #buildstream | 11:39 | |
| *** raoul_ has joined #buildstream | 11:39 | |
| *** slaf has quit IRC | 12:19 | |
| *** slaf has joined #buildstream | 12:23 | |
| *** slaf has joined #buildstream | 12:24 | |
| *** cs-shadow has joined #buildstream | 12:42 | |
| jmac | valentind: I believe you're correct; we can set BST_VIRTUAL_DIRECTORY in script | 12:45 |
| jmac | I tried building freedesktop-sdk, and it's thrown this error: Pip package buildstream-external does not contain a plugin named 'git_tag' | 12:45 |
| valentind | jmac, OK, I will try to build freedesktop sdk with that. | 12:45 |
| valentind | jmac, you need to install or update bst-external. | 12:46 |
| jmac | bst-external is there and at current master | 12:48 |
| jmac | Ah, it needs manually installing as well | 12:48 |
| jmac | Got it | 12:48 |
| valentind | jmac, I think you need the latest release. | 12:49 |
| valentind | You can check with git hash in .gitlab-ci.yml | 12:49 |
| jmac | It's building now, thanks | 12:50 |
| valentind | jmac, so the other issue I talked about before is with buildgrid. Apparently, it does not like when stdout or stderr of a job is too big. | 12:51 |
| jmac | Right, interesting | 12:52 |
| valentind | The workaround is to disable stdout/stderr in the response from the bot. But then we do not get anything in the log. | 12:53 |
| gitlab-br-bot | valentindavid closed issue #780 (Remote execution: TLS support for execution service) on buildstream https://gitlab.com/BuildStream/buildstream/issues/780 | 12:59 |
| gitlab-br-bot | valentindavid closed issue #631 (Remote-execution user configuration (technical debt)) on buildstream https://gitlab.com/BuildStream/buildstream/issues/631 | 12:59 |
| gitlab-br-bot | valentindavid merged MR !1030 (valentindavid/remote_execution_configuration->master: Remote execution configuration: HTTPS and user configuration) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1030 | 12:59 |
| *** finn has joined #buildstream | 13:00 | |
| *** raoul_ has quit IRC | 13:08 | |
| valentind | jmac, it looks like script runs fine remotely. | 13:29 |
| valentind | I will make a merge request. | 13:29 |
| jmac | Cool, thanks. | 13:53 |
| *** raoul_ has joined #buildstream | 13:54 | |
| gitlab-br-bot | valentindavid opened MR !1047 (valentindavid/script_virtual_directory->master: buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1047 | 14:02 |
| gitlab-br-bot | jmacarthur opened MR !1048 (jmac/rename_instance_name->master: Rename 'instance_name' option to 'instance-name') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1048 | 14:05 |
| valentind | jmac, found another bug. When an element produces an empty artifact, then we get a failure: | 14:05 |
| valentind | "FAILURE Directory '/buildstream-install' was not found inside the sandbox, unable to collect artifact contents" | 14:05 |
| valentind | In freedesktop sdk it is split-locales.bst. Sure it should be a "stack" here instead of "manual". But there is no point failing on empty artifacts in my opinion. There would be probably cases were some elements are just empty for some architectures and not other for example. | 14:07 |
| jmac | I'll take a look at what creates that directory | 14:08 |
| jmac | I agree, empty artifacts shouldn't be a problem | 14:08 |
| valentind | I am creating an issue for that. You can follow up on it. | 14:08 |
| gitlab-br-bot | valentindavid opened issue #851 (Element with empty artifacts are disallowed by remote execution build) on buildstream https://gitlab.com/BuildStream/buildstream/issues/851 | 14:12 |
| valentind | jmac, #851 | 14:13 |
| mablanch | valentind: For the BuildGrid log problem, you can try that: https://gitlab.com/BuildGrid/buildgrid/merge_requests/138 | 14:13 |
| valentind | mablanch, yes, I am going to launch it now. | 14:13 |
| mablanch | valentind: Great, thanks! | 14:13 |
| *** finn has quit IRC | 14:18 | |
| valentind | mablanch, It does not fail. | 14:26 |
| valentind | But I am wondering why I do not see much in the logs in buildstream. | 14:27 |
| jmac | valentind: The confusing thing is that by my reading, we've never allowed empty artifacts before. Is it possible that element isn't meant to have empty output? | 14:27 |
| valentind | I wonder if maybe in BuildStream for remote jobs, we do not log the commands and output. | 14:27 |
| valentind | jmac, Well, the artifact is not empty since it contains other stuffs than the files. | 14:28 |
| valentind | But there is no file. | 14:28 |
| valentind | I can rephrase the issue. | 14:29 |
| valentind | jmac, I rephrased to make the issue a bit more precise. | 14:32 |
| mablanch | valentind: Yes, I'm not sure either if BuildStream is actually making any use of the returned logs. Do you remember seeing any of it before you switch to that BuildGrid branch? | 14:33 |
| *** finn has joined #buildstream | 14:35 | |
| gitlab-br-bot | jmacarthur approved MR !1047 (valentindavid/script_virtual_directory->master: buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1047 | 14:43 |
| valentind | The command logs are missing in "bst artifact log". I will try with buildgrid master to see. | 14:43 |
| gitlab-br-bot | valentindavid closed issue #850 (script plugin should support virtual directories) on buildstream https://gitlab.com/BuildStream/buildstream/issues/850 | 14:44 |
| gitlab-br-bot | valentindavid merged MR !1047 (valentindavid/script_virtual_directory->master: buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1047 | 14:44 |
| jmac | juergbi: !1048 changes instance_name to instance-name in the config, as you spotted. If you can approve that I'll merge it. | 14:45 |
| gitlab-br-bot | MR !1048: Rename 'instance_name' option to 'instance-name' https://gitlab.com/BuildStream/buildstream/merge_requests/1048 | 14:45 |
| gitlab-br-bot | valentindavid approved MR !1048 (jmac/rename_instance_name->master: Rename 'instance_name' option to 'instance-name') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1048 | 14:47 |
| valentind | mablanch, It is a BuildStream issue that the logs do not appear. | 14:48 |
| valentind | jmac, I see that the logs of commands in remote execution are not added to the logs in the artifact. Is there an issue tracking the work for that? | 14:51 |
| jmac | Not that I know of. | 14:53 |
| valentind | I am sure it is an important feature we need. | 14:53 |
| valentind | Interestingly, if a build fails, it also complains of the install-root directory not to be there. But only this. We do not get the output from the failing command. | 14:56 |
| valentind | Oh, but I guess what happens is that we are supposed to get the logs through the artifact. But we cannot create the artifact because we are missing the output directory. | 14:57 |
| *** nimish has joined #buildstream | 14:59 | |
| jmac | I haven't paid much attention to build logs yet, to be honest. I do think they're important, but I'd like to get more basic functionality working with CI tests before adding more. | 14:59 |
| jennis | Is it possible to obtain an element's cache key in the test suite? | 15:04 |
| *** nimish has quit IRC | 15:04 | |
| *** nimish has joined #buildstream | 15:05 | |
| valentind | jennis, cli.get_element_key maybe. | 15:06 |
| jennis | ahh, i missed that, that sounds like exactly what I need, thanks valentind | 15:08 |
| laurence | benschubert, I can see a lachlan....is your query from Friday still relevant ? | 15:09 |
| benschubert | Yep! thanks for pinging me laurence ! | 15:09 |
| benschubert | lachlan: I have questions on benchmarking when you have time! I want to compare my branch against master, in terms of profile (pstat), memory usage, and speed (on three different benchmarks, to keep things separate). Ideally with a set of canonical repositories used for Benchmarking. Is there currently a way of doing (part) of this? | 15:10 |
| *** mohan43u has quit IRC | 15:14 | |
| *** mohan43u has joined #buildstream | 15:18 | |
| lachlan | benschubert: Hi at the minute none of the tests a.s | 15:21 |
| lachlan | benschubert: Hi at the minute none of the tests check for memory usage - but speed is determined by the time taken to execute tests | 15:22 |
| benschubert | lachlan: I am more interested in benchmarks than the tests, since the tests are not representative of real life scenarios (in terms of sizes) | 15:23 |
| *** nimish has quit IRC | 15:25 | |
| *** nimish has joined #buildstream | 15:25 | |
| lachlan | benschubert: Sorry to clarify - the benchmarks we have represent a subset of the tests that are defined in the benchmarking repo. As to whether they act as representative real life scenarios, currently one benchmarking test (which represents a real life build) is currently broken due to an architecture nomenclature change. | 15:29 |
| benschubert | lachlan: oh that makes more sense I see. And for those, we only get the timing, not profile neither memory usage, correct? | 15:30 |
| gitlab-br-bot | jmacarthur merged MR !1048 (jmac/rename_instance_name->master: Rename 'instance_name' option to 'instance-name') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1048 | 15:30 |
| gitlab-br-bot | jonathanmaw opened issue #852 (Cover the use of WSL in CI) on buildstream https://gitlab.com/BuildStream/buildstream/issues/852 | 15:31 |
| lachlan | At the minute no memory usage - but I would have thought it would be possible to craft some script that could profile memory usage at the same time as running the underlying build activity | 15:31 |
| lachlan | It is certainly something I would like to look at adding | 15:32 |
| benschubert | ok! And lastly, I ahve looked at the repo, but I am not sure how I can run it, to compare two different branches at their latest commit | 15:32 |
| lachlan | At the minute you would need to craft a benchmarking configuration for a specific branch/commit. | 15:33 |
| benschubert | could you point me to an example? And I assume we then need to compare by hand or do we have utilities for this? | 15:36 |
| lachlan | This can be done through the option to add extra versions to be tested in another file which can be appended to the test execution | 15:39 |
| *** tristan has joined #buildstream | 15:39 | |
| lachlan | If you look in the repo under bst_benchmarks/template.benchmark you can see the general form | 15:39 |
| benschubert | perfect, I'll look into this soon, thanks! | 15:41 |
| lachlan | If you create a file containing the versions you require and then add this to the end of the call as "-c <filename.benchmark>" it does these extra versions | 15:42 |
| lachlan | The results will be output with the results for master | 15:42 |
| benschubert | lachlan: perfect thanks! For requests for improvements on the benchmarking side, I guess an issue on the repo is the best, or is there other discussions in other channels about it? (ML?) | 15:51 |
| adds68 | jonathanmaw, thanks for the merge! :) | 15:55 |
| adds68 | johnward, when will the next release be done? | 15:55 |
| *** nimish has quit IRC | 16:05 | |
| lachlan | benschubert: At the moment raising an issue would be best. | 16:07 |
| laurence | adds68, did you mean jonathanmaw there? | 16:07 |
| adds68 | laurence correct sorry johnward :) | 16:07 |
| laurence | benschubert, yes, gitlab issue on that repo pls | 16:08 |
| *** nimish has joined #buildstream | 16:08 | |
| jonathanmaw | adds68: I can do one now | 16:09 |
| *** nimish has quit IRC | 16:13 | |
| *** nimish has joined #buildstream | 16:14 | |
| *** nimish has joined #buildstream | 16:14 | |
| adds68 | jonathanmaw, awesome :) | 16:14 |
| jonathanmaw | adds68: done | 16:17 |
| adds68 | \o/ | 16:17 |
| benschubert | lachlan: I'll do thanks! | 16:18 |
| *** nimish has quit IRC | 16:24 | |
| *** nimish has joined #buildstream | 16:25 | |
| *** nimish has quit IRC | 16:35 | |
| *** nimish has joined #buildstream | 16:35 | |
| *** nimish has quit IRC | 17:10 | |
| *** nimish has joined #buildstream | 17:11 | |
| *** nimish has quit IRC | 17:26 | |
| *** nimish has joined #buildstream | 17:26 | |
| *** nimish has quit IRC | 17:30 | |
| *** jonathanmaw has quit IRC | 18:26 | |
| *** toscalix has quit IRC | 18:37 | |
| *** raoul_ has quit IRC | 18:44 | |
| gitlab-br-bot | BenjaminSchubert opened issue #853 (BUG hit: "no module named 'grpc.framework') on buildstream https://gitlab.com/BuildStream/buildstream/issues/853 | 19:02 |
| benschubert | ^ If somebody wants to bang their head | 19:07 |
| gitlab-br-bot | tristanvb opened MR !1051 (tristan/detox-tests->master: Allow using detox, and fix/refactor collection of coverage reports) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1051 | 19:15 |
| *** ChanServ sets mode: +o tristan | 19:16 | |
| tristan | benschubert, !1051 is going to fix #844 :) | 19:16 |
| gitlab-br-bot | Issue #844: Coverage reports inacurate with multiple runs of `tox` https://gitlab.com/BuildStream/buildstream/issues/844 | 19:16 |
| benschubert | Oh great! Thanks | 19:17 |
| tristan | detox is nice with python 3.5, 3.6 and 3.7 | 19:17 |
| *** phildawson_ has quit IRC | 19:32 | |
| *** tristan has quit IRC | 19:40 | |
| *** tristan has joined #buildstream | 19:45 | |
| *** alatiera has quit IRC | 19:49 | |
| *** lachlan has quit IRC | 20:05 | |
| gitlab-br-bot | tristanvb closed issue #844 (Coverage reports inacurate with multiple runs of `tox`) on buildstream https://gitlab.com/BuildStream/buildstream/issues/844 | 20:36 |
| gitlab-br-bot | tristanvb merged MR !1051 (tristan/detox-tests->master: Allow using detox, and fix/refactor collection of coverage reports) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1051 | 20:36 |
| *** mohan43u has quit IRC | 20:45 | |
| gitlab-br-bot | tristanvb opened MR !1052 (tristan/refine-coverage->master: .coveragerc: Omit some things which are irrelevant to cover.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1052 | 20:47 |
| gitlab-br-bot | tristanvb merged MR !1052 (tristan/refine-coverage->master: .coveragerc: Omit some things which are irrelevant to cover.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1052 | 21:12 |
| *** mohan43u has joined #buildstream | 21:36 | |
| *** tristan has quit IRC | 21:47 | |
| *** tristan has joined #buildstream | 22:08 | |
| *** ChanServ sets mode: +o tristan | 22:44 | |
| tristan | Interestingly, this test https://gitlab.com/BuildStream/buildstream/blob/tristan/element-processing-order/tests/frontend/buildorder.py shows that the order of builds are not stable | 22:44 |
| tristan | For me, the second test passes on py36 and py37, but finds a different order on py35... and it does it consistently | 22:44 |
| *** tristan has quit IRC | 23:19 | |
| *** tristan has joined #buildstream | 23:19 | |
| *** ChanServ sets mode: +o tristan | 23:19 | |
| tristan | scratch that, not consistent; different runs yield different results | 23:25 |
| tristan | looks like a dictionary randomization is playing a part possibly | 23:26 |
| tristan | or maybe set() | 23:26 |
| *** tristan has quit IRC | 23:35 | |
| gitlab-br-bot | jjardon closed issue #351 (Allow deploying directly to ostree) on buildstream https://gitlab.com/BuildStream/buildstream/issues/351 | 23:43 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!