| *** Prince781 has joined #buildstream | 00:07 | |
| *** Prince781 has quit IRC | 00:33 | |
| *** Prince781 has joined #buildstream | 00:33 | |
| *** tristan has joined #buildstream | 01:00 | |
| *** Prince781 has quit IRC | 01:53 | |
| tristan | jjardon, https://github.com/go-debos/fakemachine looks pretty neat | 04:25 |
|---|---|---|
| *** tristan has quit IRC | 07:03 | |
| *** coldtom has joined #buildstream | 07:52 | |
| *** Phil has joined #buildstream | 08:13 | |
| *** jonathanmaw has joined #buildstream | 08:43 | |
| gitlab-br-bot | buildstream: merge request (phil/437-workspaces-tutorial->master: Phil/437 workspaces tutorial) #519 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/519 | 08:55 |
| gitlab-br-bot | buildstream: issue #446 ("Stack trace shown if you accidentially `bst COMMAND <directory>`") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/446 | 09:00 |
| jennis | What do we think about adding a "newcomers" label to the project? I've found some bugs that are suitable for a newcomer, yet I've noticed we don't have this label...? | 09:03 |
| paulsherwood | +1 | 09:04 |
| Phil | jennis, that seems very sensible | 09:06 |
| jennis | Ok i'll do it for this issue, I guess toscalix can always revert the change if he strongly disagree's when he comes online | 09:07 |
| *** obi_wan has joined #buildstream | 09:15 | |
| *** dominic has joined #buildstream | 09:16 | |
| *** jonathanmaw has quit IRC | 09:30 | |
| *** jonathanmaw has joined #buildstream | 09:30 | |
| gitlab-br-bot | buildstream: issue #447 ("Strack trace shown when trying to open a workspace for an unbuilt element") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/447 | 09:39 |
| juergbi | jmac: you mentioned issues with buildbox dependencies. what version incompatibilities have you discovered? | 09:43 |
| jmac | Lots of them | 09:44 |
| jmac | Firstly, php_namespace isn't supported by the version of grpc I have | 09:44 |
| juergbi | nothing with PHP should be relevant to buildbox | 09:45 |
| juergbi | ah, proto declaration | 09:45 |
| jmac | No, but it stops it building | 09:45 |
| jmac | Your code also asks for grpc/channel.h which isn't in my version of the grpc libraries | 09:45 |
| juergbi | I'm testing with the latest stable releases of the dependencies, afaict | 09:45 |
| juergbi | grpc 1.12.0 | 09:45 |
| juergbi | fuse 3.2.3 | 09:45 |
| jmac | I tried patching in the latest upstream but then ran into further problems | 09:45 |
| jmac | I will dump a list of them on a gitlab issue if that helps | 09:46 |
| juergbi | ok, ta | 09:47 |
| gitlab-br-bot | buildstream: issue #448 ("Autocompletion defaults to the project's root before the element path") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/448 | 10:02 |
| *** obi_wan has quit IRC | 10:03 | |
| gitlab-br-bot | buildstream: merge request (gokcen/source_transform->master: WIP: Source transform plugin) #505 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/505 | 11:05 |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 11:06 |
| gitlab-br-bot | buildstream: issue #449 ("'bst workspace open -f' does not behave as documented") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/449 | 11:19 |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 11:58 |
| *** jonathanmaw has quit IRC | 13:00 | |
| *** jonathanmaw has joined #buildstream | 13:16 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 13:25 |
| tiago | Is there any known issue with the unit tests on master? The test test_missing_project_conf is failing to me | 13:27 |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 13:36 |
| gitlab-br-bot | buildstream: merge request (franred/show-nice-error-when-try-to-load-a-directory->master: Add error message when running commands on directories) #520 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/520 | 14:26 |
| persia | tiago: There shouldn't be: generally unit tests are expected to be clean on merge. Failure in master is usually a regression. | 14:43 |
| tlater | tiago: That said, there is a known issue that causes nondeterministic test failures. Could you link me to the pipeline? If you'd like to debug yourself, restarting it normally resolves that particular issue. | 14:48 |
| tlater | I don't think that particular test is prone to the failure, but we have no idea what causes it. It's usually something related to unmounting a sandboxed directory. | 14:49 |
| tiago | I was thinking more in different versions of the dependencies | 14:50 |
| tlater | tiago: You mean you might be running versions of software different from those in our CI? | 14:51 |
| tiago | yup | 14:51 |
| tlater | If there are regressions in versions we don't test against, then I don't think we're currently aware of any - unless you can find any in the issue list :) | 14:52 |
| tlater | tiago: You're not running into issues with versioneer, are you? | 14:53 |
| tiago | tlater I don't think so. This is what I get https://paste.fedoraproject.org/paste/dPzq5cdTeN-1SIbb6jFB7A | 14:56 |
| tlater | tiago: I've never seen that before. If you can confirm it in a fresh buildstream installation, you've found a new bug :) | 14:57 |
| tiago | Well it is passing on the CI… | 14:58 |
| tlater | There *is* a chance it's caused by debris in temporary directories | 14:59 |
| tlater | If you want to test it manually, just run `bst workspace list` in an empty directory | 14:59 |
| tlater | It should fail and prompt you for a project name | 14:59 |
| *** jcampbell has quit IRC | 15:00 | |
| tlater | `bst workspace list | cat` in an empty directory should also fail | 15:00 |
| *** tristan has joined #buildstream | 15:00 | |
| tlater | If both those things happen, you have debris lying around somehow. | 15:00 |
| *** jcampbell has joined #buildstream | 15:02 | |
| *** Prince781 has joined #buildstream | 15:09 | |
| tiago | It fails and prompts | 15:14 |
| tlater | tiago: That's what the test checks. It's definitely a curious failure | 15:18 |
| tlater | But I don't think it occurs upstream then | 15:19 |
| gitlab-br-bot | buildstream: merge request (tiagogomes/issue-289->master: Tiagogomes/issue 289) #521 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/521 | 15:21 |
| tristan | tiago, I've never seen that fail, very strange - do you have local modifications ? | 15:33 |
| tiago | No :( | 15:38 |
| tristan | tiago, that test asserts that we get the expected error when we try to startup and there is no project.conf | 15:53 |
| tristan | aaaaaahhh | 15:53 |
| tristan | that will be cs_shadow's patch | 15:54 |
| tristan | tiago, https://gitlab.com/BuildStream/buildstream/merge_requests/428 is what is causing this | 15:55 |
| tristan | tiago, the workaround for now, is to move the `buildstream` checkout directory to a directory which does not have a `project.conf` in one of it's parent directories | 15:56 |
| tiago | aah, that explains it | 15:56 |
| cs_shadow | it's a feature! ;) | 15:57 |
| cs_shadow | that maybe we need to communicate to users more clearly | 15:57 |
| tlater | Ah! | 15:58 |
| tlater | cs_shadow: Users will probably rarely run the test suite, but perhaps we can avoid parsing parent's buildstream.conf in the test suite | 15:59 |
| tlater | This will be an issue when we start developing buildstream in buildstream. | 15:59 |
| cs_shadow | i meant that as a general comment as in it may not be immediately obvious to users that bst does that | 16:00 |
| cs_shadow | but your point about testsuite is also valid | 16:01 |
| gitlab-br-bot | buildstream: issue #450 ("Tests are non deterministic after supporting running commands in project subdirs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/450 | 16:04 |
| * tristan files #450 to track this | 16:04 | |
| tristan | tlater, better to remove the test than to disable the functionality in a special case scenario; which is for the purpose of the test itself | 16:06 |
| tlater | Can't we ensure that we stay inside a the test directory somehow? | 16:08 |
| tlater | Without changing the actual functionality, of course | 16:08 |
| tlater | Maybe screw with what python reports for parent directories, exclusively for that test? | 16:08 |
| tristan | With a chroot / mount trick in the runner ? | 16:08 |
| cs_shadow | one thing that might help is that if we say that `-C` option takes precedence over this search | 16:09 |
| tristan | wouldn't solve running tests for users though | 16:09 |
| tristan | cs_shadow, that is an idea | 16:09 |
| cs_shadow | so, we don't do this lookup if -C option was provided and we can run our tests with that option | 16:09 |
| tlater | Yes, that's probably good :) | 16:09 |
| tlater | tristan: I was moreso thinking about mocking out some python call, but this seems like a better solution | 16:10 |
| cs_shadow | my other idea (which i don't really like much) is to look for a specially named file and stop the search if we see that file. But I don't think that is going to be of much use outside the tests | 16:11 |
| tristan | the UX should take precedent over the testability though, so if we think "-C means act like BuildStream started here" is better than "-C means the project directory explicitly" then we should choose that | 16:13 |
| tristan | right now it's kind of on the verge that there are starting to be differences, so we can make that choice | 16:13 |
| tristan | -C is also used in `bst init` for instance | 16:13 |
| tristan | (to *create* the project.conf, also a "project directory") | 16:14 |
| *** bethw has joined #buildstream | 16:15 | |
| cs_shadow | That is an interesting point, that will also matter if we want to allow running commands from workspace directories. Keeping that in mind, I personally favor the 2nd option, i.e. `-C` means "here's the project directory" | 16:15 |
| tlater | How does git behave in this case? | 16:15 |
| tlater | Since this is modeled after git, and most people will probably be familiar with that, perhaps we should mimick them fully? | 16:15 |
| tristan | I was thinking an argument for "-C means a specific project directory" was that it was slightly more explicit | 16:16 |
| tristan | like, it ensures that scripted usage of BuildStream will never accidentally stray into a parent directory in the case that you screwed up and your project.conf is missing | 16:16 |
| cs_shadow | FWIW git will work the way `bst -C` works now, i.e. even when you provide -C, it will still lookup .git in parent directories | 16:16 |
| tristan | Yeah, Sander mentioned that's what git does | 16:17 |
| tristan | but then, it's not what make does | 16:17 |
| cs_shadow | that said, i do think there is a need for an option (maybe its `-C`, maybe something else) where we don't try to "guess" anything | 16:17 |
| tristan | I see | 16:18 |
| tristan | Hmmm | 16:18 |
| tristan | cs_shadow, it's desirable to avoid, though | 16:18 |
| tristan | i.e. adding an option is huge compared to the extra level of control the user gets (paying a high price in burden of knowledge) | 16:19 |
| cs_shadow | that's true and adding another option will also likely be a source of confusion as we'll have two similar options. That's why I was trying to define `-C` as that option | 16:21 |
| tristan | I think the argument for "-C means BuildStream was invoked here" gets us the ability to do `bst -C <workspace_dir> build`, when we land that thing which lets you run bst commands from a workspace | 16:21 |
| tristan | but, that feature is... tricky | 16:23 |
| * paulsherwood wonders if he can consider deprecating ybd yet... | 16:23 | |
| paulsherwood | can buildstream... | 16:24 |
| paulsherwood | - build bootable systems? | 16:24 |
| paulsherwood | - be run in a large ci with reliably caching across many builds/pipelines? | 16:24 |
| tristan | paulsherwood, I think to do that you need to move the projects which you think are important off of YBD first | 16:25 |
| paulsherwood | pardon? | 16:25 |
| tristan | I mean, you could deprecate it with active users | 16:25 |
| paulsherwood | i'm involved in a production project, as you know. i can't offer to move to buildstream until (at least) the above questiosn are yes | 16:26 |
| tristan | but a pragmatic approach to deprecation, can be "Lets move project(s) over to use BuildStream, am I happy ? -> deprecate" | 16:26 |
| * paulsherwood notices tristan has not answered the questions | 16:27 | |
| tristan | the answer to your first point is yes, we can build bootable intel images and have a scriptable interface for it (so should be adaptable for other machines) | 16:27 |
| tristan | ... | 16:27 |
| paulsherwood | (not arm images?) | 16:28 |
| tristan | the answer to your second is that the current climate is going to be rocky | 16:28 |
| paulsherwood | please admit that you should have used kbas :-) | 16:28 |
| tristan | paulsherwood, the thing is it's mostly a script which creates the image but is not all fancy to do every different kind of image | 16:28 |
| tristan | for arm, it's a matter of creating the filesys but the kernel bootloader is different etc | 16:29 |
| *** coldtom has quit IRC | 16:29 | |
| tristan | so yeah I mean, you can do it but there is no stock feature which will say "I'll do this complicated thing based on you're having selected 'arm'" | 16:29 |
| paulsherwood | right but that's already in (say) the applicable baserock definitions... can't bst handle the same processes? | 16:29 |
| tiago | I think noisecell is planning to work on deploying to arm | 16:30 |
| paulsherwood | i thought he was blocked, but may be wrong | 16:30 |
| tiago | What's left to do on https://gitlab.com/BuildStream/buildstream/issues/287? I just tried adding those special variables to project config and elements configuration and they are already not allowed | 16:30 |
| noisecell | there is no arm64 deployment pluging, nor ostree base for arm64. the second is tangential to buildstream the first, depends if Buildstream cares about deploying embedded systems or images | 16:31 |
| noisecell | one other point related to deployment that I can discuss is how buildstream is thinking to change final images, as we were doing before with write-extensions and configure-extensions | 16:32 |
| *** dominic has quit IRC | 16:33 | |
| tristan | paulsherwood, it doesnt work the same as with definitions no, those require super user permissions to mount loopbacks etc, and we don't have this concept of "deployment" as a semi-deterministic step. Instead, to do arm, one can look at the intel script: http://buildstream.gitlab.io/bst-external/elements/x86image.html#module-elements.x86image, | 16:33 |
| tristan | paulsherwood, and can concoct something similar for arm | 16:33 |
| noisecell | I have seen the "script" element which could help, but still unsure of how to use it | 16:33 |
| paulsherwood | but no caching at scale any time soon? can you support multiple caches and use kbas (seriously)? | 16:34 |
| paulsherwood | s/you/we/ | 16:34 |
| tristan | noisecell, with BuildStream, you can only get output, you cannot add a deployment element which for instance, has super user privs, or access to a specific device, or something outside of the sandbox | 16:34 |
| tristan | I didnt say no caching at scale any time soon... but am talking about two things at once | 16:35 |
| paulsherwood | 17:28 < tristan> the answer to your second is that the current climate is going to be rocky | 16:35 |
| tristan | From what I've seen, *current* OSTree solution is limited to about 4 concurrent full builds, all of which do up to 4 concurrent pulls/pushes each | 16:36 |
| tristan | when using a 4 core machine | 16:36 |
| paulsherwood | which is pretty awful | 16:36 |
| tristan | Right | 16:36 |
| paulsherwood | and *i told you so* | 16:36 |
| paulsherwood | sorry | 16:36 |
| * paulsherwood has had alcohol | 16:37 | |
| tristan | Well, I'm not defending that; it's still a better approach, the implementation is not good, though | 16:37 |
| tristan | yes, presently pulls are slow, pushes are pretty good as they are done in a single stream, and some benchmarks are saying that flatpak does the pulls faster | 16:37 |
| tristan | so we may be able to optimize | 16:37 |
| tristan | not sure how it's possible because it's still just ostree | 16:38 |
| * paulsherwood fears that if he spent two days implementing the kbas alternative for bst, tristan would reject if for purist reasons, hence won't bother | 16:38 | |
| tristan | But... the point is juergbi is bringing the CAS server very soon | 16:38 |
| tristan | which is why I would say the current climate is in a state of flux regarding what the performance will be | 16:38 |
| * paulsherwood predicts it will turn out that there are other issues around the CAS implementation... so this will slip. and until the bst cache story is solid, there's no point in starting down the road | 16:39 | |
| tristan | paulsherwood, there was no reason to suspect that a deduplicated file store was going to be unbearably unperformant (I'm not sure it's that unbearable today, though, and havent seen comparisons with kbas in terms of performance) | 16:40 |
| paulsherwood | and while i understand the 'no special privileges' reasoning, bst still needs a deployment story | 16:40 |
| tristan | paulsherwood, and changing something like this costs a lot, so you don't do it whimsically; now we're trying CAS which fits into the whole distributed build story | 16:40 |
| paulsherwood | tristan: i think you just haven't looked :) | 16:40 |
| tristan | paulsherwood, at numbers ? | 16:41 |
| * paulsherwood understands the costs quite well | 16:41 | |
| paulsherwood | kbas is just a cherrypi server... it scales way beyond 4 up and downloads | 16:42 |
| tristan | Well, there is distributed building, which is coming... that will cost LOC for CAS... this is reason to make sure we shed any OSTree/tar based code which serves a similar purpose | 16:42 |
| paulsherwood | hmmm | 16:43 |
| paulsherwood | while i'm in favour of CAS, part of me believes bst already has way too many dependencies | 16:43 |
| tristan | btw: it's up to 4*4 with the quad core if I understood jjardon | 16:43 |
| tristan | paulsherwood, right but we will shed the OSTree dep for CAS (rather, the OSTree stuff will become specific to downloading from ostree repos) | 16:44 |
| tristan | And technically, CAS will be *in BuildStream* | 16:44 |
| paulsherwood | tristan: baserock's kbas is an arm server. the production project supports 500 engineers with a single kbas in aws and i think it's not as powerful as the machine gnome is using | 16:45 |
| * tristan thinks we gain this google protocol python dep | 16:45 | |
| jmac | Yes we do | 16:45 |
| * paulsherwood will have to wait and see. and still thinks bst should be able to use a kbas | 16:45 | |
| * paulsherwood goes back to nursing his beer | 16:46 | |
| tristan | noisecell, script element is a little bit tricky to setup but straight forward to use once you have | 16:48 |
| tristan | noisecell, it lets you: A.) Stage the tools you want to use at "/" ... B.) Stage the dependencies which you want to manipulate at "location of your choosing"... C.) Run commands in the runtime to manipulate what you've staged | 16:49 |
| tristan | noisecell, usually, you will use "compose" first so that you create a single composed artifact with all the integration commands run | 16:49 |
| tristan | then stage that in the %{build-root} | 16:49 |
| tristan | and stage the tools you need to create an image in / | 16:50 |
| tristan | there is an example of this but it's way out of date... | 16:50 |
| tristan | it also creates an initrd separately with a compose element and then pops that into a /boot partition | 16:50 |
| * Phil wonder's what kbas is. Google provides lots of results about "key bio diversity areas"... | 16:51 | |
| Phil | wonders | 16:51 |
| jjardon | tristan: I'd say ~20 jobs and it doesn't seem to be CPU bound (in a 4 core machine you never get more that 50% CPU use) | 16:52 |
| jjardon | Phil: https://gitlab.com/baserock/ybd/tree/master/kbas | 16:52 |
| tristan | I thought this was it: https://gitlab.com/BuildStream/buildstream-tests/tree/build-gnome/elements/gnome | 16:52 |
| gitlab-br-bot | buildstream: merge request (franred/show-nice-error-when-try-to-load-a-directory->master: Add error message when running commands on directories) #520 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/520 | 16:53 |
| Phil | thanks jjardon :) | 16:53 |
| tristan | noisecell, ^^^ but it looks like that was converted to use the convenience x86image element instead of doing it manually | 16:53 |
| jjardon | Phil: basically is the cache server ybd use to store artifacts | 16:53 |
| *** jonathanmaw has quit IRC | 16:55 | |
| *** Phil has quit IRC | 16:59 | |
| finn | juergbi, I have the artifact cas server set up, I'm able to push keys from buildstream to buildgrid but I'm having trouble downloading artifacts | 17:08 |
| finn | I'm trying to query the cas server using buildstream.proto | 17:11 |
| finn | I always get "Not Found" | 17:12 |
| finn | you can see my attempt here: https://gitlab.com/BuildGrid/buildgrid/blob/finn/fuse/app/commands/cmd_bot.py#L73 | 17:15 |
| finn | Expecting a hash from here: | 17:17 |
| finn | https://gitlab.com/BuildStream/buildstream/blob/jmac/source_pushing_experiments/buildstream/sandbox/_sandboxbwrap.py#L98 | 17:17 |
| finn | Does this look sensible to you? | 17:17 |
| juergbi | finn: the BuildStream ArtifactCache interface is only for full artifacts. for lower level CAS digests, you need to use the lower level CAS interface | 17:27 |
| juergbi | i.e., ArtifactCache interface maps from BuildStream cache keys to CAS directory digests | 17:28 |
| finn | where can I find this? | 17:29 |
| juergbi | finn: the protocol is described in the Remote Execution API design document | 17:33 |
| juergbi | you can also read the proto files and/or the BuildStream implementation | 17:33 |
| jennis | Are you pubbing? | 17:40 |
| jennis | whoops haha | 17:40 |
| gitlab-br-bot | buildstream: merge request (franred/show-nice-error-when-try-to-load-a-directory->master: Add error message when running commands on directories) #520 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/520 | 17:41 |
| *** tristan has quit IRC | 17:43 | |
| gitlab-br-bot | buildstream: merge request (jennis/output_useful_error_when_trying_to_load_directory->master: Jennis/output useful error when trying to load directory) #522 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/522 | 17:46 |
| gitlab-br-bot | buildstream: merge request (jennis/output_useful_error_when_trying_to_load_directory->master: Jennis/output useful error when trying to load directory) #522 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/522 | 17:47 |
| gitlab-br-bot | buildstream: merge request (jennis/output_useful_error_when_trying_to_load_directory->master: Output a useful error message when trying to load a directory) #522 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/522 | 17:47 |
| gitlab-br-bot | buildstream: merge request (franred/show-nice-error-when-try-to-load-a-directory->master: Add error message when running commands on directories) #520 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/520 | 17:50 |
| *** bethw has quit IRC | 18:08 | |
| *** Prince781 has quit IRC | 21:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!