| *** nimish2711 has joined #buildstream | 00:15 | |
| *** nimish2711 has quit IRC | 00:21 | |
| *** nimish2711 has joined #buildstream | 00:41 | |
| *** nimish2711 has quit IRC | 01:10 | |
| *** nimish2711 has joined #buildstream | 01:26 | |
| *** nimish2711 has quit IRC | 03:06 | |
| *** swick has joined #buildstream | 04:59 | |
| *** mark has joined #buildstream | 05:39 | |
| *** mark has quit IRC | 05:40 | |
| *** mohan43u has joined #buildstream | 06:46 | |
| *** toscalix has joined #buildstream | 07:42 | |
| *** toscalix has quit IRC | 07:44 | |
| *** swick has quit IRC | 08:14 | |
| *** mohan43u has quit IRC | 08:43 | |
| *** rdale has joined #buildstream | 09:02 | |
| *** toscalix has joined #buildstream | 09:09 | |
| benschubert | juergbi: !1070 should address all points raised. Would you have time to review? I would say it is ready now | 09:12 |
|---|---|---|
| juergbi | benschubert: great, on my list for review today | 09:13 |
| benschubert | thanks! | 09:14 |
| *** alatiera has joined #buildstream | 09:22 | |
| valentind | doras[m], I am not sure why "script" plugin forces to have something in /. But it does unfortunately. Maybe because it expects some commands. | 09:33 |
| valentind | doras[m], The way we use "script" in the bootstrap of freedesktop SDK is probably not a use case that was thought of originally. | 09:34 |
| valentind | doras[m], So for the moment just put something in / | 09:34 |
| valentind | Eventually you can put something in /, then add your stuff in /newroot, and then set variable "install-root" to be /newroot, so you do not get any file from what was in /. | 09:35 |
| *** raoul has joined #buildstream | 09:37 | |
| gitlab-br-bot | willsalmon opened issue #963 (Improve workspace docs) on buildstream https://gitlab.com/BuildStream/buildstream/issues/963 | 09:48 |
| *** jonathanmaw has joined #buildstream | 09:58 | |
| gitlab-br-bot | willsalmon closed issue #904 (Wrong suggestion for opening non tracked workspaces) on buildstream https://gitlab.com/BuildStream/buildstream/issues/904 | 10:05 |
| juergbi | benschubert: interesting that it's slightly slower with 4 builders | 10:17 |
| *** alatiera has quit IRC | 10:18 | |
| *** alatiera has joined #buildstream | 10:19 | |
| benschubert | juergbi: it is also with 8 actually on my new benchmark (by 1 sec) | 10:20 |
| benschubert | juergbi: we are doing more work at the start and that is enough | 10:20 |
| benschubert | juergbi: however refactoring update_state to only update what is needed at each step is possible now and we should gain performance with a subsequent refactor :) | 10:21 |
| benschubert | juergbi: actually, I could have a "if element.__cache_key is not None" at the start of `update_state_recursively`. That could give us a bit of performance back | 10:22 |
| benschubert | juergbi: I mean just before https://gitlab.com/BuildStream/buildstream/merge_requests/1070/diffs#ecad1ef0b2d7e0597dd3d592a7617f7c23465911_2918_2989. Does that make sense? | 10:23 |
| juergbi | benschubert: only execute _udpate_state if cache key is not None? eh, how could that work? | 10:26 |
| juergbi | or do you mean the inverse? | 10:26 |
| juergbi | that would also not be correct as e.g. we have to check 'cached' state after pull_done | 10:26 |
| benschubert | juergbi: have a "if element.__cache_key is not NOne: continue" at that place | 10:26 |
| benschubert | juergbi: ah ok | 10:26 |
| benschubert | so I guess, better to keep it like that now and fix everything after by splitting update_state | 10:27 |
| juergbi | yes, I think so | 10:27 |
| benschubert | ok! | 10:27 |
| benschubert | So small users will indeed have a small hit until we have this in (small being 8 builders or less) | 10:27 |
| juergbi | benschubert: the MR description still shows a small improvement for 8+ | 10:28 |
| juergbi | if you have updated benchmarks, can you please update the description? | 10:28 |
| benschubert | juergbi: sure, Will need to wait tonight, my results are at home | 10:29 |
| juergbi | ta | 10:29 |
| benschubert | added a warning that I'll update this | 10:30 |
| doras[m] | valentind: I put the thing I wanted to move under %{install-root}/some/path, and it worked. I just needed to create a filter element that removed the runtime dependencies off of the element I wanted to move. | 10:30 |
| doras[m] | After the fix for the filter element in 1.2.4, it doesn't stage its "parent's" runtime dependencies like every other element. | 10:31 |
| doras[m] | I ended up having to create symlinks, so I did need bootstrap-import staged at "/". | 10:32 |
| doras[m] | juergbi: do you know why script elements can't have runtime dependencies? I have a few use cases for it. | 10:37 |
| juergbi | doras[m]: I can't really think of a reason why it's forbidden except that it might be unlikely to be needed with the original use case for script elements | 10:40 |
| juergbi | maybe jonathanmaw remember why he added this but it was a very long time ago | 10:40 |
| juergbi | *remembers | 10:40 |
| juergbi | https://gitlab.com/BuildStream/buildstream/merge_requests/22 | 10:42 |
| *** lachlan has joined #buildstream | 10:43 | |
| jonathanmaw | juergbi: same, I think it was just that we couldn't think of any reason why it'd be wanted, and the original use-case resembled an x86image | 10:45 |
| doras[m] | I see. The output of my script element can run. It can also perform integration commands. However, without runtime dependencies, it can't to either of those on its own. | 10:47 |
| doras[m] | My current workaround is to include the script's runtime dependencies in the compose element that runs its integration commands. It's not ideal, because it makes me compose elements that I don't really want to include in the composition. | 10:51 |
| raoul | juergbi (or anyone else), any thoughts and naming the queue for pushing sources, the send queue? Realised naming it push messed up the pipeline summary, but I'm unsure if send is specific enough. Though fetch and pull are similarly used to specify source pulling and artifact pulling | 11:08 |
| benschubert | My take on that would be that it might be nicer to have "artifact-push, artifact-pull, source-push, source-pull" but that's bigger. Otherwise not really opinionated about it | 11:12 |
| raoul | Yeah that's definitely clearer, but keeping everything aligned does mean we'll take a lot more space per line | 11:14 |
| juergbi | not sure, don't have another idea right now | 11:15 |
| raoul | I'll do an asciinema of artifact-push etc. to get an idea if it's alright | 11:16 |
| benschubert | raoul: using "apush, apull, spush, spull" would be shorted but might be harder to grasp | 11:19 |
| raoul | yeah I did spush and didn't hugely like the look of it | 11:24 |
| juergbi | a middle way could be src-push but not sure | 11:29 |
| raoul | here's with long names https://asciinema.org/a/iDny5v7Q7IEvsMIUPPrAlOWdW | 11:33 |
| raoul | src-push could work, though would we want a short name for artifacts too? We could just leave them as push and pull | 11:34 |
| *** lachlan has quit IRC | 11:56 | |
| *** lachlan has joined #buildstream | 11:57 | |
| *** raoul has quit IRC | 12:02 | |
| benschubert | art-push ? | 12:05 |
| benschubert | but yeah that long is too long :) | 12:05 |
| *** lachlan has quit IRC | 12:24 | |
| *** nimish2711 has joined #buildstream | 12:32 | |
| *** raoul has joined #buildstream | 12:55 | |
| raoul | here's how src-push etc. looks for reference https://asciinema.org/a/mHrvF8F8nFExHYDpUq74xvTvp, I'll push current remote source cache and point out in !1214 that what to call pipelines is still up for debate | 13:18 |
| gitlab-br-bot | MR !1214: Remote source cache https://gitlab.com/BuildStream/buildstream/merge_requests/1214 | 13:18 |
| raoul | actually this one has src-push https://asciinema.org/a/OIObuXAhwhYi39nXQWkXAJkUW, forgot to rebase --continue | 13:20 |
| *** nimish2711 has quit IRC | 13:44 | |
| *** nimish2711 has joined #buildstream | 13:48 | |
| *** lachlan has joined #buildstream | 14:27 | |
| benschubert | juergbi: for an element, if the last source declared is cached, does that means all the sources are cached? | 14:54 |
| raoul | Atm the source cache cache all the sources together so yes | 14:56 |
| raoul | what's the context though? sounds like it should potentially be clearer | 14:56 |
| benschubert | raoul: I was looking at https://gitlab.com/BuildStream/buildstream/commit/8998788de605fd5e1f558cae505b7bb66d254994#ecad1ef0b2d7e0597dd3d592a7617f7c23465911_2095_2136 | 14:57 |
| benschubert | raoul: then why the last one? Would it be correct if I take the first one too? | 14:58 |
| raoul | No, the sources key is built of it's unique key and the keys of all the previous sources, because they need to be staged and cached together because of sources like patch that don't make sense on their own | 14:59 |
| benschubert | ok | 14:59 |
| benschubert | thanks! | 14:59 |
| raoul | There was discussion that we might want to add a flag in sources for requiring previous sources, similar to the track and fetching flags, such that sources are stored separately where possible, but for now it's all together under the last source of an element | 15:01 |
| *** lachlan has quit IRC | 15:05 | |
| *** lachlan has joined #buildstream | 15:06 | |
| WSalmon | so before raoul's big change, get_unique_key was as far as i can tell always called after preflight. as per the other plug-ins i set host_tool = util.get_host_tool() in the preflight. but i use the host tool to get my unic key this worked well until it broke, was i just taking advantage of a happy coincidence or has this change broken a contract? | 15:06 |
| WSalmon | juergbi, tristan ^ | 15:07 |
| raoul | what do you mean by using the host tool to get the unique key? | 15:11 |
| WSalmon | https://docs.buildstream.build/buildstream.plugin.html?highlight=preflight#buildstream.plugin.Plugin.preflight says i should be working this way but https://docs.buildstream.build/buildstream.plugin.html?highlight=preflight#buildstream.plugin.Plugin.get_unique_key dose not explisitly say if i can use host_tool | 15:11 |
| WSalmon | so i run the host tool, eg svn to get info that i want to create the unique key | 15:11 |
| WSalmon | raoul, i want the repo's UUID for the unique key, and i get it by running `svn info` | 15:12 |
| juergbi | oh, that indeed doesn't sounds correct | 15:17 |
| juergbi | preflight() should be called first | 15:18 |
| raoul | hmm yeah the keys are generated when the sources are created from meta, but I guess they should be somewhere after preflight | 15:21 |
| raoul | didn't realise that preflight was ever needed for generating the unique keys | 15:21 |
| raoul | Could just move the sources _generate_key call, to immediately after the source._preflight in the elements preflight? | 15:23 |
| WSalmon | it cant be for any of the core plug-ins or the tests would not be passing, hence my question as to if i was taking advantage of convention or contract. i would like it that way round but i think this is one for juergbi / tristan as i think they have clear ideas as to how it should work. | 15:25 |
| WSalmon | if we agree how this is contract it might be nice to write it down some were | 15:27 |
| juergbi | WSalmon: can you elaborate what you use `svn info` for in `get_unique_key()`? you can't rely on the source already being local there | 15:28 |
| juergbi | (and there should also be no remote access in that method) | 15:28 |
| juergbi | i.e., while I'd still say preflight() should be called first, there might still be a bug in the svn plugin | 15:30 |
| WSalmon | juergbi, i might well be braking the remote access bit then, i was including the repo UUID, in the unique key, which i get from `svn info` | 15:32 |
| juergbi | WSalmon: maybe that should rather retrieved at track time and stored in the 'ref'? | 15:34 |
| juergbi | haven't used svn in a long time, so don't remember the details of what the UUID is for | 15:34 |
| raoul | yeah that's what I thought, though I'm even more unfamiliar with svn | 15:35 |
| WSalmon | juergbi, i do put it in to the ref as a hash so i cant currently get it back, maybe i should change that | 15:36 |
| juergbi | or just use the hash in get_unique_key()? | 15:36 |
| juergbi | if you have a bst project with sources tracked/resolved, the strict cache keys should be fully defined (at least for a given buildstream/plugin version) | 15:37 |
| juergbi | i.e., nothing remote/external should have any influence anymore | 15:37 |
| juergbi | also, after 'source fetch' the expectation is that you can even build the whole thing offline | 15:37 |
| WSalmon | thats true, i think i have build this back to front as i had taken bits from local but that is a bad example | 15:38 |
| WSalmon | i had been using get_unique_key to build my ref but it should be the other way round | 15:39 |
| juergbi | yes | 15:41 |
| juergbi | raoul: hm, we're calling get_unique_key() even when the source is not resolved? | 15:41 |
| raoul | hmm yeah I think we are, though we definitely shouldn't | 15:45 |
| *** alatiera has quit IRC | 15:51 | |
| *** lachlan has quit IRC | 15:52 | |
| *** swick has joined #buildstream | 15:52 | |
| juergbi | I'm surprised this didn't break any tests | 15:53 |
| *** alatiera has joined #buildstream | 15:53 | |
| juergbi | raoul: can you please look into this (and preflight). would be good to have additional tests | 15:54 |
| raoul | yeah I'm having a look at where the keys should be generated | 15:59 |
| raoul | I'll try and get an MR for this soonish | 15:59 |
| juergbi | ta | 16:04 |
| *** lachlan has joined #buildstream | 16:11 | |
| raoul | oh wait, I've got it called in _tracking_done() so update the key if necessary, is that sufficient? | 16:12 |
| raoul | s/so/to/ | 16:12 |
| juergbi | raoul: we shouldn't attempt to generate a key beforehand at all, though | 16:32 |
| juergbi | might cause errors in plugins and we don't need the key earlier anyway, right? | 16:32 |
| raoul | No that's true, should it be called at the beginning of the fetch stage then, as the track stage might not be called | 16:33 |
| juergbi | raoul: the key is needed in the main process before fetching | 16:34 |
| juergbi | at least to determine whether the sources are already in CAS | 16:36 |
| juergbi | raoul: maybe we should just do it on the first call to get_source_name? | 16:38 |
| juergbi | or possibly in the _key property | 16:39 |
| raoul | Though we'll need previous sources to generate the key | 16:40 |
| raoul | I guess we could have the sources remember previous sources but that doesn't sound ideal | 16:41 |
| *** toscalix has quit IRC | 16:43 | |
| raoul | maybe in __ensure_previous_sources? that way it's generated when we know it's resolved | 16:47 |
| raoul | wait, no that's only used when require previous flags are set | 16:49 |
| *** lachlan has quit IRC | 16:49 | |
| doras[m] | What would be the meaning of a runtime dependency in an element whose output is a library? | 17:01 |
| doras[m] | Is it that when this library is dynamically loaded during runtime, which other libraries it depends upon? | 17:02 |
| juergbi | doras[m]: yes, or if the library spawns an executable, that executable would also be a runtime dependency | 17:33 |
| juergbi | raoul: I've commented on !1214. Overall it looks good to me. Just some small things besides the queue/job naming (and scheduling) question | 17:35 |
| gitlab-br-bot | MR !1214: Remote source cache https://gitlab.com/BuildStream/buildstream/merge_requests/1214 | 17:35 |
| raoul | Sweet, thanks juergbi, not sure if I have an answer for those bits unfortunately so maybe some more eyes would be good | 17:36 |
| raoul | I've put the generating keys into the Element._source_cached() method which seems to work well. Means it generates the key for the final source when it needs it, which will be after tracking | 17:37 |
| juergbi | raoul: have you tested / do we have tests that workspaces work fine with configured remote source cache? | 17:37 |
| *** lachlan has joined #buildstream | 17:38 | |
| raoul | Not the remote side, a quick check seems to work, but I'll add a proper test | 17:40 |
| *** nimish2711 has quit IRC | 17:42 | |
| *** nimish2711 has joined #buildstream | 17:42 | |
| *** lachlan has quit IRC | 17:44 | |
| *** lachlan has joined #buildstream | 17:59 | |
| raoul | juergbi, I've put a commit for generating source keys later here, will think of some additional test(s) and create an MR tomorrow: https://gitlab.com/BuildStream/buildstream/tree/raoul/source-key-fix | 18:04 |
| *** jonathanmaw has quit IRC | 18:28 | |
| *** raoul has quit IRC | 18:28 | |
| *** nimish2711 has quit IRC | 18:30 | |
| *** nimish2711 has joined #buildstream | 18:30 | |
| *** nimish2711 has quit IRC | 18:39 | |
| *** lachlan has quit IRC | 18:42 | |
| *** rdale has quit IRC | 18:52 | |
| *** lachlan has joined #buildstream | 18:57 | |
| *** lachlan has quit IRC | 19:07 | |
| *** lachlan has joined #buildstream | 19:14 | |
| *** lachlan has quit IRC | 19:18 | |
| *** lachlan has joined #buildstream | 19:20 | |
| *** lachlan has quit IRC | 19:25 | |
| *** kapil___ has quit IRC | 19:31 | |
| *** lachlan has joined #buildstream | 19:35 | |
| *** csoriano has joined #buildstream | 20:29 | |
| csoriano | hey all, I'm trying to build and generate a tarball for a gnome library called gnome-autoar | 20:30 |
| csoriano | for that, I did bst build, then entered a shell | 20:30 |
| csoriano | a build shell with bst shell --build core-deps/gnome-autoar.bst | 20:30 |
| csoriano | here, I expect make and make install to work as expected | 20:30 |
| csoriano | however with make install I get a "read-only" file system | 20:30 |
| csoriano | what's the expected workflow there? | 20:31 |
| *** lachlan has quit IRC | 20:39 | |
| *** gokcennurlu has quit IRC | 21:22 | |
| gitlab-br-bot | mbodmer opened issue #964 (UX: Development process, using IDE, index code) on buildstream https://gitlab.com/BuildStream/buildstream/issues/964 | 21:57 |
| *** alatiera has quit IRC | 23:14 | |
| *** nimish2711 has joined #buildstream | 23:17 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!