| *** tristan has quit IRC | 05:49 | |
| *** tristan has joined #buildstream | 06:06 | |
| *** ChanServ sets mode: +o tristan | 06:06 | |
| *** hasebastian has joined #buildstream | 06:42 | |
| tristan | Hmmmmmmmm | 06:43 |
|---|---|---|
| tristan | juergbi, what is your take on this line: https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_variables.pyx#L302 ? | 06:43 |
| tristan | I think this is pretty weak, we should be able to detect a cyclic variable resolution the first time we encounter the same variable twice during resolution | 06:44 |
| tristan | And it's not impossible that we need to resolve 1000 variables to expand one variable :-S | 06:44 |
| juergbi | tristan: the question is, would the required state tracking to detect cyclic cases have a significant performance overhead? | 06:45 |
| juergbi | I suppose we could store the expanded value in each variable | 06:46 |
| juergbi | and add a boolean flag to track expansion | 06:47 |
| juergbi | if it's marked as in progress, we have a cycle | 06:47 |
| juergbi | not sure whether that's feasible/sensible with our variable inheritance etc., though | 06:47 |
| juergbi | I don't think I've been involved in variable expansion code, maybe benschubert has some thoughts on this | 06:48 |
| tristan | Yeah, I've pinged him on https://gitlab.com/BuildStream/buildstream/-/merge_requests/1951#note_365364450 | 06:51 |
| tristan | I think regardless of this, we should probably make a policy to avoid raising Python core defined exceptions | 06:52 |
| tristan | This is in fact BuildStream raising an exception which is saying "I found a cyclic dependency !", but disguising it as if it were a python core "Maximum stack depth reached !" error | 06:53 |
| juergbi | probably not ideal here, yes. there may be valid situations of raising built-in exceptions, not sure | 06:54 |
| tristan | Maybe I should not block !1951 on this, not sure | 06:55 |
| tristan | juergbi, fwiw, I've reconsidered Element.search() (I rebased that on the juncle this morning), and found that... we would really need to leverage the loader code for Element.search() instead of a blind dependency walk | 06:58 |
| tristan | Since project relative element paths can be changed depending on overrides and links, we cannot really expect the project author to ever have exact knowledge of what a fully resolved path would be | 06:59 |
| tristan | This might actually be more performant, depending on how it's implemented | 07:00 |
| tristan | A bit of a can of worms though, might require some rules to follow, like "Element.search() must only be called in Plugin.configure() or Plugin.preflight() implementations" | 07:01 |
| tristan | If we want to reserve the right to detach the Loader tree from elements and discard it after the load | 07:02 |
| juergbi | the main (only?) use case is for something like script `layout` where plugin-specific metadata is configured for dependencies, right? | 07:02 |
| tristan | It can only be useful for an element's dependencies (for now) | 07:03 |
| tristan | I.e. yes, it's "A user provides a path, and the plugin uses that to find an element within it's scope" | 07:03 |
| juergbi | in which case the most important aspect is that it finds the correct element when passing exactly the same string that is used in the dependency list | 07:03 |
| tristan | juergbi, I could imagine it being similarly useful in "Composite plugins" if they do materialize too | 07:03 |
| tristan | Yes | 07:04 |
| tristan | Well | 07:04 |
| tristan | No | 07:04 |
| tristan | juergbi, There is no requirement, and I don't think there should be, that the dependency must be explicitly listed in the dependency list itself | 07:04 |
| tristan | I.e. it can very well be anything which one of it's explicitly listed dependencies depended on | 07:05 |
| tristan | Not having it work that way could be quite confusing | 07:05 |
| juergbi | right, there might be valid use cases for that. however, resolution/search should be identical as if the string was listed as a dependency | 07:05 |
| tristan | Yes | 07:06 |
| juergbi | so, yes, making use of loader facility sounds sensible | 07:06 |
| tristan | If you've taken any look at the juncle patches, I think you'll agree that the search paths through the Loader are optimal | 07:06 |
| tristan | I've added Loader.get_loader(... load_subprojects=True/False ...) | 07:07 |
| tristan | Which now allows lightweight traversals | 07:07 |
| tristan | What I've been thinking though is, after obtaining the element, how do we know it's in the scope ? | 07:08 |
| juergbi | still on my todo list, planning to take a look later today | 07:08 |
| juergbi | ah, we don't want non-dependencies in there, of course | 07:08 |
| tristan | yeah thats a bit tricky | 07:08 |
| tristan | Maybe we already have some reverse dependency caching | 07:09 |
| juergbi | we could still traverse after the lookup, of course, but that's not optimal | 07:09 |
| tristan | I think that also, it would not be a problem to slightly break the Element.search() API at this point to remove the `scope` argument, if that simplifies things | 07:09 |
| juergbi | that said, I don't think it's in a performance critical path at all | 07:09 |
| tristan | No ? | 07:10 |
| tristan | Maybe not | 07:10 |
| juergbi | it's normally used as part of assemble, isn't it? | 07:10 |
| tristan | It only adds a bit | 07:10 |
| juergbi | not as part of loading | 07:10 |
| tristan | Not necessarily no, and I think it would make sense to restrict it to loading | 07:10 |
| tristan | To keep our options open | 07:10 |
| juergbi | without `scope` my fear is that plugins then start to actually inspect non-dependency elements in the pipeline | 07:10 |
| juergbi | and this would cause all kinds of problems | 07:10 |
| juergbi | including cache key not being good enough because it's not affected by changes outside the dependency chain | 07:11 |
| tristan | If we allow invoking loader code in assemble(), that paints us into a corner where the Loader must be present at assemble() time (which may mean things like pickling or such, depending on future refactors) | 07:11 |
| tristan | juergbi, The `scope` only limits the search criteria though | 07:12 |
| tristan | i.e. RUN/BUILD/ALL | 07:12 |
| tristan | If it's got an element name/path, there can really only be one | 07:12 |
| tristan | That said, we could keep it, I'm just thinking it might be more performant to not have it | 07:12 |
| juergbi | tristan: yes, the actual scope aspect is not that important to me. however, we need to make sure the returned element is actually a dependency (of any kind) of `self` | 07:12 |
| tristan | if we could solve the issue of "is this a dependency" more efficiently | 07:13 |
| juergbi | or do we guarantee this some other way even by going through the loader? | 07:13 |
| juergbi | sure, as long as we have that check I'm happy | 07:13 |
| tristan | No no... I was just saying that Element.search(path, scope) could lose the scope argument | 07:13 |
| tristan | Ofcourse it must still return only dependencies | 07:13 |
| juergbi | good | 07:14 |
| juergbi | tristan: regarding 'only as part of loading'. scriptelement currently uses it in stage() | 07:14 |
| tristan | Yeah, we could break that | 07:14 |
| tristan | Currently Element.search() does a dependency walk, which means it's safe to use for the element's lifetime | 07:15 |
| tristan | If we change it to leverage the Loader code, it could be in our interest to enforce that it be used only in configure/preflight | 07:15 |
| tristan | Raise an error if it's called out of bounds (not a hard thing for plugins, just hold onto the instance after looking it up at load time) | 07:15 |
| juergbi | seems somewhat inconvenient and at that point performance is actually relevant | 07:15 |
| tristan | It also seems quite inconvenient to promise that the loader structures are around after the load :-S | 07:16 |
| juergbi | yes... | 07:16 |
| tristan | I mean, we could code it that way, and refactoring would just be more expensive | 07:17 |
| tristan | Like, a way out of that mess would be to retain some shallow/mirror copy of element names paths post-loader-era | 07:17 |
| juergbi | I think for many/most use cases the ideal solution would actually be to allow plugin-specific config directly in the dependency list | 07:17 |
| juergbi | but we probably don't want to go there now | 07:17 |
| tristan | I think we need the element to interact with dependency parsing if we wanted that | 07:18 |
| tristan | (but yeah, it would reduce typing if we could do that, seems a bit of an obscure/unclear road to go down right now though) | 07:20 |
| tristan | Ok so for a first iteration, I will do a Loader based search + an Element.dependencies() walk and not change API | 07:21 |
| tristan | Scope reduces the cost of a dependencies() walk, but if we had a different cached approach, it might make sense to remove later | 07:22 |
| tristan | And allow calling Element.search() at any point, at the cost of tying the Loader to the Element lifecycles | 07:22 |
| juergbi | tristan: or maybe a compromise, require preflight() to call search() for the loader-based lookup and then allow search() to be called again in stage/assemble but only for those strings that have been searched for already in preflight() | 07:24 |
| juergbi | the latter would result in a local hash table lookup instead of loader | 07:24 |
| *** benschubert has joined #buildstream | 07:24 | |
| juergbi | might be too confusing | 07:24 |
| juergbi | maybe it would be less confusing to use two different methods | 07:25 |
| juergbi | the idea of this would be that individual plugins don't have to fill a hash table on their own for this (as I imagine that would be a common pattern in plugins if we restrict to preflight) | 07:26 |
| tristan | Hmmm, sounds not very pretty | 07:32 |
| tristan | we'd have an additional API like Element.resolve_search_path(), to allow Element.search() to be called later | 07:33 |
| tristan | iiuc | 07:33 |
| tristan | Morning benschubert :) | 07:33 |
| benschubert | It seems I made a habit of landing in the middle of important conversations :'D | 07:34 |
| benschubert | morning! | 07:34 |
| tristan | It's just fun times these days | 07:34 |
| tristan | I had a question for you, pinged you on https://gitlab.com/BuildStream/buildstream/-/merge_requests/1951#note_365364450 | 07:34 |
| benschubert | yeah, will have a look | 07:34 |
| tristan | benschubert, this conversation was about how Element.search() needs to leverage Loader code, because true resolved element paths cannot realistically be known by project authors, fwiw | 07:36 |
| benschubert | ok, anything I can help with for this discussion or is it pretty much settled? | 07:37 |
| tristan | benschubert, it could certainly benefit from your input if you want to look over it | 07:38 |
| benschubert | ok, let me dig the logs | 07:38 |
| tristan | I've got an idea of what to do for now, you might have a different one :) | 07:38 |
| tristan | benschubert, starts on this line: https://irclogs.baserock.org/buildstream/%23buildstream.2020-06-22.log.html#t2020-06-22T06:58:49 (not too long) | 07:39 |
| benschubert | Ok, went through it. | 07:45 |
| benschubert | First question: Do we know (or remember) how much % of memory is used by the loader caches? | 07:45 |
| tristan | benschubert, I would hazard a guess that it's >= 50% | 07:46 |
| tristan | benschubert, given that we have LoadElements and then another pass of MetaElements, it could even be as much as 60% or 70% | 07:46 |
| tristan | But that is less of a concern than the tangling of unneeded/redundant data structures in the data model | 07:47 |
| tristan | Clarity and refactorability is severely compromised I think by having the Loader around | 07:47 |
| tristan | severely might be too strong a word, I mean it can be livable | 07:48 |
| tristan | but still | 07:48 |
| benschubert | fair point | 07:49 |
| benschubert | Should we actually in the loader generate the whole graph and keep the graph around? Possibly pruned to keep only what we want | 07:50 |
| benschubert | and provide APIs to access it | 07:50 |
| benschubert | it's not pretty, and the loader is already a hot spot pre-scheduling | 07:50 |
| benschubert | but it would at least not keep the loader around | 07:50 |
| tristan | Maybe something similar to that | 07:51 |
| tristan | Integrated somehow directly into the Element data model | 07:51 |
| tristan | additional metadata cherry picked from the loader | 07:51 |
| *** tomaz has joined #buildstream | 07:51 | |
| tristan | Currently we do have these references to the loader still around, dangling on Project | 07:52 |
| tristan | And we just don't bother NULL-ing them out | 07:52 |
| tristan | (a side effect of this was that for some reason, the pickle jobber was pickling loaders) | 07:52 |
| tristan | benschubert, However we are not very good I think, for instance, I'm not convinced that in cli.py, the entire element data model gets properly GC'ed after a Stream invocation | 07:53 |
| benschubert | tristan: do you have an example of where that happens? | 07:54 |
| tristan | I recall I had the WeakValueTable thing in Plugin for some time and when it was provoked to actually work, things crashed | 07:54 |
| WSalmon | hi tomaz | 07:54 |
| WSalmon | o/ | 07:54 |
| tomaz | o/ | 07:55 |
| tristan | benschubert, Well, we could check, apparently if we run `bst --debug` we should hit: https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/plugin.py#L277 | 07:56 |
| tristan | For all instantiated plugins | 07:56 |
| tristan | benschubert, honestly I havent looked for a very long time, maybe that is working now | 07:56 |
| benschubert | tristan: There are a few things at play, but the main one is that the GC for circular definitions runs periodically, so it migth not trigger before we end the program, in which case all the `__del__` are skipped since we are just exiting | 07:57 |
| benschubert | If you believe we have things that should be destroyed earlier, I'm happy to have a look so we can save even more ram :) | 07:58 |
| benschubert | I also answered on !1951, does it make sense? | 07:58 |
| tristan | Well, it can make sense to manually provoke the GC at the exit of app.initialized() context manager, e.g.: https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_frontend/cli.py#L448 | 07:58 |
| benschubert | manually provoking it is not the best way of going through. We had tried this before and it turned out to be worse performance wise than letting it do its job | 07:59 |
| benschubert | We _could_ help it by explicitely deleting a few items in a cycle we want gone though | 08:00 |
| tristan | Ah, I don't mean for performance, just for correctness | 08:00 |
| tristan | benschubert, We could have an internal BST_TEST environment check in app.initialized(), and have a test case which scans result.stderr() counting the amount of elements/sources which get destroyed | 08:01 |
| benschubert | Well, the objects would not be accessible by anyone unless they go through the GC, which... well if someone goes that far to get something I'd rather give them an API :'D | 08:01 |
| tristan | only manually provoking the GC under test | 08:01 |
| tristan | But ensuring beyond any doubt that we properly cleanup everything upon session completion | 08:01 |
| benschubert | that could be nice indeed, to ensure we don't keep them around because we didn't cleanup | 08:01 |
| benschubert | but do we have cases where we call app.initialized multiple times? | 08:01 |
| *** samwilson has joined #buildstream | 08:02 | |
| benschubert | If not, I'm not sure it's worth the effort, since it's basically the end of the program at that point | 08:02 |
| tristan | benschubert, Not real world ones afaict | 08:02 |
| tristan | only via test cases | 08:02 |
| tristan | benschubert, Right, I brought this up in conjunction with keeping the Loader around and destroying it early on, I mean; if we're going to clean up it seems to make sense to ensure it at all levels | 08:03 |
| tristan | Nulling out the Project.loader members after load time doesnt appear to be meaningful unless we can make some guarantees that we don't ever leak things in general | 08:04 |
| tristan | or maybe that's just my own perception of the thing :) | 08:04 |
| benschubert | Ideally I'd say yes. | 08:05 |
| benschubert | Practically, gc has a cost, and unless the relieve memory pressure is a benefit I'd rather not mock around with it. | 08:05 |
| benschubert | If you tried now calling the gc at the end of the build, it _could_ just hang your process for a few seconds, and the users would be like `wut?` | 08:05 |
| tristan | Yes I agree, which is why I would only want to do it in CI | 08:06 |
| benschubert | so if we keep the assertions in tests and it's not too invasive I'm happy with it (We should definitely help the gc), but I think we need to be careful | 08:06 |
| benschubert | yep righ | 08:06 |
| benschubert | *right | 08:06 |
| tristan | It's just a matter of knowing that you've actually freed something | 08:06 |
| tristan | Setting a member to None doesnt really mean anything to me :-S | 08:06 |
| benschubert | When do you think you will be looking at the scheduler / frontend split ? | 08:11 |
| benschubert | tristan: ^ | 08:11 |
| tristan | Am I ? | 08:11 |
| benschubert | Just to ensure we don't conflict too much with my POC on the multiprocessing | 08:12 |
| benschubert | ah I thought you said you wanted to look at it again? | 08:12 |
| benschubert | Nevermind then :) | 08:12 |
| tristan | benschubert, I think last I heard of that, I pasted the issue number here and asked you if you wanted to do the honors of closing it | 08:12 |
| tristan | Is it dead or not ? | 08:12 |
| benschubert | ah sorry | 08:12 |
| benschubert | yeah it's dead at least for now | 08:12 |
| benschubert | completely misunderstood :) | 08:12 |
| tristan | My impressions from our last conversation was that it was considered a failed experiment, which ended up being even less performant | 08:13 |
| benschubert | yes that is true | 08:13 |
| tristan | So we should close it for clarity | 08:13 |
| benschubert | I still thought we wanted to bring in some cleanups to clearly separate the frontend and state? | 08:14 |
| tristan | We do want to do that, but some of the measures taken in that code which landed in master actually made things more complex (i.e. the Scheduler grew new state change delivery codepaths instead of encapsulating this in State()) | 08:15 |
| benschubert | ah true, so you meant cleaning that up gotcha | 08:15 |
| tristan | I would want to purify State() further, make sure its the thing that you pass around in the frondend for notification receivals, and the thing you pass around to core components to notify of state changes | 08:15 |
| tristan | And if there is ever a scheduler/frontend process split, ensure that that stays encapsulated inside _state.py | 08:16 |
| benschubert | seems sensible | 08:16 |
| tristan | by way of Queues and such | 08:16 |
| benschubert | I'll close the MR then | 08:16 |
| * tristan goes to grab a bite... | 08:17 | |
| *** tristan has quit IRC | 08:20 | |
| *** santi has joined #buildstream | 08:21 | |
| *** cphang has joined #buildstream | 08:22 | |
| coldtom | hi! anyone seen anything like https://gitlab.com/libreml/libreml/-/jobs/604142338 before? | 08:23 |
| * juergbi hasn't | 08:32 | |
| WSalmon | tristan how do i get a web link to things sent to the new mailing list | 08:33 |
| *** tristan has joined #buildstream | 09:12 | |
| *** ChanServ sets mode: +o tristan | 09:12 | |
| douglaswinship | coldtom: no, never seen it :( | 09:18 |
| *** hasebastian has quit IRC | 09:24 | |
| *** Frazer has joined #buildstream | 09:41 | |
| tristan | benschubert, I replied to the RecursionError thing, I'm wondering a couple of things... (A) Should I block !1951 on fixing this together ? I worry that when we leave these things aside they have a tendency of being forgotten... and (B) Do you think it will be okay to do the cyclic variable detection with a hashtable/dict like we used to ? | 09:42 |
| tristan | Regarding (B), I have a suspicion that you would not have left it like that if it did not represent a performance issue, I can't fathom why it would be like that otherwise | 09:43 |
| benschubert | tristan: A) we should not get it in if we only rely on the RecursionError, otherwise we'll get nice segfaults | 09:43 |
| benschubert | Regarding B) I think we should try again, a lot of this code has changed now and it might be worth a new benchmark | 09:44 |
| benschubert | because it is indeed simpler | 09:44 |
| tristan | Ok, is the benchmarking thing that you do available for anyone to do ? | 09:44 |
| tristan | is it finally the 'benchmarks' repo in the BuildStream group on gitlab which "just works" and get's everything right and gives us a nice report ? | 09:45 |
| benschubert | I don't have any benchmarking for variables :/ | 09:45 |
| benschubert | It is, but doesn't contain any variables | 09:46 |
| benschubert | it'I think valentind had a nice test which should be linked somewhere, give me a sec | 09:46 |
| * tristan watches !1951 scope creep... | 09:46 | |
| tristan | This fixes a real problem which we need to backport to 1.x pronto also :-S | 09:46 |
| benschubert | https://gitlab.com/BuildStream/buildstream/-/merge_requests/1949#note_352922097 | 09:47 |
| tristan | Ah nice | 09:47 |
| tristan | Ok, I'll give this a little go then | 09:47 |
| benschubert | thanks. let me know if I cna help | 09:47 |
| Frazer | hi, has there been any work on https://gitlab.com/BuildStream/buildstream/-/issues/408 ? planning on starting work on it and just checking if there's anything I should know first or any tips to get started | 09:51 |
| Frazer | ? | 09:51 |
| tristan | Frazer, I don't know of any, I would suggest whatever you do, start by formulating a plan of what precisely you intend to do and post it to the mailing list | 09:57 |
| tristan | I.e. follow the proposal process, it's unclear to me what can/should be done | 09:57 |
| *** tristan changes topic to "BuildStream 1.4.3 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://lists.apache.org/list.html?dev@buildstream.apache.org | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmaps" | 09:57 | |
| *** tristan changes topic to "BuildStream 1.4.3 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://lists.apache.org/list.html?dev@buildstream.apache.org" | 09:58 | |
| * tristan nukes the 'roadmaps' from the subject line, that's been stale for years | 09:58 | |
| Frazer | thanks tristan, will investigate a solution | 10:02 |
| tristan | benschubert, So maybe we need to keep raising RecursionError but never catch it ? At least until such a time that the code gets refactored to do variable resolution iteratively instead of recursively ? | 10:09 |
| tristan | Make sense ? Add cyclic dependency of variable checks separately, but keep the counter in place and raise an unhandled RecursionError with a corresponding comment ? | 10:10 |
| * tristan wonders if he can easily make this iterative | 10:11 | |
| tristan | Create a list, scan the dictionary in a loop, searching for referenced variables and building a list ? | 10:13 |
| tristan | Then the list is already ordered correctly, and resolve it ? | 10:13 |
| valentind | tristan, we cannot resolve all variables. | 10:14 |
| valentind | That was the point of the merge request. | 10:14 |
| tristan | valentind, I mean in the context of one variable resolution | 10:14 |
| valentind | Sure then. | 10:14 |
| tristan | valentind, it appears that we need to care about reaching the upper limits of the stack, ergo the RecursionError, I'm trying to differentiate this RecursionError from a cyclic dependency error | 10:15 |
| tristan | It appears that we just assume that a RecursionError means there was a cyclic dependency | 10:15 |
| tristan | So, maybe just eliminate the recursion altogether is a good first step | 10:16 |
| benschubert | tristan: would make sense yes | 10:16 |
| tristan | strange, we also have _check_for_cycles() here | 10:17 |
| tristan | So at least in the context of _variables.pyx, we don't have anything causing a subproject to be loaded | 10:17 |
| tristan | cycle_check is also recursive itself | 10:18 |
| tristan | sigh, ok... lets do this | 10:18 |
| tristan | Do we want to remove _check_for_cycles() and _check_for_missing() altogether ? | 10:19 |
| tristan | It seems to me that we assert the validity of unused variables this way | 10:19 |
| valentind | Yes, it is for the unused one. | 10:19 |
| valentind | If we had the evaluation to better support evaluation, then for full check we could just evaluate everything. | 10:20 |
| tristan | So how about this... (A) Make Variables.subst() handle missing variables and cyclic dependencies... (B) Resolve them on the fly, keeping a cache of resolved variables... (C) Resolve every unresolved variable (at some point) | 10:22 |
| tristan | valentind, right I think I'm saying the same thing | 10:23 |
| tristan | however | 10:23 |
| tristan | valentind, Right now I'm based on your 'partial-variables' branch | 10:23 |
| tristan | valentind, does it make sense that Variables.__init__() still does _check_for_missing() and _check_for_cycles() unconditionally ? | 10:24 |
| tristan | Should there not be an explicit API to call from outside of Variables() at some point when the core knows that everything is fully loaded ? | 10:24 |
| tristan | Ahhhh | 10:24 |
| tristan | valentind, Scratch that | 10:24 |
| tristan | For some reason I had my branch based on your branch, but my buffer still had a copy of master _variables.pyx | 10:25 |
| tristan | So you have check() for that | 10:25 |
| tristan | Which remains undocumented ! | 10:25 |
| tristan | And strangely sitting amongst the __special_python__(functions, such, as, __init__) | 10:26 |
| tristan | interesting, python has intern strings too, natively ? | 10:27 |
| benschubert | yep, though I'm not sure how much we gain by that | 10:29 |
| benschubert | nor even if we gain | 10:29 |
| benschubert | but lacking good benchmarks for that... :'D | 10:29 |
| tristan | Oddly, we're using the interns for the values, not the keys :-/ | 10:29 |
| * tristan struggles to find the best pun about hiring these interns, and gives up | 10:31 | |
| valentind | tristan, the problem is that most of variables are shared for all elements. Using sys.intern can help with using less memory. | 10:38 |
| valentind | At least that was my understanding why sys.intern was used for the values. | 10:38 |
| benschubert | valentind: correct, it also allows comparing them with a pointer comparison instead, which is much faster than __eq__. At least in theory | 10:59 |
| tomaz | when I compile different versions of a software do I keep the cache or buildstream nukes everything? | 11:06 |
| tristan | benschubert, right, we use intern strings in glib for address comparisons too... and the docs for sys.intern() in python mention that dictionaries are more performant when using them as dictionary keys (which makes sense) | 11:59 |
| * tristan got caught up with a few other topics, guess I will get to this tomorrow finally :-S | 11:59 | |
| tristan | Anyway... I'll step out for now, have a later meeting lined up | 12:00 |
| coldtom | tomaz: cache keys are based on the element definitions, so if you change the version of an element, it's no longer cached at all. | 12:01 |
| coldtom | if you're building in strict mode (the default), then you will have to build all of a changed element's reverse dependencies again too, in non-strict you will only have to rebuild the element itself | 12:01 |
| tristan | benschubert, if you could also take a look at !1901 (juergbi will also take a look), that would be helpful, lets land this huge branch :) | 12:01 |
| *** phildawson has joined #buildstream | 12:01 | |
| tristan | tomaz,https://docs.buildstream.build/master/using_config.html#local-cache-expiry | 12:02 |
| tristan | I hope those docs are still relevant, I would guess that they are, though | 12:03 |
| tristan | tomaz, this is also supported in bst-1 if that's what you're using, expiry itself is more expensive in bst-1, though | 12:03 |
| benschubert | tristan: added to my todos for today | 12:06 |
| tristan | Thanks ! | 12:07 |
| *** tristan has quit IRC | 12:17 | |
| *** lantw44 has joined #buildstream | 12:33 | |
| *** lantw44 has quit IRC | 12:35 | |
| *** xjuan has joined #buildstream | 12:54 | |
| Frazer | hi im trying to do the BuildStream example but seem to be getting this error https://pastebin.com/eur6Q73K i seem to get it when trying to build freedesktop-sdk too | 13:48 |
| douglaswinship | Can anyone shed any light on https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/603658395#L1974 ? | 13:58 |
| douglaswinship | I'm getting the error "linux32: failed to set personality to linux32: Invalid argument" | 13:58 |
| douglaswinship | That's on our gitlab-CI runners, for freedesktop-sdk | 13:58 |
| douglaswinship | I tried googling for the error, and the only result was chat logs from this channel, from February | 13:59 |
| douglaswinship | juergbi: looks like you encountered it in some situation? | 13:59 |
| juergbi | Frazer: can you paste your ~/.config/buildstream.conf ? | 14:03 |
| juergbi | douglaswinship: this sounds like the CI server (kernel or hardware) doesn't support 32-bit ARM | 14:04 |
| juergbi | or there is a bug | 14:04 |
| Frazer | juergbi thanks, i think found the issue. was a broken .conf from previous testing | 14:04 |
| juergbi | Frazer: it would be good to fix the error message in such cases. can you please open issue? unless that's already fixed in master | 14:05 |
| Frazer | will have a look | 14:06 |
| juergbi | douglaswinship: mayit could also be that you don't even want to run 32-bit ARM code and `build-arch` isn't set correctly in the sandbox config | 14:07 |
| *** tristan has joined #buildstream | 14:17 | |
| *** ChanServ sets mode: +o tristan | 14:17 | |
| *** xjuan has quit IRC | 14:22 | |
| *** tpollard has joined #buildstream | 14:25 | |
| douglaswinship | So I had an interesting overlap-whitelist issue. | 16:19 |
| douglaswinship | https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/elements/vm/config/resolved.bst#L20 | 16:19 |
| douglaswinship | That line "- etc/systemd/resolved.conf", presumably works fine for the master branch of freedesktop-sdk (or it would have been fixed by now) | 16:20 |
| douglaswinship | But it didn't work for me on the bst 2 branch i'm trying. | 16:20 |
| douglaswinship | I had to put a slash at the beginning. | 16:20 |
| douglaswinship | I got an "overlap" complaint, and the build failed. | 16:20 |
| douglaswinship | When I changed it to "- /etc/systemd/resolved.conf", it built okay. | 16:20 |
| douglaswinship | (at least locally). | 16:20 |
| tpollard | Maybe it enforces absolute paths now? | 16:21 |
| douglaswinship | That's just what I was thinking. | 16:21 |
| douglaswinship | Although it didn't complain about the path being in the wrong format, or anything. | 16:22 |
| douglaswinship | I was wondering if it was somehow treating it as a relative path, starting from some place that isn't root. So it was a valid statement but not the correct valid statement. | 16:22 |
| douglaswinship | Was just wondering if anyone's familiar. | 16:23 |
| *** tpollard has quit IRC | 16:26 | |
| tristan | douglaswinship, that rings a bell | 16:37 |
| tristan | douglaswinship, 1ae1796854055f4b9e3202d9629e059b327f3a8d | 16:38 |
| tristan | https://gitlab.com/BuildStream/buildstream/-/issues/721 | 16:38 |
| douglaswinship | Aha! thanks. | 16:39 |
| douglaswinship | Good to know. | 16:39 |
| tristan | that apparently was fixed a very long time ago though | 16:39 |
| tristan | maybe only in master but not in 1.x | 16:39 |
| tristan | Ah yes indeed | 16:40 |
| tristan | douglaswinship, note https://gitlab.com/BuildStream/buildstream/-/commit/3ae078ce0547842b26785638671a94bbd4854e59 | 16:40 |
| tristan | this backport to 1.x specifically continues to support lack of leading slash, for back compat in 1.x but not in master | 16:41 |
| tristan | makes sense | 16:41 |
| *** samwilson has quit IRC | 17:20 | |
| *** santi has quit IRC | 17:28 | |
| *** santi has joined #buildstream | 17:28 | |
| *** santi has quit IRC | 18:35 | |
| *** santi has joined #buildstream | 18:47 | |
| *** santi has quit IRC | 19:09 | |
| *** tomaz has quit IRC | 19:47 | |
| *** toscalix has joined #buildstream | 20:09 | |
| *** toscalix has quit IRC | 23:07 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!