| *** tristan_ has joined #buildstream | 01:36 | |
| *** tristan_ has quit IRC | 01:38 | |
| *** tristan_ has joined #buildstream | 02:31 | |
| *** tristan_ has quit IRC | 02:38 | |
| *** tristan_ has joined #buildstream | 03:37 | |
| *** tristan has quit IRC | 03:39 | |
| *** tristan_ has quit IRC | 04:28 | |
| *** tristan has joined #buildstream | 04:36 | |
| juergbi | tristan: no second Sunday I'm aware of ;) | 05:16 |
|---|---|---|
| juergbi | just different time zones, afaict | 05:16 |
| tlater[m] | We'll have a second sunday on the 26th iirc | 07:19 |
| tlater[m] | Also guadec is coming up :) | 07:19 |
| *** bochecha has joined #buildstream | 07:58 | |
| *** bochecha_ has joined #buildstream | 08:00 | |
| *** bochecha has quit IRC | 08:02 | |
| *** bochecha_ is now known as bochecha | 08:02 | |
| * tlater[m] wonders why `tox -e venv` takes so long | 08:10 | |
| tlater[m] | Surely it should just reuse the same ol' venv every time | 08:11 |
| tlater[m] | But it looks to be rebuilding it constantly | 08:11 |
| jennis | Installing pyroaring? | 08:11 |
| jennis | Oh right, does it look like it's reinstalling into a venv each time? | 08:11 |
| tlater[m] | Yeah | 08:12 |
| tlater[m] | This is what I get, in case you're curious: https://pastebin.com/raw/KcKdpiPm | 08:14 |
| tlater[m] | Is it because bst is not installed as editable? | 08:14 |
| tlater[m] | The whole game takes about a minute :| | 08:15 |
| jennis | <tlater[m]> Is it because bst is not installed as editable? | 08:28 |
| jennis | Potentially | 08:28 |
| jennis | Have you tested without? | 08:28 |
| tlater[m] | jennis: Frankly, I have no idea how tox works | 08:29 |
| * tlater[m] will need to check docs | 08:29 | |
| jennis | hehe, me neither, but having looked at the tox.ini, it doesn't seem too complicated | 08:29 |
| tlater[m] | No, but there's no setting for "don't reinstall the package every time I run you you dimwit" anywhere so far | 08:31 |
| jennis | tlater[m], does your `tox` on it's own reinstall everytime? | 08:31 |
| jennis | So running the test suite | 08:32 |
| *** tme5 has joined #buildstream | 08:32 | |
| tlater[m] | It looks like it | 08:33 |
| tlater[m] | Well, it reinstalls buildstream every time | 08:33 |
| tlater[m] | Which I don't normally notice because the test suite takes forever anyway, but I'm starting to use it for my one-off smoke test commands as well. | 08:34 |
| tlater[m] | And now that extra minute is too long x) | 08:34 |
| jennis | ok, mine doesn't | 08:37 |
| jennis | Mine seems to only reinstall if I pass the --recreate (or -r) flag | 08:38 |
| * tlater[m] wonders if a bst remote key store in ~/.local/share/bst/keys would make sense | 08:39 | |
| tlater[m] | Then just read the keys from a remote-specific dir with default names | 08:39 |
| tlater[m] | jennis: Hm, I missed that, does this also happen when you run tox -e venv? | 08:58 |
| gitlab-br-bot | marge-bot123 closed issue #1096 (BuildStream cannot show logs of a workspaced build) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1096 | 09:19 |
| gitlab-br-bot | marge-bot123 merged MR !1536 (jennis/fix_failed_workspaces->master: Don't reset a failed (but cached) workspaced Element) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1536 | 09:19 |
| *** jonathanmaw has joined #buildstream | 09:21 | |
| *** lachlan has joined #buildstream | 09:26 | |
| jennis | tlater[m], I'll have a try | 09:29 |
| jennis | tlater[m]: Error: No such command "bst". | 09:44 |
| jennis | ha | 09:44 |
| *** traveltissues has joined #buildstream | 09:44 | |
| jennis | `tox -e venv -- bst --no-colors --on-error terminate -C tests/integration/project bst build target.bst` | 09:44 |
| tlater[m] | Beautiful | 09:45 |
| tlater[m] | Have you tried re, in case it already exists and is messed up?? | 09:45 |
| tme5 | how do I run tests locally? i'm getting setup errors with pytest | 09:48 |
| tlater[m] | tme5: Just run `tox -e py37` in the buildstream directory | 09:49 |
| tlater[m] | Caveat, you may need to have tox installed | 09:49 |
| tlater[m] | Refer to your distribution's package manager for information on how to install that | 09:49 |
| tme5 | ahh ok, can i select tests with that? | 09:49 |
| tlater[m] | Yeah, it's something like `tox -e py37 -- tests/artifactcache/something.py::test_the_thing` | 09:50 |
| tlater[m] | And you can also do things like `tox -e py37 -- --no-cov --integration` | 09:50 |
| tlater[m] | See CONTRIBUTING.md for more :) | 09:50 |
| tme5 | great, ty :) | 09:50 |
| jennis | tme5, or just `pip install tox` | 09:50 |
| * tlater[m] generally recommends distribution packages over pip if they exist | 09:51 | |
| tlater[m] | But yeah, if there is no such package, pip will do :) | 09:51 |
| jennis | I do too, but I've sometimes had the problem of having a version which is not up-to-date enough | 09:51 |
| tlater[m] | Hehe, the downsides of ubuntu ;p | 09:52 |
| * tlater[m] will not further engage in the distro wars, don't worry | 09:52 | |
| jennis | hehe | 09:53 |
| jennis | tlater[m], didn't work with the recreate flag either | 09:53 |
| adds68 | Could anyone give this review an look over, i think the CI that failed is a known failure? https://gitlab.com/BuildStream/buildstream-docker-images/pipelines/75988489 | 09:53 |
| adds68 | I think my non zero exit status errors may be being caused by this | 09:53 |
| tme5 | tlater[m], jennis, unfortunately the case with pytest on stretch, two major versions behind | 09:54 |
| tlater[m] | jennis: I'm going to have to really dig into tox to get comfortable with it I suppose |: | 09:55 |
| tlater[m] | Yeah, these sorts of things is why I'm on rolling release stuff now. That and I have too much free time. | 09:56 |
| tlater[m] | tme5: tox will install the correct versions for you though | 09:56 |
| tlater[m] | If you can get the right version of tox, which is pretty easy with pip | 09:56 |
| gitlab-br-bot | tpollard approved MR !1524 (jennis/push_unbuilt_artifact->master: Ensure push fails when trying to push an unbuilt element) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1524 | 09:57 |
| tlater[m] | It also does everything in virtualenvs, so you won't end up with frankenpython | 09:57 |
| jennis | adds68, not sure about that one, perhaps retry that specific test to check? | 09:59 |
| adds68 | jennis, yea i've just clicked retry, lets see what happens | 09:59 |
| gitlab-br-bot | jennis opened issue #1103 (Implement `bst artifact show`) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1103 | 10:16 |
| *** traveltissues has quit IRC | 10:28 | |
| *** traveltissues has joined #buildstream | 10:28 | |
| tme5 | i'm getting a lint error for something I haven't changed :( | 10:32 |
| tlater[m] | tme5: Mind sharing the exact error? | 10:33 |
| tlater[m] | Preferably via pastebin ;) | 10:34 |
| tme5 | there's a warning for "Redefining name 'artifact' from outer scope" on this line https://gitlab.com/BuildStream/buildstream/merge_requests/1541/diffs#48a5425791d6d9d502dfdb9fad1e14e38ec05629_962_962 | 10:41 |
| tpollard | ah, click funs | 10:41 |
| tlater[m] | tme5: Don't name your variable artifact :) | 10:42 |
| laurence | BuildStream is now being tested in the RE-API test suite project ! https://gitlab.com/remote-apis-testing/remote-apis-testing | 10:42 |
| laurence | Building against BuildGrid only for now | 10:42 |
| tlater[m] | tme5: The convention is artifact_ if you can't think of anything else. | 10:42 |
| laurence | Hopefully Buildfarm and Buildbarn soon as well | 10:42 |
| tme5 | oh i see! line 962 is the *original* definition | 10:42 |
| tlater[m] | In a week or three laurence! :) | 10:43 |
| tme5 | 1013 is the offending line, got it, thank you | 10:43 |
| tlater[m] | tme5: Yup. Unfortunately python shares scope between functions and also shares names between variables and functions, so these errors are pretty common. | 10:43 |
| tlater[m] | Especially in cli.py, because everything is an artifact | 10:44 |
| tlater[m] | Hehe | 10:44 |
| tlater[m] | Does anyone know if we ever pull anything from remote source caches? | 10:45 |
| * tlater[m] only sees code to make buildstream push sources in `steam.build` | 10:45 | |
| jennis | raoul ^ | 10:46 |
| *** alexandrufazakas has joined #buildstream | 10:46 | |
| jennis | I'm looking at being able to load a dependency graph given an artifact ref and I'm wondering what dependencies we need to add to the proto | 10:52 |
| *** lachlan has quit IRC | 10:52 | |
| jennis | Currently the proto stores build deps (their name, cache key and whether it was workspaced), now, I think to reconstruct the graph, I'm going to need build + transitive runtimes of the build deps and these will need to be stored in the proto | 10:52 |
| jennis | Is that correct? | 10:53 |
| jennis | If so, i'm unsure whether I should maintain two dependency "messages" (artifact proto speak), one for build and one for runtimes, or just add another member the the Dependency message in the proto | 10:54 |
| *** cs-shadow has joined #buildstream | 10:57 | |
| *** phildawson_ has quit IRC | 10:57 | |
| *** phildawson_ has joined #buildstream | 10:57 | |
| *** lachlan has joined #buildstream | 11:06 | |
| tme5 | am I ok to work on this? https://gitlab.com/BuildStream/buildstream/issues/1078 | 11:13 |
| tme5 | asking because it's a breaking UI change | 11:14 |
| raoul | tlater[m], yeah we should be pulling sources, it's just combined in the same queue as fetching. The logic for it is in `Element._fetch` | 11:22 |
| tlater[m] | Ta raoul | 11:29 |
| tlater[m] | tme5: That would be appreciated | 11:29 |
| tlater[m] | We're preparing to release 2.0, which obviously breaks UI/API | 11:29 |
| tlater[m] | We got some things wrong, 2.0 is the big "let's fix and get everything consistent" release | 11:30 |
| adds68 | can i have a review of: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/129/diffs | 11:33 |
| adds68 | thanks | 11:33 |
| *** lantw44 has quit IRC | 11:41 | |
| tme5 | this click parameter naming is so annoying.. | 11:42 |
| tme5 | you can give different names nicely for options but not arguments | 11:42 |
| *** lantw44 has joined #buildstream | 11:47 | |
| *** phildawson_ has quit IRC | 11:58 | |
| *** lachlan has quit IRC | 11:59 | |
| *** lachlan has joined #buildstream | 12:14 | |
| *** phildawson_ has joined #buildstream | 12:31 | |
| traveltissues | juergbi, i only had a few comments on your mr | 12:56 |
| juergbi | thanks for the review | 12:57 |
| tlater[m] | juergbi: So, after that patch, expiry will happen entirely on the casd side? | 13:07 |
| juergbi | yes | 13:07 |
| tlater[m] | How does it keep track of what is being expired, is it still mtimes? | 13:07 |
| juergbi | yes but now on a per blob basis | 13:08 |
| juergbi | i.e., like expiry already was in bst-artifact-server, but different from what it was on the client side | 13:08 |
| tlater[m] | Perfect, that means my issues with server expiry are entirely resolved with this :) | 13:08 |
| tlater[m] | Wait, bst-artifact-server did use artifact-wise mtimes last I checked | 13:09 |
| juergbi | sounds good but, ooi, what issues do you mean? | 13:09 |
| juergbi | initially, yes, but valentind replaced it with blob-based expiry a while ago | 13:09 |
| tlater[m] | juergbi: As part of splitting artifact proto/blob servers I needed to get rid of code that updated artifact file mtimes | 13:09 |
| tlater[m] | I assumed that was for expiry | 13:09 |
| tlater[m] | Is that just leftover deadcode then? | 13:09 |
| juergbi | might have been | 13:10 |
| juergbi | if you mean the artifact proto files themselves | 13:10 |
| tlater[m] | Specifically: https://gitlab.com/BuildStream/buildstream/merge_requests/1540/diffs#d10b1e972922f297b08c3c672d861e73ef4e75f1_445_459 | 13:10 |
| tlater[m] | (Yeah, horrible diff, I need to actually remove that code instead of commenting it out) | 13:11 |
| tme5 | what's the difference between PipelineSelection and Scope? | 13:12 |
| juergbi | tlater[m]: that's actually still relevant, and I understand why it's an issue for you | 13:15 |
| tlater[m] | juergbi: So I take it that's what's keeping track of what blobs where modified when? | 13:16 |
| tlater[m] | Instead of (what I assumed to be) something that sets mtimes on protos? | 13:16 |
| tlater[m] | In which case the server will need to know whether it's running in artifact or cas mode... | 13:17 |
| juergbi | yes, this updates timestamps on artifact files when clients ask for an artifact | 13:17 |
| juergbi | either bst-artifact-server has to have a connection to the external cas server and call FindMissingBlobs on these | 13:17 |
| juergbi | or maybe it would be better to handle this client site | 13:18 |
| tlater[m] | Hmm, do we need to do that? | 13:18 |
| tlater[m] | Surely the cas server can itself decide when to expire files | 13:18 |
| juergbi | well, there is one issue | 13:18 |
| tlater[m] | And we simply assume that files that aren't used are not part of artifacts often accessed? | 13:18 |
| juergbi | the client right now only asks for blobs that it doesn't already have in the local cache | 13:19 |
| juergbi | when these blobs are downloaded, the server indeed updates the file timestamps | 13:19 |
| tlater[m] | Yeah, but is it really a problem if we expire blobs that aren't being used? We can treat artifacts with missing blobs as "missing" on the server side. | 13:19 |
| juergbi | however, the blobs that were already in the local cache would never get the timestamp updated (if all clients already have them) | 13:19 |
| tlater[m] | And if all clients already have them, that means that they probably also have the rest of the artifact | 13:20 |
| juergbi | difficult to say whether it would be an issue in practice | 13:20 |
| juergbi | we anyway need to deal with partial artifacts, so it definitely shouldn't cause a fundamental issue | 13:21 |
| juergbi | however, it might lead to some blobs being expired earlier than they should be | 13:21 |
| juergbi | probably a non-issue for environments with sufficient number of clients | 13:22 |
| tlater[m] | IMO this shouldn't be that big of a problem, considering this is a cache and we're already expecting artifacts to randomly disappear anyway. | 13:22 |
| juergbi | but there might be cases where a server is used by very few clients and in that case it will not be optimal | 13:22 |
| tlater[m] | Hm, well, I suppose it's worth a larger discussion. I don't like the idea of making the servers talk to each other, and it might be a bit fiddly to do on the client end. | 13:23 |
| tlater[m] | Perhaps I should make a ML post about it. | 13:23 |
| juergbi | an option could be to leave it in place for combined artifact/cas server but don't worry about it if the servers are separate until a use case arises where it's a problem | 13:24 |
| tlater[m] | Yeah, that was the approach I intended to take | 13:24 |
| juergbi | I'd assume we anyway want a flag for bst-artifact-server to disable the CAS part | 13:24 |
| tlater[m] | ooi, juergbi, is there something in the protocol that allows me to say "please realize that I asked for these artifacts"? | 13:24 |
| juergbi | do you mean blobs? | 13:25 |
| tlater[m] | Yeah, sorry | 13:25 |
| tlater[m] | Because the alternative is downloading full artifacts and piping them to /dev/null | 13:25 |
| juergbi | FindMissingBlobs should be sufficient for this, although it's not its real use case | 13:25 |
| tlater[m] | Ok, maybe not that bad to try on the client end then. | 13:26 |
| juergbi | we certainly don't want that | 13:26 |
| *** phildawson_ has quit IRC | 13:26 | |
| tlater[m] | Anyway, I need to run, -> dentist | 13:26 |
| *** phildawson_ has joined #buildstream | 13:26 | |
| tlater[m] | o/ | 13:26 |
| juergbi | good luck | 13:26 |
| jennis | good luck tlater[m] | 13:26 |
| *** coldtom has quit IRC | 13:27 | |
| *** coldtom has joined #buildstream | 13:27 | |
| tme5 | what's the difference between PipelineSelection and Scope enums? they seem to have almost identical meanings, just PipelineSelection has more choices | 13:33 |
| tme5 | ahh, got it: PipelineSelection is higher-level and uses Scope in Pipeline.get_selection | 13:38 |
| *** dftxbs3e has joined #buildstream | 13:40 | |
| *** dftxbs3e has joined #buildstream | 13:40 | |
| *** dftxbs3e has joined #buildstream | 13:40 | |
| *** tpollard has quit IRC | 13:41 | |
| *** dftxbs3e has joined #buildstream | 13:41 | |
| *** dftxbs3e has quit IRC | 13:45 | |
| *** tpollard has joined #buildstream | 13:46 | |
| *** toscalix has joined #buildstream | 13:58 | |
| *** phildawson has joined #buildstream | 14:46 | |
| *** phildawson_ has quit IRC | 14:46 | |
| *** lachlan has quit IRC | 15:16 | |
| *** lachlan has joined #buildstream | 15:21 | |
| *** lachlan has quit IRC | 15:29 | |
| *** lachlan has joined #buildstream | 15:30 | |
| *** tpollard has quit IRC | 15:36 | |
| *** tpollard has joined #buildstream | 15:37 | |
| ironfoot | Is there a way to figure out what element is creating a given file? | 15:41 |
| *** ChanServ sets mode: +o tristan | 15:47 | |
| tristan | not a nice one no, that might be interesting to bring to the list (I think the `bst artifact show` thread is on topic of late) | 15:47 |
| tristan | ironfoot, I have had that question before, and I use a rotten trick of `find extract_directory -name "file path"` | 15:49 |
| tpollard | do we still have the extract directory though? | 15:50 |
| tristan | I know that's going away, I don't know that it's gone yet in master | 15:51 |
| tristan | probably gone once all of the cas / buildbox stuff gets through | 15:51 |
| tristan | buildbox / buildbox-casd | 15:51 |
| juergbi | extract directories have been gone for a while already in master | 15:53 |
| juergbi | we directly stage from CAS to the sandbox directory | 15:54 |
| ironfoot | oh, I might be able to use that trick :) | 15:56 |
| ironfoot | also, I was wondering, is there a way to inject all the metadata (like the elements, and files provided by every element) into the final system? | 15:59 |
| ironfoot | so that you can look inside a buildstream built system, and at least have an idea of what it is, and how it was built | 16:00 |
| ironfoot | where things came from, etc | 16:00 |
| tpollard | bst-external has a manifest plugin | 16:00 |
| tpollard | but I don't think it produces detail to that extent | 16:01 |
| tpollard | it might meet your needs though :) | 16:01 |
| tpollard | https://gitlab.com/BuildStream/bst-external/blob/master/bst_external/elements/collect_manifest.py | 16:01 |
| ironfoot | oh, ace | 16:01 |
| tpollard | same for all integration commands too | 16:02 |
| ironfoot | thanks :D | 16:03 |
| *** toscalix has quit IRC | 16:21 | |
| *** tme5 has quit IRC | 16:22 | |
| *** tristan_ has joined #buildstream | 16:25 | |
| *** lachlan has quit IRC | 16:28 | |
| *** lachlan has joined #buildstream | 16:36 | |
| *** tristan_ has quit IRC | 16:38 | |
| *** lachlan has quit IRC | 16:46 | |
| *** lachlan has joined #buildstream | 16:53 | |
| *** jonathanmaw has quit IRC | 16:57 | |
| *** traveltissues has quit IRC | 17:14 | |
| *** lachlan has quit IRC | 17:17 | |
| *** tristan_ has joined #buildstream | 17:37 | |
| *** tristan_ has quit IRC | 17:38 | |
| *** phildawson has quit IRC | 17:42 | |
| *** phoenix has joined #buildstream | 18:08 | |
| *** tristan_ has joined #buildstream | 18:37 | |
| *** tristan_ has quit IRC | 18:39 | |
| *** tristan_ has joined #buildstream | 19:14 | |
| *** tristan_ has quit IRC | 19:15 | |
| *** tristan_ has joined #buildstream | 19:37 | |
| *** tristan_ has quit IRC | 19:39 | |
| *** tristan_ has joined #buildstream | 20:00 | |
| *** tristan_ has quit IRC | 20:01 | |
| *** tristan_ has joined #buildstream | 20:04 | |
| *** tristan_ has quit IRC | 20:28 | |
| *** phoenix has quit IRC | 21:40 | |
| *** tristan_ has joined #buildstream | 22:07 | |
| *** tristan_ has joined #buildstream | 22:08 | |
| *** narispo has quit IRC | 22:37 | |
| *** narispo has joined #buildstream | 22:37 | |
| *** tristan_ has quit IRC | 22:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!