| *** tristan has quit IRC | 05:07 | |
| *** tristan has joined #buildstream | 05:16 | |
| tristan | Anyone tried pdb with buildstream ? | 08:28 |
|---|---|---|
| * tristan doesnt even know how to invoke, `pdb bst [args...]` exits with: Error: bst does not exist | 08:29 | |
| *** jude has joined #buildstream | 09:06 | |
| *** tlater has joined #buildstream | 09:11 | |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 09:18 |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 09:18 |
| tristan | Aha ! | 09:20 |
| tristan | weird | 09:20 |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 09:20 |
| *** adds68 has joined #buildstream | 09:20 | |
| * tristan is debugging https://gitlab.com/BuildStream/buildstream/issues/137 | 09:20 | |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 09:21 |
| tristan | And finds that... when we run `bst track --except foo.bst bar.bst`... and if the pipeline is rather large.... the place we are spending time after the whole thing completes is... calculating cache keys !!! | 09:21 |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 09:21 |
| tristan | freaky | 09:21 |
| tristan | So, since we loaded everything in a mode where the cache key is intentionally unset until we discover it... I suppose that the base element that is excluded with --except, never gets a cache key | 09:22 |
| tristan | and it causes printing of a highlevel cache key to become very expensive | 09:23 |
| *** adds68_ has joined #buildstream | 09:23 | |
| *** adds68 has quit IRC | 09:24 | |
| tristan | at the same time, I've learned to have `python3-dbg and python3-gi-dbg` installed, and it makes gdb stack traces of: "python3-gdb `which bst` [args...]" quite informative | 09:24 |
| tristan | ahhha ! | 09:28 |
| tristan | Ok I can work around it at *least* - and I think it needs to be done anyway for tlater's work | 09:28 |
| tristan | tlater, you are doing multiple targets right now right ? | 09:28 |
| tlater | tristan: yup | 09:28 |
| tristan | https://gitlab.com/BuildStream/buildstream/issues/129 | 09:29 |
| tristan | Right | 09:29 |
| tlater | I thought it was 131 for some reason but yes | 09:29 |
| tristan | tlater, so there is one thing I intend to handle right away, which I think you need, not sure if you've considered it yet | 09:30 |
| tristan | That is, if there is multiple targets; we no longer have a general element for general printing purposes (instead, we'd have multiple) | 09:30 |
| tlater | tristan: Right, I was about to ask you about that :) | 09:30 |
| tristan | tlater, basically Pipeline.message() needs an element | 09:30 |
| tristan | Ah | 09:30 |
| tlater | What's your idea for that? I couldn't come up with anything and ignored it for the time being. | 09:31 |
| tristan | Ok, well, if I rework that to not use an element, I think it does nothing to damage the logs - it will solve your issue of needing to decide on an ambiguous target element | 09:31 |
| tristan | And it will solve https://gitlab.com/BuildStream/buildstream/issues/137 | 09:32 |
| tlater | That seems *very* handy, especially considering how often it turns up. | 09:32 |
| tristan | by not attempting to calculate a cache key | 09:32 |
| tlater | Wait, how is that related? | 09:32 |
| tristan | tlater, I've updated that bug, it's not a hang - when you track a large pipeline and --except the root... | 09:32 |
| tristan | what happens is it takes a loooooooooooooong time at 100% cpu | 09:33 |
| tristan | trying to impossibly resolve a cache key for the target element | 09:33 |
| tlater | And it does that only because it's trying to show it in the message? | 09:33 |
| tristan | because the cache key tries to get resolved but it never gets cached, it cant because dependencies have been chopped off | 09:33 |
| tristan | exactly ! | 09:34 |
| tlater | Heh, fair enough. Well, it's two birds with one stone I suppose :) | 09:34 |
| tlater | tristan: As an aside, do you know why there is an edge case for RUN dependencies here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_pipeline.py#L79 ? | 09:34 |
| tristan | maybe not the ideal fix, but seeing as we need more generalized messaging at the top level anyway, better do that... | 09:35 |
| tlater | I assume we want to build run dependencies with their parents, but looking at the run dependencies I actually get that doesn't seem to be it. | 09:35 |
| tristan | tlater, I cant say I've ever understood that code perfectly | 09:40 |
| tlater | Oh, who wrote it? | 09:41 |
| tristan | juergbi, wrote that part... | 09:41 |
| tlater | Fair enough :) | 09:41 |
| tristan | the thing is, its a bit recursive I think, and ... if you require a build dependency, you require only it's runtime dependencies, but never build dependencies of build dependencies, in a build plan | 09:41 |
| tristan | that will of course depend if all of your runtime dependencies of build time dependencies are cached, or not. | 09:42 |
| * tlater seems to have misunderstood runtime dependencies | 09:42 | |
| juergbi | yes, you want all runtime dependencies. build dependencies you need only for elements that you actually want to build | 09:43 |
| juergbi | i.e., aren't cached | 09:43 |
| tlater | Ah, I understand. My actual question is why runtime dependencies aren't considered depth + 1 though | 09:44 |
| juergbi | ah, you can build those at the same time | 09:45 |
| juergbi | to build target element foo you don't need runtime dependency bar | 09:45 |
| juergbi | unless it happens to be a build dependency as well | 09:45 |
| tlater | juergbi: Awesome, ta :) | 09:46 |
| *** valentind has quit IRC | 09:50 | |
| tlater | juergbi: Does this mean that the root element is always planned? | 09:52 |
| juergbi | planned, yes | 09:54 |
| tristan | the root being the target ? | 09:54 |
| juergbi | it might not need to be built but it's never kept out of the plan right now, iirc | 09:54 |
| juergbi | yes | 09:54 |
| tristan | Sounds reasonable - but I suppose desirable to exclude it (if only for the purpose of displaying a better session/total count in the logs and status area) | 09:55 |
| tristan | juergbi, tlater is working on making buildstream support a target list in every command instead of a single target | 09:56 |
| tristan | which is the whole context behind this, he just needs to tweak the planner code to handle a list instead of one target | 09:56 |
| *** ssam2 has joined #buildstream | 10:00 | |
| gitlab-br-bot | push on buildstream@options (by Mathieu Bridon): 1 commit (last: doc: Explain how to configure project options) https://gitlab.com/BuildStream/buildstream/commit/2252e78b0d98508fe3e37ad9f8180a00a5bbc803 | 10:11 |
| gitlab-br-bot | buildstream: merge request (options->master: doc: Explain how to configure project options) #130 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/130 | 10:12 |
| *** tiagogomes has joined #buildstream | 10:17 | |
| tlater | \o/ I think I just re-implemented the planner using Kahn's algorithm, which manages multiple top level nodes correctly :) | 10:25 |
| * tlater is off to meeting | 10:25 | |
| *** cs_shadow has joined #buildstream | 10:38 | |
| *** semanticdesign has joined #buildstream | 10:54 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 4 commits (last: tests/frontend/track.py: Added tests for recursive tracking and --except) https://gitlab.com/BuildStream/buildstream/commit/e472f5abd7f9aeff994a14e581c7a89199bd255f | 11:13 |
| gitlab-br-bot | buildstream: issue #137 ("buildstream hangs in operations with --except") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/137 | 11:13 |
| ssam2 | this wacky FUSE error happened: https://gitlab.com/baserock/definitions/-/jobs/37944721 | 11:15 |
| ssam2 | luckily I can reproduce it locally, so i will dig into wtf is going on... | 11:16 |
| ssam2 | i guess something has generated files with garbage names somehow | 11:16 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: utils.list_relative_paths: avoid os.path.relpath) https://gitlab.com/BuildStream/buildstream/commit/4bc656c598d5eaa47bfe52c4956311a0ef64aacc | 11:18 |
| ssam2 | interesting! this is the failed build log... https://pastebin.com/E9V5xjDs | 11:18 |
| ssam2 | ldconfig is outputting line noise | 11:18 |
| ssam2 | which triggers a backtrace in buildstream as it expects logs to be valid UTF-8 | 11:18 |
| gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 11:19 |
| ssam2 | oh, the real problem is that i'm doing `/bin/sh ./tools/sbin/ldconfig` on a cross-built binary, which causes sh to try and execute it as a shell script | 11:25 |
| ssam2 | but i guess there's a question on buildstream assuming the world is valid UTF-8; probably worth adding some testcases for that | 11:26 |
| tristan | We should not make that assumption | 11:31 |
| tristan | We cannot know really what the output is gonna be | 11:32 |
| tristan | So that's worth being a bug | 11:32 |
| tristan | Also, we dont have very good coverage of the fuse layer, we should add some integration test to exercise all the fs operations | 11:32 |
| tristan | I was thinking, we should probably stress test that, and consider enabling the multithreads option on it | 11:33 |
| tristan | I *think* the way we currently have it setup is to synchronize all fs operations through one single thread | 11:34 |
| tristan | (this doesnt matter *much*, because we intentionally dont use it in /buildstream/build, where we know we havent staged any hardlinks anyway) | 11:34 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: context.py: Fixed project override validation to allow new 'strict' keyword) https://gitlab.com/BuildStream/buildstream/commit/459cc8a2c16c1485cf891485caa61fec51e91b23 | 11:57 |
| tristan | ssam2, you have push access to the artifact share correct ? | 12:00 |
| tristan | ssam2, Can you try a handshake with: ssh -l artifacts -p 10007 gnome7.codethink.co.uk -oConnectTimeout=3 bst-artifact-receive artifacts | 12:00 |
| tristan | ? | 12:00 |
| tristan | I've been refused pushes since friday, I think it's not fail2ban | 12:01 |
| ssam2 | I get connection refused | 12:01 |
| tristan | crap | 12:01 |
| tristan | me too, ok -> broken | 12:01 |
| tristan | none of the local config has changed, must be something out of our control that's blocking us | 12:02 |
| ssam2 | yeah | 12:02 |
| ssam2 | hmm, i can't ssh to gnome 7 from the jumpserver either | 12:02 |
| ssam2 | perhaps the node has crashed ? | 12:02 |
| tristan | I'm on it | 12:02 |
| tristan | ssam2, I mean, I'm logged in to it | 12:03 |
| ssam2 | ah, right | 12:03 |
| tristan | not like "I'm on it", as if I knew how to fix such a network related kind of thing | 12:03 |
| tristan | heh | 12:03 |
| ssam2 | oh yeah, i was logging in wrong | 12:03 |
| ssam2 | right, yeah it must be the port forwarding broken | 12:04 |
| tlater | tristan: Would you rather have the planner or the loader sort out which elements in the pipeline are the same? Currently I have two metaelements that don't at all link together, which obviously won't work... o\ | 12:09 |
| tlater | -> They depend on metaelements which in turn are independent, so we create multiple metaelements from the same file | 12:12 |
| * tlater thinks this is a loader issue, actually. | 12:12 | |
| *** bochecha has joined #buildstream | 12:12 | |
| tristan | ssam2, Dave says he'll look into it after a meeting he is in right now... | 12:14 |
| tristan | tlater, neither | 12:14 |
| tristan | tlater, rewind to our conversation last week: It must be the loader which accepts multiple filename targets, and returns a list of metaelements in the same order | 12:15 |
| tristan | * tlater thinks this is a loader issue, actually. | 12:15 |
| tristan | That ^^^ ++ | 12:15 |
| tristan | exactly | 12:15 |
| tlater | tristan: So, it should accept multiple filename targets and return a list of metaelements in the same order. The dependencies of these metaelements must overlap, rather than just blindly creating two trees? | 12:16 |
| bochecha | hi, I have a module to build that only publishes its releases as... zip | 12:16 |
| bochecha | is that a source plugin which would be acceptable in buildstream? | 12:16 |
| *** givascu has joined #buildstream | 12:19 | |
| *** jude has quit IRC | 13:02 | |
| *** jude has joined #buildstream | 13:02 | |
| *** jude has quit IRC | 13:05 | |
| *** jude has joined #buildstream | 13:09 | |
| *** jude has quit IRC | 13:11 | |
| tristan | tlater, I dont know what you mean by "must overlap" | 13:16 |
| tristan | I.e. it's totally possible that you build two targets which dont overlap at all, there's nothing wrong with that | 13:16 |
| tlater | tristan: I want to create one graph, instead of as many trees as there are targets | 13:16 |
| tristan | bochecha, Naaaah I need to create one :-S | 13:17 |
| tlater | tristan: Ah, right, that scenario is fine. But if the graph contains dependencies that are common to some targets, they should be in the same graph. | 13:17 |
| bochecha | tristan: not sure I understand | 13:17 |
| tristan | bochecha, I think we need a "download a file" kind of source, maybe we can support some features like auto-unzip if it's a zip file | 13:18 |
| tristan | bochecha, you mean you want a source which supports zip right ? | 13:18 |
| bochecha | basically I want the tar source, for zips :) | 13:18 |
| tristan | bochecha, that happens, I mean, we could treat it very similarly to how we do tarballs | 13:18 |
| bochecha | the release tarballs are zips for this project | 13:18 |
| tristan | exactly, and no... we dont have one | 13:18 |
| tristan | We need one, that's what I mean | 13:18 |
| bochecha | Iknow there isn't one, I found out when BuildStream refused my zip file :P | 13:19 |
| tristan | bochecha, I think maybe we want something even more abstract, though | 13:19 |
| bochecha | I was asking if it was desired, before I start working on it | 13:19 |
| tristan | I mean, just download a file and checksum it | 13:19 |
| tristan | bochecha, how would you feel about depending on having a program in your runtime that unzips ? | 13:19 |
| bochecha | why would a program be needed? | 13:19 |
| tristan | Well, it's an easy feature to have, I guess no reason to omit it from a Source implementation | 13:20 |
| tristan | bochecha, so we could have one Source that deals with "whatever else you wanna download" | 13:20 |
| bochecha | Python has a zipfile module, with a very similar API to its tarfile module which BuildStream already uses | 13:20 |
| tristan | Today it's zip | 13:20 |
| tristan | Tomorrow, it's I dont know what | 13:20 |
| tlater | rar! | 13:20 |
| bochecha | tlater: I.... I'm going to leave now :P | 13:20 |
| tristan | :) | 13:21 |
| tristan | Well, I dont mind a zip plugin, I think we discussed this before specifically about weird tracking options for tarballs | 13:21 |
| bochecha | zip is unfortunately quite common for Python modules handled by distutils/setuptools (IIRC it's even the default format, or it used to be, I forgot) | 13:21 |
| tristan | bochecha, I guess it would be nice if we can internally code share between tar and zip plugins | 13:21 |
| bochecha | I wonder if it shouldn't just be a "tarball" source, which automatically handles .zip, .tar, .tar.gz, ... | 13:22 |
| bochecha | after all, the "tar" source already handles .tar, .tar.gz, .tar.xz, ... | 13:22 |
| bochecha | even though they really are different formats | 13:22 |
| tlater | That would be an "archive" source then, I suppose? | 13:22 |
| bochecha | yeah, I couldn't find a better name than "tarball", that's the obvious one I was missing :) | 13:23 |
| tlater | bochecha: I guess what tristan is saying, it would be really neat if that one eventually has a way to manage other formats besides .zip and .tar.* later down the line. | 13:23 |
| bochecha | that's what flatpak-builder does | 13:23 |
| * tristan has to run... time to get a pizza before it closes | 13:25 | |
| tristan | um um ... so no that's not *exactly* what I was saying | 13:26 |
| tristan | What I was saying... is that one day you need to download a .zip... another day, you might need to download a .ttf font somewhere | 13:26 |
| tristan | Or an anything | 13:26 |
| bochecha | tristan: I have to run to, enjoy your dinner :) | 13:26 |
| tristan | I think it's close to what I was saying | 13:26 |
| tristan | What I mean is, if we're gonna put any effort, we should consider that it can be an "anything" | 13:27 |
| tlater | tristan: Add to that that unpacking is optional? | 13:27 |
| tristan | tlater, or not even add to that ... just that | 13:27 |
| tristan | I.e., I would add to that some other thoughts, if I had space to think about it more :D | 13:27 |
| *** kailueke[m] has joined #buildstream | 13:28 | |
| *** ssam2 has quit IRC | 13:30 | |
| *** ssam2 has joined #buildstream | 13:31 | |
| *** tristan has quit IRC | 13:34 | |
| *** jude has joined #buildstream | 13:35 | |
| kailueke[m] | Hey, what do you think about adding options to bst shell for disabling bubblewrap isolation? Using the D-Bus session of the host makes sense as long as running in a VM is not ready (otherwise certain programs won't start) | 13:42 |
| kailueke[m] | bind mounting /run/user/ID seems to be needed, maybe more | 13:44 |
| *** tristan has joined #buildstream | 14:03 | |
| *** bochecha has quit IRC | 14:06 | |
| *** jude has quit IRC | 14:13 | |
| *** jude has joined #buildstream | 14:14 | |
| kailueke[m] | ah, --uid $(id -u) --gid $(id -g) might be enough | 14:16 |
| tristan | kailueke[m], I think two things... for `bst shell --build` we want to maintain total isolation | 14:28 |
| tristan | kailueke[m], and I think for regular relaxed testing purpose we probably want cooperation from the project.conf | 14:28 |
| tristan | But yes, more relaxed at the bwrap level as well would be nice | 14:28 |
| kailueke[m] | ok, that's why I am asking. anyway it would have needed the content of /run/user and a writeable /run/user/ID | 14:28 |
| tristan | I was thinking maybe an interesting thing would be to let the project declare some commands to run at `bst shell` time | 14:29 |
| tristan | For instance, if you want to muck about with /etc/resolv.conf | 14:29 |
| tristan | Then it would be up to you to know that such a file exists in the system you're developing | 14:29 |
| tristan | There is also some hard coded pass through env vars that need to be gotten rid of, that should go into a project.conf declared "shell profile" or such | 14:30 |
| kailueke[m] | using the host D-Bus session has its downsides with different client library versions | 14:30 |
| *** juergbi has quit IRC | 14:32 | |
| *** jude has quit IRC | 14:38 | |
| *** juergbi has joined #buildstream | 14:38 | |
| *** jude has joined #buildstream | 14:38 | |
| kailueke[m] | the interesting thing with jhbuild is that running system deamons from jhbuild/install/libexec works and it finds the right libraries. developing is maybe a bit easier if the process you run is not isolated. | 14:41 |
| kailueke[m] | anyway, what is roughly the current state with building vm images? | 14:43 |
| ssam2 | the 'x86image' element exists, so support is there in principle | 14:47 |
| ssam2 | but it's not being actively used at present | 14:47 |
| ssam2 | i would like to get a demo of VM image builds going soon though | 14:47 |
| *** jude has quit IRC | 14:47 | |
| kailueke[m] | ok, so just not exposed as bst subcommand | 14:48 |
| ssam2 | right. it probably wouldn't be a subcommand but an element, whose artifact would be a disk image file | 14:50 |
| ssam2 | so you'd do `bst checkout foo-vm-image.bst ./vm`, then run `qemu -hda ./vm/hda.img` or some such thing | 14:51 |
| ssam2 | or the artifact could generate a script for running qemu with the appropriate commandline | 14:51 |
| kailueke[m] | thanks for explanation :) I hope to follow closer, but rightnow bst can't replace my current local-build/jhbuild usage | 14:57 |
| *** semanticdesign has quit IRC | 15:02 | |
| *** semanticdesign has joined #buildstream | 15:03 | |
| *** jude has joined #buildstream | 15:03 | |
| *** bochecha has joined #buildstream | 15:20 | |
| tlater | What should the little header thing with the elements next to it be now that there is no clear 'target' anymore? | 15:23 |
| tlater | ^ tristan, or anyone who wants to chime in I suppose. I can't really think of anything sensible, the project name maybe? | 15:23 |
| tristan | project name sounds sensible | 15:24 |
| tristan | "project (session / total)" instead of "element.bst (session / total)" | 15:24 |
| tristan | that's what you mean right ? | 15:24 |
| tlater | tristan: yup | 15:25 |
| tristan | tlater, for the startup summary, probably better to actually list the targets, though | 15:25 |
| tlater | A bit unfortunate that you can't distinguish between builds of the same project this way, but we don't have enough info to do that anymore. | 15:26 |
| tristan | not sure what you mean | 15:26 |
| tristan | the header status thing is just user feedback | 15:26 |
| tlater | tristan: Well, if you have multiple terminals with running builds you don't really know which is which anymore. | 15:27 |
| tristan | the startup heading goes into the log | 15:27 |
| tristan | myeahhhhh | 15:27 |
| tristan | Do you really do that ? | 15:27 |
| tlater | tristan: Sometimes when testing things, I suspect most users probably won't. | 15:27 |
| tristan | I think right now it's not worth bothering about | 15:28 |
| * tlater uses the project name | 15:28 | |
| tristan | we could one day create a unique token or something, and tie that together in the individual build logs or such | 15:28 |
| tristan | but meh, for now I dont see the point unless there's really a need | 15:29 |
| tlater | "TARGETS DO NOT EXIST ANYMORE" was a cute heading. You will be missed little note written into the UI of buildstream. | 15:29 |
| bochecha | tristan, tlater: so, I've got a bit of time right now, do I add a new dedicated `zip` source? an `archive` one which takes over `tar`? something else? :) | 15:32 |
| tristan | bochecha, Ok so - as far as Elements go, we have some abstract classes in the core which we use to simplify implementation of some things | 15:35 |
| tristan | bochecha, like ScriptElement and BuildElement, cant be used directly, but things in the plugins/ dir subclass them | 15:35 |
| tristan | bochecha, how about we create something like DownloadSource, and make tar derive from that | 15:36 |
| tristan | bochecha, that way the tar source only takes care of the staging, but the downloading and tracking of "downloadable files" is taken care of by the base class | 15:36 |
| tristan | And you can have an additional zip, suddenly it's not an explosion of duplicated code | 15:36 |
| tristan | And also, it will be easy to implement one that doesnt have any fancy un-archiving thing either (in the case that you want to download a font file directly from some hosting, or similar) | 15:37 |
| tristan | We can later think about fancy tracking mechanisms in more specialized subclasses ... another time | 15:38 |
| tristan | for now the download-every-track-and-raise-warning-if-checksum-changed seems alright-ish | 15:39 |
| bochecha | tristan: pretty much every source except local ones are "download sources" (e.g ostree), but we probably don't want to factor them all? | 15:40 |
| tristan | To me, what I wanted is A.) Not have to think about this all over again on the day we want to download a .ttf that is not in any kind of archive... B.) Not reimplement stuff we already have working | 15:40 |
| tristan | bochecha, they are not downloadable files, though - maybe another name would be more suitable | 15:40 |
| tlater | tristan: Couldn't a lot of these plugins just be handled with a clever (>)? | 15:41 |
| tristan | i.e. the "norm" is a VCS, which provides features for tracking and fetching etc | 15:41 |
| tristan | tlater, more context | 15:41 |
| tlater | tristan: If we have an arbitrary 'download' command that does no unpacking, there could be some sort of a 'post-download' group of commands that allow you to run whatever on the downloaded file | 15:42 |
| tristan | tlater, it's not time to rearchitecture the whole world, but yeah I see your meaning | 15:42 |
| tlater | s/command/plugin/ | 15:43 |
| tristan | tlater, the truth is, we could rip the guts out of tar source, never make a zip source, and require that the runtime have a tar or unzip program | 15:43 |
| * bochecha just wants to build pytz ^_^ | 15:43 | |
| tristan | but meh, these are convenient anyway | 15:43 |
| tristan | bochecha, right now you can copy paste tar source into your project's local tree and implement a zip source based on that | 15:44 |
| tristan | to upstream it, we should do it nicely, and not duplicate code, though | 15:44 |
| bochecha | yeah, I know... but I usually try to avoid downstream plugin and just get things done upstream when I can :) | 15:45 |
| bochecha | and I don't have any urgency on this thing, so I'd rather take the time to do it directly upstream, rather than work around it downstream (which is pretty much what I've been doing since I've started looking at BuildStream) | 15:45 |
| tristan | I think what I'm proposing is pretty sensible no ? its not time to start thinking of removing tar or any wild things like that | 15:45 |
| tristan | I mean, API is almost sealed | 15:45 |
| bochecha | sure | 15:46 |
| tristan | So seems logical, like 80% of what tar does, is what you'd want in zip | 15:46 |
| bochecha | I can't think of any difference I'd want between the two | 15:46 |
| bochecha | except the archive format | 15:46 |
| tristan | And is also useful for a no-extract-download-a-file thing if we want to create one (which we *definitely will*) | 15:46 |
| tristan | so that 80% goes into -> CommonSourceAbstractClass, tar unpacking and staging goes into one file (that exists) and zip unpacking goes into a new Source plugin | 15:48 |
| tristan | if you dont want to futz about with making a nice common abstract class API, we could keep that abstract class private until a later time, too | 15:49 |
| tristan | _CommonAbstractSourceWithSomeNameThatIsNotAsLongAsThisOne() :) | 15:50 |
| bochecha | I'll try something :) | 15:50 |
| bochecha | you'll tell me what's wrong in the review, and it will get good before it's merged ;) | 15:50 |
| * tlater is watching two specified targets build in parallel :) | 15:53 | |
| tristan | tlater, awesome sauce :) | 15:59 |
| tristan | tlater, I think depth-sort done by the planner is the most sensitive part of this | 16:00 |
| tristan | Also, I think it wouldnt hurt if say; we broke it up into two parts - if that's more readable | 16:00 |
| tristan | i.e. a depth sort, and a filtering process which removes elements that need not be in the plan | 16:01 |
| tristan | but either way, not regressing depth sort is important | 16:01 |
| * tristan has for a long time been hopeful that we can use Element.dependencies(), which are already *pre sorted* by the loader, to help us ditch at least some of the planner code | 16:03 | |
| *** ssam2 has quit IRC | 16:07 | |
| *** ssam2 has joined #buildstream | 16:09 | |
| tlater | tristan: Probably a good idea if you had an initial look through what my solution is once it's roughly "working". I'll hopefully be far enough with that bit today. | 16:15 |
| * tlater will have to write lots of tests, but is pretty confident this works from manual testing. | 16:15 | |
| bochecha | huh | 16:57 |
| bochecha | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugins/sources/tar.py#L176 | 16:57 |
| bochecha | TarInfo objects don't have a `path` attribute... or am I missing something? | 16:57 |
| bochecha | they do, it's just not documented :o | 16:59 |
| *** valentind has joined #buildstream | 16:59 | |
| *** bochecha has quit IRC | 17:32 | |
| *** bochecha has joined #buildstream | 17:33 | |
| *** bochecha has quit IRC | 17:35 | |
| *** bochecha has joined #buildstream | 17:36 | |
| * bochecha yells at tarfile/zipfile for having such a similar API, yet subtly different enough | 17:45 | |
| bochecha | tarfile has name, zipfile has filename | 17:46 |
| bochecha | tarfile has getmembers, zipfile has infolist | 17:46 |
| bochecha | tarfile has add (which recursively adds directories by default), zipfile has write (which doesn't recurse through directories at all) | 17:46 |
| bochecha | looking at the differences between TarSource and ZipSource makes me want to have custom Archive and ArchiveMember classes which just abstract both tarfile/zipfile differences | 17:48 |
| bochecha | and of course: isdir -> is_dir | 17:52 |
| ssam2 | the Zen of Python :-) | 17:53 |
| tlater | Ouch, that one's mean. | 17:55 |
| bochecha | tlater: the worst was not adding directories recursively | 17:55 |
| bochecha | I copied the tar implementation and tests, adapted things, and all tests were passing except 2, due to this behaviour difference | 17:56 |
| bochecha | took me ages to figure out >_< | 17:56 |
| tlater | Can we change the project name of gnome-modulesets to 'Gnome'? Lower case looks like a typo now that it's an actual heading :/ | 17:58 |
| ssam2 | and the same for every BuildStream project ? that would mean filenames like ./.cache/buildstream/logs/Baserock/... | 17:59 |
| ssam2 | which is ok I guess, but i kind of instinctively dislike case in filenames | 17:59 |
| ssam2 | uppercase, i mean | 17:59 |
| bochecha | and how does zipfile determine whether a member is a directory? easy! it ends with a / | 18:00 |
| tlater | Umm... | 18:00 |
| *** ssam2 has quit IRC | 18:00 | |
| bochecha | which caused all sorts of fun when buildstream was passing it members with empty file names | 18:00 |
| tlater | Well, I it's simple at least. | 18:00 |
| bochecha | because ""[-1] doesn't do well | 18:01 |
| tlater | s/I /I guess / | 18:01 |
| bochecha | heh, yeah, dirs in tarfile don't have a trailing / | 18:05 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 20 commits (last: _pipeline.py: Make fetching of remote refs optional) https://gitlab.com/BuildStream/buildstream/commit/5c2ef6d076921bc0121e61efaa7a719c34ea1912 | 18:05 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:05 |
| bochecha | but in zipfile they do | 18:05 |
| bochecha | so here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugins/sources/tar.py#L176-177 | 18:05 |
| bochecha | with tarfile, the member whose path is base_dir doesn't have the trailing /, and therefore member.path.startswith(base_dir) is False | 18:06 |
| bochecha | with zipfile, it's true, and I was setting the path of a member to the empty string | 18:06 |
| bochecha | joy :) | 18:06 |
| gitlab-br-bot | push on buildstream@zip (by Mathieu Bridon): 2 commits (last: tar: Move most of the code to a new base class) https://gitlab.com/BuildStream/buildstream/commit/c850a1ac6387903f2e307bb4f728905f57f3d99c | 18:08 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 5 commits (last: WIP: Add support for multiple arguments (*very* WIP)) https://gitlab.com/BuildStream/buildstream/commit/4346148f33a20d206620f5408b4e00f0818e6ad5 | 18:09 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:09 |
| gitlab-br-bot | buildstream: merge request (zip->master: Add a new zip source) #131 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/131 | 18:10 |
| bochecha | anyway, here's the merge request :) | 18:10 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 5 commits (last: WIP: Add support for multiple arguments (*very* WIP)) https://gitlab.com/BuildStream/buildstream/commit/3b5b0a10cfb22793c7030eb33424196009f95397 | 18:12 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:12 |
| *** tlater has quit IRC | 18:22 | |
| tristan | bochecha, oh crap - I'm so sorry you fell into that trap too :-S | 18:34 |
| tristan | I just took a look btw, the code looks promising | 18:34 |
| tristan | bochecha, I need to remove tests/sources completely | 18:34 |
| tristan | that old stuff is still lying around because the existing stuff covers a liiiitle bit that I still have to factor into the new, better way of testing | 18:35 |
| tristan | I'm not adding any more anything to tests/sources | 18:35 |
| tristan | bochecha, what you need to do, is add a new repo implementation (test scaffolding) for zip source in tests/testutils/repo, and update tests/testutils/repo/__init__.py so that it's a part of ALL_REPO_KINDS | 18:37 |
| tristan | that action alone will cause your new source to be tested under all of the frontend tests | 18:38 |
| bochecha | tristan: ah, ok | 19:33 |
| bochecha | I wrote that in the pull request, so I don't forget :) | 19:35 |
| tristan | bochecha, I'll take a look at the code tomorrow (very, very late here) | 19:46 |
| * tristan will make sure to put up a big DONT_ADD_TESTS_README file in that directory, I think this is the second time | 19:47 | |
| * tristan is sorry to leave that around - it really is a trap that I should clean up | 19:48 | |
| bochecha | tristan: heh, starting wednesday I'll be in a timezone similar to yours :) | 19:51 |
| tristan | nice ! | 19:51 |
| bochecha | so I won't be bugging you at awfully unreasonable times ;) | 19:52 |
| *** bochecha has quit IRC | 20:13 | |
| *** bochecha has joined #buildstream | 20:19 | |
| *** bochecha has quit IRC | 20:33 | |
| *** bochecha has joined #buildstream | 21:03 | |
| *** givascu has quit IRC | 21:43 | |
| *** bochecha has quit IRC | 21:52 | |
| *** valentind has quit IRC | 23:35 | |
| *** semanticdesign has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!