| *** bochecha_ has quit IRC | 00:08 | |
| *** bochecha_ has joined #buildstream | 00:08 | |
| *** bochecha_ has quit IRC | 00:12 | |
| *** bochecha_ has joined #buildstream | 00:12 | |
| *** bochecha_ has quit IRC | 00:23 | |
| *** bochecha_ has joined #buildstream | 00:27 | |
| *** bochecha_ has quit IRC | 00:30 | |
| *** cs_shadow has quit IRC | 02:21 | |
| *** Prince781 has joined #buildstream | 04:53 | |
| *** mohan43u has quit IRC | 04:55 | |
| *** slaf has quit IRC | 04:55 | |
| *** pro[m] has quit IRC | 04:55 | |
| *** connorshea[m] has quit IRC | 04:55 | |
| *** oknf[m] has quit IRC | 04:55 | |
| *** segfault3[m] has quit IRC | 04:55 | |
| *** alatiera has quit IRC | 04:55 | |
| *** awacheux[m] has quit IRC | 04:55 | |
| *** rafaelff[m] has quit IRC | 04:55 | |
| *** asingh_[m] has quit IRC | 04:55 | |
| *** theawless[m] has quit IRC | 04:55 | |
| *** Demos[m] has quit IRC | 04:55 | |
| *** jjardon[m] has quit IRC | 04:55 | |
| *** kailueke[m] has quit IRC | 04:55 | |
| *** albfan[m] has quit IRC | 04:55 | |
| *** inigomartinez has quit IRC | 04:55 | |
| *** m_22[m] has quit IRC | 04:55 | |
| *** ptomato[m] has quit IRC | 04:55 | |
| *** waltervargas[m] has quit IRC | 04:55 | |
| *** mattiasb has quit IRC | 04:55 | |
| *** ssssam[m] has quit IRC | 04:55 | |
| *** abderrahim[m] has quit IRC | 04:55 | |
| *** cgmcintyre[m] has quit IRC | 04:55 | |
| *** ernestask[m] has quit IRC | 04:55 | |
| *** csoriano has quit IRC | 04:55 | |
| *** csoriano has joined #buildstream | 04:56 | |
| *** mohan43u has joined #buildstream | 04:57 | |
| *** Demos[m] has joined #buildstream | 04:57 | |
| *** slaf has joined #buildstream | 04:57 | |
| *** pro[m] has joined #buildstream | 04:57 | |
| *** connorshea[m] has joined #buildstream | 04:57 | |
| *** oknf[m] has joined #buildstream | 04:57 | |
| *** segfault3[m] has joined #buildstream | 04:57 | |
| *** alatiera has joined #buildstream | 04:57 | |
| *** awacheux[m] has joined #buildstream | 04:57 | |
| *** rafaelff[m] has joined #buildstream | 04:57 | |
| *** asingh_[m] has joined #buildstream | 04:57 | |
| *** theawless[m] has joined #buildstream | 04:57 | |
| *** jjardon[m] has joined #buildstream | 04:57 | |
| *** kailueke[m] has joined #buildstream | 04:57 | |
| *** albfan[m] has joined #buildstream | 04:57 | |
| *** inigomartinez has joined #buildstream | 04:57 | |
| *** m_22[m] has joined #buildstream | 04:57 | |
| *** ptomato[m] has joined #buildstream | 04:57 | |
| *** waltervargas[m] has joined #buildstream | 04:57 | |
| *** mattiasb has joined #buildstream | 04:57 | |
| *** ssssam[m] has joined #buildstream | 04:57 | |
| *** abderrahim[m] has joined #buildstream | 04:57 | |
| *** cgmcintyre[m] has joined #buildstream | 04:57 | |
| *** ernestask[m] has joined #buildstream | 04:57 | |
| *** csoriano has quit IRC | 04:57 | |
| *** csoriano has joined #buildstream | 04:57 | |
| *** Prince781 has quit IRC | 06:17 | |
| *** ernestask has joined #buildstream | 06:48 | |
| *** Prince781 has joined #buildstream | 07:20 | |
| *** bochecha_ has joined #buildstream | 07:33 | |
| *** toscalix has joined #buildstream | 07:55 | |
| finn | The email was waiting for approval by a mod on the bst mailing list | 07:56 |
|---|---|---|
| finn | I don't like BREWS, reminds me too much of a bruise. | 08:08 |
| finn | (Buildstream Remote Execution and Worker Service) | 08:08 |
| ltu | who's a moderator on that list apart from tristan? jjardon? | 08:17 |
| *** jonathanmaw has joined #buildstream | 08:18 | |
| ltu | we should get that approved asap | 08:18 |
| *** Prince781 has quit IRC | 08:19 | |
| *** Prince781 has joined #buildstream | 08:21 | |
| juergbi | finn: are you subscribed to the list? | 08:40 |
| juergbi | if you are, it should normally get through immediately | 08:41 |
| juergbi | otherwise, cancel the pending mail, subscribe now, and resend it | 08:41 |
| finn | I should be | 08:41 |
| finn | Actually, I wonder if it's on the @codethink.com alias rather than .co.uk | 08:41 |
| juergbi | yes, the address must match | 08:41 |
| juergbi | the pending approval mail should state the reason | 08:42 |
| finn | "Post by non-member to a members-only list | 08:42 |
| finn | " | 08:42 |
| juergbi | yes, so you sent it from a mismatching address | 08:45 |
| *** solid_black has joined #buildstream | 08:46 | |
| *** solid_black has quit IRC | 08:48 | |
| toscalix | finn: that is expected when redirecting mails from a different mailing list | 08:49 |
| toscalix | I shouldn't have sent it that way in first place | 08:50 |
| *** bethw has joined #buildstream | 08:58 | |
| *** tiago has joined #buildstream | 09:03 | |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 09:13 |
| *** dominic has joined #buildstream | 09:24 | |
| *** aday has joined #buildstream | 09:25 | |
| finn | I've applied for my .co.uk alias on the mailing list, just waiting for approval either tristain or jjardon | 09:29 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 09:53 |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 10:19 |
| *** jonyvx has joined #buildstream | 10:49 | |
| *** jonyvx has left #buildstream | 10:49 | |
| jmac | I've updated !445 (above) to move the virtual directory things into 'storage/' instead of 'sandbox/'. I'm now getting intermittent pylint errors about importing. I've been tearing my hair out trying to figure out what's going on there - can anyone else take a look? | 11:20 |
| toscalix | jmac: is iot possible to add to the !445 the issue it is related to? | 11:23 |
| jmac | Certainly, apologies for not doing that before | 11:23 |
| toscalix | no problem | 11:23 |
| toscalix | the description is clear, thanks | 11:24 |
| jmac | Actually, it is - https://gitlab.com/BuildStream/buildstream/issues/394 is the issue and that shows !445 as linked. But the merge request doesn't show it in the other direction. | 11:24 |
| toscalix | sadly gitlab has severe limitations related with the issue tracker | 11:25 |
| jmac | We should list the issue number in the commit messages we use, which I will try to remember to do when I squash this | 11:26 |
| toscalix | I included it in the policy draft already | 11:26 |
| toscalix | hummm, let me improve it so it clearer | 11:28 |
| jmac | It's mentioned in HACKING.rst | 11:29 |
| toscalix | It is considered in the merge request template, which I hope to implement later today | 11:31 |
| *** Prince781 has quit IRC | 11:50 | |
| *** aday has quit IRC | 12:21 | |
| gitlab-br-bot | buildstream: issue #403 ("Stripping of whitespace from tracking refs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/403 | 12:36 |
| *** tiagogomes has joined #buildstream | 13:04 | |
| *** tiago has quit IRC | 13:05 | |
| *** aday has joined #buildstream | 13:41 | |
| toscalix | valentindavid: it seems https://gitlab.com/BuildStream/buildstream/issues/387 will be readu for testing tomorrow and it might fix https://gitlab.com/BuildStream/buildstream/issues/276 | 13:43 |
| toscalix | which was the main task assigned yesterday to you | 13:43 |
| toscalix | so the recommendation is that you document your findings on #276 | 13:44 |
| toscalix | and stop working on it | 13:44 |
| toscalix | so you can focus on #331 if a I recall correctly | 13:45 |
| toscalix | unless you have a different plan | 13:45 |
| *** tiagogomes has quit IRC | 13:54 | |
| jennis | jmac, i've got the linting passing locally | 13:59 |
| jennis | Did you manage to sort it out? | 13:59 |
| jmac | jennis: Can you try it again? It appears to be intermittent? | 13:59 |
| jennis | seems fine | 14:00 |
| *** tiagogomes has joined #buildstream | 14:00 | |
| valentind | toscalix, OK. | 14:00 |
| jennis | jmac, do you have an init file? | 14:00 |
| jmac | jennis: Weird. Thanks for trying. | 14:00 |
| jmac | jennis: Yes, there's an __init__.py in storage, written in the same way as sandbox/__init__.py. | 14:01 |
| jennis | If so, have you actually commited it, because when I cloned your directory, I had to create my own | 14:01 |
| jmac | Oh, might have missed it | 14:01 |
| jennis | So that's why it may be failing in CI | 14:01 |
| jennis | Also, I suspect you'll get a warning regarding the grouping of your imports | 14:02 |
| jennis | Also, I suspect you'll get a warning regarding the grouping of your imports | 14:03 |
| jennis | whoops | 14:03 |
| tlater | Heh, good job spotting that one jennis :) | 14:08 |
| jmac | I'm not sure what you mean by grouping | 14:09 |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 14:10 |
| jennis | So convention is that you group standard library imports, related third part imports and then local application/library imports | 14:11 |
| jmac | That's not the reason for the failure, though | 14:11 |
| jmac | Right, that's a minor issue, not the problem I'm seeing | 14:11 |
| jennis | yes it is, but I'm surprised pylint isn't complaining about it | 14:11 |
| tlater | jmac: Here's the relevant pep https://www.python.org/dev/peps/pep-0008/#imports | 14:12 |
| tlater | jennis: Only recent versions of pylint support warning about it | 14:12 |
| * jennis has that link open already :p | 14:12 | |
| tlater | jmac may be using an older one :) | 14:12 |
| jennis | tlater, I remember getting this warnings before though, and I'm not getting them now | 14:12 |
| * tlater isn't sure why he doesn't get them either | 14:13 | |
| tlater | I thought I was just using the debian version of pytest | 14:13 |
| tlater | Which is far too old for this | 14:13 |
| ltu | jonathanmaw, did you have an open issue to improve the documentation for the filter element? | 14:24 |
| ltu | I'm struggling to find it (and starting to appreciate how useful labels gitlab will be!) | 14:25 |
| ltu | gitlab labels* | 14:25 |
| jonathanmaw | ltu: there was https://gitlab.com/BuildStream/buildstream/issues/278 | 14:25 |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 14:25 |
| *** aday has quit IRC | 14:27 | |
| *** jennis has left #buildstream | 14:29 | |
| *** jennis has joined #buildstream | 14:29 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 14:31 |
| *** aday has joined #buildstream | 14:43 | |
| ltu | thanks jonathanmaw - i think it may need re-opening? | 14:51 |
| ltu | will comment on the issue | 14:51 |
| jmac | OK, my pylint problems have magically disappeared | 14:56 |
| *** bochecha_ has quit IRC | 15:34 | |
| *** cs_shadow has joined #buildstream | 15:35 | |
| *** Prince781 has joined #buildstream | 15:37 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:39 |
| jennis | juergbi, tristan is concerned about how regularly my code will be calling a _get_dir_size() function, that essentially walks through the cache and determines the size of every file then sums it. Now, with my current implementation, this will only be called if the user sets a cache quota. However, when a quota is set, it will be called every time OSTreeReceiver is run, so, every time bst-artifact-receive is called | 15:41 |
| jennis | Do you know if there will be a more optimal way to do this with CAS? | 15:42 |
| jennis | Rather than Python | 15:42 |
| jennis | Because I'm not sure it's worth trying to pursue circumventing this with OSTree (as opposed to Python) when CAS is about to land, I would like to know your thoughts? | 15:43 |
| jmac | It's certainly possible but with the current implementation, I think asking for the size of a directory will be just as expensive | 15:44 |
| * paulsherwood believes that CAS will be an optional dependency, not guaranteed? | 15:44 | |
| jennis | paulsherwood, the CAS branch landing == no more ostree repo | 15:44 |
| jmac | Is _get_dir_size asking for the total size of the cache, or a subdirectory of it? | 15:44 |
| jennis | jmac, so it's similar, just a walk? | 15:45 |
| jennis | jmac: It calculates the size of the cache | 15:45 |
| paulsherwood | jennis: so bst will have a hard dependency on CAS??? | 15:45 |
| jennis | by walking through all files and directories | 15:45 |
| jennis | paulsherwood not sure what the exact name is, but yes | 15:45 |
| jennis | afaik | 15:45 |
| * paulsherwood is surprised, to say the least | 15:46 | |
| jennis | grpcio is the dependency | 15:46 |
| jennis | dependency | 15:47 |
| jmac | It's a bit tricky with CAS as there is no single root directory, so you would have to walk the underlying files and query all their sizes, AIUI | 15:47 |
| jennis | C&P didn't work | 15:47 |
| jennis | https://pypi.python.org/pypi/grpcio | 15:47 |
| jennis | ahh, so similar idea | 15:47 |
| paulsherwood | does this mean that bst will require to access google infrastructure? | 15:48 |
| paulsherwood | or some equivalent that supports grpc? | 15:48 |
| jmac | No | 15:48 |
| jennis | paulsherwood, I believe this was mentioned on Juergs proposal email for CAS | 15:48 |
| jennis | And everything I know about the hard dependency stems from that :p | 15:49 |
| paulsherwood | :) | 15:49 |
| *** bochecha_ has joined #buildstream | 15:49 | |
| jennis | So an alternative implementation that tlater suggested, was to maintain a file, on the remote machine, which just keeps a record of the cache size | 15:50 |
| jmac | Hopefully juergbi will be able to chip in soon but as far as I know, the google imports are just the protocols; all the implementation is still our work | 15:51 |
| jennis | Which is prone to falling out of sync, however we could implement some sort of cron job that re-invokes the _get_dir_size() fn | 15:51 |
| paulsherwood | ok so juergbi's proposal says optional dependency on grpcio, while retaining ostree as an optional dependency. i assume that means bst will need one, or the other (not possible to use it practically without either)? | 15:51 |
| *** Prince781 has quit IRC | 15:52 | |
| jmac | "This will add grpcio as a hard dependency" | 15:52 |
| * paulsherwood goes to read it again | 15:53 | |
| tlater | paulsherwood: ostree will remain an optional dependency for source plugin reasons. | 15:53 |
| jennis | Paul, I (think I) have forwarded you the email | 15:54 |
| paulsherwood | i have it | 15:54 |
| paulsherwood | i literally misread it | 15:54 |
| juergbi | yes, the plan is to replace the ostree-based artifact cache with the grpcio/CAS-based one | 15:55 |
| jennis | ahh ok | 15:55 |
| juergbi | there is no need for google infrastructure access for this | 15:55 |
| paulsherwood | juergbi: is it 'trivial' to setup a grpcio/CAS-based cache server? | 15:55 |
| juergbi | yes | 15:55 |
| jmac | Easier if anything than setting up an ostree based one | 15:56 |
| *** ernestask has quit IRC | 15:56 | |
| juergbi | client and server side will be part of buildstream | 15:56 |
| paulsherwood | juergbi: even to the extent of being able to run the cache server on (say) a developer laptop? | 15:56 |
| juergbi | yes | 15:56 |
| paulsherwood | (this was a use-case for ybd a couple of times) | 15:57 |
| paulsherwood | cool then. i'll stop worrying :-) | 15:57 |
| paulsherwood | sorry for the noise | 15:57 |
| juergbi | :) | 15:57 |
| paulsherwood | :-) | 15:57 |
| jmac | "Content-Addressable Storage" might sound a lot more complicated than it actually is | 15:58 |
| paulsherwood | heh | 15:59 |
| finn | Quick question, will BuildStream's REAPI/RWAPI implementation be hosted on github or gitlab? | 15:59 |
| paulsherwood | so does 'Keyed Binary Artifact Server' :) | 15:59 |
| jmac | The storage implementation is about 20 lines of Python, plus some autogenerated protocol source | 15:59 |
| juergbi | yes, it's kept very simple | 15:59 |
| * paulsherwood is delighted to hear it | 15:59 | |
| tlater | grpc is also not a cloud interface, but a library that basically helps creating protocols :) | 16:00 |
| * paulsherwood obviously had completely the wrong impression | 16:00 | |
| jmac | As did I before I started on this :) | 16:01 |
| paulsherwood | so much to learn, so little time... | 16:01 |
| juergbi | it's a bit more than 20 but the MR deletes more lines than it adds if we ignore generated code | 16:01 |
| jmac | I hope that didn't sound like I was trivialising your work juergbi | 16:03 |
| jennis | So what do we think about maintaining a file on the server where we can keep track of the cache size? | 16:03 |
| jennis | haha | 16:03 |
| juergbi | ;) | 16:03 |
| jmac | jennis: I think that's the best way to go | 16:03 |
| juergbi | jennis: it's definitely an option | 16:03 |
| * jennis waits for a but | 16:04 | |
| juergbi | there is no atomic compare and replace in POSIX, though, so it might need a lock file or similar, and tristan doesn' | 16:04 |
| juergbi | t like that much | 16:04 |
| juergbi | (in POSIX filesystem API, I mean) | 16:04 |
| tlater | For what it's worth, I'd be looking to implement this on the client side as well if it's acceptable | 16:05 |
| jmac | Just a thought, what would performance be like if we made a whole new filing system on the server, maybe loopback mounted? | 16:05 |
| tlater | Though perhaps as a future optimization | 16:05 |
| jmac | Then we could use the file system to keep track of free space | 16:05 |
| tlater | jmac: That sounds like implementing yet another FUSE mount. | 16:06 |
| juergbi | i.e., FUSE based on a single file instead of a native directory? | 16:06 |
| jmac | jennis: Yes | 16:06 |
| jmac | Er | 16:06 |
| jmac | juergbi: Yes | 16:06 |
| jmac | Not a whole new filing system, that was a bad phrase - I meant a new ext4 fs in a file | 16:07 |
| juergbi | that'd be platform specific | 16:07 |
| jmac | Yes, that's true. | 16:07 |
| tlater | I wonder how python's API for creating filesystem-in-files is | 16:07 |
| juergbi | it would also require root | 16:08 |
| juergbi | or userspace ext4 | 16:08 |
| juergbi | it is an interesting option but there are some caveats, so I'm not considering it for right now | 16:09 |
| jmac | It just popped into my head, probably not workable | 16:09 |
| *** bethw has quit IRC | 16:09 | |
| * tlater likes it, but probably just because he likes filesystem-in-files | 16:10 | |
| paulsherwood | jmac: what's the problem this would be solving? | 16:10 |
| juergbi | if we support both repository size and free disk space based configuration, anyone with root access could setup such a loopback without any special code in buildstream | 16:10 |
| jmac | paulsherwood: It's expensive to calculate how much space is being used by the artifact cache at the moment. | 16:10 |
| paulsherwood | ack | 16:11 |
| juergbi | a 'repository size' file could work pretty well, if we're ok with using a lock file or equivalent | 16:12 |
| juergbi | we might anyway need it for safe purging | 16:12 |
| ltu | finn, that's one to run past sstriker, without a doubt | 16:13 |
| * tlater figures not hosting it on gitlab would be a little odd, seeing as everything else buildstream does is in the buildstream group | 16:13 | |
| juergbi | oh, missed that question. I would go for a new gitlab group | 16:14 |
| ltu | tlater, this isn't going to be part of the BuildStream group | 16:15 |
| tlater | Ah, I missed that. Is there a reason for that? | 16:15 |
| tlater | Just curious :) | 16:15 |
| cs_shadow | Is it just me or does GitLab thinks that you are a bot as well when trying to post something? I don't think I have ever managed to create anything on GitLab without seeing "We detected potential spam in the issue. Please solve the reCAPTCHA to proceed" | 16:16 |
| tlater | cs_shadow: It might not like your name | 16:16 |
| cs_shadow | :( | 16:17 |
| tlater | tbf, your name specifically contains odd characters iirc? | 16:17 |
| tlater | "odd" | 16:17 |
| cs_shadow | nothing too exotic - just a "-" | 16:17 |
| juergbi | I've never seen that message | 16:18 |
| ltu | tlater, BuildStream will be a client of the implementation of BuilfGrid (or whatever name we decide), but the implementation of it should be totally generic | 16:18 |
| * cs_shadow blames our corp proxies, nevermind | 16:18 | |
| * tlater has seen it, probably because he switches devices and location often | 16:19 | |
| jennis | Tristan has suggested cleaning up the remote server based on how much disk space remains on the partition (see the last paragraph of his comment here: https://gitlab.com/BuildStream/buildstream/merge_requests/421#note_72664689) | 16:19 |
| tlater | jennis: That's also an option, but less user friendly | 16:20 |
| tlater | Would probably work best when hosting an artifact cache with docker, since you can then partition the disk size without having to worry about other applications. | 16:20 |
| tlater | But this isn't really acceptable for the local cache, and would be a bit weird for the server cache. | 16:20 |
| toscalix | a new subgroup can be created. later than sooner, the current repos will also be moved to its own subgroup | 16:21 |
| toscalix | as they grow | 16:21 |
| finn | Builfgrid - I like it | 16:22 |
| jennis | tlater, I guess we should ask tristan about the lock file then | 16:22 |
| tlater | jennis: He was pretty opposed to it when I suggested it :) | 16:22 |
| *** toscalix has quit IRC | 16:23 | |
| ltu | finn, :) | 16:23 |
| jmac | The partition's free space approach requires little effort to implement so if we want something that works right now, that's probably the best way | 16:25 |
| tlater | Hm, I think jmac might be right here, actually | 16:26 |
| tlater | We can recommend using docker with appropriate flags in the documentation for this, if multiple applications are a consideration | 16:26 |
| jmac | I think the lockfile is the only reliable and 'correct' way to do it but there's also a lot of room for mistakes in it. | 16:27 |
| tlater | It certainly would drag the branch out further | 16:27 |
| * juergbi has been mentioning this for a long time | 16:29 | |
| tlater | jennis: Think that's enough people voting in favor? | 16:31 |
| jennis | I was going to leave a comment for Tristan on the MR outlining thetwo options | 16:33 |
| tlater | I think the old comment you mentioned earlier is enough to show his opinion :) | 16:33 |
| jennis | perhaps he's reconsidered how he feels about lock files | 16:34 |
| jennis | I will leave the comment and look into implementing the free space approach | 16:35 |
| *** bethw has joined #buildstream | 16:43 | |
| jmac | jennis: Your description of the lockfile approach on the issue wasn't quite what I had in mind | 16:43 |
| jennis | What did you have in mind? | 16:44 |
| jmac | I think the plan there was to record the size of each artifact at the time it was created, updated or deleted | 16:44 |
| jmac | and record the change in size of the whole cache in a file somewhere, which needs to be atomically updated, i.e. locked | 16:45 |
| jennis | yes that is what I meant | 16:45 |
| jmac | So you know the size of the cache by recording everything going in or out, and you'd never need to call get_walk_dir. | 16:45 |
| jennis | I figured we'd need to call _get_dir_size() just once, at the beginning and then manually update when artifacts are pushed/expired from there | 16:45 |
| juergbi | jennis: I forgot to mention, another option tristan and I discussed was to use SQLite to store the repository size, or a somewhat similar storage approach in our own code | 16:45 |
| jmac | Right, in that case we are thinking of the same thing | 16:46 |
| jennis | and then the cron job, just to make sure we're not out of sync | 16:46 |
| jennis | jmac I'll update the comment to make it more clear | 16:46 |
| jmac | jennis: Yes, OK then, we are on the same page. Sorry for the noise. | 16:46 |
| jennis | juergbi, ok thanks, I have no idea what that is, but I'll update the comment | 16:47 |
| jennis | Assuming Tristan is going to be the one who gives an ultimatum? | 16:47 |
| tlater | jennis: sqlite is a nice database that's fairly easy to depend on by software. Databases handle all the issues of concurrent access & co for you | 16:47 |
| tlater | Sqlite could also be used to keep a manifest of the files in the cache | 16:48 |
| jmac | It seems somewhat over the top to use a relational database to store one integer, but yes, it does handle all the atomic stuff | 16:48 |
| tlater | jmac: Fully agree there, but something something on the shoulders of giants ;) | 16:48 |
| juergbi | we have to keep in mind that we might anyway need a lock file for safe purging | 16:50 |
| tlater | jennis: One note about sqlite is that it doesn't require a daemon, which is fairly unique among databases. | 16:50 |
| juergbi | where sqlite doesn't really help | 16:50 |
| tlater | juergbi: It could if it also keeps track of the artifacts. You'd have to just first remove the note that the artifact exists from the database | 16:51 |
| tlater | Oh, safe purging in terms of ostree committing/pruning, right | 16:52 |
| juergbi | the tricky aspect is to not delete a blob that is still needed | 16:52 |
| juergbi | unless we want to completely move away from the filesystem-based approach, sqlite doesn't help here | 16:52 |
| juergbi | I don't think we want to duplicate everything | 16:52 |
| tlater | Yeah, that's unfortunate | 16:52 |
| *** tiagogomes has quit IRC | 16:53 | |
| *** dominic has quit IRC | 16:55 | |
| *** aday has quit IRC | 17:06 | |
| *** aday has joined #buildstream | 17:07 | |
| *** Prince781 has joined #buildstream | 17:09 | |
| *** jonathanmaw has quit IRC | 17:09 | |
| *** bethw has quit IRC | 17:20 | |
| *** Prince781 has quit IRC | 17:27 | |
| *** Prince781 has joined #buildstream | 19:09 | |
| *** aday has quit IRC | 19:33 | |
| *** Prince781 has quit IRC | 20:58 | |
| *** Prince781 has joined #buildstream | 21:19 | |
| *** Prince781 has quit IRC | 21:54 | |
| *** bochecha_ has quit IRC | 22:17 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!