| *** valentind has quit IRC | 01:27 | |
| *** tiagogomes has quit IRC | 01:27 | |
| *** jmac_ has quit IRC | 01:28 | |
| *** tiagogomes has joined #buildstream | 01:28 | |
| *** jennis has quit IRC | 01:28 | |
| *** tiagogomes_ has quit IRC | 01:28 | |
| *** qinusty has quit IRC | 01:28 | |
| *** aidenhjj has quit IRC | 01:28 | |
| *** adds68 has quit IRC | 01:28 | |
| *** coldtom has quit IRC | 01:28 | |
| *** tiagogomes_ has joined #buildstream | 01:29 | |
| *** jennis has joined #buildstream | 01:29 | |
| *** qinusty has joined #buildstream | 01:29 | |
| *** coldtom has joined #buildstream | 01:29 | |
| *** valentind has joined #buildstream | 01:29 | |
| *** adds68 has joined #buildstream | 01:29 | |
| *** aiden has joined #buildstream | 01:30 | |
| *** jmac has joined #buildstream | 01:30 | |
| *** leopi has joined #buildstream | 03:55 | |
| gitlab-br-bot | buildstream: merge request (tristan/source-fetcher-changes-1.2->bst-1.2: Source fetcher changes 1.2) #773 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/773 | 06:23 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (tristan/source-fetcher-changes->master: Source fetcher changes) #772 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/772 | 06:23 |
| *** tristan has joined #buildstream | 06:36 | |
| gitlab-br-bot | buildstream: merge request (tristan/source-fetcher-changes-1.2->bst-1.2: Source fetcher changes 1.2) #773 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/773 | 06:50 |
| *** bochecha has joined #buildstream | 06:59 | |
| *** ChanServ sets mode: +o tristan | 07:00 | |
| gitlab-br-bot | buildstream: merge request (tristan/update-docs->bst-1.2: Updating docs session files before the release) #774 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/774 | 07:06 |
| *** alatiera_ has joined #buildstream | 07:06 | |
| gitlab-br-bot | buildstream: issue #623 ("Stack trace when running out of disk space") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/623 | 07:13 |
| gitlab-br-bot | buildstream: merge request (tristan/update-docs->bst-1.2: Updating docs session files before the release) #774 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/774 | 07:25 |
| *** toscalix has joined #buildstream | 07:39 | |
| *** rdale has joined #buildstream | 07:46 | |
| *** tiagogomes has quit IRC | 07:50 | |
| *** tiagogomes has joined #buildstream | 07:53 | |
| *** tristan changes topic to "BuildStream 1.2.0 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://buildstream.gitlab.io/buildstream | IRC logs: https://irclogs.baserock.org/buildstream | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmap" | 07:53 | |
| *** phildawson has joined #buildstream | 07:59 | |
| *** solid_black has joined #buildstream | 08:12 | |
| *** WSalmon has joined #buildstream | 08:21 | |
| *** tpollard has joined #buildstream | 08:32 | |
| *** jonathanmaw has joined #buildstream | 08:54 | |
| *** alatiera_ has quit IRC | 08:55 | |
| *** rdale has quit IRC | 08:56 | |
| *** rdale has joined #buildstream | 09:02 | |
| *** tiagogomes has quit IRC | 09:09 | |
| *** tiagogomes has joined #buildstream | 09:09 | |
| *** leopi has quit IRC | 09:20 | |
| *** lachlan has joined #buildstream | 09:27 | |
| gitlab-br-bot | buildstream: merge request (tpollard/483->master: plugins/git.py: Warn if ref is not in given track) #564 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/564 | 09:44 |
| cs-shadow | 1.2 \o/ | 09:46 |
| cs-shadow | tristan: shall we publish it to PyPI as well now? | 09:47 |
| tristan | cs-shadow, Yes ! lets see how the PyPI badge looks after the 1.2.0 release | 09:56 |
| cs-shadow | cool, let me upload it now | 09:56 |
| tristan | cs-shadow, I plan to release a 1.3.1 because 1.3.0 never had a tarball... so the 1.3.0 badge is broken... please do NOT upload that to PyPI :) | 09:57 |
| cs-shadow | sure thing | 09:57 |
| *** lachlan has quit IRC | 09:57 | |
| tristan | 1.3.x tarballs will trickle out throughout the cycle, but we need to fix some important stuff in master before we can recommend testing the dev work to those who want to try the experimental stuff | 09:58 |
| tristan | (and as mentioned before, those dev snapshots will never be accessible through PyPI) | 09:58 |
| cs-shadow | we can in theory release them to pypi as dev releases and users won't get them unless they specify --dev (or a similar option, can't remember its name). In any case, for the automation to happen, we need a way of differentiating stable releases vs developmental ones | 09:59 |
| tlater[m] | Hmm... What's the currently recommended buildstream docker image for simply running buildstream? | 10:03 |
| tlater[m] | Still buildstream-fedora? | 10:03 |
| cs-shadow | tlater[m]: yes, but that builds from master at the moment so we need to change that to use stable releases intead | 10:03 |
| tlater[m] | Thanks, master will do fine for me for the moment, but I agree | 10:04 |
| cs-shadow | (the name of the image will also change soon....) | 10:04 |
| toscalix | tiagogomes: , is it possible that, in he top banner, instead of having BuildStream, we have BuildStream, the software integration tool | 10:13 |
| gitlab-br-bot | buildstream: merge request (chandan/pypi-badge->master: README.rst: Add status badges for PyPI release and Python versions) #719 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/719 | 10:14 |
| *** lachlan has joined #buildstream | 10:14 | |
| tristan | tlater[m], Also, I have quota set to "20G" in my buildstream.conf... `du -hs ~/.cache/buildstream/artifacts/cas` says "23G", and I've not seen any cleanup jobs :-/ | 10:14 |
| tlater[m] | tristan: Is that build still ongoing? | 10:15 |
| tristan | Yes | 10:15 |
| tlater[m] | `du` computes a slightly different disk size iirc | 10:15 |
| tiagogomes | It only checks the size after build is completed | 10:15 |
| gitlab-br-bot | buildstream: merge request (chandan/pypi-badge->master: README.rst: Add status badges for PyPI release and Python versions) #719 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/719 | 10:16 |
| tlater[m] | So perhaps it will trigger soon | 10:16 |
| tristan | cache_size.log files just say START: cache_size, SUCCESS: cache_size | 10:16 |
| tristan | perhaps | 10:16 |
| cs-shadow | tristan: 1.2 is on pypi now (https://pypi.org/project/BuildStream/1.2.0/) and the badge seems better now as well: https://gitlab.com/BuildStream/buildstream/merge_requests/719 | 10:16 |
| tlater[m] | tiagogomes: It checks after each element is built, so I would expect it to trigger at some point sooner or later | 10:16 |
| gitlab-br-bot | buildstream: merge request (tpollard/483->master: plugins/git.py: Warn if ref is not in given track) #564 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/564 | 10:16 |
| tlater[m] | It can be a bit fuzzy | 10:16 |
| tiagogomes | Not very useful. https://gitlab.com/BuildStream/buildstream/merge_requests/671 adds some logging :) | 10:17 |
| tristan | Ok here's a fun fact | 10:17 |
| tristan | du -s says 24053760 (is that KB ?) and ~/.cache/buildstream/artifacts/cache_size file (added by... jonathanmaw recently ?) says 16635262095, which is just over 16G | 10:18 |
| tlater[m] | Hm, if disk size jobs are being triggered that might be a bit problematic | 10:20 |
| tlater[m] | Because it would mean that our calculations are just wildly different from those `du` has | 10:20 |
| tlater[m] | I don't think they should be, though. | 10:21 |
| tlater[m] | jonathanmaw: From your proposals I gather that cache_size file should always be up-to-date with how large buildstream thinks it is, is that correct? | 10:22 |
| jonathanmaw | tlater[m]: it should be, yes | 10:22 |
| jonathanmaw | well, ostensibly | 10:23 |
| tristan | jonathanmaw, tlater[m]; see the description text here: https://gitlab.com/BuildStream/buildstream/merge_requests/762 | 10:23 |
| tristan | That might help | 10:24 |
| tristan | Also, I'm not sure if we're at least calculating it at startup time, or assuming that it is correct always if it's there | 10:24 |
| tristan | Also, it can help to ensure that cache size jobs only ever run one at a time | 10:25 |
| jonathanmaw | tristan: when I wrote it, it wouldn't calculate cache size at startup time, and would get reconciled when something caused the artifact cache to recalculate its size | 10:25 |
| jonathanmaw | (i.e. when the size gets close to the quota size) | 10:26 |
| jonathanmaw | iiuc this shouldn't threaten actually running out of disk space, because the artifact cache checks whether the quota size would cause you to run out of disk space before it reaches the quota | 10:28 |
| tlater[m] | That is unless for some reason the file is really wrong and thinks we have a lot less than we do. But that can't realy happen unless the user mucks around and adds stuff to the cache through other means. | 10:31 |
| tlater[m] | tristan: Perhaps you jumped to a different version of buildstream at some point that didn't update the quota file? | 10:31 |
| *** leopi has joined #buildstream | 10:36 | |
| *** jonathanmaw_ has joined #buildstream | 10:44 | |
| *** jonathanmaw has quit IRC | 10:45 | |
| tristan | tlater[m], jonathanmaw_ ok so; at this point the build has been running a bit more `du -hs` now says 26G (so 2 GB were actually added), and the cache size file still says 16635225166, so it's *still* floating at around 16.6 GB | 10:52 |
| *** jonathanmaw_ is now known as jonathanmaw | 10:52 | |
| *** lachlan has quit IRC | 10:53 | |
| tlater[m] | tristan: That's the same value as before, so it looks like the file isn't being updated | 10:53 |
| tlater[m] | tristan: Can you see any cache size jobs running? | 10:53 |
| tristan | tlater[m], is it ? I thought before it was 16635262095 | 10:53 |
| tristan | yeah, it was | 10:54 |
| tristan | so it certainly added something | 10:54 |
| tristan | Is it "adding the size of the newly added artifact" ? or is it "recalculating total size" ? | 10:54 |
| tlater[m] | That should be recalculating from scratch if memory serves | 10:54 |
| tristan | If it is adding a size of a newly added artifact, then certainly the description about necessary locking in !762 above is relevant | 10:55 |
| tristan | Right, that is interesting that the size differs so much; I wonder if it's * 1000 vs * 1024 | 10:55 |
| tlater[m] | Hmm, that wouldn't change the number of bytes | 10:55 |
| tristan | Or, maaaaybe it's not counting directory nodes, which cost 4K each on disk ? | 10:56 |
| tlater[m] | I think that's more likely. I'll compare this on my end, just getting gnome build meta to track right now... | 10:56 |
| tristan | tlater[m], yeah but maybe the number of bytes being so different from the `du -hs` output is a factor of 1024 issue ? | 10:56 |
| *** lachlan has joined #buildstream | 10:57 | |
| tlater[m] | tristan: I guess if du -s is in KB, that's possible | 10:57 |
| tlater[m] | tristan: Could you try `du -hbs`? | 11:00 |
| tristan | tlater[m], 24674604383 | 11:00 |
| tristan | oooh but now cache_size *jumped* to 25588344368 | 11:00 |
| tristan | Ok now I have something interesting | 11:01 |
| tristan | A cleanup job was triggered ! | 11:01 |
| tlater[m] | Eum | 11:02 |
| * tlater[m] needs to have a look at this cache_size file code | 11:02 | |
| tristan | tlater[m], this is what happened: https://gitlab.com/BuildStream/buildstream/issues/623#note_98475350 | 11:04 |
| *** leopi has quit IRC | 11:04 | |
| toscalix | valentind: created https://gitlab.com/BuildStream/website/issues/16 | 11:04 |
| toscalix | please add there your progress on the installation page | 11:04 |
| *** leopi has joined #buildstream | 11:05 | |
| toscalix | valentind: I added there the move to the semantic versioning section | 11:07 |
| tlater[m] | tristan: Thanks, looks like we need to be careful to prune some orphaned stuff before we try and follow refs. | 11:08 |
| tristan | tlater[m], seems there are two issues at work here | 11:10 |
| tlater[m] | Yep. The first issue is that we don't catch disk-overflowing exceptions. We should catch those and trigger a cleanup instantly | 11:10 |
| tristan | tlater[m], I'm not sure how well that will play out honestly; sounds like it will be a complex fix | 11:11 |
| tlater[m] | The second is that if we fail to write certain files at some point, when we clean up later we'll try to access files that don't exist because we ran out of space. | 11:11 |
| *** leopi has quit IRC | 11:11 | |
| tristan | tlater[m], i.e. we need those tasks to "Fail temporarily"... then launch a cleanup from the main process, and then *retry* the temporarily failed builds/pulls | 11:12 |
| tlater[m] | I think the latter issue is almost a bit more problematic, since this basically means the cache is corrupted. | 11:13 |
| tristan | tlater[m], I think that for the second issue; a quick fix is totally possible, we could be more fault tolerant at prune time | 11:13 |
| tristan | Ah, indeed, there is that | 11:13 |
| tristan | a partially committed artifact should not be represented as an expected complete artifact | 11:13 |
| tristan | So commit needs to be altogether much more robust, probably doing something in a temp directory and ending with a batch os.rename() | 11:14 |
| *** leopi has joined #buildstream | 11:14 | |
| tlater[m] | That would also make detecting a ran-out-of-space problem a lot easier... | 11:15 |
| tiagogomes | tristan that can't be done. You need to place the files at a particular locations, and the location depends on the file hash | 11:16 |
| tristan | I'm a bit surprised if it doesnt already work in that way; it's better to ask juergbi how he intended commits to be atomic and safe | 11:16 |
| tlater[m] | tristan: On a slightly unrelated note, how do you get gnome build meta to build? I keep having issues when trying to track... | 11:16 |
| tristan | tlater[m], what kind of issues ? I run `bst build --track-all core.bst` for instance | 11:16 |
| tristan | missing tarballs or such ? | 11:17 |
| tiagogomes | So you can't atomically commit an artifact. The only thing that could can do is after successfully placing all files, create the ref | 11:17 |
| tlater[m] | Git repositories that fail to clone | 11:17 |
| tristan | tiagogomes, can't you do all of that such that the hashes are in a temp dir anyway ? | 11:17 |
| tristan | tiagogomes, if I understand correctly anyway, the hashes of actual files are content hashes | 11:18 |
| tristan | the path cannot be a part of that hash | 11:18 |
| tiagogomes | You can do that, but then you will have to do multiple os.rename() | 11:18 |
| juergbi | files are already written in temp dir and only moved when complete | 11:18 |
| juergbi | it's not artifact-atomic, but per-object atomic | 11:19 |
| tristan | right, I kind of expected juergbi thought about this :) | 11:19 |
| juergbi | and artifact ref should be written last | 11:19 |
| tlater[m] | juergbi: any idea how this could happen then? https://gitlab.com/BuildStream/buildstream/issues/623#note_98475350 | 11:19 |
| tristan | tlater[m], probably there is another explanation for https://gitlab.com/BuildStream/buildstream/issues/623#note_98475350 | 11:19 |
| tristan | beat me to it :) | 11:19 |
| juergbi | the whole expiry procedure is known to not be safe with regards to other operations | 11:21 |
| juergbi | the same logic as in OSTree was used with the intention of fixing this later | 11:21 |
| juergbi | i.e., concurrent prune and commit can result in partial artifacts | 11:21 |
| juergbi | which we should really fix asap | 11:22 |
| tlater[m] | When that happens no other operations should be in progress... I'm a bit suspicious of that `pull` log though. | 11:22 |
| juergbi | well, at the point of the stack trace, the ref is already dangling | 11:23 |
| juergbi | so the issue must have happened earlier | 11:23 |
| juergbi | system crash could also result in such issues but probably not the cause here | 11:24 |
| tiagogomes | commit and pull can run in parallel with the artifact cleaning, as the build and job and pull job don't require the CACHE resource to be available | 11:24 |
| tlater[m] | We didn't have a cleanup since the crash, I think. | 11:25 |
| tristan | juergbi, the crash in the referred issue occurred yesterday in the same cache... this cleanup job was the first one to run since then | 11:33 |
| tristan | So, it *could* be the cause here ? | 11:33 |
| tristan | those were crashes related to out of space conditions | 11:34 |
| tristan | <tiagogomes> commit and pull can run in parallel with the artifact cleaning, as the build and job and pull job don't require the CACHE resource to be available | 11:36 |
| tristan | tiagogomes, (tlater ^^^) That is supposed to be entirely false | 11:36 |
| tristan | That was the whole point of resources.py | 11:36 |
| tristan | HOWEVER... seeing as I am seeing 8 pull tasks getting terminated in response to the BUG raised in the cleanup job | 11:37 |
| tristan | it MUST be wrong | 11:37 |
| tristan | Can we fix that first ? | 11:37 |
| toscalix | tiagogomes: can you have a look at the https://gitlab.com/BuildStream/website/merge_requests/22 | 11:37 |
| tristan | tlater[m], looking at the log, clearly we have pull running concurrently with cleanup | 11:39 |
| juergbi | tristan: creating the ref is done as last part of commit, so I don't see how we can have incomplete artifact because of that | 11:39 |
| toscalix | and https://gitlab.com/BuildStream/website/merge_requests/21 I would need a review, please | 11:39 |
| tristan | juergbi, I was referring to your statement "system crash could also result in such issues but probably not the cause here" but wasn't sure why... but yeah I agree with you | 11:40 |
| tiagogomes | You can't simply add ResourceType.CACHE to resources of the build and pull jobs, as the maximum number of processes that can have that resource is 1. So goodbye parallelism | 11:40 |
| tiagogomes | toscalix commented | 11:43 |
| *** tristan has quit IRC | 11:43 | |
| tlater[m] | Hrm | 11:44 |
| tlater[m] | Sounds like we'll need distinction between read/write | 11:44 |
| tlater[m] | But wait, how are builds parallelized then? | 11:45 |
| tlater[m] | Surely pull just needs the same resources as build? | 11:45 |
| *** tristan has joined #buildstream | 11:52 | |
| *** ChanServ sets mode: +o tristan | 11:53 | |
| tristan | oops | 11:53 |
| tristan | not sure my last got through but... here comes something... | 11:53 |
| * tiagogomes sets the scene with some drumming | 11:53 | |
| gitlab-br-bot | buildstream: merge request (tristan/fix-cache-exclusivity->master: _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE) #775 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/775 | 11:55 |
| tristan | tlater[m], looking at only the API bits of the scheduler and ignoring the implementation, this patch SHOULD fix the crash in cleanup: https://gitlab.com/BuildStream/buildstream/merge_requests/775 | 11:56 |
| tristan | tlater[m], Unless, it never actually worked in the way it was intended to | 11:56 |
| tristan | Maybe there was some regression where the pull/build jobs stopped asking for shared access to the CACHE resource ? | 11:57 |
| toscalix | tiagogomes: related with tool vs toolset, I honestly think that the latter reflects better what we are doing | 11:57 |
| tiagogomes | No strong opinions here :) | 11:57 |
| tristan | tlater[m], Looks like the Queues dont have a way to ask for exclusive access, but the CacheSize job does in fact expose exclusive_resources as a constructor argument | 11:58 |
| tristan | So that shouldnt be a big deal... tlater[m]; do you expect my trivial patch to fix it ? | 11:58 |
| * tristan wonders if it will even pass CI | 11:58 | |
| tiagogomes | tristan that won't work because of I wrote above | 11:59 |
| tiagogomes | "as the maximum number of processes that can have that resource is 1. So goodbye parallelism" | 12:00 |
| tristan | tiagogomes, Then resources.py is *broken*... one sec, hold on... | 12:00 |
| tiagogomes | ¯\_(ツ)_/¯ | 12:01 |
| tristan | tiagogomes, that's not how it's supposed to work, it's a read-write lock, not a dumb bit; but it might be implemented wrong | 12:01 |
| tristan | tiagogomes, see this LOC: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_scheduler/scheduler.py#L324 | 12:01 |
| tlater[m] | tiagogomes is wrong here | 12:01 |
| tristan | tiagogomes, When we launch the CleanupJob, it asks for exclusive access | 12:02 |
| tlater[m] | Beat me to it :) | 12:02 |
| tlater[m] | I'm pretty sure that patch will fix it | 12:02 |
| tristan | I think he is wrong too, but he might have discovered something in the implementation such that he is right, and we are only right in theory, but not in practice | 12:02 |
| tiagogomes | My observations are only theoretical based on looking at the codebase in the past :) | 12:03 |
| tlater[m] | I won't argue with that... Though I think we'd notice if only one build job ran at a time | 12:03 |
| tristan | Okay, I do think that we could probably use more commenting and boiler plate around there | 12:04 |
| tristan | I mean... the fact that queue.py does not clearly offer support for it's subclasses to request exclusive access to a resource, is misleading; it leads one to believe there is no such thing as exclusive access | 12:04 |
| tristan | Even if that facility is currently unused by Queues | 12:05 |
| tristan | Now, I guess I have to delete my whole artifact cache | 12:06 |
| tlater[m] | tristan: perhaps not | 12:06 |
| tiagogomes | To be clear, I don't think that patch will solve it the way of want because of this: https://paste.fedoraproject.org/paste/TPdkqMno1oD1AO1kH0Ayjw | 12:07 |
| tlater[m] | I do suspect that the concurrency caused that... | 12:07 |
| tristan | If I have corrupt artifacts due to this issue, a cleanup job will still hit those bad refs | 12:07 |
| tlater[m] | Ah, right | 12:07 |
| tristan | tiagogomes, Ah I see why you would think that | 12:07 |
| tristan | And you'd also be right ! | 12:07 |
| tristan | Those, afaik, are supposed to be the caps on how many SHARED locks can occur | 12:08 |
| tristan | tlater[m], maybe that should be 0 ? and 0 should be taken to mean "no limit" ? | 12:08 |
| tlater[m] | Hmm | 12:09 |
| tristan | Well, the patch I put up *would work*, but it would also cause every build / pull to be limited to one task at a time | 12:09 |
| * bochecha just updated the Fedora package to 1.2.0 | 12:09 | |
| * tlater[m] takes a look at the implementation again | 12:09 | |
| tristan | bochecha, Nice ! | 12:09 |
| tristan | bochecha, I updated the pre-built sessions in the docs for that :D | 12:10 |
| bochecha | :) | 12:10 |
| *** solid_black has quit IRC | 12:13 | |
| tlater[m] | tristan: 0 does currently mean no limit | 12:13 |
| tlater[m] | So I think it should indeed be 0 | 12:13 |
| *** solid_black_ has joined #buildstream | 12:14 | |
| tlater[m] | That makes me wonder if we somehow missed build jobs not being concurrent, or if something else is going on here. | 12:14 |
| tristan | tlater[m], I only just now added in my patch that build jobs require the CACHE resource at all | 12:16 |
| tristan | tlater[m], so my patch would cause them to not be concurrent | 12:16 |
| *** solid_black_ has quit IRC | 12:17 | |
| tlater[m] | Ah, that does explain it. I'm not quite sure how this went wrong then, but setting the cache limit to 0 should do what we want. | 12:17 |
| *** solid_black_ has joined #buildstream | 12:17 | |
| gitlab-br-bot | buildstream: merge request (tristan/fix-cache-exclusivity->master: _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE) #775 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/775 | 12:19 |
| tristan | tlater[m], resources.py indeed has a "> 0 and..." check for max_resources being 0 meaning unlimited | 12:20 |
| tristan | Added that to 775 above | 12:20 |
| tristan | that should at least in theory fix this exclusivity problem | 12:20 |
| tiagogomes | :) | 12:21 |
| tristan | tlater[m], it's all your fault ! | 12:21 |
| tristan | hehe | 12:21 |
| * tristan couldnt resist | 12:21 | |
| * tlater[m] sits in the corner | 12:23 | |
| tristan | haha | 12:27 |
| tristan | Hmmm, ok so, launched a build with this backported to bst-1.2 | 12:29 |
| tristan | It has concurrent pulls going on | 12:29 |
| tristan | It did not launch a size check or cleanup at startup, but I suppose it's not too awful when things are working as intended | 12:30 |
| tristan | lets see what happens after a pull task completes | 12:30 |
| tlater[m] | It also shouldn't starve size checks because exclusive jobs will be prioritized... I really wonder why I set that to 1. | 12:32 |
| *** alatiera_ has joined #buildstream | 12:40 | |
| tlater[m] | Oof, building gnome certainly won't happen in 5GB, hm | 12:56 |
| * tlater[m] wonders if there's a different project that he could use to test | 12:57 | |
| coldtom | tlater: maybe try freedesktop bootstrap? | 13:03 |
| tlater[m] | Well, gnome includes freedesktop, which is taking up a lot of storage/time already | 13:04 |
| tlater[m] | Hrm, I guess I'll just have to bite the bullet and accept this is going to take a little while | 13:04 |
| tristan | bochecha, Will that be in the official repo soon ? | 13:08 |
| bochecha | tristan, I haven't started the work to push it there yet | 13:08 |
| tristan | bochecha, since we are trying to sort out our buildstream.build website and migrating the install guide to there instead of the reference docs / user guide... we'd like to know when we can update http://buildstream.gitlab.io/buildstream/install_linux_distro.html#fedora :) | 13:08 |
| bochecha | once I start, it might take a bit of time, because each required package needs to be reviewed | 13:08 |
| tristan | Right, that is fine | 13:08 |
| tristan | I think one of us needs to setup a ... what is it... "ppa" ? for debian also | 13:09 |
| bochecha | there are 3 required new packages, and one that needs to be updated in Fedora | 13:09 |
| tristan | So we can have at least debian and fedora in the works as well as arch | 13:09 |
| bochecha | if we find Fedora packagers to review the packages, it can go pretty quickly :) | 13:09 |
| bochecha | ppa are for Ubuntu, I'm not sure what the equivalent is for Debian | 13:09 |
| tristan | Yeah indeed; have to find that out | 13:10 |
| tristan | But, one thing at a time; we probably land the website without that, but hopefully can follow up soon after | 13:10 |
| bochecha | this might be the most problematic part: https://bugzilla.redhat.com/show_bug.cgi?id=1616075 | 13:11 |
| bochecha | blessings is too old in Fedora 28 | 13:11 |
| tristan | Gah, damn blessings | 13:12 |
| bochecha | and the maintainer hasn't replied | 13:12 |
| tristan | bochecha, you know what; it will be easier to *REMOVE* blessings as a dependency | 13:12 |
| bochecha | but then, if I take too long to push buildstream through the review, Fedora 29 will be released and we can just say "packages only for Fedora >= 29 :D | 13:12 |
| tristan | That is ridiculously small amount of python code that I have been wanting to just replace for a long time | 13:12 |
| tristan | Why depend on something when all it does is (A) Check if stdin isatty() ... and (B) #define some ANSI escape sequences ? | 13:13 |
| bochecha | ANSI for the colors? | 13:13 |
| tristan | bochecha, no, even those are taken care of by click | 13:13 |
| bochecha | so which ones are taken from blessings? | 13:14 |
| tristan | bochecha, the ones we use from blessings are the cursor repositioning stuff (move the cursor up and down a line) | 13:14 |
| bochecha | oh | 13:14 |
| toscalix | tiagogomes: I will amend the MR for the release announcement this afternoon | 13:14 |
| bochecha | tristan, just this? :x | 13:15 |
| tristan | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_frontend/status.py | 13:15 |
| tristan | bochecha, see the _move_up(), _clear_line() things | 13:15 |
| bochecha | tristan, did you know about sys.stdout.isatty() ? | 13:15 |
| bochecha | (same for stderr obviously) | 13:16 |
| tristan | bochecha, yes I did, I ended up looking into the blessings code and found out that it did... nothing really special | 13:16 |
| bochecha | :) | 13:16 |
| tristan | and since then, I have been wanting to rip out this "Terminal()" object | 13:16 |
| tristan | we also check Terminal().does_styling, which I think also just stupidly checks if it's a tty | 13:17 |
| tristan | really silly | 13:17 |
| bochecha | my biggest problem with blessings is the way it handles unknown attributes on its terminal object | 13:17 |
| bochecha | you'd expect an AttributeError | 13:17 |
| bochecha | it just returns an empty string | 13:17 |
| bochecha | no warning, nothing | 13:17 |
| tristan | It's python I guess</rant> | 13:18 |
| *** phildawson has quit IRC | 13:18 | |
| bochecha | because of that, and the blessings package in Fedora being too old (it didn't have the does_styling attribute) I just didn't have any status bar in buildstream | 13:18 |
| *** phildawson has joined #buildstream | 13:18 | |
| bochecha | took me a while to figure this out | 13:18 |
| tristan | hah, wow | 13:18 |
| bochecha | well, no, not "it's python"… more like "someone decided to do more work in order ot have less result" | 13:19 |
| bochecha | the blessings dev decided to implement the special __getattr__ method and make it return "" on unknown attributes | 13:19 |
| tristan | Yeah maybe; I constantly get frustrated that python culture is "We live in a bubble and don't care about API stability, just pin your version", and am conflating it with "The maintainer got tired of API stability and just hacked around it by saying 'it'll do what it can and just not crash if you have an older version'" | 13:20 |
| tristan | bochecha, I'm guessing the idea is that... most of the Terminal attributes are strings to render, whether they are functions or just values | 13:22 |
| tristan | And if you render a "", because too old version; "no harm done I guess" | 13:22 |
| bochecha | yeah | 13:23 |
| bochecha | that status bar is not useful anyway, no harm done :] | 13:23 |
| tristan | bochecha, Ok I can put that on my plate I think and get that done by 1.2.1, it's seriously trivial | 13:24 |
| tristan | let's just dump blessings | 13:25 |
| tristan | a bit unorthodox but not an API break of any sort on our side | 13:25 |
| *** adds68 has quit IRC | 13:31 | |
| *** qinusty has quit IRC | 13:32 | |
| *** jennis has quit IRC | 13:32 | |
| *** tiagogomes_ has quit IRC | 13:32 | |
| *** coldtom has quit IRC | 13:32 | |
| *** qinusty has joined #buildstream | 13:37 | |
| *** jennis has joined #buildstream | 13:37 | |
| *** coldtom has joined #buildstream | 13:37 | |
| *** tiagogomes_ has joined #buildstream | 13:37 | |
| *** adds68 has joined #buildstream | 13:37 | |
| *** phildawson has quit IRC | 13:50 | |
| toscalix | I suggest to remove the beaver from the upper banner. I would prefer to include it as image on the page | 13:50 |
| toscalix | or not include it at all | 13:50 |
| *** tristan has quit IRC | 13:54 | |
| toscalix | tiagogomes: since we have all the pages already except the roadmap, is it possible to focus some energy in getting the complete website structure implementing the sitemap? | 13:58 |
| tiagogomes | toscalix, where are the updates page, meetings and events page | 14:05 |
| toscalix | the update page is, in this case, the release announcement | 14:06 |
| toscalix | the meeting page will be the wiki page where we have the meeting logs for now. We will not create a new page | 14:06 |
| toscalix | I am really out of context with the events page | 14:07 |
| toscalix | did you see it in the sitemap ? | 14:07 |
| tiagogomes | In the sitemap is written "Meetings and events" | 14:08 |
| toscalix | that will be the wiki page for now. Let me add the link in the ticket | 14:09 |
| tiagogomes | The realease announcement should be an article, in a separate page to show the articles | 14:09 |
| toscalix | we call that.... updates | 14:09 |
| toscalix | just like the codethink page | 14:09 |
| toscalix | so we include there blogs and announcements | 14:09 |
| toscalix | added to the ticket description: https://gitlab.com/BuildStream/website/issues/9 | 14:11 |
| *** leopi has quit IRC | 14:14 | |
| *** leopi has joined #buildstream | 14:15 | |
| *** leopi has quit IRC | 14:19 | |
| *** leopi has joined #buildstream | 14:19 | |
| *** lachlan has quit IRC | 14:35 | |
| *** tristan has joined #buildstream | 14:37 | |
| *** ChanServ sets mode: +o tristan | 14:37 | |
| tristan | tlater[m], tiagogomes; the good news is that testing the MR I put up backported to 1.2, when launching the build it did notice that it needed to launch a cleanup job; even though no cache_size job ran first; presumably because of the same cache_size file | 14:38 |
| tristan | Further... it launched exactly 4 pull jobs, and no more | 14:38 |
| tristan | It waited for the 4 pull jobs to complete and did not queue any builds or pulls | 14:39 |
| tristan | Then it ran the cleanup | 14:39 |
| tristan | The bad news is, since my artifact cache is corrupt from the last stack trace where concurrent pull + prune was happening; I got the same stack trace again | 14:39 |
| tristan | I wonder if there is a way to delete the corrupt artifacts | 14:40 |
| tristan | And keep my current cache, so I can easily test this with a pretty full cache | 14:40 |
| tristan | juergbi, any ideas if I can recover and try running more tests on this ? | 14:40 |
| juergbi | tristan: if it's just about a single (or very few) artifacts, you can simply delete the corresponding refs/heads/... file | 14:42 |
| juergbi | may need to print out the ref name | 14:42 |
| tristan | juergbi, would that be knowable from say, this error: | 14:48 |
| tristan | FileNotFoundError: [Errno 2] No such file or directory: '/home/tristan/.cache/buildstream/artifacts/cas/objects/7f/4c864e8edd35a496c17285a1035b7221d3d317f90deb65a964c1d77c5c4364' | 14:48 |
| tristan | I guess not, cause the file is not found | 14:48 |
| tristan | like, I *could* do it and retry a few times | 14:49 |
| tristan | if I knew how to get the ref name | 14:49 |
| juergbi | tristan: if you don't want to add a debug print for this, you could probably grep for 7f4c86e8edd... in the refs directory | 14:49 |
| juergbi | although, that would only help if it's the toplevel | 14:50 |
| tristan | juergbi, I've updated https://gitlab.com/BuildStream/buildstream/issues/623 and it's late... ok I'll take note of that | 14:50 |
| tristan | hmmm | 14:50 |
| tristan | if you have any ideas, comment on the issue, otherwise I'll give up and zap the cache | 14:50 |
| tristan | juergbi, maybe a couple choice debug prints could help me, though; ones that I would not commit in any MR | 14:51 |
| tristan | well, maybe it's overkill, and we can find a way to test these changes | 14:51 |
| tristan | or to reproduce and fix | 14:51 |
| juergbi | tristan: yes, could catch FileNotFoundError in prune() in the first loop and print out 'filename' (ref name) in that case | 14:52 |
| juergbi | we might want to make this more robust in any case | 14:52 |
| tristan | yeah, that wouldnt be debug print, that would just be a more descriptive exception I guess when failing :) | 14:53 |
| tristan | if that's what you mean | 14:53 |
| tristan | Probably an enhanced error could be useful there | 14:53 |
| tristan | "Tried to delete a corrupt artifact: details: bla bla bla... standard print of a list of missing objects for given ref, etc" | 14:54 |
| *** lachlan has joined #buildstream | 14:57 | |
| *** lachlan has quit IRC | 15:02 | |
| *** lachlan has joined #buildstream | 15:23 | |
| toscalix | which 3 features do you consider the most relevant ones on this release? I have CAS server, config file (includes) and..... | 15:28 |
| *** lachlan has quit IRC | 15:51 | |
| *** phildawson has joined #buildstream | 15:52 | |
| *** lachlan has joined #buildstream | 16:06 | |
| *** solid_black_ has quit IRC | 16:10 | |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 16:14 |
| *** leopi has quit IRC | 16:17 | |
| *** lachlan has quit IRC | 16:59 | |
| *** lachlan has joined #buildstream | 17:08 | |
| *** leopi has joined #buildstream | 17:12 | |
| *** lachlan has quit IRC | 17:14 | |
| *** lachlan has joined #buildstream | 17:47 | |
| *** jonathanmaw has quit IRC | 17:48 | |
| *** leopi has quit IRC | 18:34 | |
| *** rdale has quit IRC | 18:57 | |
| *** toscalix has quit IRC | 19:12 | |
| *** lachlan has quit IRC | 19:18 | |
| *** gitlab-br-bot has quit IRC | 21:17 | |
| *** gitlab-br-bot has joined #buildstream | 21:17 | |
| *** gitlab-br-bot has joined #buildstream | 21:17 | |
| *** tristan has quit IRC | 21:40 | |
| *** alatiera_ has quit IRC | 21:53 | |
| *** bochecha has quit IRC | 22:11 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!