| *** slaf has quit IRC | 04:31 | |
| *** ssssam[m] has quit IRC | 04:31 | |
| *** awacheux[m] has quit IRC | 04:31 | |
| *** ptomato[m] has quit IRC | 04:31 | |
| *** segfault3[m] has quit IRC | 04:31 | |
| *** kailueke[m] has quit IRC | 04:31 | |
| *** abderrahim[m] has quit IRC | 04:31 | |
| *** asingh_[m] has quit IRC | 04:31 | |
| *** alatiera has quit IRC | 04:31 | |
| *** theawless[m] has quit IRC | 04:31 | |
| *** pro[m] has quit IRC | 04:31 | |
| *** connorshea[m] has quit IRC | 04:31 | |
| *** inigomartinez has quit IRC | 04:31 | |
| *** waltervargas[m] has quit IRC | 04:31 | |
| *** mattiasb has quit IRC | 04:31 | |
| *** cgmcintyre[m] has quit IRC | 04:31 | |
| *** bochecha has quit IRC | 04:31 | |
| *** m_22[m] has quit IRC | 04:31 | |
| *** ernestask[m] has quit IRC | 04:31 | |
| *** jjardon[m] has quit IRC | 04:31 | |
| *** rafaelff[m] has quit IRC | 04:31 | |
| *** csoriano has quit IRC | 04:31 | |
| *** slaf has joined #buildstream | 04:32 | |
| *** ssssam[m] has joined #buildstream | 04:32 | |
| *** awacheux[m] has joined #buildstream | 04:32 | |
| *** ptomato[m] has joined #buildstream | 04:32 | |
| *** segfault3[m] has joined #buildstream | 04:32 | |
| *** kailueke[m] has joined #buildstream | 04:32 | |
| *** abderrahim[m] has joined #buildstream | 04:32 | |
| *** asingh_[m] has joined #buildstream | 04:32 | |
| *** alatiera has joined #buildstream | 04:32 | |
| *** theawless[m] has joined #buildstream | 04:32 | |
| *** pro[m] has joined #buildstream | 04:32 | |
| *** connorshea[m] has joined #buildstream | 04:32 | |
| *** inigomartinez has joined #buildstream | 04:32 | |
| *** waltervargas[m] has joined #buildstream | 04:32 | |
| *** mattiasb has joined #buildstream | 04:32 | |
| *** cgmcintyre[m] has joined #buildstream | 04:32 | |
| *** bochecha has joined #buildstream | 04:32 | |
| *** m_22[m] has joined #buildstream | 04:32 | |
| *** ernestask[m] has joined #buildstream | 04:32 | |
| *** jjardon[m] has joined #buildstream | 04:32 | |
| *** rafaelff[m] has joined #buildstream | 04:32 | |
| *** alatiera has quit IRC | 04:32 | |
| *** alatiera has joined #buildstream | 04:32 | |
| *** csoriano has joined #buildstream | 04:32 | |
| *** tristan has joined #buildstream | 05:31 | |
| gitlab-br-bot | buildstream: merge request (tristan/dependency-docs-fixes->master: doc/source/format.rst: Some enhancements in how we document dependencies) #398 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/398 | 06:04 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (jjardon/order_elements->master: source/format.rst: Order of components in "elements:" doesn't matter) #390 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/390 | 06:06 |
| gitlab-br-bot | buildstream: merge request (tristan/dependency-docs-fixes->master: doc/source/format.rst: Some enhancements in how we document dependencies) #398 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/398 | 06:15 |
| gitlab-br-bot | buildstream: issue #296 ("Lightweight artifacts") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/296 | 06:20 |
| gitlab-br-bot | buildstream: issue #361 ("Store junction refs separately to project.refs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/361 | 06:26 |
| gitlab-br-bot | buildstream: issue #358 ("Documentation: Element's dependencies order") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/358 | 06:34 |
| gitlab-br-bot | buildstream: merge request (tristan/dependency-docs-fixes->master: doc/source/format.rst: Some enhancements in how we document dependencies) #398 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/398 | 06:34 |
| paulsherwood | tristan: are you around? | 07:34 |
| tristan | paulsherwood, Certainly | 07:37 |
| tristan | Back from your trip then ? | 07:37 |
| *** toscalix has joined #buildstream | 08:03 | |
| paulsherwood | yes, maybe jetlag explains my irritability :-) | 08:04 |
| *** jennis has joined #buildstream | 08:12 | |
| * paulsherwood is disappointed by the 'fix' to missing project.conf file | 08:16 | |
| paulsherwood | as a driveby user i don't want to answer three confusing questions... that's **worse** than having to create a one-line project.conf file | 08:17 |
| tristan | You dont have to answer the other two questions really, only the first one | 08:17 |
| paulsherwood | i don't want to answer any at all :) | 08:18 |
| tristan | Creating a one-line project.conf is still an option, or just typing `bst init --project-name my-project` also works | 08:19 |
| tristan | (or bst --directory /path/to/desired/project/location init --project-name my-project) | 08:20 |
| tristan | that wont ask any questions | 08:20 |
| paulsherwood | you can understand, i'm sure, why that doesn't help for my 'i just want to try buildstream' usecase | 08:34 |
| *** jonathanmaw has joined #buildstream | 08:37 | |
| *** noisecell has joined #buildstream | 08:38 | |
| tristan | paulsherwood, what would help would be a getting started guide - however this still *helps*, because it doesnt give you a crash, and it immediately informs you that you *need* a project.conf, and holds your hand while creating one | 08:40 |
| tristan | I would say this goes above and beyond when it comes to user friendlyness | 08:40 |
| *** tiago has quit IRC | 09:08 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-expire-remote-cache-artifacts->master: WIP:136 expire remote cache artifacts) #364 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/364 | 09:09 |
| *** tiago has joined #buildstream | 09:11 | |
| * tlater thinks the coverage thing is cute but is annoyed that it's ~60% out of whack | 09:13 | |
| juergbi | yes, I commented this as well. Doesn't make sense to merge it if it's wrong | 09:13 |
| tristan | No no, it's not ready | 09:16 |
| tristan | we wont merge that, I think jjardon was just running some test using the regex in the gitlab UI | 09:17 |
| tristan | which we should disable if it's misinformed | 09:17 |
| tlater | I thought it already was merged? | 09:17 |
| tlater | I see coverages on master's pipeline at least | 09:17 |
| tristan | tlater, see: https://gitlab.com/BuildStream/buildstream/settings/ci_cd | 09:18 |
| juergbi | it seems the regex is now in the project settings, but the badge hasn't been merged | 09:18 |
| tristan | right, we should disable that as it's nonsense | 09:18 |
| juergbi | and we should also remove the regex again if it's not correct | 09:18 |
| tristan | I just cleared the regex in settings | 09:19 |
| tristan | I think the feature is broken on the gitlab side, either that or just not documented | 09:19 |
| tlater | I wonder how that regex could even work, considering we want the *final* coverage number, as opposed to random percentages that occur everywhere in our CI output. | 09:19 |
| tristan | https://docs.gitlab.com/ce/ci/yaml/#coverage | 09:20 |
| tristan | That is what should be used in the .gitlab-ci.yml | 09:20 |
| tristan | but I tried some things and it just doesnt work | 09:20 |
| tristan | I dont think gitlab understands that we want *one job* to define the coverage for the *whole pipeline* | 09:21 |
| *** dominic has joined #buildstream | 09:26 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 09:29 |
| tlater | tristan: Looking at #209, back when that issue was opened we also wanted to document the workspaces.yml file so that IDEs can use it | 09:43 |
| tlater | Do we want to make that file public API? | 09:43 |
| tristan | tlater, no | 09:46 |
| tristan | tlater, we explicitly made `bst workspace list` dump machine readable YAML, in order to avoid making that local state file public API | 09:46 |
| tlater | Ah, right | 09:47 |
| tlater | So that use case is covered too, I'll close #209 then :) | 09:47 |
| tristan | Note that I pushed a fix today which adds *Since 1.2* to the docs for Element.prepare() | 09:47 |
| tristan | tlater, just pointing it out, new public API should be documented as Since the next stable version | 09:48 |
| * tlater didn't realize we did that | 09:48 | |
| tlater | Makes perfect sense though, thanks for pointing it out | 09:48 |
| gitlab-br-bot | buildstream: issue #209 ("Dont run configure commands more than once for incremental workspace builds") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/209 | 09:50 |
| tristan | tlater, so that means incremental builds for workspaces is DONE ! | 09:52 |
| tristan | Nice :) | 09:52 |
| tlater | \o/ | 09:52 |
| jjardon | tristan: juergbi see my comments at https://gitlab.com/BuildStream/buildstream/merge_requests/397#note_68583108 | 10:09 |
| juergbi | jjardon: ok, but I don't understand why you're requesting to merge this when it's completely broken | 10:10 |
| juergbi | also, https://gitlab.com/BuildStream/buildstream/badges/master/coverage.svg is not identical to https://gitlab.com/BuildStream/buildstream/badges/master/coverage.svg?job=coverage | 10:11 |
| juergbi | the latter is off even worse, though | 10:11 |
| gitlab-br-bot | buildstream: merge request (jmac/artifact-receive-profiling->master: WIP: Add artifact cache receive profiling domain and instructions.) #383 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/383 | 10:11 |
| * tlater wonders if this can be fixed by setting the regex to some variant of '\d+.\d+%' | 10:16 | |
| juergbi | I would assume so | 10:16 |
| Nexus | Are there currently plans for any further modifications to the workspace+artifact cache areas of bst? tristan juergbi | 10:17 |
| juergbi | tlater: in the pipeline status it might still show the wrong percentage, though, because I don't see where you could specify which job to take it from. It should work for the badge with ?job=coverage, though | 10:17 |
| tristan | Nexus, tlater is working on local artifact expiry, which I expect will drastically change things too | 10:17 |
| juergbi | Nexus: there are artifact cache changes planned for remote execution | 10:18 |
| tristan | Also that | 10:18 |
| Nexus | tristan: Do you think it is worth me continuing with the "opening workspace using cached buildtrees" code at this time? It may end up having to be re-written if things change too much | 10:19 |
| tlater | Nexus: I thought that was on the edge of done? | 10:19 |
| tlater | My branch will take a while still, considering the concurrency issues I realized I had. | 10:20 |
| tlater | And I think jennis isn't very far either | 10:20 |
| Nexus | tlater: it is, but it looks like the cache key generation code was made private, which changes some things for me, and i wasn't sure if many more things like that might be happening | 10:20 |
| tlater | These kinds of conflicts should be very easy to resolve, don't think it will take you long enough for jennis' or my branches to affect you, so you should be looking at the final set of conflicts, save for refactoring work. | 10:21 |
| tristan | Nexus, do not expect code to not change | 10:21 |
| Nexus | kk | 10:22 |
| * Nexus will try to get this done asap to avoid the other branches | 10:26 | |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: WIP: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 10:56 |
| gitlab-br-bot | buildstream: issue #350 ("Add plugin to generate system images") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/350 | 11:14 |
| tristan | jjardon, I commented back on the issue, looks like my original regex actually *did* work: https://gitlab.com/BuildStream/buildstream/-/jobs/62751023 | 11:20 |
| tristan | juergbi, jjardon Notice that in the right hand side the coverage total is correctly reported | 11:20 |
| tristan | problem is it does not show up on the pipeline page or in merge requests | 11:20 |
| tristan | so a piece is missing | 11:20 |
| jjardon | tristan: ah, nice! | 11:20 |
| tristan | I mean, it's not really an improvement, because you *still* have to click on the "Coverage" job in order to see anything useful | 11:21 |
| tristan | But, at least it's a step to the right direction, and we could probably use it for a badge if we specify the job=coverage thing | 11:22 |
| tristan | but I'd really like to see it on the pipelines page and in merge requests | 11:22 |
| juergbi | gitlab just broke | 11:26 |
| tristan | yeah :-/ | 11:26 |
| jjardon | tristan: I'd imagene the pipeline will use the expression you set in the UI only against the "coverage" job, if you have one. Seems It's not the case | 11:28 |
| tristan | juergbi, I just streamlined the instantiation process, and have _loader.py using Element._stage_sources_at(), coincidentally, it fixes the workspace-on-junctions bug | 11:28 |
| tristan | jjardon, I think you are imagining that gitlab has some special case checking if a job is named "coverage" | 11:29 |
| juergbi | tristan: nice | 11:29 |
| tristan | jjardon, instead there is a setting in the .gitlab-ci.yml which explicitly sets the coverage regex on a job; *I* expected it to assume that if you only have one regex set; it would assume that it's "the one for the whole pipeline" | 11:29 |
| jjardon | yeah, the docs are a bit vague and seems that would be the case | 11:30 |
| tristan | jjardon, both assumptions seem wrong though | 11:30 |
| jjardon | tristan: indeed | 11:30 |
| tristan | I think we can go ahead and merge the commit which adds the regex to the coverage job | 11:31 |
| tristan | it's a step forward, and we wont forget that way | 11:31 |
| jjardon | agree | 11:31 |
| tristan | I'm going to remove this useless warning https://gitlab.com/BuildStream/buildstream/issues/360 | 11:37 |
| tristan | unless anyone has some objection | 11:37 |
| tristan | I.e. currently, when we type `bst build foo.bst` and there is a workspace open on `bar.bst` that is not a dependency of `foo.bst`, BuildStream gives you a warning about all the open workspaces which will not be used in the current build plan | 11:38 |
| tristan | This seems first of all, completely irrelevant information | 11:38 |
| tristan | Second of all, it is worsened by being a warning (as if there was some cause to worry) | 11:38 |
| tristan | it weakens our warnings to have messages like this (i.e. people will take warnings less seriously if we say: WARNING: You happen to have pants on at the moment) | 11:39 |
| * tlater always wondered why we wanted it | 11:40 | |
| tristan | tlater, I concur :) | 11:40 |
| tristan | I'm not sure, it's possible I asked for it, I can't recall | 11:41 |
| tristan | but I've been seeing it too and thinking, why is this relevant to me ? | 11:41 |
| tlater | I guess the idea was that someone might try and modify a build but not realize an element wasn't actually part of the pipeline | 11:41 |
| tlater | But I really don't think it's very useful to try and point that out | 11:42 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: WIP: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 11:42 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: WIP: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 11:45 |
| tristan | tlater, right, the user could just as well notice the absence of "Workspace: ~/workspace" beside any elements in the heading | 11:45 |
| tlater | Well, I don't think it's buildstream's business to autocorrect pipelines for the user anyway | 11:45 |
| tlater | If nothing changes they'll backtrace and realize they didn't include a library or something | 11:46 |
| tlater | The build won't take longer anyway, since all elements would be cached | 11:46 |
| tlater | Hrm, does anyone know what method is currently supposed to call `Element.__schedule_assemble()`? The buildqueue doesn't implement prepare(), so I'm wondering if it's perhaps expected to be called by `Element.status()`. | 11:50 |
| gitlab-br-bot | buildstream: merge request (tristan/instantiation-paths->master: Streamlining instantiation paths) #399 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/399 | 11:50 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: WIP: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 11:56 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: WIP: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 12:04 |
| Nexus | So, i need some way of checking if the build-tree has been cached before. Previously, i was attempting to extract the artifact and returning either the path of the build tree of "None", however the change which throws an error if you attempt to extract a non-existant artifact, has made this no longer possible. An option that tlater came up with in our discussions was some kind of variable external to | 12:06 |
| Nexus | the cache which contains the status of the cached build tree (existant or not). Does anyone object to this method/have any other ideas of how this could be done? | 12:06 |
| Nexus | tlater: did i put across your idea correctly? | 12:06 |
| * tlater thinks so, but has no valuable input on the how bit | 12:16 | |
| Nexus | i'm just hoping for no objections, i'll have to put some thought into how it would work | 12:17 |
| tlater | Nexus: If storing data outside the artifact is too hard you can store it in the artifact metadata | 12:18 |
| tlater | Oh, wait | 12:19 |
| tlater | Nexus: Your issue is that if there is no artifact, you can't tell if there's a build-tree inside the nonexistent artifact? | 12:19 |
| Nexus | that gave me an idea... | 12:21 |
| Nexus | possibly the same one | 12:21 |
| *** valentind has quit IRC | 12:21 | |
| Nexus | i wasn't checking if there was an artifact before trying to extract, i'd moved around the code and forgotten to move that too | 12:22 |
| *** valentind has joined #buildstream | 12:22 | |
| tlater | Nexus: If there is no artifact, then there is no cached build tree - you can solve your predicament with a simple try-catch; though, this won't solve the UX stuff we discussed. | 12:23 |
| Nexus | no, you're correct, that still needs to be addressed | 12:23 |
| Nexus | but this fixes my immediate problem | 12:24 |
| * tlater just wanted to keep Nexus from wasting time ;) | 12:24 | |
| Nexus | so, assuming there are no objections to the external variable being stored, would you have time to sit down and discuss options? | 12:24 |
| * tlater does, technically, but has no input beyond "Hm, would be nice if that was possible" | 12:25 | |
| tlater | Unless you have anything specific you're thinking about? | 12:26 |
| Nexus | are there any other examples of this kind of thing? | 12:26 |
| gitlab-br-bot | buildstream: merge request (tristan/instantiation-paths->master: Streamlining instantiation paths) #399 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/399 | 12:26 |
| tlater | Nope, I don't think we've had to deal with this sort of thing before. There's the 'tainted' metadata, which is similar, but you still have to extract the artifact to see if it's tainted | 12:26 |
| Nexus | tlater: what about something similar to the way that track works? Where it updates the element files? | 12:28 |
| tlater | Would you like to add a flag to user-facing data that just buildstream should handle? | 12:28 |
| * tlater thinks that's a poor idea | 12:29 | |
| tlater | We could have something in .bst/workspaces.yml, or a similar file, or you could investigate how well ostree/tar support extracting a sub-directory of an artifact or somesuch | 12:29 |
| tristan | tlater, good point, Element._schedule_assemble() should be moved down to Element.__schedule_assemble() in local private area | 12:29 |
| tristan | tlater, I missed that in my refactor it seems | 12:30 |
| tristan | tlater, to answer the question, it's resolved by the first calls to _update_state() | 12:30 |
| tlater | Oh, I didn't realize it wasn't properly private | 12:30 |
| tlater | Ok, ta, now to figure out why that isn't called then... | 12:31 |
| tristan | tlater, _update_state() looks daunting in it's current form | 12:32 |
| Nexus | holy coverage, why do i have hundreds of coverage files? | 12:33 |
| * tlater agrees, refactoring it will be poking in a hornet's nest | 12:33 | |
| tlater | Nexus: You're running a test, they'll be cleaned up in a bit | 12:33 |
| tristan | but it seems to be called appropriately if it's not cached, remotely cached, already scheduled to be assembled, or already assembled | 12:33 |
| tlater | If they aren't, you interrupted the tests at an unfortunate moment | 12:33 |
| Nexus | ah i see | 12:34 |
| tristan | I think they are deleted automatically at the beginning of the next `./setup.py test` run | 12:36 |
| tristan | you get one coverage file for each spawned process... which is a damn lot | 12:36 |
| Nexus | yup, the went away when my tests finished | 12:36 |
| Nexus | tlater: We could have something in .bst/workspaces.yml where's that? | 12:38 |
| tlater | Nexus: In the project directory after a workspace was opened. Obviously that particular file isn't suitable, but we could have another | 12:39 |
| tlater | I'm not sure I like that though. | 12:39 |
| tristan | jjardon, w00t! https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 12:39 |
| tristan | jjardon, the coverage showed up in the merge request this time | 12:39 |
| tristan | jjardon, how about I just unmark WIP and merge now ? | 12:39 |
| * tlater suggests discussing that particular option with someone with a better eye on how we manage state | 12:39 | |
| tristan | gitlab has been acting up today | 12:39 |
| tlater | tristan: Oh, nice :) | 12:40 |
| * tristan has been having trouble with getting pipelines to build here: https://gitlab.com/BuildStream/buildstream/merge_requests/399 | 12:40 | |
| tristan | tlater, yeah I know ! I feel like, asking in #gitlab caused them to fix the bug quickly and restart some services | 12:40 |
| tristan | I mean, it's the *same patch* for the coverage regex, but it was only showing up in the job before | 12:41 |
| tlater | Heh, pretty quick on their end then :) | 12:41 |
| tristan | Well, it seems pretty grass roots hehe | 12:41 |
| tristan | Oh, embarrassing bug, lets fix and push to production right now and hope nobody noticed | 12:42 |
| tristan | to be fair, the service has been more stable this year | 12:42 |
| tlater | Don't jynx it, we're only like a third in so far | 12:42 |
| Nexus | tlater: so, could i, in theory, create a "cached-build-tree" file in the .bst directory containing the path (or lack there of) of the cached build tree for that project and pull that into the code instead of extracting the artifact? | 12:42 |
| tlater | Nexus: I'd just keep track of whether a specific artifact *has* one, as the exact path might not be as hard as you'd like, and can be figured out at runtime. | 12:43 |
| tlater | But yes, technically that's possible and would be using what buildstream has in place for this sort of scenario | 12:44 |
| tlater | I'm not sure the slight improvement in UX is worth the overhead of keeping track of another one of these files though | 12:44 |
| tlater | I'm also not sure how robust it would be, or whether we'd like to keep a manifest of all local artifacts in a project-specific file | 12:45 |
| tlater | Perhaps we'd prefer adding actual manifest to the artifact cache? | 12:46 |
| tlater | But now we need to start thinking about whether this will be useful outside of this one specific case, and I don't think I have the foresight to actually judge that | 12:47 |
| tlater | juergbi might have some comments considering his work on CAS | 12:47 |
| gitlab-br-bot | buildstream: issue #292 ("bst doesn't use an open workspace for a junction") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/292 | 12:49 |
| gitlab-br-bot | buildstream: merge request (tristan/instantiation-paths->master: Streamlining instantiation paths) #399 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/399 | 12:49 |
| gitlab-br-bot | buildstream: merge request (tristan/unused-workspace-warning->master: _pipeline.py: Stop printing warning when a build plan does not use some existing workspaces) #400 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/400 | 12:52 |
| tlater | Nexus: I think the better option is to read into ostree to see if it supports extracting *just* the build tree, or reporting whether that directory exists (I'm pretty sure ostree can do these things). Then we simply accept that tar has slightly worse UX, as we're in the process of obsoleting that anyway. | 12:54 |
| Nexus | i quite liked the idea of a cache manifest, it could be very useful in the future | 12:54 |
| tlater | Nexus: If you'd like to go that route, figure out if it will be useful first, but I don't think I can help with that ;) | 12:56 |
| tristan | tlater, it's totally possible to extract just a part of a tarball | 12:58 |
| tristan | tlater, we actually do that in our `tar` plugin | 12:58 |
| tristan | even with wildcard search support | 12:58 |
| tlater | tristan: I thought you still had to first decompress it? | 12:58 |
| tristan | Ah yeah, that at least indeed | 12:59 |
| tristan | but that you can do in memory and avoid needless disk scratching, at least | 12:59 |
| juergbi | I don't think we should do any build tree caching with the tar backend | 13:05 |
| tristan | juergbi, there are two sides of that coin... On the one hand, you are right; we're going to squash both caches with one cache to rule them all so that makes sense | 13:06 |
| tristan | juergbi, on the other hand; if it's going to introduce conditional code paths to the rest of BuildStream, caching the build tree everywhere will be more desirable | 13:07 |
| tristan | even if it's inefficient with the tar backend, it's certainly should not constitute a lot of custom code I think | 13:07 |
| juergbi | right, essentially I would minimize the effort spent with regards to the tar cache | 13:07 |
| tristan | exactly, i.e. it would be fine if for now we just extract everything into the extract/ directory, and not have custom stuff in the artifact caches for it | 13:08 |
| tristan | not sure how much that would slow things down, but it would seem saner to do that first, replace with common artifact cache, and then optimize after | 13:09 |
| juergbi | yes, that could be the simplest route | 13:10 |
| juergbi | Nexus: what would be the content and purpose of the cache manifest? | 13:11 |
| juergbi | we will likely require an external manifest to support incremental non-workspace builds when the element sources have been updated | 13:12 |
| Nexus | just containing a list of everything in the cache | 13:12 |
| juergbi | however, that feature is still quite a bit off | 13:12 |
| juergbi | everything in the artifact cache? what information would be in the manifest that can't easily be retrieved directly from the artifact cache? | 13:13 |
| Nexus | nothing, but you wouldn't have to extract it first, which can be slow | 13:14 |
| juergbi | OSTree should allow listing contents without actually extracting anything | 13:14 |
| juergbi | we probably don't have code for this in buildstream but it would definitely be better to add support for this instead of adding an external manifest that would duplicate information | 13:15 |
| Nexus | if that's possible, then i agree it would be better | 13:15 |
| Nexus | and it would allow me to check if i have a cached build tree before accessing the cache | 13:15 |
| tlater | Nexus: It's most definitely possible, look at the pygobject-introspection docs for OSTree :) | 13:15 |
| tlater | Nexus: https://lazka.github.io/pgi-docs/OSTree-1.0/index.html | 13:16 |
| juergbi | Nexus: you could take a look at tlater's diff() method that already does something like this | 13:16 |
| juergbi | except it uses the information for a particular purpose | 13:16 |
| tlater | It's a bit sad that tar performance will get even worse, but I don't think this is really a use case people on UNIX platforms have anyway. | 13:17 |
| tlater | Or let's say non-Linux for clarity here | 13:18 |
| *** bethw has joined #buildstream | 13:18 | |
| juergbi | if caching build trees is optional in general, people on non-Linux could easily disable it for the time being | 13:18 |
| juergbi | and hopefully we can replace it soon enough | 13:19 |
| gitlab-br-bot | buildstream: issue #360 ("Stop printing unused workspaces warning") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/360 | 13:24 |
| gitlab-br-bot | buildstream: issue #360 ("Stop printing unused workspaces warning") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/360 | 13:24 |
| gitlab-br-bot | buildstream: merge request (tristan/unused-workspace-warning->master: _pipeline.py: Stop printing warning when a build plan does not use some existing workspaces) #400 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/400 | 13:24 |
| *** solid_black_ has joined #buildstream | 13:28 | |
| gitlab-br-bot | buildstream: merge request (tristan/radon->master: .gitlab-ci.yml: Perform some python code analysis with radon) #401 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/401 | 13:29 |
| gitlab-br-bot | buildstream: merge request (tristan/radon->master: .gitlab-ci.yml: Perform some python code analysis with radon) #401 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/401 | 13:31 |
| *** solid_black_ has quit IRC | 13:33 | |
| *** solid_black_ has joined #buildstream | 13:35 | |
| *** solid_black_ has quit IRC | 13:38 | |
| *** solid_black_ has joined #buildstream | 13:39 | |
| *** solid_black_ has quit IRC | 13:48 | |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 13:51 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 13:52 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 13:52 |
| jjardon | tristan: \o/ thanks to you and juergbi :) | 13:53 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 13:53 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 13:54 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 13:56 |
| gitlab-br-bot | buildstream: merge request (tristan/radon->master: .gitlab-ci.yml: Perform some python code analysis with radon) #401 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/401 | 13:57 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: Add coverage badget) #397 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 13:58 |
| *** tristan has quit IRC | 13:59 | |
| *** tristan has joined #buildstream | 14:05 | |
| tristan | ugh, such a bad day for gitlab /o\ | 14:07 |
| tristan | jjardon, in other news; since we were enhancing the pipeline, notice the new job which just landed: https://gitlab.com/BuildStream/buildstream/pipelines/20573071 | 14:08 |
| tristan | juergbi, you can see that Element._update_state() gets the worst marks (an E) in Cyclomatic Complexity ;-) | 14:10 |
| *** solid_black_ has joined #buildstream | 14:11 | |
| tristan | and that we are currently < 10,000 lloc, with 24,638 loc, not bad marks all around | 14:11 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 14:12 |
| gitlab-br-bot | buildstream: merge request (jjardon/test_coverage->master: Add coverage badget) #397 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/397 | 14:31 |
| toscalix | I was wondering if there is any comment to the metrics mail I sent a few days back | 14:56 |
| tlater | No wonder things break if I try to build the same element repeatedly... Curse you, concurrency! | 14:57 |
| Nexus | lol. | 14:59 |
| tristan | toscalix, I've been so tied up, I'll try to respond to that thread, as well as juergbi's remote execution proposal, first thing tomorrow | 15:03 |
| tristan | toscalix, in general I think we should certainly have strategies for observing these metrics; the technical scope of this is more blurry | 15:03 |
| tristan | i.e. we want to avoid growing weird limbs and becoming some kind of CI framework, but we need to have answers for people who want to observe these metrics when using BuildStream in the context of their own CI | 15:04 |
| toscalix | np, if you guys think the general approach is valid, I am willing to participate in how to move it down to execution | 15:04 |
| toscalix | if there are other indicators proposed, I am willing to evaluate them | 15:05 |
| tristan | jjardon, now that CI has run on master once successfully, https://gitlab.com/BuildStream/buildstream/ coverage badge is up to date :) | 15:06 |
| tristan | 84.34% code coverage | 15:07 |
| jjardon[m] | I noticed :) | 15:07 |
| tristan | yeah, it was at 27% because it hadn't run on master yet :) | 15:07 |
| jjardon[m] | I wonder: even the badge is only trackinour coverage job, maybe we want to show the individual coverage in the other jobs as well ? | 15:08 |
| tristan | I dont understand | 15:08 |
| jmac | toscalix: I have some comments which I'll send shortly | 15:08 |
| tristan | Ahhh jjardon[m] you dont understand :) | 15:08 |
| tristan | jjardon[m], the coverage job is what calculates the accumulative coverage of all runs; so it is the right one to show | 15:09 |
| toscalix | jmac: great | 15:09 |
| jjardon | tristan: I know | 15:09 |
| jjardon | https://gitlab.com/BuildStream/buildstream/pipelines/20576676/builds | 15:09 |
| tristan | jjardon[m], for instance, once we implement a BSD platform, I'm not really interested to know what the coverage was separately for each version of BSD under test; and I do *not* want gitlab to try to sum up the differences and think it's correct in assuming a total coverage for the pipeine | 15:10 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:10 |
| jjardon | tristan: I suggesting to show how much coverage in "tests-debian-8" only | 15:10 |
| tristan | jjardon, it's important that we use the coverage.py aggregator to properly aggregate the results | 15:10 |
| jjardon | tristan: ok | 15:10 |
| tristan | Yeah, but I'm not really interested in that statistic; I do want to know that, after running on every platform, how much coverage we got | 15:11 |
| tristan | jjardon, technically speaking, it would be possible to have each job show it's coverage, and that would not be harmful; but in the case of gitlab, it actually *is* harmful | 15:11 |
| tristan | because gitlab bakes it's own average from the collective coverages, so we'd get a distorted number that doesnt come from coverage.py's better aggregation | 15:12 |
| tristan | if that was not the case, I'd just say go ahead and have a blast :) | 15:12 |
| tlater | Ack, that should really be at least a toggleable thing :| | 15:13 |
| tristan | tlater, or rather, the "Coverage for the pipeline" should be something which you should be able to have some control over; like deciding which job provides the total pipeline coverage | 15:17 |
| * tristan was unable to find a tool that transforms radon json output into a pretty html file | 15:17 | |
| tristan | that could have been cool also, instead of just the text results | 15:18 |
| jjardon | tristan: yeah, I will try to investigate if it's possible and if not I will open an issue in gitlab | 15:30 |
| juergbi | jjardon, tristan: there already is https://gitlab.com/gitlab-org/gitlab-ce/issues/19409 | 15:32 |
| jjardon | juergbi: cheers | 15:33 |
| * jjardon subscribes | 15:33 | |
| juergbi | and there is https://gitlab.com/gitlab-org/gitlab-ee/issues/413 for the graph but in -ee... | 15:33 |
| * tristan subscribes | 15:34 | |
| tristan | yeah I dont like this approach of theirs | 15:34 |
| tristan | actually I dont want gitlab to understand coverage and make graphs | 15:34 |
| tristan | I want it to allow me to contribute to it's analytics with whatever data I want, and create my own badges to display on pipelines, etc | 15:35 |
| gitlab-br-bot | buildstream: merge request (ps-update-readme->master: Update README.rst with more detail) #402 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/402 | 15:39 |
| gitlab-br-bot | buildstream: merge request (devcurmudgeon/buildstream-ps-update-readme->master: Update README.rst with more detail) #403 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/403 | 15:41 |
| *** juergbi has quit IRC | 15:43 | |
| *** juergbi has joined #buildstream | 15:45 | |
| tlater | I'd just like to repeat this question here: <jmac> Is there any documentation on workspaces that I'm missing? I can't see any in the main docs | 15:49 |
| tlater | Specifically, do we ever explain what workspaces *are*? | 15:49 |
| * tlater would open an issue but is too afraid he's not browsing the docs right | 15:50 | |
| gitlab-br-bot | buildstream: merge request (328-support-for-downloading-sources-from-mirrors->master: WIP: Resolve "Support for downloading sources from mirrors") #404 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/404 | 15:59 |
| gitlab-br-bot | buildstream: merge request (devcurmudgeon/buildstream-ps-update-readme->master: Update README.rst with more detail) #403 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/403 | 16:10 |
| gitlab-br-bot | buildstream: merge request (jennis/136-expire-remote-cache-artifacts->master: WIP:136 expire remote cache artifacts) #364 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/364 | 16:19 |
| gitlab-br-bot | buildstream: merge request (devcurmudgeon/buildstream-ps-update-readme->master: Update README.rst with more detail) #403 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/403 | 16:19 |
| tristan | tlater, we very well might not; the parts of the documentation we do have are reference material, for which we have near 100% coverage of the yaml format and plugin facing API; we have yet to make proper user facing documentation | 16:21 |
| tristan | tlater, there is a plan for that in place, which I thought jennis was going to take charge of, but it seems he doesnt have the time to invest | 16:21 |
| tlater | Ah, right, I suppose it does fall under the general improvement mantle | 16:22 |
| tristan | tlater, in other words, there is an initial "big effort" which needs to be made for user facing docs, such that we set a proper precedent to follow and can after accept small patches more easily | 16:22 |
| tristan | right now there is not even a designated place for those docs to live, indeed | 16:23 |
| jmac | I find the doc folder a pretty good source for user-facing documentation | 16:24 |
| tristan | jmac, err, my head hurts | 16:29 |
| tristan | jmac, I've shared a google doc with you which explains the general plan | 16:31 |
| tristan | jmac, "dumping a whole lotta stuff into the doc/ dir and just sorting out style, consistency etc later", is not going to be an option | 16:32 |
| jmac | Unfortunately, I think that means I will not be able to contribute at all. | 16:32 |
| tristan | it should be easy once there is some structure in place and at least one example to follow as a template | 16:34 |
| tristan | but dumping a whole lot of chicken scratch into the docs/ dir is getting off on the wrong foot | 16:34 |
| Nexus | Could someone point me at a test that changes a project config var pls? | 16:39 |
| tristan | Nexus, most tests use static project.conf added in the test directories, some of them modify project.conf inline | 16:43 |
| tristan | Nexus, for instance in tests/frontend, they all share a common project / project.conf, and there is configure_project() defined in tests/frontend/__init__.py, which you can grep for in frontend tests | 16:44 |
| Nexus | tristan: i'd like to change a confirg variable mid way through a test, is that possible? | 16:44 |
| *** toscalix has quit IRC | 16:45 | |
| tristan | Nexus, ummm, yeah ? | 16:45 |
| tristan | Nexus, every invocation of BuildStream will re-load the project.conf and elements, so you can test things where project data has changed in between invocations | 16:46 |
| *** noisecell has quit IRC | 16:55 | |
| gitlab-br-bot | buildstream: merge request (328-support-for-downloading-sources-from-mirrors->master: WIP: Resolve "Support for downloading sources from mirrors") #404 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/404 | 16:55 |
| *** dominic has quit IRC | 16:56 | |
| gitlab-br-bot | buildstream: merge request (devcurmudgeon/buildstream-ps-update-readme->master: Update README.rst with more detail) #403 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/403 | 16:58 |
| gitlab-br-bot | buildstream: merge request (jmac/artifact-receive-profiling->master: WIP: Add artifact cache receive profiling domain and instructions.) #383 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/383 | 16:58 |
| gitlab-br-bot | buildstream: merge request (jjardon/doc_readme->master: source/index.rst: Use the README as our intro) #405 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/405 | 17:07 |
| gitlab-br-bot | buildstream: merge request (328-support-for-downloading-sources-from-mirrors->master: WIP: Resolve "Support for downloading sources from mirrors") #404 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/404 | 17:11 |
| gitlab-br-bot | buildstream: merge request (jjardon/doc_readme->master: source/index.rst: Use the README as our intro) #405 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/405 | 17:11 |
| gitlab-br-bot | buildstream: merge request (jjardon/doc_readme->master: source/index.rst: Use the README as our intro) #405 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/405 | 17:12 |
| gitlab-br-bot | buildstream: merge request (jjardon/doc_readme->master: source/index.rst: Use the README as our intro) #405 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/405 | 17:14 |
| gitlab-br-bot | buildstream: merge request (jjardon/doc_readme->master: source/index.rst: Use the README as our intro) #405 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/405 | 17:18 |
| gitlab-br-bot | buildstream: issue #182 ("Closing non-existing workspace") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/182 | 17:22 |
| *** jonathanmaw has quit IRC | 17:39 | |
| *** cs_shadow has joined #buildstream | 18:22 | |
| *** bethw has quit IRC | 18:28 | |
| gitlab-br-bot | buildstream: issue #362 ("Allow workspace commands to work on multiple elements") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/362 | 22:11 |
| gitlab-br-bot | buildstream: merge request (reset-all-workspaces->master: Add option to reset multiple workspaces) #374 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/374 | 22:13 |
| *** zalupik has quit IRC | 23:38 | |
| *** zalupik has joined #buildstream | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
