| gitlab-br-bot | buildstream: merge request (fedora_update_deps_docs->master: Docs: Update the required build packages for fedora based systems.) #542 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/542 | 00:01 |
|---|---|---|
| *** lantw44 has joined #buildstream | 00:06 | |
| *** lantw44 has quit IRC | 00:11 | |
| *** tintou has quit IRC | 00:12 | |
| *** tintou has joined #buildstream | 00:12 | |
| gitlab-br-bot | buildstream: merge request (willsalmon/documentation_dependency->master: Added dependency to the Docs) #534 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/534 | 00:15 |
| *** lantw44 has joined #buildstream | 00:16 | |
| gitlab-br-bot | buildstream: merge request (willsalmon/documentation_dependency->master: Added dependency to the Docs) #534 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/534 | 00:59 |
| *** leopi has joined #buildstream | 02:30 | |
| *** Prince781 has joined #buildstream | 03:00 | |
| *** leopi has quit IRC | 04:04 | |
| *** jsgrant has quit IRC | 04:22 | |
| *** jsgrant has joined #buildstream | 04:23 | |
| *** leopi has joined #buildstream | 04:39 | |
| *** tristan_ has joined #buildstream | 04:42 | |
| *** jsgrant has quit IRC | 05:03 | |
| *** ernestask has joined #buildstream | 05:04 | |
| *** Prince781 has quit IRC | 05:05 | |
| *** Prince781 has joined #buildstream | 05:09 | |
| *** mohan43u has quit IRC | 05:27 | |
| *** mohan43u has joined #buildstream | 05:31 | |
| *** tristan_ has quit IRC | 07:32 | |
| *** Prince781 has quit IRC | 07:46 | |
| *** tpollard has joined #buildstream | 07:53 | |
| *** coldtom has joined #buildstream | 07:56 | |
| *** finn has joined #buildstream | 08:00 | |
| *** thomascoldrick has joined #buildstream | 08:00 | |
| *** bethw has joined #buildstream | 08:01 | |
| *** coldtom has quit IRC | 08:01 | |
| *** finn has quit IRC | 08:06 | |
| *** finn_ has quit IRC | 08:06 | |
| *** finn has joined #buildstream | 08:06 | |
| *** qinusty has joined #buildstream | 08:09 | |
| *** jonathanmaw has joined #buildstream | 08:12 | |
| *** ernestask has quit IRC | 08:42 | |
| gitlab-br-bot | buildstream: merge request (phil/437-junction-tutorial->master: WIP Phil/437 junction tutorial) #550 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/550 | 08:44 |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 08:46 |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: WIP: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 08:56 |
| gitlab-br-bot | buildstream: merge request (tiagogomes/tarball_checkout->master: Add support for creating artifact tarballs) #546 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/546 | 09:07 |
| *** mohan43u has quit IRC | 09:13 | |
| *** mohan43u has joined #buildstream | 09:14 | |
| gitlab-br-bot | buildstream: merge request (phil/437-junction-tutorial->master: WIP Phil/437 junction tutorial) #550 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/550 | 09:18 |
| *** tristan has joined #buildstream | 09:19 | |
| gitlab-br-bot | buildstream: merge request (phil/437-junction-tutorial->master: WIP Phil/437 junction tutorial) #550 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/550 | 09:21 |
| *** CTtpollard has joined #buildstream | 09:22 | |
| tristan | rdale, regarding building gcc with the tiny C compiler, I haven't tried it; theoretically I suppose it can work, but remember that since fairly recently gcc needs to be built with a C++ compiler | 09:22 |
| tristan | which significantly changes the bootstrapping process | 09:22 |
| *** tpollard has quit IRC | 09:22 | |
| *** CTtpollard is now known as tpollard | 09:23 | |
| tristan | (I'm not sure if the tiny c compiler is even a C++ compiler) | 09:24 |
| tiagogomes | One thing I didn't find properly explained in the documentation is the relationship between elements and sources | 09:24 |
| tristan | http://buildstream.gitlab.io/buildstream/format_declaring.html#sources ? | 09:27 |
| tristan | closest I can find | 09:27 |
| tristan | I think the reference needs a bit of a restructuring though, I dont like how the introduction ends up containing everything about the YAML directives | 09:28 |
| tristan | and the whole "Element Basics" section http://buildstream.gitlab.io/buildstream/format_declaring.html#element-basics, while attempting to provide a single all purpose example, ends up being a little redundant with later, more complete explanations of concepts | 09:29 |
| tristan | The sentence at the end: "refer to the Source specific documentation for meaningful attributes for the particular Source", should certainly link to http://buildstream.gitlab.io/buildstream/core_plugins.html#sources | 09:32 |
| tiagogomes | Maybe "At the core of BuildStream is a data model of Elements which are parsed from .bst files in a project directory and configured from a few different sources. | 09:35 |
| tiagogomes | " should be rephrased to give a stronger idea that Elements can contain Sources, and Sources are not usable by their own | 09:35 |
| gitlab-br-bot | buildstream: issue #263 ("Allow checking out an artifact as a 'tar' stream") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/263 | 09:37 |
| gitlab-br-bot | buildstream: merge request (tiagogomes/tarball_checkout->master: Add support for creating artifact tarballs) #546 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/546 | 09:37 |
| rdale | tristan: i'm trying to build gcc-4.5.3 but i need to build gmp, mpc and elf utils with tiny c and i don't think it is possible, but shall carry on trying for a bit | 09:38 |
| tristan | tiagogomes, no objections here :) | 09:38 |
| tristan | rdale, note that if it's a system with gcc that you're after, attempting to bootstrap one with tiny c might not be the most practical approach | 09:39 |
| * tlater thinks using a different base platform is definitely the way to go, having attempted just this | 09:40 | |
| tristan | rdale, i.e. the way we've started out with this in freedesktop-sdk, is to use a more complete runtime for the initial bootstrap but then make it circular | 09:40 |
| paulsherwood | hmmm | 09:41 |
| tiagogomes | tristan when fixing 263 on bst-1.12 as well, is there anything different to do regarding the NEWS file? Or should still have an entry on "buildstream 1.3.1" mirroring what's being done on master | 09:41 |
| tristan | I think freedesktop-sdk project still uses a foreign runtime for the bootstrap but it's beside the point | 09:41 |
| tristan | i.e. they just didnt get around to closing the loop | 09:41 |
| paulsherwood | in other contexts we have a purist argument that we don't want the circular shortcut | 09:41 |
| tristan | rdale, the point is; once you've done the exercise once, you've isolated yourself from anything else and the base runtime need not change too often (until you need to bootstrap something which requires new toolchain features) | 09:42 |
| tristan | paulsherwood, I suppose it's a purist argument from both sides yes, you should be able to always seed the bootstrapping with a different runtime | 09:42 |
| tristan | but if you've completed the loop, you don't require anything external anymore either | 09:43 |
| rdale | yes - baserock bootstraps from a larger gcc based tar ball than the buildstream alpine one - i might try that next | 09:43 |
| tristan | rdale, I do suggest however that you look at the freedesktop-sdk bootstrap instead; because it solves an interesting problem | 09:43 |
| rdale | ah ok, that might be a better choice then | 09:44 |
| tristan | rdale, which is that it separates the part where you first build the initial tooling, and the part when you have to first start using that tooling | 09:44 |
| tristan | which means... you can cross build the first stage to use on a foreign architecture and then continue | 09:44 |
| rdale | right, that's what i want to do certainly | 09:44 |
| paulsherwood | flatmush: ^^ what is your view of this? | 09:47 |
| tristan | paulsherwood, for complete purism in that sense, it makes sense to seed the initial bootstrap with a compiler that is not self hosting, if you can get your hands on one (is tiny C one ?) | 09:47 |
| tristan | it does sound like an interesting exercise, though | 09:47 |
| tristan | even if you can only get off the ground initially with an architecture specific assembler, at least you start with assembly | 09:47 |
| tristan | Nexus, so today my main objective was to unlock this git history thing in whatever way possible; I'm confused though what happened to the issues | 09:49 |
| tristan | Nexus, my understanding was that 455 was the workaround where we just nuke .git/, did you reverse that in the issues ? | 09:50 |
| flatmush | I'm not sure really, I mean you could always build gcc with clang, but I don't know what it gains you | 09:50 |
| * tristan 's suspicion is that this is in the context of provenance, trustability and being able to audit the full build | 09:50 | |
| tristan | flatmush, i.e. the desire would be to ensure that you are not using a compiler with a built-in backdoor somehow | 09:51 |
| tristan | which I think requires you dont use a self hosting compiler | 09:51 |
| jmac | That's right | 09:52 |
| flatmush | it's a difficult scenario for me to entertain really, and if someone was so determined, they could make a tarball of clang that puts a backdoor in gcc | 09:53 |
| flatmush | seems highly theoretical | 09:53 |
| jmac | flatmush: You can get around it | 09:53 |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: WIP: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 09:53 |
| jmac | If you want to do diverse double-compilation you need at least two compilers both capable of building themselves and each other | 09:53 |
| jmac | You would almost certainly want to do this as a test, you wouldn't want to do it to build software normally | 09:54 |
| gitlab-br-bot | buildstream: issue #486 ("Capturing binary output using pytest capsysmodule doesn't work") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/486 | 09:57 |
| tristan | if you had a minimal, well audited C compiler in assembly which you trust, you could roll back to an old enough version of gcc which can build with that; and then use that gcc to compile modern gcc, and recompile itself with that | 09:57 |
| tristan | then, if gcc is a reproducible build, you could compare the results of that crazy build, with a normal one to detect binary changes | 09:58 |
| jmac | Right, the problem is that it's not just the C compiler. Compiler is just the common example, but you can put a self-perpetuating backdoor into the assembler, shell, terminal, hard disc controller, you name it | 09:58 |
| tristan | jmac, it's an interesting question whether it's turtles all the way down or not | 09:59 |
| flatmush | well I'm happy to start writing an assembler in machine code if we want ;) | 09:59 |
| jmac | So writing a compiler in a language other than what it's hosted in doesn't get you that much | 09:59 |
| tristan | With perfect isolation, you might find a way; but it might require rewrites of a lot of the lower levels of the stack | 09:59 |
| Nexus | tristan: it stil just nukes .git, but it also does some other things to allow it to not break things unnecissarily | 10:00 |
| flatmush | I think we're going a bit far down the rabbit hole here, we can't be perfect, how would we determine which level of risk is acceptable for bootstrapping? I guess in the past, baserock considered the 3 stage approach and using tarballs to be good enough. | 10:01 |
| flatmush | My thought is that removing the tarballs is fairly easy, so it's worth doing | 10:01 |
| flatmush | I guess compiling gcc with clang and vice versa is also easy enough to consider? | 10:02 |
| gitlab-br-bot | buildstream: merge request (bst_workspace_open_force_does_nothing->master: _stream.py: Added functionality for workspace open -f) #549 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/549 | 10:02 |
| jmac | flatmush: Yes, that would be a good experiment | 10:03 |
| tristan | Nexus, ok, I was going to do something different with the issues; e.g. I was going to: A.) land and close the workaround where we nuke .git/, B.) Open a regression bug about git describe not working any more (probably with a supporting freedesktop-sdk build failure), C.) Point that new bug back to #376 | 10:03 |
| jmac | I don't know what your goal is here flatmush | 10:04 |
| tristan | Nexus, how do you suggest we proceed, then ? I still want to do things in that order, I mean just with the issues | 10:04 |
| tristan | Nexus, also, I don't see an MR with passing tests, which MR has the workaround to start with ? | 10:05 |
| Nexus | tristan: so #455 will still need fixing, and will still be the first once closed | 10:05 |
| flatmush | jmac: me neither yet, the ultimate goal is to certify a linux distribution and hardware as safe, does the Buildstream project itself have a requirement that's applicable to this? | 10:05 |
| Nexus | there currently isn't one | 10:05 |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: WIP: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 10:05 |
| Nexus | tristan: when i finiush #455, git describe still won't work, there will be either all history or no history, so nothing has changed on that front, but the plan that is currently on the issue avoids things like, having ro rebuild even if you opt to not remove your history | 10:07 |
| tristan | Nexus, that's not what we're going to do, though | 10:08 |
| jmac | flatmush: No, we've got a longstanding desire to support and test for bit-for-bit reproducibility but I don't think anyone is working on it now. | 10:08 |
| tristan | Nexus, the first step is no configuration or anything, just the breaking change of removing all history, unblocking other things while we deal with that separately | 10:08 |
| tristan | Nexus, also, reading through the proposed changes now appearing on #455, it appears that you think we can behave differently based on what minimum format version the project requires | 10:09 |
| tristan | which is also incorrect | 10:09 |
| tristan | For right now, we just need to unlock development on build tree caching, for the git part of this, we have a less developped approach which still needs some more discussion, i.e. juergbi & skullman's scripts do it, but there is the issue that we cannot consider the .git/ directory in cache key calculation | 10:11 |
| *** tpollard has quit IRC | 10:11 | |
| *** tpollard has joined #buildstream | 10:11 | |
| tristan | I should note that allowing this temporary breakage on master is basically the main motivator behind early branching of 1.2 | 10:12 |
| skullman | I thought the point was to allow the regression of the stored git history when we *do* need history, until reduced history was possible. | 10:13 |
| skullman | Wouldn't BuildStream be effectively broken if it can't build its usual systems, since they need history. | 10:14 |
| tristan | skullman, that would imply that creation of artifacts with build trees included would have to be conditional | 10:14 |
| tristan | Having alternative definitions of what an artifact is, is something we really want to avoid | 10:15 |
| skullman | tristan: not as I understood it, just that we'll have awkwardly large artifacts until we have a reduced history option. | 10:15 |
| tristan | skullman, seeing as it amounts to about the same, I'd rather live with broken freedesktop-sdk builds from bst master than carrying over an extra 6GB for every webkit push I think | 10:16 |
| tristan | It will still be important to support git describe for the modules who need it, of course | 10:17 |
| *** Phil has quit IRC | 10:18 | |
| tristan | skullman, in the BoF, I thought we could just go ahead with the script approach; but as we can't calculate it in the cache key as we had discussed, I think the faster way to unblock things is just employ the workaround of nuking .git/ unconditionally | 10:18 |
| tristan | Not to mention, we can unblock things without having to add any option, or figure out how the option works | 10:20 |
| skullman | I can't say I'd consider the work complete if it breaks everything, and pushing an extra 6GB seems like a less worse regression than not being able to build it at all. | 10:21 |
| skullman | My concern is that while it unblocks things it'll cause us more trouble until reduced history is completed. | 10:25 |
| *** Phil has joined #buildstream | 10:27 | |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: WIP: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 10:28 |
| skullman | Mostly I don't want us to be blamed for everything breaking as a result. | 10:28 |
| tiagogomes | I don't understand the NEWS usage on BuildStream. I was expecting to see a "buildstream 1.2" section on both master and bst-1.2, but that section isn't in either | 10:35 |
| tristan | tiagogomes, we have not reached 1.2 yet, that's in... a bit over a month; with feature freeze of 1.2 stable coming up in about a week or two | 10:36 |
| tristan | tiagogomes, the early branching is a special case because of all the on going work which needs to integrate | 10:36 |
| tristan | 1.2 NEWS section will appear with the 1.2 release | 10:37 |
| tristan | skullman, there are a few ways to look at it; one of them is that: Having the full .git/ being a part of a build tree is something we dont want to accept at all for stable 1.4, ensuring that we never go there at all and instead have broken builds is a higher incentive to fixing it properly | 10:39 |
| tristan | the temporary breakage is entirely expected, though | 10:40 |
| *** Prince781 has joined #buildstream | 10:40 | |
| tiagogomes | In master there is "buildstream 1.3.1" for the tarball support, but not 1.2. Which is odd | 10:41 |
| tristan | tiagogomes, good point | 10:42 |
| tristan | tiagogomes, I expect this will happen with a few features we backport to 1.2 branch; we can tidy up the NEWS in 1.2 to reflect this better | 10:42 |
| tristan | we probably need to before the next 1.1.x release | 10:42 |
| tiagogomes | Ok, so should I merge the tarball thing in 1.2 without doing anything different regarding the NEWS file | 10:45 |
| tristan | tiagogomes, since we branched early, I think factual story is that in bst-1.2 branch; the feature was added in 1.1.5 | 10:46 |
| tristan | even if in master, it will have appeared in 1.3.1, it's not a big deal I think, and accurately reflects what happened (a backport to the early branched 1.2 series) | 10:47 |
| *** tpollard has quit IRC | 10:50 | |
| *** tpollard has joined #buildstream | 10:51 | |
| gitlab-br-bot | buildstream: merge request (franred/catch-exception-when-umounting-devices->master: WIP: _sandboxbwrap.py: Catch ENOTCONN when umounting) #539 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/539 | 10:58 |
| gitlab-br-bot | buildstream: merge request (Qinusty/275->master: Indicate where artifacts are going to and coming from in the log) #553 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/553 | 11:01 |
| qinusty | Ready for review @ https://gitlab.com/BuildStream/buildstream/merge_requests/553/ | 11:03 |
| tlater | qinusty is just churning out MRs | 11:05 |
| * tlater is impressed | 11:05 | |
| * qinusty is churning out one MR which didn't go well | 11:05 | |
| tristan | qinusty, nice ! I suppose the INFO lines will show up for every artifact remote which is contacted ? | 11:05 |
| tristan | it has been a pretty confusing living without this :) | 11:05 |
| qinusty | Yes! You get one for each remote. | 11:06 |
| qinusty | Obviously, an individual remote might skip etc | 11:06 |
| qinusty | The question is, does this fix https://gitlab.com/BuildStream/buildstream/issues/481? | 11:06 |
| tristan | qinusty, I dont think it does | 11:07 |
| qinusty | Yeah the whole SUCCESS Push thing, is pretty confusing when it actually just skipped | 11:07 |
| tristan | I mean, there are multiple ways of seeing it of course... currently there are a few things which can "skip", a lot of those things are detected before ever launching any job | 11:08 |
| tristan | so you don't get START/SUCCESS messages for obviously skippable things, but when you do (whether it's for pull or whatever), it seems desirable do 481 | 11:08 |
| qinusty | But from what I can tell, SUCCESS Push comes from MessageType, which only really has START, SUCCESS, FAIL for that type of message | 11:08 |
| tristan | *to do 481 | 11:08 |
| tristan | uhhh | 11:09 |
| tristan | yeah | 11:09 |
| tristan | qinusty, I think enhancing that is what 481 is about, right ? | 11:09 |
| qinusty | It seem so | 11:09 |
| tristan | so one of the activities would be to add a new MessageType I would expect | 11:10 |
| tristan | the brunt of it seems to be to communicate that to the parent Job instance at the right time | 11:10 |
| qinusty | Yup, then theres the task of figuring out when a job actually skipped from within _child_action() | 11:10 |
| tristan | so that the generic Job harness will be able to know more than whether a job failed or succeeded | 11:10 |
| tristan | qinusty, if you've got your mind in that area... I think I have another sister issue for you... | 11:11 |
| qinusty | Yeah, child_process() and related functions will require a slight revamp from True/False | 11:11 |
| qinusty | Sounds good | 11:11 |
| tristan | where is it... | 11:12 |
| qinusty | job.py | 11:13 |
| gitlab-br-bot | buildstream: merge request (sam/fix-debug-crash->master: Fix crash when --debug is passed) #530 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/530 | 11:15 |
| tristan | found it | 11:16 |
| tristan | qinusty, https://gitlab.com/BuildStream/buildstream/issues/397 | 11:16 |
| tristan | qinusty, a nice way to word any feature around that would be... whether a job failed temporarily or permanently | 11:17 |
| jonathanmaw | tristan: Is https://gitlab.com/BuildStream/buildstream/merge_requests/404 still on your radar as something to review? Given the new reviews policy can I find someone else to check it over if you're busy? | 11:17 |
| tristan | jonathanmaw, this week I've mostly been trying to squash the pressing issues causing panic; I will try to get into full review mode after unblocking this build tree fiasco | 11:18 |
| jonathanmaw | okie doke, ta tristan | 11:19 |
| tristan | jonathanmaw, on a similar note, I recall leaving all of the filter element related stuff strictly in juergbi's hands since it was pretty complex and he was already reviewing that | 11:19 |
| tristan | jonathanmaw, I suppose that landed already by now and we can sleep soundly without worrying about filter elements biting us ? ;-) | 11:20 |
| qinusty | Okay tristan, So overhaul the job results to be more specific. This will allow us to solve the retry problem and allow for more descriptive messages when skipping etc | 11:20 |
| jonathanmaw | tristan: yep. | 11:20 |
| tristan | jonathanmaw, great :) | 11:20 |
| tristan | qinusty, basically that issue touches a very similar part of the code yeah; so I thought if you look at one, might as well consider the other | 11:21 |
| qinusty | Alright I'll take a look today | 11:21 |
| tristan | qinusty, for 397, I think we can communicate that through the SourceError and ElementError classes (allowing elements and sources to communicate that to the Job class) | 11:22 |
| tristan | for the SKIP messages, harder to say; maybe it could also use an exception ? | 11:22 |
| tristan | raise Skip("reason why I skipped this job") might be nice API for plugins ? hard to say | 11:23 |
| qinusty | Potentially | 11:24 |
| gitlab-br-bot | buildstream: merge request (tiagogomes/tarball_checkout_1.2->bst-1.2: Add support for creating a tarball on bst checkout) #554 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/554 | 11:29 |
| tristan | juergbi, unfortunately, the regression where test_push_pull fails 100% of the time is not related to the racy thing | 11:41 |
| tristan | juergbi, commit a161c746b4a0774c1799e4c7c0e0bdad1141c437 adds the line "pytest_cov.embed.cleanup_on_sigterm()", which is an AttributeError on the pytest_cov.embed object | 11:41 |
| * tristan thinks requiring a new pytest-cov will fix it, but waits for tests to pass... | 11:43 | |
| tristan | the fact that ArtifactShare.run() (in testutils) crashes in a blocked SIGINT state is also a bit weird | 11:44 |
| qinusty | Hmmm, Would you say retry by default of retry explicitly? | 11:47 |
| tristan | qinusty, I think the wording I suggested for the proposed API for SourceError / ElementError is bad, we probably want the error to identify whether the failure was permanent or temporary | 11:49 |
| tristan | qinusty, then we want our retry logic to only retry things which failed temporarily | 11:49 |
| qinusty | tristan: In that case, which would be the default? Would you want to explicitly declare your failure as temporary? | 11:49 |
| qinusty | or should you have to explicitly declare as permenant | 11:50 |
| gitlab-br-bot | buildstream: merge request (tristan/fix-artifact-tests->master: setup.py: Specify minimum required version of pytest-cov plugin) #555 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/555 | 11:53 |
| tristan | qinusty, yeah... naming things is always a pain... I guess the former; by default errors are permanent | 11:55 |
| tristan | this does constitute a minor API break on the plugin surface, though | 11:55 |
| tristan | which is something else to consider, although seeing the clear lack of many Source implementations outside of BuildStream and the fairly low impact, maybe it's not too bad of a break | 11:56 |
| tristan | qinusty, currently Sources behave in such a way that all failures are temporary; that behavior would be changed basically | 11:56 |
| qinusty | yes | 11:56 |
| qinusty | I agree, on a similar note. ElementError and SourceError are based on BstError. Should they be based on some other Error type which can be Permenant or Temporary? | 11:57 |
| qinusty | I don't think adding a permenant/temporary state to BstError is necessary | 11:57 |
| gitlab-br-bot | buildstream: merge request (tiagogomes/tarball_checkout_1.2->bst-1.2: Add support for creating a tarball on bst checkout) #554 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/554 | 11:59 |
| tristan | qinusty, Probably it makes more sense to have ElementError and SourceError derive from PluginError, which tends to be raised by the common base class | 11:59 |
| tristan | of course it means separate handling of PluginError / BstError in job.py | 12:00 |
| qinusty | except PluginError, BstError as e? | 12:00 |
| qinusty | Currently job.py just handles BstError and a broad Exception (for bugs) | 12:01 |
| tristan | true, then "temporary = isinstance(e, PluginError) and e.temporary" | 12:01 |
| qinusty | Yup | 12:01 |
| qinusty | My thoughts exactly | 12:01 |
| tristan | qinusty, it might be better to put it in BstError and document it as something only considered when a Job is handling the error | 12:07 |
| tristan | qinusty, well, not sure; just keep in mind that generic push/pull tasks can fail temporarily too | 12:07 |
| tristan | qinusty, and if you *do* have it in a separate class, there would be no need for the "except PluginError, BstError as e" part (as PluginError is also a BstError anyway) | 12:08 |
| *** Prince781 has quit IRC | 12:10 | |
| qinusty | Yes, ofcourse. Which leads us to ArtifactError too. | 12:12 |
| qinusty | I'll take a first shot at it after lunch. Is there an easy way to recreate the bad checksums issue? | 12:13 |
| * qinusty realises he can just make up a checksum | 12:13 | |
| gitlab-br-bot | buildstream: merge request (tristan/fix-artifact-tests->master: setup.py: Specify minimum required version of pytest-cov plugin) #555 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/555 | 12:23 |
| *** ernestask has joined #buildstream | 12:39 | |
| gitlab-br-bot | buildstream: merge request (tristan/remove-history-from-build-trees->master: Remove history from build trees) #556 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/556 | 12:49 |
| gitlab-br-bot | buildstream: merge request (reduce_history_in_cache->master: Reduce history in cache) #482 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/482 | 12:53 |
| gitlab-br-bot | buildstream: issue #487 ("Implement a mechanism to allow builds to leverage git describe") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/487 | 13:01 |
| gitlab-br-bot | buildstream: issue #376 ("Sources to only stage what is required for building, omitting large parts of VCS history") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/376 | 13:09 |
| * tiagogomes doesn't understand why there is a filter element when compose seems to do something similar | 13:10 | |
| skullman | difference in defaults IIRC | 13:10 |
| skullman | and intent | 13:11 |
| tristan | Alright, time to get dinner, https://gitlab.com/BuildStream/buildstream/merge_requests/556 will close #455, #376 is also closed, new regression issue to track at https://gitlab.com/BuildStream/buildstream/issues/487 | 13:11 |
| *** tristan has quit IRC | 13:16 | |
| tiagogomes | I can't see the filter element being used on either the freedesktop-sdk or baserock | 13:16 |
| noisecell | I've tested https://gitlab.com/BuildStream/buildstream/merge_requests/530 and it resolves the issue which describes, is ok for me to set it as "merge when pipeline succeeds" | 13:17 |
| noisecell | ? | 13:17 |
| gitlab-br-bot | buildstream: merge request (sam/fix-debug-crash->master: Fix crash when --debug is passed) #530 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/530 | 13:17 |
| gitlab-br-bot | buildstream: merge request (sam/fix-debug-crash->master: Fix crash when --debug is passed) #530 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/530 | 13:17 |
| * noisecell press the button | 13:20 | |
| tiagogomes | I wonder if one of the tests you use `--debug` to catch regressions | 13:21 |
| tlater | Yep, I agree, we should have added a test case for that bug | 13:22 |
| tlater | Although arguable we should *also* have a look at reducing the number of tests | 13:23 |
| tlater | Because our suite is starting to just blow up in terms of test time... | 13:23 |
| noisecell | tlater, maybe breaking them and run them in parallel in the pipeline? | 13:28 |
| tiagogomes | Possibly the integration and not integration tests could run in parallel easily. I have no idea about the gain though | 13:29 |
| tlater | tiagogomes: That might speed up CI, but I'm more concerned about the number of tests devs need to run | 13:31 |
| tlater | And devs already don't run the "integration" tests | 13:31 |
| tlater | (We also should change terminology here, all our tests are technically integration tests, these should be referred to as "tests that require sandboxing") | 13:31 |
| tiagogomes | Indeed. That name is misleading | 13:32 |
| gitlab-br-bot | buildstream: merge request (sam/fix-debug-crash->master: Fix crash when --debug is passed) #530 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/530 | 13:49 |
| paulsherwood | i'm looking at https://gitlab.com/trustable/distros/minimal-distro/-/jobs/83241028 | 14:08 |
| paulsherwood | and wondering, not for the first time... | 14:08 |
| paulsherwood | - why lines are duplicated | 14:09 |
| paulsherwood | [00:00:02][61cc39ce][build:gnu-toolchain/stage2-binutils.bst] SUCCESS Running strip-commands | 14:09 |
| paulsherwood | [00:00:02][61cc39ce][build:gnu-toolchain/stage2-binutils.bst] SUCCESS Running strip-commands | 14:09 |
| paulsherwood | - why all the space given over to empty and zero timeastamps? | 14:09 |
| paulsherwood | [--:--:--][61cc39ce][build:gnu-toolchain/stage2-binutils.bst] START Running strip-commands | 14:09 |
| paulsherwood | [--:--:--][61cc39ce][build:gnu-toolchain/stage2-binutils.bst] START Running strip-commands | 14:09 |
| paulsherwood | - why can't i see how long the whole task has been running | 14:09 |
| paulsherwood | (from the logs) | 14:10 |
| paulsherwood | - why no record of the actual config in the logs? | 14:10 |
| paulsherwood | Configuration File: /root/.config/buildstream.conf | 14:10 |
| paulsherwood | no idea what that contains | 14:11 |
| paulsherwood | actually maybe i should mail the list | 14:12 |
| persia | Issues might be better for some of these things. | 14:12 |
| persia | Definitely for the config (as it is essential for reproduction) | 14:12 |
| persia | Logging issues are likely to be more amenable to discussion: might just be a docs issue. | 14:13 |
| paulsherwood | last time i raised an issue tristan closed it and called my baby ugly :) | 14:14 |
| * paulsherwood won't forget this :-) | 14:14 | |
| persia | Which issue? | 14:14 |
| laurence | maybe add to https://gitlab.com/BuildStream/buildstream/issues/478 | 14:14 |
| persia | Is the issue with the terminal output, or the logs (from bst log and similar)? | 14:15 |
| paulsherwood | the things i'm mentioning are both, i believe | 14:15 |
| tlater | Hm, half and half | 14:16 |
| tlater | Printing the configuration to stdout feels like it would just add *more* clutter to the output | 14:16 |
| tlater | But it would be good to have in logs | 14:16 |
| persia | +1 on config in logs and not output | 14:16 |
| tpollard | has anyone had 'AttributeError: 'AutotoolsElement' object has no attribute '_Element__cache_key'' before? | 14:19 |
| persia | Is there any documentation on how the cache works? Alternately, are there known issues about caching for which reading provides good insight? | 14:21 |
| persia | I'm working on a comparison of BuildStream to some other tooling, and am hitting the limits of my knowledge in this area | 14:21 |
| *** noisecell has quit IRC | 14:22 | |
| tlater | persia: There is https://buildstream.gitlab.io/buildstream/additional_cachekeys.html | 14:25 |
| tlater | Not sure how up to date or how relevant it is | 14:25 |
| tlater | I don't know of any open issues I could link; or at least they're not active memory right now. | 14:26 |
| persia | tlater: My questions are different, like "what sorts of things are cached?" (the answer to which I think depends on the status of the source caching work), when/how is cache expiry triggered, etc. | 14:26 |
| tlater | Yep, I think most of that is documented in comments around relevant code, unfortunately. | 14:28 |
| persia | Heh | 14:28 |
| gitlab-br-bot | buildstream: issue #488 ("Follow-up from "Remove history from build trees"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/488 | 14:28 |
| gitlab-br-bot | buildstream: merge request (tristan/remove-history-from-build-trees->master: Remove history from build trees) #556 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/556 | 14:29 |
| Kinnison | At some point I'd like an answer to the question "How does the CAS anchor things?" but I think I should go rummaging for docs on the CAS on-disk structure first | 14:29 |
| paulsherwood | persia: https://gitlab.com/BuildStream/buildstream/issues/193 | 14:30 |
| persia | Hrm, and no code map in the docs either. Will I likely find most of what I want in buildstream/_artifactcache/ ? | 14:30 |
| jmac | Kinnison: What does 'anchor' mean? | 14:30 |
| tlater | persia: Yes, code in there should detail everything you want to know | 14:30 |
| paulsherwood | i still think that ybd's time logging is better than bst. takes up less space, covers more usecases | 14:30 |
| tlater | If you have specific questions, I'm happy to answer too :) | 14:31 |
| persia | Kinnison: Be careful with that: my impression (which may not be correct) is that CAS only specifies a means to interact with objects, but any implementation ultimately depends on some other backing store (which would itself deal with on-disk layout, if it was an on-disk store). | 14:31 |
| Kinnison | jmac: in this context i'm using it to mean "How does the CAS know what entries are needed and what can be expired" | 14:31 |
| Kinnison | persia: any such backing store would still need some kind of strong concept of anchors, no? | 14:31 |
| jmac | paulsherwood: BST's logging is configurable, although we lack documentation for it, and I don't think it can be configured to the extent of providing a running record of total elapsed time. | 14:32 |
| persia | Kinnison: Depends on the number of abstraction layers between the physical storage medium and the CAS interface. | 14:32 |
| Kinnison | jmac: think like how the refs in a git repository anchor the set of objects which need to be present in the object store, and how git gc can remove all the rest | 14:32 |
| jmac | Kinnison: There is no expiry. | 14:32 |
| Kinnison | jmac: aah, okay, cool | 14:32 |
| jmac | Yet. | 14:32 |
| tlater | jmac: There is as of two days ago! | 14:32 |
| Kinnison | such time travelling code changes! | 14:32 |
| tlater | Haha | 14:33 |
| jmac | Oh! Apologies tlater! | 14:33 |
| paulsherwood | jmac: out-of-the-box experience is more important than config options ime | 14:33 |
| jmac | paulsherwood: Yes, I agree | 14:33 |
| persia | tlater: The four specific questions I'm currently trying to answer are 1) "What sorts of things does BuildStream cache?", 2) "When/how is cache expiry triggered?", 3) "Is there a prioritisation of types for expiry? (e.g. if build results are cached (for incremental build support), are those evicted before binary artifacts used in sandbox construction?)", and 4) Is the cache cleared in a separate thread/process from other operations? | 14:35 |
| tlater | persia: Is this looking at a *remote* caching perspective or local developer machine caching? | 14:35 |
| tlater | There are slight differences between the two, currently | 14:36 |
| persia | tlater: Local. For remote, I think I have to learn a lot more about the remote protocol, etc. to understand the actual questions I want to ask (or the answers to them). | 14:36 |
| Kinnison | persia: My understanding (loose though it is) on 3 is that an 'artifact' consists of all of its cached outputs (to include cached work trees etc) and is considered integral -- i.e. there will be no partial removal | 14:36 |
| tlater | persia: 1) Buildstream stores sources, built artifacts (or the relevant build tree if a build fails) and build logs for each element in a pipeline. | 14:37 |
| persia | Kinnison: Do you know if there is support for partial transport to/from a remote? For example, if I want to compose something, do I need to download the build logs for each element used to construct the sandbox? | 14:37 |
| Kinnison | persia: I'm not sure about that, since I've not yet learned about that side of things, sorry | 14:38 |
| * Kinnison has only been looking at the project for a few days | 14:38 | |
| persia | tlater: What about build trees for successful (but maybe incorrect) builds? | 14:38 |
| tlater | persia: That's WIP still | 14:38 |
| persia | Kinnison: No worries :) Thanks for trying to help answer as you learn. | 14:38 |
| persia | tlater: WIP as in I can look at code on gitlab, or WIP as in "someone intends to get around to that"? | 14:39 |
| tlater | persia: Nexus is currently working on it, it was one of the main driving forces behind our git history removal discussions | 14:39 |
| tlater | I *think* we're a week or so away from caching those trees on the unstable branch | 14:40 |
| persia | Oh, I thought both success and failure trees were stuck on that, but happy to have my understanding corrected. | 14:40 |
| tlater | persia: Remotely we don't currently cache either | 14:40 |
| tlater | For remote caching both are blocking :) | 14:40 |
| tlater | 2) Cache expiry is triggered when buildstream runs out of disk quota - this is configurable, and the entire disk space by default. There is some fuzziness about what "running out" is - we try to be sensible here. | 14:41 |
| Nexus | did someone mention buildtrees? | 14:41 |
| persia | Which addresses a good chunk of (3): I suppose I need to watch the proposal queue to get the answer to that (over time). | 14:41 |
| persia | Nexus: Yes, but incidentally, as part of a list of the things BuildStream caches (or intends to cache). | 14:41 |
| tlater | Yep, we currently just remove any old data | 14:41 |
| * Nexus goes back into iding until needed | 14:42 | |
| Nexus | hiding* | 14:42 |
| persia | For (2), is there also a user interface for "clear cache now" or docs on how to use rm(1) to achieve that? | 14:42 |
| tlater | Also, 4, the cache is indeed cleared in a separate process. For atomicity it is in practice usually synchronous with any other build processes, however. | 14:42 |
| persia | tlater: synchronous is a tricky word: do builds block on cache expiry completion? | 14:43 |
| tlater | persia: Yes | 14:44 |
| tlater | *But* only build/pull/push operations | 14:44 |
| tlater | Track/fetch don't block | 14:44 |
| tlater | In practice that still means usually everything blocks | 14:44 |
| tlater | We do not currently have a UI for cache clearing. To manually remove cached artifacts with fine granularity is quite difficult, there is UI planned for this but not yet being worked on. To remove everything `rm -rf ~/.cache/buildstream/artifacts` works. | 14:45 |
| persia | Aha, so the model is that commands that depend on cache state need the cache to have a consistent state from the time they check cache until they complete, but other commands may operate at any time. | 14:45 |
| persia | And "~/.cache/buildstream/artifacts" is stored in config somewhere discoverable? | 14:45 |
| tlater | persia: That's almost correct substitute "commands" with "operations" - any command may fetch, after all :) | 14:45 |
| tlater | persia: Yep, user config defines that directory, that's just the default | 14:46 |
| persia | Yes. "operations" is a better word for that. Thanks for the correction. | 14:46 |
| persia | tlater: Thanks a lot. That answers my questions for today. | 14:46 |
| tlater | np - we should get documentation about this up at some point, lots of what you asked about is still very very fresh feature wise | 14:47 |
| persia | That's part of why I'm not asking about remote caches yet :) I know I don't know enough about the changing landscape, and I'm not even very confident of whether current answers will remain unchanged for the next couple weeks. | 14:49 |
| persia | I do remember hearing about a good chance of stability there in the next couple weeks at GUADEC, so I'll probably return with more questions after reviewing the schedule, etc. | 14:50 |
| gitlab-br-bot | buildstream: issue #455 ("Remove VCS History") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/455 | 14:52 |
| gitlab-br-bot | buildstream: merge request (tristan/remove-history-from-build-trees->master: Remove history from build trees) #556 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/556 | 14:52 |
| gitlab-br-bot | buildstream: merge request (caching_build_trees->master: Caching buildtrees) #474 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/474 | 14:57 |
| qinusty | Anyone know the max line length for our pylint config? | 15:15 |
| Phil | 120 chars I believe | 15:15 |
| qinusty | <120 or <=120? | 15:16 |
| tlater | qinusty: Should be <= | 15:16 |
| tlater | The actual configuration is 121 | 15:17 |
| qinusty | Cutting it close with an inline comment :/ | 15:17 |
| tlater | ;) | 15:17 |
| tiagogomes | That looks quite high | 15:17 |
| tlater | tiagogomes: We do have a longer length than pep suggests | 15:17 |
| tlater | qinusty: For such a long inline comment, just put it on a separate line and line wrap it | 15:17 |
| qinusty | That is a better idea than trying to figure out which words I don't need | 15:18 |
| * tlater admittedly also often falls for that trap ;) | 15:18 | |
| qinusty | comment golf best golf | 15:19 |
| tpollard | heh vscode defaults to 119, presumably as it's github review width | 15:23 |
| tlater | Hm, odd, I expected IDEs to respect our linter configuration | 15:24 |
| *** leopi has quit IRC | 15:25 | |
| *** jennis2 has joined #buildstream | 15:57 | |
| *** rdale has quit IRC | 15:58 | |
| *** jennis has quit IRC | 15:59 | |
| *** finn has quit IRC | 15:59 | |
| *** jennis2 has quit IRC | 16:21 | |
| *** tpollard has quit IRC | 16:40 | |
| *** jennis2 has joined #buildstream | 16:52 | |
| *** qinusty has quit IRC | 16:52 | |
| *** jennis2 has quit IRC | 16:53 | |
| *** thomascoldrick has quit IRC | 16:56 | |
| *** jonathanmaw has quit IRC | 17:03 | |
| *** toscalix has joined #buildstream | 17:03 | |
| jjardon | Hi, trying to upgrade to 1.1.4 I keep getting this error: https://gitlab.gnome.org/GNOME/gnome-build-meta/-/jobs/67576 | 17:07 |
| jjardon | (I've never seen that before with 1.1.3) | 17:08 |
| *** toscalix has quit IRC | 17:08 | |
| jjardon | "GPGME: Too many open files" I think someone had the same problem the other day? | 17:09 |
| *** toscalix has joined #buildstream | 17:13 | |
| *** bethw has quit IRC | 17:16 | |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 17:21 |
| *** tiagogomes has quit IRC | 17:34 | |
| *** xjuan has joined #buildstream | 17:39 | |
| *** Prince781 has joined #buildstream | 18:06 | |
| *** Prince781 has quit IRC | 18:08 | |
| *** Prince781 has joined #buildstream | 18:24 | |
| *** toscalix has quit IRC | 18:34 | |
| *** Prince781 has quit IRC | 19:13 | |
| *** ernestask has quit IRC | 19:46 | |
| *** xjuan has quit IRC | 20:25 | |
| *** mohan43u has quit IRC | 20:35 | |
| *** mohan43u has joined #buildstream | 20:39 | |
| *** bochecha has quit IRC | 20:50 | |
| gitlab-br-bot | buildstream: issue #489 ("Install instructions don't mention how to get 'grpc'") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/489 | 22:13 |
| *** bethw has joined #buildstream | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!