| *** valentind has quit IRC | 00:14 | |
| *** brlogger has joined #buildstream | 04:28 | |
| *** tristan has quit IRC | 05:44 | |
| *** tristan has joined #buildstream | 05:54 | |
| *** semanticdesign_ has joined #buildstream | 06:33 | |
| *** semanticdesign has joined #buildstream | 06:33 | |
| *** adds68 has joined #buildstream | 06:47 | |
| *** jude has joined #buildstream | 06:49 | |
| *** adds68 has quit IRC | 06:52 | |
| *** bochecha has joined #buildstream | 07:06 | |
| *** igor has joined #buildstream | 07:24 | |
| *** igor has quit IRC | 07:25 | |
| *** adds68 has joined #buildstream | 07:48 | |
| *** adds68_ has joined #buildstream | 08:03 | |
| *** adds68 has quit IRC | 08:03 | |
| *** gitlab-br-bot has joined #buildstream | 08:19 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: _pipeline.py: Improve error message when --except fails.) https://gitlab.com/BuildStream/buildstream/commit/640e7e57842ca4c2ec3a8641258f54ff7c42d3c0 | 08:24 |
|---|---|---|
| tristan | hrmmm, no more notifications when issue state changes ? | 08:27 |
| gitlab-br-bot | buildstream: issue #132 ("Loading external plugins works without explicit requirement in project.conf") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/132 | 08:34 |
| tristan | Aha, there we are | 08:36 |
| gitlab-br-bot | push on buildstream@jonathan/external-plugins (by Tristan Van Berkom): 8 commits (last: Issue #124: Add test for staging to element build directory) https://gitlab.com/BuildStream/buildstream/commit/35d7051a07451e0647f60963d0485908b1d0139a | 08:37 |
| tristan | Ah I see, gitlab-br-bot lost it's connection for a moment | 08:37 |
| gitlab-br-bot | buildstream: merge request (jonathan/external-plugins->master: Remove and refer to elements that have been moved to bst-external) #122 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/122 | 08:37 |
| tristan | now if only we can get it to stop notifying about pushes to branches that are not protected branches | 08:37 |
| *** adds68__ has joined #buildstream | 08:40 | |
| *** adds68_ has quit IRC | 08:41 | |
| *** valentind has joined #buildstream | 08:41 | |
| *** jonathanmaw has joined #buildstream | 08:59 | |
| *** ssam2 has joined #buildstream | 09:07 | |
| *** tlater has joined #buildstream | 09:08 | |
| *** adds68__ has quit IRC | 09:36 | |
| *** adds68__ has joined #buildstream | 09:36 | |
| tlater | tristan: For the tracking changes, do you think it would be sensible to have a switch such as `--track-only-inconsistent` or somesuch, to only track elements that don't have a ref? It would work like `--track-save`. | 09:49 |
| tlater | This is brought up by juergbi in one of the issues, your response was to have tracking domains but I don't think they quite cover the same use case | 09:50 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: Remove dpkg and x86image elements) https://gitlab.com/BuildStream/buildstream/commit/47bf951c2ade22d1bc2120c49a732345b05f26d0 | 09:51 |
| gitlab-br-bot | buildstream: merge request (jonathan/external-plugins->master: Remove and refer to elements that have been moved to bst-external) #122 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/122 | 09:51 |
| gitlab-br-bot | buildstream: Jonathan Maw deleted branch jonathan/external-plugins | 09:51 |
| tristan | tlater, can we close https://gitlab.com/BuildStream/buildstream/issues/107 please, before talking about other stuff ? | 09:55 |
| *** adds68__ has quit IRC | 09:56 | |
| tristan | this is a game of backgammon | 09:56 |
| tristan | last things first :) | 09:56 |
| tlater | Alright, sure :) | 09:56 |
| tlater | tristan: Alright, #107 looks to be completely independent from #128 - going back to the commit before what caused #107 I still have the issues from #128. | 10:10 |
| tlater | However, when #128 happens, SIGINT is ignored now, since the ticker is required for python to respond to that - apparently that's a python bug. | 10:11 |
| tlater | So with that branch it's currently impossible to interrupt the tracking stage - though it seems that that has been the case for a few months now. | 10:12 |
| *** adds68 has joined #buildstream | 10:14 | |
| *** valentind has quit IRC | 10:15 | |
| jonathanmaw | tristan: I'm looking at #97 (pep3102), iiuc, the methods that need to be altered are all the public ones in buildstream/*.py that don't start with underscores? | 10:16 |
| jonathanmaw | plus finding every use of that method and making sure it's calling the positional args correctly. | 10:21 |
| jonathanmaw | Is any arg that sets a default meant to be a keyword option? | 10:22 |
| jonathanmaw | e.g. Context has load(self, config=None). is config a keyword argument, or a positional arg with a default? | 10:23 |
| tristan | jonathanmaw, there will be some decision making in the process | 10:24 |
| tristan | jonathanmaw, so how about the bugs I filed in bst-external before moving on ? | 10:25 |
| tristan | jonathanmaw, looks like we have to be careful, it might just be that your CI was only passing because the elements are not yet removed from buildstream ? | 10:25 |
| jonathanmaw | tristan: ah, ok. I hadn't noticed. | 10:25 |
| tristan | also https://gitlab.com/BuildStream/buildstream/issues/132 | 10:26 |
| jonathanmaw | tristan: I ran the tests locally, too. I'll run them locally again just to be sure that I was using a version of buildstream without them internally. | 10:26 |
| tristan | jonathanmaw, ^^^ that one I filed in buildstream, I cant say for sure how it's possible that bst-external integration tests work, and maybe it needs a fix in buildstream | 10:26 |
| tristan | It should not be possible without any mention in project.conf I think | 10:26 |
| *** adds68 has quit IRC | 10:27 | |
| tristan | I mean, it should *explicitly* fail, because we're not telling buildstream how to look for the plugins, they dont magically appear out of thin air, if they do; it seems we're picking up random stuff; which is bad | 10:27 |
| *** adds68 has joined #buildstream | 10:29 | |
| tristan | Sigh, so the debootstrap base image for building GNOME is *HUGE* now | 10:35 |
| tristan | I think I better change https://gitlab.com/BuildStream/gnome-modulesets-base/blob/master/elements/base/base-configure.bst | 10:36 |
| tristan | To add `rm -rf a /whole/lot of/crap` | 10:36 |
| tristan | then make sure it's cached on the artifact server, and hopefully reduce the initial download size by > 50% | 10:37 |
| * tristan looks briefly at what is #107 and #128 | 10:39 | |
| tristan | tlater, OK I dont understand; one issue at a time: 107 is the freaking important one causing me nightmares | 10:40 |
| tristan | 128 is just a silly ticker | 10:41 |
| tlater | tristan: Mostly, but #128 does get worse if the branch for #107 is merged | 10:41 |
| tristan | tlater, can we fix 107 without making buildstream ignore SIGINT ? | 10:41 |
| tristan | Eh ? | 10:41 |
| tlater | tristan: The SIGINT ignoring specifically happens when the ticker doesn't respond anymore | 10:42 |
| tristan | Ok that doesnt mean much to me... you mean 'tickers tick even less' ? | 10:42 |
| tlater | tristan: No, only in the case that #128 happens does python start ignoring SIGINT. | 10:42 |
| tlater | Because the ticker wakes up the main thread for a little bit, which is enough time for it to handle the SIGINT | 10:42 |
| tristan | Ok well, I expect this means we have to order the unregistering and re-registering of event loop handlers for SIGINT in the proper way while suspended, right ? | 10:43 |
| tristan | Regarding 128 | 10:43 |
| tristan | Anyway; is #107 a sane proper fix ? does a fix exist ? | 10:44 |
| tlater | Yes, I'll put up the MR for it | 10:44 |
| gitlab-br-bot | buildstream: merge request (107-failing-child-processes-when-tracking-missing-git-branches->master: Issue #107: Stop attaching the child watcher multiple times) #124 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/124 | 10:45 |
| tristan | juergbi, any comments on that change (https://gitlab.com/BuildStream/buildstream/merge_requests/124) before merging ? | 10:46 |
| tristan | juergbi, we set (and leak ?) an event loop with every invocation of running the scheduler ? | 10:47 |
| juergbi | tristan: what do you mean by leaking? | 10:48 |
| tristan | maybe we're not | 10:48 |
| tristan | it seems weird to "set the default event loop" | 10:48 |
| tristan | but eh | 10:48 |
| juergbi | the first time there shouldn't be a default loop yet, so we just explicitly create one instead of implicitly | 10:48 |
| tristan | This might not be only for tests, too | 10:49 |
| juergbi | in the multi-test scenario, we replace the loop but the old one should get destroyed using regular ref counting | 10:49 |
| tristan | right, Ok I hope so | 10:49 |
| tlater | tristan: There's no reference to a 'delete' function for these, so presumably asyncio is smart enough to clean them | 10:49 |
| juergbi | as we're not interested in keeping multiple event loops alive at the same time, i consider this the better approach instead of explicitly attaching to a non-default loop everywhere | 10:50 |
| gitlab-br-bot | push on buildstream@107-failing-child-processes-when-tracking-missing-git-branches (by Tristan Van Berkom): 5 commits (last: _pipeline.py: Improve error message when --except fails.) https://gitlab.com/BuildStream/buildstream/commit/640e7e57842ca4c2ec3a8641258f54ff7c42d3c0 | 10:50 |
| juergbi | btw: new_event_loop docu even mentions "You must call set_event_loop() to make this the current event loop" | 10:50 |
| juergbi | i.e., this isn't considered odd use of new_event_loop | 10:50 |
| tristan | so probably it's keeping it alive, and next time we re-enter scheduler.run() is when garbage collection happens on the old loop | 10:51 |
| tristan | or, a set_event_loop(None) would do it in advance, but I dont care | 10:51 |
| juergbi | yes, not sure whether that's supported. if it is, we could definitely consider it | 10:51 |
| juergbi | i don't expect any practical difference, though | 10:52 |
| tlater | set_event_loop(None) just restores the original one, actually. | 10:52 |
| tristan | Rather, what is more delicate, and probably needs consideration separately in #124, is that tearing down scheduler.run() needs to be perfect | 10:52 |
| tristan | If we have event sources and such registered, that will cause weird side effects if it keeps the old loop alive | 10:52 |
| juergbi | if someone were to run the loop again, possibly | 10:53 |
| gitlab-br-bot | buildstream: merge request (107-failing-child-processes-when-tracking-missing-git-branches->master: Issue #107: Stop attaching the child watcher multiple times) #124 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/124 | 10:53 |
| gitlab-br-bot | buildstream: Tristan Van Berkom created branch 107-failing-child-processes-when-tracking-missing-git-branches | 10:53 |
| juergbi | i agree, we should aim to properly tear everything down | 10:53 |
| juergbi | however, i also don't expect any practical issues without it with the current code structure | 10:54 |
| tristan | I can foresee, possibly with your recursive pipeline stuff... practical use cases for invoking a scheduler twice in the same `bst command ...` | 10:54 |
| tristan | Also, fwiw; there are code paths which are known to run in child tasks, and code paths which run in the main task | 10:55 |
| gitlab-br-bot | buildstream: merge request (107-failing-child-processes-when-tracking-missing-git-branches->master: Issue #107: Stop attaching the child watcher multiple times) #124 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/124 | 10:55 |
| tristan | While it is *possible* that we call directly some code that normally runs in a child task... in the main process, I want to avoid that completely | 10:55 |
| tristan | I.e. I would hope to consider calling Source.something_that_runs_scheduled() directly in the main thread as a bug | 10:56 |
| tristan | Because; it doubles our calling context and bug surface (which is another reason to make sure we run the scheduler for anything, which might mean running it twice or more) | 10:57 |
| juergbi | we do call loop.close(), not sure whether that cleans up everything | 10:57 |
| *** adds68 has quit IRC | 10:57 | |
| juergbi | yes, we have to be careful about the main process (e.g., must never be multi-threaded) | 10:58 |
| tristan | almost 1hrs in... debootstrap base still only at 64%... bad | 10:59 |
| gitlab-br-bot | buildstream: issue #133 ("Bash completions dont complete `--except` arguments") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/133 | 11:05 |
| *** adds68 has joined #buildstream | 11:20 | |
| *** adds68 has joined #buildstream | 11:21 | |
| tristan | ltu, do we still have this infamous intel machine to run builds on ? | 11:22 |
| tlater | tristan: It's next to me | 11:31 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Issue #107: Stop attaching the child watcher multiple times) https://gitlab.com/BuildStream/buildstream/commit/6976ba5d8c222aee61bfd5666b5df1b07f76b9bd | 11:31 |
| gitlab-br-bot | buildstream: issue #107 ("Failing child processes when tracking missing git branches") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/107 | 11:31 |
| gitlab-br-bot | buildstream: merge request (107-failing-child-processes-when-tracking-missing-git-branches->master: Issue #107: Stop attaching the child watcher multiple times) #124 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/124 | 11:32 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch 107-failing-child-processes-when-tracking-missing-git-branches | 11:32 |
| *** tlater` has joined #buildstream | 11:33 | |
| *** tlater has quit IRC | 11:34 | |
| *** tlater` is now known as tlater | 11:40 | |
| tristan | grrr, looks like we're running that code which downloads the artifact summary unconditionally | 11:41 |
| tristan | so obnoxious pause when merely running `bst shell` | 11:41 |
| tristan | or things which I think, dont really need to know that | 11:41 |
| tristan | hmmm what is /usr/share/texlive ? | 11:54 |
| tristan | 213MB worth nuking from existence ? | 11:54 |
| tlater | tristan: things required for latex processing :) | 11:57 |
| * tlater wonders why writing to sys.stdout gives a BrokenPipeError | 11:59 | |
| tristan | crap | 12:00 |
| gitlab-br-bot | buildstream: Jonathan Maw created branch 132-loading-external-plugins-works-without-explicit-requirement-in-project-conf | 12:00 |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: WIP: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 12:00 |
| tristan | thats too much | 12:00 |
| tristan | 213MB for generating docs ?! | 12:00 |
| * tristan is grasping for a quick fix to reduce the size of this 3GB download | 12:00 | |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: WIP: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 12:01 |
| tlater | tristan: tbf, most of those are latex addons that are only required for very specific scientific domains. Little of it is actually useful for everybody installing the package. | 12:02 |
| ssam2 | tristan, removing locales is a big win, if you don't care about i18n working | 12:02 |
| *** adds68 has quit IRC | 12:03 | |
| tristan | ssam2, 158M | 12:03 |
| ssam2 | ah, not as big as i thought | 12:04 |
| tristan | ssam2, /usr/share/doc is 294M | 12:04 |
| tristan | the above is /usr/share/locale | 12:04 |
| tristan | 49MB of /usr/share/icons | 12:04 |
| tristan | and 24MB of /usr/share/man | 12:04 |
| tristan | only 2.3MB of /usr/share/help | 12:04 |
| tristan | All of that is safe to remove | 12:04 |
| tristan | what is poppler ? | 12:05 |
| ssam2 | pdf viewer | 12:05 |
| tristan | hrm | 12:05 |
| ssam2 | library used by evince i think | 12:05 |
| tristan | it's reached by some GNOME sysdep | 12:05 |
| bochecha | ssam2: you think right | 12:06 |
| tristan | but I wonder if the sysdep requires the 12MB of stuff in /usr/share/poppler, or just the rest of it | 12:06 |
| bochecha | tristan: might be needed by something like gtk-doc to build pdf docs ? | 12:06 |
| tristan | maybe | 12:06 |
| tristan | I would really like to get rid of some of this 219MB of /usr/share/texlive | 12:07 |
| tristan | if only I knew which part of it we dont need :-S | 12:07 |
| bochecha | all of it /o\ | 12:08 |
| bochecha | do people really build the docs? | 12:08 |
| tristan | bochecha, well yeah | 12:09 |
| tristan | they do | 12:09 |
| tristan | We dont need any docs in the buildstream build env for GNOME, but we need GNOME modules to be able to at least assert that they successfully build their docs | 12:09 |
| bochecha | ok, first thing I do when I build flatpaks/jhbuild is to disable the docs for everything :) | 12:09 |
| tristan | yeah, and then you probably take care of it at distcheck time - when things break | 12:10 |
| tristan | one thing we certainly do, is if you're going to build GNOME, even without --enable-gtk-doc on your favorite modules, is build gtk-doc itself :) | 12:11 |
| tristan | once you do that; it seems like a no brainer that you can build docs | 12:11 |
| bochecha | well then you'll have to keep those 219MB of texlive :) | 12:12 |
| tristan | uhg | 12:12 |
| bochecha | tex stuff is generally a pain, it's super hard to figure out what you need | 12:12 |
| bochecha | and when you're missing a module, the error messages are awful, not telling you "you're missing $module" but failing much later on what it thinks is a syntax error | 12:13 |
| tristan | Well - the only redeeming thing to all of this is, as with the original ostree source of the debian image, the artifact downloads is also ostree | 12:13 |
| bochecha | unfortunately, installing it all is the only way to keep your sanity | 12:13 |
| tristan | So, yeah you need the disk space, and you need to spend that time downloading something once | 12:13 |
| tristan | But, when we add dependencies and something changes; the download is only a delta | 12:13 |
| bochecha | I mean, look at the texlive spec file for the Fedora packages: https://src.fedoraproject.org/rpms/texlive/blob/master/f/texlive.spec | 12:14 |
| bochecha | (yes, it takes time to display that page, the spec file is... huge) | 12:14 |
| tristan | haha | 12:15 |
| tristan | always nice to see the warts | 12:15 |
| tristan | bochecha, at least it aint tizen ;-) | 12:15 |
| bochecha | it's actually generated from a template, by an ad-hoc tool written in C (and maintained in the same repo) | 12:16 |
| bochecha | the Fedora developer who made that thing quit his job at Red Hat shortly after, seemingly disappearing from the community... I hope he's alright :-/ | 12:16 |
| bochecha | are those 219MB really worth it? :) | 12:17 |
| tristan | I'm also not sure if we really need the 27MB in /usr/share/gir-1.0 | 12:18 |
| tristan | but I wont dare delete that | 12:18 |
| tristan | (I mean, we do build gobject-introspection and depend on the girs we generate, so it would be a mistake I think to pickup stuff from there, but... meh) | 12:19 |
| *** xjuan has joined #buildstream | 12:22 | |
| bochecha | one thing which can shave a few tens of MB sometimes is all the static libs: most of them can be safely removed if you always build shared objects (a few actually are needed no matter what) | 12:25 |
| *** cs_shadow_ has quit IRC | 12:29 | |
| tristan | bochecha, aha ! good point | 12:32 |
| *** adds68 has joined #buildstream | 12:32 | |
| tristan | I wonder what those would be though... lemme at least check how much all the .a files cost | 12:32 |
| tristan | and the .la files if they exist, they could all go | 12:32 |
| bochecha | even better is to not build those unneeded static libs at all, when the build system allows it (I'm looking at you, Boost) | 12:32 |
| bochecha | oh yeah, .la files don't take much space, but they can be such a pain | 12:33 |
| tristan | yeah, but we wont have the pain of .la files with buildstream | 12:35 |
| tristan | but if debian installed some, we can safely remove | 12:35 |
| *** xjuan has quit IRC | 12:35 | |
| tristan | hmmm: du -hs `cat list_of_files.txt` doesnt give me a summary... is there a way ? | 12:40 |
| tristan | --total | 12:42 |
| tristan | 582MB of static libs ! | 12:42 |
| tristan | https://bpaste.net/show/2582abbf0258 hard to say which ones I can remove | 12:43 |
| bochecha | only a 6th of the total, that's nothing :) | 12:43 |
| bochecha | the only one I found was really necessary was the libgcc* ones | 12:43 |
| tristan | I already removed ~500MB | 12:44 |
| *** xjuan has joined #buildstream | 12:44 | |
| tristan | So maybe together, brings 3GB down to around 2GB | 12:44 |
| tristan | bochecha, I'm skeptical about removing all the LLVM stuff in that list | 12:44 |
| tristan | I suspect it's similar to the gcc thingies, and the compiler will use them | 12:44 |
| bochecha | oh, I've never done anything with llvm, so I have no idea whether they are needed or not | 12:45 |
| bochecha | yeah, they might very well be similar to the gcc ones | 12:45 |
| tristan | also anything libc, means that maybe some intentionally statically linked stuff might not work | 12:45 |
| tristan | maybe | 12:45 |
| tristan | like libm.a ? | 12:45 |
| tristan | ehhhh | 12:45 |
| *** adds68 has quit IRC | 12:59 | |
| *** adds68 has joined #buildstream | 13:01 | |
| tristan | tlater, so weird thing, I can confirm that with the timer ticker bug, it's not happening after continue of a failed build | 13:11 |
| tristan | Very strange that this would be particular to `bst track` | 13:11 |
| tristan | strange and... telling | 13:12 |
| tlater | tristan: I've considered it being part of the trackingqueue | 13:12 |
| tristan | Or ostree | 13:12 |
| tlater | However uncommenting everything except the basic functionality doesn't change the behavior. | 13:12 |
| tlater | ostree? | 13:12 |
| tristan | ostree does some freaky things to your process | 13:12 |
| tlater | That would make a surprising amount of sense | 13:12 |
| tristan | Which is why we *never* use it in the main thread, if we do; that's gotta go | 13:13 |
| tristan | s/thread/process | 13:13 |
| * tlater will take a peek, this might be only on ostree elements failing to track | 13:13 | |
| tristan | except the tracking is done in a child process, so it logically should not effect the parent | 13:13 |
| tristan | but - something smells like dirtying the process environment at track time | 13:14 |
| tristan | something running in process and messing with signal state ? | 13:14 |
| tlater | tristan: So, this only happens when there is an ostree element in the pipeline that attempts to retry its tracking | 13:20 |
| tristan | tlater, I did not confirm that, did you ? | 13:20 |
| tlater | tristan: I just did, you can try if you want to | 13:21 |
| tristan | I'm just clarifying, I believe you :) | 13:21 |
| jonathanmaw | hrm, is anyone else having trouble pushing to repos in the buildstream project? | 13:32 |
| * tristan hasnt had trouble today | 13:32 | |
| jonathanmaw | when I'm trying to push to bst-external, I get "GitLab: API is not accessible. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists." | 13:32 |
| tristan | but... this is gitlab.com we're talking about | 13:32 |
| jonathanmaw | when trying to push the branch "1-coverage-report-not-combined" | 13:33 |
| tristan | I assume, you have been able to push before | 13:33 |
| tristan | and it suddenly doesnt work | 13:33 |
| tristan | Which would mean... try again in a couple hours, it'll probably work, cause it's gitlab.com ? | 13:34 |
| jonathanmaw | tristan: I've been able to push to master | 13:35 |
| jonathanmaw | I'll try pushing to a new branch, and check whether that branch has become protected somehow | 13:36 |
| tristan | API is not accessible to me means, similar to the error we see on the web interface at times | 13:37 |
| tristan | which means to me; just dont expect reliability from an unreliable service | 13:37 |
| tristan | tlater, is this something that is fixed by a branch you created: https://gitlab.com/BuildStream/buildstream/issues/48 ? | 13:42 |
| tlater | tristan: No, it's not fixed yet | 13:43 |
| tristan | ah | 13:43 |
| * tlater got different issues whenever I sat down to do that one, but occasionally touched it. | 13:44 | |
| tlater | tristan: Interestingly, `bst track` doesn't have the problems `bst build --track` has. It does, however, take significantly longer to terminate after an ostree element has failed. | 13:45 |
| tlater | I'm kind of stuck trying to find something evil ostree does though :/ | 13:45 |
| tristan | nah, you wont find the evil thing that ostree does | 13:47 |
| tristan | leave that to colin | 13:47 |
| tristan | if ever | 13:47 |
| *** jude has quit IRC | 14:00 | |
| tristan | artifact server back online \o/ ! | 14:06 |
| tristan | well, push-wise, pulling was working this week I think | 14:07 |
| tristan | excepting that expired cert, which is also fixed | 14:07 |
| tristan | wait a sec... is there a bug here ? | 14:07 |
| tristan | So... we had an expired cert... and when trying `bst track base/base.bst`... it was giving me an error about the cert | 14:08 |
| tristan | But... when we download the summary... I didnt get an error of even any warning ! | 14:08 |
| tristan | jaysus, I keep finding bugs | 14:09 |
| * tlater is developing a hatred for multiprocessing | 14:11 | |
| *** cs_shadow_ has joined #buildstream | 14:18 | |
| tristan | tlater, what we're doing is not supposed to be easy - add to that the python distortion in between, and it's not gonna be fun | 14:23 |
| *** adds68_ has joined #buildstream | 14:32 | |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 14:32 |
| *** adds68 has quit IRC | 14:33 | |
| *** tristan has quit IRC | 14:35 | |
| *** jjardon has joined #buildstream | 14:36 | |
| *** jjardon has left #buildstream | 14:36 | |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 14:37 |
| *** tristan has joined #buildstream | 14:39 | |
| tristan | whoa - cs_shadow_ you are quick, I filed a bug against gitlab because I thought the toggle buttons *must* have been an accident on their part ;-) | 14:43 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 14:44 |
| *** cs_shadow_ is now known as cs_shadow | 14:52 | |
| * cs_shadow updates his username first :) Not sure where the trailing underscore came from | 14:53 | |
| * tlater can reproduce the issue by raising an uncaught exception in the track function of the ostree plugin | 14:54 | |
| tristan | tlater, does that mean you've reproduced it now without ever invoking ostree in the child process ? | 14:55 |
| tlater | tristan: Yup | 14:55 |
| tristan | tlater, since this is a really tricky issue; it would be good to note these things in the issue as you find them out, this way we dont forget them once you figure out something, three things later :) | 14:56 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 14:56 |
| tristan | note I've updated the issue with our latest findings an hour or so ago | 14:56 |
| tlater | Will do, that's a good note, I just keep them in my own logs. Probably more useful to have them on gitlab though. | 14:56 |
| *** adds68_ has quit IRC | 15:00 | |
| tristan | Gah ! | 15:02 |
| tristan | Moved from home to here, and now: "Checking connectivity to remote artifact cache" | 15:02 |
| tristan | hangs | 15:02 |
| tristan | what the hell | 15:02 |
| tristan | ssam2, could that somehow be due to my changing ip address ? | 15:03 |
| ssam2 | shouldn't be | 15:03 |
| ssam2 | i think the timeout is 3 seconds | 15:03 |
| tristan | ssam2, sorry for blind ping but I remember you were looking into that auth a long time ago | 15:03 |
| ssam2 | for push | 15:03 |
| tristan | hmmm | 15:03 |
| ssam2 | if it hangs longer than that, something is definitely broken | 15:03 |
| tristan | it's hanging | 15:04 |
| tristan | not on push, but on the checking connectivity thingy | 15:04 |
| * tristan was happily pushing a while ago | 15:04 | |
| ssam2 | any child processes ? | 15:05 |
| ssam2 | the actual check calls `ssh`, so if that hangs for some reason... | 15:05 |
| tristan | yep | 15:05 |
| tristan | ssh -l artifacts -p 10007 gnome7.codethink.co.uk -oConnectTimeout=3 bst-artifact-receive artifacts | 15:05 |
| ssam2 | right | 15:05 |
| tristan | timeout 3, hangs forever | 15:05 |
| tristan | on the server... | 15:05 |
| ssam2 | if I run that command myself it hangs | 15:06 |
| ssam2 | which is weird, it should just print some garbage | 15:06 |
| tristan | ah did you ? | 15:06 |
| tristan | ssam2, are you right now ? | 15:06 |
| ssam2 | i did just try it | 15:06 |
| tristan | ok ... but you are not hanging *right now* | 15:07 |
| tristan | it looks like when you did it, it showed the bst-artifact-receive process | 15:07 |
| ssam2 | ah ok | 15:07 |
| tristan | I am hanging and I only see: | 15:07 |
| tristan | 1438 ? Ss 0:00 \_ sshd: artifacts [priv] | 15:07 |
| tristan | 1461 ? S 0:00 | \_ sshd: artifacts@notty | 15:07 |
| ssam2 | right, ostree.baserock.org hangs too, but i'm pushing stuff to that right now | 15:08 |
| ssam2 | i guess the client sends the hello message, so that's expected | 15:08 |
| tristan | ok so now... I interrupted, and that thing is still there | 15:08 |
| ssam2 | very weird | 15:08 |
| tristan | I have an idea | 15:09 |
| tristan | right, I'm stupid | 15:10 |
| tristan | I have this really bad idea in my ~/.ssh/config: | 15:10 |
| tristan | Host gnome7.codethink.co.uk | 15:10 |
| tristan | ControlMaster auto | 15:10 |
| tristan | ControlPath /run/user/1000/ssh-%r@%h:%p | 15:10 |
| tristan | ControlPersist 1h | 15:10 |
| tristan | So, if I ever change wifi, sucks to be me for an hour | 15:11 |
| tristan | OK - now we start a build of GNOME, from scratch, logging everything, pushing results to artifact share :) | 15:13 |
| tristan | Hah, oh crap :) | 15:26 |
| tristan | rm -rf /usr/share/doc was a bad idea | 15:26 |
| tristan | because ! | 15:29 |
| tristan | c++ bindings mm-common.bst needs to do: cp /usr/share/doc/*/libstdc++/user/libstdc++.tag doctags/ | 15:29 |
| tristan | to avoid trying a wget | 15:29 |
| tlater | Gah, the ostree thing looks to have been a red herring. I can reproduce everything with no changes or failures anywhere simply trying to `bst build --track core-deps/WebKit.bst` | 15:30 |
| tristan | what is a good glob pattern for rm here, rm -rf /usr/share/doc/{everything that is not */libstdc++/user/libstdc++.tag} | 15:30 |
| tlater | Not sure why that doesn't happen when building apps.bst, but possibly because it's harder to notice when you have a lot of elements tracking at the same time. | 15:30 |
| tristan | tlater, raising an exception without doing anything, on a very simple one-element pipeline, seems like your closest bet to reducing the problem to something manageable | 15:31 |
| tristan | so far | 15:31 |
| tristan | maybe you need 2 elements in order to reproduce, though | 15:32 |
| tlater | tristan: It doesn't happen then, or at least is so brief that it's impossible to notice | 15:32 |
| tristan | where the second one does something for a long time | 15:32 |
| tlater | Hmm... I guess tracking a large ostree thing would do it | 15:32 |
| * tlater tries | 15:32 | |
| tlater | tristan: `find /usr/share/doc -except 'libstdc++/usr/libstc++.tag' -exec rm -r {} \;`? | 15:33 |
| tristan | tlater, lets try that... | 15:33 |
| tristan | tlater, rm -r ? | 15:34 |
| tlater | -rf if needed, I'm overly careful | 15:34 |
| tristan | not sure, that might remove it anyway ? | 15:34 |
| tristan | it will recursively remove a parent dir that matches right ? | 15:34 |
| tlater | Yes, should do | 15:34 |
| tristan | so then it's not an option yet... | 15:35 |
| tristan | lets see, decent start... | 15:35 |
| * tlater might have misunderstood what you were trying to do | 15:35 | |
| tristan | find: unknown predicate `-except' | 15:35 |
| tlater | Aww, I could have sworn that was it | 15:35 |
| tristan | tlater, I want to remove everything in /usr/share/doc, except /usr/share/doc/*/libstdc++/user/libstdc++.tag | 15:36 |
| tlater | Oh, alright | 15:36 |
| * tlater thinks the correct flag is '-prune', but doesn't recall how to use it | 15:36 | |
| tristan | find /usr/share/doc ! -name "/usr/share/doc/*/libstdc++/user/libstdc++.tag" | 15:37 |
| tristan | will get a list excluding that file | 15:37 |
| tristan | sorry, -wholename | 15:38 |
| tlater | Ah, awesome | 15:38 |
| tristan | grrrr | 15:38 |
| tristan | tlater, yeah that doesnt solve it though | 15:38 |
| tristan | tlater, one result will be for example: /usr/share/doc/gcc-6-base | 15:39 |
| tristan | that one is removed recursively, so who cares that the other was skipped | 15:39 |
| tlater | Which still contains the parent... | 15:39 |
| tristan | I could only remove files, leaving an empty skeleton of 4kb directories lying around | 15:40 |
| tlater | tristan: You could then remove empty directories | 15:40 |
| tlater | find /usr/share/doc -empty | 15:40 |
| tristan | aha ! | 15:40 |
| tristan | that finds empty files | 15:41 |
| tristan | does it find empty directories ? | 15:41 |
| tlater | find /usr/share/doc -empty -type d | 15:41 |
| tlater | Perhaps the other way around | 15:41 |
| tlater | find is weird | 15:41 |
| tristan | tlater, I dont need a quoted '{}' ? | 15:43 |
| * tristan never uses these features | 15:43 | |
| tristan | eh, still doesnt work well | 15:49 |
| jonathanmaw | tristan: I've responded to issue #132 (https://gitlab.com/BuildStream/buildstream/issues/132) | 15:53 |
| jonathanmaw | having looked through the code and made a guess at what I think is happening | 15:53 |
| jonathanmaw | tl;dr: loading external plugins is fine, but where on earth is it getting buildstream from? | 15:54 |
| tristan | jonathanmaw, the image, maybe ? | 15:55 |
| * tlater takes a look as well | 15:55 | |
| tristan | jonathanmaw, in which case, it'll have an older version which includes the plugins you're testing ? | 15:55 |
| jonathanmaw | tristan: yeah, that's a risk | 15:56 |
| jonathanmaw | so I think I should make the CI uninstall whatever buildstream's currently installed, then install one from git | 15:56 |
| tristan | jonathanmaw, or update the image first | 15:56 |
| tristan | better that way | 15:56 |
| jonathanmaw | yeah, that's a smarter thing. | 15:57 |
| * jonathanmaw reminds himself how to do that | 15:57 | |
| tristan | jonathanmaw, ask ssam2 if you dont figure it out, but first checkout the BuildStream group - it's automated now | 15:58 |
| tristan | so I think there is no mucking about trying to understand the mystery that is docker | 15:58 |
| ssam2 | yes, MR against buildstream-docker-images repo is all that's needed :-) | 15:58 |
| *** adds68_ has joined #buildstream | 15:58 | |
| * tlater isn't sure what the issue is about, but does remember that pip-based plugins don't have to be required by the project.conf. | 16:02 | |
| tristan | tlater, then we should fix that I think, otherwise it's totally random | 16:05 |
| tristan | that said, maybe jonathanmaw doesnt have to block on it, but I would still say its a serious bug if plugins can "come from nowhere randomly" | 16:07 |
| tristan | also worth checking if we're missing API for it, but I think we already have the mechanics in place for explicitly requiring the right version of it, so it's probably sufficient api | 16:07 |
| tlater | tristan: They do require being installed through setuptools first - you need to run ./setup.py install before this works | 16:07 |
| tristan | yes | 16:08 |
| tlater | Just pointing that out because it's not exactly 'nowhere' :) | 16:08 |
| tristan | tlater, if they happen to be installed, and you pick them up without requiring them, then they come from an unknown location | 16:09 |
| tristan | we have explicit plugin paths in project.conf for loading local plugins | 16:09 |
| tristan | they can not come from nowhere, or else I'm not sure you can be sure you are getting what you want | 16:09 |
| tlater | tristan: I agree, although it will be very hard to specify a path from which to load these - setuptools will happily move files around. Only allowing to include plugins that are defined in project.conf is definitely possible though. | 16:11 |
| tristan | yes | 16:11 |
| tristan | desirable to a point where, it's alarming that it works without an explicit mention | 16:12 |
| tristan | currently pip is one source of plugin discovery, there may be others if we can successfully wean ourselves off of pip | 16:12 |
| tristan | seems to me the correct way to do plugins, is to have buildstream install a pkg-config file and have other modules query that config file for the location of where plugins are to be installed on the system | 16:13 |
| tristan | pip is a dirty workaround, cause python | 16:13 |
| * tlater can see it being messy to have the same plugin name in multiple discovery systems without explicit requirements. | 16:14 | |
| tlater | Perhaps this needs something like | 16:14 |
| tlater | plugins: | 16:14 |
| tlater | paths: | 16:14 |
| tlater | - /somewhere | 16:14 |
| tlater | pip: | 16:14 |
| tlater | - something | 16:14 |
| tristan | nod | 16:15 |
| tristan | OK we should mark this blocker | 16:15 |
| tristan | otherwise we're stuck with vague plugin loading | 16:16 |
| tristan | or, migration path could be to assume pip in the case it's not mentioned | 16:16 |
| tristan | and error out if you require a version of buildstream that has a proper fix (without marking blocker of 1.0) | 16:16 |
| tlater | tristan: A bit unrelated now, but if you're deleting empty directories with find you can use the -delete argument. Deleting all empty directories in a path is `find <path> -type d -empty -delete` | 16:16 |
| tristan | tlater, I skipped that for now, but is it true ? | 16:17 |
| tlater | tristan: Yes, I actually tried this one and didn't do it from memory :) | 16:17 |
| tristan | tlater, the find method left me with an unsorted thing which errors a bunch of times | 16:17 |
| tlater | :/ | 16:18 |
| tristan | if you do it many times, it will eventually delete all the empty dirs | 16:18 |
| tristan | otherwise it needs depth sort | 16:18 |
| * tristan goes to get food | 16:19 | |
| *** semanticdesign has quit IRC | 16:27 | |
| *** semanticdesign_ has quit IRC | 16:27 | |
| *** adds68_ has quit IRC | 16:28 | |
| *** tlater has quit IRC | 16:46 | |
| *** jonathanmaw has quit IRC | 17:00 | |
| *** jonathanmaw has joined #buildstream | 17:00 | |
| *** jonathanmaw has quit IRC | 17:01 | |
| *** ssam2 has quit IRC | 17:08 | |
| *** valentind has joined #buildstream | 17:27 | |
| *** adds68_ has joined #buildstream | 18:15 | |
| *** givascu has joined #buildstream | 18:21 | |
| valentind | Are artifacts indexed by target architecture? And can you refer to build from other architectures within the same project? | 18:28 |
| tristan | valentind, so the built-in --arch/--target-arch options and `arches` conditional statements are deprecated and to be removed | 18:30 |
| valentind | OK | 18:30 |
| tristan | valentind, in favor of project options, which are project specific | 18:30 |
| tristan | but - that doesnt exactly answer your question | 18:30 |
| tristan | the guarantee that we do try to make, is that a cache key for an artifact will differ based on anything which might effect the build output | 18:31 |
| tristan | which of course, includes the architecture | 18:31 |
| valentind | I suppose anyway I have to dump the cross compiled image and then feed it to the next builder. | 18:31 |
| tristan | http://buildstream.gitlab.io/buildstream/projectconf.html#architecture | 18:31 |
| tristan | to declare arch type options ^^^ | 18:32 |
| tristan | valentind, well - juergbi is working on the recursive pipelines feature we've been planning since forever | 18:32 |
| tristan | I expect that to land post 1.0 (we're "major feature" frozen for now) | 18:32 |
| tristan | and certainly before the end of year we'll be able to have projects which depend on other projects | 18:33 |
| *** adds68_ has quit IRC | 18:34 | |
| valentind | I suppose I will split into two projects. One for the cross compiling of the minimal image, and one for the rest. And do a "bst checkout" in between. | 18:34 |
| tristan | here is also another example of casing the arch project options: https://gitlab.com/BuildStream/gnome-modulesets-base/blob/master/elements/base/base-system.bst | 18:34 |
| tristan | valentind, and since we didnt have that part yet, Sam worked on it but it wasnt in a referrable form, he's currently working on that again so we have a good reference | 18:35 |
| tristan | valentind, so fwiw, there *is* a project which creates a bootable image with a very, minimal system, and the runtime from that can be used to re-bootstrap itself | 18:35 |
| tristan | but I just can't link to it now, cause it's only in the form of a conversion of the baserock base runtime stuff | 18:36 |
| tristan | I think by next week we'll have a url to point to | 18:36 |
| valentind | Bootable? | 18:36 |
| tristan | Yep, using the x86Image element to create an image (intel only for now) | 18:37 |
| tristan | all without need of root privileges | 18:37 |
| tristan | that's been removed from buildstream proper... because I consider it a bit fringe and not api stable | 18:37 |
| tristan | http://buildstream.gitlab.io/bst-external/elements.x86image.html#module-elements.x86image | 18:38 |
| valentind | I am trying to make an image like fd.o. This is just experimental. The point is to test multiarch (the mulilib the Debian way). | 18:38 |
| tristan | valentind, so it requires using the (new) bst-external, separate repo | 18:38 |
| valentind | I have the base sdk, now I am in the x.org libraries. Quite a lot of them. | 18:38 |
| tristan | Yeah, you wont find that in any of our work :) | 18:38 |
| valentind | bst-external? | 18:39 |
| tristan | not that multilib is not interesting, but there are a whole lot of other mountains to move before that one :) | 18:39 |
| valentind | I want to make Steam work correctly on flatpak. That is all I want. | 18:39 |
| tristan | yup https://gitlab.com/BuildStream/bst-external/ | 18:39 |
| valentind | And multiarch makes is easier than multilib. | 18:39 |
| valentind | tristan, there is a plugin source I would like to have, it is to have artifact as source. | 18:41 |
| valentind | source plugin. | 18:41 |
| tristan | That sounds a lot like working around the absence of the feature juergbi is working on | 18:41 |
| tristan | I.e. the same can be accomplished with a compose element on a dependent project | 18:42 |
| valentind | The point is to be able to use the import plugin on artifact. So you can compose artifact, extracting subpath or putting them in subpaths without needing a runtime. | 18:42 |
| tristan | or, rather a project you depend on | 18:42 |
| valentind | Because I cross compile into a sysroot directory. At the end I want to extract what is inside the sysroot. | 18:43 |
| tristan | I'm not sure artifacts themselves are quite consumable that way | 18:43 |
| tristan | rather, if you really want that; better to automate a task which does a checkout and commit to an ostree repo (i.e. a "real release" / "deployment") | 18:43 |
| valentind | That also would be nice. | 18:44 |
| valentind | Committing to ostree would great. | 18:44 |
| tristan | I'm not sure I can reason about consuming artifacts as sources well at 3:45 am, though :) | 18:44 |
| tristan | Yeah, I mean that's what we're doing for the base bootstrapping | 18:44 |
| tristan | Like, we initially bootstrap from a debian or something, then we build a runtime, that we commit to a public ostree repo | 18:45 |
| tristan | which in turn, we can use instead of the original debian runtime | 18:45 |
| tristan | and now we are free to upgrade | 18:45 |
| tristan | totally circular :) | 18:45 |
| valentind | The artifact as source is for something else. | 18:46 |
| tristan | The use case of deploying something which you can later import is the same, though | 18:46 |
| valentind | So what is the best for now? Doing a checkout and use local source? | 18:48 |
| tristan | Well, I'm not sure _exactly_ the context here | 18:48 |
| tristan | only vaguely | 18:49 |
| tristan | you say you want to do it "without a runtime", but presumably you've built something | 18:49 |
| tristan | which means, you had a runtime somewhere at your disposal | 18:49 |
| tristan | valentind, in which case, a simple compose element might do the trick | 18:49 |
| valentind | Compose cannot strip path components. | 18:50 |
| tristan | What does that mean ? | 18:50 |
| valentind | OK. So I build everything in "/cross-installation". This is my sysroot. | 18:50 |
| valentind | And I install everything with DESTDIR=%{install-root}/cross-installation". | 18:51 |
| valentind | When I am done, I want what is inside "/cross-installation". | 18:52 |
| valentind | But with "cross-installation" removed. | 18:52 |
| tristan | I see | 18:52 |
| tristan | Right now you can do it with a script element, which will let you do just about anything | 18:52 |
| tristan | Maybe we can add more features to compose elements too, not sure it's the right element for that, but maybe it makes sense | 18:53 |
| tristan | better to be careful about adding features to compose, as it's a slippery slope and stable API means we cant climb out of it | 18:53 |
| tristan | valentind, anyway with a script element, you can stage your "everything I built" using the host compat runtime, and yank the content of cross-compilation/ and put that where you collect the artifact from | 18:55 |
| valentind | Script element? | 18:55 |
| tristan | yeah | 18:56 |
| tristan | so all the buildstream plugins are documented here: http://buildstream.gitlab.io/buildstream/pluginindex.html#plugins-elements | 18:56 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:57 |
| tristan | people can also write their own plugins and include that in their project | 18:57 |
| tristan | valentind, the trick with the script element, is the "layout" part | 18:57 |
| tristan | valentind, that lets you define which dependencies you want to stage where | 18:58 |
| tristan | of course, you need to have something host compatible in / | 18:58 |
| tristan | (presumably, the whichever runtime you used to do cross compilation will do fine, to give you the simple tools you would need to manipulate the thing you stage at, say %{build-root}) | 18:59 |
| valentind | What I do is just use the "manual" plugin and put a cp in install-commands. But I need a runtime. | 19:03 |
| gitlab-br-bot | push on buildstream@master (by Pedro Alvarez Piedehierro): 3 commits (last: Issue #107: Stop attaching the child watcher multiple times) https://gitlab.com/BuildStream/buildstream/commit/6976ba5d8c222aee61bfd5666b5df1b07f76b9bd | 19:29 |
| ironfoot | ^ me testing gitlab bot still ok after recent changes | 19:29 |
| tristan | ironfoot, awesome, did you disable the notifications on non-master pushes ? | 19:48 |
| tristan | valentind, a manual element is not a script element | 19:48 |
| tristan | script, element: http://buildstream.gitlab.io/buildstream/elements/script.html#module-elements.script | 19:48 |
| *** xjuan has quit IRC | 20:45 | |
| *** tristan has quit IRC | 22:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!