| *** nimish2711 has joined #buildstream | 02:08 | |
| *** tristan has quit IRC | 04:07 | |
| *** tristan has joined #buildstream | 05:40 | |
| *** tristan has quit IRC | 06:48 | |
| *** tristan has joined #buildstream | 07:18 | |
| *** ChanServ sets mode: +o tristan | 07:34 | |
| *** tristan changes topic to "BuildStream 1.2.7 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://mail.gnome.org/mailman/listinfo/buildstream-list | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmaps" | 07:34 | |
| *** benschubert has joined #buildstream | 07:51 | |
| *** raoul has joined #buildstream | 08:36 | |
| *** bochecha has joined #buildstream | 08:48 | |
| *** rdale has joined #buildstream | 08:49 | |
| *** tpollard has joined #buildstream | 08:52 | |
| *** nimish2711 has quit IRC | 09:08 | |
| *** nimish2711 has joined #buildstream | 09:14 | |
| *** nimish2711 has quit IRC | 09:24 | |
| *** lachlan has joined #buildstream | 09:30 | |
| *** lachlan has quit IRC | 09:39 | |
| *** lachlan has joined #buildstream | 09:45 | |
| *** toscalix has joined #buildstream | 09:48 | |
| *** lachlan has quit IRC | 09:51 | |
| *** bochecha has quit IRC | 09:53 | |
| *** lachlan has joined #buildstream | 09:56 | |
| *** bochecha has joined #buildstream | 09:59 | |
| *** lachlan has quit IRC | 09:59 | |
| *** tristan has quit IRC | 10:00 | |
| *** tristan has joined #buildstream | 10:09 | |
| tpollard | tristan: re the ui as a subprocess thread, afaik the pulling of buildtree for bst shell/push does go through the scheduler | 10:15 |
|---|---|---|
| *** ChanServ sets mode: +o tristan | 10:18 | |
| tristan | tpollard, I wasn't being pedantic :) | 10:18 |
| tristan | indeed it seems that it is done properly with `bst shell` | 10:18 |
| tristan | tpollard, however I don't think it is done with workspace related fetch + track | 10:18 |
| tristan | tpollard, neither is it done for the automatic fetching of junctions at load time | 10:19 |
| * tpollard nods | 10:19 | |
| tpollard | Didn't think you were being pedantic either :) | 10:19 |
| tristan | tpollard, obviously splitting up processes is going to be a big architectural change... thinking back on that email, I think that maybe it is a good idea to consider precise synchronicity of state in messages to be a separate problem... by that I mean "the state reflected in a given log line might not be the state of the data model at the time the log line was initially issued" | 10:24 |
| tristan | tpollard, we might consider an overhaul of messaging, such that messages carry all state, as a completely separate detail | 10:25 |
| *** lachlan has joined #buildstream | 10:25 | |
| tristan | although, I think it is still an important detail | 10:25 |
| benschubert | tristan: I liked the idea in your email of having the message give all the necessary information, it might be a simpler way if it's not slowing down too much the execution :) | 10:25 |
| tristan | reliable logging is pretty paramount to a build tool | 10:25 |
| tristan | benschubert, indeed, I hope it is not too much, but we can keep the Message object with a minimal amount of fields, leaving the rendering details to the frontend | 10:26 |
| * phildawson is also quite keen on that. | 10:27 | |
| Kinnison | To keep the message size down requires that the state changes be communicated efficiently also | 10:27 |
| tristan | benschubert, performance should be helped by having the Job processes send messages directly to the frontend... and hopefully with non blocking writes | 10:27 |
| Kinnison | I think aevri was looking at ways to do that | 10:27 |
| *** lachlan has quit IRC | 10:28 | |
| benschubert | tristan: agreed! | 10:28 |
| phildawson | I don't know enough about the build shell to understand how big the issues there would be though. | 10:29 |
| gitlab-br-bot | BenjaminSchubert approved MR !1346 (danielsilverstone-ct/classmethod-inits->master: Make various Element/Source init helpers @classmethod to clarify data flow) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1346 | 10:29 |
| tristan | Now of course the crux of it is... if we put *all* state management into the subprocess, and have the entire function bodies of main Stream() entry points run in a single process (possibly running the scheduler multiple times)... and if we *dont* syncrhonize state back to the frontend, then the build shell is wildly tricky :-/ | 10:30 |
| tristan | phildawson, incidentally that's what I was just talking about | 10:30 |
| * phildawson reads backscroll :) | 10:30 | |
| tristan | phildawson, If we do syncrhonize all state back to the frontend, then it becomes a non-issue, but it would indeed be interesting to not need to | 10:31 |
| tristan | phildawson, just the last message :) | 10:31 |
| tristan | "now the crux of it..." | 10:31 |
| phildawson | Ah right | 10:31 |
| tristan | essentially you need the state of elements and their cache keys in order to launch a shell | 10:32 |
| *** lachlan has joined #buildstream | 10:32 | |
| benschubert | tristan: could the subprocess launch the shell? | 10:32 |
| tristan | and then, there is always a funky possibility of shelling in *through* the subprocess | 10:32 |
| tristan | but that is a whole separate can of worms I think | 10:32 |
| tristan | like, you need a real terminal | 10:33 |
| tristan | can you give control of the terminal to the subprocess ? I think that will be tricky, but maybe I'm wrong ? | 10:33 |
| benschubert | ^ I'm not sure I follow about the "real terminal" | 10:33 |
| tristan | Perhaps I'm imagining a difficulty where there is none ? | 10:34 |
| tristan | benschubert, is the subprocess inherently connected to the terminal if the master process is ? | 10:34 |
| tristan | Maybe | 10:34 |
| benschubert | I think so | 10:34 |
| tristan | But then, the current mechanics of the main process and Job processes does some things which cause it not to be true for jobs | 10:34 |
| tristan | and it is important for frontend signal handling | 10:35 |
| *** bilelmoussaoui has joined #buildstream | 10:35 | |
| tristan | i.e. you need at least the subprocess to be setsid() I believe, so that you are sure that you catch SIGTSTP and such *only* in the frontend | 10:35 |
| tristan | And instruct the subprocess to do things like pause jobs | 10:35 |
| *** lachlan has quit IRC | 10:36 | |
| tristan | In any case, job control is certainly tricky at least on linux - we certainly want the frontend to be the only place interacting with the OS, have the frontend reliably shut everything down and sync the logs on SIGTERM, handle ^C and ^Z properly etc | 10:40 |
| tristan | I think that whether we can do that and still easily launch a shell in the subprocess that is connected with the parent process's terminal is something we need to check | 10:40 |
| benschubert | agreed | 10:41 |
| benschubert | I should probably look at how to handle ^C correctly one day too | 10:41 |
| *** lachlan has joined #buildstream | 10:41 | |
| phildawson | Tristan, what you're saying makes sense. I think I'm going to need to do some reading to properly understand the problem here. | 10:42 |
| *** tristan has quit IRC | 10:43 | |
| *** tristan has joined #buildstream | 10:44 | |
| *** ChanServ sets mode: +o tristan | 10:44 | |
| tristan | It seems that my thinkpad doesnt like to have most of it's cpus running at 100% 24/7 for weeks on end :'( | 10:45 |
| tristan | So SIGINT (^C) is not particularly tough, however as I recall without setsid() in the child forked process, signals get sent to the group, and as such they get handled in all forked processes | 10:47 |
| *** lachlan has quit IRC | 10:47 | |
| tristan | it's been over a year since I looked at job control, but I believe that's how it was working, and different signals have different contracts | 10:47 |
| tpollard | I think there's been some improvements in multiprocessing around this but only since 3.7 | 10:48 |
| tristan | for SIGTSTP, we get one and use it to send SIGTSTP to the child jobs - this is a bit tricky, because the child jobs have no way of handling an IPC message (they are basically blocking workloads with no mainloop) | 10:49 |
| tristan | So in the child jobs we use the context managers in _signals.py | 10:49 |
| tristan | To react to signals... when a child gets SIGTSTP, it puts it's process tree to sleep, and then wakes up when it resumes | 10:49 |
| tristan | I think we then send SIGSTOP to ourself in the child job (which cannot be handled), and then wake up child jobs on the following line of code (because after sending ourself the signal, we only reach the next line of code by waking up) | 10:51 |
| tristan | tpollard, I highly doubt the python standard library is going to help us with job control, but interested to hear specifically what you are talking about :) | 10:51 |
| tristan | Anyway, what I was mostly getting at, is that we can have a much cleaner SIGTSTP handling in the frontend <--> scheduling process | 10:52 |
| tristan | since the scheduler does run a main loop, we can (A) Handle SIGTSTP in the frontend... (B) Send a *message* to the scheduling process and inform it that it needs to go to sleep... (C) Have the scheduling process put the jobs to sleep in the way it currently does | 10:53 |
| tristan | And basically the same thing is used when the frontend receives the first ^C (SIGINT) and we explicitly put the scheduler and jobs to sleep | 10:53 |
| tpollard | tristan: kill & close got added to the api in 3.7, I don't know the specifics though :) | 10:54 |
| tristan | tpollard, sounds like some prettyness, we do have os.kill() | 10:55 |
| tristan | tpollard, i.e. basically it sounds like process.kill() would just be shorthand for os.kill(process.pid, ...) | 10:55 |
| phildawson | Thanks for the description of how we're doing signal handling now, that was rather helpful :) | 10:56 |
| tristan | Of course, it's delicate and we all want it to work well :) | 10:57 |
| benschubert | Yep, the problem I had with ^C, is that we mask signals when spawning a new process, meaning I can never ^C if I have too many job spawned :'D | 10:58 |
| tristan | Essentially the gist of it is... when considering a frontend <--> scheduler separation, we can avoid some mess/tricks in the interactions with the scheduler <--> jobs | 10:58 |
| benschubert | but that's only a problem with RE/big machines | 10:58 |
| tristan | The scheduler <--> jobs relationship is more complicated because jobs are blocking, and we cannot send them instructions to handle in an event loop | 10:58 |
| WSalmon | I have messed up emailing this list again, this time i did mean to send the email to the list but i forgot to change the subject so it looks like it was a mistake when it was not, apologies. https://mail.gnome.org/archives/buildstream-list/2019-May/msg00018.html | 10:59 |
| tristan | benschubert, interesting, that is a bug which can theoretically be fixed I think | 10:59 |
| tristan | benschubert, I.e. we need to (A) Mask sigint so that it's not inherited... (B) Fork the subprocess... (C)... unmask and handle the signals which arrived while forking... ... and to top it off, we need to let the scheduler process breathe a bit if it is in the middle of spawning many jobs at the same time | 11:00 |
| tristan | benschubert, because the scheduler has it's main loop... we will only actually *handle* the SIGINT once we're done our current work and return to the loop | 11:01 |
| tristan | Of course, with a rework, we can replace "handling of SIGINT" here with "processing an explicit message from the frontend" | 11:02 |
| *** lachlan has joined #buildstream | 11:03 | |
| benschubert | tristan: agreed. Let's wait for the split before it, doesn't seem like many people had trouble with it | 11:04 |
| tpollard | WSalmon: nice email re buildbox support | 11:08 |
| WSalmon | Thanks tpollard | 11:10 |
| *** lachlan has quit IRC | 11:18 | |
| *** bilelmoussaoui_ has joined #buildstream | 11:22 | |
| *** bilelmoussaoui has quit IRC | 11:22 | |
| *** bilelmoussaoui_ is now known as bilelmoussaoui | 11:22 | |
| *** lachlan has joined #buildstream | 11:28 | |
| tpollard | WSalmon: also 100% for buildbox selection being a user option, not a project one | 11:33 |
| *** lachlan has quit IRC | 11:56 | |
| *** lachlan has joined #buildstream | 12:04 | |
| *** lachlan has quit IRC | 12:08 | |
| *** tristan has quit IRC | 12:47 | |
| *** lachlan has joined #buildstream | 13:12 | |
| *** bilelmoussaoui has quit IRC | 13:13 | |
| *** lachlan has quit IRC | 13:21 | |
| *** tristan has joined #buildstream | 13:22 | |
| *** lachlan has joined #buildstream | 13:35 | |
| gitlab-br-bot | willsalmon closed issue #963 (Improve workspace docs) on buildstream https://gitlab.com/BuildStream/buildstream/issues/963 | 13:38 |
| *** lachlan has quit IRC | 13:39 | |
| *** lachlan has joined #buildstream | 14:00 | |
| *** lachlan has quit IRC | 14:03 | |
| *** lachlan has joined #buildstream | 14:38 | |
| *** lachlan has quit IRC | 14:54 | |
| tpollard | Kinnison: I think marge is still struggling, it 1346 probably needs a manual merge | 14:57 |
| Kinnison | tpollard: then could you do it? | 14:58 |
| Kinnison | (It's not that I'm lazy, I literally don't have merge rights :D) | 14:58 |
| tpollard | Sure | 14:58 |
| gitlab-br-bot | tpollard approved MR !1346 (danielsilverstone-ct/classmethod-inits->master: Make various Element/Source init helpers @classmethod to clarify data flow) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1346 | 14:58 |
| gitlab-br-bot | tpollard merged MR !1346 (danielsilverstone-ct/classmethod-inits->master: Make various Element/Source init helpers @classmethod to clarify data flow) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1346 | 14:59 |
| Kinnison | ta | 14:59 |
| *** lachlan has joined #buildstream | 15:44 | |
| *** lachlan has quit IRC | 15:49 | |
| *** lachlan has joined #buildstream | 15:51 | |
| *** lachlan has quit IRC | 15:55 | |
| *** bochecha has quit IRC | 15:58 | |
| *** lachlan has joined #buildstream | 16:14 | |
| *** lachlan has quit IRC | 16:17 | |
| *** phildawson_ has joined #buildstream | 16:38 | |
| *** phildawson has quit IRC | 16:39 | |
| *** lachlan has joined #buildstream | 16:46 | |
| *** lachlan has quit IRC | 16:49 | |
| *** toscalix has quit IRC | 16:54 | |
| *** lachlan has joined #buildstream | 17:06 | |
| *** lachlan has quit IRC | 17:40 | |
| *** lachlan has joined #buildstream | 17:44 | |
| *** pointswaves has joined #buildstream | 17:45 | |
| *** tristan has quit IRC | 17:48 | |
| *** phildawson_ has quit IRC | 17:52 | |
| *** tristan has joined #buildstream | 18:09 | |
| *** pointswaves has quit IRC | 18:20 | |
| *** lachlan has quit IRC | 18:31 | |
| *** nimish2711 has joined #buildstream | 18:32 | |
| *** lachlan has joined #buildstream | 18:34 | |
| *** nimish2711_ has joined #buildstream | 18:39 | |
| *** nimish2711 has quit IRC | 18:39 | |
| *** nimish2711_ is now known as nimish2711 | 18:39 | |
| *** lachlan has quit IRC | 18:40 | |
| *** lachlan has joined #buildstream | 19:07 | |
| *** lachlan has quit IRC | 19:13 | |
| *** lachlan has joined #buildstream | 19:57 | |
| *** lachlan has quit IRC | 20:00 | |
| *** mohan43u has quit IRC | 21:10 | |
| *** mohan43u has joined #buildstream | 21:38 | |
| *** nimish2711 has quit IRC | 21:45 | |
| *** nimish2711 has joined #buildstream | 21:50 | |
| gitlab-br-bot | cs-shadow opened MR !1347 (chandan/coverage-doesnt-need-deps->master: tox.ini: Coverage does not need module installed) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1347 | 23:13 |
| *** nimish2711 has quit IRC | 23:15 | |
| *** nimish2711 has joined #buildstream | 23:20 | |
| *** admin has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!