| gitlab-br-bot | buildstream: merge request (chandan/pypi-badge->master: WIP: README.rst: Add status badges for PyPI release and Python versions) #719 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/719 | 00:31 |
|---|---|---|
| *** tristan has joined #buildstream | 04:36 | |
| *** ChanServ sets mode: +o tristan | 04:37 | |
| gitlab-br-bot | buildstream: merge request (tristan/reduce-filter-tests->master: tests/plugins/filter.py: Don't run redundant tests) #753 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/753 | 06:04 |
| gitlab-br-bot | buildstream: merge request (tristan/reduce-filter-tests-1.2->bst-1.2: tests/plugins/filter.py: Don't run redundant tests) #754 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/754 | 06:06 |
| *** leopi has joined #buildstream | 06:21 | |
| gitlab-br-bot | buildstream: merge request (tristan/reduce-filter-tests-1.2->bst-1.2: tests/plugins/filter.py: Don't run redundant tests) #754 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/754 | 06:34 |
| gitlab-br-bot | buildstream: merge request (tristan/reduce-filter-tests->master: tests/plugins/filter.py: Don't run redundant tests) #753 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/753 | 06:36 |
| gitlab-br-bot | buildstream: merge request (tristan/538-reenable-ostree-test->master: tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]) #755 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/755 | 06:39 |
| gitlab-br-bot | buildstream: merge request (tristan/538-reenable-ostree-test-1.2->bst-1.2: tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]) #756 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/756 | 06:41 |
| *** alatiera_ has joined #buildstream | 06:42 | |
| gitlab-br-bot | buildstream: merge request (tristan/538-reenable-ostree-test->master: tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]) #755 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/755 | 07:06 |
| gitlab-br-bot | buildstream: merge request (tristan/538-reenable-ostree-test-1.2->bst-1.2: tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]) #756 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/756 | 07:07 |
| *** tristan has quit IRC | 07:10 | |
| valentind | bochecha, that was jonathanmaw who made those fixes. | 08:02 |
| *** tristan has joined #buildstream | 08:05 | |
| gitlab-br-bot | buildstream: merge request (Qinusty/cyclic-variable-backport->bst-1.2: Backport cyclic variable fix) #757 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/757 | 08:06 |
| *** ChanServ sets mode: +o tristan | 08:09 | |
| *** toscalix has joined #buildstream | 08:11 | |
| *** toscalix has quit IRC | 08:13 | |
| *** toscalix has joined #buildstream | 08:15 | |
| *** rdale has joined #buildstream | 08:23 | |
| coldtom | Does buildstream remember the url pulled artifacts came from? | 08:26 |
| valentind | I do not think so. | 08:26 |
| coldtom | I'm trying to test if freedesktop is reproducible, but it just re-pulls from the cache rather than rebuilding locally, despite me removing the cache from project.conf | 08:26 |
| valentind | Have you configured it in .config/buildstream.conf? | 08:27 |
| coldtom | i can't find the location of my buildstream.conf, so i assume it's still the default | 08:29 |
| qinusty | It'll say in the cli output? | 08:30 |
| qinusty | Pulling <element> <- <URL> | 08:31 |
| coldtom | qinusty: it's pulling from the freedesktop cache, but it's not configured in my project.conf | 08:31 |
| qinusty | errrrrrrrrrrrrrrrrrrrrrm | 08:31 |
| coldtom | so i pulled to get a cache | 08:31 |
| coldtom | removed the `artifacts: ` from project.conf | 08:31 |
| coldtom | attempt to rebuild, but it still pulls | 08:31 |
| qinusty | so project.conf doesn't reference the cache either? | 08:31 |
| coldtom | it does not | 08:32 |
| tpollard | I think I've seen that behaviour before | 08:32 |
| gitlab-br-bot | buildstream: merge request (Qinusty/cyclic-variable-backport->bst-1.2: Backport cyclic variable fix) #757 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/757 | 08:33 |
| gitlab-br-bot | buildstream: issue #600 ("Devours memory and stack traces when recursive variables defined") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/600 | 08:33 |
| qinusty | hmmm | 08:33 |
| *** tiagogomes_ has joined #buildstream | 08:35 | |
| gitlab-br-bot | buildstream: issue #616 ("timed activities sometimes balanced with WARNING in place of FAILURE, when task retries are enabled") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/616 | 08:36 |
| *** tiagogomes__ has joined #buildstream | 08:36 | |
| gitlab-br-bot | buildstream: merge request (tristan/source-mirroring-changes->master: Minor code changes revolving around source mirroring) #758 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/758 | 08:46 |
| gitlab-br-bot | buildstream: merge request (valentindavid/post_tracking_errors->master: Report processing errors from tracking) #747 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/747 | 08:46 |
| qinusty | So coldtom, your local cache is full so that when you pull from a remote during a build, it isn't locally cached? | 08:46 |
| coldtom | qinusty: my cache is populated, but not full (i think i'm still configured to have infinite cache space) | 08:48 |
| coldtom | if a cache key has changed it'll still cache it locally | 08:49 |
| qinusty | So your script changes the cache key before the build? | 08:49 |
| coldtom | nope, it keeps the cache key but manually removes the ref and object from the cache | 08:49 |
| *** awacheux[m] has quit IRC | 08:49 | |
| *** pro[m] has quit IRC | 08:50 | |
| *** abderrahim[m] has quit IRC | 08:50 | |
| *** cgmcintyre[m] has quit IRC | 08:50 | |
| *** oknf[m] has quit IRC | 08:50 | |
| *** alatiera has quit IRC | 08:50 | |
| *** theawless[m] has quit IRC | 08:50 | |
| *** jjardon[m] has quit IRC | 08:50 | |
| *** connorshea[m] has quit IRC | 08:50 | |
| *** Demos[m] has quit IRC | 08:50 | |
| *** dineshdb[m] has quit IRC | 08:50 | |
| *** segfault3[m] has quit IRC | 08:50 | |
| *** asingh_[m] has quit IRC | 08:50 | |
| *** doras[m] has quit IRC | 08:50 | |
| *** kailueke[m] has quit IRC | 08:50 | |
| *** rafaelff[m] has quit IRC | 08:50 | |
| *** inigomartinez has quit IRC | 08:50 | |
| *** ssssam[m] has quit IRC | 08:50 | |
| *** albfan[m] has quit IRC | 08:50 | |
| *** m_22[m] has quit IRC | 08:50 | |
| *** waltervargas[m] has quit IRC | 08:50 | |
| *** tlater[m] has quit IRC | 08:50 | |
| *** mattiasb has quit IRC | 08:50 | |
| *** krichter[m] has quit IRC | 08:50 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/post_tracking_errors->master: Report processing errors from tracking) #747 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/747 | 08:50 |
| *** m_22[m] has joined #buildstream | 08:50 | |
| coldtom | buildstream knows it isn't cached locally, as the bst show bit before a build has "fetch needed" | 08:52 |
| gitlab-br-bot | buildstream: merge request (tristan/source-mirroring-changes-1.2->bst-1.2: Minor code changes revolving around source mirroring) #759 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/759 | 08:54 |
| tristan | coldtom, it's impossible to disable currently, short of workspacing the freedesktop-sdk-junction.bst and disabling *it's* cache server there | 08:57 |
| tristan | nice pipeline qinusty https://gitlab.com/BuildStream/buildstream/pipelines/29021229, only 20 min ! | 09:09 |
| qinusty | indeed! So much better :D | 09:10 |
| tristan | qinusty, since you were talking about UI docs... I think that describing the message types as I did today here https://gitlab.com/BuildStream/buildstream/issues/616, would be prime docs material | 09:10 |
| tristan | what do you think ? | 09:10 |
| * qinusty takes a look | 09:11 | |
| qinusty | Fu nfact | 09:11 |
| qinusty | fun fact, this confused me for a while when looking into the whole retry behaviour | 09:12 |
| gitlab-br-bot | buildstream: merge request (tristan/source-mirroring-changes->master: Minor code changes revolving around source mirroring) #758 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/758 | 09:12 |
| qinusty | It is indeed raised as a FAIL. But converted to WARNING during retries https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_scheduler/jobs/job.py#L404 https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_scheduler/jobs/job.py#L512 | 09:13 |
| tristan | yeah I bumped into that today while looking into https://gitlab.com/BuildStream/buildstream/issues/612#note_97813392 | 09:14 |
| tristan | and noticed that when a source mirror fails and it rolls over to the next one, it said WARNING (wtf??) | 09:14 |
| tristan | qinusty, I find it interesting that you added SKIPPED in _messages.py to the task related things, but then implemented it in cascache.py as just sending a message manually | 09:15 |
| tristan | I agree with the former, not the latter | 09:15 |
| tristan | And I think we have a follow up issue for it I filed yesterday ? | 09:16 |
| qinusty | Yeah I noticed after the merge during some other work that I had placed the MessageType.SKIPPED value below the timed activity messages | 09:16 |
| qinusty | It was unintentional and the intent of the implementation was a special kind of info message to be clearer. | 09:17 |
| tristan | I think basically, CasCache should be printing an INFO message based on whether it did or did not find an artifact, and in the case that all artifact servers did *not* find the artifact, it should then just `raise Skip("No artifact found")` | 09:17 |
| gitlab-br-bot | buildstream: merge request (tristan/source-mirroring-changes-1.2->bst-1.2: Minor code changes revolving around source mirroring) #759 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/759 | 09:17 |
| qinusty | The problem there is, the logic for pushing to remotes currently pushes to all remotes on one job | 09:18 |
| tristan | qinusty, because SKIPPED has already a very specific meaning, which is the bookkeeping in the job queues, IMO a timed activity by itself cannot be SKIPPED, only the complete task can be SKIPPED, and it will be in the summary and status bar | 09:18 |
| qinusty | agreed, but I haven't thought about this too much regarding how to implement this nicely | 09:19 |
| tristan | Then, we just handle the `Skip()` exception in job.py, and in *that case only*, we print a SKIPPED message instead of SUCCESS/FAIL, and place the element in the skipped list | 09:19 |
| tristan | We can also take the custom exception text from the Skip exception, and use it as `detail` in the SKIPPED message which closes the START | 09:19 |
| tristan | i.e. a START message is *always* terminated by either SUCCESS/FAILURE or SKIPPED | 09:20 |
| qinusty | So when is a task SKIPPED | 09:20 |
| qinusty | in terms of at the bottom | 09:20 |
| qinusty | As far as I can tell, they never happen | 09:21 |
| tristan | What do you mean ? | 09:21 |
| qinusty | If START happens, it isn't skipped | 09:21 |
| tristan | qinusty, that's what I'm saying, currently a START message is terminated by SUCCESS or FAILURE only (and I consider it a bug that it can be terminated with WARNING) | 09:22 |
| qinusty | When I run `bst build .....` and I see `Fetch Queue: processed 0, skipped 3, failed 0` | 09:22 |
| tristan | Yes | 09:22 |
| qinusty | Those skipped jobs, never happen. They're never `START`ed. So there's never a case for the SKIPPED message there | 09:22 |
| qinusty | Unless we want to start and skip them | 09:23 |
| tristan | Those skippings of tasks never have messages | 09:23 |
| qinusty | to be more clear to the user | 09:23 |
| tristan | and they never have START either | 09:23 |
| tristan | qinusty, basically, your introducing a SKIPPED messages, acknowledges the fact that a task can plausibly be skipped in mid processing | 09:24 |
| tristan | If the Queue could not determine before hand that it could be skipped | 09:24 |
| qinusty | Yes, but not necessarily the whole task... See cascache.py where individual remotes are skipped but they are all one job | 09:24 |
| *** jonathanmaw has joined #buildstream | 09:25 | |
| tristan | qinusty, if a pull task has not pulled anything at all, is it processed, skipped or failed ? | 09:25 |
| tristan | I mean, in the abstract, what *should* it be | 09:25 |
| qinusty | I'd say skipped | 09:25 |
| tristan | qinusty, if we agree that it should still be considered *processed*, then I would insist we remove the SKIPPED messages entirely, and only report these things through INFO messages | 09:26 |
| qinusty | I see your point now. You're saying that we shouldn't inform the user of a remote being skipped. But end the task with SKIPPED if all remotes were skipped | 09:26 |
| qinusty | or both | 09:26 |
| qinusty | yes | 09:26 |
| tristan | Right, and timed activities can *ONLY* end with SUCCESS/FAILURE/SKIPPED, while other messages can *NEVER* print START/SUCCESS/FAULIRE/SKIPPED | 09:27 |
| tristan | that disambiguation is paramount to me | 09:27 |
| qinusty | Sorry when I say both | 09:27 |
| qinusty | I mean INFO <REMOTE> has been skipped | 09:28 |
| tristan | right | 09:28 |
| qinusty | ended with SKIPPED Push | 09:28 |
| qinusty | etc | 09:28 |
| tristan | And that needs to be an exception basically | 09:28 |
| qinusty | I agree, I see what you mean now | 09:28 |
| tristan | because if it's skipped, it needs to be bookkept as skipped | 09:28 |
| tristan | right | 09:28 |
| qinusty | I do think perhaps it'd be nice to have the tasks skipped in the initial queue logged though | 09:29 |
| qinusty | it's a bit confusing when you see skipped <n> and you're like.... whaaaat | 09:29 |
| tristan | That would be extremely verbose for no benefit, though | 09:30 |
| tristan | I mean, over verbosity factor certainly outweighs that, IMO | 09:30 |
| tristan | jonathanmaw, please help me with https://gitlab.com/BuildStream/buildstream/issues/612#note_97721977 | 09:32 |
| tristan | jonathanmaw, I need a solve before friday | 09:32 |
| tristan | jonathanmaw, I've been tinkering and investigating this, and have an idea with the following comment | 09:33 |
| tristan | qinusty, I do see your meaning though; it would be nice if every queue status count had a matching message, although it's already a big step up if at least every START terminator is represented by a queue's element status | 09:38 |
| tiagogomes_ | tristan why can't max-jobs be set? freedesktop-sdk is setting this variable | 09:38 |
| *** tiagogomes__ has quit IRC | 09:39 | |
| tristan | tiagogomes_, that is illegal | 09:39 |
| tristan | tiagogomes_, what are they setting it to, 1 ? | 09:39 |
| tiagogomes_ | 1 or 2 | 09:39 |
| tristan | tiagogomes_, it's a mistake that we let it happen, it's what notparallel is for | 09:40 |
| valentind | tiagogomes_, where did you see that? | 09:40 |
| valentind | Ah. on some elements? | 09:40 |
| tiagogomes_ | elements/desktop/llvm6.bst: max-jobs: 2 | 09:40 |
| tiagogomes_ | elements/desktop/shared-mime-info.bst: max-jobs: 1 | 09:40 |
| tristan | if they set it to 2, they are working around a resource issue in a nasty way that we should not be allowing; it was never supposed to be settable | 09:40 |
| valentind | llvm6.bst is because of the memory usage. | 09:40 |
| tristan | right, see that is a nasty workaround | 09:41 |
| valentind | There is no alternative. | 09:41 |
| tristan | that should just be notparallel, and accept it being a bit slower | 09:41 |
| tristan | valentind, ^^^^ | 09:41 |
| tristan | notparallel is what we have exposed for this purpose | 09:41 |
| valentind | No llvm is also very big. | 09:41 |
| tristan | valentind, resource balancing is a much more tricky thing | 09:41 |
| valentind | So making it bit slower... that makes actually the whole thing a lot slower. | 09:42 |
| tristan | valentind, llvm compiles in like 24 minutes on my laptop | 09:42 |
| tristan | Yes | 09:42 |
| tristan | valentind, I really don't care: that is an unsolved problem | 09:42 |
| valentind | Yes I agree. It is unsolved. | 09:42 |
| tristan | what resources are available on your build machine, is *NOT* project data | 09:42 |
| valentind | But setting it to notparallel is not solving it either. | 09:42 |
| tristan | valentind, yes it is | 09:42 |
| tristan | valentind, it builds, it doesnt fail | 09:43 |
| tristan | I mean, that is your workaround | 09:43 |
| tristan | max-jobs should be illegal | 09:43 |
| tristan | the only reason it's there, is to put it in environment-nocache, to allow the automated number to get picked up | 09:43 |
| * paulsher1ood wonders why tristan thinks max-jobs should be illegal, but patch files aren't :-) | 09:44 | |
| tristan | paulsher1ood, I wonder why apples are red or green, but oceans are wet | 09:44 |
| paulsher1ood | heh | 09:44 |
| paulsher1ood | why should max-jobs be illegal, given "we don't dictate terms" | 09:45 |
| paulsher1ood | (you said yesterday) | 09:45 |
| tristan | paulsher1ood, max-job is an API detail, it's not meant to be written, only read | 09:46 |
| tristan | If you set max-jobs to 5, or 4, or 1, it will *not* effect the cache key of your element, assuming that you are using standard plugins | 09:46 |
| tristan | that is exactly what notparallel is for | 09:47 |
| tristan | the number of jobs is either forced to 1 (by notparallel), or it is a number that is not controlled by project data | 09:47 |
| paulsher1ood | hmmm. so no way for user to specify number of jobs? | 09:48 |
| valentind | Yes, but in this case we do not care about the cache key. There are things that build incorrect in parallel. But this is not the case. LLVM builds correctly in parallel. It might fail. But not build incorrectly. | 09:48 |
| tristan | Exactly, the user may only specify that the sources cannot handle parallelism, or not | 09:48 |
| paulsher1ood | tristan: are you sure that bst will make the right decision about njobs in all cases? | 09:49 |
| paulsher1ood | i understand the parallel vs notparallel point | 09:49 |
| tristan | paulsher1ood, No, but tying this to project configuration data is not the answer | 09:49 |
| tristan | Your project does not dictate whether it will be built on a small or big machine | 09:49 |
| paulsher1ood | ok, so how is njobs actually decided? | 09:50 |
| tristan | I.e., that is an unsolved problem, but abusing the API is not the answer | 09:50 |
| valentind | It is going to be abused until solved. | 09:51 |
| tristan | paulsher1ood, currently with an automated guess of cpu_count(), recently you commented on the issue where jjardon made it behave more like YBD | 09:51 |
| tristan | valentind, No, it will be abused until we issue an error at startup because someone abused the API | 09:51 |
| tristan | valentind, then it will will be lived with, and a better solution will have to be found | 09:52 |
| tristan | This wont be the first time, last time it was using "../outside/path" to refer to stuff that is considered project data | 09:53 |
| tristan | We had to correct that | 09:53 |
| valentind | We still use local plugin for the bootstrap phase. We just moved it within the project. | 09:54 |
| valentind | I do not mind, if you force use not to use max-jobs. But we will call ninja with 2 job queues for llvm. | 09:55 |
| tristan | valentind, this is the issue to track and work on: https://gitlab.com/BuildStream/buildstream/issues/185 | 09:55 |
| tristan | valentind, That is an acceptable hack, better than overwriting an automatic variable; that should result in an error | 09:56 |
| tristan | Still it's an unsolved problem, but having a well defined / strict API is important | 09:57 |
| valentind | Yes. For sure, this isIt is unsolved. | 09:57 |
| valentind | How is max-job going to be calculated? | 10:00 |
| tristan | valentind, the way it is currently calculated | 10:00 |
| tristan | _project.py: output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8)) | 10:00 |
| valentind | OK. | 10:01 |
| valentind | I think on our 96 core machines, this is a problem. | 10:01 |
| tristan | valentind, note min() | 10:02 |
| valentind | Ah ok | 10:02 |
| tristan | valentind, there is a related comment around there too | 10:02 |
| valentind | Was it always like that? | 10:02 |
| tristan | No it wasnt | 10:02 |
| valentind | Ah. MAybe we can test without max-jobs then. | 10:02 |
| tristan | jjardon, recently improved it (see backlog, we are just discussing that) | 10:02 |
| valentind | jjardon, do you remember the issue with llvm? Was it a problem because of the number of cores on the builders? | 10:03 |
| tristan | valentind, that's a good idea to test that... it will *still* be an unsolved problem, because it can still happen | 10:03 |
| tristan | of course, max-jobs calculation has to change with remote execution | 10:03 |
| * tristan looks at juergbi and jmac_ | 10:03 | |
| qinusty | tristan, can you think of any other urgent issues which need addressing prior to release? I was looking into https://gitlab.com/BuildStream/buildstream/issues/609 but can't really continue without reproducing this... | 10:04 |
| tristan | and winks | 10:04 |
| valentind | Maybe a better hack (but still a hack) is on the element to query for how much memory the machine has and use an heuristics to lower down the number if needed. | 10:04 |
| tristan | qinusty, my top priority right now is to discuss 612 with jonathanmaw | 10:04 |
| tristan | valentind, I dont know, it could be indeed doable because an element has knowledge of the shape of it's resource consumption, while BuildStream does not | 10:05 |
| tristan | valentind, on the other hand, a way to add metadata for an element to communicate that shape to BuildStream might be an avenue to pursue in the future | 10:06 |
| tristan | megabytes per processor is one attribute, but then it is also not a constant throughout the build | 10:07 |
| tristan | (usually it is only the final link stage which causes the computer to explode) | 10:07 |
| persia | IO bandwidth is a third part of the triad: if we're doing resource sizing, all three should be considered (and they are indeed variable over the course of the build) | 10:08 |
| juergbi | mablanch: I don't see where this issue comes from, can you please provide steps to reproduce? | 10:12 |
| valentind | Is it possible to do checkpointing? I see docker does it. | 10:13 |
| valentind | But it does it for the whole container. Not really what we want. | 10:15 |
| mablanch | juergbi: It was working with HEAD of juerg/wip pointing at 3c9db7d93a2d04be45ca378b02f6e17873ae3bed (just tested again). | 10:17 |
| gitlab-br-bot | buildstream: issue #618 ("Buildstream pulls from remote cache, even when configuration removed") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/618 | 10:18 |
| juergbi | mablanch: thanks for verifying. pretty odd error based on the diff | 10:19 |
| tristan | qinusty, looking at lists, I think all the horrible stuff is either out of the way or invalidated... | 10:20 |
| mablanch | juergbi, I'm using BuildStream from the jmac/remote_execution_client branch, bst-artifact-server from that same branch, BuildGrid from the mablanch/68-bots-multi-endpoints branch and BuildBox from juergbi/wip and building the autotool example from BuildStream documentation (with tweaks for remote-execution). | 10:21 |
| tristan | lemme check the board for a moment | 10:21 |
| qinusty | https://gitlab.com/BuildStream/buildstream/issues/609 seems pretty bad, I just had the same issue with a PULL, indicating that prehaps cascache.py is the issue rather than casserver.py | 10:21 |
| mablanch | juergbi, That's probably not easy to reproduce :) Maybe I can try to send you my CAS content + the BuildGrid command line it's running? | 10:22 |
| juergbi | you mean buildbox? | 10:22 |
| tristan | valentind, can you finish up with https://gitlab.com/BuildStream/buildstream/merge_requests/747 please ? | 10:22 |
| mablanch | juergbi, sorry yes... | 10:22 |
| valentind | Oh ok | 10:22 |
| juergbi | mablanch: if that's possible, that would be great | 10:22 |
| jonathanmaw | tristan: AIUI the problem is that GitSource instantiates its SourceFetchers at fetch-time. This means validating those URLs doesn't happen until fetching, and not at all if tracking. | 10:23 |
| jonathanmaw | I think it would make sense if the SourceFetchers were instantiated at configure-time. | 10:23 |
| jonathanmaw | This is complicated by GitSource being able to discover more submodules at fetch-time, though as said because they won't have aliases, we don't need them to be SourceFetchers presented in get_source_fetchers(). | 10:23 |
| jonathanmaw | I wonder if an appropriate solution would be to have git.py have two different kinds of SourceFetcher: a MainSourceFetcher, which owns the GitMirror for the top-level URL, and GitMirrors for any submodules discovered that aren't part of config; and a SubmoduleSourceFetcher, which owns the GitMirror for a specific submodule. | 10:23 |
| jonathanmaw | I suppose at that point the tricky part is making sure that we update the GitMirrors for the SourceFetchers when update_submodules() is called. | 10:23 |
| tristan | qinusty, this one, would be good to know if it's valid, and is currently unassigned: https://gitlab.com/BuildStream/buildstream/issues/551 | 10:24 |
| qinusty | Woah, I just received like 5 messages from jonathan insatntly | 10:24 |
| tristan | qinusty, which you already worked on | 10:24 |
| tristan | qinusty, jonathanmaw types really fast | 10:24 |
| jonathanmaw | qinusty: yep, I tried to keep my thoughts in order before pasting into IRC | 10:24 |
| qinusty | Ah, thought you were seeing network troubles :D | 10:24 |
| qinusty | I'll check tristan, last time I checked it seemed pretty real and inconsistent | 10:25 |
| tristan | qinusty, if we cannot solve 551, it can be nice to do https://gitlab.com/BuildStream/buildstream/issues/616 | 10:25 |
| tristan | which we were discussing earlier, at least stop making WARNING appear for failed tasks which are going to be retried | 10:25 |
| tristan | qinusty, and then maybe the other thing that is related, the Skip exception idea; having logging much more consistent by those two things is low risk and high value | 10:26 |
| gitlab-br-bot | buildstream: merge request (valentindavid/post_tracking_errors->master: Report processing errors from tracking) #747 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/747 | 10:27 |
| gitlab-br-bot | buildstream: merge request (tpollard/591->master: buildstream/_project.py: Report if project.conf is missing name) #680 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/680 | 10:27 |
| * tristan turns his attention to jonathanmaw :) | 10:27 | |
| gitlab-br-bot | buildstream: merge request (tpollard/483->master: plugins/git.py: Warn if ref is not in given track) #564 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/564 | 10:28 |
| tristan | jonathanmaw, the approach you suggest afaics, will introduce redundant fetching behaviors for unaliased urls, which might be discovered in track (added submodules) | 10:29 |
| tristan | jonathanmaw, i.e.; If we instantiate all the fetchers at configure time; we will still use *those* instantiated fetchers during Source.__do_fetch() | 10:29 |
| tristan | jonathanmaw, and so if one of them is responsible for fetching multiple urls, and fails one, they will all be retried on the next | 10:30 |
| *** abderrahim has joined #buildstream | 10:30 | |
| jonathanmaw | tristan: ah, yes. | 10:30 |
| tristan | jonathanmaw, otherwise, I diskile instantiating them once at configure time, and then strangely allowing them to differ from the originals | 10:30 |
| tristan | (i.e. we should only list them once) | 10:31 |
| tristan | I think that for one, we still need to document Source.get_source_fetchers() to return an iterable, and explain that it will be dynamically called at fetch time | 10:32 |
| *** abderrahim has quit IRC | 10:32 | |
| tristan | i.e. it is currently documented to return a list, which git.py breaks with valentind's patch which fixes it's bug | 10:32 |
| tristan | jonathanmaw, So the approach that I was thinking is to *mandate* that Source.mark_download_url() be called on every URL in the element configuration | 10:33 |
| tristan | jonathanmaw, we could do that by keeping a list around, but unfortunately it would be a late BUG | 10:33 |
| tristan | then again, it would not be so bad because it would be a BUG, and it would only need to be fixed once, for a misbehaving plugin | 10:34 |
| mablanch | juergbi, Does buildbox download the input data to it's local cache before executing the command? (ie. to you need the remote CAS content or the local one?) | 10:35 |
| tristan | jonathanmaw, i.e.; After configure, we seal this list of known user inputted urls discovered at `Plugin.configure()` time, and then later we assert that urls passed to translate_url and mark_download_url on the same source are in that list | 10:35 |
| tristan | jonathanmaw, or at least, for the ones which have an alias | 10:35 |
| juergbi | mablanch: it currently only downloads on demand. everything up to the point of error might be local | 10:35 |
| *** abderrahim has joined #buildstream | 10:37 | |
| tristan | valentind, asides from https://gitlab.com/BuildStream/buildstream/merge_requests/747, are there any merge requests pending from you that I need to think about for 1.2 ? | 10:37 |
| tristan | tiagogomes_, I think there was a fairly simple one from you that I commented on... trying to remember what it was | 10:38 |
| tiagogomes_ | was it the protected variables stuff? | 10:38 |
| tristan | Ahhh yeah maybe it was that | 10:39 |
| tristan | tiagogomes_, the only other one I can think of atm is the rather complex reworking of the scheduler | 10:39 |
| tristan | jonathanmaw, a separate subject; I merged this this morning: https://gitlab.com/BuildStream/buildstream/merge_requests/753 | 10:41 |
| jonathanmaw | tristan: okay, so we end up with two kinds of warning: 1) In configure, mark_download_url or translate_url were called on a URL without an alias, an error on the .bst writer's part. 2) Later, mark_download_url or translate_url was called on an aliased URL that hasn't been registered in configure, an error on the plugin author's part | 10:41 |
| valentind | tristan, No other merge request. I see you merged 748. | 10:41 |
| tiagogomes_ | I don't think that we should land the protected variables just before the release, as it will break fdsdk | 10:41 |
| tristan | jonathanmaw, can you think of any reason why those filter tests should have to loop over every source kind ? | 10:41 |
| tristan | tiagogomes_, yeah... but then I still want to land it in 1.2.1; post fixing freedesktop-sdk | 10:42 |
| tristan | It's not an API change, but it's an enforcement of expected behavior | 10:42 |
| toscalix | tow new pages structure merged. 1.2 feature page is next | 10:43 |
| toscalix | s/tow/two | 10:43 |
| tristan | jonathanmaw, well, the warning in configure is the reason we noticed that we are not seeing all the relevant data in `Plugin.configure()` | 10:43 |
| tiagogomes_ | toscalix can you make sure it builds in the CI? | 10:43 |
| tristan | jonathanmaw, but yeah, that warning will eventually be added... and what you list as (2), will not be a warning, it will be an `assert` directly in the code; with a custom message | 10:44 |
| mablanch | juergbi, How should I send you this? | 10:44 |
| *** awacheux[m] has joined #buildstream | 10:45 | |
| *** theawless[m] has joined #buildstream | 10:45 | |
| *** waltervargas[m] has joined #buildstream | 10:45 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/roundtripping_only_when_modified-1.2->bst-1.2: Disable round-tripping when element is not modified) #760 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/760 | 10:45 |
| *** tlater[m] has joined #buildstream | 10:45 | |
| *** ssssam[m] has joined #buildstream | 10:45 | |
| tristan | jonathanmaw, something like `assert marked_alias in self.__expected aliases "Plugin did not call translate_url() on all input URLs at configure time"` | 10:45 |
| toscalix | tiago, no, I can't. I am unfamiliar with setting Gitlab CI. Can you point me to some docu to see if I have time to learn before the release? | 10:45 |
| *** rafaelff[m] has joined #buildstream | 10:45 | |
| *** albfan[m] has joined #buildstream | 10:45 | |
| *** abderrahim[m] has joined #buildstream | 10:45 | |
| *** asingh_[m] has joined #buildstream | 10:45 | |
| *** mattiasb has joined #buildstream | 10:45 | |
| *** cgmcintyre[m] has joined #buildstream | 10:45 | |
| *** kailueke[m] has joined #buildstream | 10:45 | |
| *** krichter[m] has joined #buildstream | 10:45 | |
| *** jjardon[m] has joined #buildstream | 10:45 | |
| *** segfault3[m] has joined #buildstream | 10:45 | |
| *** connorshea[m] has joined #buildstream | 10:45 | |
| *** pro[m] has joined #buildstream | 10:45 | |
| *** dineshdb[m] has joined #buildstream | 10:45 | |
| *** oknf[m] has joined #buildstream | 10:45 | |
| juergbi | mablanch: depends on how big it is, i guess. if it's small email otherwise maybe on fs (CT) or a public file upload site | 10:46 |
| tristan | jonathanmaw, the result of that will be a BUG message, indicating that the plugin in question has a bug. As I mentioned before, I don't like that the bug can happen late like this; but on the bright side, it only needs to be fixed once for a misbehaving plugin and should never happen after that | 10:47 |
| toscalix | tiagogomes_: I see two folders: current pages and pages | 10:48 |
| toscalix | I see that pages has an extra line included title: xxxx | 10:48 |
| tristan | jonathanmaw, there also needs to be an error at load time for a URL that refers to an undefined alias | 10:48 |
| toscalix | I mean, the pages inside the folder pages | 10:48 |
| toscalix | are those the ones you render? | 10:48 |
| valentind | tiagogomes_, have you asked for SSL certificates for the website? | 10:49 |
| tiagogomes_ | toscalix The CI is already in place. Pages needs to go in the pages folder and have at least a "title:" attribute. The tool used to generate those is Pelican | 10:49 |
| tristan | jonathanmaw, but that's rather orthogonal to the subject, it's only relevant because we need to ensure all data has been seen at Plugin.configure() time | 10:49 |
| tristan | jonathanmaw, in order to ensure we fail early for things we can fail early for | 10:49 |
| toscalix | tiagogomes_: I would like to avoid having two folders, pages and current pages. Should I move all the content to pages and erase current_pages folder ? | 10:50 |
| tristan | toscalix, you trigger CI by creating a merge request | 10:50 |
| tristan | Ah, so you have some weird temporary setup it seems, while the stuff toscalix added gets added to the generated website | 10:51 |
| toscalix | MR for changes in pages folder? | 10:51 |
| jonathanmaw | tristan: looking at the filter tests now, I can't see any reason to use different source kinds. wrt the URL errors, acknowledged. | 10:51 |
| * tristan returns to "keeping his nose out of this website stuff" :) | 10:51 | |
| tiagogomes_ | toscalix I will move the pages from current pages to pages | 10:51 |
| toscalix | and erase the current_pages folder please | 10:52 |
| tristan | toscalix, yes, iiuc; in any case looks like tiagogomes_ will take care of that for you | 10:52 |
| tristan | toscalix, and in the future, always creating a merge request ensures that CI passes before merge | 10:52 |
| toscalix | on pages folder, got it | 10:52 |
| tristan | toscalix, even if it's a merge request that you self approve, doesnt matter | 10:52 |
| toscalix | tiagogomes_: you will have to add the title attribute to the pages I have created | 10:53 |
| tiagogomes_ | I know | 10:53 |
| tiagogomes_ | Btw this is a thing http://buildstream.build/ . I am working on https | 10:53 |
| toscalix | tiagogomes_: I will give you the sitemap end of today | 10:54 |
| valentind | tiagogomes_, I see that links on the page have full urls, and it redirects to buildstream.gitlab.io. This is maybe something you want to look at at some point. | 10:54 |
| mablanch | juergbi, Ok, I've emailed it, it's quite small if I don't include the remote cache content. | 10:54 |
| tristan | valentind, nice spot | 10:55 |
| paulsher1ood | tiagogomes_: i evolved devcurmudgeon.com and trustable.io from that template, and they look better imo. you'd be welcome to take my tweaks from either of those projects | 10:55 |
| gitlab-br-bot | buildstream: issue #619 ("Fail messages without logfile set cause traceback") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/619 | 10:56 |
| juergbi | mablanch: ta | 10:57 |
| tristan | valentind, thanks for backporting https://gitlab.com/BuildStream/buildstream/merge_requests/760 :) | 10:57 |
| valentind | tristan, I think !747 is ready. | 11:00 |
| tiagogomes_ | Can some one approve https://gitlab.com/BuildStream/website/merge_requests/7 I can't merge without approval | 11:02 |
| *** tpollard has quit IRC | 11:02 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/roundtripping_only_when_modified-1.2->bst-1.2: Disable round-tripping when element is not modified) #760 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/760 | 11:03 |
| jonathanmaw | tristan: is there more that needs to be discussed for #612? | 11:05 |
| tristan | jonathanmaw, right, just stepped out for a sec | 11:06 |
| tristan | jonathanmaw, so; I guess we have a decent approach ? Just... since we use the first input from `Source.translate_url()` as input to later be used at track time, regardless of the presence of SourceFetchers, do you think the plan will work well ? | 11:07 |
| tristan | jonathanmaw, Have any other ideas which don't require adding even more API surface ? | 11:07 |
| tristan | jonathanmaw, On my part, I think I'm satisfied with the solution, I guess we just need to document it well | 11:08 |
| tristan | jonathanmaw, beyond that, I think I can try to do it tomorrow, but the only thing which really blocks the release is documenting the API contract that will work | 11:08 |
| jonathanmaw | tristan: hm, I think it would require the SourceFetcher to have access to the Source's list of previously-marked URLs, and we don't currently mandate that SourceFetchers have the Source | 11:08 |
| tristan | jonathanmaw, how so ? | 11:09 |
| tristan | jonathanmaw, So just to make sure we're on the same page; I'm going to basically document that "A source plugin *must* call either Source.mark_download_url() OR Source.translate_url() on ALL urls that are present in the Source's configuration during the Plugin.configure()` method" | 11:10 |
| toscalix | paulsher1ood: we will have a look, thianks. We will need all the help we can get when it comes to contents and design | 11:10 |
| tristan | toscalix, tiagogomes_; btw, not sure where this fits into the design of the site; but I think this will be useful for the site | 11:11 |
| tristan | toscalix, tiagogomes_: <object data="https://buildstream.gitlab.io/buildstream/_static/release.svg" type="image/svg+xml"/> | 11:11 |
| tristan | toscalix, tiagogomes_: <object data="https://buildstream.gitlab.io/buildstream/_static/snapshot.svg" type="image/svg+xml"/> | 11:12 |
| tristan | Latest release badge, and latest development snapshot badge | 11:12 |
| tristan | Always up to date from BuildStream CI | 11:12 |
| jonathanmaw | tristan: the SourceFetchers have access to aliased URLs, and call their own mark_download_url() method. SourceFetchers are not required to be instantiated in configure(), so if the plugin author is being lax then URLs that are only used in SourceFetchers may not have been checked in configure() | 11:12 |
| tristan | Using <object/> is very important, if you use <img/> then you dont get the link encoded in the SVG | 11:12 |
| tristan | jonathanmaw, Exactly, that's why we hold on to them in the Source at configure time, and make the assertion, later raising a BUG | 11:13 |
| toscalix | tristan: included here: https://gitlab.com/BuildStream/website/issues/3 | 11:14 |
| jonathanmaw | tristan: by "them" do you mean the URLs or the SourceFetchers? | 11:14 |
| tristan | jonathanmaw, So, during configure, not only do we hold onto the first Source.mark_download_url() url, we hold on to a list of ALL of them, and for any of them which had an alias, we assert that in the core later, when a SourceFetcher calls SourceFetcher.mark_download_url() | 11:14 |
| tristan | jonathanmaw, I mean the URLs | 11:15 |
| tristan | jonathanmaw, So basically, we don't assert for non-aliased URLs, they can be dynamically discovered; but for anything with an alias at least; we issue a BUG (by assert statement), and force that plugin to not be lax | 11:15 |
| jonathanmaw | tristan: okay, though if the source is holding on to the list of all urls marked during configure, then SourceFetcher must be able to access that list, i.e. it must have access to the Source | 11:16 |
| juergbi | mablanch: please try again with updated branch | 11:16 |
| jonathanmaw | which we don't currently ensure, AIUI | 11:16 |
| jonathanmaw | a minor API change that SourceFetcher.__init__() now includes "source" in the arg, and that SourceFetcher authors are expected to chain up with a passed-in source should be sufficient, I think | 11:19 |
| tristan | jonathanmaw, Ah, but under the hood that is quite a trivial implementation detail | 11:19 |
| tristan | jonathanmaw, there need not be any change to the API for this, and the Plugins should not see that | 11:19 |
| tristan | jonathanmaw, although, while there is not really much reason to deny it from the API, there is neither any need to introduce it (it can be SourceFetcher._set_source() under the hood) | 11:20 |
| tristan | jonathanmaw, Ohhh, you mean the SourceFetcher() instantiation is active and not passive ! | 11:20 |
| mablanch | juergbi, Tests all pass and building a simple example works fine! Cheers! | 11:21 |
| *** jonathanmaw_ has joined #buildstream | 11:21 | |
| juergbi | thanks for confirming | 11:21 |
| jonathanmaw | okay | 11:21 |
| *** jonathanmaw has quit IRC | 11:21 | |
| *** jonathanmaw_ is now known as jonathanmaw | 11:22 | |
| tristan | jonathanmaw, Indeed, the Git source actually calls the mark_download_url() from there... *however*, it can still be asserted | 11:22 |
| tristan | I guess we need not modify that | 11:22 |
| tristan | jonathanmaw, we can still make the assertion after instantiating the SourceFetcher, and checking what URL it was marked with against our list | 11:23 |
| tristan | jonathanmaw, around this line: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/source.py#L873 | 11:24 |
| jonathanmaw | tristan: ah, I see | 11:24 |
| jonathanmaw | It sounds like I've been an effective rubber duck, if nothing else | 11:26 |
| tristan | jonathanmaw, Well, it's your area; I need someone who is confident in "owning source mirroring" to bounce ideas off of for this | 11:27 |
| tristan | that person is currently you :) | 11:28 |
| toscalix | tiagogomes_: can you track your work on the website here, please? https://gitlab.com/BuildStream/website/issues/9 | 11:28 |
| jonathanmaw | tristan: a bouncy rubber duck! even better! | 11:28 |
| toscalix | tiagogomes_: add in the description the main tasks, so the next person that takes over the web responsibility knows what you did? | 11:29 |
| gitlab-br-bot | buildstream: merge request (coldtom/strip-rules->master: WIP: Upstream freedesktop-sdk strip rules) #750 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/750 | 11:30 |
| tiagogomes_ | ok | 11:31 |
| tristan | valentind, ah I didnt notice !747, indeed, seems gitlab didnt say "this line of code changed" so the discussion page remained uneffected by your changes | 11:35 |
| gitlab-br-bot | buildstream: issue #533 ("Warning instead of error when failing to write tracking results") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/533 | 11:36 |
| gitlab-br-bot | buildstream: merge request (valentindavid/post_tracking_errors->master: Report processing errors from tracking) #747 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/747 | 11:36 |
| tristan | valentind, please backport https://gitlab.com/BuildStream/buildstream/merge_requests/747 to 1.2 and merge :) | 11:36 |
| tristan | Nice... it's all coming together !!! | 11:37 |
| gitlab-br-bot | buildstream: merge request (coldtom/strip-rules->master: WIP: Upstream freedesktop-sdk strip rules) #750 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/750 | 11:37 |
| gitlab-br-bot | buildstream: issue #620 ("Adjust Source / SourceFetcher API contract") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/620 | 11:44 |
| tristan | jonathanmaw, I will take care of https://gitlab.com/BuildStream/buildstream/issues/620 tomorrow | 11:45 |
| tristan | And will release over the weekend, as we agreed - the actual release happens end of this week; and the post release shinanigans of officialness and websity promotional charades happen on Sept 5 | 11:46 |
| juergbi | jonathanmaw: don't know whether you're already aware but cache_size write is not atomic, which constantly breaks things here on master | 11:54 |
| tristan | valentind, you are the winner so far at 19 minutes and 5 seconds (https://gitlab.com/BuildStream/buildstream/pipelines/29033000) | 11:54 |
| tristan | Eeek | 11:54 |
| tristan | juergbi, Would utils.save_file_atomic() make sense for it ? or must it use a lock I guess ? | 11:55 |
| juergbi | save_file_atomic would definitely fix the biggest issue | 11:56 |
| juergbi | I'm not familiar enough with how it's implemented to know whether lock file would be required for proper operation | 11:56 |
| juergbi | right now I get parser errors, presumably due to the file being written and read concurrently | 11:57 |
| tristan | juergbi, from my abstract understanding of it, it gets written when the cache grows; and that can happen concurrently - which means that if two processes just use save_file_atomic(), added bytes will be missed | 11:57 |
| juergbi | FAILURE Size '' parsed from '.../.cache/buildstream/artifacts/cache_size' was not an integer | 11:57 |
| tristan | So we would need to read/add/write atomically | 11:58 |
| juergbi | yes, that could well be the case. however, right now it's unusable while save_file_atomic() would at least make a proper fix less urgent | 11:58 |
| tristan | juergbi, definitely agree | 11:58 |
| tristan | Ok so... does someone already have a patch for this or shall I shove one into master and bst-1.2 ? | 11:59 |
| tristan | Somehow looking at that line of code, I feel like I already commented on this requiring utils.save_file_atomic() | 12:00 |
| juergbi | I don't have a patch yet. have gone back to older version as stop gap | 12:00 |
| tristan | well, whatever | 12:00 |
| tristan | I'll push an MR pronto | 12:00 |
| juergbi | tahnks | 12:01 |
| gitlab-br-bot | buildstream: merge request (tristan/atomic-cache-size-file->master: _artifactcache/artifactcache.py: Write the cache_size file atomically) #762 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/762 | 12:04 |
| juergbi | oh, and that's actually stored as failed build, so I have to explicitly retry | 12:06 |
| gitlab-br-bot | buildstream: merge request (valentindavid/post_tracking_error-1.2->bst-1.2: Report processing errors from tracking) #761 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/761 | 12:10 |
| gitlab-br-bot | buildstream: merge request (tristan/atomic-cache-size-file-1.2->bst-1.2: _artifactcache/artifactcache.py: Write the cache_size file atomically) #763 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/763 | 12:11 |
| tristan | juergbi, fyi, it would be good if you could take a look at this: https://gitlab.com/BuildStream/buildstream/issues/615 | 12:13 |
| tristan | juergbi, I expect it doesnt happen frequently, but when it does we fail badly and hang around for hours | 12:13 |
| juergbi | hm, maybe bad handling of connection issues | 12:15 |
| valentind | tristan, I have had this problem as well. There was one artifact that was taking much time to download. Though not hours. But like 10 minutes. I expected it to be done in few seconds. | 12:15 |
| juergbi | probably need to figure out how to simulate this in a test case | 12:15 |
| tristan | juergbi, yeah not sure; it looks like it's stuck in a very forgiving retry loop, where maybe something actually timed out | 12:16 |
| valentind | Ah this one does not exit? Interesting. | 12:16 |
| tristan | valentind, yeah it's not the performance issue; that is separate it seems | 12:17 |
| tristan | juergbi, on a side note about simulation and CI... I've been wondering "What is the plan for CI of remote execution ?" | 12:18 |
| tristan | juergbi, just throwing that out on the table, I won't be thinking of it for another week or so hehe | 12:18 |
| * tristan has to run now... but think we did very well this week ! | 12:18 | |
| juergbi | it's not completely clear yet. we have relatively contained tests for most of the individual components. I mentioned this as open point for the remote execution client MR | 12:19 |
| juergbi | a test environment is being setup, but we'll have to see how we can integrate this with CI | 12:19 |
| tristan | I think we definitely need some kind of end-to-end tests; but I don't know how they will work on GitLab | 12:19 |
| tristan | yeah | 12:20 |
| tristan | We'll also need OSX and Windows runners for those other issues | 12:20 |
| *** tristan has quit IRC | 12:24 | |
| gitlab-br-bot | buildstream: issue #601 ("PermissionError when calling `vdirectory.import_files()`") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/601 | 12:27 |
| gitlab-br-bot | buildstream: merge request (tristan/atomic-cache-size-file->master: _artifactcache/artifactcache.py: Write the cache_size file atomically) #762 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/762 | 12:27 |
| gitlab-br-bot | buildstream: merge request (tristan/atomic-cache-size-file-1.2->bst-1.2: _artifactcache/artifactcache.py: Write the cache_size file atomically) #763 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/763 | 12:32 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 12:36 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 12:38 |
| valentind | Gitlab is giving me a captcha for anything I am doing now. | 12:42 |
| coldtom | me too, valentind | 12:43 |
| coldtom | i had to do 2 captchas for checking some items on a checklist earlier | 12:43 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 12:45 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 12:56 |
| *** xjuan has joined #buildstream | 13:25 | |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 13:28 |
| *** alatiera_ has quit IRC | 13:32 | |
| gitlab-br-bot | buildstream: merge request (bzr_fix->master: Replacing string 'bzr' with value from host tools) #764 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/764 | 13:41 |
| Nexus | can someone give https://gitlab.com/BuildStream/buildstream/merge_requests/764/ a quick look please? it already got approval from tristan earlier https://gitlab.com/BuildStream/buildstream/merge_requests/726#note_97629122 but it'd be nice just o have a quick look before it can be merged | 13:43 |
| *** finn has joined #buildstream | 13:43 | |
| gitlab-br-bot | buildstream: merge request (bzr_fix->master: Replacing string 'bzr' with value from host tools) #764 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/764 | 13:45 |
| *** finn has quit IRC | 13:45 | |
| *** finn_ has quit IRC | 13:46 | |
| *** finn has joined #buildstream | 13:46 | |
| gitlab-br-bot | buildstream: merge request (jmac/remote_execution_client->master: WIP: Remote execution client) #626 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/626 | 13:58 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 14:10 |
| *** mohan43u has quit IRC | 14:18 | |
| flatmush | Any ideas what could be causing this obscure error in a docker runner for a repository that builds fine locally? | 14:32 |
| flatmush | https://gitlab.com/trustable/distros/minimal-distro/-/jobs/93411861 | 14:32 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 14:33 |
| *** alatiera has joined #buildstream | 14:34 | |
| *** alatiera is now known as alatiera_ | 14:36 | |
| *** dtf has joined #buildstream | 14:48 | |
| qinusty | I'm unable to run my test sequence locally with my setup. I'm not sure what's going on but I seem to get my pytest/pylint combo working for like 2-3 days before it breaks again | 14:53 |
| toscalix | tiago, can we have a short talk today to sync up? | 14:53 |
| toscalix | tiagogomes_: ^ | 14:54 |
| qinusty | Now I'm getting issues _pytest.python cannot import Package. Apparently related to the use of __init__.py files within test directories (e.g. like in our test plugins) | 14:54 |
| *** xjuan has quit IRC | 14:55 | |
| qinusty | Ha | 14:55 |
| qinusty | version flatmush | 14:55 |
| qinusty | coldtom will like this one | 14:55 |
| flatmush | 1.1.7 | 14:55 |
| qinusty | You've got a recursive variable | 14:55 |
| qinusty | We detect and prevent them in 1.2. But if you look at your "long path" | 14:56 |
| qinusty | lib/debug/tools/lib/debug/tools/lib/debug/tools...... | 14:56 |
| flatmush | what would that look like? I'm making a minimal distro based on definitions, so I doubt I introduced that variable | 14:56 |
| qinusty | Um, coldtom saw the issue when modifying the prefix variable | 14:57 |
| flatmush | I assume it's somewhere in here: https://gitlab.com/trustable/distros/minimal-distro/blob/master/elements/gnu-toolchain/stage1-gcc.bst | 14:57 |
| qinusty | Basically it happens when variable a substitues a variable with "%{a}" in it | 14:57 |
| flatmush | in a bst file presumably? | 14:58 |
| qinusty | yup, lemme try building with 1.2 | 14:59 |
| qinusty | It'll flag it | 14:59 |
| flatmush | also, it builds fine with 1.1.7 in a vm locally | 15:00 |
| qinusty | Weird one | 15:00 |
| qinusty | it definitely looks like a recursive problem. To some extent, its a very repetitive path | 15:00 |
| flatmush | I imagine it's an issue with my docker image, but I don't know where to start figuring out what could cause it | 15:00 |
| flatmush | I was assuming something weird with symlinks, but the recursive variable thing is another possible | 15:01 |
| qinusty | What're you building flatmush? which element | 15:01 |
| coldtom | qinusty: gnu-toolchain/stage1-gcc.bst | 15:02 |
| *** mohan43u has joined #buildstream | 15:05 | |
| flatmush | seems like the issue is in project.conf | 15:08 |
| flatmush | but that's just copied from definitions with no changes | 15:08 |
| qinusty | Where does the debugdir variable come from? | 15:08 |
| coldtom | flatmush: i think the strip-binaries command is somehow defining debugfile recursively? | 15:08 |
| coldtom | debugdir is a buildstream default | 15:08 |
| qinusty | So it is | 15:09 |
| coldtom | that's not recursively defined though | 15:10 |
| qinusty | yup | 15:10 |
| qinusty | it is | 15:10 |
| qinusty | well... | 15:10 |
| qinusty | debugfile is `debugfile="%{install-root}%{debugdir}/$(basename "$1")"` | 15:11 |
| flatmush | that's not a definition though, it's part of a bash command | 15:11 |
| flatmush | unless I'm reading wrong | 15:11 |
| qinusty | hmmm | 15:11 |
| qinusty | debugdir: "%{libdir}/debug" | 15:11 |
| coldtom | that should (and does) resolve correctly, as you can see in the logs | 15:11 |
| coldtom | debugdir resolves to /tools/lib/debug | 15:12 |
| qinusty | Expands to `/buildstream-install/usr/lib/debug/$(basename "$1") | 15:13 |
| *** noisecell has quit IRC | 15:14 | |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 15:19 |
| qinusty | coldtom, x220 isn't liking building gcc | 15:33 |
| gitlab-br-bot | buildstream: issue #594 ("pull step reported a SUCCESS even if the a artifact was not found in the cache server") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/594 | 15:35 |
| gitlab-br-bot | buildstream: merge request (Qinusty/skipped-rework->master: Add SkipError for indicating a skipped activity) #765 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/765 | 15:39 |
| qinusty | benschubert, I picked up something from a discussion with tristan without looking at any issue. Submitting my MR, I've looked through issues and found https://gitlab.com/BuildStream/buildstream/issues/614 which is assigned to you and mostly covered by my MR. I hope I've not just pushed into your work D: | 15:39 |
| qinusty | Could you take a look at the MR and let me know what you think? https://gitlab.com/BuildStream/buildstream/merge_requests/765 | 15:39 |
| gitlab-br-bot | buildstream: merge request (bzr_fix->master: Replacing string 'bzr' with value from host tools) #764 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/764 | 15:47 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 15:57 |
| gitlab-br-bot | buildstream: merge request (Qinusty/retries-should-fail->master: Retries log as failures) #766 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/766 | 16:06 |
| gitlab-br-bot | buildstream: merge request (Qinusty/retries-should-fail->master: Retries log as failures) #766 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/766 | 16:07 |
| tiagogomes_ | toscalix do still want that talk | 16:11 |
| toscalix | tomorrow morning | 16:11 |
| tiagogomes_ | ok | 16:11 |
| gitlab-br-bot | buildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/726 | 16:12 |
| qinusty | Any luck figuring out the problem flatmush? | 16:14 |
| qinusty | Anyone fancy reviewing https://gitlab.com/BuildStream/buildstream/merge_requests/766? | 16:20 |
| benschubert | qinusty: that seems good! No worries, I was going to look at it tomorrow. You can assign yourself to the issue :) | 16:20 |
| benschubert | The only thing I would do differently is https://gitlab.com/BuildStream/buildstream/merge_requests/765/diffs#1fbe2f0374c772cfd0b9ec5d4df332f28be9910e_410_409 where I would except the two exceptions separately :) | 16:21 |
| qinusty | hmmm, you're probably right there it's around to same number of lines to print the message within it's own except | 16:22 |
| qinusty | and readability etc | 16:23 |
| toscalix | tiagogomes_: one important point will be how to create ToC with pelican | 16:25 |
| toscalix | if we can use a plugin, fine. If not, we will need to use HTML syntax for titles | 16:25 |
| tiagogomes_ | I think the codethink website is mostly html | 16:25 |
| gitlab-br-bot | buildstream: merge request (Qinusty/skipped-rework->master: Add SkipError for indicating a skipped activity) #765 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/765 | 16:26 |
| gitlab-br-bot | buildstream: merge request (Qinusty/skipped-rework->master: Add SkipError for indicating a skipped activity) #765 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/765 | 16:32 |
| toscalix | tiagogomes_: if you can investigate and find out, it might save us some effrt next week | 16:33 |
| toscalix | tables do not see to have borders either | 16:33 |
| toscalix | another point to investigate how to do them or if there is a plugin we can use | 16:34 |
| toscalix | otherwise... html | 16:34 |
| gitlab-br-bot | buildstream: issue #621 ("Unhandled exception when terminating failed build") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/621 | 16:35 |
| tiagogomes_ | toscalix which license would you like for the website | 16:56 |
| *** xjuan has joined #buildstream | 16:59 | |
| adds68 | Where are config.log files stored? | 17:13 |
| *** rdale has quit IRC | 17:17 | |
| toscalix | tiagogomes_: I need to discuss that with tristan to ensure the user docu and the website does not have incompatible licenses | 17:35 |
| toscalix | I do not see license file in the docu | 17:36 |
| toscalix | it has copyright but no license, which is nt good | 17:38 |
| toscalix | tiagogomes_: needs a discussion | 17:39 |
| *** jonathanmaw has quit IRC | 17:40 | |
| toscalix | I would like to go for a FSF one for docu, rather than CC one, so code-docu is easier to handle | 17:40 |
| *** bochecha has quit IRC | 17:42 | |
| toscalix | tiagogomes_: https://gitlab.com/BuildStream/website/merge_requests/11 if I merge it... I call it a day | 17:52 |
| *** toscalix has quit IRC | 18:01 | |
| jjardon | toscalix there is an old MR to add a license to the docs: https://gitlab.com/BuildStream/buildstream/merge_requests/336 | 18:43 |
| gitlab-br-bot | buildstream: merge request (tpollard/591->master: buildstream/_project.py: Report if project.conf is missing name) #680 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/680 | 19:14 |
| *** leopi has quit IRC | 19:28 | |
| *** bochecha has joined #buildstream | 19:56 | |
| *** bochecha has quit IRC | 19:58 | |
| *** bochecha has joined #buildstream | 19:59 | |
| *** bochecha has quit IRC | 20:20 | |
| *** bochecha has joined #buildstream | 20:20 | |
| *** alatiera_ has quit IRC | 20:35 | |
| gitlab-br-bot | buildstream: merge request (tpollard/591->master: buildstream/_project.py: Report if project.conf is missing name) #680 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/680 | 20:52 |
| *** bochecha has quit IRC | 20:57 | |
| *** bochecha has joined #buildstream | 20:57 | |
| gitlab-br-bot | buildstream: issue #591 ("Missing required key on project.conf causes wrong provenance file name on error message") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/591 | 21:29 |
| *** bochecha has quit IRC | 21:50 | |
| *** bochecha has joined #buildstream | 22:13 | |
| *** bochecha has quit IRC | 23:43 | |
| *** bochecha has joined #buildstream | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
