| *** mcatanzaro has joined #buildstream | 03:29 | |
| *** mcatanzaro has quit IRC | 04:49 | |
| *** jude has joined #buildstream | 07:36 | |
| *** tristan has joined #buildstream | 07:41 | |
| *** ernestask has joined #buildstream | 07:47 | |
| *** valentind has joined #buildstream | 09:04 | |
| *** noisecell has joined #buildstream | 09:20 | |
| gitlab-br-bot | buildstream: issue #184 ("Add additional information to metadata/log file") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/184 | 09:50 |
|---|---|---|
| *** valentind has quit IRC | 09:51 | |
| jjardon[m] | Hi, is max-jobs part of the cache key? | 10:04 |
| tristan | jjardon[m], no. | 10:05 |
| tristan | jjardon[m], we have a weird little hacky mechanism to prevent that | 10:06 |
| jjardon[m] | mmm, thats bad | 10:06 |
| tristan | jjardon[m], see 'environment-nocache' setting in plugins like http://buildstream.gitlab.io/buildstream/elements/autotools.html#module-elements.autotools | 10:06 |
| tristan | jjardon[m], max-jobs should never be in the cache key | 10:07 |
| jjardon[m] | I think | 10:07 |
| tristan | jjardon[m], if you want to specify that the build can not handle parallelism in the build, use notparallel for that. | 10:07 |
| tristan | those are different things | 10:08 |
| jjardon[m] | ah, there is a different option to disable paralellism; that makes things better | 10:08 |
| jjardon[m] | but what about this use case: https://gitlab.com/baserock/ybd/issues/249 | 10:09 |
| tristan | jjardon[m], that is the use case which this covers | 10:10 |
| jjardon[m] | I mean, a system can build with 6 cores fine but fail with 8 cores | 10:10 |
| tristan | oh, that's not a use case. | 10:10 |
| jjardon[m] | not because parallelism problems, but because it gets out of RAM memory or something else | 10:11 |
| tristan | well, you can file a bug, I dont think it's immensely high priority since the cases of this must be few, and can be mitigated with notparallel | 10:12 |
| tristan | also, I'm not convinced that that out of resource conditions are things you want to fix with project configuration | 10:13 |
| tristan | rather, after typing that, I'm now quite convinced they are not the kind of thing you want to handle with project configuration | 10:13 |
| *** jonathanmaw has joined #buildstream | 10:13 | |
| jjardon[m] | rigth, I think having a nonparallel option cover most use cases; you are rigth that maybe project resources need to be fixed in another way. I will file a issue later so at least we have the problem into account | 10:15 |
| tristan | jjardon[m], yes please, always good to have on the radar | 10:15 |
| tristan | jjardon[m], what I mean by that is; the definition of how to build the project is not connected to the machine and resources used to build it | 10:16 |
| tristan | thus it should not differ for that sake | 10:16 |
| tristan | jjardon[m], plausible approaches include site related configuration in user config files; or crazy stuff like automated detection of OOM conditions causing the build to fail and retrying with less jobs | 10:17 |
| tristan | jjardon[m], also I think that is a very rare case you must have ran into | 10:19 |
| tristan | normally the OOM condition is not parallel, but at the final link stage of something huge | 10:19 |
| jjardon[m] | tristan: indeed; only for llvm compiling in ARM | 10:19 |
| jjardon[m] | tristan: I have to set max-jobs: 6; it will fail otherwise | 10:20 |
| tristan | heh | 10:20 |
| jjardon[m] | (aarch64 mahine is 8 cores, 8GB RAM0 | 10:20 |
| jjardon[m] | context: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/merge_requests/60 | 10:20 |
| tristan | jjardon[m], you *can* work around it and force it directly in the build-commands though, fwiw | 10:21 |
| tristan | not really an ultimate solution for this vector of issue | 10:22 |
| *** ssam2 has joined #buildstream | 10:22 | |
| jjardon[m] | I think max-jobs is a bit cleaner, so I can do a conditional for only ARM arches and the build-commands will still be the same | 10:22 |
| tristan | but not horrible either | 10:22 |
| tristan | max-jobs is automatically set by buildstream | 10:22 |
| tristan | and not considered in cache key intentionally because of that. | 10:23 |
| tristan | So that is afaics not an option in any scenario. | 10:23 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 10:59 |
| gitlab-br-bot | buildstream: merge request (sam/local-source-cachekey-fix->master: utils.py: Sort the results of list_relative_paths()) #216 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/216 | 11:00 |
| tristan | ssam2, thanks for comment, was going to ask if you'd like to look at it :) | 11:01 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key-backport->bst-1.0: Fix local source cache key backport) #218 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/218 | 11:09 |
| tristan | Well | 11:16 |
| tristan | That didnt work | 11:16 |
| tristan | ssam2, can you try the fix-local-source-cache-key branch ? | 11:24 |
| tristan | ssam2, and just run ./setup.py test --addopts 'tests/cachekey/' | 11:24 |
| tristan | Right now I'm a bit at a loss, gitlab behaves differently, but gitlab is gitlab | 11:24 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 11:27 |
| tristan | certainly some randomness going on: https://gitlab.com/BuildStream/buildstream/pipelines/15989477 | 11:27 |
| tristan | pytest_linux fails, pytest_unix passes | 11:27 |
| tristan | lets see how a single sorted list behaves https://gitlab.com/BuildStream/buildstream/pipelines/15990791 | 11:28 |
| tristan | equally fails | 11:29 |
| ssam2 | tristan, works for me locally | 11:35 |
| ssam2 | this is bizarre | 11:35 |
| tristan | yes | 11:39 |
| tristan | the unix thing is SKIPPED | 11:39 |
| tristan | that's because cache key test will require ostree I guess | 11:39 |
| tristan | so its consistent, maybe | 11:40 |
| tristan | maybe symlink issue I wonder... but that makes little sense... | 11:42 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 11:44 |
| tristan | crap ok... so this seems indeed to be a symlink issue :-S | 11:49 |
| ssam2 | symlinks treated differently for different backends ? | 11:50 |
| ssam2 | not sure i follow | 11:50 |
| tristan | nope | 11:50 |
| tristan | Well, if I remove the symlink from the cache key test, it passes on gitlab | 11:50 |
| tristan | that far is all I got | 11:50 |
| tristan | So I have a feeling it might have to do with pytest-datafiles, but that hasnt had a release in a long time | 11:51 |
| tristan | but... I mean; we must have symlinks on gitlab right ? | 11:51 |
| tristan | of course, or other tests would certainly be failing | 11:51 |
| ssam2 | yeah... | 11:52 |
| tristan | hmm | 11:57 |
| tristan | fails when testing local file with a symlink and not using datafiles | 11:57 |
| tristan | passes when removing the symlink, so that's not the issue | 11:57 |
| tristan | ssam2, Ok - I can testify that the cache key being generated on gitlab reads the content of the file and reports the hash for the symlink, instead of reporting the symlink | 12:31 |
| * tristan added print statements and collected output from failed run on gitlab | 12:31 | |
| tristan | And the datafiles change did not change that | 12:32 |
| tristan | Ah | 12:33 |
| tristan | Found the issue, crap that was difficult. | 12:33 |
| tristan | ssam2, it's fraking `./setup.py sdist` that is screwing with us, not gitlab | 12:34 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 12:45 |
| tristan | ssam2, what python version do we have on the gitlab test docker image ? | 12:58 |
| ltu | juergbi, are the changes to element state handling dependent on recursive pipelines? so does recursive pipelines have to land first? | 13:02 |
| juergbi | ltu: no, no direct dependency | 13:05 |
| juergbi | element state branch builds upon master | 13:05 |
| ltu | ok, thanks for the info | 13:08 |
| *** xjuan has joined #buildstream | 13:14 | |
| ssam2 | tristan, it's Python 3.6.2 | 13:22 |
| ssam2 | tristan, Fedora 26 base | 13:22 |
| ssam2 | tristan, latest images are Fedora 27 based, but buildstream's .gitlab-ci.yml isn't updated to use the latest image | 13:23 |
| ssam2 | also, that sounds hideous about sdist | 13:23 |
| tristan | ssam2, it's a regression in setuptools :-S | 13:25 |
| tristan | not to mention underspecified and never worked consistently | 13:25 |
| ssam2 | of course | 13:25 |
| tristan | I'm kicking it in the balls with a `os.unlink() && os.symlink()` directly in the cachekey test | 13:26 |
| ssam2 | seems reasonable | 13:26 |
| tristan | While I'm at it; we can consider the link target as a part of the cache key | 13:26 |
| ssam2 | yeah, we should do that | 13:26 |
| tristan | I guess we break cache keys for this fix anyway | 13:26 |
| ssam2 | yeah, but anyone using 'local' source will find they're already broken | 13:27 |
| tristan | Right | 13:28 |
| tristan | And this is going onto bst-1.0 also | 13:28 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 13:34 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key-backport->bst-1.0: Fix local source cache key backport) #218 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/218 | 13:36 |
| jonathanmaw | tristan: I've put some thought into the error handling problem you pointed out in https://gitlab.com/BuildStream/buildstream/merge_requests/181 | 13:58 |
| jonathanmaw | in my reply, I'm suggesting that I should go to various places that ultimately call stage_dependency_artifacts and make sure I have a case to catch exceptions and redirect them as appropriate. | 14:00 |
| jonathanmaw | though perhaps I've misunderstood something and there's a better way to go about it. | 14:00 |
| *** tristan has quit IRC | 14:01 | |
| *** tristan has joined #buildstream | 14:04 | |
| *** xjuan has quit IRC | 14:29 | |
| tristan | ssam2, do you have something I can use to easily test this multiple artifact cache initialization scenario ? | 14:56 |
| *** jonathanmaw has quit IRC | 14:56 | |
| ssam2 | depends if you like faffing with docker | 14:57 |
| tristan | oh crap | 14:57 |
| ssam2 | docker pull buildstream/artifact-cache:latest; docker run -i -t buildstream/artifact-cache:latest | 14:57 |
| ssam2 | oh, with also --publish 8080:8080 --publish 22200:22200 | 14:57 |
| tristan | so your docker environment provides something with multiple remote caches ? | 14:57 |
| ssam2 | no, one cache | 14:57 |
| ssam2 | but you can set up several | 14:58 |
| ssam2 | its not perfect though, i'm wrangling with it right now | 14:58 |
| tristan | I mean, I guess it's not correct to test with the local file:// cache | 14:58 |
| ssam2 | yeah | 14:58 |
| tristan | or is this the branch that changes all that ? | 14:58 |
| ssam2 | alternately you can use the gnome7 cache and the baserock.org cache | 14:58 |
| *** jonathanmaw has joined #buildstream | 14:58 | |
| tristan | Ah, that's an idea | 14:58 |
| ssam2 | this branch doesn't remove the file:// cache stuff | 14:59 |
| ssam2 | not sure what happened to that actually; maybe juergbi is sat on it waiting for this change ? | 14:59 |
| tristan | yeah, I think it's somehow part of juergbi's recursive pipelines still | 14:59 |
| tristan | probably | 14:59 |
| tristan | but it's something that happened :) | 14:59 |
| juergbi | yes, top commit of recursive pipelines | 15:01 |
| juergbi | (doesn't remove file: handling but uses the pushreceive code path for that as well) | 15:01 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key->master: Fix local source cache key) #217 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/217 | 15:03 |
| gitlab-br-bot | buildstream: merge request (fix-local-source-cache-key-backport->bst-1.0: Fix local source cache key backport) #218 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/218 | 15:05 |
| tristan | ssam2, I guess this is the baserock cache right ? https://gitlab.com/baserock/definitions/blob/master/project.conf#L129 | 15:05 |
| ssam2 | yyyeah | 15:06 |
| tristan | ssam2, so iiuc, your branch continues to support a single `artifacts:` dict as written there, and also allows it to be a list ? | 15:06 |
| ssam2 | yeah | 15:06 |
| tristan | nice | 15:07 |
| tristan | dont sound so sure about that URL :) | 15:07 |
| ssam2 | actually that's some wayland bug that causes keys to get stuck down while i'm typing | 15:07 |
| ssam2 | it can produce much more ridiculous results at times :-) | 15:07 |
| tristan | haha | 15:08 |
| ssam2 | wrt. to parallel remote init, if we can get it working we can also do multiple pushes in parallel | 15:08 |
| ssam2 | will be more or less the same thing | 15:09 |
| ssam2 | but i'll do sequential pushes initially, i don't want to fall into the same rabbit hole as before | 15:09 |
| tristan | technically, we *do* multiple pushes in parallel | 15:10 |
| tristan | perhaps it would be simpler to do that were they to be scheduled separately | 15:11 |
| ssam2 | it might, yeah | 15:11 |
| ssam2 | slightly awkward that we have the single ArtifactCache object that wraps all the remotes | 15:11 |
| ssam2 | but, we could have some way of creating a push job for a specific remote | 15:12 |
| tristan | it would require quite some churn anyway | 15:12 |
| tristan | I think it's less of a concern than startup | 15:13 |
| tristan | the current wait time when you have to download that summary is already horrendous | 15:13 |
| tristan | well, somewhere before horrendous, but beyond scratching your head and wondering why we're still waiting | 15:14 |
| gitlab-br-bot | buildstream: issue #185 ("max-jobs is not part of the cache key: Think in a way on how to handle builds that can be build in 6 cores but not in 8, because memory exhaustion or other reasons") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/185 | 15:14 |
| jonathanmaw | hrm, there seems to be an error in the instructions on how to install buildstream - we're telling people with debian stretch to add jessie-backports to their sources list, then install ostree from stretch-backports | 15:15 |
| ssam2 | tristan, i think most people are only going to use one cache, though | 15:15 |
| jonathanmaw | I'm going to make a MR to change that to stretch-backports, as that actually worked for me. | 15:16 |
| ssam2 | tristan, what might be nicer is if we could somehow initialize the remote caches in parallel with loading and resolving the pipeline | 15:16 |
| ssam2 | tristan, not sure how we'd actually do that though | 15:16 |
| ssam2 | jonathanmaw, sounds good | 15:16 |
| tristan | ssam2, not possible really | 15:16 |
| *** mcatanzaro has joined #buildstream | 15:16 | |
| tristan | ssam2, what is present in the remote cache informs the local build plan (and allows us to chop out entire portions of it) | 15:17 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-stretch-install->master: Fix inconsistency in debian stretch install instructions) #219 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/219 | 15:17 |
| *** jennis has joined #buildstream | 15:18 | |
| * tristan not even gonna look at 219 | 15:18 | |
| tristan | just merge | 15:18 |
| ssam2 | tristan, rtue | 15:19 |
| tristan | so, this is the relevant part of the log with GNOME cache + baserock cache: | 15:20 |
| tristan | [--:--:--] START Initializing remote caches | 15:20 |
| tristan | [00:00:44] SUCCESS Initializing remote caches | 15:20 |
| tristan | I guess it pushes headscratchingly long -> horrendously long in this case hehe | 15:20 |
| ssam2 | that is nasty | 15:20 |
| ssam2 | is your connection really slow? | 15:20 |
| ssam2 | both caches have fairly large summary files I imagine | 15:21 |
| tristan | My connection is one of the fastest on the planet | 15:21 |
| ssam2 | ok | 15:21 |
| tristan | doesnt change that I'm still ~350ms round trip from manchester | 15:21 |
| ssam2 | for me it takes a few seconds there | 15:21 |
| ssam2 | is this with SSH or HTTP ? | 15:21 |
| ssam2 | with HTTP/S there's really not much protocol overhead, we're simply downloading one file in the usual way | 15:22 |
| ssam2 | with the SSH protocol there's more overhead, we do the SSH handshake, then ask where the pull URL is, /then/ do the HTTP/S request | 15:22 |
| ssam2 | latency could perhaps start to bite a lot in the 2nd case | 15:22 |
| tristan | ssam2, remember that pushing a base system artifact went from > 24hrs to about 8hrs by dropping sshfs (round trips for every ostree fs access syscall), and then it went from ~8hrs to about 20min after serializing the transfers with python's TarStream | 15:23 |
| ssam2 | sure | 15:23 |
| ssam2 | that said, if this is SSH protocol only, it's a pretty rare use case | 15:24 |
| ssam2 | i don't see many situations where individual developers should be writing to project-wide caches | 15:24 |
| ssam2 | if are writing to a site-specific cache or whatever, you shouldn't have latency issues | 15:24 |
| tristan | removing the ssh from my ~/.config/buildstream.conf brings it down to around 15 seconds | 15:25 |
| ssam2 | and if it's a project wide cache, you should leave the CI to do its job | 15:25 |
| ssam2 | ok, which is still not OK for each startup | 15:25 |
| ssam2 | but better | 15:25 |
| tristan | also, I'm now uncertain about how the old user configuration plays nicely with the new project configuration | 15:25 |
| tristan | How is it that my local config gets to override the https cache ? | 15:25 |
| tristan | or rather, what is the expected result ? | 15:26 |
| ssam2 | i need more specifics to answer that | 15:26 |
| tristan | ssam2, so BEFORE = project.conf declares "the cache" and buildstream.conf can override "the cache" on a per project basis | 15:27 |
| tristan | ssam2, what happens in the regular case, where I dont change my buildstream.conf and I update to a new project.conf ? | 15:27 |
| tristan | I.e. the expectation in BEFORE, is that I override the project | 15:27 |
| tristan | I guess it can have few different answers, one of them is, all of the caches listed in the project.conf are overridded by my user config ? | 15:28 |
| tristan | i.e. it took 44 seconds to query only 1 cache over ssh ? | 15:28 |
| ssam2 | the only thing that this branch changes is what 'override' means, effectively | 15:28 |
| ssam2 | before, if you had a higher priority cache, lower priority caches would be totally ignored | 15:29 |
| ssam2 | now, all the caches in your config go in a list and we talk to all of them | 15:29 |
| tristan | So 3 caches | 15:29 |
| ssam2 | in the documented priority order | 15:29 |
| ssam2 | yeah | 15:29 |
| ssam2 | we did discuss in the original feature proposal having some way to 'remove' a cache from the list | 15:29 |
| ssam2 | and decided against it | 15:29 |
| tristan | yeah, will probably be interesting but not necessary to land at once | 15:30 |
| tristan | ssam2, you cannot put an exception in a Queue() | 15:37 |
| ssam2 | ? | 15:38 |
| tristan | we never pass exceptions between processes, they tend to refer to almost the whole program | 15:38 |
| ssam2 | i'm pretty sure we already do that in master | 15:38 |
| tristan | nope | 15:38 |
| tristan | certainly not | 15:38 |
| ssam2 | um, we do | 15:38 |
| ssam2 | in the fetch_remote_refs() method of OSTreeCache | 15:39 |
| tristan | then that is a bug | 15:39 |
| ssam2 | ok | 15:39 |
| tristan | luckily, we dont hit the exception | 15:39 |
| ssam2 | how should we propagate errors from the child then ? | 15:39 |
| tristan | for child tasks we take special care, we create a string for the exception and pass a Message() to the parent to print in the UI | 15:40 |
| tristan | i.e. using traceback to format it | 15:40 |
| ssam2 | ah, ok | 15:40 |
| tristan | for ostree specific hacks that have been inserted, we dont have policy for this | 15:40 |
| ssam2 | *perhaps* that's why i was getting weird errors here in the past | 15:40 |
| ssam2 | also, FWIW | 15:41 |
| ssam2 | [--:--:--] START Initializing remote caches | 15:41 |
| ssam2 | Got pull url https://ostree.baserock.org/cache/ for push url https://ostree.baserock.org/cache/ | 15:41 |
| ssam2 | Got pull url http://172.17.0.2:8080/artifacts/ for push url ssh://artifacts@172.17.0.2:22200/artifacts/ | 15:41 |
| ssam2 | Got pull url http://172.17.0.2:8080/artifacts/ for push url ssh://artifacts@172.17.0.3:22200/artifacts/ | 15:41 |
| ssam2 | [00:00:13] SUCCESS Initializing remote caches | 15:41 |
| ssam2 | there's 15 seconds contacting 3 caches ... 2 of which are on /my local machine/ | 15:41 |
| ssam2 | well, 13 seconds | 15:41 |
| tristan | this looks like it has other issues too, regardless of parallelism, I dont think we're handling ^C properly here either | 15:41 |
| tristan | (a think very likely to happen when waiting 13 seconds) | 15:42 |
| tristan | So, it looks like there are two avenues to follow; the irresponsible one is to say "There is problems with ostree initializations from before, this adds more of them, and we never had a strategy for handling it right - so lets fix that separately" | 15:44 |
| tristan | That has the benefit of unblocking other branches | 15:44 |
| tristan | And the downside that it will never be fixed | 15:44 |
| tristan | The other is to grok the scheduler code a bit, and properly schedule these initializations in child tasks which are not really well handled at all first | 15:45 |
| tristan | ssam2, one take home here is that now we understand better what is causing https://gitlab.com/BuildStream/buildstream/issues/141 | 15:46 |
| ssam2 | right | 15:47 |
| ssam2 | yeah, if using the scheduler is possible that would be good | 15:47 |
| ssam2 | my previous attempt to solve this involved effectively implementing a new one, which was messy and also broken | 15:47 |
| ssam2 | i don't see much harm in splitting that work out beyond the risk that i might get immediately whisked onto something else and nobody ever gets the chance to look at it | 15:48 |
| ssam2 | it's not like we're introducing a regression though, as with only one cache the new branch will behave the same as master | 15:49 |
| ssam2 | i guess i'll try to finish the other changes without tackling that, but go straight onto the scheduler stuff without declaring the work 'finished' | 15:49 |
| ssam2 | but leaving MR !166 in a state where it could be merged as-is | 15:50 |
| gitlab-br-bot | buildstream: merge request (juerg/element-state->master: WIP: element state updates) #215 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/215 | 15:50 |
| tristan | Well, we're amplifying 141 certainly | 15:50 |
| ssam2 | surely only if there's > 1 cache configured ? | 15:50 |
| tristan | ssam2, My worry about the merge now fix later approach is indeed, that those who are funding developer hours want features, and this might be my only chance to make sure we also create decent software in the process | 15:51 |
| tristan | lemme smoke on it at least hehe :) | 15:53 |
| tristan | it does only amplify 141 in the case of > 1 cache, and also it makes things unbearably slow under the same condition | 15:54 |
| ssam2 | probably enough that those who requested the feature will find it annoying to use | 15:54 |
| ssam2 | so let's have a go at fixing it | 15:55 |
| ssam2 | if i fall down another rabbit hole and don't get anywhere by the end of the week, we can think again | 15:55 |
| ssam2 | but using the scheduler and not trying to return exceptions in a queue may improve matters | 15:55 |
| gitlab-br-bot | buildstream: merge request (juerg/element-state->master: WIP: element state updates) #215 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/215 | 16:01 |
| * tristan returns from smoke... | 16:04 | |
| tristan | ssam2, I think we can start with avoiding serializing the exception, and just serialize the formatted one, i.e: str(e) | 16:05 |
| tlater | /me wonders what the difference between these two for our tests is: | 16:06 |
| tlater | printf "${ECHO_TEST_KEY}\n" | 16:06 |
| tlater | echo ${ECHO_TEST_KEY} | 16:06 |
| tristan | my quick attempt at parallelizing them with the same code resulted in a zombie child + deadlock parent | 16:06 |
| tristan | tlater, doesnt look particularly different | 16:07 |
| tlater | Really curious why we have two test cases then | 16:07 |
| tristan | which | 16:07 |
| tristan | tlater, ? | 16:08 |
| tlater | https://gitlab.com/BuildStream/buildstream/blob/master/integration-tests/shell-test/run-shell-test.sh#L50 | 16:08 |
| tlater | I can't really see the difference between the first and last case there | 16:08 |
| tlater | They seem to be testing the exact same thing | 16:08 |
| tlater | Well, apart from the different executables | 16:08 |
| tristan | tlater, originally we assumed that a runtime has a shell; and we passed strings to an assumed to exist interpretor in the runtime | 16:11 |
| tlater | I can see that, that's why we test echo vs /bin/echo, I assume? | 16:11 |
| tristan | which is a pitfall which is decent to avoid falling into; but it looks like this test does not guarantee it, even | 16:11 |
| tlater | Actually, no that doesn't guarantee t | 16:13 |
| tlater | *it | 16:13 |
| tlater | Hm, a good test for that would be running with a base platform that actually lacks a shell | 16:14 |
| * tlater writes one | 16:14 | |
| tristan | tlater, perhaps, but keep in mind both echo and printf are in fact files in PATH | 16:15 |
| tristan | tlater, sounds like a good test, though | 16:15 |
| tlater | Yeah, it seems a little pointless to have both an echo and printf test | 16:15 |
| tristan | tlater, I wouldnt block your branch on adding it, but nice to have | 16:15 |
| tlater | I'd also like to test the various std* redirections you can make with the shell | 16:15 |
| tlater | If only to confirm the suite is working now | 16:15 |
| * tlater thinks this shouldn't take too long anyway, and is close to finishing | 16:16 | |
| tristan | ssam2, this fetch_remote_refs() method... is the one you replace in the branch right ? | 16:16 |
| *** xjuan has joined #buildstream | 16:16 | |
| ssam2 | yeah | 16:16 |
| * tristan takes a stab at 141 on your branch | 16:19 | |
| tristan | Ok that was easy enough to fix | 16:25 |
| tristan | ssam2, alright so I can at least reduce this to; we just take a damn long time to initialize | 16:26 |
| tristan | and also close #141 | 16:26 |
| ssam2 | progress! | 16:26 |
| tristan | ssam2, other than this; how stands the branch in readiness, the other stuff is all taken care of yes ? | 16:26 |
| tristan | `push:` flag | 16:26 |
| ssam2 | i'm nearly done with that | 16:27 |
| ssam2 | not yet pushed | 16:27 |
| ssam2 | that said, it'll do multiple pushes in sequence in a single job | 16:27 |
| tristan | And I think there was a private function in utils lacking an underscore | 16:27 |
| tristan | ssam2, that is much less of a concern to be honest | 16:27 |
| ssam2 | right | 16:27 |
| ssam2 | it seems to be working, i'm just retesting, updating docs etc. | 16:28 |
| tristan | ssam2, at least at that point; we're doing a ton of parallel work already; and we're not blocking on the built artifact locally to build reverse deps | 16:28 |
| ssam2 | yeah | 16:28 |
| tristan | so, at push time, the performance could be improved when say, you just want to explicitly push one artifact to all your caches | 16:28 |
| tristan | but meh - still startup being slow annoying and worth fixing, but I guess in the current environment we should not block on it | 16:29 |
| ssam2 | good or bad phrasing: "You can enable this on a cache-by-cache basis ..." | 16:34 |
| tristan | ssam2, so I just added couple comments regarding how to handle that exception | 16:36 |
| tristan | ssam2, and I pushed this: https://gitlab.com/BuildStream/buildstream/commit/da6cc4fea16d6ac07300aec6509292fba373e0c9 which I'd like you to cherry-pick on top of your branch after you've done your fixes | 16:36 |
| ssam2 | will d | 16:37 |
| ssam2 | o | 16:37 |
| tristan | thats on the tip of a branch called tristan/multiple-caches fwiw | 16:37 |
| tristan | ssam2, "... on a per-cache basis ..." I think is better | 16:38 |
| tristan | or `per artifact cache` if specificity is needed, probably that is in context of your paragraph though :) | 16:38 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-stretch-install->master: Fix inconsistency in debian stretch install instructions) #219 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/219 | 16:39 |
| tristan | suspending buildstream with ^Z also works nicely while fetching the artifact lists | 16:41 |
| tristan | good good | 16:41 |
| * tristan notes it would be really great to have interactive tests too | 16:43 | |
| tristan | terminal behavior, is mostly up to developers to ensure works goddamn well before committing at this point | 16:43 |
| tristan | and we fail at that | 16:43 |
| tristan | (well, we come close, but fail) | 16:44 |
| tlater | Hm, I wonder how interactive tests would work | 16:44 |
| tristan | tlater, yeah :) | 16:44 |
| * tlater guesses there are few frameworks for this stuff, but there might be something | 16:44 | |
| tlater | This is python after all | 16:44 |
| tristan | I think pytest may have stuff builtin | 16:44 |
| tristan | when running `bst build --track-all <element>` I'm sometimes getting bad results with ^C now | 16:45 |
| tristan | this seems new | 16:45 |
| ssam2 | i think I had issues with that before | 16:46 |
| * tlater thinks that may be a bug filed a few months ago | 16:46 | |
| ssam2 | up to and including a deadlock, although that's element-state related no doubt | 16:47 |
| tristan | I need to make sure that is not from the branch... | 16:47 |
| tlater | I was waiting for the element state branch to be fixed to check it again, which I think should land soon? | 16:47 |
| *** noisecell has quit IRC | 17:02 | |
| tristan | verified that this is also happening with master | 17:04 |
| *** mcatanzaro has quit IRC | 17:08 | |
| *** mcatanzaro has joined #buildstream | 17:08 | |
| *** jude has quit IRC | 17:18 | |
| gitlab-br-bot | buildstream: merge request (tristan/fix-cache-init->bst-1.0: _artifactcache/ostreecache.py: Handle ^C and shutdown child task when initializing cache) #220 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/220 | 17:18 |
| tristan | gah, ssam2; lemme give you a different commit to cherry-pick... mine has a bogus import | 17:23 |
| gitlab-br-bot | buildstream: merge request (tristan/fix-cache-init->bst-1.0: _artifactcache/ostreecache.py: Handle ^C and shutdown child task when initializing cache) #220 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/220 | 17:25 |
| *** jude has joined #buildstream | 17:26 | |
| tristan | ssam2, updated on tristan/multiple-caches: https://gitlab.com/BuildStream/buildstream/commit/0bd2e3b1fa0df81e0f9b00eaee08a877b4499847 | 17:27 |
| tristan | I was thinking of using sys.exit(-1) but then realised raise was correct, forgot to remove sys | 17:27 |
| ssam2 | go tit | 17:28 |
| ssam2 | ah | 17:28 |
| ssam2 | got it | 17:28 |
| ssam2 | not sure i will get this done today, I'm not really happy with the tests now | 17:28 |
| tristan | ssam2, I like your wayland bugs :D | 17:28 |
| ssam2 | that might have been a finger and thumbs issue :-) | 17:29 |
| ssam2 | using real pushes and pulls to test that we combined the list of artifact caches correctly is not really that great | 17:30 |
| ssam2 | without having real caches | 17:30 |
| ssam2 | the addition of the 'push' flag just makes the current push/pull tests even less suitable for what i think we actually want to test, which is that combining the list of artifact caches from the different config files is working | 17:31 |
| ssam2 | then, we also want to test that pushes and pulls actually work | 17:31 |
| ssam2 | but i'm not liking trying to both at the same time | 17:31 |
| ssam2 | *to test both | 17:31 |
| ssam2 | maybe merging in juergbi's changes to this will help | 17:32 |
| *** jude has quit IRC | 17:45 | |
| *** ssam2 has quit IRC | 17:53 | |
| *** valentind has joined #buildstream | 18:23 | |
| gitlab-br-bot | buildstream: merge request (tristan/fix-cache-init->bst-1.0: _artifactcache/ostreecache.py: Handle ^C and shutdown child task when initializing cache) #220 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/220 | 18:42 |
| *** valentind has quit IRC | 18:52 | |
| *** valentind has joined #buildstream | 18:59 | |
| *** ernestask has quit IRC | 19:21 | |
| *** valentind has quit IRC | 19:35 | |
| *** valentind has joined #buildstream | 19:51 | |
| *** xjuan has quit IRC | 20:23 | |
| *** jonathanmaw has quit IRC | 21:23 | |
| *** tristan has quit IRC | 22:01 | |
| *** mcatanzaro has quit IRC | 22:02 | |
| *** valentind has quit IRC | 22:07 | |
| *** jennis has quit IRC | 23:52 | |
| *** paulsherwood has quit IRC | 23:52 | |
| *** nexus has quit IRC | 23:52 | |
| *** tlater has quit IRC | 23:52 | |
| *** tiago has quit IRC | 23:52 | |
| *** gitlab-br-bot has quit IRC | 23:52 | |
| *** lantw44 has quit IRC | 23:52 | |
| *** saxa has quit IRC | 23:52 | |
| *** ptomato[m] has quit IRC | 23:52 | |
| *** waltervargas[m] has quit IRC | 23:52 | |
| *** ernestask[m] has quit IRC | 23:52 | |
| *** m_22[m] has quit IRC | 23:52 | |
| *** pro[m] has quit IRC | 23:52 | |
| *** mrmcq2u[m] has quit IRC | 23:52 | |
| *** cgmcintyre[m] has quit IRC | 23:52 | |
| *** mattiasb has quit IRC | 23:52 | |
| *** kailueke[m] has quit IRC | 23:52 | |
| *** csoriano has quit IRC | 23:52 | |
| *** ironfoot has quit IRC | 23:52 | |
| *** jjardon[m] has quit IRC | 23:52 | |
| *** bochecha has quit IRC | 23:52 | |
| *** jennis has joined #buildstream | 23:53 | |
| *** paulsherwood has joined #buildstream | 23:53 | |
| *** nexus has joined #buildstream | 23:53 | |
| *** tlater has joined #buildstream | 23:53 | |
| *** tiago has joined #buildstream | 23:53 | |
| *** ernestask[m] has joined #buildstream | 23:53 | |
| *** saxa has joined #buildstream | 23:53 | |
| *** ptomato[m] has joined #buildstream | 23:53 | |
| *** waltervargas[m] has joined #buildstream | 23:53 | |
| *** m_22[m] has joined #buildstream | 23:53 | |
| *** pro[m] has joined #buildstream | 23:53 | |
| *** mrmcq2u[m] has joined #buildstream | 23:53 | |
| *** cgmcintyre[m] has joined #buildstream | 23:53 | |
| *** mattiasb has joined #buildstream | 23:53 | |
| *** kailueke[m] has joined #buildstream | 23:53 | |
| *** gitlab-br-bot has joined #buildstream | 23:53 | |
| *** bochecha has joined #buildstream | 23:53 | |
| *** lantw44 has joined #buildstream | 23:53 | |
| *** csoriano has joined #buildstream | 23:53 | |
| *** ironfoot has joined #buildstream | 23:53 | |
| *** jjardon[m] has joined #buildstream | 23:53 | |
| *** irc.acc.umu.se sets mode: +o ironfoot | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!