IRC logs for #buildstream for Thursday, 2019-04-25

*** jswagner has quit IRC00:56
*** nimish2711 has joined #buildstream01:21
*** rdale has quit IRC01:22
*** nimish2711 has quit IRC02:06
*** nimish2711 has joined #buildstream02:07
*** paulsherwood has quit IRC02:07
*** benbrown has quit IRC02:07
*** paulsherwood has joined #buildstream02:07
*** laurence has quit IRC02:07
*** laurence has joined #buildstream02:07
*** nimish2711 has quit IRC02:22
*** nimish2711 has joined #buildstream02:48
*** nimish2711 has quit IRC02:53
*** nimish2711 has joined #buildstream02:53
*** nimish2711 has quit IRC03:03
*** nimish2711 has joined #buildstream03:03
*** nimish2711 has quit IRC03:08
*** nimish2711 has joined #buildstream03:09
*** nimish2711 has quit IRC03:29
*** nimish2711 has joined #buildstream03:29
*** nimish2711 has quit IRC03:34
*** nimish2711 has joined #buildstream03:34
*** nimish2711 has quit IRC03:39
*** nimish2711 has joined #buildstream03:40
*** nimish2711 has quit IRC03:55
*** nimish2711 has joined #buildstream03:55
*** nimish2711 has quit IRC04:05
*** nimish2711 has joined #buildstream04:05
*** nimish2711 has quit IRC04:10
*** nimish2711 has joined #buildstream04:11
*** nimish2711 has quit IRC04:21
*** nimish2711 has joined #buildstream04:21
*** nimish2711 has quit IRC04:26
*** nimish2711 has joined #buildstream04:32
*** nimish2711 has joined #buildstream04:32
*** nimish2711 has quit IRC04:36
*** alatiera has joined #buildstream05:26
*** alatiera has quit IRC06:08
*** alatiera has joined #buildstream06:12
*** bochecha has joined #buildstream06:49
*** toscalix has joined #buildstream07:27
*** toscalix has quit IRC07:29
*** toscalix has joined #buildstream07:29
gitlab-br-botjjardon approved MR !1308 (tristan/fdsdk-ci-refresh->master: .gitlab-ci.yml: Updating overnight tests to test ported freedesktop-sdk) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/130807:30
*** benbrown has joined #buildstream07:34
bochechahi, has anybody ever seen something like this? https://paste.gnome.org/pqoziwlri07:58
*** nimish2711 has joined #buildstream08:06
benschubertbochecha: ouch that seems bad. Do you have something with which we could reproduce it?08:08
bochechabenschubert: not sure, I just installed buildstream and bst-external so I'm wondering whether it's my installation that is missing something08:09
bochechabenschubert: in any case, this is after cloning https://gitlab.com/freedesktop-sdk/freedesktop-sdk/ and checking out the 18.08 branch08:09
benschubertbochecha: I can't have a look in depths right now, but I might be able to have a look tonight if nobody else had a chance. This error shouldn't break anything though, it's in the UI reporting. Seems quite bad though08:15
*** nimish2711 has quit IRC08:15
bochechayeah, bst track actually did its job fine08:16
*** tpollard has joined #buildstream08:19
*** rdale has joined #buildstream08:30
*** nimish2711 has joined #buildstream08:31
*** nimish2711 has quit IRC08:51
*** nimish2711 has joined #buildstream08:52
*** nimish2711 has joined #buildstream09:02
*** nimish2711 has joined #buildstream09:03
*** nimish2711 has joined #buildstream09:03
*** nimish2711 has quit IRC09:08
*** phildawson has joined #buildstream09:15
phildawsonPlease could I get a quick review on updating the docker image for bst-experimental: https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/609:17
*** nimish2711 has joined #buildstream09:19
*** nimish2711 has quit IRC09:29
*** nimish2711 has joined #buildstream09:29
*** nimish2711 has quit IRC09:34
*** lachlan has joined #buildstream09:35
*** dtf has joined #buildstream09:38
*** tristan has joined #buildstream09:41
gitlab-br-bottristanvb merged MR !1308 (tristan/fdsdk-ci-refresh->master: .gitlab-ci.yml: Updating overnight tests to test ported freedesktop-sdk) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/130809:45
phildawsonCould anyone with merge rights to bst-experimental push the button on https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests09:46
phildawsonAlso, who should I ask about getting merge rights myself?09:47
*** ChanServ sets mode: +o tristan09:48
tristanphildawson, Sec, lemme check how that is setup09:48
phildawson:)09:49
tristanmerged !609:50
gitlab-br-botMR !6: Expand ~ in paths https://gitlab.com/BuildStream/buildstream/merge_requests/609:50
phildawsoncheers09:50
tristanphildawson, I guess that will make the mr !5 work, cause of missing gcc in the image09:51
gitlab-br-botMR !5: Dependency types https://gitlab.com/BuildStream/buildstream/merge_requests/509:51
tristanyeah09:51
tristangitlab-br-bot, wrong repo man... keep up ! ;-)09:51
tristanlooks like it was allowing merge without rebase... fixed that09:52
phildawsonactually, I think !5 still needs !130709:52
gitlab-br-botMR !1307: Make Integration cache fixture available in testing package https://gitlab.com/BuildStream/buildstream/merge_requests/130709:52
tristanHmmm that is gonna be a bit tricky09:53
tristanIt's a fork of bst-external, lets see how that is setup09:53
tristanI think it's like BuildStream09:53
*** nimish2711 has joined #buildstream09:55
tristanphildawson, you should already have rights09:55
tristanthe repo settings say for the protected branch "Maintainers" are allowed both to push and merge09:55
tristanand in the little drop down thing, you are listed as a maintainer :-/09:56
tristanoh no I'm reading it wrong09:56
phildawsonhmm, I wonder why I can't merge then :/09:56
tristanui changed it seems09:56
*** bochecha has quit IRC09:56
phildawsonAh :)09:57
tristanOk now should work, what I did is just add you as a maintainer09:57
phildawsonThanks!09:57
tristannot sure it's the end-all strategy, bit different to what we had in BuildStream, but what we have in BuildStream is a bit broken because of marge09:57
*** nimish2711 has quit IRC10:00
*** nimish2711 has joined #buildstream10:00
*** nimish2711 has quit IRC10:13
*** lachlan has quit IRC10:37
jennistristan, for our cache expiry tests we say that '0' is a valid value10:37
tristanjennis, For headroom10:38
jennisNow, in the test suite, this test passes because our headroom is also zero, however, in reality, we'll get a LoadError10:38
jennisNo, for the quota10:38
tristanoh ?10:38
jennisAre we saying that quota: 0, in userconf is essnetially 'infinity'?10:38
tristanThat is a bit strange, but I guess 0 is *anyway* a valid value, regardless of how nonsensical it is ?10:38
tristanI mean the env var we use for tests is to circumvent the hard coded headroom iirc10:39
jennisWell, in our tests, we test to ensure that if the cache quota is set to 0, the test passes (i.e. there isn't a load error)10:39
*** lachlan has joined #buildstream10:39
jenniswhich to me sounds like we're saying, "zero quota" == "infinity"10:39
tristanjennis, as I recall, 'infinity' means 'the rest of your hard drive'10:39
jennisYep10:39
tristanjennis, and that is very efficient to calculate (it's a single stat(), unlike calculating usage)10:40
tristanNow, I don't recall a change, it's possible that the way the setting is expressed may have changed10:40
phildawson I think that the references to ostree in cache expiry tests (https://gitlab.com/BuildStream/buildstream/blob/master/tests/artifactcache/expiry.py#41)  are hangovers from the old cache and are no longer correct. Am I right? If so I'll fix them.10:40
jennisYes I'm aware of that, but I'm asking what *should* the real world behaviour be if we explicitly set the quota to zero10:40
tristanBut as I recall we allowed setting the string 'infinity' to ask for this, not '0'10:41
tristanIt's a decent question10:41
tristanjennis, I would be happy with interpreting it as "not allowed to add any data to the cache"10:41
tristanAnd fire a warning when trying to cache anything10:41
tristanhonestly, it's not that dissimilar from 110:41
tristanI can't imagine anything that could be stored in an artifact cache of 1 byte10:42
jennisahh, so the other extreme10:42
tristanI mean, if we're deciding something that is still not clearly defined, that would be my vote10:42
jennisYeah, I think it makes sense10:43
jennisbecause if we want the opposite behaviour, i.e. "add as much data to the cache as possible" we have the default behaviour, or we could explicitly set quota: 100%10:44
jennisSo i guess we should also provide this (hopefully intuitive) method of, "don't add to my cache"10:44
*** tristan has quit IRC10:46
*** tristan has joined #buildstream10:46
*** ChanServ sets mode: +o tristan10:46
* tristan changes his config to only allow 2 or 3 builders10:47
tristanit's a sad day when I cannot build the whole stack with 4 parallel builds without overheating :-S10:48
jennis:(10:52
jennistristan, I'm unsure how we should raise this when the quota is explicitly set to zero10:53
jennisI wouldn't exactly say it's a LoadError10:53
jennisCASError?10:54
tristanjennis, Wouldnt it be okay to not raise an error ?10:54
jennisSo to just info?10:54
tristanjennis, and live with a cache that is full ?10:54
jennisThis is what I was wondering10:54
tristanI mean it's a zero size cache, which is used 100%10:54
tristanThen as soon as anything tries to cache something, we bail out in the normal way as we would when we hit the end of the cache10:54
tristanIt's a nonsensical situation to be sure, but I don't think it's any more nonsensical as a value of 110:55
jennisI mean, when we call CASCache.full(), if this returns true we raise a CASCacheError10:55
tristanSo might as well just treat it the same way10:55
*** nimish2711 has joined #buildstream10:56
tristanjennis, Do we ? I mean when do we do that ?10:56
tristanI think when CASCache.full() returns true, we stop any ongoing jobs and do a cleanup10:57
benschuberttristan: we don't stop jobs, we just don't enqueue new ones10:57
tristanbenschubert, Yes that is more accurate10:57
tristanIndeed10:57
benschubertmeaning on a relatively powerfull machine, you will get out of disk space completely and multiple builds/pulls will fail10:57
tristanWell, if you have any idea of how to handle that better :)10:58
tristanWe *could* start killing off jobs for the sake of a cleanup10:58
jennisahh apologies, this raising of the CASCacheError is in the cleanup job10:58
tristanBut headroom should be able to handle it in normal operation10:58
tristanOr, we could make headroom configurable10:59
tristanFor powerful machines which build many many things in parallel to have a bigger safety margin10:59
tristanjennis, Right, that's when it's still full and there are artifacts we cannot delete10:59
jennisSo if I have set my cache cache to zero, and  then `bst build foo.bst`, what should I expect to see? An info message saying that we're not going to add data to the cache?10:59
tristanjennis, Just normal startup, and a failure once you try to cache an artifact (or source) ?11:00
tristanjennis, Also, you will see 100% in red in the statusbar11:00
*** shash has joined #buildstream11:00
tristanjennis, i.e. if you set your cache size to zero, it's pretty much equal to setting it to 111:01
benschuberttristan: yep configurable/a function of the number of builders would be better :D11:01
tristanjennis, which is what I keep saying... if we have to draw the line somewhere, lets just not draw it11:01
jennisOk, bare in mind that I've made changes so that we're only calulating the quota when we actually need to write to the cache11:01
tristan:)11:01
tristanjennis, no that is not true !11:02
tristanjennis, your lazy changes defer calculating it until it needs to be *known*11:02
tristanthat is different11:02
tristanIt needs to be known for instance, when a real build session starts and we try to print it in the log11:02
jennisthe ones you saw yesterday? I've followed your suggestion and changed these11:02
jennisAhh, but CASCacheUsage just needs to know about it11:03
tristanjennis, that said, if you have some abstract "get_cache_size()" which is used by all calling code paths, including the CASCacheUsage() etc, then you know that it will only be resolved when asked for11:03
tristanSo, you might not have noticed that you will discover it earlier than that :)11:03
*** nimish2711 has quit IRC11:06
jennistristan, and *don't* trigger a cleanup job for this?11:09
tristanjennis, "I don't think it needs to be handled in any different way than any other condition when the cache becomes full"11:11
tristanDoes that answer the question ?11:11
tristanI.e., no conditional statement that handles it at all, and trust the code does the right thing, even it if seems ridiculous11:11
tristanbecause it is just as ridiculous as any other severely small number11:12
jennisOk, well without conditional statements, we see that this is less than the headroom and get a LoadError right away...11:12
tristanAhhh11:12
jennisSo anything defined between 0 and 2G11:13
tristanSo we already had that in place11:13
tristanRight11:13
tristanNot enough side for the headroom11:13
jennisYes, but the test suite passed because the headroom is 0 for the test suite11:13
tristanI am confused, I think that load error is intentional11:13
jennisand in the test suite, we have "Valid configurations":, where an explicitly set cache quota of zero, should be a success11:14
tristanRight11:14
jenniswhich to me indicates that we were treating this edge case as, 0 is infinity11:14
tristanBecause the test suite needs to run in conditions where we don't require 2G of actual cache, and we want to hit the limit in under that size11:14
tristanjennis, Ok that part I dont think so... the 0 is clearly less than headroom11:15
jennisYep, so if we want to keep this behaviour, we could set the test suite headspace to be 1 Byte11:15
* tristan is not clear why any of this has to change just because we resolve things lazily11:15
jennisbecause of the failing tests11:16
jenniswhich assume that a cache quota of 0 should be a success11:16
tristanyou could have lead with that ;-)11:16
tristanWhat is the failing test ?11:16
tristanAnd why would that now fail when before it didnt ?11:16
jennistests/artifactcache/expiry.py::test_invalid_cache_quota[0-success-None]11:17
tristanjennis, perhaps the test is using a bst command which need not resolve the headroom ?11:17
tristanlike, is it triggering `bst workspace list` and imagining that is enough ?11:17
shashHey guys, I've recently joined Bloomberg and will be working with the BuildStream team for the next 6 months. I've been mainly tasked with working on the container plugins. As such, would it be possible to have developer access to the org. My Gitlab account is: shashwatdalal. Looking forward to working with all of you for the coming months :)11:17
benschubertlaurence: ^ can you help? :)11:17
tristanshash, I think there is a button in the UI to request access, but certainly :)11:18
tpollardwelcome shash :)11:18
jennisHi shash, welcome11:19
shashtristan: I've just requested access through the UI, thanks :)11:19
tristanjennis, I just looked at the test, and indeed it is11:19
tristanjennis, that means you are using a codepath which does *not* have any reason to try to calculate the cache size at all11:20
jennisOk tristan, so I've changed it so that we're only calculating the cache quota when we need it, so I've changed the workspace list command in that test to a build11:20
tristanjennis, that would explain it I think11:20
tristanRight, and it still doesnt pass ?11:20
jennisnow that problem is that we're now trying to divide by zero to show the cache usage stats11:20
jennisWhich then lead me to ask, what *should* the behaviour of a cache quota explicitly set to zero be11:21
tristanjennis, So that means you have not marshalled the codepaths to the right places11:21
tristanI think it's what that means anyway11:21
tristanI.e. you want all of that to happen in response to trying to get the cache size, so all that load time cache quota validation cannot be known until you actually check11:22
tristanjennis, so you have some case where a 0 is appearing as a value before having been calculated11:22
tristansomething is accessing the value directly, without going through the right codepath11:22
*** tristan has quit IRC11:26
*** nimish2711 has joined #buildstream11:28
*** tristan has joined #buildstream11:30
*** ChanServ sets mode: +o tristan11:30
tristanjennis, I gotta run... actually I think the machine needs a bit of a break, overheated 3 times today :-/11:32
jennisNo worries, there is confusion added from the head space being 2G in reality and 0 in the test suite11:34
*** nimish2711 has quit IRC11:52
*** nimish2711 has joined #buildstream11:53
*** nimish2711 has quit IRC12:03
*** nimish2711 has joined #buildstream12:03
*** jonathanmaw has joined #buildstream12:04
*** nimish2711 has quit IRC12:08
*** nimish2711 has joined #buildstream12:08
*** nimish2711 has quit IRC12:16
*** bochecha has joined #buildstream12:23
*** nimish2711 has joined #buildstream12:40
*** nimish2711 has quit IRC12:45
*** nimish2711 has joined #buildstream12:45
*** bochecha has quit IRC13:26
*** nimish2711 has quit IRC13:30
*** nimish2711 has joined #buildstream13:30
jennistristan, I've now uncovered that we weren't really fully supporting a cache quota of '0' in master anyway... I've left a summary comment here: https://gitlab.com/BuildStream/buildstream/merge_requests/1311/diffs#note_16422769613:32
jennisSo to me, it looks like we definitely need to specialcase this behaviour13:32
jennisI'm not sure what we agreed from our conversation this morning, I'm still a bit confused. So I will wait for your thoughts on this comment13:33
*** nimish2711 has quit IRC13:35
*** nimish2711 has joined #buildstream13:36
*** nimish2711 has quit IRC13:51
*** nimish2711 has joined #buildstream13:51
*** nimish2711 has quit IRC13:56
*** nimish2711 has joined #buildstream13:56
*** nimish2711 has quit IRC14:01
*** nimish2711 has joined #buildstream14:02
*** tristan has quit IRC14:07
*** nimish2711 has quit IRC14:12
*** nimish2711 has joined #buildstream14:12
*** bochecha has joined #buildstream14:16
*** nimish2711 has quit IRC14:27
*** nimish2711 has joined #buildstream14:27
*** nimish2711 has quit IRC14:32
tpollardanybody else been suffering from https://gitlab.com/BuildStream/buildstream/-/jobs/202329002 today?14:35
tpollard'runner system failure' is really hitting this pipeline hard it seems14:37
jennisYeah, not just that one too14:41
phildawsonI've been seeing a lot of that14:48
*** nimish2711 has joined #buildstream14:54
*** nimish2711 has quit IRC15:24
*** nimish2711 has joined #buildstream15:30
*** toscalix has quit IRC15:33
jjardonshash: done15:39
*** nimish2711 has quit IRC15:55
*** nimish2711 has joined #buildstream16:16
*** lachlan has quit IRC16:18
*** nimish2711 has quit IRC16:26
*** nimish2711 has joined #buildstream16:26
*** lachlan has joined #buildstream16:28
*** bochecha has quit IRC16:28
*** nimish2711 has quit IRC16:31
*** nimish2711 has joined #buildstream16:37
*** nimish2711 has quit IRC16:42
*** nimish2711 has joined #buildstream16:42
*** lachlan has quit IRC16:49
*** nimish2711 has quit IRC16:52
*** nimish2711 has joined #buildstream16:53
*** lachlan has joined #buildstream16:57
*** nimish2711 has quit IRC16:58
*** nimish2711 has joined #buildstream16:58
*** jonathanmaw has quit IRC17:04
shashjjardon thank you :)17:07
gitlab-br-botshashwatdalal opened MR !1313 (fixed-bug-for-pip-test->master: tests/testutil/python_repo.py: fixed executable path when running pip) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/131317:20
*** nimish2711 has quit IRC17:22
*** lachlan has quit IRC17:43
*** nimish2711 has joined #buildstream17:46
*** nimish2711 has quit IRC17:51
*** nimish2711 has joined #buildstream17:52
*** nimish2711 has quit IRC18:02
*** nimish2711 has joined #buildstream18:02
*** nimish2711 has quit IRC18:22
*** nimish2711 has joined #buildstream18:28
*** nimish2711 has quit IRC18:33
*** nimish2711 has joined #buildstream18:33
*** nimish2711 has quit IRC18:53
*** nimish2711 has joined #buildstream18:53
*** nimish2711 has quit IRC19:08
*** nimish2711 has joined #buildstream19:09
*** nimish2711 has joined #buildstream19:09
*** nimish2711 has quit IRC19:14
*** toscalix has joined #buildstream19:17
*** toscalix has quit IRC19:19
*** nimish2711 has joined #buildstream19:25
*** nimish2711 has quit IRC19:35
*** nimish2711 has joined #buildstream19:35
*** nimish2711 has quit IRC19:40
*** nimish2711 has joined #buildstream19:40
*** nimish2711 has quit IRC19:50
*** nimish2711 has joined #buildstream19:51
*** nimish2711 has quit IRC20:11
*** nimish2711 has joined #buildstream20:11
*** nimish2711 has quit IRC20:26
*** nimish2711 has joined #buildstream20:37
*** nimish2711 has quit IRC20:39
gitlab-br-botcs-shadow approved MR !1313 (fixed-bug-for-pip-test->master: tests/testutil/python_repo.py: fixed executable path when running pip) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/131323:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!