IRC logs for #buildstream for Tuesday, 2018-09-25

gitlab-br-botbuildstream: merge request (chandan/bst-checkout-build->master: Add `--deps build` option to `bst checkout`) #819 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81900:08
*** abderrahim has quit IRC00:12
*** abderrahim has joined #buildstream00:12
*** abderrahim has quit IRC00:39
*** abderrahim has joined #buildstream00:40
gitlab-br-botbuildstream: merge request (chandan/source-checkout->master: Add `bst source-checkout` command) #820 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82000:46
gitlab-br-botbuildstream: merge request (chandan/source-checkout->master: Add `bst source-checkout` command) #820 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82001:06
*** WSalmon_ has quit IRC04:09
*** bochecha has joined #buildstream06:09
*** iker has joined #buildstream06:56
*** iker has quit IRC07:04
*** iker has joined #buildstream07:06
*** toscalix has joined #buildstream07:37
gitlab-br-botbuildstream: issue #226 ("Document incompatibilities between bst releases") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/22607:43
*** tiagogomes has joined #buildstream07:51
*** finn has quit IRC08:22
*** finn has joined #buildstream08:27
*** finn has quit IRC08:35
gitlab-br-botbuildstream: issue #675 ("Batch commands for remote execution") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/67508:39
*** finn has joined #buildstream08:42
gitlab-br-botbuildstream: issue #676 ("CAS Server: Implement BatchUpdateBlobs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/67608:44
gitlab-br-botbuildstream: issue #677 ("CAS Client: Use BatchUpdateBlobs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/67708:45
gitlab-br-botbuildstream: merge request (juerg/cas-batch->master: _artifactcache/cascache.py: Use BatchReadBlobs) #813 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81308:48
gitlab-br-botbuildstream: merge request (chandan/bst-checkout-build->master: Add `--deps build` option to `bst checkout`) #819 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81908:52
gitlab-br-botbuildstream: merge request (juerg/cas-batch->master: _artifactcache/cascache.py: Use BatchReadBlobs) #813 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81308:55
gitlab-br-botbuildstream: issue #678 ("CAS: Avoid double write for received blobs") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/67808:58
*** jonathanmaw has joined #buildstream09:01
gitlab-br-botbuildstream: merge request (juerg/cas-batch->master: _artifactcache/cascache.py: Use BatchReadBlobs) #813 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81309:02
gitlab-br-botbuildstream: issue #653 ("bst checkout fails due to artifact cache error when remote execution is enabled") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/65309:07
tiagogomesjuergbi no tests for !813 ?09:20
jonathanmawadds68: Would you be interested in becoming a co-maintainer for bst-external? I'm having trouble finding time to support bst-external at the moment, and Kinnison suggested you might be able to help with the freedesktop-sdk-related issues and MRs09:21
gitlab-br-botbuildstream: merge request (mablanch/668-remote-build-failure->master: WIP: Better handle remote build failures) #825 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82509:22
juergbitiagogomes: the existing tests will use batching. it's not a user option09:25
adds68jonathanmaw, as much as i'd love to help, i've got quite a bit on my plate at the moment also :(09:26
*** lachlan has joined #buildstream09:26
jonathanmaw:(09:27
gitlab-br-botbuildstream: issue #554 ("Pulling artifacts is very slow with high latency connections") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/55409:31
gitlab-br-botbuildstream: merge request (juerg/cas-batch->master: _artifactcache/cascache.py: Use BatchReadBlobs) #813 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/81309:31
adds68Is it possible to depend on a junction that is inside a project you are junctioning from?09:33
skullmanadds68: I think freedesktop-sdk has a junction in itself, might be worth looking at that.09:39
toscalixadds68: who from freedesktop-sdk should do it then?09:40
adds68skullman, that is only a straight junction though, i am wondering if it's possible for a project to depend on a freedesktop-sdk junction for example that inside gnome-build-meta ?09:40
* tlater[m] wonders if it would be acceptable to place the artifact server's to-be-processed artifacts in /tmp09:40
tlater[m]Since that would use memory instead of potentially-about-to-explode disk space09:40
adds68toscalix, it's a buldstream project, so i am not sure what you mean, jonathanmaw i assume was asking me directly, which at the moment isn't feasible09:41
tlater[m]But I suppose we would still run out of memory, except that in that case we would slow down the system |:09:41
tiagogomestlater[m] probably not. If it is a tempfs you won't be able to use it for large artifacts09:41
skullmantlater[m]: plus you can't atomically move it into place if it's on another filesystem09:42
tlater[m]Ah, right09:42
tiagogomesjmac can https://gitlab.com/BuildStream/buildstream/merge_requests/818 be merged?09:42
jmacI'm trying to figure that out09:43
jmacThe new CONTRIBUTING.rst doesn't say much about what approvals are needed09:43
jmacIf the policy is still "eh, whatevs" then yes, I'll merge it09:44
tlater[m]jmac: iirc the email asked you to get approval from someone proficient in the specific code area09:45
* tlater[m] isn't sure who that is at present09:46
jmacWell, that would be Jürg (who has reviewed it). I'm just trying to find the email.09:46
jonathanmawtoscalix: I'd be happy if someone from freedesktop-sdk could have time allocated to be a co-maintainer for bst-external09:46
jonathanmawif not I can't guarantee I'll have time to make changes other than review MRs and occasionally push new tags09:47
jonathanmaw(well, whether I have the time is independent of whether someone from freedesktop-sdk can be co-maintainer)09:47
toscalixjonathanmaw: bst-external will gain relevance. It will require more attention. We need more effort. I get it09:47
gitlab-br-botbuildstream: issue #679 ("Checkout gives an empty directory when using --deps none") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/67909:48
tlater[m]jmac: Hrm, looks like it's technically still just a proposal09:50
tlater[m]It's an email from tristan titled "Proposal for commiter policy update"09:51
jjardontoscalix: maybe we can move the relevant plugins to buildstream/ instead, so the maintenance is shared by everyone?09:52
*** finn has quit IRC09:52
jmacYeah, I've read that; it's not a policy though, it just tells people where to look for reviewers, rather than saying what's required09:52
* jjardon still doesnt know the criteria to have some plugin in buildstream and others in bst-external09:53
*** gitlab-br-bot has quit IRC09:53
*** finn has joined #buildstream09:53
juergbione reason is that we guarantee stability / backward compatibility for plugins in buildstream core09:54
juergbii.e., it's not suitable to move plugins there before stabilization09:55
*** finn has quit IRC09:56
jjardonok, so the idea is that bst-external is a temporal place, until plugins are stable; the final objective is move relevant plugin to buildstream. Or are plugins mean to be forever in bst-external?09:57
*** finn has joined #buildstream09:57
toscalixjjardon: juergbi I would like to discuss that in more detail09:58
juergbijjardon: afaik, the former09:58
toscalixhaving a plugin strategy where the criteria for "adopting as official" plugin is maturity..... is questionable09:59
juergbiit's not the only criterion09:59
juergbibut I don't see an issue with this being a criterion09:59
toscalixjuergbi: what are the other criterion10:00
toscalix?10:00
toscalixwork with master? work with the latest stable release?10:00
juergbiI'd say being useful for more than just a corner case and at least someone, better multiple devs willing to maintain it upstream10:01
juergbiit obviously needs to work with the version it's merged into and from that point forward keep working10:01
juergbiand have test cases10:01
toscalixthose are process and tech criterion. Now we are talking10:01
toscalixtech policy10:02
juergbiand I wouldn't typically accept new plugins into stable branches10:02
*** lachlan has quit IRC10:02
toscalixas I said, we should discuss them as guide. Not set them in stone, but have a couple of clear paragraphs that describes the conditions10:02
juergbithis is how I see it and I expect tristan to have a similar view but we'll have to clarify this with him10:03
toscalixI would like to see the maintainers/devs of the plugins providing input10:03
tiagogomesHow do you handle a plugin being moved to buildstream but an older version of bst-external being installed10:04
*** gitlab-br-bot has joined #buildstream10:04
toscalixtiagogomes: let's not digg in :-) Let's leave it as open question we need to clarify. I find it very interesting for our gathering10:04
bochechacs-shadow: about #670… it is fixed by !819, but that also introduces a new feature10:05
toscalixthe model might have a huge impact in future business models around the tool10:05
bochechacs-shadow: will both be applied to the 1.2 branch? or does it make sense to split the commit in two, so that the fix for #670 can be cherry-picked without the new feature?10:05
toscalixdo you guys remember the PHP framework catarsis ?10:05
toscalixthis topic became crutial for the survival of some of them10:06
toscalixit can become a huge differentiator factor for us10:06
toscalixif instead of considering it a downstream topic we consider it business model topic10:07
tlater[m]tiagogomes: Note that BuildStream already has features to specify from *where* you use a plugin10:09
tlater[m]In that case, if your project.conf specifies "use the plugin from x" it will override any default plugins.10:10
toscalixadded the topic to the list for the gathering https://wiki.gnome.org/Projects/BuildStream/Events#Proposed_topics10:11
*** gitlab-br-bot has quit IRC10:14
*** gitlab-br-bot has joined #buildstream10:14
*** lachlan has joined #buildstream10:17
*** solid_black has joined #buildstream10:18
juergbiany takers for !291 (git 'ref' improvement) review? jjardon: maybe you want to verify that this works for your purposes10:19
* tlater[m] managed to reproduce his first artifact server error :)10:26
tlater[m]Also running two buildstream instances concurrently is freezing up my laptop10:26
* tlater[m] needs a better laptop10:27
* jjardon will check later10:28
*** gitlab-br-bot has quit IRC10:30
*** gitlab-br-bot has joined #buildstream10:30
gitlab-br-botbuildstream: issue #680 ("FileBasedDirectory can contain stale cached information") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/68010:33
tpollardjuergbi: with mr813 is it expected that a user would upgrade their bst-artifact-server instance to utilise the batching by default?10:33
tpollardI ask because I'm adding to cascache.py, and I'm not 100% sure if that's the assumed environment to work against10:34
toscalixtlater[m]: maybe the solution is to provide a testing environment since I assume we will have to do many tests. I believe it is the intention. Can anybody confirm?10:34
tlater[m]toscalix: A testing environment for performance-intensive testing might be useful long term10:36
gitlab-br-botbuildstream: merge request (jmac/stop-caching-vdirs->master: Stop caching virtual directories if get_directory is used.) #818 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81810:36
gitlab-br-botbuildstream: merge request (mablanch/668-remote-build-failure->master: Better handle remote build failures) #825 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82510:37
juergbitpollard: while !813 requires an updated CAS server for batching, it supports fallback for old CAS servers10:37
juergbialso, 1.2.1/1.2.2 already include the server side10:37
tlater[m]If it's just this branch I don't think it will pay off, though. This is also the very first issue for which I've lacked performance, and it works just about. I'm not sure it's worth spending much effort on.10:37
juergbitpollard: for your changes to cascache.py: everything should keep working with servers that don't support batch downloads, but optimizations can be skipped in that case, if necessary10:38
tpollardjuergbi: thanks :)10:38
jmactiagogomes: !818 is now set to merge if & when the CI pipeline succeeds.10:39
jmacThanks juergbi & Kinnison for the reviews.10:39
*** lachlan has quit IRC10:40
mablanchIf anyone feels like looking at a remote-execution path, !825 needs a review. jmac, finn, if you have some time, comments are welcome :) What's the build failure handling patch.10:41
jmacI will take a look, mablanch10:42
mablanchjmac, Thanks! Notes on how to test it are on the related issue btw.10:43
tiagogomesjmac thanks10:44
juergbitlater[m]: have you seen yesterday's discussion between valentind and me with regards to expiry?10:47
tlater[m]juergbi: Not much of it, no, is there a mailing list thread about that?10:48
*** finn has quit IRC10:48
valentindtlater[m], I found 3 issues.10:49
valentindNot sure if this will help the bug reported. But I did get another bug when filling up the cache.10:49
tlater[m]valentind: Is this the one you commented about on the issue, where we set refs to nonexistent objects?10:50
valentindOne issue is that we remove objects that have no reference yet.10:50
valentindSo at the end of an upload we are in a bad state were a ref is incomplete.10:51
valentindYes I commented about the first issue.10:51
juergbitlater[m]: it started here: https://irclogs.baserock.org/buildstream/%23buildstream.2018-09-24.log.html#t2018-09-24T11:55:1010:52
valentindThen there are other issues. We check we have enough space for an object, but we store it then step by step without allocating the space. So concurrent upload could go over the limit.10:52
valentindThird issue was that we duplicate data by copying temporary files. So we need twice the size than we actually check for.10:53
juergbithe third one is now #67810:53
* tlater[m] thinks the second one is the issue actually causing #60910:54
tlater[m]I also think we can concurrently add and remove artifacts at the same time10:54
valentindOne problem with the third one is that this is where we actually get the error in the case of the ticket. It is possible we would get it somewhere else once fixed. But this is the copy that triggers the ENOSPC.10:54
juergbiI could look into solving that third issue now10:55
*** finn has joined #buildstream10:55
tlater[m]juergbi: Does grpc have any protections against concurrency issues?10:56
valentindI have a patch for the 3 issues. But it is intended to try to reproduce. Not that clean. Go ahead and fix things if you want.10:56
*** lachlan has joined #buildstream10:57
gitlab-br-botbuildstream: merge request (jmac/stop-caching-vdirs->master: Stop caching virtual directories if get_directory is used.) #818 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/81810:58
juergbitlater[m]: grpc server runs multi-threaded, so it's all up to the implementation10:59
tlater[m]Hrm, I wonder if we'll need more than allocating space to solve issue 2 then10:59
*** finn has quit IRC10:59
tlater[m]Because two concurrent instances will have different ideas of how much space is available/allocated11:00
juergbitry fallocate and handle error case by cleaning cache?11:00
juergbipossibly in a loop11:00
tlater[m]juergbi: What if one process is removing?11:01
tlater[m]I.e., cache cleanup/addition concurrency, like on the client end11:01
juergbifor that we need to solve issue 1 but I don't see how it affects issue 2 on its own11:01
*** lachlan has quit IRC11:02
cs-shadowbochecha: that's a good point, if we want the bug fix to be ported to 1.2, I can split it into two separate commits11:02
juergbivalentind: pushed somewhere? or are these workarounds that don't help with the proper fix at all?11:02
bochechacs-shadow: I'd really like it backported into 1.2, personally11:02
cs-shadowbochecha: okay, I'll do that. No problem11:03
juergbisounds sensible11:03
cs-shadowIt'll have to wait for 1.2.3 though I guess :)11:03
bochechaI'm reporting bugs to libabigail (ABI checker) regularly for the freedesktop sdk, and right now I always have to `bst checkout` a full element, then remove everything from the tree that is irrelevant11:03
bochechawith --deps=none I'd directly get the tree I want11:03
valentindjuergbi, not yet. I want to do some testing to understand.11:03
bochechathen I put that in a tarball and attach it to the libabigail bug report :)11:03
tlater[m]juergbi: Is issue 1 related to this case at all? I thought the problem was that we remove artifacts before setting their ref, and *then* set a ref to nonexistent objects.11:04
bochechacs-shadow: thanks!11:04
valentindjuergbi, I think we really need to manually test well filling up a cache server.11:04
tlater[m]valentind: If you like, I have a little script that starts a really small cache server in a docker container11:05
valentindtlater[m], I set up a cache on a mounted fs from a loop file.11:05
tlater[m]Fair enough11:05
juergbitlater[m]: yes, so that's concurrent remove/add, which we need to fix, e.g., with my suggestion of not purging recent blobs11:07
tlater[m]Ah, that would solve that too11:08
juergbivalentind: agreed, however, some improvements will make sense independent of whether they already fully fix all symptoms11:08
juergbivalentind: so is your solution to issue 3 something that could be useful as (base for) upstream fix? in that case i'll leave it for now. otherwise I'll look into a fix for that part11:09
tlater[m]valentind: I think it will be easiest if you just push those branches without MRs, we can help reproduce then, too :)11:11
valentindjuergbi, tlater[m], OK let me push a branch today. You can both look at it tomorrow and comment on it.11:17
valentindI am going to test it.11:18
juergbita11:18
NexusI'm getting this error when running a test, does anyone recognise it/know a fix?11:37
Nexushttps://pastebin.com/rdjju9s511:37
Nexusi can run the commands manually, it only fails when run as part of the test11:37
tlater[m]Nexus: Are you still playing with cross-platform support?11:39
Nexusyeeees?11:39
Nexus¬.¬11:39
tlater[m]Make sure you don't normally run as root11:40
tlater[m]Because that could a) screw up your cache on some platforms and b) hide errors like that11:40
Nexusyou have to on mac and WSL11:40
tlater[m]Nexus: In which case, make sure that CI is configured to run as root.11:40
juergbiNexus: why would root be required on either of those?11:41
Nexusthese are just local tests atm, ther is no CI11:41
juergbifor remote execution?11:41
Nexusjuergbi: no bubblwrap11:41
juergbi(or local non-sandbox tests)11:41
juergbithere is no sandbox run11:41
juergbithe dummy sandbox should not require this11:41
tlater[m]Ah, that's right.11:41
*** iker has quit IRC11:41
tlater[m]Nexus: In this particular case, it looks like you ran the test suite as root once and then as a normal user11:42
tlater[m]I'd remove the test temporary directories and try again11:42
Nexustlater[m]: i ran as root both times, and i have done, same issue11:42
*** lachlan has joined #buildstream11:43
*** iker has joined #buildstream11:43
Nexusjuergbi: i get `E           py.error.EACCES: [Permission denied]: rmtree('/Users/codething/buildstream/tmp',)11:43
Nexuswhen not running as root11:43
tlater[m]Nexus: That's a Windows path, no? Perhaps some setting of the host OS?11:44
NexusMacOS11:44
tlater[m]In any case, that's not related to sandboxing. It should run as non-root.11:46
juergbiNexus: something in tmp still has root permissions from earlier runs?11:48
juergbihave you tried tlater[m]'s suggestion of manually removing this first?11:48
juergbis/root permissions/root ownership/11:48
Nexusjuergbi: not that i can see11:49
*** finn has joined #buildstream11:52
tlater[m]toscalix: One note about the website - known issues that are fixed will show up as (closed), but may still need to be backported to the latest stable version. Is this intended?11:54
Nexusjuergbi: it seems to be a concurrency issue12:00
jmacI've got `bst` executing the wrong version of buildstream again... clearly executing an installed version which doesn't match the checked out code, despite installing with `pip3 install --user -e .`12:01
Nexusjmac: is it possible yu have 2 buildstreams installed?12:01
Nexusi have had that problem before12:01
skullmanjmac: long shot, have you run `hash -d bst`?12:02
jmacYes, but where? I've cleared out ~/.local/lib/python3.612:02
tlater[m]jmac: /usr/local/lib/python3.6 might also contain one12:03
jmacskullman: Yep - good advice, but it's not that this time12:03
toscalixyes, we only remove them when the fixes are released12:03
tlater[m]jmac: *also*, pip creates egg link cruft, so you may just have a symlink to a nonexistent binary lying around now12:03
Nexusjmac: i also have bst installed at ~/.local/bin/bst12:03
jmacbst --version returns 1.3.0+530.g662c729f which is the exact hash I'm on12:04
toscalixsince the target audience are users that consume releases, not contributors nor testers12:04
jmacbut it shows error messages which clearly don't exist anywhere in the source tree12:04
tlater[m]jmac: You definitely have leftover modules. Have you tried `pip remove buildstream; sudo pip remove buildstream`?12:04
jmactlater[m]: It's 'uninstall' rather than remove, but yes, done that12:06
gitlab-br-botbuildstream: merge request (willsalmon/outOfSourecBuild->master: Out of source builds) #776 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77612:06
tlater[m]jmac: Best way to get rid of the issue is to use `pdb` to figure out where the actual files for your modules are then.12:07
jmacCan I just set fire to my laptop and get a new one instead12:07
Nexusreinstall os12:07
jmacsnap12:07
*** lachlan has quit IRC12:08
tlater[m]jmac: Alas, usually when I accidentally screw up my python that's what I resort to...12:08
Nexusthis is why everyone should use VMs12:08
skullmanchroots normally enough12:09
jmacThat's somewhat artifical, unless we're recommending all users do that12:09
tlater[m]jmac: This is a problem that only crops up during dev though, and something that is caused by pip.12:09
Nexusi need to reinstall my OS at some point, i bricked my python about 2 months ago12:10
tlater[m]virtualenv can also help without masking the issue.12:10
Nexusbut that's effort12:10
*** solid_black has quit IRC12:10
Nexusi'm at the point where my Mac test system is more stable than my local one, so i run tests on there instead12:11
tlater[m]Does this warrant a known_issues entry? https://gitlab.com/BuildStream/buildstream/issues/66512:21
tlater[m]The stacktrace might creep up on and confuse new users, although it's technically not harmful in that BuildStream wouldn't continue doing anything anyway12:22
* tlater[m] is going through the issues list and distilling a number of issues that seem likely to cause problems for users12:23
Nexusthat's on my todo list i think12:24
Nexus66512:24
* tlater[m] would be happy to have a look at it after finishing the known issues page, too12:26
tlater[m]Just wondering if it's high enough impact to list, so that I can get a feel for what to include :)12:26
jonathanmawjuergbi: do you know the specific use-case behind https://gitlab.com/BuildStream/buildstream/issues/666? I've been trying to implement the tests for https://gitlab.com/BuildStream/buildstream/issues/667, but I don't think I understand it properly12:39
jonathanmawsince when I revert the fix to 666, the tests I did still pass12:39
Kinnisonjonathanmaw: and nothing since that fix could be masking the thing that 666 was about?12:43
adds68Was there any update on why the search functionality has stopped working on the documentation page?12:44
jonathanmawKinnison: hrm, possibly. I tried testing by rolling back to before that commit and trying a real repo instead of tests12:44
jonathanmawI'll give it a try12:45
tlater[m]valentind: Should known issues include a description of when the issue occurs that's less technical?12:47
tlater[m]Currently we only describe who is affected...12:47
gitlab-br-botbuildstream: issue #681 ("Search functionality in the Documentation has stopped working") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/68112:48
jonathanmawKinnison: correct, when I roll back to before and cherry-pick my tests back in, they pass :-/12:49
* jonathanmaw still doesn't understand the failure case12:49
KinnisonOh dear12:49
Kinnisonjuergbi: ^^^12:49
juergbiwill take a look12:52
gitlab-br-botbuildstream: issue #605 ("linux.py uses context message API before initialization") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/60512:52
adds68There is nothing in the buildstream documentation that states the limit of junctions: https://buildstream.gitlab.io/buildstream/advanced-features/junction-elements.html12:53
adds68Would anyone know if nested junctions possible?12:53
skullmanadds68: I junction in stuff from freedesktop-sdk and that has a junction to something, which suggests it works to some degree.13:00
adds68skullman, ah yes sorry i understand your point now, however i'm still not sure how to tell buildstream to look to that element inside the junction13:01
skullmanah, so it's a matter of how you would specify syntax for following the junction, hmm, I don't know that13:03
adds68skullman, any idea who would?13:04
adds68Who worked on junctions?13:04
adds68I guess maybe having 2 separate junction files would also solve this though13:04
adds68But it would be nice to know either way13:04
coldtomadds68: you can solve using two junctions for sure13:09
tlater[m]adds68: For reference, juergbi implemented junctions.13:09
tlater[m]I think he's by far the most knowledgeable.13:09
*** lachlan has joined #buildstream13:11
gitlab-br-botbuildstream: issue #429 ("BuildStream refuses to cache artifacts with files that have very strict permissions") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/42913:11
tlater[m]Our issues need a cleanup...13:14
* tlater[m] is unintentionally doing so by trying to reproduce old bugs13:14
gitlab-br-botbuildstream: merge request (richardmaw/push-failed-build-regression->master: tests: Add regression test for pushing failed builds) #824 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82413:22
tlater[m]Hmm... Should I detail know issues for which we don't really have a clue what's causing them yet?13:35
tlater[m]s/know/known/13:35
tlater[m]Things such as: https://gitlab.com/BuildStream/buildstream/issues/615 - we have no idea under which circumstances this occurs, and it clearly isn't every build.13:36
gitlab-br-botbuildstream: merge request (willsalmon/outOfSourecBuild->master: Out of source builds) #776 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77613:46
*** ikerperez has joined #buildstream13:56
*** finn_ has joined #buildstream13:56
*** iker has quit IRC13:57
*** finn has quit IRC13:57
*** ikerperez has quit IRC14:04
*** iker has joined #buildstream14:04
tlater[m]I'd appreciate it if someone could have a look at this: https://gitlab.com/BuildStream/website/merge_requests/9314:14
* tlater[m] hopes this matches what people had imagined the known_issues page to look like14:14
tlater[m]I'm also starting to wonder if we should include issues that have been fixed since (i.e. in the 1.2.x releases)14:15
toscalixReminder: please add yourself to the participants list if you are attending to the Gathering on week 42 https://wiki.gnome.org/Projects/BuildStream/Events14:24
*** catonano has joined #buildstream14:25
tlater[m]toscalix: That page presents itself as "immutable" to me14:28
toscalixtlater[m]: I mentioned in the ML that you need to have an account in GNOIME and be on a whitelist. jjardon can you add tlater[m] ?14:31
toscalixtlater[m]: nice job with the known issues14:32
*** solid_black has joined #buildstream14:39
*** raoul has joined #buildstream14:40
juergbijonathanmaw: I don't have any more information about #666. however, it's possible it was discovered on 1.2. have you tried the revert on 1.2 instead of master?14:48
juergbii.e., maybe it was avoided in master due to some other commits14:49
*** abderrahim has quit IRC14:49
*** abderrahim has joined #buildstream14:50
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72614:55
*** iker has quit IRC14:58
jjardontoscalix: sure, but anyone already in the whitelis  can add more people15:03
jjardontlater[m]: what is your username in gnome wiki?15:03
toscalixjjardon: did not know that. Checking how15:03
*** alatiera_ has joined #buildstream15:05
toscalixjjardon: do not see where15:05
jjardontoscalix: https://wiki.gnome.org/TrustedEditorGroup15:06
toscalixah, it is general for the wiki15:06
toscalixthought it was "per page"15:07
jmacLooks like my earlier problem which I thought was bst executing the wrong version was actually bst caching a build failure in CAS15:10
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:10
raoulTrying to import freedesktop-sdk with a junction but get a complaint about ostree namespace when build anything that depends on the junctions elements. As far as I'm aware I shouldn't be using ostree, but I have it installed anyway so not sure what's going wrong. Files and traceback: https://pastebin.com/JGpBGP8K15:11
jonathanmawjuergbi: I've just tried rolling back to before the commit "source.py: Fix re-instantiation" and trying my tests. They still pass :-/15:11
tpollardjuergbi: would you expect an older bst artifact server to be throwing exception errors for bath not to be implemented (whilst still seemingly behaving as before?)15:11
tpollard*batch15:12
juergbijonathanmaw: odd. unfortunately, I don't know any more than what tristan wrote on gitlab15:13
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:13
jonathanmawjuergbi: okay. It's not massively urgent, so I can wait until he gets back to explain what I did wrong15:14
juergbitpollard: hm, it's possible that the check whether batch is implemented will produce a server side exception that is logged but not otherwise harmful15:14
tpollardjuergbi: that seems to be the case15:15
tpollardjuergbi: I'm simply running it in a screen session, it's logging this constantly but not falling over https://paste.gnome.org/pql59loej15:16
juergbinot ideal but it's backported to 1.2, so shouldn't be a long term concern15:19
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:22
jjardonraoul: you probably need the python bindings for ostree15:23
jjardonraoul: what distro you use?15:23
tpollardjuergbi: agreed15:23
raouljjardon, on debian. I installed ostree with pip3, is that the bindings and I'm missing the actually library?15:24
jjardonraoul: if It's debian stable, check you have installed gir1.2-ostree-1.0 from backports15:24
KinnisonThere used to be docs about that15:25
jjardonraoul: ostree can not be installed with pip, so probably you actually dont have ostree installed, but only the bindings15:25
Kinnisonraoul: http://buildstream.gitlab.io/buildstream/install_source.html#stretch15:25
toscalixjjardon: in theory, we have a table on the feature page to reflect dependencies. I wonder if this is too specific to go there15:25
jjardontoscalix: don't know how raoul have setup buildstream or what guide is being used15:26
raoulah, I clearly missed that section, I'll do that now and see if that fixes it15:26
jjardontoscalix: but is certainly documented in the ling Kinnison just posted15:27
toscalixok then15:27
jjardonraoul: ostree is not used anymore in buildstream insternally, but It's used in freedesktop-sdk because we are using the ostree source plugin15:27
Kinnisonraoul: If any of that isn't clear, do let us know so we can improve the section15:27
raoulI did the debian section and then ignored stretch and bust and sid sections, apparently just need to read more carefully15:29
Kinnison:-)15:29
juergbiNexus: oh, I forgot to 'start discussion' for one comment: https://gitlab.com/BuildStream/buildstream/merge_requests/726#note_10419187215:36
juergbiNexus: also recheck your branches. right now workspace_list_error_message does NOT contain the workspace list change, but mac_fixes still contains that15:37
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:38
Nexusjuergbi: yup just noticed that xD15:38
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:41
Nexusjuergbi: resolved the commit message tweak15:41
juergbita15:47
*** lachlan has quit IRC15:47
*** jonathanmaw has quit IRC15:56
*** lachlan has joined #buildstream15:56
juergbijjardon: not urgent but it would be great if you (or anyone else with access) could upgrade the fdo/gnome artifact server to 1.2.2. this enables faster (batch) pulling of artifacts with bst master clients (the client side I'll backport to 1.2 after this real life testing)16:09
gitlab-br-botbuildstream: merge request (mac_fixes->master: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72616:10
jjardonjuergbi: ack, I will do the freedesktop-sdk one, for the GNOME one abderrahim if you have some time ^ (although I'd wait after the release maybe)16:11
gitlab-br-botbuildstream: merge request (mablanch/668-remote-build-failure->master: Better handle remote build failures) #825 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82516:13
abderrahimI already updated to 1.2.1, I think it has the fix16:13
* abderrahim hopes this bug is the reason why the aarch64 runner is slow16:14
*** toscalix has quit IRC16:14
juergbijjardon: thanks16:17
juergbiabderrahim: yes, 1.2.1 already has the fix. 1.2.2 is not needed on the server16:17
juergbiNexus: I'm seeing local test errors with mac_fixes branch, checking16:17
Nexusweird16:18
*** lachlan has quit IRC16:19
gitlab-br-botbuildstream: merge request (tiagogomes/issue-514->master: CI: test building freedesktop-sdk overnight) #826 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82616:24
*** catonano has quit IRC16:31
*** catonano has joined #buildstream16:31
gitlab-br-botbuildstream: merge request (tiagogomes/issue-514->master: CI: test building freedesktop-sdk overnight) #826 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82616:38
gitlab-br-botbuildstream: merge request (tiagogomes/issue-514-backport->bst-1.2: CI: test building freedesktop-sdk overnight) #827 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82716:41
*** lachlan has joined #buildstream16:42
adds68When junctioning from a project, i am seeing inconsistent pipeline errors, which is suggesting i track for a version16:48
adds68But when i track the junction element this does not resolve it16:48
adds68Any idea if i need to pass a flag to "bst track" in order to track the junction elements?16:48
valentindadds68,  you can pass the name of that junction element.16:50
adds68valentind, it does not fix the issue16:50
valentindStrange. Is it really the junction or an element in that junction?16:50
tpollardadds68: have you tried the deps & cross junction flags?16:50
adds68valentind, an element inside that junction16:50
mablanchjmac: Thanks for the review on !825. I don't have write access, would you mid merging it if your still ok with it?16:50
valentindYes, then there is a cross junction option.16:50
jmacmablanch: Sure.16:51
valentindBut you have to track something that depends on it.16:51
valentindOr you can always do "bst track junction.bst:element.bst"16:51
adds68tpollard, just --deps?16:51
valentindadds68, no, there is one more option.16:52
adds68I want a way to resolve all deps at once, there are several errors16:52
adds68All missing ref errors16:52
jmacmablanch: Can you rebase mablanch/668-remote-build-failure onto current master?16:52
tpollardadds68: --deps all16:52
valentindadds68,  --cross-junctions16:52
valentindBoth options.16:52
adds68haha good timing16:53
gitlab-br-botbuildstream: merge request (mablanch/668-remote-build-failure->master: Better handle remote build failures) #825 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82516:53
mablanchjmac: Done.16:54
adds68valentind, tpollard "bst track --deps all --cross-junctions"16:54
adds68Did not fix the error16:54
valentindadds68, what element did you pass as parameter?16:54
adds68Also passing the name of the junction file did not work16:55
valentindDo not pass the junction. Pass something using the junction.16:55
jmacmablanch: Thanks. I've set it to merge when the CI succeeds.16:55
adds68valentind, ok16:56
adds68Untrackable sources16:56
adds68    Requested to track sources across junction boundaries16:56
adds68    in a project which does not use project.refs ref-storage.16:56
valentindadds68, is the subproject using project.refs?16:56
adds68I guess not?16:56
valentindSo you cannot do that, YOu need to track manually in the source of the subproject.16:57
valentindOr convert it project.refs16:57
mablanchjmac: Cheers!16:57
adds68What is project.refs? Why is this optional?16:57
valentindadds68, what source type is the junction using?16:57
adds68git16:57
valentindadds68, project.refs is a file containing all the refs. Instead of having them in the elements.16:58
adds68valentind, why is this not automated by BuildStream?16:58
valentindThe problem in tracking non project.refs is that the files need to be modified. But buildstream is not going to commit changes to remote git repository for you.16:58
valentindBut project.refs however has one top file with all refs for all projects. So you do not need to modify any file.16:59
adds68valentind, yes i get that, but surely keeping an extra file containing all the refs isn't practical, or am i missing something here?16:59
valentindWell, in this case you have to track separately and commit your track in the subproject.17:00
adds68valentind, so this file can be generated by bst?17:02
adds68it just needs to be run once and the file committed to the project?17:02
valentindadds68, Checkout your sub project separately. Then track it. Commit. Then next track your junction to get that coomit.17:03
gitlab-br-botbuildstream: issue #514 ("BuildStream CI is not configured to perform any system level testing") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/51417:03
gitlab-br-botbuildstream: merge request (tiagogomes/issue-514->master: CI: test building freedesktop-sdk overnight) #826 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/82617:03
adds68valentind, so i need to bst track the project locally and push the file generated back to that project?17:08
valentindadds68, yes. Then you understand why project.refs can be useful.17:08
gitlab-br-botbuildstream: merge request (tpollard/494->master: WIP: Don't pull artifact buildtrees by default) #786 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/78617:08
adds68valentind, i am unsure if you mean my project, or the junction project17:09
valentindadds68, the subproject.17:09
adds68I have pulled the junction project and run bst track, but nothing was produced17:09
valentindMaybe there is nothing to track then.17:09
valentindadds68, why are you tracking?17:10
*** tpollard has quit IRC17:10
gitlab-br-botbuildstream: merge request (tiagogomes/issue-514-backport->bst-1.2: CI: test building freedesktop-sdk overnight) #827 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/82717:10
adds68valentind, because i am trying to junction the project, but bst is complaining about refs17:10
adds68Untrackable sources17:11
adds68    Requested to track sources across junction boundaries17:11
adds68    in a project which does not use project.refs ref-storage.17:11
valentindWhat command gives you this error?17:11
adds68valentind,  bst track giggle.bst --deps all --cross-junctio17:12
adds68ns17:12
adds68giggle.bst is juntioning gnome-build-meta17:12
valentindAnd gnome-build-meta uses project.refs, right?17:12
adds68valentind, i am not sure17:13
adds68this project also junctions freedesktop-sdk17:13
*** phildawson has quit IRC17:13
adds68But the error is giving me nothing to suggest what is the issue17:13
valentindadds68, You mean giggle.bst is depends on the junction element to gnome-build-meta? Or you mean that it is the junction element to gnome-build-meta?17:14
adds68valentind, the first point17:14
valentindadds, --except freedesktop-sdk/all.bst17:16
valentindadds, --except freedesktop-sdk.bst:all.bst17:16
valentindThe second one.17:17
valentindGiven freedesktop-sdk.bst is the junction freedesktop-sdk17:17
valentindadds68, I missed the "68"17:17
*** solid_black has quit IRC17:18
adds68valentind, ok that fixed the error, but now i see: Failed to load Element plugin 'collect-integration': No module named 'pluginbase._internalspace._sp25013cb2b5486823f1a1b94f9752f747.collect-integration'17:20
adds68valentind, do we need to add project.refs to freedesktop also?17:20
adds68valentind, i also have bst-external installed17:21
valentindadds68, you need bst-external.17:21
valentindIt is collect_integration.17:22
valentindI think you have an old version of freedesktop-sdk.17:22
valentindYou want to track the junction.17:22
adds68valentind, i am junctioning 18.0817:22
adds68valentind, so after tracking the freedesktop-sdk junction, i now see the Untrackable sources error again haha17:23
juergbiKinnison: I've pushed mac_fixes with two new commits at the bottom but it probably will require some further tweaks17:26
gitlab-br-botbuildstream: merge request (mac_fixes->master: Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72617:27
valentindadds68, do you have a repository ready to test?17:31
valentindadds68, it sounds to me like there is a bug. But I am to lazy to reproduce your project.17:31
*** lachlan has quit IRC17:43
Kinnisonjuergbi: I shall look at those commits17:48
Kinnisonjuergbi: I'm currently eating my dinner, so it won't be quick17:48
*** lachlan has joined #buildstream17:55
*** lachlan has quit IRC17:57
*** finn_ has quit IRC18:00
Kinnisonjuergbi: Looks good to me18:08
Kinnisonjuergbi: do you want me to say so on-MR ?18:08
Kinnisonjuergbi: Interestingly some of the tests are failing, and it doesn't seem to be proceeding.  You may need to reinvestigate :(18:33
*** catonano has quit IRC18:40
*** catonano has joined #buildstream18:50
*** finn has joined #buildstream19:08
*** finn has quit IRC19:43
*** catonano has quit IRC21:19
*** bochecha has quit IRC21:32
*** alatiera__ has joined #buildstream23:29
*** alatiera_ has quit IRC23:31
gitlab-br-botbuildstream: merge request (chandan/bst-checkout-build->master: Add `--deps build` option to `bst checkout`) #819 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81923:48
gitlab-br-botbuildstream: merge request (chandan/fix-checkout-none-1.2->bst-1.2: Ensure `--deps=none` option works for `bst checkout`) #828 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/82823:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!