IRC logs for #buildstream for Tuesday, 2018-02-06

gitlab-br-botbuildstream: issue #233 ("Do not push artifact that have been downloaded from the cache") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/23301:17
*** ruben has quit IRC01:25
*** ruben has joined #buildstream01:28
*** ruben has quit IRC01:31
*** Prince781 has joined #buildstream02:06
*** Prince781 has quit IRC03:57
*** mcatanzaro has quit IRC04:16
*** noisecell has joined #buildstream06:34
*** noisecell has quit IRC06:37
*** noisecell has joined #buildstream06:39
*** noisecell has quit IRC06:53
*** noisecell has joined #buildstream06:53
*** aday has joined #buildstream09:17
*** adds68 has joined #buildstream09:37
*** adds68_ has joined #buildstream09:58
*** adds68 has quit IRC09:59
*** ssam2 has joined #buildstream10:13
*** adds68_ has quit IRC10:44
*** adds68_ has joined #buildstream10:44
ssam2juergbi, my worry about overlayfs, which i haven't researched much yet, is how well it works at layering many filesystems10:47
ssam2the interface seems to only support layering 210:47
juergbiyes, i don't know whether it would be suitable for the many-layer case10:48
juergbihowever, it would be very useful for the current hardlinking approach10:48
ssam2ah, yes it would10:49
juergbias we can separate the writes into the upper layer (which could still be FUSE for #38 and other things)10:49
juergbihttps://gitlab.com/BuildStream/buildstream/commit/0e0990c5557f3b454f54f412d3c28c4bd4b4aaf710:49
juergbissam2: does the above patch look good to me as a short term solution for now?10:50
juergbieh, to you10:50
ssam2makes sense, yeah10:50
ltureminder that there's the monthly meeting at 14:00 UTC today. Any additions to the agenda, please go ahead and respond to my mail to the last from last week.10:51
juergbiit fixed the issue for vala for me even with -j3210:51
ssam2tlater, can you reply to the agenda mail  to add this UNIX-platform staging iussue ?10:52
*** aday has quit IRC10:53
* tlater is in the process of doing so10:54
gitlab-br-botbuildstream: merge request (juerg/fuse-rlimit->master: Increase the soft limit for open file descriptors) #260 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/26010:55
*** adds68_ has quit IRC10:56
gitlab-br-botbuildstream: issue #232 ("fd leak when generating epiphany tarball") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/23210:57
gitlab-br-botbuildstream: merge request (juerg/fuse-rlimit->master: Increase the soft limit for open file descriptors) #260 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/26010:57
*** adds68 has joined #buildstream11:00
*** aday has joined #buildstream11:07
*** adds68 has quit IRC11:31
*** adds68 has joined #buildstream11:31
*** toscalix has joined #buildstream12:02
*** noisecell has quit IRC12:02
*** aday has quit IRC12:03
*** aday has joined #buildstream12:04
tlaterDo workspace commands not fetch anymore?12:05
* tlater sees a lot of workspace-related test failures while rebasing his completions branch12:05
*** adds68 has quit IRC12:08
*** adds68 has joined #buildstream12:09
*** ruben has joined #buildstream12:14
*** ruben has quit IRC12:26
*** jonathanmaw has joined #buildstream12:30
juergbihm, i'm not aware of a change in that regard12:37
tlaterI'm not sure what's happening here12:38
tlaterAs far as I can tell it looks like the elements are put in the FetchQueue as expected12:39
tlaterBut they are never actually fetched12:39
tlaterTrying to set `skip_cached=False` makes buildstream get stuck calculating some cache key12:39
* tlater thinks about just rewriting the branch from scratch12:40
tlaterHm, the process() method of the fetch queue isn't even called12:42
*** tristan has joined #buildstream12:46
* tlater suspects this broke with the element state changes12:58
juergbitlater: if skip_cached=False didn't work anymore, `bst fetch` would also be broken, wouldn't it?13:02
tlaterYeah, which confuses me13:02
gitlab-br-botbuildstream: merge request (completion-optimizations->master: WIP: Completion optimizations) #256 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25613:04
gitlab-br-botbuildstream: issue #234 ("Allow viewing build logs from artifacts") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/23413:07
*** mcatanzaro has joined #buildstream13:14
gitlab-br-botbuildstream: merge request (completion-optimizations->master: WIP: Completion optimizations) #256 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25613:21
*** sstriker has joined #buildstream13:36
gitlab-br-botbuildstream: merge request (completion-optimizations->master: WIP: Completion optimizations) #256 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25613:44
gitlab-br-botbuildstream: merge request (212-git-source-needs-a-way-to-disable-checking-out-submodules->master: WIP: Resolve "Git source needs a way to disable checking out submodules") #259 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25913:46
gitlab-br-botbuildstream: merge request (completion-optimizations->master: Completion optimizations) #256 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25613:51
ssam2buildstream team IRC meeting is starting in 9 minutes  in #buildstream-meetings13:51
ssam2all welcome13:51
*** sstriker has quit IRC13:51
*** sstriker has joined #buildstream13:52
gitlab-br-botbuildstream: merge request (completion-optimizations->master: Completion optimizations) #256 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25613:53
*** mib_u3lusx has joined #buildstream13:55
*** mib_u3lusx has left #buildstream13:56
*** mib_u3lusx has joined #buildstream13:57
*** gnurlu has joined #buildstream13:58
*** dplayle has joined #buildstream13:58
*** dplayle has quit IRC14:01
*** Benjamin has joined #buildstream14:10
*** noisecell has joined #buildstream14:26
*** hashpling has joined #buildstream14:46
*** ssam2 has quit IRC14:50
*** ssam2 has joined #buildstream14:50
*** noisecell has quit IRC15:01
mcatanzaroRegarding https://gitlab.com/BuildStream/buildstream/issues/232. "Disable parallel builds" is not a really great solution for build software....15:05
ssam2agreed :-)15:06
ssam2https://gitlab.com/BuildStream/buildstream/commit/0e0990c5557f3b454f54f412d3c28c4bd4b4aaf7 should hopefully fix the issue15:06
ssam2if not ... we are going to be reworking all this code in due course anyway15:06
ssam2we need to fix the infamous issue #38 for example (https://gitlab.com/BuildStream/buildstream/issues/38)15:06
ssam2so now that we know it's an issue, we can mitigate it15:07
*** gnurlu has quit IRC15:08
juergbimcatanzaro: please let me know whether the issue is fixed for you as well in master15:09
mcatanzarojuergbi: I'll try master in a sec (but I doubt raising the rlimit for open files actually does anything on Linux?)15:09
juergbiit fixed the issue for me. why don't you think this would help on Linux?15:10
juergbiit does depend on what the soft and hard limits are set to on your system / for your user, though15:10
juergbiwe can hopefully avoid requiring that many file descriptors in the future but that will take a while15:11
mcatanzarojuergbi: Is the soft limit really lower than the hard limit? OK, I'll try15:12
mcatanzaroAnyway, I'm now more worried about https://gitlab.com/BuildStream/buildstream/issues/227#note_57840688 (new comment)15:13
juergbiyou can check ulimit -n vs. ulimit -n -H15:13
juergbiit's 1024 vs 4096 here15:13
juergbitaking a look15:14
mcatanzaroOh cool!15:16
*** hashpling has quit IRC15:18
juergbioh, maybe that's a cache key conflict15:18
mcatanzarojuergbi: I tried the new version and it did avoid the fd limit, thanks15:18
juergbithanks for testing15:18
juergbitristan: the potential cache key issue we talked about a week ago or so: the cache key of the platform-specific bootstrap/toolchain does not apply to weak cache keys15:19
juergbiso it's actually trivial to get (weak) cache key conflicts right now with different platforms and i suspect that's what's happening here15:20
juergbior maybe not15:27
*** noisecell has joined #buildstream15:29
tristanjuergbi, whooooops15:31
juergbiprobably not the exact issue here but still something we need to fix15:31
tristanjuergbi, the question is whether to backport a fix to 1.0 I guess ?15:31
tristanwell15:31
tristanahhh I guess it can happen for weak cache keys yes15:32
juergbimcatanzaro: libbsd.so exists here in the sandbox15:38
tlaterssam2: Is there a reason the docker plugin hasn't landed in buildstream-external yet?15:38
juergbierror 24 is actually the same file descriptor issue again15:38
ssam2no blockers as far as I know15:38
juergbimcatanzaro: did you see these test failures with master or was this before the update?15:38
mcatanzarojuergbi: It's from before the update, I'll try again15:39
tlaterCould we land that and the alpine image? I'd like to merge my docs for that soonish.15:39
ssam2what do you think juergbi ?15:39
ssam2sorry to pull you 3 ways at once :)15:39
juergbii haven't reviewed these changes but i don't have any general objections15:41
tlaterjonathanmaw could have a look, I suppose, he's probably closest to the plugin code atm15:42
jonathanmawtlater: would that be !9: "Add Docker source plugin" by ssam2 ?15:44
* tlater believes so15:44
tristanjuergbi, I started hacking something for it just to see, and here is my conundrum15:44
* jonathanmaw can have a look15:45
tristanjuergbi, I want to ultimately use the execution environment to inform cache keys, and have the sandbox inform a unique key much like plugins do15:45
tristanbut there is no sandbox at cache key calculation time :-S15:46
tristanjuergbi, I guess it would be safe to assume at least that once a project is loaded, the execution environment of an element is resolved and will not change15:47
persiaCan we be confident that a sandbox is deterministic based on description, and do we have the sandbox description at cache key calculation time?15:47
tristanWe dont want that rule by sandbox type, at least15:47
persiaAh, right.15:48
tristanI.e. we want a sandbox type which can provide alternative execution enviroments15:48
tristanCreating a sandbox for each element will increase the memory footprint a lot (but I'm not really sure what that footprint *is* right now)15:49
tristanhmm, it's not a lot of data though, mostly static code15:49
tristanhrm, requires churn anyway15:50
tristansandbox directory configuration at least15:50
jmacI keep getting  "bwrap: execvp sh: No such file or directory" while trying to write manual install rules, and I can't remember what it means15:53
juergbitristan: i would definitely expect the information at load time to suffice15:53
ssam2jmac, it either means your sandbox has no /bin/sh, or it has a symlink at /bin/sh that points to something that doesn't exist, or you are missing the dynamic loader such as /lib/ld-linux.so.215:54
jmacThat's odd15:55
jmacMissing dependency on base/base-configure.bst15:56
tristanjuergbi, it's a lot of churn to do it right, maybe we need a quicker fix16:00
tristanIf we add something temporary directly in element.py, we can later delegate that to Sandbox() after some churn, without bumping the artifact version a second time16:02
juergbifor now we could put something like `uname -sm` in the cache key16:02
juergbithe cache key will still change again unless the Sandbox (at least on major platforms) produces exactly the same unique key16:03
tristanright, however it ends up being implemented (it may be that the Sandbox need only be told what the execution environment is)16:03
tristanjuergbi, I'm assuming that the bwrap and chroot sandboxes will continue to calculate the cache key in the same way16:04
tristanalthough it will merit a break later when we add something like linux vs bsd into the mix16:04
juergbiyes, we'll need that\16:05
juergbiright now, we're not at risk of conflict between linux and non-linux because ostree is linux-only16:05
tristanis there a generic way to get it in python ?16:05
persiaMaybe inject more of uname in early to avoid the break?16:06
tristanor try to parse /etc/os-release and weird things like this16:06
juergbithat's way too specific16:06
persiaI don't think /etc/os-release exists for most unices16:06
juergbiand we also don't want to make the cache key distro-specific on linux16:07
tristanmore of uname looks good16:07
persiaIndeed, I don't have /etc/os-release on a local Mach/Darwin environment.16:07
tristanand has a python layer16:07
tristanos.uname()16:08
persiaI think we need to stick to things that are required by the Single UNIX Specification16:08
tristanso will maybe do something useful on windows even16:08
persiaNot all of uname, as part of it is comment (often used for things like build ID, etc.)16:08
tristanthe first field and the arch16:09
tristanLinux and x86_6416:09
persiaThat seems a reasonable match.16:09
juergbii'm wondering whether we actually might need to change the approach a bit for weak cache keys16:10
persiaBe nice if we could get a userland string to go with that, but I'm not sure there is a portable way to do that (for things like GNU/BSD or similar)16:10
juergbiproject options/variants that influence the bootstrap or something else low level (e.g., whether libselinux is used) are currently not taken into account for weak cache keys either16:11
juergbiwell libselinux would be in the recursive dependency list but there might be something else that is not covered by dependency names16:11
persiajuergbi: Unless the base runtime included libselinux, making it opaque to the build scripts16:12
juergbiindeed16:12
juergbione alternative would be to bring weak cache keys closer to strong cache keys. just stripping out the source refs of dependencies but leave all other key contributions in16:13
juergbii.e., command and option changes in dependencies would be reflected in weak cache keys, only code changes would be ignored16:14
juergbihm, that's actually still not sufficient for the example of base image variants with project option conditionals16:15
juergbi(as the unique key doesn't include project options per se)16:15
tristanoptionality significantly changes things which individually contribute to cache key16:15
juergbiyes but when stripping out source refs, we would lose some of that again16:16
tristanyes it's possible to weak cache key collide with something built against a depenedency with a different option turned on16:16
tristanstripping out source refs ??16:16
juergbifor weak cache keys16:16
tristanOh you mean the cache keys of the dependencies16:16
juergbiyes16:16
juergbii was simply thinking about alternative possibilities16:16
*** csoriano has quit IRC16:16
tristanyes well, it is possible to collide with alternative configurations16:17
*** csoriano has joined #buildstream16:17
juergbiyes and i this might be a practical issue for GNOME where non-strict is even in the recommendation16:17
tristanjuergbi, I'm not sure how problematic that is in practice though16:17
juergbiit depends on who has push access to the remote cache, i suppose16:17
tristanI'm reluctant to strengthen cache keys with optionality :-/16:18
juergbithe biggest practical risk i see right now for GNOME is weak cache key conflicts for different architectures16:21
juergbithe rest is mainly theoretical for now but we should further look into it16:22
tristanjuergbi, is there an issue number for this ?16:22
tristanarch/os thing ?16:22
juergbii don't think so. can file one16:23
tristanhold on...16:23
gitlab-br-botbuildstream: merge request (cache-keys-os-arch->master: Cache keys to consider OS and machine arch) #261 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/26116:27
tristanjuergbi, !261 above should do it I think16:27
juergbiyes, that looks like a reasonable change. we may have to give this more thoughts in the future, though16:32
gitlab-br-botbuildstream: issue #235 ("Add the ability to generate manifest files") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/23516:39
jmacIs there a pre-written element for getting Python setuptools onto a system?16:42
ssam2how about https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/sdk/elements/base/python3-setuptools.bst ?16:45
jmacI'll give it a try, thanks16:49
*** ruben has joined #buildstream16:57
jmacSo many dependencies...17:00
tlaterAck17:00
tlaterDid I just accidentally push to origin/master?17:00
tlaterWhy is the branch not protected? ;-;17:00
tlaterIs it safe to force-push a branch without my last commit?17:01
persiajmac: Maybe base your work on a platform that contains the necessary dependencies?  I know there are some based on debian, freedesktop-sdk, and baserock floating around.17:01
persiatlater: It is safe if nobody else has pulled.  There is no logging of pulls.  Less time is better when you change history.  Better rule: don't change history.17:02
persiaIn an environment with active CI, CI always pulls very quickly, making force-push impossible to be safe.17:02
* tlater is highly tempted to change history before someone pull17:02
tlater8s17:02
persiaCommon workaround for CI environments is to push a reversion commit, undoing all the changes.17:03
jmacThat's an option17:03
persiagood practice is to protect branches that are commonly pulled :)17:03
* persia doesn'T actually know the policy for buildstream17:03
ssam2i think we don't have one17:03
persiatristan: juergbi: ^^ ??17:03
ssam2and i think if it is protected, those with superpowers can push anyway17:04
ssam2so i would say let's enable protection for 'master'17:04
persiassam2: Yes they can.17:04
* persia believes in only giving robots superpowers, but that requires lots of time programming the robots17:04
*** ruben has quit IRC17:05
tlaterI wasn't even aware I had permissions to push to master :|17:05
*** ruben_ has joined #buildstream17:05
* tlater figures this commit can't hurt much, since it's only docs, but still feels annoyed at the implications on history17:05
juergbiforce push on public master branches should really be avoided17:06
*** ruben_ is now known as ruben17:06
juergbinobody might notice in this case but we should probably still not do it17:06
tlaterI'll just keep the commit and hope for it to get lost under lots of merges this week17:06
jmacI don't think anyone intends to17:07
persiatlater: Submit an MR correcting the bits that need correcting soonest, maybe?17:07
*** noisecell has quit IRC17:07
persiajuergbi: Do you have sufficient superpowers to protect public master so that it is hard to accidentally push there?17:07
tlaterpersia: Far from complete, unfortunately, I pushed a .rst file to see what gitlab would spit out when rendering it17:07
juergbimaster is already protected17:08
tlaterAh17:08
juergbionly 'Masters' can push17:08
persiatlater: Were I in your shoes, I'd consider finishing that bit of work my highest priority.  I have made that class of mistake before when working with CI, which led to more intensity than I usually commit in trying to restore the branch to a status I was happy others using (at the time, I was in a far different timezone than other developers, which made that easier).17:09
persiajuergbi: Are any other permissions associated with "Master"?17:09
juergbimaybe membership management? i'm not sure17:09
*** toscalix has quit IRC17:10
tlaterYou can specify permissions for individual users - perhaps 'Master' is just too generic17:10
persiaThat was my thought.  If someone can override protection without knowing, perhaps we're not being careful enough about permissioning.17:11
*** ruben has quit IRC17:12
ssam2i presume tlater was given master permissions to be able to configure CI17:12
*** adds68_ has joined #buildstream17:14
*** ruben has joined #buildstream17:14
*** ruben has quit IRC17:14
*** adds68 has quit IRC17:15
*** ruben has joined #buildstream17:16
persiaThat makes sense, and is a good thing.  I suppose it's just communications that need improvement.  I don7t want to take away superpowers, just reduce accidents.17:17
*** ruben has quit IRC17:17
tlaterHrm, I suppose I'd have made the same mistake otherwise - did not realize I had my master branch checked out17:18
* tlater is sorry for the disruption :|17:18
persiaThe nice part of the disruption is that we now have a draft policy for force push: "force push on public master branches should really be avoided".17:19
jmacIt needs work17:19
*** ernestask has joined #buildstream17:20
*** ssam2 has quit IRC17:41
*** jonathanmaw has quit IRC18:01
jjardon[m]Hi, can anyone here guide me where do you want this info? https://gitlab.com/BuildStream/buildstream/issues/22618:04
jjardon[m]also, any news about the possibility to have a pre-release tag?18:05
persiajjardon[m]: We discussed having a dev release next week in today's meeting.  Would that meet your needs?18:07
jjardon[m]sure18:08
*** adds68_ has quit IRC18:08
jjardon[m]Its more a conviniant thing so I can tell possible users of our project to "Use this tag" instead "use this SHA"18:09
gitlab-br-botbuildstream: issue #230 ("Error when trying to "bst shell" in a component") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/23018:32
gitlab-br-botbuildstream: merge request (image-authoring->master: Image authoring documentation) #262 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/26218:44
tlater^ That is the MR to fix my oopsie from earlier, but I don't think it should be merged before a few other things are merged18:46
tlaterI guess this will have to happen tomorrow18:46
* tlater departs, still somewhat shaken18:48
jjardon[m]tlater: in our project no one can push directly to master, all the changes should go through merge request. we can do the same here but maybe still allow tristan juergbi to commit directly to master19:40
jjardon[m]tristan: do you want me to activate review aprovals, so at least 1 person needs to approve before merge?19:41
*** adds68_ has joined #buildstream19:45
persiaWhile I personally like the "all changes need to be approved by the robot" model, it does break common git workflow and require using a web UI to do things.19:46
persia(even if the robot is just counting folk who pressed the "approve" button)19:47
*** mcatanzaro has quit IRC19:55
jjardon[m]persia: you can approve from the command line. But yeah as I said is totally configurable20:00
juergbistrictly applied it can conflict with the current policy of only fast forward merges20:01
juergbias branches in other repositories can't be rebased in the web UI20:01
jjardon[m]persia: not sure if it breaks the git workflow though: you still need someone to aprove your patch even if you send it attached in an email20:01
jjardon[m]ah, thats a limitation yes20:02
persiaThe webUI doesn't have a rebase button?  Wow!  This is the first time I've ever heard about a UI feature where gerrit is ahead of other tooling :)20:05
juergbipersia: it normally does20:06
juergbithe issue is that the source branch may be in a different repo and then there is no rebase possibility for me as approver20:07
persiaBut not when you enable the feature jjardon[m] is describing?20:07
persiaAh, right, that makes sense.  The approver would need to have access to the source branch to do that.20:07
juergbiexactly20:07
juergbiwe grant developer access to contributors, so everyone could create branches in the main repo at least after the first contribution20:08
juergbibut it's not always the case20:08
persiaDoes that mean that approvers would be able to rebase those contributions using the UI?20:08
juergbiyes, it works for branches in the main repo20:09
juergbi(if the rebase is possible without conflicts)20:09
persiaAh, right, then my preference would be to turn it on, and allow a very small number of special people to push manually for the special case of rebasing out-of-repo MRs.20:09
persiaAnd, yes, I don't imagine any robot can safely rebase where the rebase isn't clean: git assumes that requires human intelligence, and I suspect it is right.20:10
juergbithis could indeed be sensible20:10
juergbione thing that gitlab could support but doesn't is allowing to change the source branch of an existing MR20:10
juergbithen i could manually rebase it and push it to a new branch in the main repo, updating the MR20:10
persiaOr allow collaboration on MR source branches in some way.20:11
juergbiyes20:11
connorshea[m]this'd be the issue for that https://gitlab.com/gitlab-org/gitlab-ce/issues/2229220:15
*** ernestask has quit IRC20:19
connorshea[m]also this is the best issue I can find for changing the source branch https://gitlab.com/gitlab-org/gitlab-ce/issues/3295220:19
connorshea[m]also Carlos opened an issue for this, apparently https://gitlab.com/gitlab-org/gitlab-ce/issues/3975620:22
*** valentind has joined #buildstream21:04
persiaconnorshea[m]: Thanks!  Those are great to be able to track easily.21:19
jjardon[m]Hi! Is it possible to build for armv7 with aarch64 runners? I think I'm hitting a incompatibility when trying to doing that: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/merge_requests/54#note_5792177721:31
*** mcatanzaro has joined #buildstream22:13
*** aday has quit IRC22:32
*** mcatanzaro has quit IRC22:40
*** adds68_ has quit IRC23:06
*** valentind has quit IRC23:12
*** tristan has quit IRC23:27
*** mcatanzaro has joined #buildstream23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!