IRC logs for #buildstream for Tuesday, 2018-08-14

*** tristan has joined #buildstream05:00
*** tristan has quit IRC05:20
*** tristan has joined #buildstream05:20
*** ChanServ sets mode: +o tristan05:21
*** tristan has quit IRC06:23
*** toscalix has joined #buildstream06:23
*** toscalix has quit IRC06:53
*** toscalix has joined #buildstream06:56
*** tristan has joined #buildstream07:09
*** ChanServ sets mode: +o tristan07:17
*** finn_ is now known as finn07:19
valentindtristan, I think an easy fix for the mirroring thing is provide the fetchers as a generator rather than a list. So that first fetcher, fetches main mirror, once it is done we refresh the submodules and provide the rest of the fetchers.07:52
*** toscalix has quit IRC08:03
tristanvalentind, you are also requesting an API break ?08:04
valentindtristan, I think that the current way it is written we can use generators.08:05
valentindI still have to test.08:05
valentindYes. It works. 3 lines change.08:06
valentindRunning the full test suite now.08:07
tristanvalentind, check https://irclogs.baserock.org/buildstream/%23buildstream.2018-08-08.log.html#t2018-08-08T12:34:3908:07
tristanvalentind, and my responses to those queries08:07
tristanand consider if your implementation would still work08:07
valentindtristan, It will work like before. I am not changing behavior.08:10
valentindWhat I do is only delaying scanning for submodules just a bit later.08:10
valentindFor my case it does not matter if using aliases and mirrors or not. They will all behave the same.08:11
qinustyI have removed fatal-warnings: True and addressed all review points tristan, https://gitlab.com/BuildStream/buildstream/merge_requests/627/ is ready when you get some spare time.08:12
tristanqinusty, I know :)08:13
qinustyOkay :D08:14
tristanvalentind, I'm a bit concerned mostly because I don't have the time to properly digest all the knowledge I need to evaluate that08:15
tristanvalentind, could you harden the git tests for mirroring + submodules, to check the edge cases discussed in that IRC log ?08:15
valentindtristan, yes, I will do that.08:16
tristanvalentind, thanks !08:16
valentindAll backported!08:19
tristanvalentind, :)08:19
tristanyep, I babysat them this morning, I have my finger on the release trigger but was waiting for those to come through :)08:19
valentindThanks for clicking on merge tristan. It took sometime for the queue to go through08:19
tristanThanks for the backporting !08:19
tristanyes, it's annoying that CI takes so long in the cloud :-S08:20
tristanat least I've seen no spurious errors since the last ones were fixed, so that's something08:20
*** toscalix has joined #buildstream08:45
*** jonathanmaw has joined #buildstream08:47
*** rdale has joined #buildstream09:00
*** bethw has quit IRC09:11
*** bethw has joined #buildstream09:11
*** johnward has quit IRC09:11
*** milloni has quit IRC09:11
*** johnward has joined #buildstream09:11
*** milloni has joined #buildstream09:12
*** Nexus has quit IRC09:12
*** Nexus has joined #buildstream09:12
*** tristan changes topic to "/BuildStream 1.1.6 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://buildstream.gitlab.io/buildstream | IRC logs: https://irclogs.baserock.org/buildstream | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmap"09:20
*** tristan changes topic to "BuildStream 1.1.6 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://buildstream.gitlab.io/buildstream | IRC logs: https://irclogs.baserock.org/buildstream | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmap"09:20
tiagogomesWe should time the tests, check the ones that take longer and see what we can do to optimize them09:26
tiagogomesImproving the bst startup time would reduce the time for them to finish09:26
tristantiagogomes, true; I still feel that the real bottleneck though is going to be I/O in the cloud09:31
tristantiagogomes, what takes the longest time is staging runtimes in the integration tests; or worse, downloading those runtimes when the cache is not up to date and needs to be recreated (the gitlab cache)09:32
tristanwhat normally takes around 10min (or 12 ?) to run locally on my laptop, takes > 40min (or > 1hrs ?) to run on a much "more powerful" build server09:33
*** bochecha has joined #buildstream09:37
paulsherwoodare we using ephemeral runners?09:40
paulsherwoodwould be better to have some permanent ones maintaining caches and gits09:40
*** CTtpollard has joined #buildstream09:44
*** leopi has joined #buildstream09:45
*** tpollard has quit IRC09:46
*** leopi has quit IRC09:48
*** sergey_ has joined #buildstream09:49
*** sergey_ has joined #buildstream09:51
*** sergey_ has quit IRC09:52
*** solid_black has joined #buildstream09:52
tristanpaulsherwood, I think we do use caches, jjardon knows the config best10:00
tristanpaulsherwood, what I am not 100% certain of, is if they have good SSD, and if the CPU and memory is guaranteed to be on the same machine as the SSD10:01
tristanI think so though, because apparently it's one "big" machine10:02
jjardonbuildstream uses the ephemeral runners from baserock, as the ones from gitlab.com was not so good10:02
tristanoh10:02
valentindtristan, !656 for #537 will be ready for review when the CI has succeeded. There is no limitation on submodule discovery (mirroring or not). The fix is 3 lines. There are 4 new tests (2 from jonathanmaw, 2 from me). I will have some lunch, then I will look at the ostree mirroring issue.10:05
tristanvalentind, awesome sauce... I have two other MRs in my queue and yet another email to consider; not sure I will get to review it today :)10:07
jmacjuergbi, is there a more up-to-date branch of buildbox I should use than juerg/wip?10:08
juergbino, that's up-to-date10:09
jonathanmawvalentind: ooh, that's clever!10:09
juergbi(latest change yesterday)10:09
jmacTa juergbi10:11
*** toscalix has quit IRC10:30
WSalmonI was looking for more sources that might have the missing track/ref error as prompted by tristan in https://gitlab.com/BuildStream/buildstream/merge_requests/621#note_93616304 and the only place i could find a issue was in ostree but i cant see many tests mentioning ostree and there doesn't seem to be a set of ostree specific tests, is this, fine/know issue/should i make a issue?10:48
qinustyDoes anyone recall the discussion surrounding buildstreams UI, what information is provided in different regions of the UI etc. Was there an issue raised?10:50
tristanWSalmon, I'm pretty sure it can happen for `bzr` as well as `ostree`10:57
WSalmoni had a look and it looked like track was mandatory so not a issue but i will relook now10:57
tristanWSalmon, iirc (ask jonathanmaw for more detail), with bzr we use both the track and ref to construct the full ref; where a ref is a sub-version of track (not sure I recall it properly), still there is an opportunity to specify a version in the ref which does not exist for the track specification10:58
WSalmonI thought there was a different ticket for ref not in track? It looked to me like the configure would go pop under most failure modes already but i am looking again11:00
*** WSalmon has quit IRC11:04
phildawsonqinusty, I believe https://gitlab.com/BuildStream/buildstream/issues/552 is related to that discussion. I'm not sure it was a direct result though.11:05
*** WSalmon has joined #buildstream11:05
*** CTtpollard is now known as tpollard11:13
WSalmontristan looking at bzr configure the lack of default will, if i understand correctly, cause a failure if track is not specified, and thus not alow our error condition. As for the ref not in track is this not a different issue? both in terms of failure mode and GL ticket, i think tpollard was looking at it. Or do you want me to suck more stuff in to this MR? I have a patch ready to go to add the no track or ref for ostree but i am not sure about11:18
WSalmonwere to add the tests as ostree seems to be very lightly tested11:18
jmacjuergbi: Is there any way to tell the difference between a command failing inside buildbox, or buildbox itself failing (e.g. due to bad arguments)?11:36
juergbijmac: not yet but we should definitely add this11:37
tpollardWSalmon: yep, my MR was to cover the specified ref not existing in the specified track11:41
WSalmontpollard, did you check that your issue was only in git, it looks like it might be a issue in bzr too11:45
tpollardI kept it to the scope of the issue which was for git, I can well believe it would apply to a bzr source too11:45
jmacActually, I suppose a lack of a digest being printed on stdout by buildbox is enough to distinguish the cases11:46
*** leopi has joined #buildstream11:47
juergbijmac: please note that the latest version only supports --input-digest= and --output-digest= instead of the original stdin/stdout11:48
juergbito allow stdout (and stdin) to be passed through from the command11:49
juergbian empty (or not existing) output digest file is indeed a possibility to check whether buildbox failed11:49
jmacOh, interesting, I don't think the buildgrid plugin knows about that yet11:49
finnit won't do, I haven't touched that file since the demonstration at GUADEC11:51
finnit won't do, I haven't touched that file since the demonstration at GUADEC11:52
jmacnp, looks like an easy fix11:52
tristanWSalmon, I have a feeling we've been talking about different issues indeed, my mistake12:08
tristanWSalmon, lemme take a look at the link you provided12:08
tristanWSalmon, ah right... ok this is a different issue indeed :)12:11
tristanWSalmon, so to answer one question; the majority of coverage we get for Source plugins is by virtue of implementing a `Repo` for the source in tests/testutils/repo12:12
tristanWSalmon, so while some ostree related edge cases might not be covered, there is still a lot of coverage by that avenue12:12
tristanWSalmon, indeed, bzr would not have this issue12:13
tristanWSalmon, it could make sense to add a little test for this, it would only need to run `bst show` (as it's a load time error)12:14
tristancs-shadow, have a moment to chat about source transform / pip ?12:30
WSalmontristan i will have ago at a test for ostree, thanks for the info12:33
tristanWSalmon, unfortunately I did not get a chance to look at your email regarding out of tree builds today, it's in my queue, though12:37
tristancs-shadow, I'll have to run, but I've left some more comments on the `pip` plugin12:55
tristancs-shadow, a couple of last things I think we overlooked; but the tests look good !12:56
*** TingPing has joined #buildstream12:56
cs-shadowtristan: sorry, I was afk. Just got back. I'll go through your comments in a bit12:56
*** edb has joined #buildstream13:00
TingPingHey, so I'm looking into the issue about generating a manifest as part of the output. One piece of information missing from that would be the version of each module, I imagine just adding a version property to everything is the only real solution to that, does that sound reasonable?13:02
*** edb has quit IRC13:02
*** edb has joined #buildstream13:04
tristancs-shadow, mostly just (A) Lets be symmetric with `directory`, unless you think there is a good case to not do what I suggested there13:04
tristancs-shadow, and (B), I think there is a side effect of inadvertent tracking, we should make sure13:05
tristancs-shadow, sorry I was not able to pay more attention today :-S13:05
tristanI think it's mostly golden though, docs might get improved later but definitely good enough for landing13:06
tristanI didn't go through the `pip` implementation in minute detail, but found one issue with it; I think we've discussed it enough too for it to be ready13:06
tpollardwith the talk around generating manifests, elements don't currently have a concept of licensing do they?13:09
jmacNope13:10
tpollardjmac: cheers13:12
jmacMy element's `build-root` is set to /buildstream/base-sdk-bootstrap/gcc-stage1.bst" - is it normal to have the element name on the end?13:12
tristanjmac, yes13:13
qinustyTingPing, We don't exactly have a version of sources. This issue has been discussed on the mailing list  (see https://mail.gnome.org/archives/buildstream-list/2018-August/msg00045.html).13:13
tpollardprojectconfig.yaml sets 'build-root: /buildstream/%{project-name}/%{element-name}' by default jmac13:13
valentindLook at ostree, I think we had a bug where if you changed the url after you had already fetched something in the cache. And then change the ref. It would still try to pull from the old url.13:13
tristanjmac, we did that mostly because we want to be able to stage all the sources optionally in a shell, and we want them built with distinct paths13:14
valentinds/Look at/Looking at/13:14
tristanjmac, for debugability in the shell13:14
jmactristan: Seems a bit surprising. I expected something called *-root to be a directory13:14
jmacEspecially as it's passed into sandbox.set_work_directory13:14
qinustyThe author of bst files could indicate the versions of their sources in a property similar to url: or ref: but that is not currently in place.13:14
qinustyTingPing ^13:14
qinustytiagogomes are you working on #561?13:15
TingPingqinusty, right, are unknown properties already just ignored?13:15
qinustyProperties are validated and loaded in to sources in their Configure() method13:15
tristanjmac, I think its rather inconsequential; alternatively the whole thing could be prefixed `/buildstream/build/%{project-name}/%{element-name}`13:15
tristanjmac, if we wanted that communicated more clearly; then we'd have every build staged into a shell under `/buildstream/build`, might be more discoverable13:16
jmacOh wait, you're saying gcc-stage1.bst is the name of a directory13:17
tristanjmac, since we haven't implemented the feature yet which uses it, it might make sense in 1.3 to prettify the directory name; it would be another core artifact version bump13:17
tristanjmac, yes13:17
jmacOK, that makes more sense13:17
qinustyTingPing, What you're suggesting is a change to Source.py, probably adding a "version" tag to Source.COMMON_CONFIG_KEYS and loading it within source.py.13:17
tiagogomesqinusty not at the moment, but I plan to work on it when I finish some work I am doing13:18
qinustyOkay no worries tiagogomes13:18
TingPingqinusty, alright, and from what i'm reading tristan thinks the manifest generation belongs out of tree?13:18
tristanI think manifest generation needs to be supported by features that we need to add, but not added as a concept in BuildStream proper, right13:19
qinustyThe way the discussion had progressed is that if buildstream can provide a formattable output with the necessary data, then a bash script can read and parse the necessary information13:19
* tristan passed 10pm... gonna run now :)13:19
* TingPing just wrote this a few minutes ago: https://gist.github.com/TingPing/958554c5a8f82ef84fe6e5a4434d84fd13:19
qinustyLater tristan13:19
TingPingqinusty, version probably belongs as part of the top level elements not sources, for my needs at least13:20
*** tristan has quit IRC13:23
qinustyOkay, I do think tristan is along the right lines with producing a parse-able output. The idea behind the manifest was originally to collate Elements and their source urls/refs from all bst files used.13:23
TingPinghonestly what is there now is good enough, just a list of yaml files to read13:24
qinustyMy problem there is that you're having to parse a directory tree of files which may not be loaded in a build13:25
* skullman grumbles about having to patch ruamel 0.14.12 to compile for python 3.713:25
qinustyThe manifest produces a list of elements included within the build13:25
TingPingi assumed bst show knew which files are used13:25
* qinusty thanks skullman for his suffering13:25
qinustyIt does13:26
qinustySo as long as we can get bst show to produce the necessary data we need, we're good13:26
TingPingwell thats what i used13:26
qinustyAhhhhhh, you're loading files from the elements shown in bst show. That'll do it13:27
TingPingqinusty, some of what it returns aren't file paths though: bootstrap-junction.bst:attr.bst13:28
TingPingah thats a subproject?13:29
qinustyYes it appears the colon seperates a junction13:29
qinustyYou'd need the path to the junction for that13:29
TingPingso then i have to bst show each of those13:30
* qinusty assumes it bst shows the junction too, just not the full path13:30
qinustyit seems "bootstrap-junction.bst:" is effectively an alias for the junction project path13:30
TingPingbut that path isn't known13:31
qinustynot in the current UI13:31
qinustybuildstream knows it13:31
TingPingok, i guess ill see if i can improve that13:31
*** edb has quit IRC13:32
TingPing(import.py is a terrible python module name)13:46
qinustyimport import13:49
TingPingfrom import import Import13:51
qinusty:/13:51
TingPinginvalid syntax13:51
qinustyfrom .import import Import?13:51
qinustyThe word import is starting to look wrong13:51
TingPingno you simply can't use that keyword to import it, you need to import as a string with __import__ or implib, etc13:52
qinustyNot even with the .?13:52
TingPingnot even13:52
qinustyrip, slight flaw there13:52
TingPingqinusty, you happen to know how to get a path from a junction - https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugin.py#L706-70913:56
qinustyUmmmmm, Junction is an element, Element inherits from Plugin, Plugin has access to __project...... TingPing.14:14
qinustyNot sure what public channels there are14:14
qinustyhmm, but the Junction element __project is the main project.14:15
qinustyjunction has self.path14:15
TingPingqinusty, junction.path is empty14:16
qinustyself.path = self.node_get_member(node, str, 'path', default='')14:17
qinustyIf the junction is configured within the project, it should be loaded once the element is configure()'d14:17
TingPingmaybe show never gest that far14:18
qinustyPerhaps14:19
WSalmonshow will call configure on sources and there for presumably elements too14:20
qinustyNope TingPing, configure is definitely called when bst show is ran14:21
qinusty^14:21
tiagogomespytest is not showing stack traces. Is there a way of seeing those?14:30
WSalmonthey are usually quite a long way up the terminals scroll back before the coverage report, your not just missing them there? other wise i dont know14:32
qinustyDid you run with -s?14:32
*** tpollard has quit IRC14:35
*** WSalmon has quit IRC14:35
*** tpollard has joined #buildstream14:36
*** leopi has quit IRC14:37
tiagogomesYes I am running with -s14:41
NexusI've come across a blocker, in that certain os commands don't work on Mac, do i raise an issue, or just fix it?14:43
cs-shadowthank the gods! Links in GitLab email notifications appear to be fixed now14:51
laurencecs-shadow, yes ! return of the 'view on gitlab' button in email notifications !14:59
laurenceNexus, are you blocked on WSL side of things?15:01
Nexuslaurence: not entirely15:02
laurenceNexus, ok cool, then raise it and move on is my advice :)15:09
*** leopi has joined #buildstream15:32
qinustyCan someone review https://gitlab.com/BuildStream/buildstream/merge_requests/662 for merge on pipeline succession?15:51
Nexus qinusty: commented15:55
Nexusqinusty: basically, tests?15:55
*** solid_black has quit IRC16:08
*** adds68 is now known as ads6816:11
*** ads68 is now known as adds6816:11
qinustyNexus, good shout. Probably best for a regression. I'll take a look16:13
*** edb has joined #buildstream16:17
*** xjuan has joined #buildstream16:19
Nexusis there a reason we're using `IS_LINUX = os.getenv('BST_FORCE_BACKEND', sys.platform).startswith('linux')` rather than `IS_LINUX if "Linux" in os.uname().release` ?16:21
Nexussorry, os.uname().sysname16:22
juergbios.uname() may not be available on all platforms16:27
juergbiwe should aim to keep direct OS conditionals to a minimum, though16:28
juergbiif it's about a particular feature that is currently only supported on Linux but might be supported on other systems later, we should define a separate condition for that feature, imo16:28
Nexusjuergbi: it came up because i needed a way of differenciating WSL and true Linux16:29
Nexusos.uname().release tells me it's a Windows Kernel16:29
Nexusthe rest returns Linux16:29
juergbiit might be better to detect the absence of FUSE if that's the reason for the conditional16:31
*** mohan43u has quit IRC16:32
*** mohan43u has joined #buildstream16:32
Nexusjuergbi: there may be other things, and WSL intend to eventually support FUSE16:32
Nexusi think it's still best we have a flag for WSL16:33
juergbithe question is what do you use the flag for16:33
Nexusskipping tests mostly16:33
juergbiif you want to skip tests that require FUSE, it should be a FUSE conditional, not a WSL conditional16:33
juergbiotherwise 1) we have to update all these tests again when WSL finally supports FUSE16:33
juergbi2) we'd have to add another conditional for other FUSE-less Linux cases16:33
juergbithat said, it could make sense to initialize the FUSE flag based on whether we're running on WSL - if there is no simple more direct check for FUSE16:34
juergbior maybe the conditional should be even more generic: support for local builds16:35
Nexushow would i check that?16:36
juergbithis is mainly about the name of the conditional. as currently we expect FUSE available for local builds, we can set the 'local build' conditional based on FUSE availability16:36
juergbihowever, if and when we support local builds with native WinAPI at some point, we most likely will not use FUSE for local builds16:37
Nexustrue16:37
Nexusassuming we always require fuse to build locally16:37
juergbistrictly speaking, we only need it for integration commands and other read-write root cases, but not for simple builds16:38
juergbidon't know whether it's worth differentiating16:38
juergbiprobably not16:39
juergbibtw: I think IS_LINUX should remain True on WSL. WSL should be considered an alternative implementation of the Linux API. can require extra quirks but it should still be considered IS_LINUX16:40
Nexus i was planning on keeping it true16:40
juergbiok, good16:41
*** tristan has joined #buildstream16:58
* tiagogomes shakes a fist to the artifact cache and parallel jobs16:59
*** jonathanmaw has quit IRC17:05
jmacRemote execution is updated and running the alpine example again \o/17:14
jmacTrying it on freedesktop-sdk/bootstrap overnight17:15
*** phildawson has quit IRC17:16
juergbigreat17:24
jjardonHi updateing from 1.1.5 to 1.1.6 our CI is rebuilding the workd: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/8920455217:31
jjardonis this expected? there was a change in the cache key calculation?17:32
edbnice jmac17:45
*** leopi has quit IRC17:46
*** leopi has joined #buildstream17:52
*** edb has quit IRC18:39
*** edb has joined #buildstream19:07
*** leopi has quit IRC19:25
*** edb has quit IRC19:52
adds68has anyone ever seen an "inconsistent pipeline"error when trying to build an element?20:08
*** ChanServ sets mode: +o tristan20:10
tristanadds68, that happens when you try to build, but some sources don't have a ref set; the error should mention which specific elements have sources that are in an inconsistent state; and need either to be tracked or have a ref set manually20:11
adds68tristan, thanks it does give the element, which does have a ref and it has not been modified either, so i am not sure why it has started to fail20:11
adds68This is seen on both 1.1.5 and 1.1.620:12
adds68Running bst track also did not fix the issue20:12
tristanadds68, please file an issue with the steps to reproduce then, and we'll prioritize that for 1.220:13
adds68tristan, will do, thanks :)20:13
tristanadds68, no, ... thank _you_ :)20:14
*** leopi has joined #buildstream20:15
adds68tristan, i think i found the issue20:24
adds68tristan, i opened a workspace for that element, then just rm -r the directory once i had finished, as i did not know you had to "close" the workspace20:25
adds68Ever since i did that it would report that error20:25
adds68Now i've actually closed the workspace it builds \o/ i'll open an issue with these steps though :)20:25
tristanadds68, yes please do; it would be helpful to improve the error for that case20:26
jjardontristan: did you see my message about the cache key?20:37
tristanjjardon, nope20:39
tristanjjardon, it's mentioned in the release email for 1.1.6, though20:39
jjardonoh, I only checked the NEWS file20:40
* jjardon looks20:41
jjardontristan: perfect, then it works as expected. Sorry for the noise20:42
tristanjjardon, yeah I didnt mention all the specific fixes in the NEWS, but indeed, sad rebuild of the world20:47
tristanit's for the sake of improved determinism at source staging time20:48
*** tristan has quit IRC21:47
*** xjuan has quit IRC22:02
*** leopi has quit IRC22:09
*** bochecha has quit IRC23:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!