IRC logs for #buildstream for Thursday, 2018-03-08

*** Prince781 has joined #buildstream00:58
*** Prince781 has quit IRC01:55
*** Prince781 has joined #buildstream01:56
*** mcatanzaro has quit IRC03:12
*** tristan has joined #buildstream05:13
*** tristan has quit IRC05:15
*** tristan has joined #buildstream05:15
*** Prince781 has quit IRC05:35
gitlab-br-botbuildstream: issue #285 ("Get story straight with private symbols") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/28505:52
gitlab-br-botbuildstream: issue #286 ("Impossible to distinguish sandboxing errors from build errors") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/28608:11
juergbitristan: regarding the symlink sdist issue. if we have to keep working around it, i think we should at least check for accidental symlinks in CI to avoid mysterious failures08:20
juergbijmac: btw: nice catch08:20
tristanHmmm08:31
tristanjuergbi, maybe a part of the regular `./setup.py test` which ensures there are no revisioned symlinks ?08:32
tristanPlus a CI step which runs the tests on an uninstalled buildstream from git ?08:32
tristanWhich, would probably also be a good test anyway ?08:33
tristan(sans the integration part, perhaps)08:33
* tristan doesnt like the idea of adding tests which "only happen on gitlab"08:33
juergbii think we should guarantee that the test files are the same and then we don't need to run them twice08:34
juergbiideally, our dist tarball would just be git archive, as we desire for other projects as well08:35
tristanI'm not sure thats plausible for distribution via pip, though08:35
juergbiwhy do we actually need sdist at this point?08:35
tristanWe need it until we switch to meson or something, and mandate real host installation08:36
tristanBecause of the magic stuff sdist does I think08:36
tristanOr, maybe we dont ?08:36
juergbipip locally works from git, though, doesn't it?08:36
* tristan on crack ?08:36
tristanright08:36
juergbiat least that's how i install it08:36
tristanyeah, and since we are anyway not suitable for pypa... it's probably a song and dance we dont need08:37
tristanjuergbi, there is one reason we need it though, I feel anyway08:37
tristanwhich is accurate versioning information, to be present regardless of whether the history is there08:38
tristanThere needs to be this distribution song-and-dance where we auto generate the version metadata which ends up in a release08:38
tristanand ensure that a git describe thing is appended to the base tag version08:39
tristanThat will be true regardless of what mechanism is chosen for distribution08:39
juergbiright, hm08:39
juergbieven keeping that, it may be worth switching from sdist to something very simple (essentially git archive with the version file added)08:41
juergbigiven that sdist has bad behavior08:42
juergbi(or switching to meson or so but that's a bigger change)08:42
juergbialthough there is already !196 for meson08:42
juergbii would definitely miss -e, though08:43
tristanYes me too :)08:45
tristanUgh... python dist mess better set aside in some way for now :-/08:45
tristanthen again, jhbuild requires make install on every change, I expect I'd get used to it08:46
tristanbut -e is really nice to have08:46
juergbimaybe we could support running it without any installation at all08:47
juergbidirectly from the source directory08:47
tristanthe story is probably bigger than just this, too08:47
tristanBecause of how we rely on setuptools for pure python dependencies08:47
tristanI suspect in a bright shiny future, we bundle those into our installation and guarantee what we release is exactly what we release08:48
tristansupporting running from source directory is one thing; but it cannot be completely self contained (we need to introspect host man page directory and where to install bash completion scripts)08:49
juergbimay be useful for end users but would be distro unfriendly, so should be a separate option08:49
* tristan thinks this talk is premature08:50
juergbiyes, but maybe get rid of sdist in the short term08:50
tristanSo get rid of sdist, in favor of... a script which does the version generation ?08:51
juergbiyes. it doesn't sound ideal but as it's so simple and sdist doesn't support symlinks...08:53
tristanYeah that could be okay08:53
tristanI'd keep the dist step in CI08:53
tristanand nuke the MANIFEST.in08:53
juergbiyes, that sounds fine08:54
juergbiwe'd just fix dist, not replace it08:54
tristanAs a side effect, it would be *nice* if we fixed `-e` such that the version info is introspected dynamically, *when* running from a git checkout08:55
tristanright now you only get a new `bst --version` output for every `./setup.py install`08:55
tristanhaven't been strictly considering that a bug, but we probably should08:56
*** benzea has quit IRC09:04
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28309:14
gitlab-br-botbuildstream: merge request (jmac/restore-brackets-282->master: status.py: Restore brackets to time in job display area) #308 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/30809:14
*** jonathanmaw has joined #buildstream09:17
jonathanmawHi, I'm looking into source mirroring for buildstream, and was planning to compare it to buildstream's remote artifact cache, but I'm having trouble finding where the code for that is09:18
tristanjonathanmaw, I would say, don't jump the gun on source mirroring; there are a bunch of aspects to consider09:19
tristanOn the one hand, if we had a single list of mirrors configuration, and a server side helper which facilitates mirroring using already built-in features (like it's mostly a variant of `bst fetch`)09:20
*** valentind has joined #buildstream09:21
tristanThen we could have a simple configuration story, and pretty easy to setup mirroring09:21
tristanOn the other hand, we could have a more complicated config story; allowing say, alternative urls for each source alias09:21
tristanAnd then we could externalize the actual mirroring process entirely; plausibly supporting more exotic setups09:22
tristanOr... <insert your creative and better idea here>.09:23
tristanthere is always that third alternative :D09:23
tristanAnother thought is that the second approach does not really prevent the first, more hands-on / automated approach09:24
tristanSo it could be a matter of getting some functionality now, while leaving the door open for more functionality later09:24
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28309:26
*** tiago has joined #buildstream09:48
Nexusis tristan still around?09:52
tristanYep09:53
Nexustristan: please could you have a look at https://gitlab.com/BuildStream/buildstream/merge_requests/310 for me? :)09:53
tristanNexus, you mean https://gitlab.com/BuildStream/buildstream/merge_requests/310#note_62214798 ?09:54
Nexusoh yeah :D09:54
Nexusty09:54
* Nexus wonders why he didn't get an email09:55
*** dominic has joined #buildstream10:06
gitlab-br-botbuildstream: merge request (jmac/remove-sequence-id->master: Remove sequence ID.) #309 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30910:07
gitlab-br-botbuildstream: merge request (jmac/remove-sequence-id->master: Remove sequence ID.) #309 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30910:09
NexusSo how do i define a new %{foo} variable for projectconfig.yaml to use?10:20
*** valentind has quit IRC10:20
tristanNexus, I think one part, is that you would define `element` as an empty string, beside `max-jobs`, with a similar note about how that is a magically automated variable10:23
tristanNexus, then you would define %{build-root} as `/buildstream/%{element}`10:24
tristanNexus, the final part, would happen in Element initialization10:24
juergbiwe already have project-name somewhat similar10:24
juergbimaybe call it element-name for consistency?10:25
tristanin a similar way that you had the original patch, only you would be assigning the element name to the variable10:25
tristanjuergbi, +110:25
tristanAlso10:25
tristan  build-root: '/buildstream/%{project-name}/%{element-name}'10:25
tristanMight be a good idea10:25
tristanWould be a good idea I think10:25
juergbisounds sensible10:26
juergbifollows the approach in the artifact cache10:26
juergbiwe just can't have a project called 'install'....10:26
Nexusso do project-name in the same way as element?10:27
tristanAs a nitpick, `project-name` is code in _project.py, it could also use an empty declaration in projectconfig.yaml for documentation purposes, at the same place.10:27
tristanNexus, project-name is already assigned, undocumented10:27
* tlater wonders if source-bundle with junctions may potentially break lacking %{project-name} in the build path.10:28
Nexusso define element in element __init__ and project-name exists but should be added to projectconfig?10:28
tristanjuergbi, oops, good point10:28
tristanjuergbi, OR, we set %{install-root} = `/buildstream-install`10:28
tristanI think that is nicer, because when you want to be in a shell and have your build trees around; it's just sexier without the extra `/build/` path component10:29
Nexuswant me to change that too?10:30
tristanNexus, Yep :)10:30
tristanand yes to your last, sounds about right10:30
Nexuskk :)10:30
tristanDo we need a specific test case for this ?10:30
tristanseems like, if current tests pass, it works10:31
tristanNexus, there is always a chance that there is stray hard code referring to `/buildstream/build` and `/buildstream/install`, that may have to be fixed consequently; I suspect in the bst-external repo that may happen10:33
Nexusi'll have a look10:34
Nexustristan: is ` self._variables['project-name'] = self.name` the line you were refering to that sets the projectconfig value?10:44
tristanhmmm, still no backpointer from Source -> Element ?10:44
tristanNexus, it is, it's a line which exists in _project.py10:45
Nexusok so  self._variables['element'] = self.normal_name would work in the element init?10:45
tristanerrrm10:46
tristanself.name I think ?10:46
tristanWhat do we still use normal_name for...; oh yeah, artifact cache strings ?10:46
Nexusi THINK normal name replaces spaces with '-' but i'm not certain10:46
tristanit does10:46
Nexuswhich is what would be needed for a path10:46
tristanno it does not10:46
tristanit only replaces the `.`10:47
tristanartifact cache does a more extensive thing10:47
tristanAnd normal_name, probably needs to go10:47
Nexusright ok, so use self.name10:47
tristanYeah, self.name is the element directory relative bst filename10:47
tristanSo, it is already a valid filename10:48
tristanand it also happens to be what we mostly use to display to users10:48
Nexuscool10:48
tristanah right, normal_name splits the path separators, so core/totem.bst becomes core-totem.bst10:48
tristanin *any* case, stick with element.name please :)10:49
* tristan thinks we use normal_name for build directories in ~/.cache/buildstream/build, but even that is unnecessary10:50
tlaterIs the utils._process_list (and utils.{link,copy}_files) helper guaranteed to work with files in any order? https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/utils.py#L61410:51
* tlater suspects directories need to be listed before files contained in them10:51
tlaterBit of a shame, I'd love to use set operations for some stuff here :|10:52
tristantlater, required, see comment in beginning of same function10:56
tristantlater, I think we have a corresponding test case too which exposes the problem if you dont have that sort10:56
tlatertristan: My question is if that sorted(filelist) call there is enough to do the sorting for me should I give it an unsorted list10:57
tristanIt should be10:58
tristanheh10:58
tristantlater, that is a mind-numbing topic, cant answer off the top of my head to be honest10:58
* tlater hoped someone could ;)10:59
tristanI guess a better answer is, if it breaks; we need a test case + fix10:59
Nexusah, normal name replaces the '/' with - and removes the .bst11:00
tristantlater, which means "trust it"11:00
* tlater will trust it11:00
tlater:)11:00
tlaterWhoo, cheap set operations!11:00
tristanNexus, right11:00
tristanNexus, probably *also* used in artifact paths; and cached on Element to avoid repeating the thing11:01
tristanNexus, since you have me thinking of this... I've spotted another bug11:03
tristanIf a user sets `project-name`, `max-jobs` or now `element-name` in their project.conf or element.bst files; we should probably bail out with LoadError11:04
tristanNexus, I'll file it separately11:05
Nexusit seems like _project self.name is not declared early enough to work in this? it's coming up as blank11:05
tristanOh ?11:05
tristanThat smells like a regression11:05
tristanWe chopped up project load at some point I think11:06
Nexusrunning `print("project-name: ".format(self.name))` the line before `self._variables['project-name'] = self.name` in _project.py returns project-name:11:06
tristanNexus, that seems impossible, it's correct11:06
tristanWeird11:07
tristanNexus, look at the _load() function though, name is the first thing that is every loaded11:07
tristanwaaaay before that statement11:07
tristanAh11:08
tristanNexus, your format string is to blame !11:08
Nexus??11:08
tristantry "project-name: {}"11:08
Nexusdid i do a dumb?11:08
Nexusfk.11:08
NexusxD11:08
Nexusproject-name: gnome11:08
Nexusneat11:09
NexusSO11:09
Nexusthat works, but build-root: "/buildstream/%{project-name}/%{element}"  seems to run with %{element} having no value11:09
Nexushmmmm11:09
tristanRight, that *all* happens before Variables() is ever instantiated11:09
tristanso all of that manually setting of stuff, happens in yaml loaded nodes11:10
tristanOnce you instantiate the Variables() instance, all bets are off; build-root wont resolve again11:11
Nexusi'm doing it way before then11:11
tristanI dont see how11:11
Nexusself.__variables = Variables(variables) iscalled on ln 183 for me, i'm running my code on ln 15411:12
Nexusbut my code may be wrong, as i was trying to hack togather something similar to _project11:12
tristanWouldnt it be done after __extract_variables() ?11:12
Nexusit says i can't modify that value11:13
tristanIt says ?11:13
Nexus1 sec11:13
NexusVariables' object does not support item assignment11:14
Nexusbut i did it too late11:14
Nexusdoing it after __extract_variables works11:14
tristanRight11:14
Nexusthat gives me /buildstream/gnome/base-ninja11:14
Nexusas my path11:14
tristanNot /buildstream/gnome/base/ninja.bst ?11:15
tristanWhat do we prefer then ?11:15
Nexusno, atm i'm using normal_name11:15
tristanjuergbi, thoughts ?11:15
tristanI think... use name11:15
Nexusthat is a lot of subdirs though11:15
tristanBecause we always show the user the name11:15
juergbihaving the directory ending with .bst is a bit odd11:15
juergbibut it doesn't really matter to me11:16
juergbibut then we should use it everywhere11:16
tristanI agree, and would want to push for that11:16
juergbialso for build directories11:16
Nexus/buildstream/gnome/base/ninja.bst <-- output of name11:16
tristanI would want to kill normal_name entirely frankly11:16
Nexushavign a dir end with .bst seems a bit confusing to me11:17
tristanit's a source of confusion internally, and externally; I dont see why to have it either; give the user always one name11:17
Nexusbut i can see why you don't want normal_name11:17
juergbiwe do currently use normal_name in the artifact cache but i don't think we really need to11:17
tristanYeah, it would be an artifact version bump there too11:18
Nexusso, decision?11:27
tristanNexus, element.name11:28
Nexusso /buildstream/foo/bar/baz11:28
Nexusbar.baz*11:28
Nexusare you making a seperate issue for removing instances of "normal_name"?11:29
gitlab-br-botbuildstream: issue #287 ("Special / Magic variables are unprotected") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/28711:29
tristanYes11:32
gitlab-br-botbuildstream: issue #288 ("Kill Element.normal_name variable") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/28811:33
Nexuslol11:34
gitlab-br-botbuildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31011:38
gitlab-br-botbuildstream: issue #260 ("Regression: Not possible to import/export artifacts using different build architecture.") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/26011:40
gitlab-br-botbuildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31011:46
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28311:52
jennisjuergbi, tlater re what we discussed about the pylint configuration file, the earlier commits on that branch should (hopefully) be what you wanted11:53
jennis^^11:53
juergbijennis: fedora-6? or is that a typo?11:55
tlaterjuergbi: I used fedora-6 afaik11:56
tlaterImage ID 422dc563ca32, to be exact x)11:57
juergbiit's just an odd version number11:57
juergbii assume it has nothing to do with Fedora 611:57
juergbii.e., i have no idea how to get from fedora-6 to the version11:58
Nexusomg all of the /buildstream/install hard codes D:11:59
tristanheh11:59
tlaterjuergbi: Huh, docker has odd versioning here, it seems. Looking at it more closely it's 2712:00
tlaterjennis: Probably want to amend that commit then, my bad12:00
Nexustristan: in ./buildstream/plugins/elements/script.yaml how should i deal with the buildroot def?12:00
jennistlater, just reword yeah?12:01
tlaterYep12:01
tristanNexus, remove the assignments in there, but make it a documented comment in same file12:01
tristanNexus, in tests, looks like there are not that many12:01
tlaterAlso, jennis, there are some changes here that I'd probably not make: https://gitlab.com/BuildStream/buildstream/merge_requests/283/diffs?commit_id=10aed0531244d04591ae8e8db6c1e038e48c570512:01
tristanUmmm12:01
Nexustristan: i've changed them all, there were ~12 i think12:02
tristanNexus, might be a bit more complicated for script element...12:02
tristanWe may need to change that12:02
tlaterjennis: I feel that's probably down to the confusing way in which we patched this... Mind if I just amend that commit for you?12:02
jennisgo for it12:03
tristanNexus, ok - I got it12:04
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28312:04
* tlater wonders if there's a nice way to make a temporary commit so I can switch branches *without* having to reset said commit later12:04
juergbitlater: stash12:04
tristanNexus, remove the mention of %{build-root} in that .yaml file altogether12:05
* tlater will finally read up on stash next time he comes across this problem12:05
tristanNexus, its documented for no reason, absolutely pointless and unused12:05
tlaterta juergbi ;)12:05
Nexusfair enough12:05
juergbitlater: for the simplest case of quick branch switching it's just: git stash && git checkout foo && do something && git checkout - && git stash pop12:05
juergbi(assuming you don't stash while on the other checkout)12:06
tristanNexus, for install-root, comment it as I said, it is basically API, the element declaration author is allowed to set it and it will be used12:06
tristanNexus, and of course, update the commented value to reflect the new default; I think it's worthwhile to duplicate that value in that comment anyway12:07
gitlab-br-botbuildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31012:10
NexusCan someone please have a look at the fails i'm getting in those tests? I'm not sure what they mean, it seems to have something to do with cache keys?12:10
jmacLooking12:11
Nexusty12:11
jmacThat test is still running, I don't know how to get back to a previous result12:12
jmacAh, found it12:12
Nexuswait until the test is done, i fixed a lot of errors in the last one12:12
jmacIs it just saying you changed the cache key calculation, so you need to regenerate it?12:17
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28312:18
tristanNexus, yep that's expected12:18
tristanNexus, when changing %{build-root} and %{install-root}, it will effect cache keys of elements which format instructions based on those values12:19
juergbitristan: btw, as you looked at #10. do you think we should depend on host dpkg or directly access the ar archive (pyar)?12:19
tristangood question12:20
tristanjuergbi, how difficult of a dependency is pyar is a factor; pure python ?12:20
juergbiactually, i meant arpy12:21
juergbiyes, i think it's pure python, it's on pypi12:21
juergbihttps://pypi.python.org/pypi/arpy12:21
tristanjuergbi, another factor is; running host commands is pretty robust as far as child process control goes; is arpy going to be trouble in suspend/resume ?12:21
juergbii don't expect it to be more problematic than tarfile, which we also use12:22
juergbibut i have no experience with it12:22
tristanAnd another consideration is; is there *any* chance that host dpkg is going to extract anything based on host site assumptions ?12:22
juergbino idea12:22
tristanmaybe munge package metadata ?12:22
tristanyeah; I have no idea of these things either12:22
tlaterjennis: Really appreciate the branch, I'm looking forward to "M-x flycheck-list-errors" being useful :)12:28
tlaterThere's one final comment from me, would you mind taking a look again after, juergbi?12:28
juergbii will12:29
tlaterThis MR, for reference: https://gitlab.com/BuildStream/buildstream/merge_requests/28312:29
tristanwith all the activity today, I've made progress on cleaning up my project.refs branch... much less messy, still lacks some needed optionality; namely explicit track on junctioned elements12:29
tristanAnd of course, a crap-ton of test cases12:30
tristanif I'm lucky, it's all done over this weekend12:30
* tlater wonders how many bytes of test cases fit in a crap-ton12:31
*** tristan has quit IRC12:36
*** tristan has joined #buildstream12:39
tristanNexus, https://gitlab.com/BuildStream/buildstream/-/jobs/56407299 seems to be hung, you might try running the integration tests locally12:50
tristanoh wait, it didnt hang for linux ?12:50
tristanweird, that means... probably unrelated, but bad spurious error ?12:50
tlatertristan: I've seen this on occasion, I think it's a gitlab bug12:51
tlaterIt happens when docker fails to spin up on the host for some reason12:51
tlaterOh, it unhanged itself :)12:52
paulsherwoodis there a hello-world bst example anywhere?12:55
paulsherwoodie minimal set of bst definitions to start from creating a totally new thing?12:55
tlaterNexus^^^?12:57
paulsherwoodi can't find a single *example* bst file in the docs... how is a new user supposed to get started?13:01
paulsherwoodall the quotes seem to be parts of things, not a full example. maybe it's me misunderstanding13:03
tlaterpaulsherwood: There is this repo, which is still WIP, but we would like to have some example projects there soonish: https://gitlab.com/BuildStream/buildstream-examples13:04
tlaterAtm there is something like this in the docs somewhere13:04
tlaterBut I didn't pay close enough attention to know how to find it :(13:04
tlaterPerhaps some restructuring would be nice13:04
paulsherwood/src/netsurf-flatpak> bst build netsurf-flatpak.bst13:11
paulsherwood[--:--:--] START   Loading pipeline13:11
paulsherwood[00:00:00] FAILURE Loading pipeline13:11
paulsherwoodError loading pipeline: netsurf.bst [line 4 column 12]: Unexpected key: junction13:11
* paulsherwood creates an empty repo... creates an empty bst file ... tries bst build...13:22
paulsherwoodError loading project: Could not find file at /src/project.conf13:22
* paulsherwood touches project.conf13:22
paulsherwoodError loading project: /usr/local/lib/python3.6/site-packages/buildstream/data/projectconfig.yaml [line 6 column 0]: Dictionary did not contain expected key 'name'13:23
* paulsherwood finds that bst build foo fails to notice foo.bst13:25
tristanpaulsherwood, I just got back from obtaining food and going to eat; first errors is due to you not having an up-to-date buildstream, and whatever project netsurf-flatpak.bst comes from not requiring the minimal format version where junctions exist13:29
tristanThe latter part would have given you a more comprehensive error13:29
tristanA project.conf cannot not have a name, that is not allowed13:30
tristanthe error message being wrong however, really sucks13:30
paulsherwoodtristan: thanks. how do i update my buildstream?13:31
tristangit pull --rebase should be enough13:31
paulsherwood(am using bst-here)13:31
tristanI've never used that... hrmmm, tlater ?13:31
tristanssssam[m] ?13:31
paulsherwoodnext up: i have a bst file that has literally only "kind: manual" in it...13:32
tristanthat should do absolutely nothing I suppose, if not raise an error13:32
paulsherwoodtristan: guess how many lines of output it gives me? :)13:32
*** ChanServ sets mode: +o ironfoot13:33
paulsherwoodhttps://paste.baserock.org/xobanacepo13:33
paulsherwoodis bst making a whole load of linux assumptions?13:33
tristanlooking at bst-here, it should pull the latest build of the fedora based docker images13:34
paulsherwoodevery time bst-here is run?13:34
tlaterAh, sorry, was away for a sec13:35
tristanI guess, if I had any clue whatsoever about anything about docker, I might be able to tell you13:35
tristanhttps://gitlab.com/BuildStream/buildstream-docker-images/pipelines looks like it has not produced a fedora image in a while13:35
tristanSo perhaps these builds are not automated ?13:35
tlaterpaulsherwood: bst-here is partially designed to be used on macs13:36
tlaterI'm not aware of any linux assumptions13:36
paulsherwoodtlater: i know. that's why i'm using it. now how do i update its bst ?13:36
* skullman ran bst-here in his buildstream checkout and ran python3 setup.py install to get a recent version13:36
tristanpaulsherwood, ah, your manual element is default inheriting the strip-commands, which seems reasonable13:36
tlaterpaulsherwood: I think you need to manually update the docker image13:36
* tlater figures out the command13:36
tristanbut; we could do better to report an error when someone tries to run something in a runtime with nothing in it13:37
tlaterpaulsherwood: `sudo docker pull buildstream/buildstream-fedora` should do it13:37
paulsherwoodskullman: ack, i'll try that13:37
* tlater wonders if that script should have an 'update' command or something13:37
gitlab-br-botbuildstream: issue #289 ("Need better errors when running something on nothing") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/28913:40
tlatertristan: Should we perhaps make bst-here check for and update to new docker images, as well?13:40
tristantlater, it should have something that is self sustainable, or buildstream-docker-images should have a maintainer, or something :)13:40
paulsherwoodneeds better *behaviour* when run on nothing too... shouldn't need a project.conf13:40
tristanthat is a rando script hanging out in contrib/ to me13:41
gitlab-br-botbuildstream: issue #290 ("retry doesn't always work when a build error is found") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/29013:41
tlaterThe problem in this case isn't that the docker image is out-of-date afaik13:41
tlaterIt's just that `docker run` doesn't actually run the latest image13:41
tristanpaulsherwood, you absolutely need a project.conf with a name, that wont change; a better error message if a project.conf is missing maybe13:41
tlaterSo you need to manually ensure your docker has the latest image13:42
tristanproject.conf is the root of what defines your project, basically13:42
paulsherwoodtristan: why cant $name be whatever the target is that i specify?13:42
tristanmaaaaybe a default name but, that seems rather weird13:42
jmacThe target name you specify is an element name, not the project13:42
tristanthe target you are building, is not the project13:42
paulsherwood(if no project.conf exists?)13:42
* paulsherwood is a n00b on bst, doesn't yet understand what an element is13:43
tristanelement = target; bst is a pipeline of elements; each one can be a target13:43
paulsherwoodif the 'project' is in effect everything in the directory i'm in, then default to $dirname ?13:44
tristanartifacts and everything need to be namespaced13:44
tristanone pipeline can always have multiple projects, projects depend on eachother via junction elements13:44
jjardon[m]paulsherwood: pip3 uninstall buildstream; pip3 install --user buildstream_repo/13:44
tristanjjardon[m], bst-here script in contrib/13:44
tlaterjjardon[m]: That won't work inside the docker container ;)13:44
jmacI'd rather bst just show a very clear error message explaining how to create a minimal project.conf in that case13:45
jjardon[m]tlater I don't use docker containers13:45
* Nexus was afk, has anyone pointed paulsherwood to my beginner docs?13:45
jjardon[m]tristan I know nothing about that script :)13:45
tristanjjardon[m], the person you are recommending to use pip, is13:45
tristanjjardon[m], I know very little about it also, only that it's in use by people who want it to exist in contrib/13:46
jjardon[m]tristan I simply follow the docs to install and update buildstream; it works OK for me13:47
tristan(who are more numerous than paulsherwood, I know it's regularly used)13:47
dominic Nexus, I would like to see those if you have a link?13:47
tristanjjardon[m], this is not a meaningful conversation :)13:47
tristanjjardon[m], are you asking for help to upgrade your bst ?13:47
gitlab-br-botbuildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30513:47
*** ernestask has joined #buildstream13:47
jjardon[m]http://buildstream.gitlab.io/buildstream/install.html#installing13:48
Nexusdominic: https://gitlab.com/BuildStream/buildstream/issues/16913:48
jjardon[m]Tristan not me, paulsherwood have asked13:48
dominicNexus, thanks :)13:48
tristanjjardon[m], and we both told you that he is using bst-here, not pip, so this thread can easily just... not happen :)13:49
Nexusdominic: the links in there are all related to branches in that repo too13:49
Nexusnamed things like "install"13:49
jjardon[m]tristan yep sorry I missed that; maybe a good moment to document bst-here in the installation guide?13:51
gitlab-br-botbuildstream: merge request (sam/compose-symlinks-issue->master: Avoid compose element dropping files that are staged inside directory symlinks) #295 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/29513:53
gitlab-br-botbuildstream: issue #291 ("Updating buildstream is non-trivial with bst-here") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/29113:56
*** jonathanmaw has quit IRC13:58
tristanjjardon[m], <I'm really gonna eat my cooling off food this time/>... I suppose it could live in main bst docs; although the script itself is meant to be stand-alone and self-documenting; it's rather a "external thing we keep in contrib/" right now14:05
tristanjjardon[m], I wouldnt oppose a patch which documents it, though14:06
tristan(in the html)14:06
tristanoh look at that, it *is* in the docs, but a bit too obscure14:07
tristanhttp://buildstream.gitlab.io/buildstream/docker.html#docker14:08
tristanlinked from first section in http://buildstream.gitlab.io/buildstream/install.html#installing14:08
jjardon[m]oh nice, sorry for the noise then14:22
tristanno worry, it needs better documentation14:22
tristanand it needs to be self sustainable in general14:22
paulsherwoodNexus: no, where are your beginner docs?14:22
paulsherwoodjjardon[m]: i'm using the docker route, tlater's magic worked for me14:23
dominicpaulsherwood, <Nexus> dominic: https://gitlab.com/BuildStream/buildstream/issues/16914:23
Nexus^14:24
jjardon[m]paulsherwood: cool :)14:25
paulsherwoodreally? you kids today a actually think that documentation should be in the issue tracker?14:25
* paulsherwood is just jealous about others being young, but really14:25
jennispaulsherwood, there is also this https://wiki.gnome.org/Newcomers/BuildSystemComponent14:25
jennisNot sure if you've come across it14:25
paulsherwoodcool. can there be a link to that, so that when n00b lands at bst via google, it's obvious where to start?14:26
paulsherwoodactually, yes i've seen that... but it seems to be more about starting on gnome than starting with bst14:27
gitlab-br-botbuildstream: merge request (sam/compose-symlinks-issue->master: Avoid compose element dropping files that are staged inside directory symlinks) #295 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/29514:27
jennisIt was a good starting point to actually get `bst build foo` working, as you mentioned earlier14:27
paulsherwoodjennis: true, however... i'm expressly acting like a busy user who doesn't have much time (because actually i am) and i just want a painless 'get started' experience... i was hoping to be up-and-running in way less than an hour...14:29
paulsherwoodas much as i admire bst from afar, i really don't have time to chase down docs via the issue tracker for example14:30
jennishaha no I totally understand14:30
paulsherwoodso, back to this... who is actually documenting that 'getting started' experience, please?14:32
gitlab-br-botbuildstream: issue #292 ("bst doesn't use an open workspace for a junction") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/29214:32
tlaterpaulsherwood: The docs you just looked at are exactly that, actually. We use the gnome base for historical reasons, but it's not actually related to gnome.14:33
paulsherwoodthey're not that. they are *really* not that14:34
dominicyeah, I had trouble when just using those14:34
jennis+114:34
*** mcatanzaro has joined #buildstream14:35
jjardon[m]tlater: getting started should really be at https://buildstream.gitlab.io/buildstream/, not in a GNOME; thats something me and tristan wrote as a replacement for the previous jhbuild guide we had14:35
paulsherwoodi am a n00b. i don't know what(tf) bst is. tell me a) what it's for in a couple of sentencess. b) how to install it (via a link if it's complex) c) how to do 'hello-world' within 10 minutes, and without any words i don't understand d) where to dig deeper14:35
* tlater thought that was what the docs Nexus linked to contained, but will concede14:36
paulsherwoodbonuse points if you make the doc attractive and fun14:36
sstriker#292, workspacing a junction.  Hadn't thought of that one yet.14:38
gitlab-br-botbuildstream: issue #293 ("Write "getting started" guide") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/29314:39
* paulsherwood wonders what is a 'junction' is, and finds that there's no glossary for bst?14:40
jennisI think the information is there, but atm I think there is "somewhere"14:40
* tlater agrees with jennis - we have a lot of branches with almost-done documentation lying around14:40
paulsherwood:/14:40
jjardon[m]paulsherwood: hope you dont mind I stole your words to open that issue14:41
paulsherwoodjjardon[m]: no problem :-)14:42
sstrikerI thought there was more here https://buildstream.gitlab.io/buildstream/ at first.  Seems to have shed some content?  Hmm, where did the useful TOC go?  I can find what I need via google but not direct: https://buildstream.gitlab.io/buildstream/elements/junction.html14:45
juergbi'Authoring Projects' is where the information is for people who write/modify elements (.bst files)14:57
juergbi'Using BuildStream' => 'Invoking BuildStream' is for the CLI documentation14:58
juergbimaybe we should flatten 'Using BuildStream' as that just contains two links14:59
juergbiand maybe insert the word API in the link/title of 'Core documentation and reference'14:59
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28315:07
gitlab-br-botbuildstream: merge request (sam/compose-symlinks-issue->master: Avoid compose element dropping files that are staged inside directory symlinks) #295 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/29515:09
gitlab-br-botbuildstream: issue #282 ("Status bar regression after enhancing log configurability") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/28215:23
*** Prince781 has joined #buildstream15:24
gitlab-br-botbuildstream: merge request (bst-here-update->master: bst-here: Add '-u' flag to upgrade buildstream (Issue #291)) #311 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31115:27
gitlab-br-botbuildstream: issue #258 ("Configurable log line formatting") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/25815:29
jennisHas anyone already achieved a work-around to having "../<to_plugin>" in the project.conf15:40
jennis?15:40
paulsherwoodok, so in the presence of a project.conf file, what's the simplest single bst file that will actually pass for bst build?15:44
paulsherwood`kind: manual` hits the strip commands fail i mentioned earlier15:45
* paulsherwood is considering using bst to model a pipeline of mapped documents (eg some yaml structures with no real code underneath them)15:45
* paulsherwood feels like he may have scared the bst folks into silence16:03
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28316:03
jennisjuergbi, tlater has reviewed !283 and resolved the discussions. Would you be able to add this to your list? :)16:04
juergbiyes16:04
tlater\o/16:04
* tlater is *so* looking forward to clean emacs fringes16:05
juergbipaulsherwood: i'd recommend you to disable strip-commands in project.conf in that case16:05
juergbinot quite sure i understand your pipeline, though. if you want to process documents, the corresponding processing tools need to be either built or imported into buildstream16:06
ssssam[m]paulsherwood: you won't ever really have a project with a single .bst file16:06
juergbii assumed that would just be that start but maybe i misunderstood16:07
ssssam[m]probably the most minimal would be; a base.bst file that is an 'import' element that pulls in binaries of the tools you need; then a 'do-stuff.bst' that uses those tools to do something16:07
juergbiyou can have a single element that imports a document and doesn't do anything16:07
juergbiit's not particularly useful, of course ;)16:08
paulsherwoodcan the strip-commands be made to succeed on empty?16:08
ssssam[m]i'm not sure i'd call that a "build pipeline" :-)16:09
* paulsherwood is expecting to have multiple bst files... but is also trying to iron out this hello-world usecase on the way16:09
paulsherwoodssssam[m]: agreed. but it could be a "hello world" demo of bst?16:09
ssssam[m]the issue there is that if you want stripping but it's not working, you might not notice16:10
*** Prince781 has quit IRC16:10
ssssam[m]oh, you mean succeed if there's nothing to strip? that could make sense16:10
paulsherwoodyes, that's what i mean16:10
ssssam[m]Nexus: the docs at https://buildstream.gitlab.io/bst-external/ have a link to the 'quild' plugin, that seems wrong16:12
paulsherwoodso https://paste.baserock.org/ebisubazif gives a successful pipeline16:12
paulsherwoodhow would i get hello-world to appear in its (admittedly lovely) output?16:12
juergbihello-world is a file in the same repo?16:13
paulsherwoodnot yet it isn't... but if that's the best way, let's saay yes :)16:14
Nexusssssam[m]: did i add that?16:14
juergbihttps://paste.baserock.org/nenujabeca16:14
juergbipaulsherwood: if it's not in the same repo, you have to use a different 'kind' of source. e.g., git16:15
ssssam[m]Nexus: I thought you were working on a quilt source plugin, did someone else do that?16:15
Nexusssssam[m]: i worked on the quilt soruce plugin, but i never wrote docs for it that i can recall16:16
paulsherwoodjuergbi: nice try... but there's no sign of 'hello-world' in the output :-)16:16
Nexusapparently i did...16:16
paulsherwoodi appreciate this is a silly example. ideally causing 'hello-world' to appear woudl demonstrate the power of (say) script running on the pipeline16:17
ssssam[m]all new features should be documented, so it would be kinda bad if you didn't ...16:18
paulsherwoodeg like adding - configure-command: 'echo hello world' in a morph file16:18
juergbipaulsherwood: ah, i thought you wanted it in the built artifact, not in the log16:18
juergbipaulsherwood: to get echo working in the sandbox, you need to import an echo binary (or a shell with echo built-in) into the sandbox16:19
paulsherwoodaha... there's a built artifact? the log doesn't tell me that!!!! :)16:19
juergbithere is an open bug about that16:19
paulsherwood:)16:19
Nexusssssam[m]: hmm, let me have a look at what's going on16:19
juergbihttps://gitlab.com/BuildStream/buildstream/issues/25216:19
juergbipaulsherwood: you can get your artifact with: bst checkout yourelement.bst output-directory16:20
paulsherwoodeek... still seems to me that bst folks don't understand, then. cache-keys should be reported at the start, not at the end16:20
juergbithey are, where possible16:20
juergbiit's not always possible, though16:20
juergbisee linked issue description16:21
paulsherwoodhttps://paste.baserock.org/fevegodile - what's my artifact?16:21
juergbi      cached 2beb60c1 foo.bst16:22
juergbithat's the (abbreviated) cache key but you don't actually need it to checkout16:22
paulsherwoodwhich means precisely nothing to me as a drive-by user16:22
paulsherwoodsorry, i'm being harsh today16:22
juergbithe planned summary for #252 and #247 should address this16:22
paulsherwoodack16:22
juergbithat's for the summary after build16:23
juergbiwe're aware of the issue and it's already in our 'immediate priorities' milestone16:23
paulsherwoodcool16:23
*** Prince781 has joined #buildstream16:23
juergbibtw: i just noticed that you already have a local '.' source16:24
juergbithat's normally not recommended as this means you import the whole repository including project.conf and the .bst file16:24
juergbialways use a path, can be a subdirectory or a single file16:24
Nexusssssam[m]: that's been updated, just needs to be checked and merged16:25
paulsherwoodjuergbi: ack. i was just trying to get something to pass :)16:26
juergbisure, understood16:26
juergbibtw: if you want to import a basic system so you can have echo and co., you could use ssssam[m]'s reasonably small alpine base image: https://paste.baserock.org/uhilaroxut16:27
juergbiwith this alias in project.conf: https://paste.baserock.org/uwibowoyig16:28
paulsherwoodjuergbi: tvm16:28
paulsherwoodi'm still trying to understand the previous example...16:28
paulsherwoodso in theory my empty hello-world file would be in the artifact?16:28
juergbiyes, you can test with bst checkout16:28
paulsherwoodbut when i do bst checkout foo.bst bobbins -- my bobbins directory is empty16:28
juergbihm, it isn't here16:29
paulsherwoododd... do i have to checkin hello-world?16:29
juergbino, local doesn't know about git16:29
paulsherwoodjuergbi: ah, probably pebkac16:31
* paulsherwood is *nearly* there, but .... https://paste.baserock.org/zotixopuhu16:45
juergbipaulsherwood: project name with space that we don't support?16:46
juergbiwe should really fix that, either error out or properly support it16:46
paulsherwoodack :)16:47
jmacI hit exactly the same problem16:47
* paulsherwood has been failing to run software as its creators intend for some decades now :)16:47
juergbihehe16:49
paulsherwoodhttps://paste.baserock.org/unayogukis16:54
tlaterzo/16:55
tlaterErr16:55
tlaterThat wasn't supposed to be a 'z'16:55
juergbi:)16:55
tlaterjuergbi: Do you know what causes those ugly empty brackets btw?16:55
* tlater believes this to not have been an issue before16:56
juergbimight be a regression of the configurable log line16:56
juergbibrackets are part of the format and if the corresponding item is an empty string, we still see the brackets16:56
tlaterOh, right16:56
juergbinot sure how to best handle that16:57
jmacIt did come in with the configurable log line, yes16:57
jmacI chose to handle them by considering them beautiful instead of ugly16:57
* tlater isn't particularly happy with that solution, but not unhappy enough to immediately figure out how templating works here :|16:58
jmacIf it's really annoying then I will figure out a solution16:59
tlaterHeh, I was considering changing the format in my local config :)16:59
jmacYou can remove the brackets from the config, but they you won't get any brackets at all17:01
tlaterI like the brackets if they're around something though17:02
* tlater is hard to please17:02
jmacI think if I just add a render-with-brackets function to the Widget base class we can get the old behaviour back17:02
tlaterHm, weren't we using a templating language for this, as we are for `bst show`?17:03
tlaterIf it's the same language I think it already has a feature to allow this sort of stuff17:03
tlaterJust changing the default template might be better17:03
tlaterThen again, I might misremember17:04
jmac... you can't do that17:04
tlaterAww :(17:04
jmacThe log line has a format string which you can change but there's nothing that formatting string that's clever enough to support optional brackets17:05
jennisjuergbi, re the dir vs dir_, I can change this to directory as I'm reluctant to just disable the warning for this17:18
juergbiwe do use a _ suffix in other places, so maybe we should leave that in the branch as is17:18
juergbiit just feels odd having to accommodate for the pretty much useless dir() built-in17:19
juergbii wish we could disable the dir built-in instead17:19
juergbior globally ignore the warning for 'dir'17:19
juergbimostly a nitpick. i should probably not care that much about it17:19
tlaterWe can technically disable that, but I think it's generally bad - someone's going to spend 30 minutes debugging this only to remember dir() at some point17:20
juergbiyou mean disable the whole warning class for builtins? i think the warning generally makes sense17:21
juergbior could we disable it only for dir?17:21
tlaterWe can disable the warning specifically for dir, I *think*17:21
juergbiok, so what would be the downside?17:21
tlaterSomeone might use dir() for its built-in reason17:21
tlaterI.e., for a type check or something17:22
tlaterUnlikely to cause issues, but a massive pain if it does17:22
juergbihm, yes, if this is considered useful outside python interactive shell or similar, then we should probably not disable it17:23
juergbi"Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names"17:26
juergbii say it's worse to actually use the built-in dir() in regular code than to use dir as variable name17:26
juergbibut maybe that's just me17:26
juergbii want to invert the warning17:26
* tlater thinks that is also possible17:28
juergbibad_builtin plugin17:28
tlaterI think I agree here, actually, considering the doc17:28
juergbiok, let's do that, then17:28
gitlab-br-botbuildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30517:31
juergbiadded comment on gitlab17:32
gitlab-br-botbuildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30517:32
*** noisecell has quit IRC17:43
Nexusneed to add dpkg to CI :p17:43
juergbiNexus: you appear to have imported lots of binaries. accidentally?17:53
Nexusjuergbi: it's the contents of the .deb i'm using in the tests17:54
Nexusif you have a smaller one i can use, i can swap it out, but the contents need to be there afaik17:54
Nexusalso tristan, you were saying about the cache keys changing due to what i did with %{build-root}  and %{install-root}. Can i fix that in the tests?17:58
juergbimaybe reference a small one from snapshot.debian.org?17:58
juergbiwe shouldn't import binaries into git like that17:58
juergbian alternative may be to create a .deb as preparatory step in the test itself18:00
tristanNexus, yes; just do as the error message says and commit the result separately18:00
Nexusjuergbi: how does http://ftp.debian.org/debian/pool/main/c/clod/lua-clod_1.0.2-3_all.deb look?18:02
juergbisize is good but i think we should use snapshot.debian.org to reduce risk of breaking over time18:03
juergbiolder debs get removed from the main archive at some point18:03
Nexuscan you order by tiny?18:04
juergbihttp://snapshot.debian.org/archive/debian/20180308T040556Z/pool/main/c/clod/lua-clod_1.0.2-3_all.deb18:06
Nexusty18:06
*** dominic has quit IRC18:14
*** mcatanzaro has quit IRC18:41
*** valentind has joined #buildstream18:42
gitlab-br-botbuildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30518:52
Nexusjuergbi: replaced :)18:52
*** xjuan has joined #buildstream18:58
*** tristan has quit IRC19:34
gitlab-br-botbuildstream: merge request (richardmaw-271-workspaces-versioning->master: WIP: Add versioning to workspaces yaml configuration) #312 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31219:45
*** Prince781 has quit IRC20:08
*** ernestask has quit IRC20:11
*** xjuan_ has joined #buildstream21:23
*** xjuan has quit IRC21:25
*** Prince781 has joined #buildstream21:49
*** noisecell has joined #buildstream22:21
*** noisecell has quit IRC22:24
*** valentind has quit IRC22:52
*** xjuan_ has quit IRC22:52
*** Prince781 has quit IRC23:00
*** Prince781 has joined #buildstream23:05
*** Prince781 has quit IRC23:26
*** Prince781 has joined #buildstream23:31
*** mcatanzaro has joined #buildstream23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!