IRC logs for #buildstream for Wednesday, 2017-04-12

*** tristan has joined #buildstream06:37
*** ChanServ sets mode: +o tristan06:37
*** brejoc has joined #buildstream07:28
*** jonathanmaw has joined #buildstream08:22
*** ssam2 has joined #buildstream09:23
tristanchrispolin, So, did you ever get passed the hurdle of installing ?10:38
chrispolinI did indeed.10:38
tristanchrispolin, I'm trying to rewrite the user story for getting started and make virtualenv the "right way"10:38
tristanNice10:38
tristancuriously, how did you get passed the initial problems ?10:39
chrispolinIt was an issue with ruamel.yaml, as you suggested. Was getting that error that ironfoot and I posted when trying to install with pip.10:39
tristanHmmm10:40
tristanOk, curiously, did you do _any_ system wide installation with pip ?10:40
tristanDid you `sudo pip` at all ?10:40
ironfootI'm intereseted in knowing the solution to the problem too10:41
tristanits a god damned mess10:41
tristanpip and setuptools are just awful10:41
tristanironfoot, so what distro do you run and were you ever able to run buildstream without the docker crazy ?10:42
chrispolinI didn't 'sudo pip', but I did end up having to use apt-get to get over that initial issue. Which I realise contradicts what buildstream is for, but I wanted to get it built first and readdress issues afterwards.10:43
tristanchrispolin, I dont think it's contradictory to use your package manager to install buildstream's base dependencies no10:43
tristanbut what did you install with the package manager btw ?10:43
tristanruamel ?10:44
chrispolinruamel.yaml10:44
tristanI see10:44
tristanyeah that should be OK10:44
tristanbut it should not be required10:44
ironfoottristan: what do you mean. What distro I used to install buildstream?10:44
ironfoot(note, i always used docker)10:44
tristanchrispolin, what distro do you have and what version of python ?10:44
tristanironfoot, yes, what distro is running on the machine where you can run buildstream, and what python version is there ?10:45
ironfootbut distro used was debian stretch, (which is debian testing)10:45
ironfootpython 3.510:45
tristanironfoot, oh, then you should have no problem _at all_ to just run buildstream locally10:45
tristanWhat was the reason you reached for a docker in the first place ?10:45
chrispolinDebian 8.7, Python 3.410:45
ironfootlocally i use debian jessie10:45
chrispolin(jessie)10:46
tristanironfoot, Oh, OK so your docker image is that10:46
tristanstretch10:46
tristanI see10:46
tristanironfoot, and in stretch you have 3.4, as does chrispolin10:46
ironfootin jessie, i have 3.4, as chrispolin10:46
tristanironfoot, but at the time we required 3.5, and only recently I patched some things so that jonathanmaw could run on 3.410:46
tristanSo that's why10:47
tristanAlright, that explains that, but it also makes you an interesting candidate10:47
tristanto test something :)10:47
tristanironfoot, can you try a few commands for me ?10:47
ironfootsure10:47
tristanironfoot, A.) ensure that you have virtualenv, pygobject, ostree and bubblewrap installed with package manager10:48
tristanostree and bubblewrap are available through jessie-backports10:48
tristancd /path/to/buildstream10:48
tristanvirtualenv --system-site-packages -p python3.4 sandbox10:49
tristansource sandbox/bin/activate10:49
tristanpip install -e .10:49
tristan~~~~~~~~~~~~~10:49
tristanThats all10:49
tristanThat should install buildstream (and any dependencies it needs) entirely inside the virtualenv sandbox, without littering your system at all with pip10:50
ironfoothm.. is it python-gi the package for pygobject?10:50
tristanit's either python-gi10:51
tristanOr its pygobject310:51
tristanweird, I dont find it10:51
tristanon my stretch, python-gi title says it's for python2 :-S10:52
tristanironfoot, python3-gi10:52
tristanthat's what it is, python3-gi10:52
ironfoottrue, py310:52
tristanSo what I'm thinking now, is I want to be able to remove --system-site-packages from that user story10:54
tristanBut for that, I need to be able to get pygobject via pip10:55
tristaninto the virtual env :-S10:55
tristanBut it should work without that... the --system-site-packages only makes it fragile, in case the user has already garbaged their system with `sudo pip` in the past10:55
ironfoottristan: http://paste.baserock.org/upumuqokep10:56
ironfooti may need to install some ostree python library first, maybe10:56
tristanyes10:56
tristangir1.2-ostree-1.010:57
tristanShould be it10:57
tristanlemme check what it is here10:57
tristanthat's exactly what it is here10:57
tristanironfoot, sorry I did not mention that; the important part is to not only have the library, but also have the introspection data for PyGObject to access the library10:58
tristanthat is the "gir"10:58
ironfootinstalled10:58
tristanSo from there you can run bst right ?10:58
ironfootyep10:58
tristanOk, good to know10:58
tristanNow, what should we do10:59
ironfootxD10:59
tristanI also have a trick for generating man pages10:59
tristanIf you install in that same venv: pip install click-man11:00
tristan(or click_man)11:00
ironfoot`which bst` spits pwd/sandbox/bin/bst, which is good11:00
tristanThen you can do: python3 setup.py --command-packages=click_man.commands man_pages11:00
ironfoothttp://paste.baserock.org/ivizafuxeb11:01
tristanseriously ?!11:02
tristancrap11:02
tristanWell, man pages aside11:03
tristanI have an idea that I may (ugh, I hate this), just commit generated man pages11:04
tristanbecause with that, I can tell setup.py to distribute them as data_files11:04
tristanand they will "just work" the moment this is actually installed on a real system11:04
tristanBut for the time being...11:04
tristanironfoot, I think maybe a helper script for running bst in a venv would be helpful ?11:05
tristanOr... just some instructions on using venv ?11:05
ironfoottox helps with the creation of venvs, not sure if we can use it nicely here11:06
tristanI was thinking, if I wrap it up with our own script11:06
ironfootis more tests oriented11:06
tristanThen I can additionally set MANPATH in it11:06
tristanand when you enter the env you also have the man pages11:06
tristanThe helper script could create a venv only if it doesnt exist, and activate it11:07
* albfan[m] sent a long message: albfan[m]_2017-04-12_12:11:35.txt - https://matrix.org/_matrix/media/v1/download/matrix.org/hyETClapyQWdlXrokFLJUgzx12:11
tristanalbfan[m], the thing is, with flatpak-builder you are leveraging build metadata that exists12:14
tristani.e. org.gnome.Todo.Test.json12:15
tristanalbfan[m], it's planned that we create a conversion script for those which should work pretty much seamlessly, but there are a couple blockers to that12:15
tristani.e. jonathanmaw is working on the Source implementation that would be needed12:15
albfan[m]tristan: Let me know If I can go on that blockers or test anything12:16
tristanright now, without that, you would need to A.) Build everything from git (no tarballs or hg/bzr Sources) and B.) Manually translate the build instructions12:17
tristanSimilar to what I did for the gedit demo12:17
tristanwhich pulls in the flatpak runtime/sdk required, and builds some stuff12:17
tristanNow lets assume that there is a conversion script; to run tests there would be a manual or an automated approach12:18
jonathanmawtristan: speaking of which, I finally have a merge request for you https://gitlab.com/BuildStream/buildstream/merge_requests/1812:18
tristanmanual approach would be `bst shell --deps runtime gnome-todo.bst`, and then run the tests12:18
tristanautomated approach would be to create an element which includes instructions for running the tests, and then `bst build gnome-todo-tests.bst`12:19
tristanjonathanmaw, yay !12:19
tristanAt least your day was productive, while mine was lost in a mine field of pip and virtual env :'(12:19
jonathanmawtristan: one thing that surprised me while testing my tar implementation is that having "track: false" in the yaml doesn't work for me, I guess because it was treating it as a string, not a boolean.12:20
tristanjonathanmaw, I'm curious how your tarball plugin fares with suspend/resume/terminate btw, have you tried that ?12:20
jonathanmawtristan: I have not12:20
jonathanmawI might have to fetch a pretty big tarball to get the opportunity.12:21
tristanThere is a bug with ostree that I should file, which is a bit awkward and corner case; seems to suspend/resume alright, but when terminating a long standing ostree fetch; other processes (like ongoing git clones) become orphaned12:21
jonathanmawtristan: I assume if I wanted track to be a true/false, then I ought to pass a different type into node_get_member.12:22
tristanjonathanmaw, Oh12:23
albfan[m]tristan: got it12:23
tristanjonathanmaw, what would track true/false do ?12:23
jonathanmawtristan: I have no use for setting "track" to anything specific, because there's no specific tracking ref to fetch and set the ref to. So it might as well be a binary true/false.12:23
tristanjonathanmaw, sorry I missed that part yesterday; I think that 'track' as a configuration option of the tar Source was not needed12:23
tristanand we would just not have it... just that the _behaviour_ of tracking a tarball would be to get the sha256sum12:24
tristanautomatically12:24
tristanjonathanmaw, what would it mean if track were false ?12:24
tristandisable the feature ?12:24
jonathanmawtristan: yep. but it looks nicer for it to be set to false, than to be commented-out12:25
tristanjonathanmaw, what about if it's just not there at all ?12:25
tristaninstead of commented out :)12:25
jonathanmawtristan: true, it's fair to expect that they're working in git, so don't lose anything by deleting the track line12:26
tristanjonathanmaw, I have some ideas how we *could try* to implement a more meaningful track experience in the future (i.e. most tarball release servers have some structure, could be that track could eventually point to a subdir on an ftp server, and we resolve to the latest available tarball of a given release directory)12:27
tristanbut I think its a bit crack12:27
tristananyway probably not something to spend time worrying about now12:27
jonathanmawok, I should probably update the comments pointing out that _any_ value for track is treated as true.12:28
tristanjonathanmaw, rather I think we should just remove it entirely for the tar plugin for now12:28
tristanIt could become a feature in the future, for right now, the behavior of tracking a tarball is just to automatically update the sha256sum12:29
jonathanmawtristan: ok, so "bst track foo" should happen unconditionally, for the time being?12:30
tristanYes, if the user asks to track an element with tarball sources, the way a tarball source handles that is to update the sha256 sum12:30
tristanbetter to not provide additional API unless it's meaningful12:31
tristanjonathanmaw, note that tracking is not something that happens unconditionally12:32
tristanit's something the user asks for12:32
tristanAnd it doesnt really fit with tarballs the way it does with VCSs, only that tarballs are a thing, and it seems to me that automatically updating the sha256sum is a useful thing to do :)12:33
jonathanmawtristan: up until someone tries to update all refs by running 'bst track' on every element, but with many disparate source types, that's a risky thing to do, anyway.12:33
tristanNot entirely I think12:34
tristanWell, I would assume that you generally revision your BuildStream project12:34
tristanSo it's really up to you what you commit as a result of tracking12:34
tristanOf course, it's true that it's undesirable to repeatedly update the sha256sum over time12:35
jonathanmawespecially because you have to fetch the entire tar each time to do it :)12:35
tristanand if it ever changes for a tarball, it may be an indication of some evil12:35
tristanjonathanmaw, I think you don't, though12:36
tristanI mean, it will mirror unconditionally, for either track or fetch12:37
tristanOnce it's been mirrored, it's there, and track will only calculate the checksum of the downloaded tarball12:37
tristanRight ?12:37
jonathanmawaha, sloppy wordage. I was using "fetch" to mean all kinds of downloading.12:37
tristanYes I think I understood, but in any case... `bst track foo.bst` twice... will only download foo.bst's tarballs once12:38
tristanThis is correct right ?12:38
tristanI think so12:39
tristanlooking at _ensure_mirror()12:39
jonathanmawtristan: I don't think there's any way of knowing whether the sha256sum of the file has changed between downloads without actually downloading it.12:39
tristan:-/12:39
tristanI dont think that is the desire, though12:40
tristanright now it looks like it works well enough12:40
tristanjonathanmaw, there is a strange case in which a remote tarball being served has changed, or a mitm attack occurs12:40
tristanin which case an sha256sum changes12:40
tristanjonathanmaw, it's pretty standard practice to hold on to tarball shas12:41
tristanexactly to prevent against this kind of mitm12:41
tristanor ensure you've got the right thing12:41
tristanSo, track is something you do, because you are either creating a new element, or you are updating to a new tarball12:41
tristanAfter which point you've committed your sha12:42
tristanThe next person who fetches, should get an error if the sha is a mismatch12:42
tristan(i.e. they never mirrored that tarball before)12:42
tristanSo far so good12:43
tristanjonathanmaw, that said, it *may* be of interest to add a warning if track() is called, and self.ref is not None, and the resulting ref is *different* than what was already there ?12:44
tristanThat is A.) It could be the user changed the tarball URI (new tarball is intended anyway, so it's not a problem) .. or B.) User was refreshing all branches in a larger project, and for _some reason_ they have obtained a different checksum this time around ?12:45
tristanWhere (B) may be an attack ?12:45
tristanI dont know, I think it's overkill12:45
tristanto consider this right now: tracking a tarball is only a convenience for updating the for you, so that you dont have to12:46
tristansomething jjardon[m] will be happy about when crawling through huge moduleset files and updating to latest release tarballs12:46
jonathanmawtristan: I think it's something we should look at again if we ever expect track to be used automatically12:47
tristanSure12:48
tristanI do expect it to be used automatically, by surrounding scriptage12:48
jonathanmawand if instructions ever appear to run `bst track` on every tar source, just before building, they need to be chastised for blatantly ignoring security.12:49
tristanFor instance: A CI server does `bst track --deps all target.bst && bst build target.bst`12:49
tristanAnd if the build passes, it submits a candidate branch for review12:49
tristanI.e. that's the regular case: Get me the latest of everything on their tracking branches and try it12:50
tristanif it's good, lets commit12:50
tristanit's true, tarballs are a bit different in that regard12:50
jonathanmawtristan: review gives a chance to check whether the changes to sha256sums are trustworthy, at least.12:50
tristanat least yeah, and thats why I wonder if a warning can be good12:50
tristani.e.: If you change the tarball url and delete the ref, you dont get a warning12:51
jonathanmawhmm, AIUI, an element can have multiple sources, and those sources can be of varying types. Will we have to worry about different sources treating track slightly differently?12:51
tristanIf you change the tarball url and _dont_ delete the ref, track produces a warning saying this sha256sum you already had may have changed12:51
jonathanmawI suppose it makes sense to make these changes now, rather than forget about them when they're needed later :)12:52
tristanjonathanmaw, I'm not really worried about sources treating that differently, although we can never be certain for sources that are not first class citizens12:52
tristanI.e. projects are free to include their own Source implementations in a subdir, and those can go an hack the fbi database and automatically send out personal information about the president in their track() implementations for all I care :)12:53
tristanhttps://gitlab.com/jonathanmaw/buildstream/commit/1653dac7c9baf591dfbcc9776c0f3917880f9e41#f352dc2162c6fa9a9e75c276dd143ab3270b9063_0_10112:54
tristanjonathanmaw, I dont really like this line ^^^^^^12:54
tristantry: ... raise SourceError() ... except Exception as e: ??12:55
tristanlooks like that surrounding try/except is just noise12:55
tristanjonathanmaw, rather, _ensure_mirror() should be normalizing the errors it produces at the source of the error12:56
tristanjonathanmaw, would you rather I comment this in the MR than chatting about it here ?12:56
jonathanmawtristan: on the MR makes it a bit more transparent, I think12:56
tristanSure12:56
jonathanmawtristan: is preflight the right place to check whether ref has been set? I don't know whether `bst track` sidesteps that12:58
jonathanmawI suppose it can't, because preflight is used to check that necessary binaries exist.12:58
tristanAdded some comments13:03
tristanjonathanmaw, I suppose that could be better documented for plugin authors indeed13:04
tristanjonathanmaw, What's going to happen, is I believe fetch will never be called if the Source is INCONSISTENT13:04
tristanif it's INCONSISTENT, then it's considered that track() is required, the user will be told they must track at this point13:05
tristanif its RESOLVED but not CACHED, then fetch() is needed13:05
jjardon[m]someone mentioned me here :) currently we use scripts that grab the tarballs for the locations specify in jhbuild + a control file (https://git.gnome.org/browse/releng/tree/tools/smoketesting/tarball-conversion.config) to then generate new modulesets using tarballs instead. Script here: https://git.gnome.org/browse/releng/tree/tools/smoketesting/convert-to-tarballs.py13:05
jjardon[m]tristan: jonathanmaw in case Its useful ^13:06
tristanjjardon[m], yeah I was just talking about how it can be a convenience for someone who has to update a bunch of tarballs, that we can automatically update the sha256 sums13:06
tristanjjardon[m], so maybe you have some thoughts on the topic13:07
tristanjjardon[m], The question we're asking... is: When you have a big project and you want to automatically "track" the latest of everything13:07
tristanjjardon[m], and some of your sources are tarballs13:08
jjardon[m]the current script do exactly that: they generate new modulesets with reference to the tarballs and the sha256: https://www.mirrorservice.org/sites/ftp.gnome.org/pub/GNOME/teams/releng/3.24.0/gnome-suites-core-3.24.0.modules13:08
jjardon[m]ah13:08
tristanjjardon[m], now there is a weird, rare chance, that your tarball already had an sha256sum specified... but `bst track` found a DIFFERENT sha256sum13:08
tristanthat can be because the ftp server is fubar13:09
tristanor corrupted13:09
tristanOr because there was some mitm attack13:09
jjardon[m]in GNOME we only use tarballs for releases; that can change in the future13:09
jjardon[m]some prople is pushing to simply use git tags instead13:09
jjardon[m]not sure if this reply to your questions?13:10
tristanjjardon[m], Right now, we prefer the convenience that `bst track` will update an sha256sum of a tarball in the project inline, but we dont really know if A.) The user just change the tarball URL to a new release, and WANTS a new sha256sum ... or B.) The user wanted to track/update everything else, and a new sha256sum was *inadvertently* added13:10
tristanjjardon[m], it does not, you wont be able to reply unless you read everything I wrote and understand it :D13:11
tristanjjardon[m], So, we are considering; perhaps BuildStream should issue a warning, in the case that an sha256sum was *already specified* and does *not match* the introspected checksum13:12
tristanI.e. something bad *could* have happened13:12
tristanjjardon[m], see what I'm getting at now ?13:12
jjardon[m]yep, Id error in that case; but let me read all first13:12
tristanIf the user wants to update to new releases, they go and change a bunch of urls, and run `bst track` to update the checksums... if they did not *remove* the previous checksums, is it an error ? or just a warning ?13:13
tristanWith an error, it would mean that you have to also delete the previous checksums whenever you update tarball uris13:14
tristanWith a warning, you would just be warned that an existing sha256sum changed13:15
jjardon[m]maybe Im misubdaerstanding some buildstreamsemantics here, but can "track" somehow means "track the latest tarballs in this folder", so when you run the tool, the sha256 gets updated to the latest tarball available in that repo?13:17
tristanjjardon[m], We discussed that earlier in the backlog, I think it would be an interesting feature but also I think it's a bit fragile and 'crack'13:19
tristanat the moment13:19
tristanjjardon[m], I'm not averse to adding something crazy like that in the future, but an ftp server really is not a VCS13:19
jjardon[m]yup, only wanted to explain how our current tools work atm13:20
tristanjjardon[m], it might for example, involve some kind of pattern matching semantics which might need to go into the configuration13:20
tristanGNOME's tarball server works one way, it may not match exactly with how sourceforge's tarballs are served, etc13:21
tristanjjardon[m], in your case I guess, you would want the 'uri' to be a base ftp.gnome.org/pub/somemodule13:22
tristanAnd then 'track' would be a release subdir, and finally the 'ref' would be a tuple of release tarball itself and sha256sum13:23
tristanjjardon[m], anyway yeah, it's feasible, but fragile... and I'd rather not spend time on that right now13:23
jjardon[m]tristan: yeah, those heuristics are in https://git.gnome.org/browse/releng/tree/tools/smoketesting/convert-to-tarballs.py (some tarballs are not in gnome repos as you pointed out)13:24
tristanyeah, heuristics is exactly the word for something not exactly reliable :)13:25
jjardon[m]:) (for example, we have had problems before with tarballs named 1.a and 1.b and the script crashes because it didn't know whats bigger)13:26
tristanI also had the thought that with ftp protocol you can probably get the datestamp before download13:27
tristanAlso GNOME servers offer a feature of providing a LATEST symlink13:27
tristanthey are all sorta different (tarball hostings)13:27
tristanjonathanmaw, anyway yeah, ultimately what we are discussing would be a full blown, more accurate interpretation of what "track" means; but lets not go down this crazy path "just yet"13:29
tristan:D13:29
tristanbrejoc, I wanted to find time to chat with you, and thanks for the comments on the blog, it sounds very interesting... but alas for today I have run out of time13:30
tristanironfoot, maybe you, as the official "docker dude", may have some more concrete ideas regarding the last comments on: https://blogs.gnome.org/tvb/2017/02/06/introducing-buildstream-2/13:31
tristanironfoot, sounds interesting to me, a deployment element to generate a docker image ?13:31
tristanAnd maybe coupled with some Source plugins which could implement our Source sementics for "importing a debian runtime" or such13:32
tristanWould allow to take someone's already build deterministic sysroot, build some things on top of it for a web hosting, and finally deploy it as a cloud ready docker image ?13:33
tristansounds pretty nice actually :)13:33
ssam2like Nix and GUIX but without the totalitarian approach13:36
tristanpizza time13:37
*** tristan has quit IRC13:40
*** brejoc has quit IRC13:40
*** tristan has joined #buildstream14:04
*** ChanServ sets mode: +o tristan14:05
jonathanmawtristan: it looks like I'm having some trouble when converting an urllib.URLError into a SourceError using "raise foo from bar"15:01
jonathanmawI get "TypeError: __init__() missing 1 required positional argument: 'message'"15:02
jonathanmaw"message" appears to be an argument passed into the SourceError's constructor.15:03
*** ssam2 has quit IRC15:08
jonathanmawI think It worked it out. raise foo from bar still needs foo to be constructed with args like normal.15:16
jonathanmaws/It/I/15:16
*** ssam2 has joined #buildstream15:25
jonathanmawtristan: merge request updated https://gitlab.com/BuildStream/buildstream/merge_requests/1815:55
jonathanmawI'm going to disappear for today, as I have to disappear to prepare for some nerd-wrangling (roleplaying games)15:55
*** jonathanmaw has quit IRC15:57
tristanright... 'from' was added in python3 and allows reraising new errors whilst preserving the originals they came from (but I see I got here late) :)16:00
chrispolinRunning into this issue, appears to be a problem involving gcc. http://paste.baserock.org/oyokinejev16:11
chrispolinI have gcc 4.9.2, so I don't think it's to do with the problem here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=6995916:13
ssam2it looks like something fails when buildstream tries to unpack the artifact, at a glance16:15
ssam2I don't think it's to do with gcc at all16:15
*** brejoc has joined #buildstream16:41
*** xjuan has joined #buildstream16:41
*** ssam2_ has joined #buildstream17:02
*** ssam2 has quit IRC17:04
*** ssam2 has joined #buildstream17:10
*** ssam2_ has quit IRC17:12
*** ssam2 has quit IRC17:16
*** tiagogomes has quit IRC17:16
*** chrispolin has quit IRC17:16
*** chrispolin has joined #buildstream17:17
*** tiagogomes has joined #buildstream17:17
*** tiagogomes_ has joined #buildstream17:22
*** ssam2 has joined #buildstream17:22
*** tiagogomes has quit IRC17:24
*** tiagogomes_ has quit IRC17:25
*** chrispolin has quit IRC17:25
*** ssam2 has quit IRC17:25
tristanwhats with the gaps ?17:26
tristanis weird JS pastebin to blame for that ?17:26
*** brejoc has quit IRC17:29
*** chrispolin has joined #buildstream17:31
*** tiagogomes_ has joined #buildstream17:31
tristanchrispolin, your paste seems to have messed up, there are weird gaps, maybe weird JS pastebin is to blame17:32
tristanthat or, the output really contains gaps ?17:32
tristanchrispolin, it looks however like you hit the race and we're casing the wrong error17:34
tristanchrispolin, maybe this line should be EEXISTS ?17:34
tristanhttps://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_artifactcache.py#L11417:35
* tristan thinks that's what's happening, and if you relaunch the build, you wont get that error again17:35
*** Chris has joined #buildstream18:28
*** chrispolin has quit IRC18:30
*** brejoc has joined #buildstream20:02
*** xjuan has quit IRC21:31
*** brejoc has quit IRC21:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!