| paulsher1ood | i've just raised https://gitlab.com/BuildStream/buildstream/issues/11 | 06:34 |
|---|---|---|
| paulsher1ood | is there anything written anywhere to tell me how to install buildstream on a new machine? | 06:35 |
| *** franred has joined #buildstream | 06:38 | |
| paulsher1ood | hmmm... 'ostree is already the newest version (2016.15-4).' | 06:52 |
| *** brejoc has joined #buildstream | 07:10 | |
| Chris | paulsher1ood, the bst command is in ~/.local/bin/ | 07:39 |
| Chris | The blog post you posted in #baserock (https://blogs.gnome.org/tvb/2017/04/10/buildstream-progress-and-booting-images/) was the best walkthrough I could find, but it's not comprehensive. | 07:41 |
| *** tristan has quit IRC | 08:11 | |
| *** brejoc has quit IRC | 08:12 | |
| Chris | Can confirm that it does build. | 08:23 |
| *** ssam2 has joined #buildstream | 08:23 | |
| *** jonathanmaw has joined #buildstream | 08:24 | |
| *** tiagogomes has joined #buildstream | 08:24 | |
| * paulsher1ood can confirm that the instructions *don't* work for him | 08:26 | |
| paulsher1ood | software rarely works for me | 08:26 |
| paulsher1ood | 'git clone git@gitlab.com:BuildStream/buildstream.git' is a fail, for example | 08:26 |
| Chris | I think gitlab is down, currently. | 08:27 |
| paulsher1ood | no, it's not that. Permission denied (publickey). | 08:27 |
| Chris | Ah. I had that, but it was because I hadn't added my public key to my gitlab profile yet. | 08:28 |
| tiagogomes | you are trying to clone over ssh | 08:28 |
| * paulsher1ood is following 'instructions' that don't work :) | 08:28 | |
| tiagogomes | They should, for the ones that set up a SSH key :) | 08:29 |
| paulsher1ood | tiagogomes: why should that be a dependency? and if it is, where is that stated? | 08:30 |
| * tiagogomes apologizes for his bad sensor of humour | 08:31 | |
| tiagogomes | Obviously the clone URL shouldn't use the ssh format | 08:32 |
| paulsher1ood | tiagogomes: no need to apologise, i appreciate your humour | 08:33 |
| paulsher1ood | but i'm interested in getting to instructions that work, with minimum fuss. | 08:33 |
| paulsher1ood | 'pip install --user -e .' - command not found | 08:33 |
| paulsher1ood | nothing has explained to me how to get a python3 default, with pip3 as default | 08:34 |
| paulsher1ood | these things are probably obvious for an experienced/active gnome person... | 08:35 |
| ssam2 | i think that's distro specific .. the instructions should probably say `pip3 install` | 08:35 |
| paulsher1ood | ssam2: is there no way we could get to an equivalent of ybd's install_dependencies.sh, which would 'just do it'? | 08:36 |
| paulsher1ood | s/equivalent/improvement on/ | 08:36 |
| ssam2 | probably | 08:37 |
| ssam2 | (was trying to look at that file to comment, but got a 500 error from gitlab) | 08:37 |
| paulsher1ood | which file? | 08:37 |
| ssam2 | install_dependencies.sh | 08:37 |
| ssam2 | https://github.com/devcurmudgeon/ybd/blob/master/install_dependencies.sh | 08:38 |
| Chris | wfm, try it again. | 08:38 |
| paulsher1ood | new question - why is buildstream LGPL, rather than GPL? | 08:38 |
| ssam2 | looking at that script I'm sure it could be modified to work for buildstream | 08:39 |
| ssam2 | that one I guess you'll have to ask tristan | 08:39 |
| paulsher1ood | would it be worth having an equivalent install script, or is it 'just me'? :) | 08:39 |
| ssam2 | well, the ideal would be that distros actually package buildstream | 08:40 |
| ssam2 | and if it becomes widely used then that will happen | 08:40 |
| paulsher1ood | right... but in the meantime, with neither egg nor chicken? :) | 08:40 |
| ssam2 | in the meantime, I don't know how many people would use an equivalent of install_dependencies.sh :-) do you know how many folk used it for ybd ? | 08:41 |
| paulsher1ood | ssam2: everyone, afaik | 08:41 |
| ssam2 | then it's probably useful for buildstream too at the moment | 08:42 |
| paulsher1ood | sorry, everyone except rjek, who refuses to use pip as root under any circumstances | 08:42 |
| paulsher1ood | ok, so after supposedly 'Successfully installed BuildStream' using setup.py ... | 08:46 |
| paulsher1ood | -bash: bst: command not found | 08:46 |
| Chris | ~/.local/bin/bst | 08:47 |
| paulsher1ood | Chris: but *why*? | 08:47 |
| Chris | Oh I agree, it shouldn't be there, or made clear in the instructions. | 08:48 |
| Chris | s/or made/or should be made/ | 08:48 |
| ssam2 | some linux distros put ~/.local/bin in your PATH | 08:48 |
| ssam2 | but others don't | 08:48 |
| * paulsher1ood notes that he tried this on debian, because that's what .gitlab-ci.yml appears to use | 08:52 | |
| * paulsher1ood has bst running at last :) | 08:54 | |
| Chris | gitlab back upm | 08:59 |
| Chris | s/m// | 08:59 |
| *** tristan has joined #buildstream | 09:08 | |
| *** ChanServ sets mode: +o tristan | 09:09 | |
| *** jude has joined #buildstream | 09:12 | |
| *** tristan has quit IRC | 09:21 | |
| *** tristan has joined #buildstream | 09:23 | |
| *** ChanServ sets mode: +o tristan | 09:23 | |
| tristan | okaaaaay | 09:25 |
| tristan | good morning | 09:25 |
| * tristan starts by going through the logger's backlog | 09:25 | |
| tristan | A.) install_dependencies.sh thing should be taken care of entirely by having a standard setup.py with pip if I understand correctly | 09:26 |
| tristan | i.e. the only thing install_dependencies.sh dealt with was distro agnostic python packages | 09:26 |
| tristan | B.) Should have some stock instructions for installing the ostree + ostree gir data + bubblewrap for known distros, but this will always be distro specific | 09:27 |
| tristan | C.) I raised the license thing months ago, nobody replied; I prefer LGPL, even if rms hates me (note flatpak-builder is also LGPL) | 09:28 |
| tristan | As someone who lived through maintaining Glade and tried to relicense it LGPL, I can say that it was sad to not have a lot of support from, say Access company (previously palmsource), when they were considering using Glade as a part of their official SDK tooling | 09:29 |
| tristan | all because of restrictive GPL-ness | 09:30 |
| tristan | After struggling for a day with pip and trying to reproduce the clean initial install, but having some cruft installed in my system dirs, I seriously discourage anyone from ever running pip under sudo | 09:31 |
| tristan | and start to agree more with rjek on this | 09:31 |
| tristan | people in #pypa may also agree | 09:31 |
| paulsher1ood | 'standard setup.py' - can't work out-of-the-box, afaict | 09:32 |
| paulsher1ood | you're presuming the box already includes specific version of pip and python | 09:32 |
| tristan | best in the horrible world of python packaging, is virtualenv; best possible future is "apt-get install buildstream", but wont happen for a while :) | 09:32 |
| tristan | Yes I am assuming that, you need to install pip from your distro | 09:33 |
| paulsher1ood | pip3, python3, git... | 09:33 |
| tristan | git is a bit of a special case, it's only projects which use git which require git | 09:34 |
| tristan | the user story for this should be that, during preflight, buildstream tells you with an error that git is not installed | 09:34 |
| tristan | (before any long time processing happens) | 09:35 |
| * paulsher1ood just wants an out-of-the-box experience that doesn't require messing around for a couple of hours | 09:35 | |
| tristan | Yes, I was looking into that story last week and trying to find the best options | 09:35 |
| tristan | Because I want to include a "Installing and Running" section in the full documentation, and I dont want Docker to be the "preferred" only story there | 09:36 |
| tristan | My hope was that everything could be done inside virtualenv, but unfortunately you need virtualenv --enable-system-site-packages (or whatever that switch was) | 09:37 |
| tristan | because you cant install the pygobject parts with pip into a virtualenv, still, I'll include that story | 09:37 |
| tristan | For a user, the best is `pip install --user .` | 09:38 |
| tristan | which can later be uninstalled with `pip uninstall buildstream`, but will have the side effect of user-installing the python deps | 09:38 |
| tristan | For a developer, adding `-e` is better, but you dont get any man pages that way | 09:39 |
| tristan | paulsher1ood, I'll try to add that section to the docs today, and change the repo's README.rst a bit to link out to things | 09:40 |
| * tristan originally didnt want to have links in the README.rst because then he is stuck with more things to maintain whenever resources change | 09:40 | |
| tristan | other than that my priorities this week are community outreach and submitting talk for guadec | 09:41 |
| paulsher1ood | afaict i need to be root to install things for buildstream in any case? | 09:46 |
| paulsher1ood | eg 'echo "deb http://ftp.uk.debian.org/debian/ testing main" >> /etc/apt/sources.list' doesn't work unless i'm root | 09:47 |
| tristan | paulsher1ood, you need to be root to install the system requirements yes, including ostree and bubblewrap and pip itself | 09:47 |
| paulsher1ood | right. so setup.py can never address these things | 09:48 |
| paulsher1ood | hence in the absence of a 'packaged buildstream' i think think we *do* need something like (or better than) install_dependencies.sh | 09:49 |
| tristan | paulsher1ood, the reason I strongly discourage using `sudo pip` is that having pip write to system folders is an invite for disaster, I had to manually delete things ("eggs" specifically) from my /usr/local/lib/python3.5 | 09:49 |
| tristan | in order to reproduce the installation, otherwise it was borked | 09:49 |
| paulsher1ood | i'm fine with that. my problems here are way earlier than that :) | 09:50 |
| tristan | paulsher1ood, instead, a copy/pastable per-distro line like: `sudo apt-get install foo, bar, baz` in the getting started docs I think should be specific | 09:50 |
| tristan | err s/specific/sufficient | 09:50 |
| tristan | I'm not fond of the idea of having a script which tries to determine what distro you have and launches the distro package manager for you | 09:51 |
| paulsher1ood | tristan: having a script is better imo, since it can be run by (say) a ci program | 09:51 |
| tristan | A ci program can know what distro it's running on | 09:51 |
| tristan | or conversely, it, itself can have it's own story of what it requires from the OS | 09:52 |
| tristan | I dont want to hear people saying "Your install_dependencies.sh does not work on arch/gentoo, this is a bug", that's a door better left closed | 09:53 |
| paulsher1ood | tristan: the current situation is worse than that, imo | 09:53 |
| tristan | paulsher1ood, let me add the docs I've been meaning to add, then | 09:54 |
| paulsher1ood | sure | 09:54 |
| tristan | jonathanmaw, I've been wanting to ask you about some things btw... | 10:00 |
| jonathanmaw | tristan: go ahead | 10:01 |
| tristan | jonathanmaw, since you are looking into some of the missing Sources, there are some things I find unsatisfactory about it's API | 10:01 |
| tristan | and wanted you to think a little on that | 10:01 |
| tristan | Some of that includes... right now we have get_unique_key(), and we have get_consistency(), which is a bit confusing... | 10:02 |
| tristan | Originally, we did not have get_consistency(), and if get_unique_key() reported None, then it was assumed to be INCONSISTENT, but then there is also the required interrogation to see if the source is CACHED or not | 10:02 |
| tristan | jonathanmaw, it could be that what we have is best, or, it could be that adding a bool cached() abstract method would be better ? | 10:03 |
| tristan | jonathanmaw, the thing is, we might potentially be nearing a point where we can no longer make such drastic changes to the API | 10:03 |
| tristan | so it may be our last chances to have the most comprehensive API surface as possible | 10:04 |
| tristan | you may have some creative ideas to make a smaller more elegant api surface while implementing Sources | 10:04 |
| tristan | jonathanmaw, another thing I'm not sure I like, is how we have track(), get_ref() and set_ref(), it feels a bit redundant | 10:05 |
| tristan | however get_ref()/set_ref() was done so that we can A.) report a newly tracked ref back to the master process after a track() activity, B.) Set the ref in the master process data model before C.) rewriting the related elements in line | 10:06 |
| jonathanmaw | tristan: hrm, I think track/get_ref/set_ref do different tasks, but they are often used together, the way we currently use them. I think they ought to be fine unless there's something wrong with the way we use them (I don't have enough experience with how that's done to have any opinions about that, currently) | 10:13 |
| jonathanmaw | as far as consistency and get_unique_key, there does seem to be a bit of overlap, so I think we ought to consider if there's a way to make sure that's appropriate. | 10:14 |
| jonathanmaw | I think get_unique_key is appropriate, so mostly I'm wondering whether there are parts of consistency that are redundant | 10:15 |
| jonathanmaw | I think if we could rely on get_unique_key to return None when having a unique key is impossible (what currently has Consistency.INCONSISTENT), they get_consistency becomes a way of checking whether the ref is cached. | 10:17 |
| jonathanmaw | then again, I haven't had a look at the exact code, there might be issues of having the appropriate information at the appropriate time. | 10:17 |
| *** jonathanmaw has quit IRC | 10:41 | |
| *** jonathanmaw has joined #buildstream | 11:34 | |
| tristan | jonathanmaw, makes sense, yeah a None get_unique_key is redundant with an Consistency.INCONSISTENT value | 11:58 |
| tristan | that is a sure thing basically | 11:58 |
| tristan | we could have a cached() which replaces it, and possibly a documented guarantee in the Source docs that cached() will never be called if get_unique_key() has returned None | 11:59 |
| tristan | and remove get_consistency() | 11:59 |
| tristan | jonathanmaw, anyway, the building is not on fire but just wanted you to keep in mind that we can still make the API better, and if you have some ideas for that, around this time would be the best time :) | 12:00 |
| tristan | paulsher1ood, https://buildstream.gitlab.io/buildstream/ now includes "Installing BuildStream" section, and the README now points to the docs | 12:02 |
| * tristan is just heading out to lunch | 12:02 | |
| jonathanmaw | tristan: after a lot of fiddling with bzr, I have a quandary about url: Should it be the full URL, including branch name (e.g. https://launchpad.net/bzr/trunk), or just the repo name, with a separate field for the branch? | 12:26 |
| jonathanmaw | going with just the repo name makes it easier to cache the repo and reduce duplication if we cache multiple branches | 12:27 |
| jonathanmaw | there doesn't seem to be any way to clone the entire bzr repo, AIUI | 12:28 |
| jonathanmaw | hrm, I'll see if trove ever found a way to do that. | 12:28 |
| jonathanmaw | aha, the tool I meant was "lorry" | 12:28 |
| *** franred has quit IRC | 12:33 | |
| jonathanmaw | looks like lorry expected it to be done by specifying a list of branches, so mystery black-box it is! | 12:33 |
| *** franred has joined #buildstream | 12:35 | |
| jonathanmaw | also, reading up on bzr, the revision number is unique to a repository, in principle, assuming every branch branched off the trunk at some point | 12:50 |
| jonathanmaw | in reality, an individual revision is identified by the branch name and revision number | 12:50 |
| jonathanmaw | so while I could copy the git model and have "track" for the branch, and "ref" for the revision, I would be using them for something significantly different compared to what git does. | 12:52 |
| tristan | jonathanmaw, I see... | 13:39 |
| tristan | So... in that case, would it make sense that "For a bzr source, the 'track' parameter is a hard requirement to specify the branch, and running track() will update ref with the latest revision of that branch" ? | 13:40 |
| tristan | sounds to me like it would work well enough | 13:41 |
| jonathanmaw | yep | 13:41 |
| jonathanmaw | and we want to make the revision number mandatory, to avoid changing from beneath us without us knowing about it. | 13:41 |
| tristan | yes, if the revision is stored as 'ref' and the regular 'ref' semantics apply, then buildstream will refuse to fetch (or consequently build) until that is resolved | 13:43 |
| tristan | sounds like bzr greatly resembles svn | 13:44 |
| jonathanmaw | hmm, to atomically branch within a repo, what I really need to do is create a tmpdir within the repo, and when I've finish branching, move the dir to its desired name | 15:48 |
| jonathanmaw | though I'm not sure if there's a way to atomically pull any new changes that might have come in | 15:49 |
| *** tristan has quit IRC | 15:54 | |
| *** jude has quit IRC | 15:55 | |
| *** franred has quit IRC | 16:02 | |
| *** tristan has joined #buildstream | 16:02 | |
| *** ChanServ sets mode: +o tristan | 16:03 | |
| jonathanmaw | tristan: should I give up on atomicity, or accept that it'll pull the entire branch each time to make sure it's got the latest verson? | 16:04 |
| tristan | I | 16:04 |
| tristan | sorry, I'm puzzled | 16:04 |
| jonathanmaw | I'm not sure of any way to check I've got the right version that won't risk leaving the repo in a dodgy state | 16:04 |
| jonathanmaw | since it doesn't looks like bzr guarantees atomicity in its operations. | 16:04 |
| tristan | Well, there is no automation on our side for *creating* branches right ? | 16:05 |
| tristan | And what we need is only to find the latest revision of a given branch... to do that part, what is missing ? | 16:05 |
| * tristan carefully rereads previous statements | 16:05 | |
| tristan | I dont get "to atomically branch within a repo" | 16:06 |
| jonathanmaw | tristan: bzr has unhelpful terminology to git users. branching with bzr includes pulling the source code in from a remote place | 16:07 |
| tristan | I see | 16:07 |
| jonathanmaw | I do it in a repository, so we can make use of shared objects (or whatever their bzr equivalent is) when mirroring multiple branches | 16:07 |
| jonathanmaw | (bzr supports having branches that don't exist in a repository, if we're not actually interested in that) | 16:08 |
| jonathanmaw | and bzr branch doesn't seem to be atomic, based on my experience of ctrl-C'ing branching from one dir to another | 16:08 |
| tristan | Ummm, ok I think I follow, so we have a "repository" for the mirror, as opposed to probably what a regular developer using bzr would use, which is something like a "checkout" | 16:08 |
| jonathanmaw | (du -sh reports different memory usage with a finished and unfinished dir) | 16:09 |
| jonathanmaw | tristan: yep, we use a repository instead of a checkout, since ideally buildstream would be able to build without being connected to the internet | 16:09 |
| jonathanmaw | assuming the code was all fetched beforehand | 16:09 |
| tristan | Yep, ok so the concern now is that failing to mirror a branch can be bad | 16:10 |
| jonathanmaw | yes | 16:10 |
| tristan | And possibly also, concurrent access to the same repo ? | 16:10 |
| tristan | the second is corner case and plausibly workaroundable with file locking | 16:10 |
| jonathanmaw | also, that I don't think I can trust `bzr pull` (which fetches any new changes) to be atomic, either. | 16:10 |
| jonathanmaw | I haven't thought about concurrent access, yet, but that's going to have trouble if we're not being atomic. | 16:11 |
| tristan | Ok, so one solution about the branch you suggested is to get that branch into a temp directory, but then we lose out on shared objects ? | 16:11 |
| jonathanmaw | tristan: unless that temp directory is in the repo, where we're using it for a random name that can't be guessed. | 16:12 |
| tristan | I was thinking that maybe, at the cost of some temporary disk space, we could A.) Mirror everything currently already mirrored into a temp dir, B.) Run the network operations in that temp dir instead and C.) If/When this succeeds, *replace* the original mirror with the now modified one ? | 16:13 |
| tristan | I have a feeling that we are going to want to fully prevent concurrency on some or all Sources | 16:14 |
| tristan | (unrelated to pull/fetch atomicity I think) | 16:14 |
| tristan | It looks like ostree handles parallel fetches into the same mirror well | 16:15 |
| tristan | and git does not really lock for you | 16:15 |
| tristan | i.e. https://gitlab.com/BuildStream/buildstream/issues/5 would also be "fixed" by this step | 16:16 |
| jonathanmaw | ok, so pulling a branch (taking in upstream changes) is a matter of: Branch into a tmpdir; Change the metadata so it recognizes the original upstream; bzr pull; move tmpdir on top of old branch. | 16:16 |
| tristan | and the core could handle it in one place so that Sources dont have to care, they just get the guarantee that "no Source shalt be operated on in parallel" | 16:16 |
| tristan | I dont understand the move tmpdir on top of the old branch | 16:17 |
| tristan | I was rather thinking, "Copy this whole mirror, all branches, everything, to tempdir" | 16:17 |
| tristan | "Do it there" | 16:17 |
| tristan | "If success, replace original mirror" | 16:17 |
| jonathanmaw | tristan: aha, because we have no guarantee things are happening nicely to the shared object. | 16:18 |
| tristan | Right | 16:18 |
| tristan | and it feels a bit evil to step in and move things around internal of the repo | 16:18 |
| tristan | s/evil/possibly error prone ? | 16:18 |
| jonathanmaw | ok | 16:21 |
| tristan | alright sounds like a sane solution | 16:21 |
| tristan | I guess nothing bad is going to happen when, e.g., we 'track' a bzr branch only to find that no new revision existed in the first place | 16:22 |
| tristan | no real need to case that differently | 16:23 |
| *** ssam2 has quit IRC | 16:27 | |
| *** xjuan has joined #buildstream | 16:31 | |
| jonathanmaw | tristan: though if every time we check for updates it takes as much work as cloning from scratch, I don't think we're saving any thing by caching. | 16:34 |
| tristan | jonathanmaw, is that really true, though ? | 16:39 |
| tristan | jonathanmaw, I mean, in the above scenario, we double the local mirror into a temp dir and do the "Get me the latest of this branch, which might already be here" in there | 16:40 |
| tristan | Just for atomicity | 16:40 |
| tristan | We still use that whatever bzr provides to get the latest of that branch in that tmp dir | 16:40 |
| jonathanmaw | tristan: hrm, I suppose not in that case. I'll have to think about the other cases. | 16:40 |
| *** ssam2 has joined #buildstream | 16:41 | |
| tristan | the case of "Ensure we have this ref" (i.e. fetch), is; we ideally want to download as little as possible | 16:41 |
| tristan | but may need the entire 'track' specified branch | 16:41 |
| tristan | only in the case that we find that, "we do not have this revision/branch locally" | 16:42 |
| *** tiagogomes has quit IRC | 16:56 | |
| *** tristan has quit IRC | 16:57 | |
| *** jonathanmaw has quit IRC | 17:11 | |
| *** ssam2 has quit IRC | 17:14 | |
| *** xjuan has quit IRC | 19:01 | |
| *** xjuan has joined #buildstream | 21:18 | |
| *** xjuan has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!