IRC logs for #baserock for Saturday, 2015-06-13

*** edcragg has quit IRC00:29
paulsherwoodKinnison: where would i find that?07:25
richard_mawpaulsherwood: it will turn up in http://blog.digital-scurf.org/07:43
*** pacon has joined #baserock08:07
KinnisonAye08:14
paulsherwoodrichard_maw: thansk!08:19
* paulsherwood wonders if 'git clone --depth 1 --no-single-branch' may work and be cheaper than wget whole tarball08:23
KinnisonIn theory, yes08:25
Kinnisonbut it won't work if anything then uses the git reponess inside the staging area08:25
Kinnisone.g. for 'git describe'08:25
paulsherwoodgit describe seems to be working? i'll need to check if it's giving the same results, though08:40
* paulsherwood notices that somehow we've ended up with ref: master for git-review. 08:42
Kinnisonit's possible newer gits do better with shallow clones08:44
Kinnisonbut I thought that what happened was essentially it would deepen the clone as soon as it needed any extra data08:44
Kinnisonwhich likely wouldn't work in the staging area08:44
paulsherwoodi'll know in an hour or so08:47
Kinnisonheh08:48
* Kinnison holds out hope that few chunks will need poking08:48
KinnisonIdeally I'd prefer it if we never had to clone into the staging area at all08:48
Kinnisonjust put a working tree there08:48
* paulsherwood still doesn't 'get' the difference, believe it or not09:00
paulsherwoodbut i'm learning :)09:01
paulsherwoodhmmm... so 'git clone --depth 1 --no-single-branch' gets branches, not tags or master afaict?09:02
paulsherwoodhence doesn't give me something that can handle, for example gawk  unpetrify-ref: gawk-4.1.109:03
KinnisonAs I said, I have no idea :-(09:03
paulsherwoodnp, i'm just musing and documenting my musings :-)09:03
* Kinnison leaves this here with no comment: https://scontent-lhr3-1.xx.fbcdn.net/hphotos-xfa1/v/t1.0-9/s720x720/11401253_815858851832788_8548523855124523314_n.jpg?oh=dc711a99f5363bd51b0089ba006f97d5&oe=55FE2D7A09:03
paulsherwoodROFL09:04
*** edcragg has joined #baserock09:13
richard_mawpaulsherwood: with a shallow clone you wouldn't get the tags, so `git describe` would never provide the expected information by upstream09:16
richard_mawIf you want fastest and safe with the current feature set, you'd do a clone with --shared, and arrange for the repository from your local repo cache to be visible in the staging area, but mounted read-only09:20
richard_mawIf you want to close the reproducibility hole of having access to the full repository but only caching based on the tree you checked out, you would either have to declare which other branches, at which versions, to make available in the build directory, or09:23
richard_mawonly give the current branch, or no repository at all, and have some other mechanism for getting the version09:23
*** rdale has quit IRC10:37
*** pacon has quit IRC11:48
*** edcragg has quit IRC13:19
paulsherwoodrichard_maw: my current attempt is ''git', 'clone', '--depth', '1', '--no-single-branch', '--mirror', '-n'' followed by ''git', 'fetch', '--tags', '--depth', '1''13:35
paulsherwoodthat seems to handle the describe situation. however when a branch has moved on and is no longer the SHA1 we've locked, it's not clear what magic to use to get the SHA from g.b.o13:37
* paulsherwood tries the brute-force ''git', 'fetch', 'origin', '--depth', '1000000', this['unpetrify-ref']15:45
richard_mawpaulsherwood: I don't believe that we currently care about the builddir having the same ref as specified in unpetrify-ref, so you may be able to do a clone with `--no-checkout` then `'git', 'checkout', this['ref']`, as morph doesn't assume `unpetrify-ref` points to anything you could fetch `ref` from21:11
richard_mawpaulsherwood: but we do assume that `ref` exists somewhere in the repository, so if you do a clone with `--mirror`, or at least fetch all refs, that it would be anchored by one of your refs, so you may call `git checkout` and be able to retrieve the tree of the commit that way21:13
paulsherwoodrichard_maw: but that clone would pull everything? which is the load i'm trying to reduce?21:13
paulsherwoods/pull/fetch/21:14
richard_mawpaulsherwood: hmm, we'd need to make the unpetrify-ref definition more strict, or find a way to do a fetch without having an anchor ref21:18
richard_mawpaulsherwood: I'm not sure the fetch protocol allows this, I'll have a look.21:19
richard_mawYou can fetch a tree's contents when you only have the sha1 if your git server enables the `git archive` extension, but not the history, which wouldn't work for `git describe`21:20
paulsherwoodrichard_maw: i think my brute-force approach will work, for now.21:23
paulsherwoodi don't want to provoke unnecessary work on gbo21:23
richard_mawthat's kind of what the git cache server is for, letting you do some operations without needing to clone the whole repository21:28
paulsherwoodyou mean /src/cache/gits ?21:29
richard_mawno, the web service running on git.baserock.org that lets you do some requests without fetching all the repositories to /src/cache/gits21:30
paulsherwoodwhat kind of requests?21:30
paulsherwoodi use it for trees, but that's not much of a saving really21:30
richard_mawa) what commit and tree sha1 does this ref resolve to; b) what file names are in the top level of this tree21:31
paulsherwoodin any case i think my solution needs to work with jrandom servers, not just gbo21:31
richard_mawpaulsherwood: this is why we fall back to cloning when we don't have the repository locally and the remote cache server doesn't have it21:32
paulsherwoodyes, i use a) already. whats's the benefit of b) ?21:32
richard_mawpaulsherwood: we need to know the cache keys of things we may not need to build, but to make the cache key we need to know what commands need to be run, and these commands can come from a guessed build system, which is based on what files are present21:33
paulsherwoodah - you mean use the cache-key to check if the cache-server has it already, therefore no need to clone the git repo?21:35
paulsherwoodactually, i think the build-system is redundant information in that calculation?21:35
richard_mawor some other way to have the artifact, but not the repo, such as if you built it earlier and wiped your cache, or if you have some other way of sharing artifacts21:36
richard_mawbuild-system is not redundant21:36
paulsherwoodfor a given SHA1, i think only one build-system will occur?21:36
paulsherwoodie for SHA1, the files are fixed, therefore the build-system is fixed?21:37
richard_mawyour code is not though21:37
richard_mawwe don't cache by the build-system directly, we cache by the commands the build system provided, so that we can change what build systems mean without having to discard the whole cache21:38
paulsherwoodah, ok.21:38
paulsherwoodi don't know if the cost is worth it, though :)21:39
richard_mawI had a look at fetching by commit sha1. The command succeeds if you already had the commit, but not if you don't, which is kind of pointless.21:40
paulsherwoodyup. thanks for looking, but i already spent a while on this. i've been told the best soln is to get someone to work on a patch for git-fetch to handle shas, and wait for it to filter down to my chosen distro :)21:42
paulsherwoodit does seem an odd omission to me, though21:42
* paulsherwood goes to sleep21:43
richard_mawat one point I think git was working towards a per-branch security model21:43
paulsherwoodoh? interesting21:43
richard_mawif you already had the sha1s and could fetch contents by that rather than the branch, that would completely break that21:43
richard_mawI think the per-branch security model was abandoned21:43
richard_mawright now, I'm just looking to see if the git fetch protocol itself can handle it, such that you might be able to write a custom client21:44
KinnisonI imagine the per-branch stuff kinda failed because it meant you couldn't optimise for only receiving what you need to move forward21:44
paulsherwoodrichard_maw: i think that may be overkill for my use-case? it's an optimisation that would only matter at initialisation of user's environment21:47
* paulsherwood was expecting this to be a one-line change... it's now been several hours of google/irc and fail attempts :-)21:48
richard_mawpaulsherwood: possibly, if we had this optimisation we may not need a local cache21:54
*** zoli__ has joined #baserock22:10
richard_mawhmm, currently it appears that failing to fetch a sha1 is a bug, since there's explicit code for allowing you to set a sha1 as the source in a refspec, but when it goes to fetch the pack, it doesn't send the commits over, and it fails a later check whether the fetch function has actually succeeded in fetching the refs that were requested22:33
*** zoli__ has quit IRC22:58
*** petefoth1ringham has joined #baserock23:28
*** petefotheringham has quit IRC23:29
* richard_maw grumbles at https://github.com/git/git/blob/master/fetch-pack.c#L53323:30
richard_maw*that* is why you can't fetch a commit by its sha123:30
richard_mawI think23:30
richard_mawhm, no, the name check should fail, and fall through23:30
richard_mawhrm, no23:35
richard_mawthe problem is fundamental23:35
richard_mawthe git protocol involves asking the remote what refs it has and comparing it against what refs you want23:36
richard_mawthen asking for those refs23:36
richard_mawso if it were to allow you to ask for commits, it would need to send you a list of every commit it has23:36
richard_mawoh, hm, perhaps not23:41
richard_mawthat's interesting…23:41
richard_mawallow_tip_sha1_in_want23:41
richard_mawso if servers set uploadpack.allowtipsha1inwant in their git config then we could fetch just the commit23:44
richard_mawuploadarchive.allowUnreachable would also be necessary if we were to enable git-archive as a substitute for the current cache server functionality on trove23:47
richard_mawI should write this up in storyboard at some point23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!