| *** edcragg has quit IRC | 00:29 | |
| paulsherwood | Kinnison: where would i find that? | 07:25 |
|---|---|---|
| richard_maw | paulsherwood: it will turn up in http://blog.digital-scurf.org/ | 07:43 |
| *** pacon has joined #baserock | 08:07 | |
| Kinnison | Aye | 08:14 |
| paulsherwood | richard_maw: thansk! | 08:19 |
| * paulsherwood wonders if 'git clone --depth 1 --no-single-branch' may work and be cheaper than wget whole tarball | 08:23 | |
| Kinnison | In theory, yes | 08:25 |
| Kinnison | but it won't work if anything then uses the git reponess inside the staging area | 08:25 |
| Kinnison | e.g. for 'git describe' | 08:25 |
| paulsherwood | git describe seems to be working? i'll need to check if it's giving the same results, though | 08:40 |
| * paulsherwood notices that somehow we've ended up with ref: master for git-review. | 08:42 | |
| Kinnison | it's possible newer gits do better with shallow clones | 08:44 |
| Kinnison | but I thought that what happened was essentially it would deepen the clone as soon as it needed any extra data | 08:44 |
| Kinnison | which likely wouldn't work in the staging area | 08:44 |
| paulsherwood | i'll know in an hour or so | 08:47 |
| Kinnison | heh | 08:48 |
| * Kinnison holds out hope that few chunks will need poking | 08:48 | |
| Kinnison | Ideally I'd prefer it if we never had to clone into the staging area at all | 08:48 |
| Kinnison | just put a working tree there | 08:48 |
| * paulsherwood still doesn't 'get' the difference, believe it or not | 09:00 | |
| paulsherwood | but i'm learning :) | 09:01 |
| paulsherwood | hmmm... so 'git clone --depth 1 --no-single-branch' gets branches, not tags or master afaict? | 09:02 |
| paulsherwood | hence doesn't give me something that can handle, for example gawk unpetrify-ref: gawk-4.1.1 | 09:03 |
| Kinnison | As I said, I have no idea :-( | 09:03 |
| paulsherwood | np, i'm just musing and documenting my musings :-) | 09:03 |
| * Kinnison leaves this here with no comment: https://scontent-lhr3-1.xx.fbcdn.net/hphotos-xfa1/v/t1.0-9/s720x720/11401253_815858851832788_8548523855124523314_n.jpg?oh=dc711a99f5363bd51b0089ba006f97d5&oe=55FE2D7A | 09:03 | |
| paulsherwood | ROFL | 09:04 |
| *** edcragg has joined #baserock | 09:13 | |
| richard_maw | paulsherwood: with a shallow clone you wouldn't get the tags, so `git describe` would never provide the expected information by upstream | 09:16 |
| richard_maw | If you want fastest and safe with the current feature set, you'd do a clone with --shared, and arrange for the repository from your local repo cache to be visible in the staging area, but mounted read-only | 09:20 |
| richard_maw | If you want to close the reproducibility hole of having access to the full repository but only caching based on the tree you checked out, you would either have to declare which other branches, at which versions, to make available in the build directory, or | 09:23 |
| richard_maw | only give the current branch, or no repository at all, and have some other mechanism for getting the version | 09:23 |
| *** rdale has quit IRC | 10:37 | |
| *** pacon has quit IRC | 11:48 | |
| *** edcragg has quit IRC | 13:19 | |
| paulsherwood | richard_maw: my current attempt is ''git', 'clone', '--depth', '1', '--no-single-branch', '--mirror', '-n'' followed by ''git', 'fetch', '--tags', '--depth', '1'' | 13:35 |
| paulsherwood | that seems to handle the describe situation. however when a branch has moved on and is no longer the SHA1 we've locked, it's not clear what magic to use to get the SHA from g.b.o | 13:37 |
| * paulsherwood tries the brute-force ''git', 'fetch', 'origin', '--depth', '1000000', this['unpetrify-ref'] | 15:45 | |
| richard_maw | paulsherwood: I don't believe that we currently care about the builddir having the same ref as specified in unpetrify-ref, so you may be able to do a clone with `--no-checkout` then `'git', 'checkout', this['ref']`, as morph doesn't assume `unpetrify-ref` points to anything you could fetch `ref` from | 21:11 |
| richard_maw | paulsherwood: but we do assume that `ref` exists somewhere in the repository, so if you do a clone with `--mirror`, or at least fetch all refs, that it would be anchored by one of your refs, so you may call `git checkout` and be able to retrieve the tree of the commit that way | 21:13 |
| paulsherwood | richard_maw: but that clone would pull everything? which is the load i'm trying to reduce? | 21:13 |
| paulsherwood | s/pull/fetch/ | 21:14 |
| richard_maw | paulsherwood: hmm, we'd need to make the unpetrify-ref definition more strict, or find a way to do a fetch without having an anchor ref | 21:18 |
| richard_maw | paulsherwood: I'm not sure the fetch protocol allows this, I'll have a look. | 21:19 |
| richard_maw | You can fetch a tree's contents when you only have the sha1 if your git server enables the `git archive` extension, but not the history, which wouldn't work for `git describe` | 21:20 |
| paulsherwood | richard_maw: i think my brute-force approach will work, for now. | 21:23 |
| paulsherwood | i don't want to provoke unnecessary work on gbo | 21:23 |
| richard_maw | that's kind of what the git cache server is for, letting you do some operations without needing to clone the whole repository | 21:28 |
| paulsherwood | you mean /src/cache/gits ? | 21:29 |
| richard_maw | no, the web service running on git.baserock.org that lets you do some requests without fetching all the repositories to /src/cache/gits | 21:30 |
| paulsherwood | what kind of requests? | 21:30 |
| paulsherwood | i use it for trees, but that's not much of a saving really | 21:30 |
| richard_maw | a) what commit and tree sha1 does this ref resolve to; b) what file names are in the top level of this tree | 21:31 |
| paulsherwood | in any case i think my solution needs to work with jrandom servers, not just gbo | 21:31 |
| richard_maw | paulsherwood: this is why we fall back to cloning when we don't have the repository locally and the remote cache server doesn't have it | 21:32 |
| paulsherwood | yes, i use a) already. whats's the benefit of b) ? | 21:32 |
| richard_maw | paulsherwood: we need to know the cache keys of things we may not need to build, but to make the cache key we need to know what commands need to be run, and these commands can come from a guessed build system, which is based on what files are present | 21:33 |
| paulsherwood | ah - you mean use the cache-key to check if the cache-server has it already, therefore no need to clone the git repo? | 21:35 |
| paulsherwood | actually, i think the build-system is redundant information in that calculation? | 21:35 |
| richard_maw | or some other way to have the artifact, but not the repo, such as if you built it earlier and wiped your cache, or if you have some other way of sharing artifacts | 21:36 |
| richard_maw | build-system is not redundant | 21:36 |
| paulsherwood | for a given SHA1, i think only one build-system will occur? | 21:36 |
| paulsherwood | ie for SHA1, the files are fixed, therefore the build-system is fixed? | 21:37 |
| richard_maw | your code is not though | 21:37 |
| richard_maw | we don't cache by the build-system directly, we cache by the commands the build system provided, so that we can change what build systems mean without having to discard the whole cache | 21:38 |
| paulsherwood | ah, ok. | 21:38 |
| paulsherwood | i don't know if the cost is worth it, though :) | 21:39 |
| richard_maw | I had a look at fetching by commit sha1. The command succeeds if you already had the commit, but not if you don't, which is kind of pointless. | 21:40 |
| paulsherwood | yup. thanks for looking, but i already spent a while on this. i've been told the best soln is to get someone to work on a patch for git-fetch to handle shas, and wait for it to filter down to my chosen distro :) | 21:42 |
| paulsherwood | it does seem an odd omission to me, though | 21:42 |
| * paulsherwood goes to sleep | 21:43 | |
| richard_maw | at one point I think git was working towards a per-branch security model | 21:43 |
| paulsherwood | oh? interesting | 21:43 |
| richard_maw | if you already had the sha1s and could fetch contents by that rather than the branch, that would completely break that | 21:43 |
| richard_maw | I think the per-branch security model was abandoned | 21:43 |
| richard_maw | right now, I'm just looking to see if the git fetch protocol itself can handle it, such that you might be able to write a custom client | 21:44 |
| Kinnison | I imagine the per-branch stuff kinda failed because it meant you couldn't optimise for only receiving what you need to move forward | 21:44 |
| paulsherwood | richard_maw: i think that may be overkill for my use-case? it's an optimisation that would only matter at initialisation of user's environment | 21:47 |
| * paulsherwood was expecting this to be a one-line change... it's now been several hours of google/irc and fail attempts :-) | 21:48 | |
| richard_maw | paulsherwood: possibly, if we had this optimisation we may not need a local cache | 21:54 |
| *** zoli__ has joined #baserock | 22:10 | |
| richard_maw | hmm, currently it appears that failing to fetch a sha1 is a bug, since there's explicit code for allowing you to set a sha1 as the source in a refspec, but when it goes to fetch the pack, it doesn't send the commits over, and it fails a later check whether the fetch function has actually succeeded in fetching the refs that were requested | 22:33 |
| *** zoli__ has quit IRC | 22:58 | |
| *** petefoth1ringham has joined #baserock | 23:28 | |
| *** petefotheringham has quit IRC | 23:29 | |
| * richard_maw grumbles at https://github.com/git/git/blob/master/fetch-pack.c#L533 | 23:30 | |
| richard_maw | *that* is why you can't fetch a commit by its sha1 | 23:30 |
| richard_maw | I think | 23:30 |
| richard_maw | hm, no, the name check should fail, and fall through | 23:30 |
| richard_maw | hrm, no | 23:35 |
| richard_maw | the problem is fundamental | 23:35 |
| richard_maw | the git protocol involves asking the remote what refs it has and comparing it against what refs you want | 23:36 |
| richard_maw | then asking for those refs | 23:36 |
| richard_maw | so if it were to allow you to ask for commits, it would need to send you a list of every commit it has | 23:36 |
| richard_maw | oh, hm, perhaps not | 23:41 |
| richard_maw | that's interesting… | 23:41 |
| richard_maw | allow_tip_sha1_in_want | 23:41 |
| richard_maw | so if servers set uploadpack.allowtipsha1inwant in their git config then we could fetch just the commit | 23:44 |
| richard_maw | uploadarchive.allowUnreachable would also be necessary if we were to enable git-archive as a substitute for the current cache server functionality on trove | 23:47 |
| richard_maw | I should write this up in storyboard at some point | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
