Gitolite is a git repository manager. Here's how to add git-annex support to gitolite, so you can git annex copy files to a gitolite repository, and git annex get files from it.

Warning : The method described here works with gitolite version g2, avaible in the g2 branch on github. There is an experimental support for g3 in the git-annex branch, if you tested it please add some feedback.

A nice feature of using gitolite with git-annex is that users can be given read-only access to a repository, and this allows them to git annex get file contents, but not change anything.

First, you need new enough versions:

  • gitolite 2.2 is needed -- this version contains a git-annex-shell ADC and supports "ua" ADCs.
  • git-annex 3.20111016 or newer needs to be installed on the gitolite server. Don't install an older version, it wouldn't be secure!

And here's how to set it up. The examples are for gitolite as installed on Debian with apt-get, but the changes described can be made to any gitolite installation, just with different paths.

Set $GL_ADC_PATH in .gitolite.rc, if you have not already done so.

echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/";' >>~gitolite/.gitolite.rc

Make the ADC directory, and a "ua" subdirectory.

   
mkdir -p /usr/local/lib/gitolite/adc/ua

Install the git-annex-shell ADC into the "ua" subdirectory from the gitolie repository.

   
cd /usr/local/lib/gitolite/adc/ua/
cp gitolite/contrib/adc/git-annex-shell .

Now all gitolite repositories can be used with git-annex just as any ssh remote normally would be used. For example:

# git clone gitolite@localhost:testing
Cloning into testing...
Receiving objects: 100% (18/18), done.
# cd testing
# git annex init
init  ok
# cp /etc/passwd my-cool-big-file
# git annex add my-cool-big-file
add my-cool-big-file ok
(Recording state in git...)
# git commit -m added
[master d36c8b4] added
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 120000 my-cool-big-file
# git push --all
Counting objects: 17, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 1.39 KiB, done.
Total 14 (delta 0), reused 1 (delta 0)
To gitolite@localhost:testing
   c552a38..db4653e  git-annex -> git-annex
   29cd204..d36c8b4  master -> master
# git annex copy --to origin
copy my-cool-big-file (checking origin...) (to origin...) 
WORM-s2502-m1318875140--my-cool-big-file
        2502 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)

sent 2606 bytes  received 31 bytes  1758.00 bytes/sec
total size is 2502  speedup is 0.95
ok

Troubleshooting

I got an error like this when setting up gitolite after setting up a local git repo and git annex:

git-annex-shell: First run: git-annex init
Command ssh ["git@git.example.com","git-annex-shell 'configlist' '/~/myrepo.git'"] failed; exit code 1

because I forgot to "git push --all" after adding the new gitolite remote.

Looks like you are missing a closing double quote on the line:

echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/;' >>~gitolite/.gitolite.rc

right after /;

I got this working by the way - great stuff.

Comment by http://www.openid.albertlash.com/openid/ Sat Dec 24 06:08:45 2011
I've fixed the typo (anyone can edit pages in this wiki FWIW.)
Comment by http://joey.kitenet.net/ Sat Dec 24 16:54:31 2011
I'm confused by the fact that the git-annex-shell adc rejects any repo names that don't start with /~/ since none of my repos start that way. It seems work ok if I just delete /\~ from the front of the regex, but I feel like I must be missing something.
Comment by bremner Fri Dec 30 21:41:13 2011

Well a repo url like gitolite@localhost:testing puts it in the gitolite user's /~/testing

This worked when I added the gitolite stuff, anyway.. Let's see if it still does:

joey@gnu:~/tmp>mkdir g
joey@gnu:~/tmp>cd g
joey@gnu:~/tmp/g>git init
Initialized empty Git repository in /home/joey/tmp/g/.git/
joey@gnu:~/tmp/g>git annex init
init  ok
joey@gnu:~/tmp/g>git remote add test 'gitolite@localhost:testing'
joey@gnu:~/tmp/g>touch foo
joey@gnu:~/tmp/g>git annex add foo
add foo (checksum...) ok
(Recording state in git...)
joey@gnu:~/tmp/g>git annex copy foo --to test --debug
git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","ls-files","--cached","-z","--","foo"]
git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","check-attr","annex.numcopies","-z","--stdin"]
git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","show-ref","--hash","refs/heads/git-annex"]
git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","show-ref","git-annex"]
git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","cat-file","--batch"]
Running: ssh ["-4","gitolite@localhost","git-annex-shell 'configlist' '/~/testing'"]

Still seems right, the ADC's regexp will match this the git-annex shell command.

Comment by http://joey.kitenet.net/ Sat Dec 31 00:29:45 2011

I guess there is some path rewriting going in in gitolite proper because if try a url of the form ssh://git@localhost/testing, then it still works with gitolite, but fails with the ADC because the repo is passed as /testing:

Running: ssh ["git@host","git-annex-shell 'configlist' '/recommend'"]
Running: ssh ["git@host","git-annex-shell 'configlist' '/recommend'"]

What I have to ask Sitaram and or find in the docs is if this is a bug or a feature in gitolite. I can see how the leading slash would get swallowed up by this line

$repo = "'$REPO_BASE/$repo.git'"

in gl-auth-command, but I guess that isn't the whole story.

Comment by bremner Sat Dec 31 01:50:49 2011

I confirmed with Sitaram that this is intentional, if probably under-documented. Since the ADC strips the leading /~/ in assigning $start anyway, I guess something like the following will work


diff --git a/contrib/adc/git-annex-shell b/contrib/adc/git-annex-shell
index 7f9f5b8..523dfed 100755
--- a/contrib/adc/git-annex-shell
+++ b/contrib/adc/git-annex-shell
@@ -28,7 +28,7 @@ my $cmd=$ENV{SSH_ORIGINAL_COMMAND};
 # the second parameter.
 # Further parameters are not validated here (see below).
 die "bad git-annex-shell command: $cmd"
-    unless $cmd =~ m#^(git-annex-shell '\w+' ')/\~/([0-9a-zA-Z][0-9a-zA-Z._\@/+-
+    unless $cmd =~ m#^(git-annex-shell '\w+' ')/(?:\~\/)?([0-9a-zA-Z][0-9a-zA-Z.
 my $start = $1;
 my $repo = $2;
 my $end = $3;
Comment by bremner Sat Dec 31 03:34:17 2011

That patch seems ok, it doesn't seem to allow through any repo locations that were blocked before.

So, it has my blessing.. but the ADC is in gitolite and will need to be patched there.

Comment by http://joey.kitenet.net/ Sat Dec 31 18:32:28 2011

After some debugging printing, here is my current understanding.

  • urls of the form git@host:~repo or ssh://git@host

    • git sends commands like "git-receive-pack '~/repo'
    • gitolite converts these to $REPO_BASE/~/repo which fails. ~/repo would also fail fwiw.
    • git-annex sends seems /~/repo, which works
  • urls of the form git@host:/repo or ssh://git@host/repo

    • git sends "git-receive-pack '/db/cs3383'"
    • gitolite converts this to $REPO_BASE/repo which works
    • git annex sends "git-annex-shell 'inannex' '/repo' ..." which works, but only with the patch above.
  • urls of the form git@host:repo

    • git sends "git-receive-pack 'repo'
    • gitolite converts this to $REPO_BASE/repo, which works
    • git-annex sends "git-annex-shell 'inannex' '/~/db/cs3383'...", which also works for git-annex-shell.

So the weird case is the last one where git and git-annex are sending different things over the wire. I don't know if you have other motivations for doing the url normalization on the client side, but it isn't needed for gitolite, and in some sense complicates things a little. On the other hand, now that I see what is going on, it isn't a big deal to just strip the leading /~ off in the adc. It does lead to the odd situation of some URLs working for git-annex but not git.

Comment by bremner Sat Dec 31 22:29:38 2011
Ah right. git-annex normalizes all git ssh style user@host:dir to valid uris, which is where the /~/ comes from. I don't anticipate this changing on the git-annex side.
Comment by http://joey.kitenet.net/ Mon Jan 2 16:27:55 2012
See http://gitolite.com/gitolite/dev-status.html for some details.
Comment by http://ertai.myopenid.com/ Mon Mar 25 12:47:21 2013

Hi,

I noticed using the git-annex branch of gitolite v3 that the same URL with ".git" at the end would not work in git-annex. For example my test repository was git@git2.mildred.fr:u/mildred/Annex.git but it didn't work until I converted it to git@git2.mildred.fr:u/mildred/Annex

On the server, the repository is in repositories/u/mildred/Annex.git

If I try a copy with git-annex for example, I would get:

$ git annex copy titi --to test
copy titi (checking test...) FATAL: u/mildred/Annex.git mildred DENIED

(unable to check test) failed
git-annex: copy: 1 failed

(test is the name of my remote and titi is my file)

Note, in my gitolite conf, I have:

repo u/CREATOR/[a-zA-Z0-9].*
  C    = @all
  RW+D = CREATOR
  RW   = WRITERS
  R    = READERS
Comment by http://mildred.fr/ Fri May 24 12:15:16 2013
latest code of gitolite does not support git-annex ? I could not find a way to make it work ?

The steps to activate git-annex integration have changed/simplified for v3.

1) during install, be sure to use the 'git-annex' branch, rather than master[fn:1].

2) to enable git-annex-shell, open ~/.gitolite.rc and insert 'git-annex-shell' => 'ua' into the hash list in the COMMANDS array.[fn:2]

'git-annex-shell' => 'ua',

[fn:1] We'd like to have this feature-branch merged to master, so please send Sitaram feedback, positive and negative, based on your experiences. [fn:2] There is no GL_ADC_PATH and no "ua" subdirectory here, and nothing to "install"; the command now comes with gitolite.

@khaije

Could you paste your config file? Here is mine: http://paste.debian.net/44856/ I don't have any COMMANDS array. Could you elaborate your modifications please?

Thanks.

@François

The proper array in .gitolite.rc seems to be the "ENABLE" array, which it appears is parsed into the COMMANDS array in src/lib/Gitolite/Rc.pm

@Khaije

Using 'git-annex-shell' => 'ua' doesn't seem to work for me. The program still fails in src/gitolite-shell around line 163 (gitolite repo version b1d3c05):

_die "suspicious characters loitering about '$soc'"
  if $rc{COMMANDS}{ words[0] } ne 'ua' and $soc !~ $REMOTE_COMMAND_PATT;

When I insert $rc{COMMANDS}{ words[0] } into the _die message, it shows up as "1" instead of "ua" as I was expecting.

When I manually set $rc{COMMANDS}{ words[0] } to 'ua' slightly earlier in the script, the git-annex-shell command gets run but it seems to fail to parse the result of the configlist command properly because then I get

Failed to get annex.uuid configuration of repository origin

Instead, got: "annex.uuid=\ncore.gcrypt=\n"

I am actually about to give up on the notion of using git-annex and gitolite together. Maybe. I am interested to know if anyone else is having similar problems.

We have reached the same point as the previous poster from 25 days ago. $ git annex copy --to origin FATAL: suspicious characters loitering about 'git-annex-shell 'configlist' '/~/testing''

Remote origin does not have git-annex installed; setting remote.origin.annex-ignore git-annex: cannot determine uuid for origin

Anyone actually have this working?

my gitolite.rc is available at https://gist.github.com/khaije1/7609848

For whatever reason I've found this to be very simple to get working so I'd guess there's a missing ingredient somewhere. The combination of gitolite and git-annex is valuable to me so I'll add documents to the url above in hopes it will assist some people with getting the same value.

Adding:

'git-annex-shell' =>1,

To the .gitolite.rc file resulted in the "FATAL: suspicous characters loitering about 'git-annex-shell 'configlist' '/~/testing''...

Gitolite source code (https://github.com/sitaramc/gitolite/commit/b1d3c0571409b7c6279fc6a77253c3bc262ab425#diff-79a3701e9e2cee0ea1316451c21a3fec) requires this entry:

'git-annex-shell ua'

Comments on this page are closed.