This page exsists to collect benchmarking data about git-annex, so it can be referred to later. If you have a specific instance where git-annex seems unncessarily slow, please file a bug report about it.
This page exsists to collect benchmarking data about git-annex, so it can be referred to later. If you have a specific instance where git-annex seems unncessarily slow, please file a bug report about it.
Built git-annex with profiling, using
stack build --profile
(For reproduciblity, running git-annex in a clone of the git-annex repo https://github.com/RichiH/conference_proceedings with rev 2797a49023fc24aff6fcaec55421572e1eddcfa2 checked out. It has 9496 annexed objects.)
Profiling
git-annex find +RTS -p
:This is interesting!
Fully 40% of CPU time and allocations are in list (really String) processing, and the details of the profiling report show that
spanList
andstartsWith
andjoin
are all coming from calls toreplace
inkeyFile
andfileKey
. Both functions nest several calls to replace, so perhaps that could be unwound into a single pass and/or a ByteString used to do it more efficiently.12% of run time is spent calculating the md5 hashes for the hash directories for .git/annex/objects. Data.Hash.MD5 is from missingh, and it is probably a quite unoptimised version. Switching to the version if cryptonite would probably speed it up a lot.
Instead of profiling
git annex copy --to remote
, I profiledgit annex find --not --in web
, which needs to do the same kind of location log lookup.The adjustGitEnv overhead is a surprise! It seems it is getting called once per file, and allocating a new copy of the environment each time. Call stack: withIndex calls withIndexFile calls addGitEnv calls adjustGitEnv. Looks like simply making gitEnv be cached at startup would avoid most of the adjustGitEnv slowdown.
(The catchIO overhead is a false reading; the detailed profile shows that all its time and allocations are inherited. getAnnexLinkTarget is running catchIO in the expensive case, so readSymbolicLink is the actual expensive bit.)
The parsePOSIXTime comes from reading location logs. It's implemented using a generic Data.Time.Format.parseTime, which uses a format string "%s%Qs". A custom parser that splits into seconds and picoseconds and simply reads both numbers might be more efficient.
catObjectDetails.receive is implemented using mostly String and could probably be sped up by being converted to use ByteString.
After all that, profiling
git-annex find
:And
git-annex find --not --in web
:So, quite a large speedup overall!
This leaves md5 still unoptimised at 10-28% of CPU use. I looked at switching it to cryptohash's implementation, but it would require quite a lot of bit-banging math to pull the used values out of the ByteString containing the md5sum.
Switched from MissingH to cryptonite for md5. It did move md5 out of the top CPU spot but the overall runtime didn't change much. Memory allocations did go down by a good amount.
Updated profiles: