Wednesday, April 6, 2011

Linux: compute a single hash for a given folder & contents?

Surely there must be a way to do this easily!

I've tried the linux command-line apps sha1sum & md5sum but they seem only to be able to compute hashes of individual files and output a list of hash values, one for each file.

I need to generate a single hash for the entire contents of a folder (not just the filenames).

I'd like to do something like

sha1sum /folder/of/stuff > singlehashvalue

Edit: to clarify, my files are at multiple levels in a directory tree, they're not all sitting in the same root folder.

From stackoverflow
  • If you just want to hash the contents of the files, ignoring the filenames then you can use

    cat $FILES | md5sum
    

    Make sure you have the files in the same order when computing the hash:

    cat $(echo $FILES | sort) | md5sum
    

    But you can't have directories in your list of files.

  • Try to make it in two steps:

    1. create a file with hashes for all files in a folder
    2. hash this file

    Like so:

    # for FILE in `find /folder/of/stuff -type f | sort`; do sha1sum $FILE >> hashes; done
    # sha1sum hashes
    

    Or do it all at once:

    # cat `find /folder/of/stuff -type f | sort` | sha1sum
    
  • You could sha1sum to generate the list of hash values and then sha1sum that list again, it depends on what exactly it is you want to accomplish.

  • I would pipe the results for individual files through sort (to prevent a mere reordering of files to change the hash) into md5sum or sha1sum, whichever you choose.

  • One possible way would be:

    sha1sum /path/to/folder/* | sha1sum
    

    If there is a whole directory tree, you're probably better off using find and xargs. One possible command would be

    find /path/to/folder -type f -print0 | xargs -0 sha1sum | sha1sum
    

    Edit: Good point, it's probably a good thing to sort the list of files, so:

    find path/to/folder -type f -print0 | sort -z | xargs -0 sha1sum | sha1sum
    

    RafaƂ Dowgird : If you sort after the first sha1sum, then a LF in a filename should do no harm.
    Aaron Digulla : Edited. Sort can work on 0 delimited lists with the -z option.
    David Schmitt : and don't forget to set LC_ALL=POSIX, so the various tools create locale independent output.
    • Commit the directory to git, use the commit hash. See metastore for a way to also control permissions.

    • Use a file system intrusion detection tool like aide.

    • hash a tar ball of the directory:

      tar cvf - /path/to/folder | sha1sum

    • Code something yourself, like vatine's oneliner:

      find /path/to/folder -type f -print0 | sort -z | xargs -0 sha1sum | sha1sum

    Bombe : +1 for the Git solution. :)
  • What's wrong with a tar -c /path/to/folder | sha1sum?

0 comments:

Post a Comment