Labels

Friday, August 23, 2013

Find duplicates in bash


find_dupes.sh


 #! /bin/bash
OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
echo ""                >> $OUTF;
find "$@" -type f -print0 | xargs -0 -n1 md5sum | sort --key=1,32 | uniq -w 32 -d --all-repeated=separate | sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF

Run the script:
./find_dupes.sh /path/to/check .\

Which will output a file called rem-duplicates.sh which can then be edited as you wish.

I stole this from here. Credit to , who in turn used code from J. Elonen


LED lights on my Vespa