md5sum vs. md5deep vs. openssl md5: MD5 calculation speed test
The following experiment is conducted to determine if md5sum, md5deep or openssl md5 hash calculations are faster than the others.
Test 1: A directory of test files consisting of disk images and extracted Windows Registry files will be scanned with the command 'find -type f | xargs -n 1 md5program'. The output of this command will be fed into each program, respectively. The time for the entire process to finish will be tracked with the time command. Xargs is being used to feed each line of output to the md5 program.
Test 1.1: After all programs have finished, the computer will be restarted, and the order the programs are ran in will be changed. For example, if md5sum was the first program in the first round, it will be the second program in the second round. This is to attempt to account for any caching that takes place in the operating system.
The result will be 9 timed runs for calculating the md5 sum of all files in a directory.
Test 2: Big files - In test 1, multiple files of different sizes will be continuously fed to the hashing program. In test 2, a single 10 GB disk image will be given to each program and timed.
Test machine:
Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz (4 cores)
8 GB RAM
Timing cached reads: 8892.40 MB/sec
Timing buffered disk reads: 403.98 MB/sec
Results:
| md5sum, run 1 | md5sum, run 2 | md5sum, run 3 |
| real 3m46.457s | real 3m46.117s | real 3m59.198s |
| user 0m42.183s | user 0m39.254s | user 0m42.235s |
| sys 3m3.595s | sys 3m6.300s | sys 3m16.096s |
| md5deep, run 1 | md5deep, run 2 | md5deep, run 3 |
| real 3m57.595s | real 3m57.666s | real 4m2.255s |
| user 0m43.743s | user 0m43.567s | user 0m44.551s |
| sys 3m17.552s | sys 3m17.192s | sys 3m22.881s |
| openssl md5, run 1 | openssl md5, run 2 | openssl md5, run 3 |
| real 3m48.130s | real 3m43.142s | real 3m50.619s |
| user 0m37.878s | user 0m37.462s | user 0m38.558s |
| sys 3m9.436s | sys 3m5.112s | sys 3m11.136s |
Overall processor time:
Run 1, md5sum 3m45.77s; md5deep 4m01.29s; openssl 3m47.32s.
Run 2: md5sum 3m45.55s; md5deep 4m00.76s; openssl 3m42.57s.
Run 3: md5sum 3m58.34s; md5deep 4m07.43s; openssl 3m49.7s.
Average (rounded to second):
md5sum: 3m50s
md5deep: 4m03s
openssl: 3m46s
Test 1 Conclusions
While the 'real' time may be an interesting factor for investigators, what we are more interested in is the time that the program was using the processor. In this case 'user' is the time taken by the program in user mode, and sys is time in kernel mode. The basic idea is that we want a program that takes the least amount of time on the processor to do the same amount of work.
Note: from the data it appears that another process, perhaps an OS update, was taking place that affected all the programs.
Based on the average of the three runs, it appears that openssl is slightly faster, followed by md5sum and then md5deep.
It should be noted, however, that md5deep is not really being used in the way it was designed. For example, md5sum does not include a recursive mode, where md5deep does. If we run md5deep using its recursive mode - instead of find and xargs - then the time is actually slightly better than the others:
md5deep recursive - restricted to 1 thread
real 3m43.963s
user 0m41.327s
sys 3m2.103s
md5deep: 3m43s
So, in conclusion, if you are are feeding a list of files into an md5 hash program, openssl appears to be a slightly better choice over md5sum and md5deep. However, if you can choose how the file list is ingested, md5deep is probably a better choice because of speed and available features.
Test 2: Testing the time for each program to hash a 10 GB disk image. This test will not use find and xargs - it will use the program directly. md5deep will be restricted to 1 thread.
| md5sum | md5deep | openssl md5 |
| real 1m44.558s | real 1m49.491s | real 1m43.475s |
| user 0m19.369s | user 0m20.421s | user 0m18.233s |
| sys 1m24.877s | sys 1m30.534s | sys 1m24.889s |
md5sum: 1m44s
md5deep: 1m51s
openssl: 1m43s
For a single large file, it appears that openssl is also the fastest, followed by md5sum and then md5deep.
Please note: all of these tests are, at best, quite shallow. A proper testing environment with many more runs should be conducted.
Bonus observation: As set, xargs will send 1 line to the hashing program and spawn 1 process. This process seems to switch between processors, even while the same hash is calculating.
