This is a first attempt to work with PyTest asserts. Test script for the md5 and crc functions of checksum_speed_test scripts.
Outputs to log the following data, tab separated:įilepath MD5/CRC32 chunk size Size in MB Time taken in seconds Python version. Passes the filepath to the following functions using timeit to record the duration taken. The script creates a filepath variable for each file, and runs a size check against it, in MB. For each path it iterates through the files within it (there is no check here for file type). Iterates through path list, stored in paths variable, and checks each path is legitimate. If you're testing in both Python2 and Python3 then you can also amend the version in line 87 and 88. To run this script you need to edit the paths variable (line 37/38) and specify a path for your log ouput (line 30). 
Mostly identical to checksum_speed_tests.py, but with print statements removed so runs silently and appends to a log file at a specified path. Tab separated: Filepath - MD5/CRC32 chunk size - Size in MB - Time taken in seconds - Python version. Also outputs to (or appends to if you have multiple attempts) a log
Outputs the time taken for each function to terminal console. md5_65536(filename): Opens the input file in bytes, splits the file into chunks and iterates through these (size 65536). Prints the MD5 checksum, formatted hexdigest. md5_4096(filename): Opens the input file in bytes, splits the file into chunks and iterates through these (size 4096). crc_65536(filename): Opens the file in bytes, and passes to zlib.crc32 in buffersizes of 65536, until the whole of the file. Prints the CRC32 checksum to the terminal output, formatted 08x. crc_4096(filename): Opens the file in bytes, and passes to zlib.crc32 in buffersizes of 4096, until the whole of the file. Makes timeit calls to the following functions supplying the filename:. If both are True it stores sys.argv (the path you supplied) as variable 'filename'. Checks the path supplied is legitimate and present. The script performs the following functions: Python checksum_speed_test.py /path_to_file/file.mkv You can drag/drop a file after the python script name to make sure the path is correct. This script allows for a single file to be input and tested against zlib CRC32 and hashlib MD5 modules of Python to see which is quicker. Tests averaged across one week of repeat testing: Result from the tests, run on 8 thread Ubuntu VM with 12GB RAM and 10Gbps network connection - testing with files on two different network shares: CRC32 chunk size 65536 using Python 3 (3.6 installed) implementation is fastest, with MD5 chunk size 4096 Python 2 (2.7 installed) implementation slowest. There are two versions of the checksum_speed_test script that allow for single use checksum testing or automated testing of directories, and both will run on Python2.7 or Python3. Timeit's default is 1,000,000 (when no number is specified) so do ensure you set a number for checksum tests that is achievable. If you are checksum testing smaller files in a collection you can change the number=1 setting to another figure, such as number=100 and run the test to receive an average time across those 100. Timeit however, was developed to create averages across multiple speed tests on short snippets of code. As the media files are generally many GBs in size the test repeat number is set to 1 in timeit, so the script is only run once per file, every four hours from crontab. They both use timeit to measure the speed that it takes to run each checksum pass. The scripts in this repository use Python standard library zlib and hashlib to generate CRC32 and MD5 hashes respectively. (Ruling out the supported SHA options as we have no need for the cryptographic functionality and prefer the speed gain over that functionality). We have decided to run some comparisons between CRC32 and MD5, as we currently only have support for these within our tape library system. One such bottleneck is caused by our use of hashlib in Python2 scripts to generate MD5 checksums for every media file before being written to LTO tape storage. Python checksum archive#
The Data and Digital Preservation team in the BFI National Archive has been running checksum speed comparisons, with an aim to reducing bottlenecks caused by an increasing volume of digital media files.
0 kommentar(er)
