>>> help(zlib.crc32)
Help on built-in function crc32 in module zlib:
crc32(...)
crc32(string[, start]) -- Compute a CRC-32 checksum of string.
An optional starting value can be specified. The returned checksum is
a signed integer.
>>> zlib.crc32("".join(chr(random.randrange(0,255)) for _ in xrange(100)))
333158331
EDIT: code that uses the start value 0xFFFF:
>>> text = "".join(chr(random.randrange(0,255)) for _ in xrange(100))
>>> zlib.crc32(text)
-964269250
>>> zlib.crc32(text, 0xFFFF)
2057263175
Answer from Matt Anderson on Stack OverflowHi,
I usually use Bash and the command line to calculate checksums (including crc32). I wanted to learn how to do it in python. I found the zlib module which handles MD5, SHA1 and SHA256 very well and fast!
However there is some problem with zlib.crc32(). For small files it returns the same results as the crc32 command in the terminal. However if I feed zlib.crc32() any file larger than roughly 2GB the results differ. I am not sure how to provide a minimal working example since I cannot attach a 2GB file. But I will show you which commands I run. Maybe there is a mistake I do not see.
In python I do:
import zlib path2file = path/to/my.file #The [2:] gets rid of the HEX 0x prefix crc_python = hex(zlib.crc32(open(path2file,'rb').read()))[2:]
In the command line I would run:
~$ crc32 path/to/my.file
As mentioned the results of these two commands are the same for smaller files but differ for larger files.
Naturally I could yield the same result as in the terminal using python by:
import subprocess
path2file = path/to/my.file
crc_shell = subprocess.run(f'crc32 "{path2file}"', shell=True, text=True, capture_output=True).stdout.strip()which kinda defeats the purpose of using python in the first place and is much slower.
Any ideas why crc_python == crc_shell for small files but crc_python != crc_shell for large files?
Thanks!
Calculating CRC using Python zlib.crc32 function - Stack Overflow
Calculate CRC32 correctly with Python - Stack Overflow
C zlib crc32 and Python zlib crc32 doesn't match - Stack Overflow
crc - Python: setting generator Polynomial in zlib.crc32 - Stack Overflow
Python 2 (unlike py3) is doing a signed 32-bit CRC.
Those sites are doing an unsigned 32-bit CRC.
The values are the same otherwise, as you can see from this:
>>> 0x100000000 - 0xb1d4025b == 0x4e2bfda5
True
One quick way to convert from 32-bit signed to 32-bit unsigned is:*
>>> -1311505829 % (1<<32)
2983461467
Or, in hex:
>>> hex(-1311505829 % (1<<32))
'0xb1d4025b'
& 0xFFFFFFFF or % 0x100000000 or & (2**32-1) or % (2**32) and so on are all equivalent ways to do the same bit-twiddling; it just comes down to which one you find most readable.
* This only works in languages that do floored integer division, like Python (-3 // 2 == -2); in languages that do truncated integer division, like Java (-3 / 2 == -1), you'll still end up with a negative number. And in languages that don't even require that division and mod go together properly, like C, all bets are off—but in C, you'd just cast the bytes to the type you want…
zlib.crc32 documentation suggests using the following approach "to generate the same numeric value across all Python versions and platforms".
import zlib
hex(zlib.crc32(b'hello-world') & 0xffffffff)
The result is 0xb1d4025b as expected.
Note that as per the docs for Python 3.10, & 0xffffffff is necessary to do only for Python 2 or earlier.
There were problems in your original C and Python code snippets. As for your second C snippet, I haven't tried to compile it, but it's not portable since byte order within an int is platform-dependant. So it will give different results depending on the endianness of the CPU.
One problem, as Serge Ballesta has mentioned, is the difference between {1, 2, 3, 4, 5, 6, 7, 8, 9} and {'1', '2', '3', '4', '5', '6', '7', '8', '9'}. Another problem is that the loop in your original C code didn't actually scan the data, since you didn't use i in the loop, as bav mentioned.
crctest.c
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <zlib.h>
#define NUM_BYTES 9
// gcc -std=c99 -lz -o crctest test.c
void do_crc(uint8_t *bytes)
{
uint32_t crc = crc32(0L, Z_NULL, 0);
for (int i = 0; i < NUM_BYTES; ++i)
{
crc = crc32(crc, bytes + i, 1);
}
printf("CRC32 value is: %lu\n", crc);
}
int main(void)
{
uint8_t bytes0[NUM_BYTES] = {1, 2, 3, 4, 5, 6, 7, 8, 9};
uint8_t bytes1[NUM_BYTES] = {'1', '2', '3', '4', '5', '6', '7', '8', '9'};
do_crc(bytes0);
do_crc(bytes1);
}
output
CRC32 value is: 1089448862
CRC32 value is: 3421780262
crctest.py
#! /usr/bin/env python
import zlib
def do_crc(s):
n = zlib.crc32(s)
return n + (1<<32) if n < 0 else n
s = b'\x01\x02\x03\x04\x05\x06\x07\x08\x09'
print `s`, do_crc(s)
s = b'123456789'
print `s`, do_crc(s)
output
'\x01\x02\x03\x04\x05\x06\x07\x08\t' 1089448862
'123456789' 3421780262
edit
Here's a better way to handle the conversion in Python:
def do_crc(s):
n = zlib.crc32(s)
return n & 0xffffffff
See the answers here for more info on this topic: How to convert signed to unsigned integer in python.
According to www.lammertbies.nl that has detailed references on CRC calculation and C routines, the CRC32 of the ASCII string 123456789 in 0xCBF43926, that is 3421780262 as an unsigned 32 integer in decimal form.
That means that your Python computation is correct, but to get same result in C you should write
uint8_t bytes[NUM_BYTES] = {'1', '2', '3', '4', '5', '6', '7', '8', '9'};
uint32_t crc = crc32(0L, Z_NULL, 0);
Alternatively, if what you want is indeed the crc 32 for uint8_t bytes[NUM_BYTES] = {1, 2, 3, 4, 5, 6, 7, 8, 9};, you must use in python 2.x:
s = ''
for i in range(10):
s += chr(i)
s
outputs : '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t'
then
zlib.crc32(s)
outputs : 1164760902
Nota: in python 3.x, you would have written : s = bytes(range(10))
I'm using right now zlib.crc32, but for C there is no such library
Um, yes, there is. It's called zlib. zlib is written in C, and it's what Python is using! Hence the name of the class.
You can use the crc32() function in zlib. That implementation is a fair bit faster than others you might find. Read zlib.h for the interface information.
You can compile zlib yourself, or it may already be installed on your system.
Update:
I now see your comment (which should be edited into the question since it is critical to getting the right answer) that you have extremely limited memory. Then you can use this:
static uint32_t crc32(uint32_t crc, unsigned char *buf, size_t len)
{
int k;
crc = ~crc;
while (len--) {
crc ^= *buf++;
for (k = 0; k < 8; k++)
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
}
return ~crc;
}
The crc is initially set to zero.
The use of ~ will give the correct result, since the uint32_t type in stdint.h is assured to be 32 bits.
If you can afford a little more code space, then unrolling the loop will likely speed it up (if the compiler doesn't already do this):
static uint32_t crc32(uint32_t crc, unsigned char *buf, size_t len)
{
crc = ~crc;
while (len--) {
crc ^= *buf++;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
}
return ~crc;
}
You said that you only have 4 KBytes of "memory". Is that just working memory for the program, or does the program have to live there as well? If you have more space in flash for example for the code, then the table can be precomputed and stored with the code. A table-driven CRC will be much faster. The zlib code provides table-driven CRCs that do one byte at a time and four-bytes at a time, requiring respectively a 1Kbyte or 4Kbyte table.
Update 2:
Since you answered in a comment that the 4KBytes are just working memory, then you should use a table-driven CRC. You can simply use the crc32() function in zlib's crc32.c and the table in crc32.h with BYFOUR undefined.
C:
UInt32
crc32(UInt32 crc, UInt8 *p, SInt len)
{
crc = ~crc;
while (--len >= 0) {
crc = crc ^ *p++;
for (SInt i = 8; --i >= 0;) {
crc = (crc >> 1) ^ (0xedb88320 & -(crc & 1));
}
}
return ~crc;
}
void
crc_unitTest(void)
{
UInt8 b1[] = { 0, 0, 0, 0, 0, 0, 0, 0 };
UInt8 b2[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
UInt8 b3[] = { 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0 };
assert(crc32(0, b1, 8) == 0x6522df69);
assert(crc32(0, b2, 10) == 0x456cd746);
assert(crc32(0, b3, 8) == 0xea8c89c0);
}
Python:
def crc32(crc, p, len):
crc = 0xffffffff & ~crc
for i in range(len):
crc = crc ^ p[i]
for j in range(8):
crc = (crc >> 1) ^ (0xedb88320 & -(crc & 1))
return 0xffffffff & ~crc
def unitTest():
b1 = [ 0, 0, 0, 0, 0, 0, 0, 0 ]
b2 = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]
b3 = [ 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0 ]
assert(crc32(0, b1, 8) == 0x6522df69)
assert(crc32(0, b2, 10) == 0x456cd746)
assert(crc32(0, b3, 8) == 0xea8c89c0)
A little more compact and optimized code
def crc(fileName):
prev = 0
for eachLine in open(fileName,"rb"):
prev = zlib.crc32(eachLine, prev)
return "%X"%(prev & 0xFFFFFFFF)
PS2: Old PS is deprecated - therefore deleted -, because of the suggestion in the comment. Thank you. I don't get, how I missed this, but it was really good.
A modified version of kobor42's answer, with performance improved by a factor 2-3 by reading fixed size chunks instead of "lines":
import zlib
def crc32(fileName):
with open(fileName, 'rb') as fh:
hash = 0
while True:
s = fh.read(65536)
if not s:
break
hash = zlib.crc32(s, hash)
return "%08X" % (hash & 0xFFFFFFFF)
Also includes leading zeroes in the returned string.