>>> import struct
>>> struct.pack('f', 3.141592654)
b'\xdb\x0fI@'
>>> struct.unpack('f', b'\xdb\x0fI@')
(3.1415927410125732,)
>>> struct.pack('4f', 1.0, 2.0, 3.0, 4.0)
'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@'
Answer from tzot on Stack Overflow>>> import struct
>>> struct.pack('f', 3.141592654)
b'\xdb\x0fI@'
>>> struct.unpack('f', b'\xdb\x0fI@')
(3.1415927410125732,)
>>> struct.pack('4f', 1.0, 2.0, 3.0, 4.0)
'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@'
Just a little addition, if you want a float number as output from the unpack method instead of a tuple just write
>>> import struct
>>> [x] = struct.unpack('f', b'\xdb\x0fI@')
>>> x
3.1415927410125732
If you have more floats then just write
>>> import struct
>>> [x,y] = struct.unpack('ff', b'\xdb\x0fI@\x0b\x01I4')
>>> x
3.1415927410125732
>>> y
1.8719963179592014e-07
>>>
Confusion about the size of floats in python
python - Converting a float to bytearray - Stack Overflow
casting - How to interpret 4 bytes as a 32-bit float using Python - Stack Overflow
numpy - How to check the size of a float in python? - Stack Overflow
Videos
Hi, not sure if this belongs here or in the main python sub, but decided to try here first. Anyways, I'm working on a project where I have a large text file which essentially contains a roughly 22,000x6,000 matrix of decimal values. This file is >2GB in size, however it's all encoded as text so the size of the data should become considerably smaller once I read them in and represent them as floats (they are mostly a single-digit integer part with 14 digit fractional part, along with the decimal point itself, each character of which UTF-8 can represent as a single byte since it's in the ascii range so each value should use 16 bytes if my calculations are correct). I figure that python represents floats as 8-byte double precision values, as the documentation says CPython will use a C double for your platform to represent them. My googling seems to support this assumption. That would mean that my original matrix in the file should be about 16220006000 ~= 2 GB (not including the whitespace separators), and the matrix in memory as floats should be 8220006000 ~= 1 GB, which while not small, should fit comfortably in memory on my laptop. However, this is not the case and I'm getting an out of memory error as a result.
As a result, I did some investigation and did >>> x = 3.14 >>> sys.getsizeof(x) 24 What the hell, how can this be??? What could CPython possibly be doing with 24 bytes? All I can figure is that there's an 8-byte pointer to a 16-byte double (huge!) and getsizeof() is counting both? Note that this is 64-bit CPython running on linux. I tried again on a 32-bit CPython running on windows and the answer I got was 16 bytes, so I figure this means a 4-byte pointer to an 8-byte double? Somewhat frustrated with this result, I tried again, this time using numpy.float16 because these are relatively small values and I don't need all that space anyways. For the linux 64-bit CPython I got 24 bytes again, and on the Windows 32-bit CPython I got 12 bytes. No idea at all what's happening here because if my previous guesses were correct I should have 8 and 4 byte pointers to 2-byte float16s, giving 10 and 6 bytes, respectively. Clearly I'm missing something here. Can anyone explain what's happening to me, and why so much memory is being used to store these values?
Also, I'm aware that my use case is memory intensive and I'm working on refactoring so that I don't need to have everything in memory at once. However, I'm still extremely curious about this behavior as it seems strange to me. Thanks!
It depends what you want, and what you are going to do with it. If all you want is a bytearray then:
import struct
value = 5.1
ba = bytearray(struct.pack("f", value))
Where ba is a bytearray. However, if you wish to display the hex values (which I suspect), then:
print([ "0x%02x" % b for b in ba ])
This gives (for value 5.1):
['0x33', '0x33', '0xa3', '0x40']
However, CPython uses the C type double to store even small floats (there are good reasons for that), so:
value = 5.1
ba = bytearray(struct.pack("d", value))
print([ "0x%02x" % b for b in ba ])
Gives:
['0x66', '0x66', '0x66', '0x66', '0x66', '0x66', '0x14', '0x40']
The result I would want from 5.1 is 0x40 a3 33 33 or 64 163 51 51. Not as a string.
To get the desired list of integers from the float:
>>> import struct
>>> list(struct.pack("!f", 5.1))
[64, 163, 51, 51]
Or the same as a bytearray type:
>>> bytearray(struct.pack("!f", 5.1))
bytearray(b'@\xa333')
Note: the bytestring (bytes type) contains exactly the same bytes:
>>> struct.pack("!f", 5.1)
b'@\xa333'
>>> for byte in struct.pack("!f", 5.1):
... print(byte)
...
64
163
51
51
The difference is only in mutability. list, bytearray are mutable sequences while bytes type represents an immutable sequence of bytes. Otherwise, bytes and bytearray types have a very similar API.
For detail see Python Struct. For your specific question:
import struct
# if input is string, per @robyschek will fail on python 3
data=b'\x64\xd8\x64\x3f'
print struct.unpack('<f', data) #little endian
print struct.unpack('>f', data) # big endian
#your input
list1=[0x64, 0xD8, 0x6E, 0x3F]
# aa=str(bytearray(list1)) # edit: this conversion wasn't needed
aa= bytearray(list1)
print struct.unpack('<f', aa)
output:
(0.8939268589019775,)
(3.193376169798871e+22,)
(0.9329893589019775,)
If you're willing to use a big library that's really for handling (large) arrays of numbers efficiently:
import numpy as np
data_bytes = np.array([0x64, 0xD8, 0x6E, 0x3F], dtype=np.uint8)
data_as_float = data_bytes.view(dtype=np.float32)
print(data_as_float)
This will also work on big byte arrays; then you get an array of floats.
Properties of a Python float can be requested via sys.float_info. It returns information such as max/min value, max/min exp value, etc. These properties can potentially be used to calculate the byte size of a float. I never encountered anything else than 64 bit, though, on many different architectures.
The items of a NumPy array might have different size, but you can check their size in bytes by a.itemsize, where a is a NumPy array.
numpy.finfo lists sizes and other attributes of float32 ..., including
nexp : number of bits in the exponent including its sign and bias.
nmant : number of bits in the mantissa.
On a machine with IEEE-754
standard floating point,
import numpy as np
for f in (np.float32, np.float64, float):
finfo = np.finfo(f)
print finfo.dtype, finfo.nexp, finfo.nmant
will print e.g.
float32 8 23
float64 11 52
float64 11 52
(Try float16 and float128 too.)
Hi all,
I'm working on an application where I need to send some time data over CAN. I have no issues doing simple bitwise manipulation to get hours and minutes into byte form, but I am having a hard time trying to figure out how to get fractional seconds into byte form. Specifically, I need to send the data with 13bits of precision.
I've seen some documentation on how to do this math wise, but I'm curious if there's a way to do this easily in Python.
Any thoughts?
Using struct
Base on the struct module documentation the struct.unpack() function must have defined exact format for unpacking multiple values at once. So if you need to use struct module, then you either have to define the format using format as '<ff' or iterate over the array using struct.iter_unpack(format, buffer).
Using array
Another option is to use array module, which allows you to decode the binary data at once. You can change endianness using array.byteswap() method. The only restriction is that the whole array must have same type.
import array
arr = array.array('f', b'\x00\x00\x80?\x00\x00\x00@')
arr.tolist()
# [1.0, 2.0]
# change endianness
arr.byteswap()
arr.tolist()
# [4.600602988224807e-41, 8.96831017167883e-44]
Python's strcut provide some essential tooling, but is a bit clumsy to work with. In particular, for this, it'd have to yield all your floats as a tuple of Python floats (which are "enormous" objetcs with 10s of bytes each), and then insert then into an array.
Fortunatelly, Python's array.array class itself can fill in its values given a bytes object, or even read then directly from a file.
import array
data = b'\x00\x00\x80?\x00\x00\x00@'
arr = array.array('f')
arr.frombytes(data)
print(arr)
Yields:
array('f', [1.0, 2.0])
(to read directly from a file, use the fromfile method)
NumPy arrays come with a tobytes method that gives you a dump of their raw data bytes:
arr.tobytes()
You can specify an order argument to use either C-order (row major) or F-order (column major) for multidimensional arrays.
Since you want to dump the bytes to a file, you may also be interested in the tofile method, which dumps the bytes to a file directly:
arr.tofile(your_file)
tofile always uses C-order.
If you need to change endianness, you can use the byteswap method. (newbyteorder has a more convenient signature, but doesn't change the underlying bytes, so it won't affect tobytes.)
import sys
if sys.byteorder=='big':
arr = arr.byteswap()
data_bytes = arr.tobytes()
You could use struct to pack the bytes like,
>>> import struct
>>> struct.pack('<f', 3.14) # little-endian
b'\xc3\xf5H@'
>>> struct.pack('>f', 3.14) # big-endian
b'@H\xf5\xc3'
Finally, I have also found this message by Ehud Karni on the mailing list. Indeed that code also gives you the correct numbers in hexadecimal. I will paste it here (it does not work when directly copy pasting from the mailing list message)
(defvar IEEE-sign-mlt 128 "multiplier for sign") ; 1 bits
(defvar IEEE-exp-bias 127 "exponent bias for float") ; 8 bits
(defvar IEEE-mantissa-divisor ?\x800000 "mantissa divisor for float") ; 23 bits
(defvar IEEE-float-mantissa-divisor (/ 1.0 (float IEEE-mantissa-divisor))
"floated 1 / mantissa divisor")
(defun IEEE-float-2-hex (FNUM)
"Convert a floating point number FNUM to 8 Hex digits string (big endian)"
(interactive "nEnter float to convert: ")
(or (floatp FNUM)
(setq FNUM (float FNUM)))
(let ((IEEE-sign 0)
(IEEE-exp 0)
(IEEE-mantissa 0)
hex)
(and (< FNUM 0) ; negative
(setq IEEE-sign IEEE-sign-mlt) ; yes, sign=1 (* 2**7)
(setq FNUM (- FNUM))) ; negate (abs) it
(cond
((= FNUM 0)) ; Zero - we all set
((= FNUM 1e+INF) ; infinite ?
(setq IEEE-exp IEEE-exp-bias)) ; exp = max, mantissa = 0
(t ;; real float
(setq IEEE-exp (floor (log FNUM 2.0))) ; exponent [ log 2 FNUM ] - integer
(if (<= IEEE-exp (- IEEE-exp-bias)) ; check for De-normalized number
(setq IEEE-exp 0 ; really -
IEEE-exp-bias
IEEE-mantissa (truncate (* IEEE-mantissa-divisor
0.5 FNUM (expt 2.0 IEEE-exp-bias))))
;; normal float
(setq IEEE-mantissa (truncate (* IEEE-mantissa-divisor
(- (/ FNUM (expt 2.0 IEEE-exp)) 1))))
(setq IEEE-exp (+ IEEE-exp IEEE-exp-bias))) ;; offset-ed exp
))
(setq hex (format "%02X%06X" (+ IEEE-sign (/ IEEE-exp 2))
(+ IEEE-mantissa
(* (% IEEE-exp 2)
IEEE-mantissa-divisor))))
;; (message "Float=%f sign=%d, exp=%d, mant=%d HEX=%s"
;; FNUM IEEE-sign IEEE-exp IEEE-mantissa hex)
(print hex)))
(defun IEEE-hex-2-float (HEX)
"Convert an 8 Hex digits string (big endian) to floating point number"
(interactive "sEnter hex value (8 hex digits): ")
(if (or (not (stringp HEX))
(/= (length HEX) 8)
(string-match "[^0-9a-fA-F]" HEX))
(error "Arg must be a string of EXACTLY 8 hex (1-8, a-f, A-F)
digits"))
(let ((S-EXP (string-to-number (substring HEX 0 2) 16))
(MANTISSA (string-to-number (substring HEX 2) 16))
(RSLT 1)
IEEE-sign
IEEE-exp
IEEE-mantissa
)
(setq IEEE-mantissa (logior MANTISSA IEEE-mantissa-divisor)) ; always set upper bit
(if (= IEEE-mantissa IEEE-mantissa-divisor) ; = --> zero mantissa, check special cases
(cond
((string-equal HEX "00000000")
(setq RSLT (float 0))) ; Zero
((string-equal HEX "3F800000")
(setq RSLT 1e+INF)) ; + infinity
((string-equal HEX "BF800000")
(setq RSLT -1e+INF)) ; - infinity
))
(print (if (/= RSLT 1)
RSLT ; special cases (0, infinity)
(setq IEEE-sign (ash S-EXP -7)) ; shift right 7 bits
(setq IEEE-exp (+ (ash (% S-EXP IEEE-sign-mlt) 1) ; shift (part) exp 1
(ash MANTISSA -23))) ; 1 bit from mantissa
(if (= IEEE-exp 0) ; De-normalized ?
(setq IEEE-mantissa (ash
(logxor IEEE-mantissa IEEE-mantissa-divisor) 1))) ; clear upper bit
;; (message "Hex=%s sign=%d, exp=%d, mant=%d"
;; HEX IEEE-sign IEEE-exp IEEE-mantissa)
(* ; result float
(- 1 IEEE-sign IEEE-sign) ; sign
(expt 2.0 (- IEEE-exp IEEE-exp-bias)) ; 2 ** exponent (un-biased)
(float IEEE-mantissa) ; mantissa part
IEEE-float-mantissa-divisor ; 1 / mantissa divisor
)))))
You will get the bytes in hexadecimal representation as a string. Unfortunately I don't know how to 'convert' those strings to hexadecimal numbers (starting with #x), but if you find a way to convert the hexadecimal strings to numbers, then you could use this solution also.
You can't. Emacs Lisp is a highly abstract, strongly typed dynamic language where everything is a reference. It determines the data type through the highest bits of pointers in C language. If you want to treat a floating point number as an integer, you have to modify some bits of the pointer. This can't be done in Emacs Lisp; you must modify it through a debugger like GDB.
But you can write a function to find out the precise representation of a floating point number. As per the elisp manual, Emacs Lisp follows IEEE-754.