You can use Python's struct module to convert the byte string to integers. It takes care of endianness and sign extension for you. I guess you are trying to interpret this 16-byte string as 8 2-byte signed integers, in big-endian byte order. The format string for this is '>8h. The > character tells Python to interpret the string as big endian, 8 means 8 of the following data type, and h means signed short integers.
import struct
nums = struct.unpack('>8h', bin_bytes)
Now nums is a tuple of integers that you can process further.
I'm not quite sure if your data is little or big endian. If it is little-endian, you can use < to indicate that in the struct.unpack format string.
python - How to convert bytes byte by byte to signed int - Stack Overflow
language design - Should bytes be signed? - Programming Language Design and Implementation Stack Exchange
Python - convert signed int to bytes - Stack Overflow
python - Using signed Bytes with GDAL - Geographic Information Systems Stack Exchange
However this does not work as the
for j in bytes()converts a bytes element directly into an intj.
As you've noticed, the bytes is already an iterable over integer values. If we have a lot of data, it would be more memory efficient to keep the bytes as is, and convert values on demand. We can simply do the math to convert the unsigned interpretation of a byte to the corresponding signed representation:
def signed_byte(b):
return b - 256 if b >= 128 else b
And wrap the indexing process:
def get_signed(data, index):
return signed_byte(data[index])
If we want or need to do all the conversion ahead of time, that feeds directly into the list comprehension:
ints = [signed_byte(b) for b in b'\xff\xff\x01']
On the other hand, we can reframe the question: to get a bytes object from the original data as a subsequence (like how it worked in 2.x), we can use a 1-element slice instead; or we can wrap the int value into a new bytes. The former will be tricky to adapt to the original code, but the latter is trivial:
ints = [int.from_bytes(bytes([b]), byteorder='little', signed=True) for b in b'\xff\xff\x01']
Another way, using Numpy:
>>> import numpy as np
>>> np.frombuffer(b'\xff\xff\x01', dtype=np.int8)
array([-1, -1, 1], dtype=int8)
Frame Challenge: Bytes should not be integers.
Bytes are bytes, a fixed-size sequence of bits, typically 8 on modern computers.
Should an array of bits be signed or unsigned? Neither, the question is nonsensical: an array cannot be signed or unsigned in the first place.
There's a long-standing tradition in programming languages to conflate integers and array of bits, which can be seen in that bit-wise operations are typically defined on integers. I argue that this "blending" of the two roles is a mistake: a violation of the Single Responsibility Principle.
Yes, it's all registers at the end... should a pointer be signed or unsigned? Surely nowadays the question seems nonsensical, even though in B pointers and integers were the same type, and in CPU they are manipulated through the same registers.
Just like pointers are different from integers, I hereby argue that array of bits should be different from integers, and bit-wise functions are only sensible on array of bits.
The conversion between integer and array of bits (of the same number of bits) should exist -- just like that between floating point or decimal an array of bits -- and it should ideally boil down to a no-op, but from a type system, the two should be separate.
Bytes should be unsigned
There are a few reasons most languages have unsigned bytes by default:
- A more useful range. Bytes can only represent 256 different numbers so using them effectively is extra important. It is more common to want to represent a bigger number like 200 than -1, so the 0-255 is more practical than -128-127.
This isn't really an issue for bigger numbers that usually default to signed. Even an signed short can go up to more than 30,000 which is not something you would typically count to.
- Bytes are most often used as a sequence, rather than as an actual number. You rarely use a type like a byte to store numbers intended as numbers, more often they represent bits of binary data, characters etc.
The "numeric properties" only get in the way of accessing the raw data. Adding a special meaning to the upper half of the byte space would just add extra complexity to any serialization, parsing, string manipulation, etc. code that may want to operate on byte sequences.
error messgae is clear , if your vaue includes signs you need to pass signed =True when you convert it to bytes:
an_int = -5
a_bytes_big = an_int.to_bytes(2, 'big', signed = True)
print(a_bytes_big)
The method to_bytes takes a third parameter: signed:
So you can modify your code to this:
an_int = -5
a_bytes_big = an_int.to_bytes(2, 'big', signed=True)
# or
a_bytes_big = an_int.to_bytes(2, 'big', True)
If you want to use signed 8bit integers for GeoTIFFs use datatype Byte together with creation options PIXELTYPE=SIGNEDBYTE
https://gdal.org/drivers/raster/gtiff.html
gdal_translate -ot Byte -a_nodata -1 -co PIXELTYPE=SIGNEDBYTE input.tif output.tif
This is an interesting problem. To my knowledge, GDAL does not support interpretation of signed 8-bit data. Even raster formats that can handle signed bytes SDAT, for instance are coerced to handle GDT_Byte I think...
The obvious solution to your underlying problem is to use a nodata value like 255 if you really care about the file storage size. Otherwise use a signed 16-bit image and some compression if the format supports it. You may find it plenty suitable.
Subtract 256 if over 127:
unsigned = ord(character)
signed = unsigned - 256 if unsigned > 127 else unsigned
Alternatively, repack the byte with the struct module:
from struct import pack, unpack
signed = unpack('B', pack('b', unsigned))[0]
or directly from the character:
signed = unpack('B', character)[0]
from ctypes import c_int8
value = c_int8(191).value
use ctypes with your ord() value - should be -65 in this case
ex. from string data
from ctypes import c_int8
data ='BF'
value1 = int(data, 16) # or ord(data.decode('hex'))
value2 = c_int8(value1).value
value1 is 16bit integer representation of hex 'BF' and value2 is 8bit representation
-2 is not correct for the values you have specified, and byte order matters. struct uses > for big-endian (most-significant byte first) and < for little-endian (least-significant byte first):
>>> import struct
>>> struct.pack('>h',-2)
'\xff\xfe'
>>> struct.pack('<h',-2)
'\xfe\xff'
>>> p1=chr(254) # 0xFE
>>> p0=chr(255) # 0xFF
>>> struct.unpack('<h',p1+p0)[0]
-2
>>> struct.unpack('>h',p0+p1)[0]
-2
Generally, when using struct, your format string should start with one of the alignment specifiers. The default, native one differs from machine to machine.
Therefore, the correct result is
>>> struct.unpack('!h',p0+p1)[0]
-20761
The representation of -2 in big endian is:
1111 1111 1111 1110 # binary
255 254 # decimal bytes
f f f e # hexadecimal bytes
You can easily verify that by adding two, which results in 0.
Use the struct module.
import struct
value = struct.unpack('B', data[0:1])[0]
We have to specify a range of 1 (0:1), because Python 3 converts automatically to an integer otherwise.
Note that unpack always returns a tuple, even if you're only unpacking one item.
Also, have a look at this SO question.
bytes/bytearray is a sequence of integers. If you just access an element by its index you'll have an integer:
>>> b'abc'
b'abc'
>>> _[0]
97
By their very definition, bytes and bytearrays contain integers in the range(0, 256). So they're "unsigned 8-bit integers".
Use a different operation for bit flipping.
E.g.:1
array[i] = 255 - array[i]
or also:
array[i] = 255 ^ array[i]
will flip all (i.e.: 8) bits.
1 the math behind this can be worked out from two's complement wikipedia page.
The solution is actually remarkably simple after playing around with a binary calculator a little bit.
Just subtract the magnitude of the SIGNED int from 256, to get the value of the UNSIGNED int with the same binary representation.
So,
-23 signed would be 233 unsigned.
Hope this helps anyone else looking for a solution :)
EDIT: For those saying answer is 255 - array[0]. In this case I'm looking for a way to go from post NOT'd int to its unsigned counter part. So I've already performed the bitwise NOT on the integer, now I'm just getting it back to a form that can be inputted into the byte-array.
So in the end it would look something like this:
tmp = ~array[0]
array[0] = 256 + tmp
or
array[0] = 256 - abs(tmp)
This gets me the correct answer :)
Assuming you're on at least 3.2, there's a built in for this:
int.from_bytes(bytes,byteorder, *,signed=False)...
The argument
bytesmust either be a bytes-like object or an iterable producing bytes.The
byteorderargument determines the byte order used to represent the integer. Ifbyteorderis"big", the most significant byte is at the beginning of the byte array. Ifbyteorderis"little", the most significant byte is at the end of the byte array. To request the native byte order of the host system, usesys.byteorderas the byte order value.The
signedargument indicates whether two’s complement is used to represent the integer.
## Examples:
int.from_bytes(b'\x00\x01', "big") # 1
int.from_bytes(b'\x00\x01', "little") # 256
int.from_bytes(b'\x00\x10', byteorder='little') # 4096
int.from_bytes(b'\xfc\x00', byteorder='big', signed=True) #-1024
Lists of bytes are subscriptable (at least in Python 3.6). This way you can retrieve the decimal value of each byte individually.
>>> intlist = [64, 4, 26, 163, 255]
>>> bytelist = bytes(intlist) # b'@\x04\x1a\xa3\xff'
>>> for b in bytelist:
... print(b) # 64 4 26 163 255
>>> [b for b in bytelist] # [64, 4, 26, 163, 255]
>>> bytelist[2] # 26