The official node.js documentation for Buffer is the best place to check for something like this. As previously noted, Buffer currently supports these encodings: 'ascii', 'utf8', 'utf16le'/'ucs2', 'base64', 'base64url', 'latin1'/'binary', and 'hex'.

The correct answer is:

p.communicate(b"insert into egg values ('egg')")

Note the leading b, telling you that it's a string of bytes, not a string of unicode characters. Also, if you are reading this from a file:

value = open('thefile', 'rt').read()
p.communicate(value)

The change that to:

value = open('thefile', 'rb').read()
p.communicate(value)

Again, note the 'b'. Now if your value is a string you get from an API that only returns strings no matter what, then you need to encode it.

p.communicate(value.encode('latin-1'))

Latin-1, because unlike ASCII it supports all 256 bytes. But that said, having binary data in unicode is asking for trouble. It's better if you can make it binary from the start.

Decode the bytes object to produce a string:

>>> b"abcde".decode("utf-8")
'abcde'

The above example assumes that the bytes object is in UTF-8, because it is a common encoding. However, you should use the encoding your data is actually in!

The most concise I could come up with is the following, which you may be able to make more concise with a few convenience functions (or even replacing/overriding the print function):

# -*- coding=utf-8 -*-
import codecs
import os
import sys

# if you include the -*- coding line, you can use this
output = 'bar' + u'→'
# otherwise, use this
output = 'bar' + b'\xe2\x86\x92'.decode('utf-8')

if sys.stdout.encoding == 'UTF-8':
    print(output)
else:
    output += os.linesep
    if sys.version_info[0] >= 3:
        sys.stdout.buffer.write(bytes(output.encode('utf-8')))
    else:
        codecs.getwriter('utf-8')(sys.stdout).write(output)

The best option is using the -*- encoding line, which allows you to use the actual character in the file. But if for some reason, you can't use the encoding line, it's still possible to accomplish without it.

This (both with and without the encoding line) works on Linux (Arch) with python 2.7.7 and 3.4.1. It also works if the terminal's encoding is not UTF-8. (On Arch Linux, I just change the encoding by using a different LANG environment variable.)

LANG=zh_CN python test.py

It also sort of works on Windows, which I tried with 2.6, 2.7, 3.3, and 3.4. By sort of, I mean I could get the '→' character to display only on a mintty terminal. On a cmd terminal, that character would display as '→'. (There may be something simple I'm missing there.)

I think struct is what you are looking for. Anyhow you have to find out the serialized data format. It is not a C/C++ struct that is sent but (hopefilly) a well described serialized format which may be binaray. Important topics are little/big endianess, size of double, size of int and padding.

from struct import *
arg1, arg2, arg3, arg4, arg5 = unpack('!ddddi', buffer)

standard sizes would require 36 bytes

>>> from struct import *
>>> a='\x00\x00\x00\x01'*9
>>> unpack('>ddddi', a)
(2.1219957915e-314, 2.1219957915e-314, 2.1219957915e-314, 2.1219957915e-314, 1)
>>> unpack('<ddddi', a)
(7.291122046717944e-304, 7.291122046717944e-304, 7.291122046717944e-304, 7.291122046717944e-304, 16777216)
>>> unpack('!ddddi', a)
(2.1219957915e-314, 2.1219957915e-314, 2.1219957915e-314, 2.1219957915e-314, 1)
>>>

EDIT: unpacking to variables, named tuple and class

unpack directly to your variables

from struct import unpack 
b = "\x00\x00\x00\x05\x00\x00\x00\x2a"

x, y = unpack('!ii', b)
print x
print y

unpack to named tuple

from struct import unpack 
from collections import namedtuple
b = "\x00\x00\x00\x05\x00\x00\x00\x2a"

Point = namedtuple('Point', 'x, y')

p = Point._make(unpack("!ii", b))
print p.x
print p.y

remember: tuple attributes are read only

unpack and initalize a class

from struct import unpack 
b = "\x00\x00\x00\x05\x00\x00\x00\x2a"

class Point(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

p = Point(*unpack('!ii', b))
print p.x
print p.y

If you look at the docs for bytes, it points you to bytearray:

bytearray([source[, encoding[, errors]]])

Return a new array of bytes. The bytearray type is a mutable sequence of integers in the range 0 <= x < 256. It has most of the usual methods of mutable sequences, described in Mutable Sequence Types, as well as most methods that the bytes type has, see Bytes and Byte Array Methods.

The optional source parameter can be used to initialize the array in a few different ways:

If it is a string, you must also give the encoding (and optionally, errors) parameters; bytearray() then converts the string to bytes using str.encode().

If it is an integer, the array will have that size and will be initialized with null bytes.

If it is an object conforming to the buffer interface, a read-only buffer of the object will be used to initialize the bytes array.

If it is an iterable, it must be an iterable of integers in the range 0 <= x < 256, which are used as the initial contents of the array.

Without an argument, an array of size 0 is created.

So bytes can do much more than just encode a string. It's Pythonic that it would allow you to call the constructor with any type of source parameter that makes sense.

For encoding a string, I think that some_string.encode(encoding) is more Pythonic than using the constructor, because it is the most self documenting -- "take this string and encode it with this encoding" is clearer than bytes(some_string, encoding) -- there is no explicit verb when you use the constructor.

I checked the Python source. If you pass a unicode string to bytes using CPython, it calls PyUnicode_AsEncodedString, which is the implementation of encode; so you're just skipping a level of indirection if you call encode yourself.

Also, see Serdalis' comment -- unicode_string.encode(encoding) is also more Pythonic because its inverse is byte_string.decode(encoding) and symmetry is nice.

UCS-2 is a character encoding standard in which characters are represented by a fixed-length 16 bits (2 bytes)

With the encoding known you can decode properly and get the data as expected:

const data = [79, 0, 98, 0, 115, 0, 116, 0, 59, 0, 80, 0, 101, 0, 114, 0, 115, 0, 111, 0, 110, 0, 59, 0]
const dataBuffer = Buffer.from(data)
const utf16Decoder = new TextDecoder('UTF-16')

console.log(utf16Decoder.decode(dataBuffer)) // Obst;Person;

I think it's trying to decode as utf8 by default and getting upset at the unrenderable null bytes between every character: