The correct answer is:

p.communicate(b"insert into egg values ('egg')")

Note the leading b, telling you that it's a string of bytes, not a string of unicode characters. Also, if you are reading this from a file:

value = open('thefile', 'rt').read()
p.communicate(value)

The change that to:

value = open('thefile', 'rb').read()
p.communicate(value)

Again, note the 'b'. Now if your value is a string you get from an API that only returns strings no matter what, then you need to encode it.

p.communicate(value.encode('latin-1'))

Latin-1, because unlike ASCII it supports all 256 bytes. But that said, having binary data in unicode is asking for trouble. It's better if you can make it binary from the start.

Python 3

From the docs:

Concatenating immutable sequences always results in a new object. This means that building up a sequence by repeated concatenation will have a quadratic runtime cost in the total sequence length. To get a linear runtime cost, you must switch to one of the alternatives below: if concatenating str objects, you can build a list and use str.join() at the end or else write to an io.StringIO instance and retrieve its value when complete

Experiment to compare runtime of several options:

import sys
import timeit
from io import StringIO
from array import array


def test_concat():
    out_str = ''
    for _ in range(loop_count):
        out_str += 'abc'
    return out_str


def test_join_list_loop():
    str_list = []
    for _ in range(loop_count):
        str_list.append('abc')
    return ''.join(str_list)


def test_array():
    char_array = array('b')
    for _ in range(loop_count):
        char_array.frombytes(b'abc')
    return str(char_array.tostring())


def test_string_io():
    file_str = StringIO()
    for _ in range(loop_count):
        file_str.write('abc')
    return file_str.getvalue()


def test_join_list_compr():
    return ''.join(['abc' for _ in range(loop_count)])


def test_join_gen_compr():
    return ''.join('abc' for _ in range(loop_count))


loop_count = 80000

print(sys.version)

res = {}

for k, v in dict(globals()).items():
    if k.startswith('test_'):
        res[k] = timeit.timeit(v, number=10)

for k, v in sorted(res.items(), key=lambda x: x[1]):
    print('{:.5f} {}'.format(v, k))

results

3.7.5 (default, Nov  1 2019, 02:16:32) 
[Clang 11.0.0 (clang-1100.0.33.8)]
0.03738 test_join_list_compr
0.05681 test_join_gen_compr
0.09425 test_string_io
0.09636 test_join_list_loop
0.11976 test_concat
0.19267 test_array

Python 2

Efficient String Concatenation in Python is a rather old article and its main statement that the naive concatenation is far slower than joining is not valid anymore, because this part has been optimized in CPython since then. From the docs:

CPython implementation detail: If s and t are both strings, some Python implementations such as CPython can usually perform an in-place optimization for assignments of the form s = s + t or s += t. When applicable, this optimization makes quadratic run-time much less likely. This optimization is both version and implementation dependent. For performance sensitive code, it is preferable to use the str.join() method which assures consistent linear concatenation performance across versions and implementations.

I've adapted their code a bit and got the following results on my machine:

from cStringIO import StringIO
from UserString import MutableString
from array import array

import sys, timeit

def method1():
    out_str = ''
    for num in xrange(loop_count):
        out_str += `num`
    return out_str

def method2():
    out_str = MutableString()
    for num in xrange(loop_count):
        out_str += `num`
    return out_str

def method3():
    char_array = array('c')
    for num in xrange(loop_count):
        char_array.fromstring(`num`)
    return char_array.tostring()

def method4():
    str_list = []
    for num in xrange(loop_count):
        str_list.append(`num`)
    out_str = ''.join(str_list)
    return out_str

def method5():
    file_str = StringIO()
    for num in xrange(loop_count):
        file_str.write(`num`)
    out_str = file_str.getvalue()
    return out_str

def method6():
    out_str = ''.join([`num` for num in xrange(loop_count)])
    return out_str

def method7():
    out_str = ''.join(`num` for num in xrange(loop_count))
    return out_str


loop_count = 80000

print sys.version

print 'method1=', timeit.timeit(method1, number=10)
print 'method2=', timeit.timeit(method2, number=10)
print 'method3=', timeit.timeit(method3, number=10)
print 'method4=', timeit.timeit(method4, number=10)
print 'method5=', timeit.timeit(method5, number=10)
print 'method6=', timeit.timeit(method6, number=10)
print 'method7=', timeit.timeit(method7, number=10)

Results:

2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
method1= 0.171155929565
method2= 16.7158739567
method3= 0.420584917068
method4= 0.231794118881
method5= 0.323612928391
method6= 0.120429992676
method7= 0.145267963409

Conclusions:

• join still wins over concat, but marginally
• list comprehensions are faster than loops (when building a list)
• joining generators is slower than joining lists
• other methods are of no use (unless you're doing something special)

From a practical standpoint, the difference is that:

x = np.fromstring(s, dtype='int8')

Will make a copy of the string in memory, while:

x = np.frombuffer(s, dtype='int8')

or

x = np.frombuffer(buffer(s), dtype='int8')

Will use the memory buffer of the string directly and won't use any* additional memory. Using frombuffer will also result in a read-only array if the input to buffer is a string, as strings are immutable in python.

(*Neglecting a few bytes of memory used for an additional python ndarray object -- The underlying memory for the data will be shared.)

If you're not familiar with buffer objects (memoryview in python3.x), they're essentially a way for C-level libraries to expose a block of memory for use in python. It's basically a python interface for managed access to raw memory.

If you were working with something that exposed the buffer interface, then you'd probably want to use frombuffer. (Python 2.x strings and python 3.x bytes expose the buffer interface, but you'll get a read-only array, as python strings are immutable.)

Otherwise, use fromstring to create a numpy array from a string. (Unless you know what you're doing, and want to tightly control memory use, etc.)

In Python, you don't have to care about memory management. You don't need to reserve "buffers", just assign the variables you want, and use them.

If you assign too many (for your RAM), you might run into MemoryErrors, indicating you don't have enough memory.

You could use a Queue as some kind of buffer, though, e.g.

import random
from Queue import Queue, Full

q = Queue(100)

for i in range(10):
    #fill queue
    while True:
        try:
            q.put_nowait(random.randint(1,10))
        except Full:
            break
    #take some values from queue
    print "Round", i,
    number_of_values_to_get = random.randint(0,20)
    print "getting %i values." % number_of_values_to_get
    for j in range(number_of_values_to_get):
        value = q.get()
        print "  got value", value

I create a Queue of size 100, i.e., a maximum of 100 entries can be stored in it. If you try to put on a full queue, it will raise the corresponding exception, so you better catch it. (This is only True if you use put_nowait or put with a timeout, just look at the docs for more details.)

In this example, for ten rounds, I fill a "buffer" (queue) of 100 random integers. Then, I pick between 0 and 20 values from this buffer and print them.

I hope this helps. The main use-case for queues is multithreaded program execution, they are thread-safe. So maybe you'll want to fill the queue from one thread and read it from another. If you don't want multithreading, collections.deque might be a faster alternative, but they are not thread-safe.