Given a file object, and a number of characters, you can use:

# build a table mapping lead byte to expected follow-byte count
# bytes 00-BF have 0 follow bytes, F5-FF is not legal UTF8
# C0-DF: 1, E0-EF: 2 and F0-F4: 3 follow bytes.
# leave F5-FF set to 0 to minimize reading broken data.
_lead_byte_to_count = []
for i in range(256):
    _lead_byte_to_count.append(
        1 + (i >= 0xe0) + (i >= 0xf0) if 0xbf < i < 0xf5 else 0)

def readUTF8(f, count):
    """Read `count` UTF-8 bytes from file `f`, return as unicode"""
    # Assumes UTF-8 data is valid; leaves it up to the `.decode()` call to validate
    res = []
    while count:
        count -= 1
        lead = f.read(1)
        res.append(lead)
        readcount = _lead_byte_to_count[ord(lead)]
        if readcount:
            res.append(f.read(readcount))
    return (''.join(res)).decode('utf8')

Result of a test:

>>> test = StringIO(u'This is a test containing Unicode data: \ua000'.encode('utf8'))
>>> readUTF8(test, 41)
u'This is a test containing Unicode data: \ua000'

In Python 3, it is of course much, much easier to just wrap the file object in a io.TextIOWrapper() object and leave decoding to the native and efficient Python UTF-8 implementation.

Rather than mess with .encode and .decode, specify the encoding when opening the file. The io module, added in Python 2.6, provides an io.open function, which allows specifying the file's encoding.

Supposing the file is encoded in UTF-8, we can use:

>>> import io
>>> f = io.open("test", mode="r", encoding="utf-8")

Then f.read returns a decoded Unicode object:

>>> f.read()
u'Capit\xe1l\n\n'

In 3.x, the io.open function is an alias for the built-in open function, which supports the encoding argument (it does not in 2.x).

We can also use open from the codecs standard library module:

>>> import codecs
>>> f = codecs.open("test", "r", "utf-8")
>>> f.read()
u'Capit\xe1l\n\n'

Note, however, that this can cause problems when mixing read() and readline().

One way might be to use Hachoir to define a file parsing protocol.

The simple alternative is to open the file in binary mode and manually initialise a buffer and text wrapper around it. You can then switch in and out of binary pretty neatly:

my_file = io.open("myfile.txt", "rb")
my_file_buffer = io.BufferedReader(my_file, buffer_size=1) # Not as performant but a larger buffer will "eat" into the binary data 
my_file_text_reader = io.TextIOWrapper(my_file_buffer, encoding="utf-8")
string_buffer = ""

while True:
    while "near the end" not in string_buffer:
        string_buffer += my_file_text_reader.read(1) # read one Unicode char at a time

    # binary data must be next. Where do we get the binary length from?
    print string_buffer
    data = my_file_buffer.read(3)

    print data
    string_buffer = ""

A quicker, less extensible way might be to use the approach you've suggested in your question by intelligently parsing the text portions, reading each UTF-8 sequence of bytes at a time. The following code (from http://rosettacode.org/wiki/Read_a_file_character_by_character/UTF8#Python), seems to be a neat way to conservatively read UTF-8 bytes into characters from a binary file:

 def get_next_character(f):
     # note: assumes valid utf-8
     c = f.read(1)
     while c:
         while True:
             try:
                 yield c.decode('utf-8')
             except UnicodeDecodeError:
                 # we've encountered a multibyte character
                 # read another byte and try again
                 c += f.read(1)
             else:
                 # c was a valid char, and was yielded, continue
                 c = f.read(1)
                 break

# Usage:
with open("input.txt","rb") as f:
    my_unicode_str = ""
    for c in get_next_character(f):
        my_unicode_str += c

You can always map a str to unicode using the latin-1 codec. Once you have a unicode, you can always encode it in utf-8:

data.decode('latin-1').encode("utf-8")

Quick and dirty and not tested:

# assumption: the file is small enough to fit into the RAM
# and also that 'abst' does not occur in the dataset
for hunk in input.split('abst')[1:]: # skip first hunk, since it is the stuff befor the first 'abst' occurence
    var1 = ord(hunk[0])
    var2 = ord(hunk[1]) + ord(hunk[2])*256 + ord(hunk[3])*256*256
    var3 = hunk[4:].split('\x00')[0]

First two bytes \xff\xfe looks like BOM (Byte Order Mark)
and table at Wikipedia page BOM shows that \xff\xfe can means encoding UTF-16-LE

So you could try

with open('file.sql', 'r', encoding='utf-16-le') as f:

EDIT:

There is module chardet which you may also try to use to detect encoding.

import chardet 

with open('file.sql', 'rb') as f:  # read bytes
     data = f.read()

info = chardet.detect(data)
print(info['encoding'])

text = data.decode(info['encoding'])

Usually files don't have BOM but if they have then you may try to detect it using example from unicodebook.readthedocs.io/guess_encoding/check-for-bom-markers

from codecs import BOM_UTF8, BOM_UTF16_BE, BOM_UTF16_LE, BOM_UTF32_BE, BOM_UTF32_LE

BOMS = (
    (BOM_UTF8, "UTF-8"),
    (BOM_UTF32_BE, "UTF-32-BE"),
    (BOM_UTF32_LE, "UTF-32-LE"),
    (BOM_UTF16_BE, "UTF-16-BE"),
    (BOM_UTF16_LE, "UTF-16-LE"),
)

def check_bom(data):
    return [encoding for bom, encoding in BOMS if data.startswith(bom)]

# ---------

with open('file.sql', 'rb') as f:  # read bytes
     data = f.read()

encoding = check_bom(data)
print(encoding)     

if encoding:
    text = data.decode(encoding[0])
else:
    print('unknown encoding')

The 'b' flag will get python to treat the file as a binary, so no modules are needed. Also you haven't provided a purpose for having python read a binary file with a question like that.

f = open('binaryfile', 'rb')
print(f.read())