Neither is better than the other, they do exactly the same thing. However, using .encode() and .decode() is the more common way to do it. It is also compatible with Python 2.

bytes

You are working with bytes which is very cumbersome.

• You have to prefix all literals
• single bytes have type int and need to be converted to to bytes again At least I cannot do this without debugging some errors.

It is much more convenient to work with strings. So I suggest to decode the bytes to a string with an 8-bit codec like 'latin-1'. No more bytes([x]) or x.to_bytes(). You can use plain comprehension, plain ''.join(), etc. If required you encode the string to bytes again with the very same codec.

dict.get()

For conditional translations there is a nice trick - dict.get() - which allows to pass a default value for non-existing keys.

translated = replacements.get(char, char)

This returns replacements[char] if char is in replacements.keys(). Otherwise it returns the default char. To use this functionality you include the escape character in the translation table (however you fill it).

replacements = {escape_char: escape_char + escape_char,
       '\n':   escape_char + '\x00',
       '\xfd': escape_char + '\x01',
       '\xfe': escape_char + '\x02',
       '\xff': escape_char + '\x03',
       }

The encoding function now reads

def encode(data: bytes) -> bytes:
    str_data = data.decode(encoding='latin-1')
    str_enc = ''.join(replacements.get(x, x) for x in str_data)
    return str_enc.encode(encoding='latin-1')

loop like a pro

In your decoding function you loop over len(data) which is an anti-pattern. You should always loop over the elements. To make it worse you use a while loop and increment the loop counter manually. You also access elements with index i+1 which again is error prone. Loop over elements only and keep a little state like a previous_byte. You cannot be out of bounds.

def decode(data: bytes) -> bytes:
    d_data = data.decode(encoding='latin-1')
    l = []
    esc = ''
    for x in d_data:
        if not esc and x == escape_char:
            esc = x
            continue
        l.append(reverse_replacements.get(esc + x, esc + x))
        esc = ''
    assert esc == ''
    return ''.join(l).encode(encoding='latin-1)')

testing

For your nicely testable functions you should provide some tests. There are unit test frameworks available. You can also do the most primitive assertions, of course in a main guard.

if __name__ == '__main__':
    assert encode(b"asdf") == b"asdf"
    assert decode(b"asdf") == b"asdf"

    assert encode(b"as\ndf") == b"as=\x00df"
    assert decode(b"as=\x00df") == b"as\ndf"
    assert encode(b"\ndf") == b"=\x00df"
    assert decode(b"=\x00df") == b"\ndf"
    assert encode(b"as\n") == b"as=\x00"
    assert decode(b"as=\x00") == b"as\n"

    assert encode(b"as\xfddf") == b"as=\x01df"
    assert decode(b"as=\x01df") == b"as\xfddf"

    assert encode(b"as\xfedf") == b"as=\x02df"
    assert decode(b"as=\x02df") == b"as\xfedf"

    assert encode(b"as\xffdf") == b"as=\x03df"
    assert decode(b"as=\x03df") == b"as\xffdf"