Your ''.join() expression is filtering, removing anything non-ASCII; you could use a conditional expression instead:

return ''.join([i if ord(i) < 128 else ' ' for i in text])

This handles characters one by one and would still use one space per character replaced.

Your regular expression should just replace consecutive non-ASCII characters with a space:

re.sub(r'[^\x00-\x7F]+',' ', text)

Note the + there.

In order to replace the content of string using str.replace() method; you need to firstly decode the string, then replace the text and encode it back to the original text:

>>> a = "hi »"
>>> a.decode('utf-8').replace("»".decode('utf-8'), "").encode('utf-8')
'hi '

You may also use the following regex to remove all the non-ascii characters from the string:

>>> import re
>>> re.sub(r'[^\x00-\x7f]',r'', 'hi »')
'hi '

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')

You need to decode the string

# -*- coding: utf-8 -*- 
clean = "you’ll".decode('utf-8')
clean = clean.replace('’'.decode('utf-8'),' ')
print clean

This prints

you ll

which is as expected

At this point w has a byte with value 160 in it, but it's encoding is 'ascii'.

You have an unicode string:

>>> w
u'\xa0 foo bar'
>>> type(w)
<type 'unicode'>

How do replace all of the \xa0 bytes with another character?

>>> x = w.replace(u'\xa0', ' ')
>>> x
u'  foo bar'

And why does BS return an 'ascii' encoded string with an invalid character in it?

As mentioned above, it is not an ascii encoded string, but an Unicode string instance.

Is there a way to convert w to a 'latin1` encoded string?

Sure:

>>> w.encode('latin1')
'\xa0 foo bar'

(Note this last string is an encoded string, not an unicode object, and its representation is not prefixed by 'u' like the previous unicode objects).

Notes (edited):

• If you are typing strings into your source files, note that encoding of source files matters. Python will assume your source files are ASCII. The command line interpreter, on the other hand, will assume you are entering strings in your default system encoding. Of course you can override all this.
• Avoid latin1, use UTF-8 if possible: ie. w.encode('utf8')
• When encoding and decoding can tell Python to ignore errors, or replace characters that cannot be encoded with some marker character . I don't recommend to ignore encoding errors (at least without logging them), except for the hopefully rare cases when you know there are encoding errors or you need to encode text into a more reduced character set, requiring replacement of the code points that cannot be represented (ie if you need to encode 'España' into ASCII, you definitely should replace the 'ñ'). But for these cases there are imho better alternatives and you should look into the magical unicodedata module (see https://stackoverflow.com/a/1207479/401656).
• There is a Python Unicode HOWTO: https://docs.python.org/2/howto/unicode.html

If all you want to do is degrade accented characters to their non-accented equivalent:

>>> import unicodedata
>>> unicodedata.normalize('NFKD', u"m\u00fasica").encode('ascii', 'ignore')
'musica'

>>> mylist = ["apple", "samsung", "toshiba", "Don’t know", "Can’t recall"]
>>> [item.replace('\xe2\x80\x99',"'") for item in mylist]
['apple', 'samsung', 'toshiba', "Don't know", "Can't recall"]

If all the items are already unicode:

>>> mylist = [u"apple", u"samsung", u"toshiba", u"Don’t know", u"Can’t recall"]
>>> [item.replace(u'’',u"'") for item in mylist]
[u'apple', u'samsung', u'toshiba', u"Don't know", u"Can't recall"]

You've got an encoding problem. Instead of trying to remove this characters, look for the encoding of the page, then when you read the file, use the codecs module instead of open(), using the proper character encoding.