Today I re-learned: Python function default arguments are retained between executions
python - "Least Astonishment" and the Mutable Default Argument - Stack Overflow
What the default arguments are?
Python function with default list argument - Stack Overflow
Actually, this is not a design flaw, and it is not because of internals or performance. It comes simply from the fact that functions in Python are first-class objects, and not only a piece of code.
As soon as you think of it this way, then it completely makes sense: a function is an object being evaluated on its definition; default parameters are kind of "member data" and therefore their state may change from one call to the other - exactly as in any other object.
In any case, the Effbot (Fredrik Lundh) has a very nice explanation of the reasons for this behavior in Default Parameter Values in Python. I found it very clear, and I really suggest reading it for a better knowledge of how function objects work.
Suppose you have the following code
fruits = ("apples", "bananas", "loganberries")
def eat(food=fruits):
...
When I see the declaration of eat, the least astonishing thing is to think that if the first parameter is not given, that it will be equal to the tuple ("apples", "bananas", "loganberries")
However, suppose later on in the code, I do something like
def some_random_function():
global fruits
fruits = ("blueberries", "mangos")
then if default parameters were bound at function execution rather than function declaration, I would be astonished (in a very bad way) to discover that fruits had been changed. This would be more astonishing IMO than discovering that your foo function above was mutating the list.
The real problem lies with mutable variables, and all languages have this problem to some extent. Here's a question: suppose in Java I have the following code:
StringBuffer s = new StringBuffer("Hello World!");
Map<StringBuffer,Integer> counts = new HashMap<StringBuffer,Integer>();
counts.put(s, 5);
s.append("!!!!");
System.out.println( counts.get(s) ); // does this work?
Now, does my map use the value of the StringBuffer key when it was placed into the map, or does it store the key by reference? Either way, someone is astonished; either the person who tried to get the object out of the Map using a value identical to the one they put it in with, or the person who can't seem to retrieve their object even though the key they're using is literally the same object that was used to put it into the map (this is actually why Python doesn't allow its mutable built-in data types to be used as dictionary keys).
Your example is a good one of a case where Python newcomers will be surprised and bitten. But I'd argue that if we "fixed" this, then that would only create a different situation where they'd be bitten instead, and that one would be even less intuitive. Moreover, this is always the case when dealing with mutable variables; you always run into cases where someone could intuitively expect one or the opposite behavior depending on what code they're writing.
I personally like Python's current approach: default function arguments are evaluated when the function is defined and that object is always the default. I suppose they could special-case using an empty list, but that kind of special casing would cause even more astonishment, not to mention be backwards incompatible.
Although an answer has been accepted, it is interesting as well as important to understand why this happens, so as to avoid not-so-obvious pitfalls.
An excerpt from the Python documentation for compound statements :
Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use None as the default, and explicitly test for it in the body of the function
Please refer to the StackOverflow Discussion here for a discussion related to mutable arguments within the Python language. The discussion points to a very interesting and informative article on Effbot - Python Default Values which gives a good explanation of why this behavior is observed and the places where such a behavior is desirable - for e.g. a calculator function that performs very computing-intensive calculations may use a dictionary mutable as a default parameter to store the results of a calculation keyed by the parameters used for the calculation. In such a case, when the client requests for a calculation to be performed, the function can look up the dictionary mutable and return the values if already present, else perform the calculation.
Hopefully, this answer provides an insight into this "astonising" behavior of python and help in designing functions that work correctly and are performant.
Don't use mutable as keyword arguments.
def test_func( newList, myList=None ):
myList = [] if myList is None else myList