You rarely need loops for vector operations in numpy. You can create an uninitialized array and assign to all entries at once:
>>> a = numpy.empty((3,3,))
>>> a[:] = numpy.nan
>>> a
array([[ NaN, NaN, NaN],
[ NaN, NaN, NaN],
[ NaN, NaN, NaN]])
I have timed the alternatives a[:] = numpy.nan here and a.fill(numpy.nan) as posted by Blaenk:
$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a.fill(np.nan)"
10000 loops, best of 3: 54.3 usec per loop
$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a[:] = np.nan"
10000 loops, best of 3: 88.8 usec per loop
The timings show a preference for ndarray.fill(..) as the faster alternative. OTOH, I like numpy's convenience implementation where you can assign values to whole slices at the time, the code's intention is very clear.
Note that ndarray.fill performs its operation in-place, so numpy.empty((3,3,)).fill(numpy.nan) will instead return None.
numpy - How to create an array of NA or Null values in Python? - Stack Overflow
How should I initialize a numpy array of NaN values?
Setting random NaN in multidimensional numpy array
numpy - What is equivalent of Matlab nan( ) in python? - Stack Overflow
You rarely need loops for vector operations in numpy. You can create an uninitialized array and assign to all entries at once:
>>> a = numpy.empty((3,3,))
>>> a[:] = numpy.nan
>>> a
array([[ NaN, NaN, NaN],
[ NaN, NaN, NaN],
[ NaN, NaN, NaN]])
I have timed the alternatives a[:] = numpy.nan here and a.fill(numpy.nan) as posted by Blaenk:
$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a.fill(np.nan)"
10000 loops, best of 3: 54.3 usec per loop
$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a[:] = np.nan"
10000 loops, best of 3: 88.8 usec per loop
The timings show a preference for ndarray.fill(..) as the faster alternative. OTOH, I like numpy's convenience implementation where you can assign values to whole slices at the time, the code's intention is very clear.
Note that ndarray.fill performs its operation in-place, so numpy.empty((3,3,)).fill(numpy.nan) will instead return None.
Another option is to use numpy.full, an option available in NumPy 1.8+
a = np.full([height, width, 9], np.nan)
This is pretty flexible and you can fill it with any other number that you want.
you could use either None or np.nan to create an array of just missing values in Python like so:
np.full(shape=5, fill_value=None)
np.full(shape=5, fill_value=np.nan)
back to your example, this works just fine:
import numpy as np
import pandas as pd
unknowns = np.full(shape=5, fill_value=None)
categories = np.random.choice(['web', 'software', 'hardware', 'biotech'], size = 15, replace = True)
categories = np.concatenate([categories, unknowns])
example = pd.DataFrame(data = {'categories': categories})
example['transformed'] = [ x if pd.isna(x) == False else 'unknown' for x in example['categories']]
print(example['transformed'].value_counts())
Lastly, this line is inefficient.
example['transformed'] = [ x if pd.isna(x) == False else 'unknown' for x in example['categories']]
You do want to avoid loops & list comprehensions when using pandas
on large data, this is going to run much faster:
example['transformed'] = example.categories.apply(lambda s: s if s else 'unknown')
There is a typing problem here.
If you're working in numpy, vectors are typed after being initialized. Assigning a np.nan value to a vector initialized with strings will try to coalesce back into a string:
import numpy as np
v1 = np.array(['a', 'b', 'c'])
v1[0] = np.nan
# v1 = array(['n', 'b', 'c'], dtype='<U1')
v2 = np.array(['ab', 'cd', 'ef'])
v2[0] = np.nan
# v2 = array(['na', 'cd', 'ef'], dtype='<U2')
v3 = np.array(['abc', 'def', 'ghi'])
v3[0] = np.nan
# v3 = array(['nan', 'def', 'ghi'], dtype='<U3')
However, if you're working with pandas in the second half of the question, there's a separate way for handling missing data:
import pandas as pd
df = pd.DataFrame({"x": [pd.NA, "Hello", "World"]})
How should I initialize a numpy array of NaN values?
Below is what I tried in the terminal:
>>> import numpy as np >>> x = np.array([0,0,0]) >>> x array([0, 0, 0]) >>> x.fill(np.nan) >>> x array([-9223372036854775808, -9223372036854775808, -9223372036854775808]) >>>
You can use np.full() to create a multi-dimensional array pre-populated with the same values:
np.full((1, 5), np.nan)
which produces:
array([[nan, nan, nan, nan, nan]])
You can use numpy.zeros(N) + numpy.nan, where N is the number of NaN you want in your array.
import numpy as np
N = 6
nan_array = np.zeros(N) + np.nan
Will produce the following array -
[array([nan, nan, nan, nan, nan, nan])]