There can be a significant performance difference, of an order of magnitude for multiplications and multiple orders of magnitude for indexing a few random values.
I was actually wondering about the same thing and came across this interesting comparison: http://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/
Answer from Mark on Stack OverflowWhy would anyone use Pandas when there's NumPy? What can Pandas do that NumPy can't do? NumPy is faster. Is the only reason to use Pandas that it may be easier to code for certain tasks? If you care more about code speed than coding ease, isn't NumPay always the better option?
FYI, my use case is generating complex strings of text from data.
python - Is there a performance difference between Numpy and Pandas? - Stack Overflow
When to use pandas series, numpy ndarrays or simply python dictionaries? - Stack Overflow
Python, numpy and pandas.
Codecademy Python 2 is free and pretty good
More on reddit.comWhat are the advantages of Pandas dataframe VS Numpy arrays?
What is Python?
What is a Python Library?
Which One Comes Out Ahead?
There can be a significant performance difference, of an order of magnitude for multiplications and multiple orders of magnitude for indexing a few random values.
I was actually wondering about the same thing and came across this interesting comparison: http://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/
In my experiments on large numeric data, Pandas is consistently 20 TIMES SLOWER than Numpy. This is a huge difference, given that only simple arithmetic operations were performed: slicing of a column, mean(), searchsorted() - see below. Initially, I thought Pandas was based on numpy, or at least its implementation was C optimized just like numpy's. These assumptions turn out to be false, though, given the huge performance gap.
In examples below, data is a pandas frame with 8M rows and 3 columns (int32, float32, float32), without NaN values, column #0 (time) is sorted. data_np was created as data.values.astype('float32'). Results on Python 3.8, Ubuntu:
A. Column slices and mean():
# Pandas
%%timeit
x = data.x
for k in range(100): x[100000:100001+k*100].mean()
15.8 ms ± 101 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# Numpy
%%timeit
for k in range(100): data_np[100000:100001+k*100,1].mean()
874 µs ± 4.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Pandas is 18 times slower than Numpy (15.8ms vs 0.874 ms).
B. Search in a sorted column:
# Pandas
%timeit data.time.searchsorted(1492474643)
20.4 µs ± 920 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# Numpy
%timeit data_np[0].searchsorted(1492474643)
1.03 µs ± 3.55 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Pandas is 20 times slower than Numpy (20.4µs vs 1.03µs).
EDIT: I implemented a namedarray class that bridges the gap between Pandas and Numpy in that it is based on Numpy's ndarray class and hence performs better than Pandas (typically ~7x faster) and is fully compatible with Numpy'a API and all its operators; but at the same time it keeps column names similar to Pandas' DataFrame, so that manipulating on individual columns is easier. This is a prototype implementation. Unlike Pandas, namedarray does not allow for different data types for columns. The code can be found here: https://github.com/mwojnars/nifty/blob/master/math.py (search "namedarray").
The rule of thumb that I usually apply: use the simplest data structure that still satisfies your needs. If we rank the data structures from most simple to least simple, it usually ends up like this:
- Dictionaries / lists
- Numpy arrays
- Pandas series / dataframes
So first consider dictionaries / lists. If these allow you to do all data operations that you need, then all is fine. If not, start considering numpy arrays. Some typical reasons for moving to numpy arrays are:
- Your data is 2-dimensional (or higher). Although nested dictionaries/lists can be used to represent multi-dimensional data, in most situations numpy arrays will be more efficient.
- You have to perform a bunch of numerical calculations. As already pointed out by zhqiat, numpy will give a significant speed-up in this case. Furthermore numpy arrays come bundled with a large amount of mathematical functions.
Then there are also some typical reasons for going beyond numpy arrays and to the more-complex but also more-powerful pandas series/dataframes:
- You have to merge multiple data sets with each other, or do reshaping/reordering of your data. This diagram gives a nice overview of all the 'data wrangling' operations that pandas allows you to do.
- You have to import data from or export data to a specific file format like Excel, HDF5 or SQL. Pandas comes with convenient import/export functions for this.
If you want to an answer which tells you to stick with just one type of data structures, here goes one: use pandas series/dataframe structures.
The pandas series object can be seen as an enhanced numpy 1D array and the pandas dataframe can be seen as an enhanced numpy 2D array. The main difference is that pandas series and pandas dataframes has explicit index, while numpy arrays has implicit indexation. So, in any python code that you think to use something like
import numpy as np
a = np.array([1,2,3])
you can just use
import pandas as pd
a = pd.Series([1,2,3])
All the functions and methods from numpy arrays will work with pandas series. In analogy, the same can be done with dataframes and numpy 2D arrays.
A further question you might have can be about the performance differences between a numpy array and pandas series. Here is a post that shows the differences in performance using these two tools: performance of pandas series vs numpy arrays.
Please note that even in an explicit way pandas series has a subtle worse in performance when compared to numpy, you can solve this by just calling the values method on a pandas series:
a.values
The result of apply the values method on a pandas series will be a numpy array!