pandas's DataFrame is a high level tool while structured arrays are a very low-level tool, enabling you to interpret a binary blob of data as a table-like structure. One thing that is hard to do in pandas is nested data types with the same semantics as structured arrays, though this can be imitated with hierarchical indexing (structured arrays can't do most things you can do with hierarchical indexing).
Structured arrays are also amenable to working with massive tabular data sets loaded via memory maps (np.memmap). This is a limitation that will be addressed in pandas eventually, though.
pandas's DataFrame is a high level tool while structured arrays are a very low-level tool, enabling you to interpret a binary blob of data as a table-like structure. One thing that is hard to do in pandas is nested data types with the same semantics as structured arrays, though this can be imitated with hierarchical indexing (structured arrays can't do most things you can do with hierarchical indexing).
Structured arrays are also amenable to working with massive tabular data sets loaded via memory maps (np.memmap). This is a limitation that will be addressed in pandas eventually, though.
I'm currently in the middle of transition to Pandas DataFrames from the various Numpy arrays. This has been relatively painless since Pandas, AFAIK, is built largely on top of Numpy. What I mean by that is that .mean(), .sum() etc all work as you would hope. On top of that, the ability to add a hierarchical index and use the .ix[] (index) attribute and .xs() (cross-section) method to pull out arbitray pieces of the data has greatly improved the readability and performance of my code (mainly by reducing the number of round-trips to my database).
One thing I haven't fully investigated yet is Pandas compatibility with the more advanced functionality of Scipy and Matplotlib. However, in case of any issues, it's easy enough to pull out a single column that behaves enough like an array for those libraries to work, or even convert to an array on the fly. A DataFrame's plotting methods, for instance, rely on matplotlib and take care of any conversion for you.
Also, if you're like me and your main use of Scipy is the statistics module, pystatsmodels is quickly maturing and relies heavily on pandas.
That's my two cents' worth
What are the advantages of Pandas dataframe VS Numpy arrays?
python - Convert structured numpy array (containing sub-arrays) to pandas dataframe - Stack Overflow
Numpy array vs Pandas Dataframe
What are the differences between Python Array, Numpy Array and Panda Dataframe? When do I use which?
With the given example, you could let
df = pd.DataFrame(test).set_index('f0')
With that, you can access, say, the row whose index is 45 through df.loc[45].
Assuming you've done import pandas as pd already:
df = pd.DataFrame(test) # converts your array to a DataFrame.
df = df.set_index('f0') # changes the index to be the first column.
I just need some good reasons to be able to differentiate them from each other in terms of pros and cons. Thank!
This can be flatten with two pd.DataFrame call
df=pd.DataFrame(arr.tolist())
df=df.join(pd.DataFrame(df[1].tolist()).add_prefix('B'))
Out[404]:
0 1 2 B0 B1 B2
0 1 [5.0, 3.0, 7.0] 6 5.0 3.0 7.0
1 2 [2.0, 1.0, 3.0] 9 2.0 1.0 3.0
2 3 [3.0, 8.0, 4.0] 3 3.0 8.0 4.0
3 4 [1.0, 7.0, 4.0] 2 1.0 7.0 4.0
You can do (assuming you know, that column B is the one to be expanded, you can iterate over dtype if you need to automate it further- to get the ones of compound type)
df=pd.DataFrame.from_records(map(lambda x: list(x), arr), columns=arr.dtype.names)
df2=pd.DataFrame(df["B"].tolist())
df2.columns=map(lambda x: f"B_{x+1}", df2.columns)
df=pd.concat([df, df2], sort=False, axis=1).drop(columns="B")
Outputs:
A C B_1 B_2 B_3
0 1 6 5.0 3.0 7.0
1 2 9 2.0 1.0 3.0
2 3 3 3.0 8.0 4.0
3 4 2 1.0 7.0 4.0
I am certain this is a silly question and just shows how much I don’t know.
But when would we use a numpy array vs pandas data frame for data?
Thank you