I think that pandas offers better alternatives to what you're suggesting (rationale below).

For one, there's the pandas.Panel data structure, which was meant for things like you're doing here.

However, as Wes McKinney (the Pandas author) noted in his book Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, multi-dimensional indices, to a large extent, offer a better alternative.

Consider the following alternative to your code:

dfs = []
for year in range(1967,2014):
    ....some codes that allow me to generate df1, df2 and df3 
    df1['year'] = year
    df1['origin'] = 'df1'
    df2['year'] = year
    df2['origin'] = 'df2'
    df3['year'] = year
    df3['origin'] = 'df3'
    dfs.extend([df1, df2, df3])
df = pd.concat(dfs)

This gives you a DataFrame with 4 columns: 'firm', 'price', 'year', and 'origin'.

This gives you the flexibility to:

• Organize hierarchically by, say, 'year' and 'origin': df.set_index(['year', 'origin']), by, say, 'origin' and 'price': df.set_index(['origin', 'price'])

• Do groupbys according to different levels

• In general, slice and dice the data along many different ways.

What you're suggesting in the question makes one dimension (origin) arbitrarily different, and it's hard to think of an advantage to this. If a split along some dimension is necessary due, to, e.g., performance, you can combine DataFrames better with standard Python data structures:

• A dictionary mapping each year to a Dataframe with the other three dimensions.

• Three DataFrames, one for each origin, each having three dimensions.

I had the same problem and resorted to dataclasses. In a shell something like:

@dataclass
class OrderDataFields:
    """
    data field names aka columns
    """
    order_id        : str = 'order_id'
    order_amount    : str = 'order_amount'
    order_date      : str = 'order_date'


@dataclass
class OrderData:
    data: pd.DataFrame
    columns: OrdrDataFields = OrderDataFields()

Now when you are using your dataframe put it in the class

order_data = OrderData(data=order_df)

If you want to perform column checks every time you instantiate Data objects you can use BaseData class and inherit it in your for example OrderData object:

@dataclass
class BaseData:
    """
    Base class for all Data dataclasses.

    Expected child attributes:
        `data` : pd.DataFrame -- data
        `columns`: object -- dataclass with column names

    Raises
    ------
    ValueError
        If columns in `columns` and `data` dataframe don't match
    """
    data: pd.DataFrame
    columns: object

    def __post_init__(self):
        self._check_columns()
    
    def _check_columns(self):
        """
        Check if columns in dataframe match the columns in `columns` attribute

        Raises
        ------
        ValueError
            If columns don't match
        """
        data_columns = set(self.data.columns)
        columns_columns = {v for k,v in asdict(self.columns).items()}
        if data_columns != columns_columns:
            raise ValueError


@dataclass
class OrderData(BaseData):
    data: pd.DataFrame
    columns: OrdrDataFields = OrderDataFields()

And now when you do some wrangling you can use dataframe and columns:

df = order_data.data
c = order_data.columns

df[c.order_amount] ....
....

Along those lines, adjust for your case

There is also library pandera : https://pandera.readthedocs.io/en/stable/

Pandas operates more like data.frame in this regard. You can check this using the memory_profiler package; here's an example of its use in the Jupyter notebook:

First define a program that will test this:

%%file df_memprofile.py
import numpy as np
import pandas as pd

def foo():
    x = np.random.rand(1000000, 5)
    y = pd.DataFrame(x, columns=list('abcde'))
    y.rename(columns = {'e': 'f'}, inplace=True)
    return y

Then load the memory profiler and run + profile the function

%load_ext memory_profiler
from df_memprofile import foo
%mprun -f foo foo()

I get the following output:

Filename: /Users/jakevdp/df_memprofile.py

Line #    Mem usage    Increment   Line Contents
================================================
     4     66.1 MiB     66.1 MiB   def foo():
     5    104.2 MiB     38.2 MiB       x = np.random.rand(1000000, 5)
     6    104.4 MiB      0.2 MiB       y = pd.DataFrame(x, columns=list('abcde'))
     7    142.6 MiB     38.2 MiB       y.rename(columns = {'e': 'f'}, inplace=True)
     8    142.6 MiB      0.0 MiB       return y

You can see a couple things:

1. when y is created, it is just a light wrapper around the original array: i.e. no data is copied.

2. When the column in y is renamed, it results in duplication of the entire data array in memory (it's the same 38MB increment as when x is created in the first place).

So, unless I'm missing something, it appears that Pandas operates more like R's dataframes than R's data tables.

Edit: Note that rename() has an argument copy that controls this behavior, and defaults to True. For example, using this:

y.rename(columns = {'e': 'f'}, inplace=True, copy=False)

... results in an inplace operation without copying data.

Alternatively, you can modify the columns attribute directly:

y.columns = ['a', 'b', 'c', 'd', 'f']