This will return the split DataFrames if the condition is met, otherwise return the original and None (which you would then need to handle separately). Note that this assumes the splitting only has to happen one time per df and that the second part of the split (if it is longer than 10 rows (meaning that the original was longer than 20 rows)) is OK. · df_new1, df_new2 = df[:10, :], df[10:, :] if len(df) > 10 else df, None · Note you can also use df.head(10) and df.tail(len(df) - 10) to get the front and back according to your needs. You can also use various indexing approaches: you can just provide the first dimensions index if you want, such as df[:10] instead of df[:10, :] (though I like to code explicitly about the dimensions you are taking). You can can also use df.ilocdf.ix to index in similar ways. · Be careful about using df.loc however, since it is label-based and the input will never be interpreted as an integer position. .loc would only work "accidentally" in the case when you happen to have index labels that are integers starting at 0 with no gaps. · But you should also consider the various options that pandas provides for dumping the contents of the DataFrame into HTML and possibly also LaTeX to make better designed tables for the presentation (instead of just copying and pasting). Simply Googling how to convert the DataFrame to these formats turns up lots of tutorials and advice for exactly this application. Answer from ely on pythonpedia.com
🌐
Statology
statology.org › home › how to split a pandas dataframe into multiple dataframes
How to Split a Pandas DataFrame into Multiple DataFrames
August 5, 2021 - Notice that df1 contains the first six rows of the original DataFrame and df2 contains the last six rows of the original DataFrame. ... import pandas as pd #create DataFrame df = pd.DataFrame({'x': [1, 1, 1, 3, 3, 4, 5, 5, 5, 6, 7, 9], 'y': [5, 7, 7, 9, 12, 9, 9, 4, 3, 3, 1, 10]}) #split into three DataFrames df1 = df.iloc[:3] df2 = df.iloc[3:6] df3 = df.iloc[6:] #view resulting DataFrames print(df1) x y 0 1 5 1 1 7 2 1 7 print(df2) x y 3 3 9 4 3 12 5 4 9 print(df3) x y 6 5 9 7 5 4 8 5 3 9 6 3 10 7 1 11 9 10
Discussions

python - Split pandas dataframe in two if it has more than 10 rows - Stack Overflow
I have a huge CSV with many tables with many rows. I would like to simply split each dataframe into 2 if it contains more than 10 rows. If true, I would like the first dataframe to contain the fi... More on stackoverflow.com
🌐 stackoverflow.com
python - Split pandas dataframe into multiple dataframes with equal numbers of rows - Stack Overflow
I have a dataframe df : a b c 0 0.897134 -0.356157 -0.396212 1 -2.357861 2.066570 -0.512687 2 -0.080665 0.719328 0.604294 3 -0.639392 -0.9129... More on stackoverflow.com
🌐 stackoverflow.com
python - Splitting dataframe into multiple dataframes - Stack Overflow
I have a very large dataframe (around 1 million rows) with data from an experiment (60 respondents). I would like to split the dataframe into 60 dataframes (a dataframe for each participant). In the More on stackoverflow.com
🌐 stackoverflow.com
ENH: Add split method to DataFrame for flexible row-based partitioning
Feature Type Adding new functionality to pandas Changing existing functionality in pandas Removing existing functionality in pandas Problem Description Currently, pandas does not provide a direct, ... More on github.com
🌐 github.com
5
March 20, 2024
🌐
Skytowner
skytowner.com › explore › randomly_splitting_dataframe_into_multiple_dataframes_of_equal_size_in_pandas
Randomly splitting DataFrame into multiple DataFrames of equal size in Pandas
The frac=1 means we want all rows returned. we then use NumPy's array_split(~,2) method to split the DataFrame into 2 equally sized sub-DataFrames. The return type is a list of DataFrames. When the number of splits do not evenly divide the number of rows, then the resulting DataFrames will ...
Top answer
1 of 9
46

I used a List Comprehension to cut a huge DataFrame into blocks of 100'000:

size = 100000
list_of_dfs = [df.loc[i:i+size-1,:] for i in range(0, len(df),size)]

or as generator:

list_of_dfs = (df.loc[i:i+size-1,:] for i in range(0, len(df),size))
2 of 9
28

This will return the split DataFrames if the condition is met, otherwise return the original and None (which you would then need to handle separately). Note that this assumes the splitting only has to happen one time per df and that the second part of the split (if it is longer than 10 rows (meaning that the original was longer than 20 rows)) is OK.

df_new1, df_new2 = df[:10, :], df[10:, :] if len(df) > 10 else df, None

Note you can also use df.head(10) and df.tail(len(df) - 10) to get the front and back according to your needs. You can also use various indexing approaches: you can just provide the first dimensions index if you want, such as df[:10] instead of df[:10, :] (though I like to code explicitly about the dimensions you are taking). You can can also use df.iloc and df.ix to index in similar ways.

Be careful about using df.loc however, since it is label-based and the input will never be interpreted as an integer position. .loc would only work "accidentally" in the case when you happen to have index labels that are integers starting at 0 with no gaps.

But you should also consider the various options that pandas provides for dumping the contents of the DataFrame into HTML and possibly also LaTeX to make better designed tables for the presentation (instead of just copying and pasting). Simply Googling how to convert the DataFrame to these formats turns up lots of tutorials and advice for exactly this application.

Top answer
1 of 13
113

Can I ask why not just do it by slicing the data frame. Something like

#create some data with Names column
data = pd.DataFrame({'Names': ['Joe', 'John', 'Jasper', 'Jez'] *4, 'Ob1' : np.random.rand(16), 'Ob2' : np.random.rand(16)})

#create unique list of names
UniqueNames = data.Names.unique()

#create a data frame dictionary to store your data frames
DataFrameDict = {elem : pd.DataFrame() for elem in UniqueNames}

for key in DataFrameDict.keys():
    DataFrameDict[key] = data[:][data.Names == key]

Hey presto you have a dictionary of data frames just as (I think) you want them. Need to access one? Just enter

DataFrameDict['Joe']
2 of 13
79

Firstly your approach is inefficient because the appending to the list on a row by basis will be slow as it has to periodically grow the list when there is insufficient space for the new entry, list comprehensions are better in this respect as the size is determined up front and allocated once.

However, I think fundamentally your approach is a little wasteful as you have a dataframe already so why create a new one for each of these users?

I would sort the dataframe by column 'name', set the index to be this and if required not drop the column.

Then generate a list of all the unique entries and then you can perform a lookup using these entries and crucially if you only querying the data, use the selection criteria to return a view on the dataframe without incurring a costly data copy.

Use pandas.DataFrame.sort_values and pandas.DataFrame.set_index:

# sort the dataframe
df.sort_values(by='name', axis=1, inplace=True)

# set the index to be this and don't drop
df.set_index(keys=['name'], drop=False,inplace=True)

# get a list of names
names=df['name'].unique().tolist()

# now we can perform a lookup on a 'view' of the dataframe
joe = df.loc[df.name=='joe']

# now you can query all 'joes'
🌐
GitHub
github.com › pandas-dev › pandas › issues › 57934
ENH: Add split method to DataFrame for flexible row-based partitioning · Issue #57934 · pandas-dev/pandas
March 20, 2024 - Currently, pandas does not provide a direct, built-in method to split a DataFrame into multiple smaller DataFrames based on a number of specified rows. Users seeking to partition a DataFrame into chunks either have to relay on workarounds using ...
Author   pandas-dev
Find elsewhere
🌐
Spark By {Examples}
sparkbyexamples.com › home › pandas › how to split pandas dataframe?
How to Split Pandas DataFrame? - Spark By {Examples}
December 6, 2024 - You can split the Pandas DataFrame based on rows or columns by using Pandas.DataFrame.iloc[] attribute, groupby().get_group(), sample() functions. It
🌐
GeeksforGeeks
geeksforgeeks.org › python › split-pandas-dataframe-by-rows
Split Pandas Dataframe by Rows - GeeksforGeeks
July 15, 2025 - In this article, we will elucidate several commonly employed methods for Split Pandas Dataframe by Rows.
🌐
Arab Psychology
scales.arabpsychology.com › home › how can a pandas dataframe be split into multiple dataframes?
How Can A Pandas DataFrame Be Split Into Multiple DataFrames?
May 5, 2024 - You can use the following basic syntax to split a pandas DataFrame into multiple DataFrames based on row number:
🌐
GITNUX
blog.gitnux.com › home › programming-advice
How do I split a Pandas DataFrame into multiple DataFrames using Python? • Gitnux
October 5, 2023 - You can use the `groupby` method in Pandas to split a DataFrame into multiple DataFrames based on a common column value.
🌐
datagy
datagy.io › home › pandas tutorials › pandas dataframes › python: split a pandas dataframe
Python: Split a Pandas Dataframe • datagy
December 16, 2022 - Let’s say we wanted to split a Pandas dataframe in half. We would split row-wise at the mid-point. The way that we can find the midpoint of a dataframe is by finding the dataframe’s length and dividing it by two.
🌐
Untitled Publication
elisa.hashnode.dev › split-a-dataframe
Split a Pandas Dataframe in Python - Elisa's Blog - Hashnode
September 29, 2021 - Split a Pandas Dataframe into two parts, either sequentially or randomly. Select the number of rows or size ratio for the split dataframes.
🌐
Delft Stack
delftstack.com › home › howto › python pandas › split pandas dataframe
How to Split Pandas DataFrame | Delft Stack
February 2, 2024 - Python ScipyPythonPython ... split a DataFrame into multiple smaller DataFrames using row indexing, the DataFrame.groupby() method, and DataFrame.sample() method....
Top answer
1 of 2
4

You could do use pd.cut on the cumulative sum of the B column:

th = 50

# find the cumulative sum of B 
cumsum = df.B.cumsum()

# create the bins with spacing of th (threshold)
bins = list(range(0, cumsum.max() + 1, th))

# group by (split by) the bins
groups = pd.cut(cumsum, bins)

for key, group in df.groupby(groups):
    print(group)
    print()

Output

   A   B
0  0  10
1  1  30

   A   B
2  2  50

   A   B
3  3  20
4  4  10
5  5  30
2 of 2
2

Here's a method using numba to speed up our for loop:

We check when our limit is reached and we reset the total count and we assign a new group:

from numba import njit

@njit
def cumsum_reset(array, limit):
    total = 0
    counter = 0 
    groups = np.empty(array.shape[0])
    for idx, i in enumerate(array):
        total += i
        if total >= limit or array[idx-1] == limit:
            counter += 1
            groups[idx] = counter
            total = 0
        else:
            groups[idx] = counter
    
    return groups

grps = cumsum_reset(df['B'].to_numpy(), 50)

for _, grp in df.groupby(grps):
    print(grp, '\n')

Output

   A   B
0  0  10
1  1  30 

   A   B
2  2  50 

   A   B
3  3  20
4  4  10
5  5  30

Timings:

# create dataframe of 600k rows
dfbig = pd.concat([df]*100000, ignore_index=True)
dfbig.shape

(600000, 2)

# Erfan
%%timeit
cumsum_reset(dfbig['B'].to_numpy(), 50)

4.25 ms ± 46.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# Daniel Mesejo
def daniel_mesejo(th, column):
    cumsum = column.cumsum()
    bins = list(range(0, cumsum.max() + 1, th))
    groups = pd.cut(cumsum, bins)
    
    return groups

%%timeit
daniel_mesejo(50, dfbig['B'])

10.3 s ± 2.17 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

Conclusion, the numba for loop is 24~ x faster.

🌐
Skytowner
skytowner.com › explore › splitting_dataframe_into_multiple_dataframes_based_on_value_in_pandas
Splitting DataFrame into multiple DataFrames based on value in Pandas
To split a DataFrame into multiple DataFrames based on values in a column in Pandas, perform a groupby(~) on the column, and then call the methods tuple(~) and dict(~).
Top answer
1 of 1
1
  • You can use .groupby on the 'Rows' and create a dict of DataFrames with unique 'Row' values as keys, with a dict-comprehension.
    • .groupby returns a groupby object, that contains information about the groups, where g is the unique value in 'Rows' for each group, and d is the DataFrame for that group.
  • The value of each key in df_dict, will be a DataFrame, which can be accessed in the standard way, df_dict['key'].
import pandas as pd

# setup data and dataframe
data = {'Rows': ['row_1', 'row_1', 'row_1', 'row_2', 'row_2', 'row_33', 'row_33', 'row_33', 'row_34', 'row_34'],
        'col1': ['var1', 'var2', 'var3', 'var1', 'var2', 'var1', 'var2', 'var3', 'var1', 'var2'],
        'value1': [12, 212, 340, 226, 323, 592, 282, 455, 457, 617],
        'value2': [3434, 546, 8686, 55, 878, 565, 343, 764, 24, 422]}

df = pd.DataFrame(data)

# split the dataframe and loop of the groupby object
df_dict = dict()

for g, d in df.groupby('Rows'):
    df_dict[g] = d


# or as a dict comprehension: the unique Row value will be the key
df_dict = {g: d for g, d in df.groupby('Rows')}


# or a specific name for the key, using enumerate
df_dict = {f'df{i}': d for i, (g, d) in enumerate(df.groupby('Rows'))}

df_dict['df0'] or df_dict['row_1']

    Rows  col1  value1  value2
0  row_1  var1      12    3434
1  row_1  var2     212     546
2  row_1  var3     340    8686

df_dict['df1'] or df_dict['row_2']

    Rows  col1  value1  value2
3  row_2  var1     226      55
4  row_2  var2     323     878

df_dict['df2'] or df_dict['row_33']

     Rows  col1  value1  value2
5  row_33  var1     592     565
6  row_33  var2     282     343
7  row_33  var3     455     764
🌐
py4u
py4u.org › blog › python-splitting-dataframe-into-multiple-dataframes-based-on-column-values-and-naming-them-with-those-values
Python: Split DataFrame into Multiple DataFrames by Column Values (Auto-Naming with Values) – Guide for Competitor Product Region Analysis
North Region DataFrame: Date Product Competitor Region Sales 0 2023-01-01 Laptop ABC Corp North 1500 1 2023-01-01 Phone XYZ Inc North 800 ... Avoids overwriting existing variables (e.g., if you already have a variable named north). Makes it easy to loop through all subsets later (e.g., for batch analysis). Scales to any number of unique values (even 100+ regions!). Always verify that the split worked correctly. Check: The number of rows in each sub-DataFrame.