python - Split pandas dataframe in two if it has more than 10 rows - Stack Overflow
python - Split pandas dataframe into multiple dataframes with equal numbers of rows - Stack Overflow
python - Splitting dataframe into multiple dataframes - Stack Overflow
ENH: Add split method to DataFrame for flexible row-based partitioning
I used a List Comprehension to cut a huge DataFrame into blocks of 100'000:
size = 100000
list_of_dfs = [df.loc[i:i+size-1,:] for i in range(0, len(df),size)]
or as generator:
list_of_dfs = (df.loc[i:i+size-1,:] for i in range(0, len(df),size))
This will return the split DataFrames if the condition is met, otherwise return the original and None (which you would then need to handle separately). Note that this assumes the splitting only has to happen one time per df and that the second part of the split (if it is longer than 10 rows (meaning that the original was longer than 20 rows)) is OK.
df_new1, df_new2 = df[:10, :], df[10:, :] if len(df) > 10 else df, None
Note you can also use df.head(10) and df.tail(len(df) - 10) to get the front and back according to your needs. You can also use various indexing approaches: you can just provide the first dimensions index if you want, such as df[:10] instead of df[:10, :] (though I like to code explicitly about the dimensions you are taking). You can can also use df.iloc and df.ix to index in similar ways.
Be careful about using df.loc however, since it is label-based and the input will never be interpreted as an integer position. .loc would only work "accidentally" in the case when you happen to have index labels that are integers starting at 0 with no gaps.
But you should also consider the various options that pandas provides for dumping the contents of the DataFrame into HTML and possibly also LaTeX to make better designed tables for the presentation (instead of just copying and pasting). Simply Googling how to convert the DataFrame to these formats turns up lots of tutorials and advice for exactly this application.
There are many ways to do what you want, your method looks over-complicated. A groupby using a scaled index as the grouping key would work:
df = pd.DataFrame(data=np.random.rand(100, 3), columns=list('ABC'))
groups = df.groupby(np.arange(len(df.index))//10)
for (frameno, frame) in groups:
frame.to_csv("%s.csv" % frameno)
You can use a dictionary comprehension to save slices of the dataframe in groups of ten rows:
df_dict = {n: df.iloc[n:n+10, :]
for n in range(0, len(df), 10)}
>>> df_dict.keys()
[0, 10, 20]
>>> df_dict[10]
a b c
10 -0.011909 -0.304162 0.422001
11 0.127570 0.956831 1.837523
12 -1.074771 0.379723 -1.889117
13 -1.449475 -0.799574 -0.878192
14 -1.029757 0.551023 2.519929
15 -1.001400 0.838614 -1.006977
16 0.677216 -0.403859 0.451338
17 0.221596 -0.323259 0.324158
18 -0.241935 -2.251687 -0.088494
19 -0.995426 0.665569 -2.228848
Can I ask why not just do it by slicing the data frame. Something like
#create some data with Names column
data = pd.DataFrame({'Names': ['Joe', 'John', 'Jasper', 'Jez'] *4, 'Ob1' : np.random.rand(16), 'Ob2' : np.random.rand(16)})
#create unique list of names
UniqueNames = data.Names.unique()
#create a data frame dictionary to store your data frames
DataFrameDict = {elem : pd.DataFrame() for elem in UniqueNames}
for key in DataFrameDict.keys():
DataFrameDict[key] = data[:][data.Names == key]
Hey presto you have a dictionary of data frames just as (I think) you want them. Need to access one? Just enter
DataFrameDict['Joe']
Firstly your approach is inefficient because the appending to the list on a row by basis will be slow as it has to periodically grow the list when there is insufficient space for the new entry, list comprehensions are better in this respect as the size is determined up front and allocated once.
However, I think fundamentally your approach is a little wasteful as you have a dataframe already so why create a new one for each of these users?
I would sort the dataframe by column 'name', set the index to be this and if required not drop the column.
Then generate a list of all the unique entries and then you can perform a lookup using these entries and crucially if you only querying the data, use the selection criteria to return a view on the dataframe without incurring a costly data copy.
Use pandas.DataFrame.sort_values and pandas.DataFrame.set_index:
# sort the dataframe
df.sort_values(by='name', axis=1, inplace=True)
# set the index to be this and don't drop
df.set_index(keys=['name'], drop=False,inplace=True)
# get a list of names
names=df['name'].unique().tolist()
# now we can perform a lookup on a 'view' of the dataframe
joe = df.loc[df.name=='joe']
# now you can query all 'joes'
chunksize
When this argument is used, read_csv returns an iterator in which each iteration returns a new chunk.
data = [*pd.read_csv(EVALUATION_FILE, chunksize=12)]
numpy.split
If by chance you have already read in your dataframe and you want to split it after the fact. Use nupmy.split with an array that defines your split points.
data = np.split(df, range(12, len(df), 12))
Check groupby after read_csv
data=[y for x , y in df.groupby(data.index//12)]
EVALUATION_FILE = 'training/evaluation.csv'
data = pd.read_csv(
EVALUATION_FILE,
engine='python',
index_col=None
)
You could do use pd.cut on the cumulative sum of the B column:
th = 50
# find the cumulative sum of B
cumsum = df.B.cumsum()
# create the bins with spacing of th (threshold)
bins = list(range(0, cumsum.max() + 1, th))
# group by (split by) the bins
groups = pd.cut(cumsum, bins)
for key, group in df.groupby(groups):
print(group)
print()
Output
A B
0 0 10
1 1 30
A B
2 2 50
A B
3 3 20
4 4 10
5 5 30
Here's a method using numba to speed up our for loop:
We check when our limit is reached and we reset the total count and we assign a new group:
from numba import njit
@njit
def cumsum_reset(array, limit):
total = 0
counter = 0
groups = np.empty(array.shape[0])
for idx, i in enumerate(array):
total += i
if total >= limit or array[idx-1] == limit:
counter += 1
groups[idx] = counter
total = 0
else:
groups[idx] = counter
return groups
grps = cumsum_reset(df['B'].to_numpy(), 50)
for _, grp in df.groupby(grps):
print(grp, '\n')
Output
A B
0 0 10
1 1 30
A B
2 2 50
A B
3 3 20
4 4 10
5 5 30
Timings:
# create dataframe of 600k rows
dfbig = pd.concat([df]*100000, ignore_index=True)
dfbig.shape
(600000, 2)
# Erfan
%%timeit
cumsum_reset(dfbig['B'].to_numpy(), 50)
4.25 ms ± 46.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# Daniel Mesejo
def daniel_mesejo(th, column):
cumsum = column.cumsum()
bins = list(range(0, cumsum.max() + 1, th))
groups = pd.cut(cumsum, bins)
return groups
%%timeit
daniel_mesejo(50, dfbig['B'])
10.3 s ± 2.17 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
Conclusion, the numba for loop is 24~ x faster.
You can use, np.array_split to split the dataframe:
import numpy as np
dfs = np.array_split(df, 161) # split the dataframe into 161 separate tables
Edit (To assign a new col based on sequential number of df in dfs):
dfs = [df.assign(new_col=i) for i, df in enumerate(dfs, 1)]
simply use
import numpy as np
df_list = np.array_split(df, 3) # replace 3 with the amount of rows you want
In you case you should switch 3 with df(len) // desired_row_amount. We use // to round the result to an integer.
Or go old school and use a for loop, something along the lines of:
rows = 100 # example number of rows
df_list = [] # list to store dfs
for i in range(len(df) // rows):
if i == len(df) // rows: # if this is the last part of the df
df_list.append(df[i*rows:]) # append the dataframe rows left
else:
# append with a dataframe which has the desired amount of rows
df_list.append(df[i*rows:(i+1)*rows])