Apply a lambda and call sample with param frac:
In [2]:
df = pd.DataFrame({'a': [1,2,3,4,5,6,7],
'b': [1,1,1,0,0,0,0]})
grouped = df.groupby('b')
grouped.apply(lambda x: x.sample(frac=0.3))
Out[2]:
a b
b
0 6 7 0
1 2 3 1
Answer from EdChum on Stack OverflowApply a lambda and call sample with param frac:
In [2]:
df = pd.DataFrame({'a': [1,2,3,4,5,6,7],
'b': [1,1,1,0,0,0,0]})
grouped = df.groupby('b')
grouped.apply(lambda x: x.sample(frac=0.3))
Out[2]:
a b
b
0 6 7 0
1 2 3 1
pandas >= 1.1: GroupBy.sample
This works like magic:
# np.random.seed(0)
df.groupby('b').sample(frac=.3)
a b
5 6 0
0 1 1
pandas <= 1.0.X
You can use GroupBy.apply with sample. You do not need to use a lambda; apply accepts keyword arguments:
df.groupby('b', group_keys=False).apply(pd.DataFrame.sample, frac=.3)
a b
5 6 0
0 1 1
You can take a randoms sample of the unique values of df.some_key.unique(), use that to slice the df and finally groupby on the resultant:
In [337]:
df = pd.DataFrame({'some_key': [0,1,2,3,0,1,2,3,0,1,2,3],
'val': [1,2,3,4,1,5,1,5,1,6,7,8]})
In [338]:
print df[df.some_key.isin(random.sample(df.some_key.unique(),2))].groupby('some_key').mean()
val
some_key
0 1.000000
2 3.666667
If there are more than one groupby keys:
In [358]:
df = pd.DataFrame({'some_key1':[0,1,2,3,0,1,2,3,0,1,2,3],
'some_key2':[0,0,0,0,1,1,1,1,2,2,2,2],
'val': [1,2,3,4,1,5,1,5,1,6,7,8]})
In [359]:
gby = df.groupby(['some_key1', 'some_key2'])
In [360]:
print gby.mean().ix[random.sample(gby.indices.keys(),2)]
val
some_key1 some_key2
1 1 5
3 2 8
But if you are just going to get the values of each group, you don't even need to groubpy, MultiIndex will do:
In [372]:
idx = random.sample(set(pd.MultiIndex.from_product((df.some_key1, df.some_key2)).tolist()),
2)
print df.set_index(['some_key1', 'some_key2']).ix[idx]
val
some_key1 some_key2
2 0 3
3 1 5
I feel like lower-level numpy operations are cleaner:
import pandas as pd
import numpy as np
df = pd.DataFrame(
{
"some_key": [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3],
"val": [1, 2, 3, 4, 1, 5, 1, 5, 1, 6, 7, 8],
}
)
ids = df["some_key"].unique()
ids = np.random.choice(ids, size=2, replace=False)
ids
# > array([3, 2])
df.loc[df["some_key"].isin(ids)]
# > some_key val
# 2 2 3
# 3 3 4
# 6 2 1
# 7 3 5
# 10 2 7
# 11 3 8
Convert sample_info to dictionary. Group population by Group ID. Pass the sample size values to DataFrame.sample using the dictionary.
mapper = sample_info.set_index('Group ID')['Sample Size'].to_dict()
population.groupby('Group ID').apply(lambda x: x.sample(n=mapper.get(x.name))).reset_index(drop = True)
I am not sure about the speed but sample the index looks like save the memory at least
d=population.groupby('Group ID').groups
a=np.concatenate([np.random.choice(d[x],y) for x, y in zip(sample_info['Group ID'],sample_info['Sample Size']) ])
population.loc[a]
Out[83]:
Group ID Response
1 1 False
1 1 False
2 1 False
0 1 True
1 1 False
3 2 True
5 2 False
3 2 True
4 2 True
5 2 False
5 2 False
You can do with shuffle and ngroup
g = df.groupby(['col1', 'col2'])
a=np.arange(g.ngroups)
np.random.shuffle(a)
df[g.ngroup().isin(a[:2])]# change 2 to what you need :-)
Shuffle your dataframe using sample, and then perform a non-sorting groupby:
df = df.sample(frac=1)
df2 = pd.concat(
[g for _, g in df.groupby(['col1', 'col2'], sort=False, as_index=False)][:3],
ignore_index=True
)
If you need the first 3 per group, use groupby.head(3);
df2 = pd.concat(
[g.head(3) for _, g in df.groupby(['col1', 'col2'], sort=False, as_index=False)][:3],
ignore_index=True
)