I don't know how you have tested it the first time, here is my logic. It supposes the first element in flag is 0!
df = pd.DataFrame({'memberid': [1]*11,
'flag': [0,0,1,1,0,1,0,0,0,1,1],
})
df['seq'] = ""
for i in range(0, len(df)):
df.loc[i, 'seq'] = 1 if df.loc[i, 'flag'] == 0 else df.loc[i - 1, 'seq'] + 1
print(df)
Another solution using lambda:
df = pd.DataFrame({'memberid': [1] * 11,
'flag': [0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]
})
def f(flag):
global previous_seq
previous_seq = 1 if flag == 0 else previous_seq + 1
return previous_seq
previous_seq = 0
df['seq'] = df[['flag']].apply(lambda x: f(*x), axis=1)
print(df)
I am not sure if it is faster than the first solution....
Answer from Frenchy on Stack ExchangeI don't know how you have tested it the first time, here is my logic. It supposes the first element in flag is 0!
df = pd.DataFrame({'memberid': [1]*11,
'flag': [0,0,1,1,0,1,0,0,0,1,1],
})
df['seq'] = ""
for i in range(0, len(df)):
df.loc[i, 'seq'] = 1 if df.loc[i, 'flag'] == 0 else df.loc[i - 1, 'seq'] + 1
print(df)
Another solution using lambda:
df = pd.DataFrame({'memberid': [1] * 11,
'flag': [0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]
})
def f(flag):
global previous_seq
previous_seq = 1 if flag == 0 else previous_seq + 1
return previous_seq
previous_seq = 0
df['seq'] = df[['flag']].apply(lambda x: f(*x), axis=1)
print(df)
I am not sure if it is faster than the first solution....
I've tried this 3 ways iterating through the flags (counting the continuous 1s) and this was the fastest for the same dataframe (large enough to neglect small variation in time on retries).
I'm maintaining two lists (one for flag, one for seq) and a counter variable.
We loop through flags and append the corresponding seq to seq_list. As you described, we keep track of count if we keep seeing 1 and reset to 1 if we see a 0 in the flags_list.
We add seq_list as a column to the dataframe once we're done.
seq_list = []
counter = 0
flag_list = list(df['flag'])
for flag in flag_list:
if(flag == 0):
counter = 1
seq_list.append(counter)
else:
counter += 1
seq_list.append(counter)
df['seq'] = seq_list
Other variants I tried include
- Directly iterating through each flag element in the dataframe (using .loc) and adding to the seq_list which gets added as a column but this took 16x of the time for the method above.
seq_list = []
counter = 0
for i in range(df.shape[0]):
if(df.loc[i,'flag'] == 0):
counter = 1
seq_list.append(counter)
else:
counter += 1
seq_list.append(counter)
df['seq'] = seq_list
- Directly iterating through each flag element in the dataframe (using .loc) and modifying the dataframe right away (again, using .loc) but this was even slower and took 5x-10x the time for the previous method.
counter = 1
df['seq'] = 0
for i in range(df.shape[0]):
if(df.loc[i,'flag'] == 0):
counter = 1
df.loc[i,'seq'] = counter
else:
counter += 1
df.loc[i,'seq'] = counter
How to add a sequential number to each row within a group in pandas
Is there a way to concatenate values within groups as a sequence
Can I create a sequence based on a specific order within groups
I stumbled upon the answer which was embarrassingly simple. The groupby statement has a 'cumcount()' option which will enumerate group items.
df['sequence']=df.groupby('patient').cumcount()
The caveat is that the records have to be in the order you want them enumerated.
If you want the sequence to be sorted based on the values of another column, first sort the dataframe and then add the new sequence column.
For instance, if we want the sequence for patients visits sorted based on their visit date, the following code can be used.
df['sequence'] = df.sort_values(by=['patient', 'date']).groupby('patient']).cumcount() + 1
As seen in the docs for pandas.Series, all that is required for your data parameter is an array-like, dict, or scalar value. Hence to create a series for a range, you can do exactly the same as you would to create a list for a range.
one_to_hundred = pd.Series(range(1,101))
one_to_hundred=pd.Series(np.arange(1,101,1))
This is the correct answer where you create a series using the numpy arange function which creates a range starting with 1 till 100 by incrementing 1.
pd.DataFrame([np.arange(6, 18, 3)]*7)
alternately,
pd.DataFrame(np.repeat([np.arange(6, 18, 3)],7, axis=0))
0 1 2 3
0 6 9 12 15
1 6 9 12 15
2 6 9 12 15
3 6 9 12 15
4 6 9 12 15
5 6 9 12 15
6 6 9 12 15
Here is a solution using NumPy broadcasting which avoids Python loops, lists, and excessive memory allocation (as done by np.repeat):
pd.DataFrame(np.broadcast_to(np.arange(6, 18, 3), (6, 4)))
To understand why this is more efficient than other solutions, refer to the np.broadcast_to() docs: https://numpy.org/doc/stable/reference/generated/numpy.broadcast_to.html
more than one element of a broadcasted array may refer to a single memory location.
This means that no matter how many rows you create before passing to Pandas, you're only really allocating a single row, then a 2D array which refers to the data of that row multiple times.
If you assign the above to df, you can say df.values.base is a single row--this is the only storage required no matter how many rows appear in the DataFrame.
Assuming you just want to create a sequence number column, you can use ngroup:
df = pd.DataFrame({'group Nr':[50,50,50,53,53,53,53,56,56,59,59,59]})
df["sequence Nr"] = df.groupby("group Nr").ngroup() + 1
ngroup numbers each group starting from 0, so you'll want to add 1.
You can reach the target by the following code.
import pandas as pd
tmp = pd.DataFrame({'group Nr':[50,50,50,53,53,53,53,56,56,59,59,59]})
tmp = tmp.sort_values('group Nr')
s_df = tmp.groupby('group Nr').head(1)
s_df['sequential Nr'] = range(1, len(s_df)+1)
tmp = tmp.merge(s_df, on='group Nr', how='left')
print(tmp)
There can be a lot of solutions. In the comments of the code block (#) you will find a few links for more information:
import pandas as pd
import numpy as np
import random
import string
k = 5
N = 10
#http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.randint.html
#http://stackoverflow.com/a/2257449/2901002
df = pd.DataFrame({ 'A' : range(1, N + 1 ,1),
'B' : np.random.randint(k, k + 100 , size=N),
'C' : pd.Series(random.choice(string.ascii_uppercase) for _ in range(N)) })
print df
# A B C
#0 1 60 O
#1 2 94 L
#2 3 10 W
#3 4 94 X
#4 5 60 O
#5 6 20 K
#6 7 58 Y
#7 8 40 I
#8 9 49 X
#9 10 65 S
Numpy solution:
import pandas as pd
import numpy as np
k = 5
N = 10
alphabet = list('ABCDEFGHIJKLMNOPQRSTUVWXYZ')
#http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html
df = pd.DataFrame({ 'A' : range(1, N + 1 ,1),
'B' : np.random.randint(k, k + 100 , size=N),
'C' : np.random.choice(np.array(alphabet, dtype="|S1"), N) })
print df
# A B C
#0 1 16 U
#1 2 76 X
#2 3 101 N
#3 4 61 F
#4 5 52 J
#5 6 62 A
#6 7 99 L
#7 8 23 N
#8 9 75 D
#9 10 16 Q
import pandas
n = 30
k = 40
pandas.DataFrame([(i, random.randint(k, k+100), chr(random.randint(ord('A'), ord('Z')))) for i in xrange(0, n)
If you want you specify the column names otherwise it is set to 0,1,2
You can set_index and reindex using a range from the Sequence's min and max values:
(df.set_index('Sequence')
.reindex(range(df.Sequence.iat[0],df.Sequence.iat[-1]+1), fill_value='')
.reset_index())
Sequence Value
0 1 x
1 2 x
2 3
3 4 x
4 5
5 6 x
6 7 x
7 8
8 9 x
9 10 x
Or do it by merging DataFrames:
seq = [1, 2, 4, 6, 7, 9, 10]
dfs0 = pd.DataFrame.from_dict({'Sequence': seq, 'Value': ['x']*len(seq)})
dfseq = pd.DataFrame.from_dict({'Sequence': range( min(seq), max(seq)+1 )})
.merge(dfs0, on='Sequence', how='outer').fillna('')
print(dfseq)
Sequence Value
0 1 x
1 2 x
2 3
3 4 x
4 5
5 6 x
6 7 x
7 8
8 9 x
9 10 x
