You're trying to split the entire review column of the data frame (which is the Series mentioned in the error message). What you want to do is apply a function to each row of the data frame, which you can do by calling apply on the data frame:
f = lambda x: len(x["review"].split("disappointed")) -1
reviews["disappointed"] = reviews.apply(f, axis=1)
Answer from hoyland on Stack OverflowYou're trying to split the entire review column of the data frame (which is the Series mentioned in the error message). What you want to do is apply a function to each row of the data frame, which you can do by calling apply on the data frame:
f = lambda x: len(x["review"].split("disappointed")) -1
reviews["disappointed"] = reviews.apply(f, axis=1)
pandas 0.20.3 has pandas.Series.str.split() which acts on every string of the series and does the split. So you can simply split and then count the number of splits made
len(reviews['review'].str.split('disappointed')) - 1
pandas.Series.str.split
You can do:
df["Shape"]=df["Shape"].str.split("\r\n")
print(df.explode("Shape").reset_index(drop=True))
Output:
Color Shape Price
0 Green Rectangle 10
1 Green Triangle 10
2 Green Octangle 10
3 Blue Rectangle 15
4 Blue Triangle 15
This might not be the most efficient way to do it but I can confirm that it works with the sample df:
data = [['Green', 'Rectangle\r\nTriangle\r\nOctangle', 10], ['Blue', 'Rectangle\r\nTriangle', 15]]
df = pd.DataFrame(data, columns = ['Color', 'Shape', 'Price'])
new_df = pd.DataFrame(columns = ['Color', 'Shape', 'Price'])
for index, row in df.iterrows():
split = row['Shape'].split('\r\n')
for shape in split:
new_df = new_df.append(pd.DataFrame({'Color':[row['Color']], 'Shape':[shape], 'Price':[row['Price']]}))
new_df = new_df.reset_index(drop=True)
print(new_df)
Output:
Color Price Shape
0 Green 10 Rectangle
1 Green 10 Triangle
2 Green 10 Octangle
3 Blue 15 Rectangle
4 Blue 15 Triangle
python - Split (explode) pandas dataframe string entry to separate rows - Stack Overflow
Python - Parsing Columns and Rows - Stack Overflow
numpy - How to split row of data into columns separated by comma in python - Stack Overflow
Pandas to split single df row into multiple
UPDATE 3: it makes more sense to use Series.explode() / DataFrame.explode() methods (implemented in Pandas 0.25.0 and extended in Pandas 1.3.0 to support multi-column explode) as is shown in the usage example:
for a single column:
In [1]: df = pd.DataFrame({'A': [[0, 1, 2], 'foo', [], [3, 4]],
...: 'B': 1,
...: 'C': [['a', 'b', 'c'], np.nan, [], ['d', 'e']]})
In [2]: df
Out[2]:
A B C
0 [0, 1, 2] 1 [a, b, c]
1 foo 1 NaN
2 [] 1 []
3 [3, 4] 1 [d, e]
In [3]: df.explode('A')
Out[3]:
A B C
0 0 1 [a, b, c]
0 1 1 [a, b, c]
0 2 1 [a, b, c]
1 foo 1 NaN
2 NaN 1 []
3 3 1 [d, e]
3 4 1 [d, e]
for multiple columns (for Pandas 1.3.0+):
In [4]: df.explode(['A', 'C'])
Out[4]:
A B C
0 0 1 a
0 1 1 b
0 2 1 c
1 foo 1 NaN
2 NaN 1 NaN
3 3 1 d
3 4 1 e
UPDATE 2: more generic vectorized function, which will work for multiple normal and multiple list columns
def explode(df, lst_cols, fill_value='', preserve_index=False):
# make sure `lst_cols` is list-alike
if (lst_cols is not None
and len(lst_cols) > 0
and not isinstance(lst_cols, (list, tuple, np.ndarray, pd.Series))):
lst_cols = [lst_cols]
# all columns except `lst_cols`
idx_cols = df.columns.difference(lst_cols)
# calculate lengths of lists
lens = df[lst_cols[0]].str.len()
# preserve original index values
idx = np.repeat(df.index.values, lens)
# create "exploded" DF
res = (pd.DataFrame({
col:np.repeat(df[col].values, lens)
for col in idx_cols},
index=idx)
.assign(**{col:np.concatenate(df.loc[lens>0, col].values)
for col in lst_cols}))
# append those rows that have empty lists
if (lens == 0).any():
# at least one list in cells is empty
res = (res.append(df.loc[lens==0, idx_cols], sort=False)
.fillna(fill_value))
# revert the original index order
res = res.sort_index()
# reset index if requested
if not preserve_index:
res = res.reset_index(drop=True)
return res
Demo:
Multiple list columns - all list columns must have the same # of elements in each row:
In [134]: df
Out[134]:
aaa myid num text
0 10 1 [1, 2, 3] [aa, bb, cc]
1 11 2 [] []
2 12 3 [1, 2] [cc, dd]
3 13 4 [] []
In [135]: explode(df, ['num','text'], fill_value='')
Out[135]:
aaa myid num text
0 10 1 1 aa
1 10 1 2 bb
2 10 1 3 cc
3 11 2
4 12 3 1 cc
5 12 3 2 dd
6 13 4
preserving original index values:
In [136]: explode(df, ['num','text'], fill_value='', preserve_index=True)
Out[136]:
aaa myid num text
0 10 1 1 aa
0 10 1 2 bb
0 10 1 3 cc
1 11 2
2 12 3 1 cc
2 12 3 2 dd
3 13 4
Setup:
df = pd.DataFrame({
'aaa': {0: 10, 1: 11, 2: 12, 3: 13},
'myid': {0: 1, 1: 2, 2: 3, 3: 4},
'num': {0: [1, 2, 3], 1: [], 2: [1, 2], 3: []},
'text': {0: ['aa', 'bb', 'cc'], 1: [], 2: ['cc', 'dd'], 3: []}
})
CSV column:
In [46]: df
Out[46]:
var1 var2 var3
0 a,b,c 1 XX
1 d,e,f,x,y 2 ZZ
In [47]: explode(df.assign(var1=df.var1.str.split(',')), 'var1')
Out[47]:
var1 var2 var3
0 a 1 XX
1 b 1 XX
2 c 1 XX
3 d 2 ZZ
4 e 2 ZZ
5 f 2 ZZ
6 x 2 ZZ
7 y 2 ZZ
using this little trick we can convert CSV-like column to list column:
In [48]: df.assign(var1=df.var1.str.split(','))
Out[48]:
var1 var2 var3
0 [a, b, c] 1 XX
1 [d, e, f, x, y] 2 ZZ
UPDATE: generic vectorized approach (will work also for multiple columns):
Original DF:
In [177]: df
Out[177]:
var1 var2 var3
0 a,b,c 1 XX
1 d,e,f,x,y 2 ZZ
Solution:
first let's convert CSV strings to lists:
In [178]: lst_col = 'var1'
In [179]: x = df.assign(**{lst_col:df[lst_col].str.split(',')})
In [180]: x
Out[180]:
var1 var2 var3
0 [a, b, c] 1 XX
1 [d, e, f, x, y] 2 ZZ
Now we can do this:
In [181]: pd.DataFrame({
...: col:np.repeat(x[col].values, x[lst_col].str.len())
...: for col in x.columns.difference([lst_col])
...: }).assign(**{lst_col:np.concatenate(x[lst_col].values)})[x.columns.tolist()]
...:
Out[181]:
var1 var2 var3
0 a 1 XX
1 b 1 XX
2 c 1 XX
3 d 2 ZZ
4 e 2 ZZ
5 f 2 ZZ
6 x 2 ZZ
7 y 2 ZZ
OLD answer:
Inspired by @AFinkelstein solution, i wanted to make it bit more generalized which could be applied to DF with more than two columns and as fast, well almost, as fast as AFinkelstein's solution):
In [2]: df = pd.DataFrame(
...: [{'var1': 'a,b,c', 'var2': 1, 'var3': 'XX'},
...: {'var1': 'd,e,f,x,y', 'var2': 2, 'var3': 'ZZ'}]
...: )
In [3]: df
Out[3]:
var1 var2 var3
0 a,b,c 1 XX
1 d,e,f,x,y 2 ZZ
In [4]: (df.set_index(df.columns.drop('var1',1).tolist())
...: .var1.str.split(',', expand=True)
...: .stack()
...: .reset_index()
...: .rename(columns={0:'var1'})
...: .loc[:, df.columns]
...: )
Out[4]:
var1 var2 var3
0 a 1 XX
1 b 1 XX
2 c 1 XX
3 d 2 ZZ
4 e 2 ZZ
5 f 2 ZZ
6 x 2 ZZ
7 y 2 ZZ
Pandas >= 0.25
Series and DataFrame methods define a .explode() method that explodes lists into separate rows. See the docs section on Exploding a list-like column.
Since you have a list of comma separated strings, split the string on comma to get a list of elements, then call explode on that column.
df = pd.DataFrame({'var1': ['a,b,c', 'd,e,f'], 'var2': [1, 2]})
df
var1 var2
0 a,b,c 1
1 d,e,f 2
df.assign(var1=df['var1'].str.split(',')).explode('var1')
var1 var2
0 a 1
0 b 1
0 c 1
1 d 2
1 e 2
1 f 2
Note that explode only works on a single column (for now). To explode multiple columns at once, see below.
NaNs and empty lists get the treatment they deserve without you having to jump through hoops to get it right.
df = pd.DataFrame({'var1': ['d,e,f', '', np.nan], 'var2': [1, 2, 3]})
df
var1 var2
0 d,e,f 1
1 2
2 NaN 3
df['var1'].str.split(',')
0 [d, e, f]
1 []
2 NaN
df.assign(var1=df['var1'].str.split(',')).explode('var1')
var1 var2
0 d 1
0 e 1
0 f 1
1 2 # empty list entry becomes empty string after exploding
2 NaN 3 # NaN left un-touched
This is a serious advantage over ravel/repeat -based solutions (which ignore empty lists completely, and choke on NaNs).
Exploding Multiple Columns
pandas 1.3 update
df.explode works on multiple columns starting from pandas 1.3:
df = pd.DataFrame({'var1': ['a,b,c', 'd,e,f'],
'var2': ['i,j,k', 'l,m,n'],
'var3': [1, 2]})
df
var1 var2 var3
0 a,b,c i,j,k 1
1 d,e,f l,m,n 2
(df.set_index(['var3'])
.apply(lambda col: col.str.split(','))
.explode(['var1', 'var2'])
.reset_index()
.reindex(df.columns, axis=1))
var1 var2 var3
0 a i 1
1 b j 1
2 c k 1
3 d l 2
4 e m 2
5 f n 2
On older versions, you would move the explode column inside the apply which is a lot less performant:
(df.set_index(['var3'])
.apply(lambda col: col.str.split(',').explode())
.reset_index()
.reindex(df.columns, axis=1))
The idea is to set as the index, all the columns that should NOT be exploded, then explode the remaining columns via apply. This works well when the lists are equally sized.
Well, I can explain the error. You're using str.split() and its usage pattern is:
str.split(separator, maxsplit)
You're using str.split(string, separator) and that isn't a valid call to split. Here is a direct link to the Python docs for this:
http://docs.python.org/library/stdtypes.html#str.split
To directly answer your question, there is a problem with the following line:
col = line.split(line, ',')
If you check the documentation for str.split, you'll find the description to be as follows:
str.split([sep[, maxsplit]])Return a list of the words in the string, using sep as the delimiter string. If maxsplit is given, at most maxsplit splits are done (thus, the list will have at most
maxsplit+1elements). If maxsplit is not specified, then there is no limit on the number of splits (all possible splits are made).
This is not what you want. You are not trying to specify the number of splits you want to make.
Consider replacing your for loop and network.append with this:
for line in filename.readlines():
# line is a string representing the values for this row
row = line.split(',')
# row is the list of numbers strings for this row, such as ['1', '0', '4', ...]
cols = [int(x) for x in row]
# cols is the list of numbers for this row, such as [1, 0, 4, ...]
network.append(row)
# Put this row into network, such that network is [[1, 0, 4, ...], [...], ...]
1.numpy solution: (because numpy tag)
Use numpy.genfromtxt for numpy array:
import numpy as np
arr = np.genfromtxt('file.txt',dtype='str',delimiter=',')
print (arr)
[['red' 'red' 'blue']
['blue' 'red' 'blue']
['blue' 'blue' 'red']]
print (arr[0])
['red' 'red' 'blue']
print (arr[0][2])
blue
2.pandas solution:
Use read_csv for DataFrame and for select values loc:
import pandas as pd
df = pd.read_csv('file.txt', header=None)
print (df)
0 1 2
0 red red blue
1 blue red blue
2 blue blue red
#select first row to Series
print (df.loc[0])
0 red
1 red
2 blue
Name: 0, dtype: object
#select value by index and column
print (df.loc[0, 2])
blue
3.pure python solutions:
If want nested lists use nested list comprehension:
data = [[item for item in line.rstrip('\r\n').split(',')]
for line in open('file.txt')]
print (data)
[['red', 'red', 'blue'], ['blue', 'red', 'blue'], ['blue', 'blue', 'red']]
Or with module csv:
import csv
reader = csv.reader(open("file.txt"), delimiter=',')
data = [word for word in [row for row in reader]]
print (data)
[['red', 'red', 'blue'], ['blue', 'red', 'blue'], ['blue', 'blue', 'red']]
print (data[0])
['red', 'red', 'blue']
print (data[0][2])
blue
Solution without any external modules :
output = []
with open('file.txt', 'r') as reading:
file_input = reading.read().split('\n')
for row in file_input:
output.append(row.split(','))
print(output)
Hi there. I'm trying to rack my brain about how to accomplish this. I have a form where each submission is one row in a csv file.
| ID | Name1 | Age | Name2 | Age |
|---|---|---|---|---|
| 001 | Joe | 25 | Mary | 89 |
| 002 | Chris | 38 |
What I want to accomplish is to splice a single row into 2 rows if Name2 has a value. I want to take this original df, and create something like this:
| ID | Name | Age |
|---|---|---|
| 001 | Joe | 25 |
| 001 | Mary | 89 |
| 002 | Chris | 38 |
That way, I can still group submissions (it's easy to see that Joe and Mary came from the same form), but I have a clean df.
df2=pd.DataFrame(columns=['ID','Name','Age])
#append just the first 3 columns 0,1,2 to new df2
for x in range(len(df):
df2.append(df[x,0:3]
#append columns 0,3,4 to df2--but only if there is data in the Name2 column
if pd.notnull(df.iloc[x,3]):
df2.append(df.iloc[x,0]
df2.append(df.iloc[x,3:5]This is my logic, but not implementing correctly. Can somebody point me to the right direction? Is this a smart way to approach this? In reality, I have a csv file with about 100 columns and I'm picking and choosing new columns to be added to a much cleaner second data frame. This involves lots of guessing about which columns I want and then getting the right index for .iloc.
Thanks for any tips.
Having the player name be it's own variable is not as helpful as you might think: you can't iterate over the collection (because there isn't one), your code is fixed and fragile (you have to add/remove lines when a player name is added or removed from your file, etc.
If, however, you have them in a dictionary -- well, now you can iterate, and you can still ask for players by name.
player_data = file("Orioles.csv", "rU")
orioles = dict()
for row in player_data:
row = row.split(',')
orioles[row[0]] = tuple(map(float, row[1:]))
print orioles.keys()
# ['Adam_Jones', 'Nick_Markakis']
print orioles['Adam_Jones']
# (0.005, 0.189, 0.07, 0.002, 0.09)
Rather than the row.split(',') trick above, you'll probably want to use the csv module in real code.
Orioles = file("Orioles.csv", "rU")
stats = dict((row[0], map(float, row[1:])) for row in Orioles)
# In Python 2.7+: stats = {row[0]: map(float, row[1:]) for row in Orioles}
Now, you can access the stats like this:
print(sum(stats['Nick_Markakis'])) # 0.356
From the docs rstrip:
Return a copy of the string with trailing characters removed.
So, in this case, it will return the "\n" at the end of the string.
Also from the docs, split:
Return a list of the words in the string, using sep as the delimiter string.
Therefore, once you have removed "\n", it will return the list of string which are separated by ",".
Example
>>> s = "a,b,c,d\n"
>>> s.rstrip("\n").split(",")
['a', 'b', 'c', 'd']
The way row = line.rstrip("\n").split(",")
If the same as following, you can reuse directly the string returned by rstrip
row = line.rstrip("\n") # remove newline char at the end
row = row.split(",") # separate one string into a list of multiple ones, based on the comma