To select the rows of your dataframe you can use iloc, you can then select the columns you want using square brackets.
For example:
df = pd.DataFrame(data=[[1,2,3]]*5, index=range(3, 8), columns = ['a','b','c'])
gives the following dataframe:
a b c
3 1 2 3
4 1 2 3
5 1 2 3
6 1 2 3
7 1 2 3
to select only the 3d and fifth row you can do:
df.iloc[[2,4]]
which returns:
a b c
5 1 2 3
7 1 2 3
if you then want to select only columns b and c you use the following command:
df[['b', 'c']].iloc[[2,4]]
which yields:
b c
5 2 3
7 2 3
To then get the mean of this subset of your dataframe you can use the df.mean function. If you want the means of the columns you can specify axis=0, if you want the means of the rows you can specify axis=1
thus:
df[['b', 'c']].iloc[[2,4]].mean(axis=0)
returns:
b 2
c 3
As we should expect from the input dataframe.
For your code you can then do:
df[column_list].iloc[row_index_list].mean(axis=0)
EDIT after comment: New question in comment: I have to store these means in another df/matrix. I have L1, L2, L3, L4...LX lists which tells me the index whose mean I need for columns C[1, 2, 3]. For ex: L1 = [0, 2, 3] , means I need mean of rows 0,2,3 and store it in 1st row of a new df/matrix. Then L2 = [1,4] for which again I will calculate mean and store it in 2nd row of the new df/matrix. Similarly till LX, I want the new df to have X rows and len(C) columns. Columns for L1..LX will remain same. Could you help me with this?
Answer:
If i understand correctly, the following code should do the trick (Same df as above, as columns I took 'a' and 'b':
first you loop over all the lists of rows, collection all the means as pd.series, then you concatenate the resulting list of series over axis=1, followed by taking the transpose to get it in the right format.
dfs = list()
for l in L:
dfs.append(df[['a', 'b']].iloc[l].mean(axis=0))
mean_matrix = pd.concat(dfs, axis=1).T
Answer from PdevG on Stack OverflowTo select the rows of your dataframe you can use iloc, you can then select the columns you want using square brackets.
For example:
df = pd.DataFrame(data=[[1,2,3]]*5, index=range(3, 8), columns = ['a','b','c'])
gives the following dataframe:
a b c
3 1 2 3
4 1 2 3
5 1 2 3
6 1 2 3
7 1 2 3
to select only the 3d and fifth row you can do:
df.iloc[[2,4]]
which returns:
a b c
5 1 2 3
7 1 2 3
if you then want to select only columns b and c you use the following command:
df[['b', 'c']].iloc[[2,4]]
which yields:
b c
5 2 3
7 2 3
To then get the mean of this subset of your dataframe you can use the df.mean function. If you want the means of the columns you can specify axis=0, if you want the means of the rows you can specify axis=1
thus:
df[['b', 'c']].iloc[[2,4]].mean(axis=0)
returns:
b 2
c 3
As we should expect from the input dataframe.
For your code you can then do:
df[column_list].iloc[row_index_list].mean(axis=0)
EDIT after comment: New question in comment: I have to store these means in another df/matrix. I have L1, L2, L3, L4...LX lists which tells me the index whose mean I need for columns C[1, 2, 3]. For ex: L1 = [0, 2, 3] , means I need mean of rows 0,2,3 and store it in 1st row of a new df/matrix. Then L2 = [1,4] for which again I will calculate mean and store it in 2nd row of the new df/matrix. Similarly till LX, I want the new df to have X rows and len(C) columns. Columns for L1..LX will remain same. Could you help me with this?
Answer:
If i understand correctly, the following code should do the trick (Same df as above, as columns I took 'a' and 'b':
first you loop over all the lists of rows, collection all the means as pd.series, then you concatenate the resulting list of series over axis=1, followed by taking the transpose to get it in the right format.
dfs = list()
for l in L:
dfs.append(df[['a', 'b']].iloc[l].mean(axis=0))
mean_matrix = pd.concat(dfs, axis=1).T
You can select specific columns from a DataFrame by passing a list of indices to .iloc, for example:
df.iloc[:, [2,5,6,7,8]]
Will return a DataFrame containing those numbered columns (note: This uses 0-based indexing, so 2 refers to the 3rd column.)
To take a mean down of that column, you could use:
# Mean along 0 (vertical) axis: return mean for specified columns, calculated across all rows
df.iloc[:, [2,5,6,7,8]].mean(axis=0)
To take a mean across that column, you could use:
# Mean along 1 (horizontal) axis: return mean for each row, calculated across specified columns
df.iloc[:, [2,5,6,7,8]].mean(axis=1)
You can also supply specific indices for both axes to return a subset of the table:
df.iloc[[1,2,3,4], [2,5,6,7,8]]
For your specific example, you would do:
import pandas as pd
import numpy as np
df = pd.DataFrame(
np.array([[1,2,3,0,5],[1,2,3,4,5],[1,1,1,6,1],[1,0,0,0,0]]),
columns=["a","b","c","d","q"],
index = [0,1,2,3]
)
#I want mean of 0, 2, 3 rows for each a, b, d columns
#. a b d
#0 1 1 2
df.iloc[ [0,2,3], [0,1,3] ].mean(axis=0)
Which outputs:
a 1.0
b 1.0
d 2.0
dtype: float64
Alternatively, to access via column names, first select on those:
df[ ['a','b','d'] ].iloc[ [0,1,3] ].mean(axis=0)
To answer the second part of your question (from the comments) you can join multiple DataFrames together using pd.concat. It is faster to accumulate the frames in a list and then pass to pd.concat in one go, e.g.
dfs = []
for ix in idxs:
dfm = df.iloc[ [0,2,3], ix ].mean(axis=0)
dfs.append(dfm)
dfm_summary = pd.concat(dfs, axis=1) # Stack horizontally
Iterating through the rows doesn't take advantage of Pandas' strengths. If you want to do something with a column based on values of another column, you can use .loc[]:
dataFrame.loc[dataFrame['Dates'] == 'Oct-16', 'Score 1']
The first part of .loc[] selects the rows you want, using your specified criteria (dataFrame['Dates'] == 'Oct-16'). The second part specifies the column you want (Score 1). Then if you want to get the mean, you can just put .mean() on the end:
dataFrame.loc[dataFrame['Dates'] == 'Oct-16', 'Score 1'].mean()
How about the mean for all dates
dataframe.groupby('Dates').['Score 1'].mean()
You can specify a new column. You also need to compute the mean along the rows, so use axis=1.
df['mean'] = df.mean(axis=1)
>>> df
Y1961 Y1962 Y1963 Y1964 Y1965 Region mean
0 82.567307 83.104757 83.183700 83.030338 82.831958 US 82.943612
1 2.699372 2.610110 2.587919 2.696451 2.846247 US 2.688020
2 14.131355 13.690028 13.599516 13.649176 13.649046 US 13.743824
3 0.048589 0.046982 0.046583 0.046225 0.051750 US 0.048026
4 0.553377 0.548123 0.582282 0.577811 0.620999 US 0.576518
We can find the the mean of a row using the range function, i.e in your case, from the Y1961 column to the Y1965
df['mean'] = df.iloc[:, 0:4].mean(axis=1)
And if you want to select individual columns
df['mean'] = df.iloc[:, [0,1,2,3,4].mean(axis=1)
You can try via filter() select columns Named like 'Week' then find mean and store that into a variable(for good performance) and finally fill NaN's by using fillna():
cols=df.filter(regex='Week').columns
m=df[cols].mean(axis=1).round()
df=df.fillna({x:m for x in cols})
output:
ID Feature Paid Week1 Week2 Week3 Week4
0 1 1 1 12 12.0 12.0 12.0
1 2 0 1 34 23.0 28.0 28.0
2 3 1 0 24 13.0 14.0 17.0
Create a dictionary that maps the Week names to the mean values of weeks along axis=1, then fill the NaN values using this dictionary
c = df.filter(like='Week').columns
df.fillna(dict.fromkeys(c, df[c].mean(1)))
ID Feature Paid Week1 Week2 Week3 Week4
0 1 1 1 12 12.0 12.0 12.0
1 2 0 1 34 23.0 28.5 28.5
2 3 1 0 24 13.0 14.0 17.0