Just to underline my comment to @maxymoo's answer, it's almost invariably a bad idea ("code smell") to add names dynamically to a Python namespace. There are a number of reasons, the most salient being:

1. Created names might easily conflict with variables already used by your logic.

2. Since the names are dynamically created, you typically also end up using dynamic techniques to retrieve the data.

This is why dicts were included in the language. The correct way to proceed is:

d = {}
for name in companies:
    d[name] = pd.DataFrame()

Nowadays you can write a single dict comprehension expression to do the same thing, but some people find it less readable:

d = {name: pd.DataFrame() for name in companies}

Once d is created the DataFrame for company x can be retrieved as d[x], so you can look up a specific company quite easily. To operate on all companies you would typically use a loop like:

for name, df in d.items():
    # operate on DataFrame 'df' for company 'name'

In Python 2 you were better writing

for name, df in d.iteritems():

because this avoids instantiating the list of (name, df) tuples that .items() creates in the older version. That's now largely of historical interest, though there will of course be Python 2 applications still extant and requiring (hopefully occasional) maintenance.

I think you think your code is doing something that it is not actually doing.

Specifically, this line: df = pd.read_csv(file)

You might think that in each iteration through the for loop this line is being executed and modified with df being replaced with a string in dfs and file being replaced with a filename in files. While the latter is true, the former is not.

Each iteration through the for loop is reading a csv file and storing it in the variable df effectively overwriting the csv file that was read in during the previous for loop. In other words, df in your for loop is not being replaced with the variable names you defined in dfs.

The key takeaway here is that strings (e.g., 'df1', 'df2', etc.) cannot be substituted and used as variable names when executing code.

One way to achieve the result you want is store each csv file read by pd.read_csv() in a dictionary, where the key is name of the dataframe (e.g., 'df1', 'df2', etc.) and value is the dataframe returned by pd.read_csv().

list_of_dfs = {}
for df, file in zip(dfs, files):
    list_of_dfs[df] = pd.read_csv(file)
    print(list_of_dfs[df].shape)
    print(list_of_dfs[df].dtypes)
    print(list(list_of_dfs[df]))

You can then reference each of your dataframes like this:

print(list_of_dfs['df1'])
print(list_of_dfs['df2'])

You can learn more about dictionaries here:

https://docs.python.org/3.6/tutorial/datastructures.html#dictionaries

I would generally discourage you from creating lots of variables with related names which is a dangerous design pattern in Python (although it's common in SAS for example). A better option would be to create a dictionary of dataframes with the key as your 'variable name'

df_dict = dict()
for df in 2011, 2012, 2013:
   df_dict["pivot_"+df.name] = pd.pivot_table(df, index=["income"], columns=["area"], values=["id"], aggfunc='count')

I'm assuming here that your dataframes have the names "2011", "2012", "2013"

This should do it:

for i in province_id:
    for j in year:
        locals()['sub_data_{}_{}'.format(i,j)] = data[(data.provid==i) & (data.wave==j)]

I initially suggested using exec, which is not usually considered best practice for safety reasons. Having said so, if your code is not exposed to anyone with malicious intentions, it should be OK, and I'll leave it here for the sake of completeness:

for i in province_id:
    for j in year:
        exec "sub_data_{}_{} = data[(data.provid==i) & (data.wave==j)]".format(i,j)

Nevertheless, for most use cases, it's probably better to use a collection of some sort, e.g. a dictionary, because it will be cumbersome to reference dynamically generated variable names in subsequent parts of your code. It's also a one-liner:

data_dict = {key:g for key,g in data.groupby(['provid','wave'])}

IIUC, I was able to achieve what you wanted.

import pandas as pd
import numpy as np

# source data for the dataframe
data = {
"ID":["x","y","z","x","y","z","x","y","a","b","x"],
"Date":["May 01","May 02","May 04","May 01","May 01","May 02","May 01","May 05","May 06","May 08","May 10"],
"Amount":[10,20,30,40,50,60,70,80,90,100,110]
}

df = pd.DataFrame(data)

# convert the Date column to datetime and still maintain the format like "May 01"
df['Date'] = pd.to_datetime(df['Date'], format='%b %d').dt.strftime('%b %d')

# sort the values on ID and Date
df.sort_values(by=['ID', 'Date'], inplace=True)
df.reset_index(inplace=True, drop=True)

print(df)

Original Dataframe:

    Amount    Date ID
0       90  May 06  a
1      100  May 08  b
2       10  May 01  x
3       40  May 01  x
4       70  May 01  x
5      110  May 10  x
6       50  May 01  y
7       20  May 02  y
8       80  May 05  y
9       60  May 02  z
10      30  May 04  z

.

# create a list of unique ids
list_id = sorted(list(set(df['ID'])))

# create an empty list that would contain dataframes
df_list = []

# count of iterations that must be seperated out
# for example if we want to record 3 entries for 
# each id, the iter would be 3. This will create
# three new dataframes that will hold transactions
# respectively. 
iter = 3
for i in range(iter):
    df_list.append(pd.DataFrame())


for val in list_id:
    tmp_df = df.loc[df['ID'] == val].reset_index(drop=True)

    # consider only the top iter(=3) values to be distributed
    counter = np.minimum(tmp_df.shape[0], iter)
    for idx in range(counter):
        df_list[idx] = df_list[idx].append(tmp_df.loc[tmp_df.index == idx])

for df in df_list:
    df.reset_index(drop=True, inplace=True)
    print(df)

Transaction #1:

   Amount    Date ID
0      90  May 06  a
1     100  May 08  b
2      10  May 01  x
3      50  May 01  y
4      60  May 02  z

Transaction #2:

   Amount    Date ID
0      40  May 01  x
1      20  May 02  y
2      30  May 04  z

Transaction #3:

   Amount    Date ID
0      70  May 01  x
1      80  May 05  y

Note that in your data, there are four transactions for 'x'. If lets say you wanted to track the 4th iterative transaction as well. All you need to do is change the value if 'iter' to 4 and you will get the fourth dataframe as well with the following value:

   Amount    Date ID
0     110  May 10  x

I got the answer which I was looking for

import pandas as pd
gbl = globals()
for i in locations:
gbl['df_'+i] = df[df.Start_Location==i]

This will create 3 data frames df_HOME, df_office and df_SHOPPING

Thanks,

You could save the dataframes to a list:

list_of_dfs = list()

for x in names:
    df = data[data['Name'].eq(x)].groupby('Part No.')['Quantity'].sum().rename("QTY").to_frame()
    list_of_dfs.append(df)

Consider using a dictionary of data frames with dst as the key building an inner list of data frames that are concatenated outside the loop:

df_dict = {}

for dst, dstkey in zip(Dst, DstKey):
    inner = []
    with arcpy.da.SearchCursor(dst, ("OBJECTID", dstkey)) as cursor:
        # returns an iterator of tuples
        for row in cursor:
            if (row[1] is None or not str(row[1]).strip()):
                inner.append(pd.DataFrame({dst.split("\\").pop(): str(row[0])}, index=[0])

    df_dict[dstkey] = pd.concat(inner, ignore_index=True)

Alternatively with list comprehension:

df_dict = {}

for dst, dstkey in zip(Dst, DstKey):
    with arcpy.da.SearchCursor(dst, ("OBJECTID", dstkey)) as cursor:
        # returns an iterator of tuples
        inner = [pd.DataFrame({dst.split("\\").pop(): str(row[0])}, index=[0]) 
                 for row in cursor if (row[1] is None or not str(row[1]).strip())]

    df_dict[dstkey] = pd.concat(inner, ignore_index=True)

For Excel export using dictionary of data frames:

writer = pd.ExcelWriter('/path/to/output.xlsx')

for i, df in df_dict.items():
   df.to_excel(writer, sheet_name=i)

writer.save()

First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only.

Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns.

With these two consideration, assuming you have defined your objects as follows:

df_a = {
    'name': 'df_a',
    'value': pd.DataFrame([[1, 2, 3, 4], [2, 4, 6, 8]])
}
df_b = {
    'name': 'df_b',
    'value': pd.DataFrame([[10, 5, 2, 1], [4, 4, 6, 2]])
}

ref_1 = {
    'name': 'ref_1',
    'value': pd.DataFrame([[5, 5, 5, 5], [5, 5, 5, 5]])
}
ref_2 = {
    'name': 'ref_b',
    'value': pd.DataFrame([[3, 3, 3, 3], [3, 3, 3, 3]])
}

I did this because I want to use the names in creating the name of the columns of your final df. Then your code would be:

from itertools import product

final_result = pd.DataFrame(
    {
        '{}_{}'.format(df['name'], ref['name']): (df['value']-ref['value']).sum(axis=1)
        for (df, ref) in product([df_a, df_b], [ref_1, ref_2])
    }
)

• I have used dictionary comprehension to skip the ugly loop/append solution.
• product function from itertools does your iteration. product on (ab, cd) gives you ac, ad, bc, bd
• as for keys, df names are joined together with _, and as for values, I have subtracted two dfs and sum over columns (axis=1)

The result would then be as you expect:

   df_a_ref_1  df_a_ref_b  df_b_ref_1  df_b_ref_b
0         -10          -2          -2           6
1           0           8          -4           4

Still if you want to expand the dictionary comprehension or do not want to define dictionaries of names/values, of course you can imagine how you can write simple for loops with the same logic:

for (df, ref) in product([df_a, df_b], [ref_1, ref_2]):
    # your desired columns
    col = (df - ref).sum(axis=1)