I think the best is create dictionary of DataFrames:
d = {}
for i in range(12,0,-1):
d['t' + str(i)] = df.shift(i).add_suffix('_t' + str(i))
If need specify columns first:
d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
d['t' + str(i)] = df[cols].shift(i).add_suffix('_t' + str(i))
dict comprehension solution:
d = {'t' + str(i): df.shift(i).add_suffix('_t' + str(i)) for i in range(12,0,-1)}
print (d['t10'])
column1_t10 column2_t10
0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 0.0 19.0
11 1.0 18.0
12 2.0 17.0
13 3.0 16.0
14 4.0 15.0
15 5.0 14.0
16 6.0 13.0
17 7.0 12.0
18 8.0 11.0
19 9.0 10.0
EDIT: Is it possible by globals, but much better is dictionary:
d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
globals()['df' + str(i)] = df[cols].shift(i).add_suffix('_t' + str(i))
print (df10)
column1_t10 column2_t10
0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 0.0 19.0
11 1.0 18.0
12 2.0 17.0
13 3.0 16.0
14 4.0 15.0
15 5.0 14.0
16 6.0 13.0
17 7.0 12.0
18 8.0 11.0
19 9.0 10.0
Top answer 1 of 2
13
I think the best is create dictionary of DataFrames:
d = {}
for i in range(12,0,-1):
d['t' + str(i)] = df.shift(i).add_suffix('_t' + str(i))
If need specify columns first:
d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
d['t' + str(i)] = df[cols].shift(i).add_suffix('_t' + str(i))
dict comprehension solution:
d = {'t' + str(i): df.shift(i).add_suffix('_t' + str(i)) for i in range(12,0,-1)}
print (d['t10'])
column1_t10 column2_t10
0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 0.0 19.0
11 1.0 18.0
12 2.0 17.0
13 3.0 16.0
14 4.0 15.0
15 5.0 14.0
16 6.0 13.0
17 7.0 12.0
18 8.0 11.0
19 9.0 10.0
EDIT: Is it possible by globals, but much better is dictionary:
d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
globals()['df' + str(i)] = df[cols].shift(i).add_suffix('_t' + str(i))
print (df10)
column1_t10 column2_t10
0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 0.0 19.0
11 1.0 18.0
12 2.0 17.0
13 3.0 16.0
14 4.0 15.0
15 5.0 14.0
16 6.0 13.0
17 7.0 12.0
18 8.0 11.0
19 9.0 10.0
2 of 2
2
for i in range(1, 16):
text=f"Version{i}=pd.DataFrame()"
exec(text)
A combination of exec and f"..." will help you do that.
If you need iterating or Versions of same variable above statement will help
Top answer 1 of 2
2
I hope this will help, as i understood from this.
gbl = globals()
lst = ['SymbolA','SymbolB', 'SymbolC' .... 'SymbolN']
for i in lst:
data = SomeFunction(lst[i])
gbl[lst[i]+str(i)] = pd.Dataframe(data)
this will create a df dynamically . for accessing those df you need to run code like this.
gbl[lst[i]+str(i)]
try this..
2 of 2
0
You input has to be like below:
lst = ({'data':['SymbolA','SymbolB', 'SymbolC', 'SymbolN']})
print pd.DataFrame(lst)
python - Create new dataframe in pandas with dynamic names also add new column - Stack Overflow
My requirement is to call test ... seperate dataframe of that yearmonth.it would be helpful if you can explain me with example what exaclty you are trying to say 2016-12-05T12:55:29.797Z+00:00 ... Is creating dynamically named variables even possible in python?... More on stackoverflow.com
Using Pandas in Glue ETL Job ( How to convert Dynamic DataFrame or PySpark Dataframe to Pandas Dataframe)
I am wanting to use Pandas in a Glue ETL job. I am reading from S3 and writing to Data Catalog. I am trying to find a basic example where I can read in from S3 , either into or converting to a Pa... More on repost.aws
1
0
April 29, 2022Dynamically assigning name of dataframe in a loop. Stuck!
This won't work. Use a dictionary. More on reddit.com
5
5
February 2, 2018python - How to create a dynamic dataframe - Stack Overflow
I was trying to create a data frame and the reason why I gave the create a data frame in the below manner is to make it dynamic but the expression is passed as a string and the exec command is not ... More on stackoverflow.com
Videos
08:45
Create Dynamic Dataframes in PySpark - YouTube
10:39
How To Create a DataFrame Dynamically in Python #coding #python ...
12:34
5. Pandas - Read and Write Multiple Sheet Dynamically - YouTube
08:45
Create Dynamic DataFrame | Exec Method | Dynamic DataFrame - YouTube
AskPython
askpython.com › home › python pandas dynamically create a dataframe
Python Pandas Dynamically Create a Dataframe - AskPython
July 20, 2023 - Dynamically creating a dataframe is important in cases where we don’t know the size of the dataframe when we create it, or maybe you would like to rename the headers dynamically without a tedious process in the background.
Top answer 1 of 2
24
Creating variables with dynamic names is typically a bad practice.
I think the best solution for your problem is to store your dataframes into a dictionary and dynamically generate the name of the key to access each dataframe.
import copy
dict_of_df = {}
for ym in [201511, 201612, 201710]:
key_name = 'df_new_'+str(ym)
dict_of_df[key_name] = copy.deepcopy(df)
to_change = df['YearMonth']< ym
dict_of_df[key_name].loc[to_change, 'new_col'] = ym
dict_of_df.keys()
Out[36]: ['df_new_201710', 'df_new_201612', 'df_new_201511']
dict_of_df
Out[37]:
{'df_new_201511': A B ID t YearMonth new_col
0 -a a 1 2016-12-05 07:53:35.943 201612 201612
1 1 NaN 2 2016-12-05 07:53:35.943 201612 201612
2 a c 2 2016-12-05 07:53:35.943 201612 201612,
'df_new_201612': A B ID t YearMonth new_col
0 -a a 1 2016-12-05 07:53:35.943 201612 201612
1 1 NaN 2 2016-12-05 07:53:35.943 201612 201612
2 a c 2 2016-12-05 07:53:35.943 201612 201612,
'df_new_201710': A B ID t YearMonth new_col
0 -a a 1 2016-12-05 07:53:35.943 201612 201710
1 1 NaN 2 2016-12-05 07:53:35.943 201612 201710
2 a c 2 2016-12-05 07:53:35.943 201612 201710}
# Extract a single dataframe
df_2015 = dict_of_df['df_new_201511']
2 of 2
1
There is a more easy way to accomplish this using exec method. The following steps can be done to create a dataframe at runtime.
1.Create the source dataframe with some random values.
import numpy as np
import pandas as pd
df = pd.DataFrame({'A':['-a',1,'a'],
'B':['a',np.nan,'c'],
'ID':[1,2,2]})
2.Assign a variable that holds the new dataframe name. You can even send this value as a parameter or loop it dynamically.
new_df_name = 'df_201612'
3.Create dataframe dynamically using exec method to copy data from source dataframe to the new dataframe dynamically and in the next line assign a value to new column.
exec(f'{new_df_name} = df.copy()')
exec(f'{new_df_name}["new_col"] = 123')
4.Now the dataframe df_201612 will be available on the memory and you can execute print statement along with eval to verify this.
print(eval(new_df_name))
AWS
docs.aws.amazon.com › aws glue › user guide › aws glue programming guide › programming spark scripts › program aws glue etl scripts in pyspark › aws glue pyspark extensions reference › dynamicframe class
DynamicFrame class - AWS Glue
It is similar to a row in a Spark DataFrame, except that it is self-describing and can be used for data that does not conform to a fixed schema. options – A list of ResolveOption objects that specify how to resolve choice types during the conversion. This parameter is used to handle schema inconsistencies, not for format options like CSV parsing. For CSV parsing and other format options, specify these in the from_options method when creating the DynamicFrame, not in the toDF method.
Towards Data Science
towardsdatascience.com › home › latest › data wrangling solutions – dynamically creating variables when slicing dataframes
Data Wrangling Solutions - Dynamically Creating Variables When Slicing Dataframes | Towards Data Science
January 29, 2025 - Therefore, we will provide a workaround solution to use the Python dictionaries. The keys in this dictionary will be the different categories of the variable. The value component of the dictionary will be the dataframe slice itself.
Pandas
pandas.pydata.org › pandas-docs › version › 0.24.2 › user_guide › cookbook.html
Cookbook — pandas 0.24.2 documentation
These examples are written for Python 3. Minor tweaks might be necessary for earlier python versions. ... In [1]: df = pd.DataFrame({'AAA': [4, 5, 6, 7], ...: 'BBB': [10, 20, 30, 40], ...: 'CCC': [100, 50, -30, -50]}) ...: In [2]: df Out[2]: AAA BBB CCC 0 4 10 100 1 5 20 50 2 6 30 -30 3 7 40 -50
AWS re:Post
repost.aws › questions › QU51ax-cVsROSuk9-tWgOhQQ › using-pandas-in-glue-etl-job-how-to-convert-dynamic-dataframe-or-pyspark-dataframe-to-pandas-dataframe
Using Pandas in Glue ETL Job ( How to convert Dynamic DataFrame or PySpark Dataframe to Pandas Dataframe) | AWS re:Post
April 29, 2022 - Would say convert Dynamic frame to Spark data frame using .ToDF() method and from spark dataframe to pandas dataframe using link https://sparkbyexamples.com/pyspark/convert-pyspark-dataframe-to-pandas/#:~:text=Convert PySpark Dataframe to Pandas DataFrame,small subset of the data.
Medium
vnimmana.medium.com › creating-dynamic-dataframes-using-dictionary-d2b0dfd3262c
Creating Dynamic Dataframes Using Dictionary | by Vijay Krishna Nimmana | Medium
December 27, 2020 - Creating Dynamic Dataframes Using Dictionary In this tutorial, we will discuss about reading multiple files in to dataframes and append all the files to form a single dataframe. Photo by Fré …
Reddit
reddit.com › r/learnpython › dynamically assigning name of dataframe in a loop. stuck!
r/learnpython on Reddit: Dynamically assigning name of dataframe in a loop. Stuck!
February 2, 2018 -
Going pseudo-code this out, perhaps somebody has encountered this sort of issue before. Have not had luck reading through stackoverflow posts.
I have a list of months and a df for each month with data that includes delivery volume and a time. These named like 'df_1701_unfiltered'.
I previously hardcoded my query logic, but on mobile now. That's not what I'm worried about so please disregard the pseudo aspect (I'm on mobile atm).
I want to create a new, separate dataframe for each month that is a filtered version of the original. Here is my thought process.
months = ['1701', '1702', '1703']
For month in month: "df_"+month+"filtered" = "df"+month+"_unfiltered".query("time > start and time < end")
I'm able to do something similar within a single dataframe using .apply to create dynamic columns. It throws an "cannot assign to operator" error each time.
Any idea how I can do this for entire dataframes?
Top answer 1 of 3
1
You could give a try with a config file like below
import json
files = json.loads('{
"fileA": {
"header": "true",
"inputFileType": "csv",
"sourceFilePath": "path_to_fileA"
},
"fileB": {
"header": "true",
"inputFileType": "parquet",
"sourceFilePath": "path_to_fileB"
}
}')
df_dict = {}
for file in files:
df_dict['file'] = spark.read.option('header',file["header"]).format(file["inputFileType"]).path(file["sourceFilePath"])
Then you get a dictionary of dataframes with different formats and file path.
2 of 3
1
Hey got the answer...
def fileReader(inputFileType,sourceFilePath,inputFileType):
value ='true'
header='header'
a= "spark.read.option('"+header+"','"+value+"')."+inputFileType+"
('"+sourceFilePath+"')"
print(a)
print(type(a))
ds = eval(a)
return 'True'
the motive behind creating this function is to dynamically create data frames using different formats of files supported by pyspark. Now using this function I can create a data frame any type of file formats that are supported by pyspark, by just passing the location and the format of files.
I appreciate all the help.
Python.org
discuss.python.org › python help
Dynamically Creating Column and setting it's value based on values present at other columns - Python Help - Discussions on Python.org
May 2, 2023 - Below i am trying to mention my requirement. In the below screenshot, the yellow columns are the columns which i have to create dynamically based on response in other columns. Q1_SL is corresponding to Q1_1, Q1_2, and Q1_3. Q1_SL should be set as 1 only in case all the three columns have same response else will be set as 0. Same logic needs to be applied for Q2_SL.
Python.org
discuss.python.org › python help
How can i access dynamic dataframe that are created using global() function? - Python Help - Discussions on Python.org
June 17, 2022 - How can i access dynamic dataframe that are created using global() function · Using the function globals() you can access the variables in the current module scope (also called global variables). The function returns a dictionary. Keys of the dictionary are the variable names · I do not know ...
GeeksforGeeks
geeksforgeeks.org › different-ways-to-create-pandas-dataframe
Different ways to create Pandas Dataframe - GeeksforGeeks
January 2, 2025 - It is the most commonly used Pandas object. The pd.DataFrame() function is used to create a DataFrame in Pandas. There are several ways to create a Pandas Dataframe in Python.Example: Creating a DataFrame from a DictionaryPythonimport pandas as pd # initialize data of lists.
Saturn Cloud
saturncloud.io › blog › how-to-create-a-new-dataframe-in-pandas-with-dynamic-names-and-add-a-new-column
How to Create a New DataFrame in Pandas with Dynamic Names and Add a New Column | Saturn Cloud Blog
September 8, 2023 - Sometimes, you may want to create a new DataFrame with a dynamic name, such as when you are creating multiple DataFrames in a loop. To do this, you can use Python’s string formatting to generate a new name for each DataFrame.
Reddit
reddit.com › r/learnpython › trying to dynamically name and reference a dataframe, getting error 'syntaxerror: can't assign to function call'
r/learnpython on Reddit: trying to dynamically name and reference a dataframe, getting error 'SyntaxError: can't assign to function call'
May 5, 2017 -
#define list of fields to run match for
fieldlist = ['MATTER NUMBER','MATTER NAME','CLAIM NUMBER LISTING']
#loop through each field in fieldlist
for field in fieldlist:
#define dfname as the field with spaces replaced with underscores
dfname = '{}'.format(field.replace(' ','_'))
#create df with dfname
'{}'.format(dfname) = checkdf['{}'.format(field)].dropna()the error is on the last line. I also tried:
'{}'.format(dfname) = checkdf['{}'.format(field)].dropna() Top answer 1 of 3
3
General rule of thumb - if you need dynamically created variable names - you are doing something wrong. Even though it is possible, doing so would be an awfully bad idea, ie how will you then reference these variables later in your code?
2 of 3
2
You'll probably want a dict of dataframes e.g. dfs = {} ... dfs[dfname] = checkdf[field].dropna() '{}'.format(string) is just the same as string you don't need '{}'.format()
Quora
quora.com › How-do-you-dynamically-select-columns-in-pandas-based-on-input-passed
How to dynamically select columns in pandas based on input passed - Quora
Answer: In pandas you declare a dataframe as: import pandas as pd df=pd.DataFrame({'one’:[1,2,3],'two':[4,5,6]} Now if you want to access one of the column dynamically, we can use a function passing the dataframe and the column as argument, where you can have the column as your input. def sel...
IncludeHelp
includehelp.com › python › dynamically-filtering-a-pandas-dataframe.aspx
Python - Dynamically filtering a pandas dataframe
November 15, 2022 - Finally, we have a method called pandas.DataFrame.query() inside which we can pass any comprehended query and it will dynamically calculate all the results.
HubSpot
blog.hubspot.com › website › creating-pandas-dataframe
How to Create a Pandas Dataframe in Python
July 4, 2022 - Then, if that list is dynamically generated based on user input, you can store that list in a variable and use that to declare the indexes. id_names = [ 'Bartender', 'Investor', 'Janitor' ] df = pd.DataFrame(data, index = id_names)
Stack Overflow
stackoverflow.com › questions › 27077180 › pandas-dynamic-dataframe
python - Pandas dynamic dataframe - Stack Overflow
November 22, 2014 - data = pd.DataFrame({"Point1":["Red","Red","Orange","Green","Yellow","Black"],"Point2": ["Blue","Yellow","Green","Black","Black","Orange"]}) data["Red"] = [random.randint(-10,10) for p in range(0,6)] data["Blue"] = [random.randint(-10,10) for p in range(0,6)] data["Yellow"] = [random.randint(-10,10) for p in range(0,6)] data["Green"] = [random.randint(-10,10) for p in range(0,6)] data["Orange"] = [random.randint(-10,10) for p in range(0,6)] data["Black"] = [random.randint(-10,10) for p in range(0,6)] data["Number"] = [8,6,np.NaN,np.NaN,np.NaN,np.NaN]