Break this up into three parts to help isolate the problem and improve readability:

  1. Build the SQL string
  2. Set parameter values
  3. Execute pandas.read_sql_query

Build SQL

First ensure ? placeholders are being set correctly. Use str.format with str.join and len to dynamically fill in ?s based on member_list length. Below examples assume 3 member_list elements.

Example

member_list = (1,2,3)
sql = """select member_id, yearmonth
         from queried_table
         where yearmonth between {0} and {0}
         and member_id in ({1})"""
sql = sql.format('?', ','.join('?' * len(member_list)))
print(sql)

Returns

select member_id, yearmonth
from queried_table
where yearmonth between ? and ?
and member_id in (?,?,?)

Set Parameter Values

Now ensure parameter values are organized into a flat tuple

Example

# generator to flatten values of irregular nested sequences,
# modified from answers http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python
def flatten(l):
    for el in l:
        try:
            yield from flatten(el)
        except TypeError:
            yield el

params = tuple(flatten((201601, 201603, member_list)))
print(params)

Returns

(201601, 201603, 1, 2, 3)

Execute

Finally bring the sql and params values together in the read_sql_query call

query = pd.read_sql_query(sql, db2conn, params)
Answer from Bryan on Stack Overflow
Top answer
1 of 3
27

Break this up into three parts to help isolate the problem and improve readability:

  1. Build the SQL string
  2. Set parameter values
  3. Execute pandas.read_sql_query

Build SQL

First ensure ? placeholders are being set correctly. Use str.format with str.join and len to dynamically fill in ?s based on member_list length. Below examples assume 3 member_list elements.

Example

member_list = (1,2,3)
sql = """select member_id, yearmonth
         from queried_table
         where yearmonth between {0} and {0}
         and member_id in ({1})"""
sql = sql.format('?', ','.join('?' * len(member_list)))
print(sql)

Returns

select member_id, yearmonth
from queried_table
where yearmonth between ? and ?
and member_id in (?,?,?)

Set Parameter Values

Now ensure parameter values are organized into a flat tuple

Example

# generator to flatten values of irregular nested sequences,
# modified from answers http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python
def flatten(l):
    for el in l:
        try:
            yield from flatten(el)
        except TypeError:
            yield el

params = tuple(flatten((201601, 201603, member_list)))
print(params)

Returns

(201601, 201603, 1, 2, 3)

Execute

Finally bring the sql and params values together in the read_sql_query call

query = pd.read_sql_query(sql, db2conn, params)
2 of 3
15

WARNING! Although my proposed solution here works, it is prone to SQL injection attacks. Therefor, it should never be used directly in backend code! It is only safe for offline analysis.

If you're using python 3.6+ you could also use a formatted string litteral for your query (cf https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep498)

start, end = 201601, 201603
selected_members = (111, 222, 333, 444, 555)  # requires to be a tuple

query = f"""
    SELECT member_id, yearmonth FROM queried_table
    WHERE yearmonth BETWEEN {start} AND {end}
      AND member_id IN {selected_members}
"""

df = pd.read_sql_query(query, db2conn)
🌐
Like Geeks
likegeeks.com › home › python › pandas › using the pandas read_sql params for dynamic data retrieval
Using the Pandas read_sql params for Dynamic Data Retrieval
December 9, 2023 - query = "SELECT * FROM call_logs WHERE network IN (:networks)" params = handle_params(networks=('Verizon', 'AT&T')) df = pd.read_sql(query, conn, params=params) In this scenario, handle_params would need to be extended to properly format lists into a tuple, as SQLite requires for IN clauses. There are two primary placeholder syntaxes used in SQL queries: %s and :name. The %s syntax is often used with databases like MySQL and PostgreSQL. It represents positional parameters in your query. import pandas as pd import psycopg2 # assuming we're using PostgreSQL conn = psycopg2.connect("dbname=telecom_data user=your_username") query = "SELECT * FROM call_logs WHERE duration > %s AND network = %s" params = [60, 'Verizon'] df = pd.read_sql(query, conn, params=params)
Discussions

python - How to pass a list of parameter to Pandas read_sql with Teradata - Stack Overflow
Is there a way or how can I correct my syntax in order to pass a list of parameter of string into SQL query in pandas? I have the following code but it it not working So i have a list of string and I More on stackoverflow.com
🌐 stackoverflow.com
read_sql should accept a sql_params parameter
You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. ... There was an error while loading. Please reload this page. ... I wonder if current use of read_sql couldn't lead to SQL injection. More on github.com
🌐 github.com
13
August 24, 2015
python - pandas read_sql_query with params matching multiple columns - Stack Overflow
I'm trying to query a table using pandas.read_sql_query, where I want to match multiple columns to python lists passed in as param arguments. Running into various psycopg2 errors when trying to More on stackoverflow.com
🌐 stackoverflow.com
Pandas Parametized Query
From the pandas.read_sql() documentation , it looks like you need the params argument. Probably like this: df1 = pd.read_sql("select * from orders where datecreated > %s and datecreated < %s ", params=(startdate, currentdate), con=db) More on reddit.com
🌐 r/learnpython
2
1
October 25, 2022
Top answer
1 of 2
134

The read_sql docs say this params argument can be a list, tuple or dict (see docs).

To pass the values in the sql query, there are different syntaxes possible: ?, :1, :name, %s, %(name)s (see PEP249).
But not all of these possibilities are supported by all database drivers, which syntax is supported depends on the driver you are using (psycopg2 in your case I suppose).

In your second case, when using a dict, you are using 'named arguments', and according to the psycopg2 documentation, they support the %(name)s style (and so not the :name I suppose), see http://initd.org/psycopg/docs/usage.html#query-parameters.
So using that style should work:

df = psql.read_sql(('select "Timestamp","Value" from "MyTable" '
                     'where "Timestamp" BETWEEN %(dstart)s AND %(dfinish)s'),
                   db,params={"dstart":datetime(2014,6,24,16,0),"dfinish":datetime(2014,6,24,17,0)},
                   index_col=['Timestamp'])
2 of 2
-1

I was having trouble passing a large number of parameters when reading from a SQLite Table. Then it turns out since you pass a string to read_sql, you can just use f-string. Tried the same with MSSQL pyodbc and it works as well.

For SQLite, it would look like this:

# write a sample table into memory
from sqlalchemy import create_engine
df = pd.DataFrame({'Timestamp': pd.date_range('2020-01-17', '2020-04-24', 10), 'Value1': range(10)})
engine = create_engine('sqlite://', echo=False)
df.to_sql('MyTable', engine);

# query the table using a query
tpl = (1, 3, 5, 8, 9)
query = f"""SELECT Timestamp, Value1 FROM MyTable WHERE Value1 IN {tpl}"""
df = pd.read_sql(query, engine)

If the parameters are datetimes, it's a bit more complicated but calling the datetime conversion function of the SQL dialect you're using should do the job.

start, end = '2020-01-01', '2020-04-01'
query = f"""SELECT Timestamp, Value1 FROM MyTable WHERE Timestamp BETWEEN STRFTIME("{start}") AND STRFTIME("{end}")"""
df = pd.read_sql(query, engine)
🌐
Plain English
plainenglish.io › home › blog › python › pandas read_sql with variable parameters — alternative way
Pandas Read_Sql With Variable Parameters — Alternative Way
November 26, 2022 - First, we need our SP. In this example, name of my SQL — related things are below. ... In this example, our SP is taking 2 date parameters: StartDate and EndDate. ... Now, we are ready to write our query in Python. We will use an awesome library: Pandas.
🌐
Like Geeks
likegeeks.com › home › python › pandas › read sql query/table into dataframe using pandas read_sql
Read SQL Query/Table into DataFrame using Pandas read_sql
October 16, 2023 - params: An optional list or dictionary of parameters to pass into the sql query. parse_dates: An optional parameter to parse columns into datetime. columns: If you’re reading a table (not a query), this allows you to select which columns to ...
🌐
Pandas
pandas.pydata.org › docs › reference › api › pandas.read_sql.html
pandas.read_sql — pandas 3.0.1 documentation - PyData |
index_colstr or list of str, optional, default: None · Column(s) to set as index(MultiIndex). ... Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. paramslist, tuple or dict, optional, default: None
🌐
Pandas
pandas.pydata.org › docs › dev › reference › api › pandas.read_sql.html
pandas.read_sql — pandas 3.0.0rc0+27.g47fea804d6 documentation
index_colstr or list of str, optional, default: None · Column(s) to set as index(MultiIndex). ... Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. paramslist, tuple or dict, optional, default: None
🌐
Pandas
pandas.pydata.org › pandas-docs › version › 1.5 › reference › api › pandas.read_sql.html
pandas.read_sql — pandas 1.5.2 documentation
index_colstr or list of str, optional, default: None · Column(s) to set as index(MultiIndex). ... Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. paramslist, tuple or dict, optional, default: None
Find elsewhere
🌐
CSDN
devpress.csdn.net › python › 630458547e6682346619a401.html
Binding list to params in Pandas read_sql_query with other params_python_Mangs-Python
August 23, 2022 - query = pd.read_sql_query( """ select member id ,yearmonth from queried_table where yearmonth between ? and ? and member_id in (%s) """ % ','.join(['?']*len(member_list), db2conn, params = [201601, 201603, tuple(member_list)])
Top answer
1 of 2
3

You have a couple of issues:

  1. The package name teradatasql is misspelled in your example tearadatasql.connect
  2. You must compose the IN-predicate with the same number of question-mark parameter markers as the number of values you intend to bind.

In your example, you intend to bind the three values contained in the column_list variable, so you must compose the IN-predicate with three question-mark parameter markers.

Generally speaking, you should dynamically compose the IN-predicate with the number of question-mark parameter markers equal to the number of values in the parameter-value list that you will bind.

Below is a modified version of your example that corrects these two issues. I actually ran this example and verified that it works.

import teradatasql
import pandas as pd 
with teradatasql.connect(host="whomooz",user="guest",password="please") as connection:
  with connection.cursor() as cur:
    cur.execute("create volatile table table1 (c1 varchar(1), c2 integer) on commit preserve rows")
    cur.execute("insert into table1 values ('A', 1) ; insert into table1 values ('B', 2)")
  column_list = ['A','B','C']
  query = "select c1, c2 from table1 where c1 in ({}) order by c1".format(','.join(['?'] * len(column_list)))
  print(query)
  print("with params={}".format (column_list))
  df = pd.read_sql(query, connection, params=column_list)
  print(df)

This example produces the following output:

select c1, c2 from table1 where c1 in (?,?,?) order by c1
with params=['A', 'B', 'C']
  c1  c2
0  A   1
1  B   2
2 of 2
2

You should fill in the %s with some parameters

df = psql.read_sql(('select "column","row" from "table1" '
                 'where "column" in %(col_list)s'), connection, params={'col_list':column_list})
🌐
GitHub
github.com › pandas-dev › pandas › issues › 10899
read_sql should accept a sql_params parameter · Issue #10899 · pandas-dev/pandas
August 24, 2015 - # Never do this -- insecure! symbol = 'RHAT' c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol) # Do this instead t = ('RHAT',) c.execute('SELECT * FROM stocks WHERE symbol=?', t) print c.fetchone() # Larger example that inserts many records at a time purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00), ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00), ('2006-04-06', 'SELL', 'IBM', 500, 53.00), ] c.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
Author   s-celles
🌐
Pandas
pandas.pydata.org › docs › reference › api › pandas.read_sql_query.html
pandas.read_sql_query — pandas 3.0.1 documentation
Column(s) to set as index(MultiIndex). ... Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point. Useful for SQL result sets. paramslist, tuple or mapping, optional, default: None · List of parameters to pass to execute method.
🌐
Reddit
reddit.com › r/learnpython › pandas parametized query
r/learnpython on Reddit: Pandas Parametized Query
October 25, 2022 -

Hello Guys,

I'm trying to query data with variables using the select statement with pandas and MySQL. Is there a way I can declare the date variable and pass them to the query during runtime. I haven't come across any ways that work from my online research.

Here is my code :

from datetime import datetime
from email import encoders
import smtplib
import pandas as pd
from sqlalchemy import create_engine
from urllib.parse import quote  
import mysql.connector as sql

db = create_engine('mysql://root:%s@localhost:3306/store' % quote('Mypass@12!'))

now = datetime.now()
startdate = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0).strftime("%y-%m-%d %H:%M:%S")
currentdate = now.replace(day=3,hour=23, minute=00, second=0).strftime("%y-%m-%d %H:%M:%S")

df1 = pd.read_sql("select * from orders where datecreated > %s and datecreated < %s ", con=db)

pdwriter = pd.ExcelWriter('report.xlsx', engine='xlsxwriter')
df1.to_excel(pdwriter, sheet_name='GEN')
pdwriter.save()

I would like to pass the date variables to the query.

Any suggestion is greatly appreciated. Thank you.

🌐
Python Forum
python-forum.io › thread-18705.html
Passing parameters to SQL Server using pandas
Hello Python forum, I'm new to python world. Please bear with me if my question sounds silly. My problem statement : Passing parameter to SQL server using pandas. Note: Have imported all the necessary library for pandas,datetime,pyodbc in my cod...
🌐
Pandas
pandas.pydata.org › pandas-docs › version › 0.19.2 › generated › pandas.read_sql_query.html
pandas.read_sql_query — pandas 0.19.2 documentation
pandas.read_sql_query(sql, con, index_col=None, coerce_float=True, params=None, parse_dates=None, chunksize=None)[source]¶
🌐
Saturn Cloud
saturncloud.io › blog › how-to-bind-list-to-params-in-pandas-readsqlquery-with-other-params
How to Bind List to Params in Pandas readsqlquery with Other Params | Saturn Cloud Blog
August 25, 2023 - We then define our SQL query and use the .format method to replace the {} placeholder with a comma-separated list of question marks, one for each element in the list. We then create a numpy array from our list using the np.array method and pass it as a parameter to the Pandas read_sql_query method.
🌐
Readthedocs
pyhelpers.readthedocs.io › en › latest › _generated › pyhelpers.dbms.PostgreSQL.read_sql_query.html
PostgreSQL.read_sql_query - PyHelpers - Read the Docs
>>> import datetime >>> import pandas as pd >>> sql_qry = 'SELECT * FROM "table_name" ' ... 'WHERE "timestamp_column_name" BETWEEN %(ts_start)s AND %(ts_end)s' >>> params = {'d_start': datetime.datetime.today(), 'd_end': datetime.datetime.today()} >>> data_frame = pd.read_sql(sql=sql_qry, con=testdb.engine, params=params)
🌐
Pandas
pandas.pydata.org › pandas-docs › stable › reference › api › pandas.read_sql.html
pandas.read_sql — pandas 2.2.3 documentation - PyData |
index_colstr or list of str, optional, default: None · Column(s) to set as index(MultiIndex). ... Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets. paramslist, tuple or dict, optional, default: None