You need to use the params keyword argument:

f = pd.read_sql_query('SELECT open FROM NYSEMSFT WHERE date = (?)', conn, params=(date,))
Answer from alecxe on Stack Overflow
🌐
Reddit
reddit.com › r/learnpython › how do i use a python variable in a where clause pd.read_sql_query query
r/learnpython on Reddit: How do i use a python variable in a where clause pd.read_sql_query query
January 17, 2020 -

Hi all,

As per heading I am creating a data frame using SQL that looks something like this:

SQL_Query = pd.read_sql_query(
"""SELECT TOP (1000)
DATA.NAME,
DATA.AGE,
DATA.ADDRESS,
DATA2.COUNTRY
FROM DATA
LEFT JOIN DATA2
ON DATA.NAME=DATA2.NAME
WHERE COUNTRY = (PYTHON VARIABLE )

What is the proper way to do this? I have seen online using %s as below

ursor1.execute("SELECT * FROM People WHERE Name=%s", (myName,))

but this doesn't seem to work. Not sure if it's because I'm using the pd.read_sql_query function.

Any help would be great :)

EDIT: worked it out :)

(Select * from data WHERE data.field = ?),con,Params=[variable])

Thank you all

Top answer
1 of 3
10

You need to use the params keyword argument:

f = pd.read_sql_query('SELECT open FROM NYSEMSFT WHERE date = (?)', conn, params=(date,))
2 of 3
5

As @alecxe and @Ted Petrou have already said, use explicit parameter names, especially for the params parameter as it's a fourth parameter in the pd.read_sql_query() function and you used it as a third one (which is coerce_float)

But beside that you can improve your code by getting rid of the for date in dates: loop using the following trick:

import sqlite3

dates=['2001-01-01','2002-02-02']
qry = 'select * from aaa where open in ({})'

conn = sqlite3.connect(r'D:\temp\.data\a.sqlite')

df = pd.read_sql(qry.format(','.join(list('?' * len(dates)))), conn, params=dates)

Demo:

Source SQLite table:

sqlite> .mode column
sqlite> .header on
sqlite> select * from aaa;
open
----------
2016-12-25
2001-01-01
2002-02-02

Test run:

In [40]: %paste
dates=['2001-01-01','2002-02-02']
qry = 'select * from aaa where open in ({})'
conn = sqlite3.connect(r'D:\temp\.data\a.sqlite')

df = pd.read_sql(qry.format(','.join(list('?' * len(dates)))), conn, params=dates)
## -- End pasted text --

In [41]: df
Out[41]:
         open
0  2001-01-01
1  2002-02-02

Explanation:

In [35]: qry = 'select * from aaa where open in ({})'

In [36]: ','.join(list('?' * len(dates)))
Out[36]: '?,?'

In [37]: qry.format(','.join(list('?' * len(dates))))
Out[37]: 'select * from aaa where open in (?,?)'

In [38]: dates.append('2003-03-03')   # <-- let's add a third parameter

In [39]: qry.format(','.join(list('?' * len(dates))))
Out[39]: 'select * from aaa where open in (?,?,?)'
🌐
Stack Overflow
stackoverflow.com › questions › 50837627 › use-variable-in-pd-read-sql
python - Use variable in pd.read.sql - Stack Overflow
This is never a good idea, it opens up the code for SQL injection attacks, and robs the database of an opportunity to reuse optimised queries. Use ? or %s placeholders (depends on the database driver) and pass in the value to read_sql() via params=(x,). 2019-01-15T12:30:35.157Z+00:00 ... var = "953381" df = pd.read_sql('select name, data, from sheet where name =' + var, conn, index_col=['date'], parse_dates=['date'])
Top answer
1 of 2
134

The read_sql docs say this params argument can be a list, tuple or dict (see docs).

To pass the values in the sql query, there are different syntaxes possible: ?, :1, :name, %s, %(name)s (see PEP249).
But not all of these possibilities are supported by all database drivers, which syntax is supported depends on the driver you are using (psycopg2 in your case I suppose).

In your second case, when using a dict, you are using 'named arguments', and according to the psycopg2 documentation, they support the %(name)s style (and so not the :name I suppose), see http://initd.org/psycopg/docs/usage.html#query-parameters.
So using that style should work:

df = psql.read_sql(('select "Timestamp","Value" from "MyTable" '
                     'where "Timestamp" BETWEEN %(dstart)s AND %(dfinish)s'),
                   db,params={"dstart":datetime(2014,6,24,16,0),"dfinish":datetime(2014,6,24,17,0)},
                   index_col=['Timestamp'])
2 of 2
-1

I was having trouble passing a large number of parameters when reading from a SQLite Table. Then it turns out since you pass a string to read_sql, you can just use f-string. Tried the same with MSSQL pyodbc and it works as well.

For SQLite, it would look like this:

# write a sample table into memory
from sqlalchemy import create_engine
df = pd.DataFrame({'Timestamp': pd.date_range('2020-01-17', '2020-04-24', 10), 'Value1': range(10)})
engine = create_engine('sqlite://', echo=False)
df.to_sql('MyTable', engine);

# query the table using a query
tpl = (1, 3, 5, 8, 9)
query = f"""SELECT Timestamp, Value1 FROM MyTable WHERE Value1 IN {tpl}"""
df = pd.read_sql(query, engine)

If the parameters are datetimes, it's a bit more complicated but calling the datetime conversion function of the SQL dialect you're using should do the job.

start, end = '2020-01-01', '2020-04-01'
query = f"""SELECT Timestamp, Value1 FROM MyTable WHERE Timestamp BETWEEN STRFTIME("{start}") AND STRFTIME("{end}")"""
df = pd.read_sql(query, engine)
🌐
Plain English
plainenglish.io › home › blog › python › pandas read_sql with variable parameters — alternative way
Pandas Read_Sql With Variable Parameters — Alternative Way
November 26, 2022 - And, generally, an SP has its parameters and these parameters need to be variables most of the time. Indeed, this issue is discussed and solved in Stack overflow. You can check it here. However, I think there is a better way doing this. First, we need our SP. In this example, name of my SQL — related things are below. ... In this example, our SP is taking 2 date parameters: StartDate and EndDate. ... Now, we are ready to write our query in Python.
🌐
Reddit
reddit.com › r/learnpython › pandas parametized query
r/learnpython on Reddit: Pandas Parametized Query
October 25, 2022 -

Hello Guys,

I'm trying to query data with variables using the select statement with pandas and MySQL. Is there a way I can declare the date variable and pass them to the query during runtime. I haven't come across any ways that work from my online research.

Here is my code :

from datetime import datetime
from email import encoders
import smtplib
import pandas as pd
from sqlalchemy import create_engine
from urllib.parse import quote  
import mysql.connector as sql

db = create_engine('mysql://root:%s@localhost:3306/store' % quote('Mypass@12!'))

now = datetime.now()
startdate = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0).strftime("%y-%m-%d %H:%M:%S")
currentdate = now.replace(day=3,hour=23, minute=00, second=0).strftime("%y-%m-%d %H:%M:%S")

df1 = pd.read_sql("select * from orders where datecreated > %s and datecreated < %s ", con=db)

pdwriter = pd.ExcelWriter('report.xlsx', engine='xlsxwriter')
df1.to_excel(pdwriter, sheet_name='GEN')
pdwriter.save()

I would like to pass the date variables to the query.

Any suggestion is greatly appreciated. Thank you.

🌐
Pandas
pandas.pydata.org › docs › reference › api › pandas.read_sql.html
pandas.read_sql — pandas 3.0.1 documentation - PyData |
>>> pd.read_sql("test_data", "postgres:///db_name") For parameterized query, using params is recommended over string interpolation.
Top answer
1 of 3
27

Break this up into three parts to help isolate the problem and improve readability:

  1. Build the SQL string
  2. Set parameter values
  3. Execute pandas.read_sql_query

Build SQL

First ensure ? placeholders are being set correctly. Use str.format with str.join and len to dynamically fill in ?s based on member_list length. Below examples assume 3 member_list elements.

Example

member_list = (1,2,3)
sql = """select member_id, yearmonth
         from queried_table
         where yearmonth between {0} and {0}
         and member_id in ({1})"""
sql = sql.format('?', ','.join('?' * len(member_list)))
print(sql)

Returns

select member_id, yearmonth
from queried_table
where yearmonth between ? and ?
and member_id in (?,?,?)

Set Parameter Values

Now ensure parameter values are organized into a flat tuple

Example

# generator to flatten values of irregular nested sequences,
# modified from answers http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python
def flatten(l):
    for el in l:
        try:
            yield from flatten(el)
        except TypeError:
            yield el

params = tuple(flatten((201601, 201603, member_list)))
print(params)

Returns

(201601, 201603, 1, 2, 3)

Execute

Finally bring the sql and params values together in the read_sql_query call

query = pd.read_sql_query(sql, db2conn, params)
2 of 3
15

WARNING! Although my proposed solution here works, it is prone to SQL injection attacks. Therefor, it should never be used directly in backend code! It is only safe for offline analysis.

If you're using python 3.6+ you could also use a formatted string litteral for your query (cf https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep498)

start, end = 201601, 201603
selected_members = (111, 222, 333, 444, 555)  # requires to be a tuple

query = f"""
    SELECT member_id, yearmonth FROM queried_table
    WHERE yearmonth BETWEEN {start} AND {end}
      AND member_id IN {selected_members}
"""

df = pd.read_sql_query(query, db2conn)
🌐
Google Groups
groups.google.com › g › python-sqlite › c › -DYiLNSvbxw
Pandas DataFrame - Using variables in SQL query
December 9, 2020 - I have the basic query and the report I want working but I need to be able to produce that report for a variety of dates. As an example; Barometric pressure, Max and Min each hour for a day, each day for a week, or a month etc. I'm using Python 3.7 and Pandas Dataframe. df = pd.read_sql("SELECT strftime('%d', dateTime, 'unixepoch')AS day, " "strftime('%m', dateTime, 'unixepoch')AS month, " "strftime('%Y', dateTime, 'unixepoch')AS year, " "round(min*33.86,2) AS low, " "round(max*33.86,2) AS high " "FROM archive_day_barometer " "WHERE month = '11' AND year='2020'", conn)
Find elsewhere
🌐
Like Geeks
likegeeks.com › home › python › pandas › using the pandas read_sql params for dynamic data retrieval
Using the Pandas read_sql params for Dynamic Data Retrieval
December 9, 2023 - In this example, the :name placeholders directly map to the keys in the params dictionary. Dynamic query building is an advanced technique that allows your Python code to generate SQL queries based on varying parameters.
🌐
Readthedocs
pyhelpers.readthedocs.io › en › latest › _generated › pyhelpers.dbms.PostgreSQL.read_sql_query.html
PostgreSQL.read_sql_query - PyHelpers - Read the Docs
'WHERE "timestamp_column_name" BETWEEN %(ts_start)s AND %(ts_end)s' >>> params = {'d_start': datetime.datetime.today(), 'd_end': datetime.datetime.today()} >>> data_frame = pd.read_sql(sql=sql_qry, con=testdb.engine, params=params)
🌐
Pandas
pandas.pydata.org › docs › reference › api › pandas.read_sql_query.html
pandas.read_sql_query — pandas 3.0.1 documentation
pandas.read_sql_query(sql, con, index_col=None, coerce_float=True, params=None, parse_dates=None, chunksize=None, dtype=None, dtype_backend=<no_default>)[source]#
🌐
Stack Overflow
stackoverflow.com › questions › 72262050 › how-to-use-a-variable-in-the-query-string-of-pd-read-sql-query
python - How to use a variable in the query string of pd.read_sql_query? - Stack Overflow
Connect and share knowledge within a single location that is structured and easy to search. Learn more about Teams ... I'm trying to write a python script that runs a few SQL queries. However, the table being queried will change on a semi-regular basis. To make the script more user-friendly I want to store the table name in a variable. ... import pandas as pd V_pct = "UACO_PCT0000003221" V_contact_df = pd.read_sql_query( sql = """" SELECT DATE(CONTACTDATETIME) AS CONTACT_DATE FROM AF.{V_pct} GROUP BY DATE(CONTACTDATETIME); """, con = V_conn)
🌐
GitHub
github.com › pandas-dev › pandas › issues › 10899
read_sql should accept a sql_params parameter · Issue #10899 · pandas-dev/pandas
August 24, 2015 - Hello, I wonder if current use of read_sql couldn't lead to SQL injection. I read in https://docs.python.org/2/library/sqlite3.html # Never do this -- insecure! symbol = 'RHAT' c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbo...
Author   s-celles
🌐
Stack Overflow
stackoverflow.com › questions › 67491758 › retrieve-the-query-with-parameters-with-pandas-read-sql-query
python - Retrieve the query with parameters with pandas.read_sql_query() - Stack Overflow
sql = ''' select foo, bar from table where table.date >= %(start_date)s and table.value <= %(threshold)s ''' params = {'start_date':'2021-01-01', 'threshold':10} df = pd.read_sql_query(sql, db_connection, params=params) return df · What I want to do is add a line like this but where I save the actual query to the object, not the sql with placeholder variables: self.query = sql