So long as dt is a datetime dtype already you can filter using date strings, if not then you can convert doing this:
df['dt'] = pd.to_datetime(df['dt'])
Then filter:
In [115]:
df[(df['dt'] > '2014-07-23 07:30:00') & (df['dt'] < '2014-07-23 09:00:00')]
Out[115]:
dt value
index
91 2014-07-23 07:35:00 0.300
92 2014-07-23 07:40:00 0.300
93 2014-07-23 07:45:00 0.216
94 2014-07-23 07:50:00 0.204
95 2014-07-23 07:55:00 0.228
96 2014-07-23 08:00:00 0.228
97 2014-07-23 08:05:00 0.228
98 2014-07-23 08:10:00 0.228
99 2014-07-23 08:15:00 0.240
100 2014-07-23 08:20:00 0.228
101 2014-07-23 08:25:00 0.216
102 2014-07-23 08:30:00 0.228
103 2014-07-23 08:35:00 0.324
104 2014-07-23 08:40:00 0.336
105 2014-07-23 08:45:00 0.324
106 2014-07-23 08:50:00 0.324
107 2014-07-23 08:55:00 0.324
Answer from EdChum on Stack OverflowSo long as dt is a datetime dtype already you can filter using date strings, if not then you can convert doing this:
df['dt'] = pd.to_datetime(df['dt'])
Then filter:
In [115]:
df[(df['dt'] > '2014-07-23 07:30:00') & (df['dt'] < '2014-07-23 09:00:00')]
Out[115]:
dt value
index
91 2014-07-23 07:35:00 0.300
92 2014-07-23 07:40:00 0.300
93 2014-07-23 07:45:00 0.216
94 2014-07-23 07:50:00 0.204
95 2014-07-23 07:55:00 0.228
96 2014-07-23 08:00:00 0.228
97 2014-07-23 08:05:00 0.228
98 2014-07-23 08:10:00 0.228
99 2014-07-23 08:15:00 0.240
100 2014-07-23 08:20:00 0.228
101 2014-07-23 08:25:00 0.216
102 2014-07-23 08:30:00 0.228
103 2014-07-23 08:35:00 0.324
104 2014-07-23 08:40:00 0.336
105 2014-07-23 08:45:00 0.324
106 2014-07-23 08:50:00 0.324
107 2014-07-23 08:55:00 0.324
You can also use query:
In [25]: df.query('"2014-07-23 07:55:00" <= dt <= "2014-07-23 08:20:00"')
Out[25]:
dt value
95 2014-07-23 07:55:00 0.228
96 2014-07-23 08:00:00 0.228
97 2014-07-23 08:05:00 0.228
98 2014-07-23 08:10:00 0.228
99 2014-07-23 08:15:00 0.240
100 2014-07-23 08:20:00 0.228
python - How to filter a Pandas dataframe by timestamp functon using .query() - Stack Overflow
python - Filtering Pandas DataFrames on dates - Stack Overflow
Python Pandas DataFrame: filter by a Timestamp column with a list of string timestamps - Stack Overflow
Can you filter pandas dataframes by Day/Month date without year? Writing a generalised function for between date that is year agnostic is proving difficult.
Try: df.query('date >= @yesterday'). You need @ so pandas recognizes it's a variable.
IIUC, you want to create an outside varible to use inside your query?
from the docs
You can refer to variables in the environment by prefixing them with an ‘@’ character like @a + b.
using pandas only
import pandas as pd
df = pd.DataFrame({'date' : pd.date_range('01-02-2020','01-03-2021',freq='D')})
df = df.set_index('date')
delta = (pd.Timestamp('today') - pd.DateOffset(days=1)).strftime('%d-%m-%y')
df.query(f"date <= @delta")
date
2020-01-02
2020-01-03
2020-01-04
2020-01-05
2020-01-06
If date column is the index, then use .loc for label based indexing or .iloc for positional indexing.
For example:
df.loc['2014-01-01':'2014-02-01']
See details here http://pandas.pydata.org/pandas-docs/stable/dsintro.html#indexing-selection
If the column is not the index you have two choices:
- Make it the index (either temporarily or permanently if it's time-series data)
df[(df['date'] > '2013-01-01') & (df['date'] < '2013-02-01')]
See here for the general explanation
Note: .ix is deprecated.
Previous answer is not correct in my experience, you can't pass it a simple string, needs to be a datetime object. So:
import datetime
df.loc[datetime.date(year=2014,month=1,day=1):datetime.date(year=2014,month=2,day=1)]
You can use .contains() by first converting them into str
res = df.loc[(df.index.astype(str).str.contains("2008-12"))
| (df.index.astype(str).str.contains('2008-11-05'))]
print(res)
ts val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100 0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643 2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837 3
second question
yes you can apply filter on normal column like
df.loc[(df.ts.astype(str).str.contains("2008-12"))
|(df.ts.astype(str).str.contains('2008-11-05'))]
This should be get going for you..
>>> df
ts val
0 2008-11-05 07:45:23.100 0
1 2008-11-17 06:53:25.150 1
2 2008-12-02 07:36:18.643 2
3 2008-12-15 07:36:24.837 3
4 2009-01-06 07:03:47.387 4
Result:
>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05')).any(axis=1)]
ts val
0 2008-11-05 07:45:23.100 0
OR ..
>>> df
ts val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100 0
2008-11-17 06:53:25.150 2008-11-17 06:53:25.150 1
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643 2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837 3
2009-01-06 07:03:47.387 2009-01-06 07:03:47.387 4
Result:
>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05')).any(axis=1)]
ts val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100 0
Looking for multiple values.
>>> df[df.apply(lambda row: row.astype(str).str.contains('2008-11-05|2008-12')).any(axis=1)]
ts val
ts
2008-11-05 07:45:23.100 2008-11-05 07:45:23.100 0
2008-12-02 07:36:18.643 2008-12-02 07:36:18.643 2
2008-12-15 07:36:24.837 2008-12-15 07:36:24.837 3