You can use the Series method str.startswith (which takes a regex):
In [11]: s = pd.Series(['aa', 'ab', 'ca', np.nan])
In [12]: s.str.startswith('a', na=False)
Out[12]:
0 True
1 True
2 False
3 False
dtype: bool
You can also do the same with str.contains (using a regex):
In [13]: s.str.contains('^a', na=False)
Out[13]:
0 True
1 True
2 False
3 False
dtype: bool
So you can do df[col].str.startswith...
See also the SQL comparison section of the docs.
Note: (as pointed out by OP) by default NaNs will propagate (and hence cause an indexing error if you want to use the result as a boolean mask), we use this flag to say that NaN should map to False.
In [14]: s.str.startswith('a') # can't use as boolean mask
Out[14]:
0 True
1 True
2 False
3 NaN
dtype: object
You can use the Series method str.startswith (which takes a regex):
In [11]: s = pd.Series(['aa', 'ab', 'ca', np.nan])
In [12]: s.str.startswith('a', na=False)
Out[12]:
0 True
1 True
2 False
3 False
dtype: bool
You can also do the same with str.contains (using a regex):
In [13]: s.str.contains('^a', na=False)
Out[13]:
0 True
1 True
2 False
3 False
dtype: bool
So you can do df[col].str.startswith...
See also the SQL comparison section of the docs.
Note: (as pointed out by OP) by default NaNs will propagate (and hence cause an indexing error if you want to use the result as a boolean mask), we use this flag to say that NaN should map to False.
In [14]: s.str.startswith('a') # can't use as boolean mask
Out[14]:
0 True
1 True
2 False
3 NaN
dtype: object
- To find all the values from the series that starts with a pattern "s":
SQL - WHERE column_name LIKE 's%'
Python - column_name.str.startswith('s')
- To find all the values from the series that ends with a pattern "s":
SQL - WHERE column_name LIKE '%s'
Python - column_name.str.endswith('s')
- To find all the values from the series that contains pattern "s":
SQL - WHERE column_name LIKE '%s%'
Python - column_name.str.contains('s')
For more options, check : https://pandas.pydata.org/pandas-docs/stable/reference/series.html
Modern alternatives to Data Science Libraries like Polars with Pandas?
python - USING LIKE inside pandas.query() - Stack Overflow
packages similar to Pandas
A Dependencyless Pure-Python Pandas-like DataFrame (for Lambdas)
I've been trying Polars and love them more than Pandas. In addition to performance, I find the API better designed (fewer ways to do the same thing) which, I think, allows memorizing the syntax faster, I would recommend Polars instead of Pandas to a new person.
Are there any modern alternatives for data visualization, algorithms, etc. that you are considering as an upgrade to your stack?
If you have to use df.query(), the correct syntax is:
df.query('column_name.str.contains("abc")', engine='python')
You can easily combine this with other conditions:
df.query('column_a.str.contains("abc") or column_b.str.contains("xyz") and column_c>100', engine='python')
It is not a full equivalent of SQL Like, however, but can be useful nevertheless.
@volodymyr is right, but the thing he forgets is that you need to set engine='python' to expression to work.
Example:
>>> pd_df.query('column_name.str.contains("abc")', engine='python')
Here is more information on default engine ('numexpr') and 'python' engine. Also, have in mind that 'python' is slower on big data.
Do you guys has any recomendation of packages similar to Python Pandas?I was searching but I'm not sure if there one option than it's better than other , I'm kinda new to golang , thanks .
After dealing with AWS Lambda layers and Python libraries (specifically, Pandas, Numpy, and PyArrow), I decided to try making my own dependency-less pure-python implementation of Pandas. The goal was to make is as identical as possible to Pandas, and even have it replicate the behavior of shared underlying data between Frames and slices/Series.
With this built, if I just needed the convenience of Pandas for dealing with a (small) dataset, but didn't want the headache of installing all the dependencies, I could just do a quick pip install pambdas. All under 1MB (or 188 KB).
Currently supported:
-
Indexing/assignment by .loc and .iloc
-
Unary operators for Series
-
Boolean Indexing
-
Apply and ApplyMap
-
Append
-
GroupBy
-
pambdas.concat
-
pambdas.read_csv
The underlying data type is a simple Python list, and views are created but adjusting the slice (start, stop, and step). It was a bit of a pain, but oddly satisfying! It's not optimized, quite verbose (I handle each .iloc/.loc case independently), and is certainly very buggy. And since it's uses only pure-python, it is obviously much slower.
A couple of comments about the process:
-
My test file is over 1000 lines, and I'm still finding bugs.
-
It was important to get the indexing right.
-
It was fun developing the very basic functions (indexing, values, operators), and then bootstrapping with these to make much higher complexity processes.
-
Should have gotten the nans right from the start.
-
Testing square dataframes (e.g. 4x4, 5x5) was a bad idea...
There is still al lot of work to be done, so I'll collect some feedback while I take a break from this project. And if you're interested in contributing let me know! Lots of low hanging fruit. And some higher hanging fruit, like garbage collection.
Check out the github (https://github.com/danhitchcock/pambdas) and the example (https://github.com/danhitchcock/pambdas/blob/master/example.py), and feel free to install and find all the bugs try it out with `pip install pambdas`! Also, let me know if there are any killer features you'd like to see.
I'm sick of Pandas and want to use something faster and more intuitive for data wrangling.
I've been given the green light at work to try out whatever package/language I want, so open to any suggestions.
I was considering something like DataFrames.jl, Tidyverse, Polars, TidyPolars, etc. but wondered what people thought was best nowadays?