It is possible to do this. The setup is... not very straightforward, but the end result is that you can search the entire web from python with few lines of code.

There are 3 main steps in total.

#1st step: get Google API key

The pygoogle's page states:

Unfortunately, Google no longer supports the SOAP API for search, nor do they provide new license keys. In a nutshell, PyGoogle is pretty much dead at this point.

You can use their AJAX API instead. Take a look here for sample code: http://dcortesi.com/2008/05/28/google-ajax-search-api-example-python-code/

... but you actually can't use AJAX API either. You have to get a Google API key. https://developers.google.com/api-client-library/python/guide/aaa_apikeys For simple experimental use I suggest "server key".

#2nd step: setup Custom Search Engine so that you can search the entire web

Indeed, the old API is not available. The best new API that is available is Custom Search. It seems to support only searching within specific domains, however, after following this SO answer you can search the whole web:

  1. From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
  2. Type a name and description for your search engine.
  3. Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
  4. Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
  5. Click any of the links under the Next steps section to navigate to your Control panel.
  6. In the left-hand menu, under Control Panel, click Basics.
  7. In the Search Preferences section, select Search the entire web but emphasize included sites.
  8. Click Save Changes.
  9. In the left-hand menu, under Control Panel, click Sites.
  10. Delete the site you entered during the initial setup process.

This approach is also recommended by Google: https://support.google.com/customsearch/answer/2631040

#3rd step: install Google API client for Python

pip install google-api-python-client, more info here:

  • repo: https://github.com/google/google-api-python-client
  • more info: https://developers.google.com/api-client-library/python/apis/customsearch/v1
  • complete docs: https://api-python-client-doc.appspot.com/

#4th step (bonus): do the search

So, after setting this up, you can follow the code samples from few places:

  • simple example: https://github.com/google/google-api-python-client/blob/master/samples/customsearch/main.py

  • cse() function docs: https://google-api-client-libraries.appspot.com/documentation/customsearch/v1/python/latest/customsearch_v1.cse.html

and end up with this:

from googleapiclient.discovery import build
import pprint

my_api_key = "Google API key"
my_cse_id = "Custom Search Engine ID"

def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res['items']

results = google_search(
    'stackoverflow site:en.wikipedia.org', my_api_key, my_cse_id, num=10)
for result in results:
    pprint.pprint(result)

After some tweaking you could write some functions that behave exactly like your snippet, but I'll skip this step here.

Note that num has an upper limit of 10 as per docs. Updating start in a loop might be necessary.

Answer from mbdevpl on Stack Overflow
🌐
PyPI
pypi.org › project › googlesearch-python
googlesearch-python · PyPI
A Python library for scraping the Google search engine.
      » pip install googlesearch-python
    
Published   Jan 21, 2025
Version   1.3.0
Top answer
1 of 3
143

It is possible to do this. The setup is... not very straightforward, but the end result is that you can search the entire web from python with few lines of code.

There are 3 main steps in total.

#1st step: get Google API key

The pygoogle's page states:

Unfortunately, Google no longer supports the SOAP API for search, nor do they provide new license keys. In a nutshell, PyGoogle is pretty much dead at this point.

You can use their AJAX API instead. Take a look here for sample code: http://dcortesi.com/2008/05/28/google-ajax-search-api-example-python-code/

... but you actually can't use AJAX API either. You have to get a Google API key. https://developers.google.com/api-client-library/python/guide/aaa_apikeys For simple experimental use I suggest "server key".

#2nd step: setup Custom Search Engine so that you can search the entire web

Indeed, the old API is not available. The best new API that is available is Custom Search. It seems to support only searching within specific domains, however, after following this SO answer you can search the whole web:

  1. From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
  2. Type a name and description for your search engine.
  3. Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
  4. Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
  5. Click any of the links under the Next steps section to navigate to your Control panel.
  6. In the left-hand menu, under Control Panel, click Basics.
  7. In the Search Preferences section, select Search the entire web but emphasize included sites.
  8. Click Save Changes.
  9. In the left-hand menu, under Control Panel, click Sites.
  10. Delete the site you entered during the initial setup process.

This approach is also recommended by Google: https://support.google.com/customsearch/answer/2631040

#3rd step: install Google API client for Python

pip install google-api-python-client, more info here:

  • repo: https://github.com/google/google-api-python-client
  • more info: https://developers.google.com/api-client-library/python/apis/customsearch/v1
  • complete docs: https://api-python-client-doc.appspot.com/

#4th step (bonus): do the search

So, after setting this up, you can follow the code samples from few places:

  • simple example: https://github.com/google/google-api-python-client/blob/master/samples/customsearch/main.py

  • cse() function docs: https://google-api-client-libraries.appspot.com/documentation/customsearch/v1/python/latest/customsearch_v1.cse.html

and end up with this:

from googleapiclient.discovery import build
import pprint

my_api_key = "Google API key"
my_cse_id = "Custom Search Engine ID"

def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res['items']

results = google_search(
    'stackoverflow site:en.wikipedia.org', my_api_key, my_cse_id, num=10)
for result in results:
    pprint.pprint(result)

After some tweaking you could write some functions that behave exactly like your snippet, but I'll skip this step here.

Note that num has an upper limit of 10 as per docs. Updating start in a loop might be necessary.

2 of 3
36

@mbdevpl's response helped me a lot, so all credit goes to them. But there have been a few changes in the UI, so here is an update:

A. Install google-api-python-client

  1. If you don't already have a Google account, sign up.
  2. If you have never created a Google APIs Console project, read the Managing Projects page and create a project in the Google API Console.
  3. Install the library.

B. To create an API key:

  1. Navigate to the APIs & Services→Credentials panel in Cloud Console.
  2. Select Create credentials, then select API key from the drop-down menu.
  3. The API key created dialog box displays your newly created key.
  4. You now have an API_KEY

C. Setup Custom Search Engine so you can search the entire web

  1. Create a custom search engine in this link.
  2. In Sites to search, add any valid URL (i.e. www.stackoverflow.com).
  3. That’s all you have to fill up, the rest doesn’t matter. In the left-side menu, click Edit search engine{your search engine name}Setup
  4. Set Search the entire web to ON.
  5. Remove the URL you added from the list of Sites to search.
  6. Under Search engine ID you’ll find the search-engine-ID.

Search example

from googleapiclient.discovery import build

my_api_key = "AIbaSyAEY6egFSPeadgK7oS/54iQ_ejl24s4Ggc" #The API_KEY you acquired
my_cse_id = "012345678910111213141:abcdef10g2h" #The search-engine-ID you created


def google_search(search_term, api_key, cse_id, **kwargs):
    service = build("customsearch", "v1", developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res['items']


results = google_search('"god is a woman" "thank you next" "7 rings"', my_api_key, my_cse_id, num=10)
for result in results:
    print(result)

Important! on the first run, you might have to enable the API in your account. The error message should contain the link to enable the API in. It will be something like: https://console.developers.google.com/apis/api/customsearch.googleapis.com/overview?project={your project name}.

You’ll be asked to create a service name (It doesn’t matter what it is), and give it Roles. I gave it Role Viewer and Service Usage Admin and it works.

🌐
Google
developers.google.com › search console api › quickstart: run a search console app in python
Quickstart: Run a Search Console App in Python | Search Console API | Google for Developers
To run the application, you need internet access, a Google account with a verified website in Search Console, and Python 3 with the Flask framework installed. The application utilizes the OAuth 2.0 protocol for authorization and the Google Search ...
🌐
GitHub
github.com › googleapis › google-api-python-client
GitHub - googleapis/google-api-python-client: 🐍 The official Python client library for Google's discovery based APIs.
🐍 The official Python client library for Google's discovery based APIs. - googleapis/google-api-python-client
Starred by 8.6K users
Forked by 2.5K users
Languages   Python 94.1% | Shell 5.7% | Makefile 0.2%
🌐
The Python Code
thepythoncode.com › article › use-google-custom-search-engine-api-in-python
How to Use Google Custom Search Engine API in Python - The Python Code
Let's build the API URL we'll request: # the search query you want query = "python" # using the first page page = 1 # constructing the URL # doc: https://developers.google.com/custom-search/v1/using_rest # calculating start, (page=2) => (start=11), (page=3) => (start=21) start = (page - 1) * 10 + 1 url = f"https://www.googleapis.com/customsearch/v1?key={API_KEY}&cx={SEARCH_ENGINE_ID}&q={query}&start={start}"
🌐
zenserp
zenserp.com › home › using google search api in python: a beginner’s guide
Use Google Search API in Python: A Beginner's Guide | Zenserp
October 17, 2024 - The most important package is the google-api-python-client, which can be installed using pip (Python package installer) by running the following command in your terminal or command prompt: ... Finally, you will need to create a Google API key to use the Google Search API.
🌐
Google
developers.google.com › programmable search engine › libraries and samples
Libraries and Samples | Programmable Search Engine | Google for Developers
Google API client libraries in various programming languages simplify using the Custom Search JSON API. Documentation and samples are available for client libraries in languages such as Java, JavaScript, .NET, Objective-C, PHP, and Python.
🌐
GitHub
github.com › abenassi › Google-Search-API
GitHub - abenassi/Google-Search-API: Python based api for searching google web, images, calc, and currency conversion.
Python based api for searching google web, images, calc, and currency conversion. - abenassi/Google-Search-API
Starred by 572 users
Forked by 215 users
Languages   HTML 94.6% | Python 5.2% | Makefile 0.2%
Find elsewhere
🌐
SerpApi
serpapi.com › integrations › python
SerpApi: Python Integration
params = { "q": "coffee", "location": "Location Requested", "device": "desktop|mobile|tablet", "hl": "Google UI Language", "gl": "Google Country", "safe": "Safe Search Flag", "num": "Number of Results", "start": "Pagination Offset", "api_key": "Your SerpApi Key", # To be match "tbm": "nws|isch|shop", # To be search "tbs": "custom to be search criteria", # allow async request "async": "true|false", # output format "output": "json|html" } # define the search search search = GoogleSearch(params) # override an existing parameter search.params_dict["location"] = "Portland" # search format return as raw html html_results = search.get_html() # parse results # as python Dictionary dict_results = search.get_dict() # as JSON using json package json_results = search.get_json() # as dynamic Python object object_result = search.get_object()
🌐
Linux Hint
linuxhint.com › google_search_api_python
Using Google Search API With Python – Linux Hint
We have done well getting the Custom Search ID and the API Key. Next we are going to make use of the API. While you can access the API with other programming languages, we are going to be doing so with Python. To be able to access the API with Python, you need to install the Google API Client ...
🌐
GitHub
github.com › RMNCLDYO › Google-Search-API-Wrapper
GitHub - RMNCLDYO/Google-Search-API-Wrapper: A simple python wrapper for Google's Search API, enabling programatic web and image searches.
Once created, find your Search Engine ID (cx parameter) in the Overview page's Basic section. Once you've obtained your API key and CX id, add them to your .env file. Create a new file named .env in the root directory, or rename the example.env file in the root directory of the project to .env. Add your API key and CX id to the .env file as follows: ... The application will automatically load and use the API key and CX id when making API requests. from google_search_api import GoogleSearchAPI print(GoogleSearchAPI().response(method='text', max_results=5, query='Your text prompt'))
Author   RMNCLDYO
🌐
PyPI
pypi.org › project › Google-Search-API
Google-Search-API · PyPI
Google_Search_API-1.1.14-py3-none-any.whl (19.5 kB view details) Uploaded Apr 2, 2019 Python 3 · Details for the file Google-Search-API-1.1.14.tar.gz. Download URL: Google-Search-API-1.1.14.tar.gz · Upload date: Apr 2, 2019 · Size: 19.4 kB · Tags: Source ·
      » pip install Google-Search-API
    
Published   Apr 02, 2019
Version   1.1.14
🌐
GitHub
github.com › serpapi › google-search-results-python
GitHub - serpapi/google-search-results-python: Google Search Results via SERP API pip Python Package
params = { "q": "coffee", "location": "Location Requested", "device": "desktop|mobile|tablet", "hl": "Google UI Language", "gl": "Google Country", "safe": "Safe Search Flag", "num": "Number of Results", "start": "Pagination Offset", "api_key": "Your SerpApi Key", # To be match "tbm": "nws|isch|shop", # To be search "tbs": "custom to be search criteria", # allow async request "async": "true|false", # output format "output": "json|html" } # define the search search search = GoogleSearch(params) # override an existing parameter search.params_dict["location"] = "Portland" # search format return as raw html html_results = search.get_html() # parse results # as python Dictionary dict_results = search.get_dict() # as JSON using json package json_results = search.get_json() # as dynamic Python object object_result = search.get_object()
Starred by 720 users
Forked by 117 users
Languages   Python 97.5% | Makefile 2.5%
🌐
Scraperapi
docs.scraperapi.com › python › making-requests › async-structured-data-collection-method › google-search-api-async
Google Search API (Async) | Python | ScraperAPI Documentation
Scrape Google SERP into structured JSON/CSV with ScraperAPI async in Python. Extract product info, filter by region and language, and automate SERP insights. This endpoint will retrieve search data from a Google search result page and transform ...
🌐
Google
docs.cloud.google.com › application hosting › app engine › standard environment › python 2 › search api for legacy bundled services
Search API for legacy bundled services | App Engine standard environment for Python 2 | Google Cloud Documentation
You can search an index, and organize and present search results. The API supports full text matching on string fields. Documents and indexes are saved in a separate persistent store optimized for search operations.
🌐
Apify
apify.com › apify › google-search-scraper › api › python
Google Search scraper and SERP API 🔎 API in Python · Apify
February 18, 2019 - The Apify API client for Python is the official library that allows you to use Google Search Results Scraper API in Python, providing convenience functions and automatic retries on errors.
🌐
Serper
serper.dev
Serper - The World's Fastest and Cheapest Google Search API
"snippet": "Google SERP APIs#. Search Engine Results Pages or SERPs are easy to scrape by using the right API. The Application Programming Interface lets...", ... "imageUrl": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRIVlwjl0M2LVTZ8xPBuWjbvvAqwiRWbhSVCsizKuVdiimaj4-3sgzpIqZLUA&s", ... "snippet": "Learn about the essential components of LangChain — agents, models, chunks and chains — and how to harness the power of LangChain in Python.",
🌐
LangChain
api.python.langchain.com › en › latest › utilities › langchain_community.utilities.google_search.GoogleSearchAPIWrapper.html
langchain_community.utilities.google_search.GoogleSearchAPIWrapper — 🦜🔗 LangChain 0.2.17
- Click Enable APIs and Services. - Search for Custom Search API and click on it. - Click Enable. URL for it: https://console.cloud.google.com/apis/library/customsearch.googleapis .com