Yes, Google Custom Search has now replaced the old Search API, but you can still use Google Custom Search to search the entire web, although the steps are not obvious from the Custom Search setup.
To create a Google Custom Search engine that searches the entire web:
- From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
- Type a name and description for your search engine.
- Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
- Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
- Click any of the links under the Next steps section to navigate to your Control panel.
- In the left-hand menu, under Control Panel, click Basics.
- In the Search Preferences section, select Search the entire web but emphasize included sites.
- Click Save Changes.
- In the left-hand menu, under Control Panel, click Sites.
- Delete the site you entered during the initial setup process.
Now your custom search engine will search the entire web.
Pricing
- Google Custom Search gives you 100 queries per day for free.
- After that you pay $5 per 1000 queries.
- There is a maximum of 10,000 queries per day.
Source: https://developers.google.com/custom-search/json-api/v1/overview#Pricing
- The search quality is much lower than normal Google search (no synonyms, "intelligence" etc.)
- It seems that Google is even planning to shut down this service completely.
Videos
What are the alternatives now that the Google web search API has been deprecated? - Stack Overflow
How do I get web search results using an api? - Google Search Community
Why people are paying for API to do google searches? Why people won't use simple Python scrapy/requests script instead of paying 200 dollars for 200k searches?
Does Google provide a search API - Stack Overflow
Yes, Google Custom Search has now replaced the old Search API, but you can still use Google Custom Search to search the entire web, although the steps are not obvious from the Custom Search setup.
To create a Google Custom Search engine that searches the entire web:
- From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
- Type a name and description for your search engine.
- Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
- Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
- Click any of the links under the Next steps section to navigate to your Control panel.
- In the left-hand menu, under Control Panel, click Basics.
- In the Search Preferences section, select Search the entire web but emphasize included sites.
- Click Save Changes.
- In the left-hand menu, under Control Panel, click Sites.
- Delete the site you entered during the initial setup process.
Now your custom search engine will search the entire web.
Pricing
- Google Custom Search gives you 100 queries per day for free.
- After that you pay $5 per 1000 queries.
- There is a maximum of 10,000 queries per day.
Source: https://developers.google.com/custom-search/json-api/v1/overview#Pricing
- The search quality is much lower than normal Google search (no synonyms, "intelligence" etc.)
- It seems that Google is even planning to shut down this service completely.
Google Custom Search (as advocated in the top rated answers) works well, but is very expensive, compared to its competitors (below) or compared to other Google API's. It has a small free tier (100 queries/day) and a very high price of $5 per 1000 query.
They offer the option to upgrade to Site Search, which has slightly better prices, but that is meant for searching one site (your own), so it is really something quite different - not an upgrade.
The main alternatives seem to be:
Bing Search API
https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44
Which has a free tier of 5000q/month, and prices starting at 5 query per penny, and no hard limit.
UPDATE: At the end of 2016 this API was shutdown in favour of its Azure counterpart "Cognitive Services Bing Search API":
https://azure.microsoft.com/en-us/services/cognitive-services/search/
See here for a pricing chart, which starts at US$3/m for 1,000 transactions. Unless I'm missing something it is quite expensive.
Yahoo BOSS Search API
UPDATE: Was discontinued on March 31, 2016.
http://developer.yahoo.com/boss/search/
With prices starting at about 12 queries/penny for whole web searches.
And some I haven't heard of before:
http://www.gigablast.com/searchfeed.html
http://www.faroo.com/hp/api/api.html
http://www.commoncrawl.org/
http://www.entireweb.com/search_api/implementation/
[discontinued - as pointed out below]
There is a bit of discussion of some of these on this SO post.
[got closed for being off-topic and is now gone]
Hi. I am going currently through different SaaS's things and I found business that is selling access to API to do google searches. And I don't understand who is paying for this.
First of all, I thought that google is selling access to their own search through their own API, but apparently it was deleted for some reason?
Out of curiosity, I checked out how hard it would be with Python and Scrapy and it is around 10 lines of code to get search response from google.
I am wondering what am I missing? I was thinking about 2 obvious(?) problems:
How well would Scrapy/Requests with let's say FastAPI scale. How slow it would be.
How much would google tolerate. I can only guess that if 1 ip will be making new search every 3 - 5 seconds non-stop. Google would ban this ip. So the more requests, the biggest IP pool would be needed?
But other than that, what am I missing? Why someone is paying 200 dollars for 200k google searches?
Be aware that for not so popular sites, even if they have been around for years, Google will ever show only about 4% of their pages.
In 2022, I tested five SEO experts sites and out of the 1566 pages listed in their sitemap.xml files, Google only listed 402, or 26%. None of them explained why all their pages weren't listed, which makes it seem that they either don't know or they are incompetent. Either way, use of their services is rather dubious if they promote Google but coverage of their sites is so woeful.
Up until August 2023, Bing would return several times more than Google, but now returns 100%, and without Google's 200 limit for the site: filter.
Bing returns all pages for those SEO sites now as well.
For more details, see Google search is lying to you!
I expect a lot of downvotes, as I already had a lot of pushback from several people on another SO forum when I previously tried to highlight Google's egregious discriminatory practice. Perhaps too many tech people are too enamoured with Search Central tools that they blindly accept that Google has valid reasons for ignoring many if not most of their site's pages. That Bing can now list all pages highlights that Google's actions are unwarranted. Hopefully Google will now live up to their implied legal responsibility to list all valid pages.
I suggested to the Bing team that they should immediately launch a worldwide advertising campaign highlighting how Google is letting down all their users by their failure to list all valid pages.
Yes, Google Custom Search does what you want. It has multiple sub-products, including Site Search which is the one that is for your own site, but also CSE which is a general web search. It is primarily intended for searching subsets of the web, but has a feature where you can flexibly and dynamically define those subsets.
https://www.google.com/cse/
https://www.google.ca/cse/docs/
There is an API for "custom search":
https://developers.google.com/custom-search/
The service is pretty good, but rather expensive at $5 / 1000 queries once you pass the 100 queries/day free tier.