It was explained in the blog a year ago, and I guess things have not really changed:
User-Agent Requests Bytes Served Yahoo! Slurp/3.0 56,331 1,124,861,780 Googlebot/2.1 56,579 773,418,834 Mediapartners-Google 30,519 671,904,609
...and then, after some specific settings did not help:
Answer from Arjan on Stack ExchangeIn the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...
Yahoo!
help.yahoo.com › kb › SLN22600.html
Why is Slurp crawling my page? | Yahoo Help
Slurp is the Yahoo Search robot for crawling and indexing web page information. Although some Yahoo Search results are powered by our partners, sites should allow Yahoo Slurp access in order to appear in Yahoo Mobile Search results.
Perishable Press
perishablepress.com › yahoo-slurp-too-stupid-to-be-a-robot
Yahoo! Slurp too Stupid to be a Robot | Perishable Press
August 8, 2024 - Yahoo's disobedient Slurp crawler tells us that Yahoo is deliberately disobeying the rules or else their robot is malfunctioning -- either case is not good..
Seoapi
seoapi.com › yahooslurp
Yahoo! Slurp API
Slurp to crawl the web, including your website. The Yahoo! Slurp API is a 'REST API' which is accessible via HTTPS using the GET method at a predefined URL.
Webmaster World
webmasterworld.com › search_engine_spiders › 4360952.htm
Yahoo! Slurp - Crawler, Spider, and User Agent ID forum at WebmasterWorld - WebmasterWorld
September 10, 2011 - Thus there's precious little benefit I can see in allowing anything other 'plain' "Yahoo! Slurp" from ".crawl.yahoo.com" access to anything other than .html files. YMMV ... I have all of the YST IPs blocked. I only allow crawls from crawl IPs. 98.137.72/24 is pretty much a slurp range but without the crawl rDNS.
TheFreeDictionary.com
thefreedictionary.com › Yahoo+Slurp
Yahoo Slurp - definition of Yahoo Slurp by The Free Dictionary
Define Yahoo Slurp. Yahoo Slurp synonyms, Yahoo Slurp pronunciation, Yahoo Slurp translation, English dictionary definition of Yahoo Slurp. n. Computers. spider . Random House Kernerman Webster's College Dictionary, © 2010 K Dictionaries Ltd. Copyright 2005, 1997, 1991 by Random House, Inc.
Search Engine Genie
searchenginegenie.com › yahoo-slurp.htm
Yahoo Slurp Article about Yahoo Search Engine Features Include Search Engine Optimization SEO
Yahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using the Yahoo! search engine.
Perishable Press
perishablepress.com › suspicious-behavior-from-yahoo-slurp-crawler
Suspicious Behavior from Yahoo! Slurp Crawler | Perishable Press
July 30th 2007, 09:19pm >> https://example.com/press/2005/11/07/dead-letter-art/feed/function.array-rand REFERRER: QUERY STRING: REMOTE ADDRESS: 74.6.26.221 USER AGENT: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Similar to the previous log entry, this one shows Yahoo!
DataDome
datadome.co › home › bots › yahoo!
What is the Yahoo! bot & How to block it?
February 4, 2025 - The Yahoo! bot, also known as Yahoo! Slurp, is a web crawler used by Yahoo! to index web pages for its search engine. It systematically browses the internet, collecting data from websites to update Yahoo!’s search index. This process allows Yahoo! to provide relevant search results to users ...
Yahoo!
uk.help.yahoo.com › kb › SLN24876.html
Slurp asking for “SlurpConfirm404” | Yahoo Help
To check a Web servers handling of page-not-found conditions, the Yahoo Search crawler, Slurp, occasionally sends URLs built from random words to websites from which no 404 responses have been seen.
BuiltWith
trends.builtwith.com › websitelist › Yahoo-Slurp-Disallow
Websites using Yahoo Slurp Disallow
Web Technology usage statistics and list of websites using web technologies.