(Originally posted this on another sub and figured it makes sense to share it here as well.)

Hi everyone, Screaming Frog is obviously one of the most essential tools for SEO. Yet I noticed many experts only use it for rather basic tasks (e.g. getting reports regarding internal linking, URL status codes, etc.). After starting to dive into the more advanced use cases, I'm honestly pretty overwhelmed as to where to start. Hence my questions leading to this post:

• Do you make use of SF beyond such common reports and possibly automate SF exporting integrated with other tools?

• Are there any general best practices to keep in mind regarding SF configs or so? (e.g. DB storage mode, crawl frequency and other options)

• Does SF serve you specifically regarding reportings for clients? (I understand it's essential when working with SEOs so I'm mostly curious if there are ways to use SF for comprehensible reportings providing relevant insights for potentially non-SEO folks)

• Are there generally any visualition methods you could share?

I understand these aspects mostly touch more advanced aspects and don't expect anyone to share specific workflows they had to work out themselves. Even general input as to which extend you professionals are utilizing Screaming Frog would be greatly appreciated as I'm quite lost right now and simply would like to know whether this is something I should generally spend time on or if it's rather negligible. So I'm grateful for any kind of input!

Hey everyone,

I'm working on a client's website and facing a bit of a head-scratcher. When I crawl the entire site, it returns a massive number of URLs, way more than I'd expected.

To narrow things down, I tried crawling just the sitemap. This resulted in a much more reasonable number of URLs, but I'm concerned that some important pages might be missing.

Any tips on how to reconcile the massive crawl results with a potentially incomplete sitemap?

Got a message from a friend the other day that made me wince - he'd just spent 11 hours manually writing meta descriptions for 139 pages. Been there, done that, and it's absolutely soul-crushing work.

Figured I'd share my setup for automating this with Screaming Frog and OpenAI. It's not perfect, but it beats the hell out of doing it manually.

What you'll need:

• Screaming Frog license

• OpenAI API key

• Basic JavaScript knowledge (really basic, I promise)

Step 1: Enable JavaScript in Screaming Frog

1. Go to Configuration > Spider > Rendering

2. Select "JavaScript" from dropdown

3. Click OK

Step 2: Prevent API Rate Limits (trust me, you want to do this)

• Go to Configuration > Speed

• Enable "Limit URL/s"

• Set Max URL/s to 0.5 or lower

• Also uncheck all resource links in Configuration > Spider > Crawl (you don't need meta descriptions for CSS files!)

Step 3: Set Up Custom JavaScript

1. Navigate to Configuration > Custom > Custom JavaScript

2. Click "+Add from Library"

3. Select "(ChatGPT) Template"

Step 4: Get Your OpenAI API Key

1. Create an OpenAI platform account (this is different than a ChatGPT account)

2. Go to Settings > API keys

3. Create new secret key

4. Copy it into your Screaming Frog setup (for security reasons you can't access it again once you close the window)

The Secret Sauce: The Prompt
Here's the prompt I use that's been working pretty well:

const question = "Identify the primary keyword from the following web page content and generate an SEO optimized meta description using that keyword in the style of a SEO expert. IMPORTANT: Do NOT start the description with the words explore, exmbark, experience, join, dive into, or discover.\n"+
"# Steps\n"+
"1. Analyze Content: Examine the provided content to determine the primary theme or subject matter.\n"+
"2. Identify Keyword: Select the primary keyword that best represents the main subject of the web page.\n"+
"3. Craft Meta Description: Write a meta description between 120-155 characters that incorporates the keyword, is engaging, and succinctly summarizes the page content using the language in the text.\n"+
"# Output Format\n"+
"**Primary keyword:** The keyword that best represents the main subject of the web page\n"+
"**Meta Description:**  Meta description between 120-155 characters\n"+
"# Examples\n"+
"**Example 1:**\n"+
"- Primary Keyword: Keyword Research Tool\n"+
"- Meta Description: The most complete keyword research tool on the market. Generate thousands of keyword ideas, group them into topical clusters, and nail search intent.\n"+
"**Example 2:**\n"+
"- Primary Keyword: Authority checker\n"+
"- Meta Description: Free tool to check the authority of any website based on the quality and quantity of its external backlinks.\n"+
"**Example 3:**\n"+
"- Primary Keyword: Screaming Frog alternatives\n"+
"- Meta Description: Looking for a Screaming Frog alternative? MetaMonster could be the tool for you.\n"+
"**Example 4:**\n"+
"- Primary Keyword: SEO spider software\n"+
"- Meta Description: Clean up your metadata in minutes instead of months with MetaMonster, the SEO spider that fixes issues for you.\n"+
"**Example 5:**\n"+
"- Primary Keyword: Page title generator\n"+
"- Meta Description: Generate page titles for your website with AI. MetaMonster is the SEO spider software that fixes issues for you.\n"+
"**Example 6:**\n"+
"- Primary Keyword: Wordpress SEO plugin\n"+
"- Meta Description: Easily publish optimizations to your site with the MetaMonster Wordpress SEO Plugin.\n"

It's broken up into separate lines because of the way SF's code editor handles long strings.

Bonus Tips to improve your output:

• Increase temperature to 1.0 for more variable responses

• Add this code to extract just the meta description:

function extractMetaDescription(content) {
    const metaDescriptionRegex = /\*\*Meta Description:\*\*\s*(.+)/;
    const match = content.match(metaDescriptionRegex);
    
    return match ? match[1].trim() : null;
}

Limitations You Should Know About:

• Each page is processed in isolation so the AI has limited context

• SF's basic code editor makes it a pain to do more sophisticated stuff like prompt chaining

• Make sure to save your changes as your own template or you'll lose everything when you close SF

Results
Running this on my friend's site would've taken about 30 minutes of setup and maybe 20 minutes of processing time instead of his 11-hour manual slog. The descriptions aren't perfect, but they're pretty solid and can be quickly edited if needed.

Disclaimer: I'm building a tool called MetaMonster that handles this stuff automatically without writing code (and with higher quality output), but this Screaming Frog setup is a solid free alternative if you already have SF.

I wrote all of this up in a blog post on our site that I'm happy to share, but didn't want to violate any self-promo rules. Hope this is helpful!