For big projects, I highly recommend Screaming Frog. Compared to SEMrush and Ahrefs, I personally like it because it catches some errors that they don’t. More on reddit.com

Gold standard product for auditing, crawling and extracting website information. If you are not used or new to the tool do some, free, training on the SF site to get used to all the bells and whistles. More on reddit.com

Screaming frog has a lot of cool uses for developers and website managers beyond typical SEO stuff also. A a few examples, Your marketing team has an embedded Marketo iframe form on various pages, but its difficult to track them down. You can crawl the entire site looking for that iframe, and extract other info and come up with a nice report that has all the pages listed with relevant info ( Page url, form ID, Form heading text ) After several years of different people creating pages on your site, you have a mismatch of CTA button text. You can scrape the site and pull a report of every button with a specific css ID and the text inside. The report can list out all the buttons and the pages each one is on for an audit to create consistent CTAs. Visualization - You can crawl just your blog area, pull publish date, author, and word count. Put that all into an excel / google sheet and import it into Google Data Studio and create a cool visual report of content created. I find this best in timeline paired with search console data and traffic. You end up getting a report that helps show after big content pushes, reach and traffic increase. More on reddit.com

You start your crawl, then see a weird pattern in URLs. Pause your crawl and go to Exclude and then exclude that subdirectory or pattern. Then unpause crawl and repeat. More on reddit.com