The SEO Spider tool from Screaming Frog is a free application that lets you crawl your site like a search engine would so you can identify any trouble spots in your site structure and take action to correct them.
The spider tool is free for up to 500 URLs of your site. If you have more than 500 URLs you’ll need to upgrade to the paid version.
In this article, we’ll cover:
What Is a Site Crawler?
But what is a site crawler, and why does it matter? As you probably already know, search engines crawl the web and index the results. If you can test your site’s “crawlability,” and see any potential pitfalls that might block or inhibit search engine bots, you can work to fix them and avoid loss of time that can risk in poor search results and delays.
It can take up to six months for changes made to your website to start showing significant search gains. Imagine if you were unaware that an important part of your site was not being crawled for six months? Here is where a site crawler can save you.
And, up to a certain number of URLs, the Screaming Frog tool is free.
How to Crawl Your Site With SEO Spider
Now it’s time to run a crawl of your site. For this part of the tutorial, just make sure that you have the primary domain of your website handy because we will use it to initiate our crawl.
- Open SEO Spider on your computer
- Fill in your URL
- Click Start
More SEO Resources
Check out these other resources from the support center and blog:
