Pages blocked by robots.txt file
When pages on your website are blocked by your robots.txt file, this means that search engines are not able to crawl these pages.
This can negatively impact your SEO, as these pages will not be visible to users in the search engine results.
It’s important to fix robots.txt file issues so that your important pages have a chance of ranking in search engines.
What is a robots.txt file?
A robots.txt file is a file that website owners create to permit or restrict access from search engine robots. It is essentially a “Go” or “Stop” sign that tells search engines whether they can crawl your website.
A “pages blocked by robots.txt” error indicates that there are pages on your website that are not accessible to search engine crawlers. These pages will not show up in the search results and will not be able to be found organically by users.
What is a robots.txt file used for?
A robots.txt file is used to manage search engine traffic to your website. You may choose to restrict access to certain pages if you don’t want them to be discovered by search engines or users.
For instance, you may want to hide certain landing pages so that users only end up there through a specific marketing campaign, not via organic search. You may also restrict access to unimportant image, style, or script files.
To see if a page has been crawled by Google, search for the page URL in Google.
How to Update Your Robots.txt File
If you discover that an important page is being blocked by your robots.txt file and you want to unblock it, or if you want to hide a page from search engines, then you will need to update your robots.txt file (or set up a new one).
How you do this will depend on the CMS and host that you use. We recommend searching for instructions on how to modify your robots.txt file for your specific host.
First, make sure that the robots.txt file is writeable by using FTP. If it is not, you may need to change user permissions.
- For WordPress, you may also use the Yoast SEO plugin to update your robots.txt file.
- For Wix, you can view robots.txt modification instructions here.
- For Squarespace, read their instructions on how to edit robots.txt here.
To test your new or updated robots.txt file, submit to Google for it to be crawled.