Wordpress offers lots of plugins that may make your robots.txt file for you. However, my suggestion is to always make and update it manually.
If you don’t want to read all that long staff of why and how – here’s a basic robots.txt example for any Wordpress site.
Just don’t forget to change the sitemap address to your real sitemap, please!!!
The explanations and some other recommendations go right after.
You should save this code as a robots.txt file and download to your website core folder via ftp or c-panel so that it would be accessible by typing: your-site-web-address.com/robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: *?replytocom* Disallow: /tag/ Disallow: /category/ Disallow: /author/ Disallow: /?s Disallow: /wp-login.php* Sitemap: https://here-goes-your-site-domain.com/sitemap.xml
Why You Need Robots.txt At All
If you don’t care about SEO very much than, probably, you won’t need it. Otherwise, robots.txt serves as a guide for bots (especially search bots in this case) which parts and pages of your website you don’t want to be indexed. It should be kept in mind, however, that bots would not necessarily adhere to your guidelines if their algorithm tells them that this particular page you want to hide may be valuable for their users.
To persuade the bots that your intentions about the page are serious enough you should also exclude it from the sitemap, and put noindex robots in its header (may look like this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
or like this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
or even like this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX“> – what you choose depends if you want the bots to follow the links on this page, or not, or don’t care 🙂 )
Thus, you need that robots file when you don’t want some of your pages to be indexed by search bots.
Good Technical SEO Practice
Which pages should you put into your robots.txt file then?
Technical SEO requires to close these pages from bots:
- print versions,
- admin pages,
- personal workspace,
- cart, orders, and related,
- all the technical pages you don’t want to be visible through search.
This is usually done by placing those addresses right after Disallow: /and-here-goes-what-you-want-to-disallow. However, you should be careful not to disallow the page you are actually promoting. For instance, you may have a page: some-site.com/order/order-page.html and then you put it into your robots.txt like this:
This will make all your pages and subfolders starting with order to be not indexed. For example, you may be promoting a page: some-site.com/order-this-wonderful-product-of-mine.html, but if you leave your Disallow the way it’s shown above, it won’t get indexed.
In this case, you may either disallow everyparticular page or put:
Anyway, be careful about that.
If you don’t want to hire a SEO expert for technical audit of your website, but want to be sure you are doing everything right, you may do as follows:
- again make sure that your promoted pages are not locked from indexing (putting Disallow: / – will lock your whole site, BTW),
- go to Google Search Console (Google Webmaster Tools) and check it there (if you don’t have an account there, I do recommend making it),
- Bing Webmaster Tools also have a tool for validation,
- scan your site with some popular SEO Tool, for example, through SemRush (they have such a tool in their projects area) to find if there’s no prohibitions conflicts,
- make sure there’s no contradiction between your allows and disallows directions (for the same section of your site),
- make sure that pages without content are unavailable as well as search results,
- if you’ve got an e-commerce project, make catalog filters that generate non-unique content or no content at all, also unavailable.
Finally, there are a lot of Facebook groups, SEO forums, Quora and other social media where you can just ask for a favor! 🙂