In this category Optimising for search engines and AI chatbots

Optimising for search engines and AI chatbots

Add rules to robots.txt

1 min read

If you want to customise your robots.txt – as part of your search engine optimisation, for example – that is no problem.

You can read more about robots.txt at Google, among other places:

Introduction to robots.txt at Google Search Central (opens in a new tab)

You add your own settings by entering them in the file directly from the menu SEO > Robots.txt

Screenshot: Add rules to robots.txt

Screenshot: Add rules to robots.txt (opens a larger image)

Test your changes

When you have made changes to your robots.txt, it is a good safety measure to check that you have not blocked anything you didn’t mean to. So check, for example, the web address of a category and of a product, as well as the one you wanted to block. You can use a free online tool such as the robots.txt tester from technicalseo.com (opens in a new tab).