In this category Optimising for search engines and AI chatbots
Optimising for search engines and AI chatbots
Add rules to robots.txt
If you want to customise your robots.txt – as part of your search engine optimisation, for example – that is no problem.
You can read more about robots.txt at Google, among other places:
Introduction to robots.txt at Google Search Central (opens in a new tab)
You add your own settings by entering them in the file directly from the menu SEO > Robots.txt

Test your changes
When you have made changes to your robots.txt, it is a good safety measure to check that you have not blocked anything you didn’t mean to. So check, for example, the web address of a category and of a product, as well as the one you wanted to block. You can use a free online tool such as the robots.txt tester from technicalseo.com (opens in a new tab).
