Sorry, first link got screwed.
https://codewithhugo.com/importance-crawler-bot-block/
Here are some helpful tips to prevent malicious bots from attacking your website:
Quote:
Adding an additional application plugin, such as the Wordfence security plugin for WordPress prevents such attacks.
It’s also advisable to set access rules to get rid of malicious requests. You can disallow a specific search engine by inputting the name of the search engine on the disallow user agent of your robots.txt file.
You can disallow other search engines from crawling your website except for Googlebot by only allowing Googlebot as your user agent in your robots.txt file.
Note: A robots.txt refers to a file that’s associated with your website. It’s used to ask web crawlers to crawl or not to crawl some parts or web pages of your website. In short, a robots.txt file specifies which of your web pages should be crawled by web crawlers or spiders.
|