Currently it ignores robots.txt rules on subdomains
Hi there, I have the same issue with my landing page and web application’s subdomain.
Recrawl the websites to apply robots.txt rules for subdomains.
Great job Ivan, thank you!
thank you!