How to achieve environment specific robots.txt

vikram.bishnoi

Hi Team,

Can you please how can we create environment specific robots.txt?

The requirment is to avoid crawling of lower environments and allow only PROD environment.

Please suggest.

Thanks

Vikram

fsmeier

Heyhey,

how are you creating the current robots.txt? And why is your robots.txt from a hopefully protected staging/testing environment publicly available to the crawlers?

All the best,

Florian