Issue
If I remove robots.txt from my webapps root directory, it's allow the Google bot to crawl pages in my site?
We have already disallowed all th bots, but we want to remove it. so pls clarify, for bots does missing robots.txt file means don't crawl into the site?
Solution
A missing robots.txt file, means it's open for unlimited crawling by anyone.
Also, most websites don’t need a robots.txt file.
It is better practice to have an robots.txt listing disallowed paths, than rejecting/blocking the HTTP requests based on the User-Agent string.
A little side note:
On dynamic web pages, it's relatively easy to filter bots on runtime, using the User-Agent string, but it may be more difficult to rejecting bots on static assets, like files or images.
Also, many bots doesn't even have the word bot or crawler in it's User-Agent string, making it harder to differentiate humans from bots.
Answered By - jpg
