peter123 wrote:
How could I customize the robot.txt to allow google? |
By default, everyone is allowed, including google.
However, if you want to disallow someone specific, for example Yahoo! then you can do this:
# robots.txt file for my website # feel free to contact us @....
User-agent: Yahoo Disallow: /
|
In the above example yahoo will be disallowed from viewing the entire site... if you want it to be disallowed from only one folder then you can sepcify just that one folder's path.
For example: disallow /temp
this will prevent yahoo from viewing the temp folder.