snapey wrote:
Is this anything I should be worried about?
Should I add this page to robots.txt or is it too late for that? |
You could add it to robots.txt, but if, as -boRg- suspects, these are email-harvesters, it'll probably make little difference.
My new site is on a test server and so the robots.txt is currently set to exclude all spiders from the entire site - yet there's still a large number of "visitors" at times (many more than the number of people I've told about the test location).
My guess is these are harvesters that simply ignore robots.txt because the originators really don't care, so long as they get any email addresses there might be about the place!