Frequently Visiting Spider – Vegabondo

Good morning,

I am just generally curious and was wondering about which and how often search engines where indexing my forum, so I added the following to the end of default.asp.

As I have virtually no visits this code is not putting much of a load onto the server, about  15 rows per day, if your forum has a lot of visitors you may not want to do this.

If Session("BrowserTypeLogged") = "" Then

    openDatabase(strCon)

    strSQL = "INSERT INTO tblActualAgents (Agent_char,OS_Char)" & _

    " VALUES ('" & left(Request.ServerVariables("HTTP_USER_AGENT"),127) & "','" & left(Request.ServerVariables("HTTP_USER_AGENT"),127) & "')"

     'Set error trapping

    On Error Resume Next

     adoCon.Execute(strSQL)                                  

     'If an error has occurred write an error to the page

    If Err.Number <> 0 Then          Call errorMsg("An error has occurred while executing SQL query on database.", "Insert Browser Type", "default.asp")

    'Disable error trapping

    On Error goto 0

    CloseDatabase()

    Session("BrowserTypeLogged") = "Done"

end if

 And via SQL Server Enterprise Manager

 CREATE TABLE [IanSmithcse_forum].[tblActualAgents](

                        Ref_id int IDENTITY(1,1) NOT NULL,

                        DateCreated_date datetime NOT NULL DEFAULT GetDate(),

                        Agent_char nchar(128) NULL,

                        OS_char nchar(128) NULL

) ON [PRIMARY]

 GO

 Note that this table is not even indexed.

Querying this table I found this

2012-08-19 09:52:09.080       Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl

2012-08-19 09:53:25.210       Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl

2012-08-19 09:54:41.237       Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-

 I left it for a while and as far as I can see Vagabondo just keeps on querying every 75 seconds. It is not me, all the other search engines seem much more sensible.

I have added Vagabondo to my robots.txt and this is respected. Anybody know anything about this spider, I have looked at http://www.wise-guys.nl and there is no obvious reason why I should care about them?

Bye

Ian