how to extract data from websites
Printed From: Web Wiz Forums
Category: General Discussion
Forum Name: Classic ASP Discussion
Forum Description: Discussion on Active Server Pages (Classic ASP).
URL: https://forums.webwiz.net/forum_posts.asp?TID=18104
Printed Date: 29 March 2026 at 2:58pm
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com
Topic: how to extract data from websites
Posted By: liza
Subject: how to extract data from websites
Date Posted: 24 January 2006 at 3:57am
|
Hi, I would like to know how data can be automatically extracted from websites, such as from asp pages.
Is it possible to automatically select from the drop-down list, type in the text boxes, and automatically click on the submit button. And then do this over and over again for all the items in the drop-down list?
It seems like it is possible, right? Otherwise, how do those shopping comparison websites do it?
Thanks in advance.
Liza
|
Replies:
Posted By: dpyers
Date Posted: 24 January 2006 at 9:51pm
|
In asp, you can use serverXMLHTTP to pull back a screen in a string that you can then search/screenscrape. You can also use it to post form data although some sites will check http_referrer to ensure that form data is only posted from their domain. Many of the comparison sites use an api to pick up a feed from the sites they compare . ------------- 
Lead me not into temptation... I know the short cut, follow me. |
Posted By: liza
Date Posted: 27 January 2006 at 7:40am
|
Thanks for your response. Could you also point me in the direction of learning about those things that you mentioned? websites, books, etc.
Thanks!
|
Posted By: dpyers
Date Posted: 28 January 2006 at 12:20am
|
There's an example of serverxmlhttp here http://new2asp.com/Section_Files/PullStrings/PullStrings.asp - http://new2asp.com/Section_Files/PullStrings/PullStrings.asp You can use either get or put. a put will ususally require adding a query string to the url. You can probably contact the sites you want to get the info from to find out what api's they use. There's industry specifix xml feeds and often site specific ones. Always get permission from the sites to either screen scrape or use their api. You'll be exposed to legal liability if you don't. They can also complain to your webhost and your isp about theft of content, badwidth theft, etc. You could wake up one morning and find yourself on a badgut list with no website and no internet connection. -------------
Lead me not into temptation... I know the short cut, follow me. |