I would like to have a page that allows a user to conduct a search. I would like the action of the search to be an ASP page that in turn queries yahoo.com, altavista.digital.com, and excite.com, and returns the results back to a string variable in my ASP page.
I would then like to parse through this string, pulling out the relevant links from each search engine and create a new HTML stream with the links from all of the search engines, indicating the original source after the link.
What is the best way to capture content from another site for parsing and reformatting?
The Microsoft Internet Transfer control makes it a snap to capture HTML from any Web site. The control has a number of properties and methods that are useful, but one of the most important is the OpenURL method. Using a single call to this method, you can retrieve HTML into a variable. You can then parse this variable in any way that you like.
The following excerpt is from a Visual Basic application that uses an Internet Transfer Control named Inet1. This procedure will call the OpenURL method and will save the HTML from the page in a variable. The URL that is to be retrieved is defined in a textbox on the form. The textbox is named txtURL. The HTML is parsed and each hyperlink that is found in the HTML is displayed in a list box on the form. The list box is named List1.
Private Sub cmdGetLinks_Click() On Error GoTo ErrorHandler Dim strHTML List1.Clear List1.AddItem "Finding Links..." cmdGetLinks.Enabled = False strfTheURL = txtURL.Text strTemp = Inet1.OpenURL(txtURL.Text) List1.Clear strHTML = LCase(strTemp) cmdGetLinks.Enabled = True intLookForNextHREFHere = 1 Do strNextHref = InStr(intLookForNextHREFHere, strHTML, "href") If strNextHref < 1 Then Exit Do intWhereIsTheEQ = InStr(strNextHref, strHTML, "=") intLookHere = intWhereIsTheEQ + 1 strNextChar = Mid(strHTML, intLookHere, 1) Const DOUBLE_QUOTE = 34 ' ASCII NUMBER If strNextChar = Chr(DOUBLE_QUOTE) Then ' The delimiter is a double-quote, look for the next one intNextDelimiter = InStr(intLookHere + 1, strHTML, Chr(34)) strTheURL = Trim(Mid(strHTML, intLookHere + 1, intNextDelimiter - intLookHere - 1)) List1.AddItem strTheURL Else ' DOUBLE QUOTE CHARACTER NOT USED, ' LOOK FOR START OF URL Do Until Len(Trim(strNextChar)) > 0 intLookHere = intLookHere + 1 strNextChar = Mid(strHTML, intLookHere, 1) Loop ' NOW LOOK FOR END OF URL BY LOCATING THE NEXT SPACE CHARACTER ' OR THE NEXT HTML CLOSE TAG CHARACTER (>) WHICHEVER IS NEAREST If InStr(intLookHere, strHTML, " ") < InStr(intLookHere, strHTML, ">") Then intNextDelimiter = InStr(intLookHere, strHTML, " ") Else intNextDelimiter = InStr(intLookHere, strHTML, ">") End If strTheURL = Trim(Mid(strHTML, intLookHere, intNextDelimiter - intLookHere)) List1.AddItem strTheURL End If ' FIND NEXT URL intLookForNextHREFHere = intNextDelimiter Loop Exit Sub ErrorHandler: Const REQUEST_TIMED_OUT = 35761 Select Case Err.Number Case REQUEST_TIMED_OUT MsgBox Err.Number & " - " & Err.Description Case Else End SelectEnd Sub