How to read a remote web page with ASP.NET 2.0

4.3 (27 votes)

Two classes in the System.Net namespace make it very easy to obtain the html of a remote web page. These are the HttpWebRequest and HttpWebResponse. Here's a quick demo.

The static method below makes an http request for a web page, and the resulting string of html within the http response is captured and returned to the caller.

[C#]

//System.Net
//System.IO
static string GetHtmlPage(string strURL)
{

  String strResult;
  WebResponse objResponse;
  WebRequest objRequest = HttpWebRequest.Create(strURL);
  objResponse = objRequest.GetResponse();
  using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
  {
    strResult = sr.ReadToEnd();
    sr.Close();
  }
  return strResult;
}
[VB]
'System.Net
'System.IO
Shared Function GetHtmlPage(ByVal strURL As String) As String

  Dim strResult As String
  Dim objResponse As WebResponse
  Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
  objResponse = objRequest.GetResponse()
  Using sr As New StreamReader(objResponse.GetResponseStream())
    strResult = sr.ReadToEnd()
      sr.Close()
  End Using
  Return strResult
End Function

And to call the method:

[C#]
string TheUrl = "http://www.mikesdotnetting.com/Default.aspx";
string response = GetHtmlPage(TheUrl);
[VB]
Dim TheUrl As String = "http://www.mikesdotnetting.com/Default.aspx"
Dim response As String = GetHtmlPage(TheUrl)

The HttpWebRequest object can be configured to supply header information. Most of the default values are null. For example, the User-Agent value is null, but can be set to anything you want. Cookies can also be "enabled" for the object through the CookiesContainer property, if required to maintain sessions.

Date Posted:
Last Updated:
Posted by:
Total Views to date: 110109

8 Comments

- titus

How to get the content with out tag,images and unwanted information from this return result.Kindly help me

- Mike

@Titus

You need to use Regualr Expressions. Google "Regular Expressions Strip HTML" for examples.

- Michael

It seems there is a loss in translation with the reader.
The original file that is read has : 7S£ §I‚xsß*ُ£
The information read
from the file only reads : 7S�� �I�xs�*ُ�

Any ideas? or are the characters just not allowed

- Michael

Hello again. Thanks all for the help.
I just converted to binary.

- kumar

Hi mike
Is there any technique through which we can read a single line or value from a webpage like reading a specific number from a webpage

thanks
Pawan Kumar

- Dipesh

Simple & Awesme... tq...

- mahendra

thanx a lot

- Tina

When I try the C# version, I just get a blank response. It's as if it's not loading the URL at all. No errors, just nothing returned. Help?

Recent Comments

Ghazanfar 30/01/2016 06:43
In response to Getting Started with ASP.NET MVC 5 using Visual Basic
Nice working. Please keep it up to convert csharp code into vb.net. Its very helpful for vb...

sara 29/01/2016 09:39
In response to Simple Login and Redirect for ASP.NET and Access
Hi there, I am trying to validate and check for inputs entered. When I don't enter any inputs and...

Martin Thatcher 28/01/2016 17:28
In response to MVC 5 with EF 6 in Visual Basic - Advanced Entity Framework Scenarios
A small typo I think. In the code section that begins Function Index(ByVal SelectedDepartment As As...

Suresh_thefame 28/01/2016 08:03
In response to Sessions and Shopping Carts
Helpful....

Andrey Kurdyumov 28/01/2016 05:47
In response to ASP.NET 5: Uploading files with ASP.NET MVC 6
@Lee IFormFile has OpenReadStream(): Stream method...

Alisa 27/01/2016 18:37
In response to A Better Way To Export Gridviews To Excel
Thanks for this example! I am also asking about the date fields having to be re-formatted when the a...

Dan Buckley 27/01/2016 00:47
In response to What ASP.NET Can And Cannot Do
New to all things programming, this was very helpful and clear. Please write more....

satyabrata 26/01/2016 16:16
In response to Request.Form Is Empty When Posting To ASPX Page
Thank you....

david sanchez 26/01/2016 09:51
In response to ASP.NET MVC 5 with EF 6 - Working With Files
Thanks! This line " <img src="~/images/@Model.FilePaths.First(f => f.FileType == alt="" />" an...

Bryon 25/01/2016 15:06
In response to Windows Authentication With ASP.NET Web Pages
Was hoping this would help solve the issues I'm having. Then I saw the dates and new it was too old....