How to read a remote web page with ASP.NET 2.0

Two classes in the System.Net namespace make it very easy to obtain the html of a remote web page. These are the HttpWebRequest and HttpWebResponse. Here's a quick demo.

The static method below makes an http request for a web page, and the resulting string of html within the http response is captured and returned to the caller.

[C#]

//System.Net
//System.IO
static string GetHtmlPage(string strURL)
{

  String strResult;
  WebResponse objResponse;
  WebRequest objRequest = HttpWebRequest.Create(strURL);
  objResponse = objRequest.GetResponse();
  using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
  {
    strResult = sr.ReadToEnd();
    sr.Close();
  }
  return strResult;
}
[VB]
'System.Net
'System.IO
Shared Function GetHtmlPage(ByVal strURL As String) As String

  Dim strResult As String
  Dim objResponse As WebResponse
  Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
  objResponse = objRequest.GetResponse()
  Using sr As New StreamReader(objResponse.GetResponseStream())
    strResult = sr.ReadToEnd()
      sr.Close()
  End Using
  Return strResult
End Function

And to call the method:

[C#]
string TheUrl = "http://www.mikesdotnetting.com/Default.aspx";
string response = GetHtmlPage(TheUrl);
[VB]
Dim TheUrl As String = "http://www.mikesdotnetting.com/Default.aspx"
Dim response As String = GetHtmlPage(TheUrl)

The HttpWebRequest object can be configured to supply header information. Most of the default values are null. For example, the User-Agent value is null, but can be set to anything you want. Cookies can also be "enabled" for the object through the CookiesContainer property, if required to maintain sessions.

Date Posted:
Last Updated:
Posted by:
Total Views to date: 103486

8 Comments

- titus

How to get the content with out tag,images and unwanted information from this return result.Kindly help me

- Mike

@Titus

You need to use Regualr Expressions. Google "Regular Expressions Strip HTML" for examples.

- Michael

It seems there is a loss in translation with the reader.
The original file that is read has : 7S£ §I‚xsß*ُ£
The information read
from the file only reads : 7S�� �I�xs�*ُ�

Any ideas? or are the characters just not allowed

- Michael

Hello again. Thanks all for the help.
I just converted to binary.

- kumar

Hi mike
Is there any technique through which we can read a single line or value from a webpage like reading a specific number from a webpage

thanks
Pawan Kumar

- Dipesh

Simple & Awesme... tq...

- mahendra

thanx a lot

- Tina

When I try the C# version, I just get a blank response. It's as if it's not loading the URL at all. No errors, just nothing returned. Help?

Recent Comments

Gayan 7/3/2015 6:20 AM
In response to 7 C# 6.0 Features That Every ASP.NET Developer Should Know About
Great Article thanks...

Semil 7/1/2015 7:03 AM
In response to iTextSharp - Drawing shapes and Graphics
I have created a rectangle using above methode. Now I want to add a text in the center of this How I...

Satyabrata Mohapatra 6/30/2015 6:12 PM
In response to Reading Excel Files Without Saving To Disk In ASP.NET
Ahh.....this is awesome. Happy to see after a long time you wrote a article on web form :D...

Marty 6/30/2015 7:16 AM
In response to Posting Data With jQuery AJAX In ASP.NET Razor Web Pages
Mike, you're the Man! Another great article. So incredibly helpful. I'm definitely going to buy your...

Rohan 6/30/2015 5:32 AM
In response to ASP.NET MVC 5 with EF 6 - Working With Files
Very good and helpful tutorial. Thanks. Just wanted to know what would be the max file size limit we...

Fernando 6/30/2015 1:59 AM
In response to Programmatically accessing data from DataSource controls
What if I want to pass parameters natively using the DataSourceSelectArguments object, instead of be...

pankaj 6/29/2015 3:13 PM
In response to How to retain carriage returns or line breaks in an ASP.NET web page
very nice i'm use this in my code thank you.... ...

Mike 6/29/2015 2:22 AM
In response to MVC 5 with EF 6 in Visual Basic - Sorting, Filtering and Paging
This is the first example that I have found that works....

Marty 6/28/2015 4:57 AM
In response to Posting Data With jQuery AJAX In ASP.NET Razor Web Pages
Mike, what if I don't want to render back the text to the browser, but I want to send it some other...

Mike 6/27/2015 4:00 PM
In response to Migrating Classic ASP To ASP.NET Razor Web Pages Part One- Razor Syntax And Visual Basic
have you used any of the code converters to convert classic asp to c#? If so, which one do you have...