How to read a remote web page with ASP.NET 2.0

4.3 (27 votes)

Two classes in the System.Net namespace make it very easy to obtain the html of a remote web page. These are the HttpWebRequest and HttpWebResponse. Here's a quick demo.

The static method below makes an http request for a web page, and the resulting string of html within the http response is captured and returned to the caller.

[C#]

//System.Net
//System.IO
static string GetHtmlPage(string strURL)
{

  String strResult;
  WebResponse objResponse;
  WebRequest objRequest = HttpWebRequest.Create(strURL);
  objResponse = objRequest.GetResponse();
  using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
  {
    strResult = sr.ReadToEnd();
    sr.Close();
  }
  return strResult;
}
[VB]
'System.Net
'System.IO
Shared Function GetHtmlPage(ByVal strURL As String) As String

  Dim strResult As String
  Dim objResponse As WebResponse
  Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
  objResponse = objRequest.GetResponse()
  Using sr As New StreamReader(objResponse.GetResponseStream())
    strResult = sr.ReadToEnd()
      sr.Close()
  End Using
  Return strResult
End Function

And to call the method:

[C#]
string TheUrl = "http://www.mikesdotnetting.com/Default.aspx";
string response = GetHtmlPage(TheUrl);
[VB]
Dim TheUrl As String = "http://www.mikesdotnetting.com/Default.aspx"
Dim response As String = GetHtmlPage(TheUrl)

The HttpWebRequest object can be configured to supply header information. Most of the default values are null. For example, the User-Agent value is null, but can be set to anything you want. Cookies can also be "enabled" for the object through the CookiesContainer property, if required to maintain sessions.

You might also like...

Date Posted:
Last Updated:
Posted by:
Total Views to date: 115777

8 Comments

- titus

How to get the content with out tag,images and unwanted information from this return result.Kindly help me

- Mike

@Titus

You need to use Regualr Expressions. Google "Regular Expressions Strip HTML" for examples.

- Michael

It seems there is a loss in translation with the reader.
The original file that is read has : 7S£ §I‚xsß*ُ£
The information read
from the file only reads : 7S�� �I�xs�*ُ�

Any ideas? or are the characters just not allowed

- Michael

Hello again. Thanks all for the help.
I just converted to binary.

- kumar

Hi mike
Is there any technique through which we can read a single line or value from a webpage like reading a specific number from a webpage

thanks
Pawan Kumar

- Dipesh

Simple & Awesme... tq...

- mahendra

thanx a lot

- Tina

When I try the C# version, I just get a blank response. It's as if it's not loading the URL at all. No errors, just nothing returned. Help?

Recent Comments

Abolfazl Roshanzamir 14/09/2016 05:36
In response to Loading ASP.NET Core MVC Views From A Database Or Other Location
Nice article. Thanke you so much ....

cyrus 02/09/2016 15:12
In response to ASP.NET Web Pages vNext or Razor Pages
I've got some news. As Damian stated in this link: https://github.com/aspnet/Mvc/issues/5208 “We...

Simon 01/09/2016 08:00
In response to Loading ASP.NET Core MVC Views From A Database Or Other Location
Thanks Mike, nice post and exactly what I was looking for. Like you said, I think I'll opt to the...

dave 20/08/2016 14:57
In response to ASP.NET Web Pages vNext or Razor Pages
Do SimplemembershipProvider in viewpages is supported?...

Steven 18/08/2016 04:40
In response to Entity Framework Code First and Stored Procedures
Can you provide the directives (using statements) you're using for EF7 example?...

yousaid 17/08/2016 22:08
In response to ASP.NET Web Pages vNext or Razor Pages
Increasingly, learning a Microsoft tool is no longer worth the return on investment. Too many tools...

jared 12/08/2016 05:54
In response to ASP.NET Web Pages vNext or Razor Pages
hi mike, just for clarification, is viewpages something different from webpages? is webpages still...

Jocke 08/08/2016 20:37
In response to Loading ASP.NET Core MVC Views From A Database Or Other Location
Good post! If this was to be implemented in a CMS where users can change the view files, how would I...

cyrus 05/08/2016 19:49
In response to ASP.NET Web Pages vNext or Razor Pages
I think adding these features to webpages make it complicated. msft forget webpages goal so we move...

Curt Smith 27/07/2016 20:38
In response to ASP.NET Web Pages vNext or Razor Pages
I am only slightly disappointed to hear that WebMatrix is officially dead, because I suspected this...