How to read a remote web page with ASP.NET 2.0

Two classes in the System.Net namespace make it very easy to obtain the html of a remote web page. These are the HttpWebRequest and HttpWebResponse. Here's a quick demo.

The static method below makes an http request for a web page, and the resulting string of html within the http response is captured and returned to the caller.

[C#]

//System.Net
//System.IO
static string GetHtmlPage(string strURL)
{

  String strResult;
  WebResponse objResponse;
  WebRequest objRequest = HttpWebRequest.Create(strURL);
  objResponse = objRequest.GetResponse();
  using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
  {
    strResult = sr.ReadToEnd();
    sr.Close();
  }
  return strResult;
}
[VB]
'System.Net
'System.IO
Shared Function GetHtmlPage(ByVal strURL As String) As String

  Dim strResult As String
  Dim objResponse As WebResponse
  Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
  objResponse = objRequest.GetResponse()
  Using sr As New StreamReader(objResponse.GetResponseStream())
    strResult = sr.ReadToEnd()
      sr.Close()
  End Using
  Return strResult
End Function

And to call the method:

[C#]
string TheUrl = "http://www.mikesdotnetting.com/Default.aspx";
string response = GetHtmlPage(TheUrl);
[VB]
Dim TheUrl As String = "http://www.mikesdotnetting.com/Default.aspx"
Dim response As String = GetHtmlPage(TheUrl)

The HttpWebRequest object can be configured to supply header information. Most of the default values are null. For example, the User-Agent value is null, but can be set to anything you want. Cookies can also be "enabled" for the object through the CookiesContainer property, if required to maintain sessions.

Date Posted:
Last Updated:
Posted by:
Total Views to date: 101055

8 Comments

- titus

How to get the content with out tag,images and unwanted information from this return result.Kindly help me

- Mike

@Titus

You need to use Regualr Expressions. Google "Regular Expressions Strip HTML" for examples.

- Michael

It seems there is a loss in translation with the reader.
The original file that is read has : 7S£ §I‚xsß*ُ£
The information read
from the file only reads : 7S�� �I�xs�*ُ�

Any ideas? or are the characters just not allowed

- Michael

Hello again. Thanks all for the help.
I just converted to binary.

- kumar

Hi mike
Is there any technique through which we can read a single line or value from a webpage like reading a specific number from a webpage

thanks
Pawan Kumar

- Dipesh

Simple & Awesme... tq...

- mahendra

thanx a lot

- Tina

When I try the C# version, I just get a blank response. It's as if it's not loading the URL at all. No errors, just nothing returned. Help?
Add your comment

If you have any comments to make about this article, please use this form to do so. Make sure that your comment relates specifically to the article above. More general comments can be posted through the form on the Contact page.

Please note, all comments are moderated, and I end up deleting quite a lot. The kind of things that will ensure your comment is deleted without ever seeing the light of day are as follows:

  • Requests to fix your code (post a question to forums.asp.net instead, please)
  • Gratuitous links to your own site or product
  • Anything abusive or libellous
  • Spam

I do not pass email addresses on to spammers, so a valid one will assist me in responding to you personally if required.

Recent Comments

kaleem 4/16/2015 9:14 AM
In response to ASP.NET MVC, Entity Framework, One-to-Many and Many-to-Many INSERTS
hi this is very helpful ASP.NET MVC, Entity Framework, One-to-Many and Many-to-Many INSERTS if to...

cranston mason 4/16/2015 8:49 AM
In response to Managing Checkboxes And Radios In ASP.NET Razor Web Pages
How do you include a checkbox at the View module and send its values to controller to be processed?...

vamsi 4/15/2015 5:40 AM
In response to Sessions in ASP.NET 5
good article...

James Chaney 4/14/2015 8:44 PM
In response to 7 C# 6.0 Features That Every ASP.NET Developer Should Know About
Question on #5 - I don't see where this gets you out of a NullReferenceException if returns null is...

Grey 4/14/2015 6:09 PM
In response to Getting the identity of the most recently added record
Nice work....

Anh Huynh 4/14/2015 3:36 PM
In response to Create PDFs in ASP.NET - getting started with iTextSharp
I was successfully create and display PDF file from my localhost but when I move the aspx to the I...

Hisham Abdullah Bin Ateya 4/14/2015 7:00 AM
In response to Sessions in ASP.NET 5
Thanks Mike for sharing this article. George it will be nice if we let the web developers implement...

CTR 4/14/2015 6:39 AM
In response to Integrating Web API with ASP.NET Razor Web Pages
Thanks Mike, new to web pages. learning a lot from your posts, Making Ajax calls to Web API extend...

Bayu Angkasa 4/13/2015 11:27 AM
In response to The Difference Between @Helpers and @Functions In WebMatrix
Four years after you wrote it, I still find out how lucky I am to read your posts ... Thank you Mike...

abedon 4/10/2015 9:17 PM
In response to ASP.NET 5 Middleware, Or Where Has My HttpModule Gone?
I feel the title of this article is not that felicitous. ASP.NET 5 is built of top of the concept...