How to read a remote web page with ASP.NET 2.0

Two classes in the System.Net namespace make it very easy to obtain the html of a remote web page. These are the HttpWebRequest and HttpWebResponse. Here's a quick demo.

The static method below makes an http request for a web page, and the resulting string of html within the http response is captured and returned to the caller.

[C#]

//System.Net
//System.IO
static string GetHtmlPage(string strURL)
{

  String strResult;
  WebResponse objResponse;
  WebRequest objRequest = HttpWebRequest.Create(strURL);
  objResponse = objRequest.GetResponse();
  using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
  {
    strResult = sr.ReadToEnd();
    sr.Close();
  }
  return strResult;
}
[VB]
'System.Net
'System.IO
Shared Function GetHtmlPage(ByVal strURL As String) As String

  Dim strResult As String
  Dim objResponse As WebResponse
  Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
  objResponse = objRequest.GetResponse()
  Using sr As New StreamReader(objResponse.GetResponseStream())
    strResult = sr.ReadToEnd()
      sr.Close()
  End Using
  Return strResult
End Function

And to call the method:

[C#]
string TheUrl = "http://www.mikesdotnetting.com/Default.aspx";
string response = GetHtmlPage(TheUrl);
[VB]
Dim TheUrl As String = "http://www.mikesdotnetting.com/Default.aspx"
Dim response As String = GetHtmlPage(TheUrl)

The HttpWebRequest object can be configured to supply header information. Most of the default values are null. For example, the User-Agent value is null, but can be set to anything you want. Cookies can also be "enabled" for the object through the CookiesContainer property, if required to maintain sessions.

Date Posted: Monday, May 21, 2007 9:38 PM
Last Updated: Friday, June 29, 2007 8:32 AM
Posted by: Mikesdotnetting
Total Views to date: 83692

8 Comments

Tuesday, June 9, 2009 12:32 PM - titus

How to get the content with out tag,images and unwanted information from this return result.Kindly help me

Tuesday, June 9, 2009 3:59 PM - Mike

@Titus

You need to use Regualr Expressions. Google "Regular Expressions Strip HTML" for examples.

Monday, June 22, 2009 8:35 PM - Michael

It seems there is a loss in translation with the reader.
The original file that is read has : 7S£ §I‚xsß*ُ£
The information read
from the file only reads : 7S�� �I�xs�*ُ�

Any ideas? or are the characters just not allowed

Sunday, August 2, 2009 11:41 PM - Michael

Hello again. Thanks all for the help.
I just converted to binary.

Friday, August 28, 2009 7:15 AM - kumar

Hi mike
Is there any technique through which we can read a single line or value from a webpage like reading a specific number from a webpage

thanks
Pawan Kumar

Tuesday, October 22, 2013 6:53 AM - Dipesh

Simple & Awesme... tq...

Saturday, March 8, 2014 6:17 AM - mahendra

thanx a lot

Wednesday, April 16, 2014 10:25 PM - Tina

When I try the C# version, I just get a blank response. It's as if it's not loading the URL at all. No errors, just nothing returned. Help?
Add your comment

If you have any comments to make about this article, please use this form to do so. Make sure that your comment relates specifically to the article above. More general comments can be posted through the form on the Contact page.

Please note, all comments are moderated, and some may not be published. The kind of things that will ensure your comment is deleted without ever seeing the light of day are as follows:

  • Not relevant to the article
  • Gratuitous links to your own site or product
  • Anything abusive or libellous
  • Spam
  • Anything in a language I don't understand including gibberish.

I do not pass email addresses on to spammers, so a valid one will assist me in responding to you personally if required.

Recent Comments

Bino 11/27/2014 7:05 PM
In response to MVC 5 with EF 6 in Visual Basic - Async and Stored Procedures with the Entity Framework
Copy +...

Manas 11/27/2014 5:30 AM
In response to Scheduled Tasks In ASP.NET With Quartz.Net
Hi Mike, Thank you for awesome article. My concern is it might impact website if we use or is...

priya 11/26/2014 6:50 PM
In response to Create PDFs in ASP.NET - getting started with iTextSharp
very nice.....its save my time...

ransems 11/24/2014 12:29 AM
In response to Adding A Controller
Love the article. I dislike that the world thinks c# articles are the way to go. Love the VB, keep...

Gautam 11/20/2014 8:01 AM
In response to I'm Writing A Book On WebMatrix
Hello Mike, I read your book, loved it! However, I have a few request/suggestions: 1) an example...

Bret Dev 11/19/2014 8:39 PM
In response to The Difference Between @Helpers and @Functions In WebMatrix
Excellent post! One concern - where can you place global @Functions code within an MVC project to Is...

Rob Farquharson 11/19/2014 4:28 PM
In response to iTextSharp - Links and Bookmarks
How can I place text at an absolute position on the page? Also, how can I rotate text?...

Andy 11/17/2014 8:08 PM
In response to MVC 5 with EF 6 in Visual Basic - Sorting, Filtering and Paging
Hello I'm testing your sorting instructions above. This is great and I was able to get it to work...

Gautam 11/17/2014 5:51 PM
In response to WebMatrix - Database Helpers for IN Clauses
Hi Mike, I am very new to programming: In the above example if I want to use a delete button the...

donramon 11/17/2014 3:22 PM
In response to Entity Framework 6 Recipe - Alphabetical Paging In ASP.NET MVC
Congratulations on your new website look and the excellent articles. Thank you!...