Why is the WebClient timeout when trying to crawl this shortened URL?
I have a small web crawler that sometimes has to crawl twitter and fetch urls. I am using a modified version of the Webclient class provided in the .net Framework.
This usually works fine, even with shortened URLs from sites.
However, when navigating to the shortened URL, the web client expires.
Do you think they filter certain clients?
How can I fix this or why is this happening?
a source to share
Are you sure you can hit this URL from your network without missing the proxy?
Does your control match the web client redirects? You can test this by creating a TinyURL and see if your web client can view it.
If you are viewing the proxy in your browser, you need to configure it in the WebClient control.
It is not easy to check if clients are filtering - set UserAgent on Request object to match FireFox object.
a source to share