How can I use .NET to traverse a directory structure in a website?
Simple question.
How can I use .NET to traverse a directory structure in a website?
What I want to do is access a specific directory and then browse the files looking for .html files and then download them.
I know how to use WebClient.DownloadString () to copy files, but how can I search directories?
a source to share
Http is missing directory listing / moving as part of the spec. The best you can do is hope that the default page for a directory is a list, and then you have to parse it looking for links to files in the same directory. There are no standards for the format, but it shouldn't be too hard to deduce the href attributes of all tags <a>
and then test them against the following conditions:
- no slashes like "file.html"
- full path to the same directory, eg "/the/directory/file.html" if you look at "/ the / directory"
- full path to the same directory on the same server, for example " http: //the.server/the/directory/file.html "
If the web server does not provide you with directory listings, you can always use the full web spider mode (just parse all the links on the page and visit all those that are on the same server and parse them, etc. and then create your own tree view structure), but many websites cannot easily do this.
a source to share
You have to analyze the downloaded files and look for tags <a>
to extract links. Repeat this process recursively until all the pages you want have loaded.
Try using the dedicated Html Agility Pack library . This .Net library has kill functions and it
is a flexible HTML parser that generates DOM read / write and support for simple XPATH or XSLT.
After that, it is easy to work with the document and it is very easy to extract any information using XPATH.
a source to share