When I was researching about downloading pages and all content from a website. I got this crawler..its very simple. and you need to take a look at it. Its real cool and simple.
http://code.google.com/p/harvestman-crawler/wiki/WorldsSimplestCrawler
windows user if you have cygwin installed it would be helpful.
Leave a Reply