About Web Crawler
Web Crawler can be used to get links, emails, images and files from a webpage or site.
Web Crawler has a simple and intuitive interface.
The crawler is multithreaded and optimized for performance. It scans the webpage based on MIME types and file extensions, so it can find hidden links.
The “Deep crawl” feature allows the crawler to search all the linked pages from the selected website.
After crawling, the Web Crawler will save all links and e-mail addresses to the selected folder, along with all the crawled files.
How to use
On the top is a box for entering the URL to crawl.
Underneath the URL box is a folder in which to save the crawled files. The last box is for file extensions that the crawler should look for. If the file extensions box is left empty, then the program only looks for links and e-mails on the page and saves them to the linkList.txt and emailList.txt files in the output directory.
The application is primarily meant for subpage crawling, but can crawl a whole website when the “deep crawl” option is checked. This option is very resource intensive as it tries to make parallel connections to the server for better performance.
0 komentar:
Posting Komentar