Awesome

S2Robot

Overview

S2Robot is a crawler library for crawling a web site and a file system.

Note

The project was moved to Fess Crawler: https://github.com/codelibs/fess-crawler