Harvest-NG is a set of tools for building a standards-compliant Web crawler. It is implemented in Perl, and can provide a complete resource discovery system, along with a modular structure which allows others to easily customise the system to their own needs. The development of Harvest-NG was started as an attempt to preserve the strong features of the Harvest architecture, but to allow more rapid developing and prototyping from a cleaner, better structured codebase.
