Skip to content
Alex Osborne edited this page Oct 17, 2024 · 20 revisions

This is the public wiki for the Heritrix archival crawler project.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Heritrix (sometimes spelled heretrix, or misspelled or mis-said as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

All topical contributions to this wiki (corrections, proposals for new features, new FAQ items, etc.) are welcome! Register using the link near the top-right corner of this page.

Webmasters!

Heritrix is designed to respect the robots.txt exclusion directives and META nofollow tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

If you notice our crawler behaving poorly – The Internet Archive uses archive.org_bot as User Agent when crawling – please send us email at [email protected].

(If you see a different User-Agent in your logs that still says 'heritrix', it may be someone else using this open-source software. In such a case, even if we can't directly change how your site is crawled, we are happy to help you interpret your logs and identify, contact, or block the source of any troublesome crawling.)

The newer wildcard extension to robots.txt is not yet supported (feature request).

Downloads

The most up to date release packages are the Heritrix 3.x dated releases. These are also available on Maven Central.

The Heritrix 3.x latest stable development snapshot is available here: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.4.0-SNAPSHOT/heritrix-3.4.0-SNAPSHOT-dist.zip

License

Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0. Some individual source code files are subject to or offered under other licenses.

For more details see https://raw.github.com/internetarchive/heritrix3/master/dist/LICENSE.txt

Latest Releases

Releases

Periodic dated releases are available from Github releases tab and Maven Central. These are currently maintained by the National Library of Australia and previously by the UK Web Archive.

Snapshot builds

Automated snapshot builds of the Heritrix git master branch can be downloaded from http://builds.archive.org/.

Outage note: Many services hosted on archive.org are currently temporarily unavailable (details)

Older releases

Heritrix 3.3.0-LBS-2016-02 (May 2016)

A 3rd party build of Heritrix. This release uses Heritrix 3.3.0 master from May 2016. It was tested by several institutions and found to be very stable. It addresses many serious issues in 3.2.0. Blog post about this release.

Binary (-dist) and source (-src) distributions are available at: https://sbforge.org/nexus/index.html#view-repositories;thirdparty~browsestorage

Source fork can be found here: https://github.com/Landsbokasafn/heritrix3/tree/LBS-2016-02

Heritrix 3.2.0 (Jan 2014)

Heritrix 3.2.0 final release is now available. See the following Release Notes page on the project wiki for full details: Release Notes - Heritrix 3.2.0

Binary (-dist) and source (-src) distributions are available at: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.2.0/

As always, problem reports, ideas, fix/feature contributions, and other kinds of feedback are all welcome here on the list and on the project wiki and JIRA issue tracker:

Heritrix Wiki: https://github.com/internetarchive/heritrix3/wiki

Heritrix JIRA: http://webarchive.jira.com/browse/HER

Heritrix 1.14.4 (May 2010)

Heritrix release 1.14.4 is now available at Sourceforge: https://sourceforge.net/projects/archive-crawler/files/

This is a 'micro' release with small bug fixes and improvements. Online release notes for 1.14.4 with a list of issues addressed are available at: Release Notes - 1.14.4

Project bug/issue tracking For Heritrix is in a JIRA-based system: http://webteam.archive.org/jira/browse/HER

If you register to report/comment on JIRA issues, that same login will work for wiki edits and comments, and we welcome people to 'be bold' in adding/updating/improving project wiki pages.

Documentation

For starting with Heritrix3, the Heritrix 3.0 and 3.1 User Guide provides getting-started and reference information.

Javadoc API documentation, which also serves as the canonical documentation for crawl operators on configuration of built-in modules:

The 1.x-based Heritrix User Manual covers getting started with Heritrix and many advanced topics. This User Manual is generally focused on Heritrix 1.X versions, not fully updated for 1.12/1.14 or the larger changes in 2.0/3.0, but provides a reasonable basis for getting started with Heritrix, especially 1.14.4.

There is a FAQ from the 1.x era: http://archive-crawler.sourceforge.net/faq.html

The Knowledge Base contains topical articles documenting parts of the crawler and common problems or usage scenarios.

For developers, the 1.x-based Heritrix Developer Manual provides a guide to extending and customizing Heritrix code for your own purposes, though of course the source code itself, which is fairly well-commented, is the best guide.

For future documentation improvements, we have a [Documentation Wishlist](Documentation Wishlist).

An Introduction to Heritrix provides more detailed information on the structure and design of Heritrix.

Some very-old info can still be gleaned from the old wiki (http://web.archive.org/web/*/http://crawler.archive.org/cgi-bin/wiki.pl?HomePage).

Mailing lists

Heritrix discussion: http://groups.yahoo.com/group/archive-crawler/

Heritrix source code commits: https://lists.sourceforge.net/lists/listinfo/archive-crawler-cvs

Development

As of October 2011, immediately following the 3.1.0 release, the Heritrix 3 source repository is hosted at github: https://github.com/internetarchive/heritrix3

Older source code can be found in sourceforge svn: http://sourceforge.net/scm/?type=svn&group_id=73833 Heritrix JIRA bug/feature/issue tracking: http://webarchive.jira.com/browse/HER [Note to the public: this site does not allow creation of an account with @gmail address :{]

Heritrix

Structured Guides:

Wiki index

FAQs

User Guide

Knowledge Base

Known Issues

Background Reading

Users of Heritrix

How To Crawl

Development

Clone this wiki locally