Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. Heritrix is designed to respect the robots.txt exclusion directives† and META nofollow tags. Please consider the load your crawl will place on seed sites and set politeness policies accordingly. Also, always identify your crawl with contact information in the User-Agent so sites that may be adversely affected by your crawl can contact you or adapt their server behavior accordingly.

Features

  • Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0
  • Heritrix is designed to respect the robots.txt exclusion directives† and META nofollow tags
  • Always identify your crawl with contact information in the User-Agent
  • Open-source, extensible, web-scale
  • Archival-quality web crawler project
  • Heritrix is primarily used on Linux

Project Samples

Project Activity

See All Activity >

Categories

Libraries

License

MIT License

Follow Heritrix

Heritrix Web Site

Other Useful Business Software
Attack Surface Management | Criminal IP ASM Icon
Attack Surface Management | Criminal IP ASM

For security operations, threat-intelligence and risk teams wanting a tool to get access to auto-monitored assets exposed to attack surfaces

Criminal IP’s Attack Surface Management (ASM) is a threat-intelligence–driven platform that continuously discovers, inventories, and monitors every internet-connected asset associated with an organization, including shadow and forgotten resources, so teams see their true external footprint from an attacker’s perspective. The solution combines automated asset discovery with OSINT techniques, AI enrichment and advanced threat intelligence to surface exposed hosts, domains, cloud services, IoT endpoints and other Internet-facing vectors, capture evidence (screenshots and metadata), and correlate findings to known exploitability and attacker tradecraft. ASM prioritizes exposures by business context and risk, highlights vulnerable components and misconfigurations, and provides real-time alerts and dashboards to speed investigation and remediation.
Learn More
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Heritrix!

Additional Project Details

Programming Language

Java

Related Categories

Java Libraries

Registered

2023-08-08