Hi Sebastian,

The main problem with running broad crawls (over a larger number of hosts) lies in the memory overhead associated with each active host.

That is, each host that is being crawled requires memory, both for host specific information and the ‘top’ of the per host queue of waiting URIs.

One of the best way to combat this is to enable site first, limiting the number of active hosts to the bare minimum needed to fully occupy the crawler. Judging from your post I’m unsure if that is acceptiable to you since this will favor certain hosts until they are exhausted before moving on.

Another thing that can be done to mitigate the memory use is to limit the number of URIs kept in memory for each host. This is done by setting the “host-queues-memory-capacity” setting to a smaller value. The default is 200, with a true broad crawl this is too much. I’d suggest something in the range of 20-50. A low value will incur more disk access but save memory.

Other then the site-first and host-queues-memory-capacity there isn’t all that much you can to limit memory use by configuring Heritrix. I know the guys at the Archive are looking at some ideas for limiting memory use further but that is still a long way away.

As for hardware and heap size. Well the latter is easy. Assign as large a heap as the hardware allows. I’ve been running on a macine with 1.5GB RAM and I usually assign 1.25GB to the java heap.

As for hardware you’ll want plenty of memory (duh) and a fast processor also helps if you are concerned about the speed the crawl runs at. The crawl tends to be limited by the processing power availible. If you choose to limit memory use by lowering the host-queues-memory-capacity significantly a fast HD will be very useful. I’d suggest having different HD for state/scratch/log files and ARC files.

- Kris