-
Notifications
You must be signed in to change notification settings - Fork 26
Requirements for different archive sizes
Here follows a short overview of hardware requirements for a fresh SolrWayback setup. This wiki page needs elaboration.
Index workflow, search engine and frontend should be able to run using a total of 4GB of RAM on just about any current machine. In case of crash: Reindex.
SSD highly recommended, 4 CPU's, 8GB of RAM (need to test this - might need 10-12), single machine setup or 2 machines for redundancy, WARC index logistics from command line
SSD essential, RAM for caching, separation of index & search, multi machine, fully live index, WARC index logistics possible from command line but consider Hadoop/netsearch/generic workflow engine
Same as single collection, but consider freezing finished collections
As above, but automated logistics system, freezing of finished collections and highly recommended, focus on Solr sharding practical limitations
If everything is to be searched in the same cloud, strong focus on freezing and minimizing of shard/collection count vs. single shard size maximum om ~1TB is needed
Uncharted territory. Trivial to do by using multiple separate clouds, but hard if full corpus search is needed. Can be helped by compromising on indexed text size and features.