suitable hosting for Nutch

[elmouad]

Hello,

I want to create a search engine website using the open source software Nutch. My question is : do I need a dedicated server for that or a virtual private server is enough?.

Has someone experience with Nutch? what is the best hosting plan for using Nutch?

PS: I want to index about 10 Millions webpages.

Mouad

 

[handsonhosting]

How quickly do you want to index those pages - that's going to be your determining factor on this. The bigger the machine (or cluster of machines) the faster it will get done. I think you'll be dragging along if you opt for a VPS, but it depends on how quickly you need the indexing done and how often you're spidering back.

I've not worked with Nutch myself, but based on some of the reading that I've done it seems to be a memory intensive program, so be sure to get enough memory. There's a number of places that recommend NOT using a VPS server for this due to the constraints.

 

[SenseiSteve]

I think on a $50 budget, you need to leave "quick' out of your vocabulary. Good luck on your project.

 

[elmouad]

How quickly do you want to index those pages - that's going to be your determining factor on this. The bigger the machine (or cluster of machines) the faster it will get done. I think you'll be dragging along if you opt for a VPS, but it depends on how quickly you need the indexing done and how often you're spidering back.

I've not worked with Nutch myself, but based on some of the reading that I've done it seems to be a memory intensive program, so be sure to get enough memory. There's a number of places that recommend NOT using a VPS server for this due to the constraints.

The speed does not really matter, I need to reindex these pages once a 3 mounths, but the most important for me is the disk space I need at least 50 GB since I index about 1 million pages.

 

[handsonhosting]

Managed VPS or self managed? 50GB for $50 with ample bandwidth for all your indexing is going to be a tall order for most places.

May want to give out some detailed specs on what you need (memory, bandwidth etc) so that others can give you some more guidance.

 

[Carbon_Neutral]

MOD NOTE: Self-advertising is not allowed.