Have Geek Brain

Amazon EC2 - network issues

We are running a hadoop cluster on amazon ec2 and lately we have been having network issues like master not being able to connect to slave. We thought the reason was Amazon throttling network connections to the limit. So, we tried to establish a connection after a random delay from each slave node. But it did not help.

Are there any other suggestions?

Thanks. Bala

+2


a source to share


3 answers


Have you tried using the hadoop-ec2 scripts from cloudera? I have used them to create random clusters for my dissertation research and I have found that they work quite well. It takes a few minutes to set up, but once set up, you just do

hadoop-ec2 launch-cluster <clustername> <number of slaves>

and it sets up everything needed, and usually does a really good job. Sometimes node won't start or whatever, but it's easy enough to shutdown the cluster and try again, and it's not too expensive.



You can find instructions on how to configure them here:

http://archive.cloudera.com/docs/ec2.html
0


a source


Do you have the correct ports open in the security group that the cluster instances are using? I am not familiar with Hadoop, but if it uses a custom TCP / IP or UDP port for communication between nodes, you need to specify it in your security group.



0


a source


Using Amazon Elastic MapReduce would address many issues and provide IO support for S3 and between nodes, as well as a few custom AWS patches to improve reliability.

It might be wise to stay away from EC2 cluster scenarios unless you need a specific version of Hadoop, but you really shouldn't.

0


a source






More articles:


All Articles