Skip to content

Commit

Permalink
Fix issue with spark_ec2 seeing empty security groups
Browse files Browse the repository at this point in the history
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR apache#827, now that the issue
has been reproduced.
  • Loading branch information
aarondav committed Sep 19, 2013
1 parent 2aff798 commit f589ce7
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions ec2/spark_ec2.py
Original file line number Diff line number Diff line change
Expand Up @@ -364,12 +364,12 @@ def get_existing_cluster(conn, opts, cluster_name, die_on_error=True):
slave_nodes = []
for res in reservations:
active = [i for i in res.instances if is_active(i)]
if len(active) > 0:
group_names = [g.name for g in res.groups]
for inst in active:
group_names = [g.name for g in inst.groups]
if group_names == [cluster_name + "-master"]:
master_nodes += res.instances
master_nodes.append(inst)
elif group_names == [cluster_name + "-slaves"]:
slave_nodes += res.instances
slave_nodes.append(inst)
if any((master_nodes, slave_nodes)):
print ("Found %d master(s), %d slaves" %
(len(master_nodes), len(slave_nodes)))
Expand Down

0 comments on commit f589ce7

Please sign in to comment.