ec2 deployment docs added, tutorial from cli added

zehrilatezab · Aug 22, 2014 · 97a09d1 · 97a09d1
1 parent 635d5c8
commit 97a09d1
Show file tree

Hide file tree

Showing 6 changed files with 194 additions and 30 deletions.
diff --git a/ec2/README.txt b/ec2/README.txt
@@ -21,6 +21,10 @@ At a minimum, you need to specify an ssh key name and a security group name.
 % ./h2o-cluster-distribute-h2o.sh  --OR--  ./h2o-cluster-download-h2o.sh
 % [optional] ./h2o-cluster-distribute-aws-credentials.sh
 
+Note:  If you run h2o with an IAM role, it is not necessary to distribute
+       the aws-credentials to all the nodes in the cluster. The temporary
+       access key should be accessible by later version of H2O.
+
 Note:  Download may be faster than distribute, since download pulls from S3.
 
 Note:  Distributing the AWS credentials copies the Amazon AWS_ACCESS_KEY_ID

diff --git a/h2o-docs/source/deployment/ec2.rst b/h2o-docs/source/deployment/ec2.rst
@@ -0,0 +1,22 @@
+.. _EC2:
+
+
+H\ :sub:`2`\ O on EC2
+=====================
+
+The H\ :sub:`2`\ O platform is tested nightly in EC2
+for both single-node and multi-node configurations.
+
+For users who want to quickly give H\ :sub:`2`\ O a try, we provide
+an Amazon Machine Image to serve as an easy starting point.
+
+The README.txt file embedded below describes scripts that can be found
+in the EC2 directory of the downloadable zip file
+:ref:`GettingStartedFromaZipFile`, or in the EC2 directory of the
+Github repository :ref:`QuickstartGit`.
+
+.. toctree::
+   :maxdepth: 1
+
+   ec2_glossary
+   ec2_tutorial_cli
diff --git a/h2o-docs/source/newuser/ec2.rst → h2o-docs/source/deployment/ec2_glossary.rst b/h2o-docs/source/newuser/ec2.rst → h2o-docs/source/deployment/ec2_glossary.rst
@@ -1,22 +1,7 @@
-.. _EC2:
+.. _EC2_Glossary:
 
-
-H\ :sub:`2`\ O on EC2
-=====================
-
-The H\ :sub:`2`\ O platform is tested nightly in EC2
-for both single-node and multi-node configurations.
-
-For users who want to quickly give H\ :sub:`2`\ O a try, we provide
-an Amazon Machine Image to serve as an easy starting point.
-
-The README.txt file embedded below describes scripts that can be found
-in the EC2 directory of the downloadable zip file
-:ref:`GettingStartedFromaZipFile`, or in the EC2 directory of the
-Github repository :ref:`QuickstartGit`.
-
-Glossery of Parameters for Launching EC2 Instances
-""""""""""""""""""""""""""""""""""""""""""""""""""
+EC2 Glossary
+============
 
 **AMI ID/Image ID**
   The ID of the AMI image to launch and run, only change this when switching between preferances between different AWS virtualization (HVM vs Paravirtualization).
@@ -33,7 +18,7 @@ Glossery of Parameters for Launching EC2 Instances
 
 ::
 
-  { 
+  {
    "Version": "2012-10-17",
    "Statement": [
      {
@@ -43,7 +28,7 @@ Glossery of Parameters for Launching EC2 Instances
      }
    ]
   }
-  
+
 **IAM Profile Resource Name**
   The resource name of the IAM Instance profile to associate to the instance. This can be used instead of IAM Profile Name.
 
@@ -58,8 +43,3 @@ Glossery of Parameters for Launching EC2 Instances
   The names of the EC2 security groups with which to associate the instance.
 
 
-.. raw:: html
-
-    <div style="margin-top:10px;">
-      <iframe width=900 height=900 src="../bits/ec2/README.txt" frameborder="0" allowfullscreen></iframe>
-    </div>
diff --git a/h2o-docs/source/deployment/ec2_tutorial_cli.rst b/h2o-docs/source/deployment/ec2_tutorial_cli.rst
@@ -0,0 +1,158 @@
+.. _EC2_Tutorial_CLI:
+
+Launch H\ :sub:`2`\ O from Command Line
+=======================================
+Launch scripts are available on our .. _github: https://github.com/0xdata/h2o/
+
+#. Prerequisite: Install boto python library.
+    In order to run the scripts you must have the boto library installed, references available:
+    http://boto.readthedocs.org/en/latest/
+
+    http://www.amazon.com/Python-and-AWS-Cookbook-ebook/dp/B005ZTO0UW/ref=sr_1_1?ie=UTF8&qid=1379879111&sr=8-1&keywords=python+aws
+
+#. First edit *h2o-cluster-launch-instances.py* launch script for parameter changes, refer to :ref:`ec2_glossary` for help.
+
+	::
+
+	  # Environment variables you MUST set (either here or by passing them in).
+	  # -----------------------------------------------------------------------
+	  #
+	  os.environ['AWS_ACCESS_KEY_ID'] = '...'
+	  os.environ['AWS_SECRET_ACCESS_KEY'] = '...'
+	  os.environ['AWS_SSH_PRIVATE_KEY_FILE'] = '/path/to/private_key.pem'
+
+	  # Launch EC2 instances with an IAM role
+	  # --------------------------------------
+	  # Change either but not both the IAM Profile Name.
+	  iam_profile_resource_name = None
+	  iam_profile_name = 'testing_role'
+	  ...
+	  # SSH key pair name.
+	  keyName = 'testing_key'
+	  securityGroupName = 'SecurityDisabled'
+	  ...
+	  numInstancesToLaunch = 2
+	  instanceType = 't1.micro'
+	  instanceNameRoot = 'amy_is_testing'
+	  ...
+	  regionName = 'us-east-1'
+	  amiId = 'ami-ed550784'
+
+#. Launch the EC2 instances using the H\ :sub:`2`\ O AMI by running *h2o-cluster-launch-instances.py*.
+
+    ::
+
+      $ python h2o-cluster-launch-instances.py
+      Using boto version 2.27.0
+      Launching 2 instances.
+      Waiting for instance 1 of 2 ...
+        .
+        .
+        instance 1 of 2 is up.
+      Waiting for instance 2 of 2 ...
+        instance 2 of 2 is up.
+
+      Creating output files:  nodes-public nodes-private
+
+      Instance 1 of 2
+        Name:    amy_is_testing0
+        PUBLIC:  ec2-54-164-161-125.compute-1.amazonaws.com
+        PRIVATE: 172.31.21.154
+
+      Instance 2 of 2
+        Name:    amy_is_testing1
+        PUBLIC:  ec2-54-164-161-149.compute-1.amazonaws.com
+        PRIVATE: 172.31.21.155
+
+      Sleeping for 60 seconds for ssh to be available...
+      Testing ssh access ...
+
+      Distributing flatfile...
+
+#. Download the latest build of H\ :sub:`2`\ O onto each of the instances using *./h2o-cluster-distribute-h2o.sh*  --OR--  *./h2o-cluster-download-h2o.sh*. Download will typically be faster than distribute since the file is being downloaded from S3.
+
+    ::
+
+      $ ./h2o-cluster-download-h2o.sh
+      Fetching latest build number for branch master...
+      Fetching full version number for build 1480...
+      Downloading H2O version 2.7.0.1480 to cluster...
+      Downloading h2o.jar to node 1: ec2-54-164-161-125.compute-1.amazonaws.com
+      Downloading h2o.jar to node 2: ec2-54-164-161-149.compute-1.amazonaws.com
+      Warning: Permanently added 'ec2-54-164-161-125.compute-1.amazonaws.com,54.164.161.125' (RSA)
+      to the list of known hosts.
+      Warning: Permanently added 'ec2-54-164-161-149.compute-1.amazonaws.com,54.164.161.149' (RSA)
+      to the list of known hosts.
+      Unzipping h2o.jar within node 1: ec2-54-164-161-125.compute-1.amazonaws.com
+      Unzipping h2o.jar within node 2: ec2-54-164-161-149.compute-1.amazonaws.com
+      Copying h2o.jar within node 1: ec2-54-164-161-125.compute-1.amazonaws.com
+      Copying h2o.jar within node 2: ec2-54-164-161-149.compute-1.amazonaws.com
+      Success.
+
+#. Distribute a flatfile.txt of all the private node IP address.
+
+    ::
+
+      $ ./h2o-cluster-distribute-flatfile.sh
+      Copying flatfile to node 1: ec2-54-164-161-125.compute-1.amazonaws.com
+      flatfile.txt                             100%   40     0.0KB/s   00:00
+      Copying flatfile to node 2: ec2-54-164-161-149.compute-1.amazonaws.com
+      flatfile.txt                             100%   40     0.0KB/s   00:00
+      Success.
+
+#. [Optional] For users that want to import data from a private S3 bucket, permission must be given to each launched node. If the cluster was launched without an IAM profile and policy, then AWS credentials would have to be distributed to each node as a aws_credentials.properties file using *./h2o-cluster-distribute-aws-credentials.sh*. If cluster was launched with IAM profile H2O will detect the temporary credentials on the cluster.
+
+    ::
+
+      $ ./h2o-cluster-distribute-aws-credentials.sh
+      Copying aws credential files to node 1: ec2-54-164-161-125.compute-1.amazonaws.com
+      core-site.xml                              100%  500     0.5KB/s   00:00
+      aws_credentials.properties                 100%   82     0.1KB/s   00:00
+      Copying aws credential files to node 2: ec2-54-164-161-149.compute-1.amazonaws.com
+      core-site.xml                              100%  500     0.0KB/s   00:17
+      aws_credentials.properties                 100%   82     0.1KB/s   00:00
+      Success.
+
+
+#. Start H\ :sub:`2`\ O by executing *./h2o-cluster-start-h2o.sh*.
+
+::
+
+  $ h2o-cluster-start-h2o.sh
+  Starting on node 1: ec2-54-164-161-125.compute-1.amazonaws.com...
+  JAVA_HOME is ./jdk1.7.0_40
+  java version "1.7.0_40"
+  Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
+  Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
+  01:55:18.438 main      INFO WATER: ----- H2O started -----
+  01:55:18.632 main      INFO WATER: Build git branch: master
+  01:55:18.633 main      INFO WATER: Build git hash: 1fbeb98671c73d4e2a61fc3defecb6bd1646c4d5
+  01:55:18.633 main      INFO WATER: Build git describe: nn-2-9356-g1fbeb98
+  01:55:18.634 main      INFO WATER: Build project version: 2.7.0.1480
+  01:55:18.634 main      INFO WATER: Built by: 'jenkins'
+  01:55:18.635 main      INFO WATER: Built on: 'Thu Aug 21 23:51:30 PDT 2014'
+  01:55:18.635 main      INFO WATER: Java availableProcessors: 1
+  01:55:18.649 main      INFO WATER: Java heap totalMemory: 0.01 gb
+  01:55:18.649 main      INFO WATER: Java heap maxMemory: 0.14 gb
+  01:55:18.650 main      INFO WATER: Java version: Java 1.7.0_40 (from Oracle Corporation)
+  01:55:18.651 main      INFO WATER: OS   version: Linux 2.6.32-358.14.1.el6.x86_64 (amd64)
+  01:55:18.959 main      INFO WATER: Machine physical memory: 0.58 gb
+  Starting on node 2: ec2-54-164-161-149.compute-1.amazonaws.com...
+  JAVA_HOME is ./jdk1.7.0_40
+  java version "1.7.0_40"
+  Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
+  Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
+  01:55:21.983 main      INFO WATER: ----- H2O started -----
+  01:55:22.067 main      INFO WATER: Build git branch: master
+  01:55:22.068 main      INFO WATER: Build git hash: 1fbeb98671c73d4e2a61fc3defecb6bd1646c4d5
+  01:55:22.068 main      INFO WATER: Build git describe: nn-2-9356-g1fbeb98
+  01:55:22.069 main      INFO WATER: Build project version: 2.7.0.1480
+  01:55:22.069 main      INFO WATER: Built by: 'jenkins'
+  01:55:22.069 main      INFO WATER: Built on: 'Thu Aug 21 23:51:30 PDT 2014'
+  01:55:22.070 main      INFO WATER: Java availableProcessors: 1
+  01:55:22.082 main      INFO WATER: Java heap totalMemory: 0.01 gb
+  01:55:22.082 main      INFO WATER: Java heap maxMemory: 0.14 gb
+  01:55:22.083 main      INFO WATER: Java version: Java 1.7.0_40 (from Oracle Corporation)
+  01:55:22.084 main      INFO WATER: OS   version: Linux 2.6.32-358.14.1.el6.x86_64 (amd64)
+  01:55:22.695 main      INFO WATER: Machine physical memory: 0.58 gb
+  Success.
diff --git a/h2o-docs/source/index.rst b/h2o-docs/source/index.rst
@@ -14,11 +14,11 @@ Getting Started with H\ :sub:`2`\ O
 
 This section is for new users. Here you can find information on
 getting started from downloaded versions of  H\ :sub:`2`\ O, and using  H\
-:sub:`2`\ O with R. 
+:sub:`2`\ O with R.
 
 .. toctree::
     :maxdepth: 1
-    
+
     newuser/top
     Ruser/top
     userguide/top
@@ -33,7 +33,7 @@ Tutorials in H\ :sub:`2`\ O
 
     tutorial/top
     tutorial/videos
-    
+
 
 Deployment and Big Data Management
 ==================================
@@ -43,6 +43,7 @@ Deployment and Big Data Management
 
     deployment/hadoop
     deployment/multinode
+    deployment/ec2
 
 Helpful Resources
 =================
@@ -52,7 +53,7 @@ in the public domain.
 
 .. toctree::
     :maxdepth: 1
-   
+
     resources/glossary
     resources/algoroadmap
     resources/publicdata

diff --git a/h2o-docs/source/newuser/top.rst b/h2o-docs/source/newuser/top.rst
@@ -11,5 +11,4 @@ New Users
 
 
    quickstart_jar
-   ec2
    java_help
Original file line number	Diff line number	Diff line change
Expand Up		@@ -11,5 +11,4 @@ New Users


		quickstart_jar
		ec2
		java_help