1.5 Install StarCluster for AWS instance provisioning (optional)

Important note: If users are using option 1 in Step 2 (i.e. Public AMI) to set up the tracking database, users DO NOT need to install the starcluster, but setting up the cluster template (i.e. configure file) is a MUST. Users should re-visit this session after setting up the tracking database AMI in Step 2 (Option 1)

Install starcluster

StarCluster is an open source cluster-computing toolkit for Amazon's Elastic Compute Cloud (EC2). StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand. In VCPA, we launch instance via starcluster.

Users can download Starcluster version 0.95.6 here https://github.com/jtriley/StarCluster.

To install starcluster:

 $ tar xvzf starcluster-0.95.6.tar.gz
 $ cd starcluster-0.95.6
 $ sudo python setup.py install

Setup StarCluster cluster template (i.e. configure file)

To successfully launch the starcluster on EC2, users must create a cluster template (i.e. Starcluster configure file). These templates are a collection of settings that define a single cluster configuration and are used when creating and configuring a cluster. Starcluster has a default template cluster smallcluster. You can modify it according to your needs. You can also define multiple cluster templates, from which you can choose your template while launching the instance. Below is an example of the "smallcluster template"

DEFAULT_TEMPLATE=smallcluster[aws info]aws_access_key_id =       ###your aws access key id hereaws_secret_access_key =   ###your secret aws access key hereaws_user_id =             ###your 12-digit aws user id here# You can create a new key with starcluster# starcluster createkey -o ~/.ssh/aws_security_key.pem aws_security_key[key aws_security_key]KEY_LOCATION=~/.ssh/aws_security_key.rsa[cluster smallcluster]KEYNAME = aws_keyCLUSTER_SIZE = 1CLUSTER_USER = sgeadminCLUSTER_SHELL = bashAWS_REGION_NAME = us-east-1#AVAILABILITY_ZONE = us-east-1aMASTER_IMAGE_ID = ami-2cb8cd53NODE_IMAGE_ID = ami-2cb8cd53 PLUGINS=sgeconfigMASTER_INSTANCE_TYPE = r3.8xlargeNODE_INSTANCE_TYPE = r3.8xlarge# Uncomment for SPOT PRICING#force_spot_master=TRUE#SPOT_BID=3.50[plugin sgeconfig]SETUP_CLASS = sgeconfig.SGEConfig

Optional: if users are not clear on how to input the contents of the template file, the following is a step-by-step guide:

1) type: sudo python setup.py install

Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [e-mail]
!!! ERROR - config file /home/user/.starcluster/config does not exist
Options:
--------
[1] Show the StarCluster config template
[2] Write config template to /home/user/.starcluster/config
[q] Quit

Please enter your selection:

This shows that your Starcluster configure file does not exist. User will need to select the second option by typing 2 and pressing enter. This will give you a template to use to create a configuration file containing your AWS credentials, cluster settings, etc. The next step is to customize this file using your favorite text-editor:

$  vi ~/.starcluster/config

2) fill in the AWS credentials and key pair info which you created in the session 1.2.

[aws info]
aws_access_key_id =       ###your aws access key id here
aws_secret_access_key =   ###your secret aws access key here
aws_user_id =             ###your 12-digit aws user id here

3) fill in your key pairs information.

If you don’t have the key pairs, you can create one from StarCluster using the following command:

$ starcluster createkey mykey -o ~/.ssh/aws_security_key.rsa

This will create key pairs called mykey on Amazon EC2 and save the private key to ~/.ssh/mykey.rsa.

4) fill-in your key pairs info in the StarCluster config file:

Keypair: Define at least one keypair section that represents one of your keypairs on Amazon EC2.

############################# Defining EC2 Keypairs #############################[key keyname]Key_location=/path/to/your/aws_security_key.rsa

Choose instance that is optimized for memory intensive applications (e.g. i3.8xlarge, r3.8xlarge, r4.8xlarge, d2.8xlarge)

Set Amazon Machine Image of ID in Starcluster config

It includes all resources required to launch an instance, which is a virtual server in the cloud. It includes the following:

• A template for the root volume for the instance

• Launch permission that control which AWS accounts can use the AMI to launch instances

• A block device that specifies the volumes to attach to the instance when it’s launched

AWS image ID – describe what is in this image ID

MASTER_IMAGE_ID = ami-2cb8cd53