NIAGADS
  • VCPA
  • Introduction
  • Step 1: Set up the Amazon Web Services (AWS) environment
    • 1.1 Create AWS account
    • 1.2 Configure your computing environment and login to AWS
    • 1.3 Setup a S3 bucket (simple storage solution for AWS) for hosting sequencing data
    • 1.4 Install AWS command line software for accessing S3 bucket via command line interface
    • 1.5 Install StarCluster for AWS instance provisioning (optional)
  • Step 2: Create your tracking database instance
    • Option 1: Setup sample tracking database using Public AMI (recommended)
    • Option 2: Setup sample tracking database using Docker
  • Step 3: Configure your project information in the tracking database
    • 3.1 List all projects in the tracking database
    • 3.2 Create the project in the tracking database
  • Step 4: Upload sequencing data to your S3 bucket
  • Step 5: Configure your samples information in the tracking database
    • 5.1 Input the sample information to the tracking database
    • 5.2 Populate the tracking database with the S3 paths for the samples to be processed
    • 5.3 Populate the tracking database with the designated result folder for each sample to be processed
    • 5.4 Input PCR protocol information into the tracking database
    • 5.5 Add the capture kit information (WES sample only) into the tracking database
    • 5.6 Generate an ID to represent the capture kit information (WES sample only)
  • Step 6: Submit a job to process one whole genome (WGS) / whole exome (WES) sample
    • 6.1 Update vcpa-pipeline bitbucket contents
    • 6.2 Choose which workflow to use
    • 6.3 Enter your AWS credentials into the workflow script
    • 6.4 Launch Amazon EC2 Spot Instances via starcluster
  • Step 7: Review quality metrics of processed data
  • Step 8: Generating Project-level VCF via joint genotyping
  • Optional: Change software versions and dependencies of the VCPA workflow
Powered by GitBook
On this page
  • Install starcluster
  • Setup StarCluster cluster template (i.e. configure file)
  • Set Amazon Machine Image of ID in Starcluster config
  1. Step 1: Set up the Amazon Web Services (AWS) environment

1.5 Install StarCluster for AWS instance provisioning (optional)

Previous1.4 Install AWS command line software for accessing S3 bucket via command line interfaceNextStep 2: Create your tracking database instance

Last updated 6 years ago

Important note: If users are using option 1 in Step 2 (i.e. Public AMI) to set up the tracking database, users DO NOT need to install the starcluster, but setting up the cluster template (i.e. configure file) is a MUST. Users should re-visit this session after setting up the tracking database AMI in Step 2 (Option 1)

Install starcluster

StarCluster is an open source cluster-computing toolkit for Amazon's Elastic Compute Cloud (EC2). StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand. In VCPA, we launch instance via starcluster.

Users can download Starcluster version 0.95.6 here .

To install starcluster:

 $ tar xvzf starcluster-0.95.6.tar.gz
 $ cd starcluster-0.95.6
 $ sudo python setup.py install

Setup StarCluster cluster template (i.e. configure file)

To successfully launch the starcluster on EC2, users must create a cluster template (i.e. Starcluster configure file). These templates are a collection of settings that define a single cluster configuration and are used when creating and configuring a cluster. Starcluster has a default template cluster smallcluster. You can modify it according to your needs. You can also define multiple cluster templates, from which you can choose your template while launching the instance. Below is an example of the "smallcluster template"

DEFAULT_TEMPLATE=smallcluster[aws info]aws_access_key_id =       ###your aws access key id hereaws_secret_access_key =   ###your secret aws access key hereaws_user_id =             ###your 12-digit aws user id here# You can create a new key with starcluster# starcluster createkey -o ~/.ssh/aws_security_key.pem aws_security_key[key aws_security_key]KEY_LOCATION=~/.ssh/aws_security_key.rsa[cluster smallcluster]KEYNAME = aws_keyCLUSTER_SIZE = 1CLUSTER_USER = sgeadminCLUSTER_SHELL = bashAWS_REGION_NAME = us-east-1#AVAILABILITY_ZONE = us-east-1aMASTER_IMAGE_ID = ami-2cb8cd53NODE_IMAGE_ID = ami-2cb8cd53 PLUGINS=sgeconfigMASTER_INSTANCE_TYPE = r3.8xlargeNODE_INSTANCE_TYPE = r3.8xlarge# Uncomment for SPOT PRICING#force_spot_master=TRUE#SPOT_BID=3.50[plugin sgeconfig]SETUP_CLASS = sgeconfig.SGEConfig

Optional: if users are not clear on how to input the contents of the template file, the following is a step-by-step guide:

1) type: sudo python setup.py install

Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu
!!! ERROR - config file /home/user/.starcluster/config does not exist
Options:
--------
[1] Show the StarCluster config template
[2] Write config template to /home/user/.starcluster/config
[q] Quit

Please enter your selection:

This shows that your Starcluster configure file does not exist. User will need to select the second option by typing 2 and pressing enter. This will give you a template to use to create a configuration file containing your AWS credentials, cluster settings, etc. The next step is to customize this file using your favorite text-editor:

$  vi ~/.starcluster/config

2) fill in the AWS credentials and key pair info which you created in the session 1.2.

[aws info]
aws_access_key_id =       ###your aws access key id here
aws_secret_access_key =   ###your secret aws access key here
aws_user_id =             ###your 12-digit aws user id here

3) fill in your key pairs information.

If you don’t have the key pairs, you can create one from StarCluster using the following command: 

$ starcluster createkey mykey -o ~/.ssh/aws_security_key.rsa

This will create key pairs called mykey on Amazon EC2 and save the private key to ~/.ssh/mykey.rsa.

4) fill-in your key pairs info in the StarCluster config file:

############################# Defining EC2 Keypairs #############################[key keyname]Key_location=/path/to/your/aws_security_key.rsa

Choose instance that is optimized for memory intensive applications (e.g. i3.8xlarge, r3.8xlarge, r4.8xlarge, d2.8xlarge)

Set Amazon Machine Image of ID in Starcluster config

It includes all resources required to launch an instance, which is a virtual server in the cloud. It includes the following:

  • A template for the root volume for the instance

  • Launch permission that control which AWS accounts can use the AMI to launch instances

  • A block device that specifies the volumes to attach to the instance when it’s launched

AWS image ID – describe what is in this image ID

MASTER_IMAGE_ID = ami-2cb8cd53

: Define at least one keypair section that represents one of your keypairs on Amazon EC2.

https://github.com/jtriley/StarCluster
Keypair