Skip to content

Commit

Permalink
Made amendments to chapters.
Browse files Browse the repository at this point in the history
  • Loading branch information
ayushpriya10 committed Mar 3, 2020
1 parent 7afaeba commit e4652cd
Show file tree
Hide file tree
Showing 15 changed files with 96 additions and 32 deletions.
6 changes: 5 additions & 1 deletion Task_Report/docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,8 @@ There are some terms used in this report which might not be common and/or might
|**Agent VM** | The VM which has a Jenkins Agent installed to use the distributed build architecture. |
|**Jenkins User** | The system user created on the VM after installing Jenkins. |
|**Infrastructure** | The environment with the two VMs, the Jenkins VM and the Production VM. |
|**EC2 Instance** | A virtual machine on AWS Cloud. |
|**EC2** | Elastic Cloud Compute; A virtual machine on AWS Cloud. |
|**ECS** | Elastic Container Service; A container orchestration service on AWS Cloud. |
|**ECR** | Elastic Container Registry; A container registry on AWS Cloud. |
|**RDS** | Relational Database Service; A managed relational DB on AWS Cloud. |
|**IAM** | Identity and Access Management; A service to manage and access resources on AWS Cloud.|
Binary file added Task_Report/docs/img/secret_value.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Task_Report/docs/img/secrets_fromvalue_arn.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Task_Report/docs/img/secrets_manager.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
21 changes: 17 additions & 4 deletions Task_Report/docs/moving_setup_to_aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Objective

The aim of this section is to shift the entire setup from the local machine to AWS Cloud to provide a solution to the 1st point of the [problem statement](problem_statement.md) under `Task 4`.
The aim of this section is to shift the entire setup from the local machine to AWS (Amazon Web Services) Cloud to provide a solution to the 1st point of the [problem statement](problem_statement.md) under `Task 4`.

## Configuring Jenkins with EC2 Instances

Expand Down Expand Up @@ -32,7 +32,7 @@ To start shifting the entire setup that I had locally on my machine, firstly I b

* Under 'Configure Security Group' page:
- I clicked on the "Add Rule" button to add a new 'Custom TCP Rule', gave '8080' as the 'Port Range' because that is where the Jenkins UI is accessible.
- Under the 'Source' column, I selected the 'My IP' option to allow access only from the office IP.
- Under the 'Source' column, I selected the 'My IP' option to allow access only from my current IP.
- I gave a brief description of both the rules I added for the instance.
![Security Rules](/img/security_rules.PNG)

Expand All @@ -43,7 +43,7 @@ To start shifting the entire setup that I had locally on my machine, firstly I b

After successfully starting the instance, I had to install Jenkins on it. I used the steps from the [previous section](/setting_up_vms/#installing-jenkins) that I wrote on the same.

Starting Jenkins after the installation, I encountered an issue that I had not faced when I was running it on my machine locally. The URLs from where Jenkins fetches plugins had a few redirects which it was not able to handle on its own and failed to install any plugin. To rectify this issue, I ended up using [Nginx](https://www.nginx.com/), which is a reverse proxy and was able to handle the redirects successfully. To install Nginx, I followed this [documentation](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04). I, however, skipped step 5 on 'Setting Up Server Blocks' as it was not needed in the context of the problem statement. Lastly, as part of configuring Nginx, I wrote a config file, `jenkins-config`, whose contents are mentioned below:
* Starting Jenkins after the installation, I encountered an issue that I had not faced when I was running it on my machine locally. The URLs from where Jenkins fetches plugins had a few redirects which it was not able to handle on its own and failed to install any plugin. To rectify this issue, I ended up using [Nginx](https://www.nginx.com/), which is a reverse proxy and was able to handle the redirects successfully. To install Nginx, I followed this [documentation](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04). I, however, skipped step 5 on 'Setting Up Server Blocks' as it was not needed in the context of the problem statement. Lastly, as part of configuring Nginx, I wrote a config file, `jenkins-config`, whose contents are mentioned below:

```nginx
server {
Expand Down Expand Up @@ -71,6 +71,19 @@ server {
}
```

* I placed this script under the `/etc/nginx/sites-available/` directory.
* Next, I created a _symlink_ to the config file, `jenkins-config`, as follows:

```bash
sudo ln -s /etc/nginx/sites-available/jenkins-config /etc/nginx/sites-enabled/
```

* Lastly, I reloaded the Nginx service to have it run the latest configuration:

```bash
sudo nginx -s reload
```

After resolving the issue, I installed the plugins that Jenkins recommends.

## Configuring SAST Tools
Expand All @@ -83,7 +96,7 @@ I, once again, made a `reports/` directory inside `Jenkins Home Directory` to st

The installation for SonarQube is divided into two halves - setting up SonarQube Server and setting up the SonarQube Scanner. I was using docker previously to run the SonarQube Server on the local setup and hence, I had to use available services on AWS to run the SonarQube server as a container. For the second half, I re-used the steps from my [documentation](/static_analysis/#sonarqube) on installing SonarQube.

* To set up a container with SonarQube Server running in it, I followed this [tutorial](https://itnext.io/run-your-containers-on-aws-fargate-c2d4f6a47fda) as it explained things in a simpler language as compared to other available articles. I, however, skipped steps `1` and `5` as I directly pulled the docker image from docker hub and I did not want to use a domain to point to the container. So, in essence, I started off by creating a cluster (which is a collective of services and tasks), created a new task definition (which is the details about the container) and lastly, created a service to run the task (container). There was one other thing where I deviated from the tutorial I used, in the security group configuration, I added rules to allow access to the container only from the Office IP by selecting the 'My IP' option under the 'Source' field and another rule to allow the Jenkins EC2 instance to access the container via its public IP.
* To set up a container with SonarQube Server running in it, I followed this [tutorial](https://itnext.io/run-your-containers-on-aws-fargate-c2d4f6a47fda) as it explained things in a simpler language as compared to other available articles. I, however, skipped steps `1` and `5` as I directly pulled the docker image from docker hub and I did not want to use a domain to point to the container. So, in essence, I started off by creating a cluster on ECS (which is a collective of services and tasks), created a new task definition (which is the details about the container) and lastly, created a service to run the task (container). There was one other thing where I deviated from the tutorial I used, in the security group configuration, I added rules to allow access to the container only from my current IP by selecting the 'My IP' option under the 'Source' field and another rule to allow the Jenkins EC2 instance to access the container via its public IP.

**Note**: I had a doubt initially whether or not I could pull images from Docker hub directly. After searching for a bit, it turned that I could. I just had to specify the repository URL structure as `docker.io/<docker-image-name>:version` while creating the task definition.

Expand Down
9 changes: 8 additions & 1 deletion Task_Report/docs/secrets_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,14 @@ aws secretsmanager create-secret --name <SECRET NAME> --secret-string <SECRET VA

**Note**: Using the console to create the secrets was a bit confusing initially. Selecting 'Other type of secrets' option prompted for either a `key-value` or a `plaintext` type secret. The field for entering a plaintext secret had a JSON-like template structure present which turned out to be unnecessary. Removing the template JSON with just the required value worked as well to create secrets from the web console.

* Here's the AWS Secrets Manager web console after I added all the secrets:
![AWS Secrets Manager](/img/secrets_manager.PNG)

* Below is an image depicting where to retrieve the Secret's ARN on the web console:
![Secret ARN](/img/secret_value.PNG)

* Now, instead of creating a _revision_ to the existing task definition, I created a new task definition following the [steps](/moving_setup_to_aws/#deploying-dvna-on-aws) I used earlier (skipping the first nine steps as they did not need to be redone) with the new task definition's name as `deployDvnaSecretsManager`. This time, however, instead of directly passing the values for the database configurations, I chose the `ValueFrom` option and passed the respective `Secret ARN` that I took note of in the first step as the value for each secret under the environment variables section while providing the container details.
![Fetching Secret with ARN](/img/secrets_fromvalue_arn.PNG)

* Next, I created a new service (`dvnaSSMDeployService`) under the previously created cluster named `deploymentCluster` and created a task from the task definition created in the previous step. This launched DVNA on a new ECS instance with the secrets retrieved from the Secrets Manager.

Expand Down Expand Up @@ -53,7 +60,7 @@ After Vault was configured and was ready to be used, I went through the steps, m

* To retrieve secrets I used the command - `vault kv get dvna/db` which printed all the secrets in that path (Database name, host, port, username, and password) in a tabular format. Now, I had to fetch just the secret out of the output, instead of formatting the output to suit my needs, Vault came with a utility to only print the value of the secret by specifying the name of the field I need. The command I used to retrieve the secrets' values was `vault kv get -field=<SECRET NAME> dvna/db` where `SECRET NAME` was one the values from - database, username, password, port, and host.

* Now, for the secrets to be retrieved in the pipeline, the vault needs to be unsealed first and then it should also be resealed after the secrets are fetched. To achieve this, firstly, I made use of the `withCredentials` section in the pipeline syntax to fetch secrets from Jenkins' credential manager to retrieve Vault Keys that we stored their earlier and used them with the unseal command in the pipeline stage. I then amended the script I used to stop the currently running ECS tasks and fetch new deployed ECS instances running the latest container images of DVNA and lastly, I sealed the vault again.
* Now, for the secrets to be retrieved in the pipeline, the vault needs to be unsealed first and then it should also be resealed after the secrets are fetched. To achieve this, firstly, I made use of the `withCredentials` section in the pipeline syntax to fetch secrets from Jenkins' credential manager to retrieve Vault Keys that we stored there earlier and used them with the unseal command in the pipeline stage. I then amended the script I used to stop the currently running ECS tasks and fetch new deployed ECS instances running the latest container images of DVNA and lastly, I sealed the vault again.

* Unsealing (and resealing) Vault with keys from Jenkins Secret Manager in the pipeline script:

Expand Down
20 changes: 18 additions & 2 deletions Task_Report/site/glossary/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -444,8 +444,24 @@ <h1 id="glossary">Glossary</h1>
<td>The environment with the two VMs, the Jenkins VM and the Production VM.</td>
</tr>
<tr>
<td align="center"><strong>EC2 Instance</strong></td>
<td>A virtual machine on AWS Cloud.</td>
<td align="center"><strong>EC2</strong></td>
<td>Elastic Cloud Compute; A virtual machine on AWS Cloud.</td>
</tr>
<tr>
<td align="center"><strong>ECS</strong></td>
<td>Elastic Container Service; A container orchestration service on AWS Cloud.</td>
</tr>
<tr>
<td align="center"><strong>ECR</strong></td>
<td>Elastic Container Registry; A container registry on AWS Cloud.</td>
</tr>
<tr>
<td align="center"><strong>RDS</strong></td>
<td>Relational Database Service; A managed relational DB on AWS Cloud.</td>
</tr>
<tr>
<td align="center"><strong>IAM</strong></td>
<td>Identity and Access Management; A service to manage and access resources on AWS Cloud.</td>
</tr>
</tbody>
</table>
Expand Down
Binary file added Task_Report/site/img/secret_value.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Task_Report/site/img/secrets_fromvalue_arn.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Task_Report/site/img/secrets_manager.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 19 additions & 4 deletions Task_Report/site/moving_setup_to_aws/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -820,7 +820,7 @@

<h1 id="shifting-local-setup-to-aws">Shifting Local Setup to AWS</h1>
<h2 id="objective">Objective</h2>
<p>The aim of this section is to shift the entire setup from the local machine to AWS Cloud to provide a solution to the 1st point of the <a href="../problem_statement/">problem statement</a> under <code>Task 4</code>.</p>
<p>The aim of this section is to shift the entire setup from the local machine to AWS (Amazon Web Services) Cloud to provide a solution to the 1st point of the <a href="../problem_statement/">problem statement</a> under <code>Task 4</code>.</p>
<h2 id="configuring-jenkins-with-ec2-instances">Configuring Jenkins with EC2 Instances</h2>
<h3 id="starting-an-ec2-instance">Starting an EC2 instance</h3>
<p>To start shifting the entire setup that I had locally on my machine, firstly I brought up an EC2 instance to install and run Jenkins on. Below mentioned are the steps to spin up an EC2 instance:</p>
Expand Down Expand Up @@ -856,7 +856,7 @@ <h3 id="starting-an-ec2-instance">Starting an EC2 instance</h3>
<p>Under 'Configure Security Group' page:</p>
<ul>
<li>I clicked on the "Add Rule" button to add a new 'Custom TCP Rule', gave '8080' as the 'Port Range' because that is where the Jenkins UI is accessible.</li>
<li>Under the 'Source' column, I selected the 'My IP' option to allow access only from the office IP.</li>
<li>Under the 'Source' column, I selected the 'My IP' option to allow access only from my current IP.</li>
<li>I gave a brief description of both the rules I added for the instance.
<img alt="Security Rules" src="/img/security_rules.PNG" /></li>
</ul>
Expand All @@ -868,7 +868,9 @@ <h3 id="starting-an-ec2-instance">Starting an EC2 instance</h3>
</ul>
<h3 id="installing-jenkins-on-ec2-instance">Installing Jenkins on EC2 Instance</h3>
<p>After successfully starting the instance, I had to install Jenkins on it. I used the steps from the <a href="/setting_up_vms/#installing-jenkins">previous section</a> that I wrote on the same.</p>
<p>Starting Jenkins after the installation, I encountered an issue that I had not faced when I was running it on my machine locally. The URLs from where Jenkins fetches plugins had a few redirects which it was not able to handle on its own and failed to install any plugin. To rectify this issue, I ended up using <a href="https://www.nginx.com/">Nginx</a>, which is a reverse proxy and was able to handle the redirects successfully. To install Nginx, I followed this <a href="https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04">documentation</a>. I, however, skipped step 5 on 'Setting Up Server Blocks' as it was not needed in the context of the problem statement. Lastly, as part of configuring Nginx, I wrote a config file, <code>jenkins-config</code>, whose contents are mentioned below:</p>
<ul>
<li>Starting Jenkins after the installation, I encountered an issue that I had not faced when I was running it on my machine locally. The URLs from where Jenkins fetches plugins had a few redirects which it was not able to handle on its own and failed to install any plugin. To rectify this issue, I ended up using <a href="https://www.nginx.com/">Nginx</a>, which is a reverse proxy and was able to handle the redirects successfully. To install Nginx, I followed this <a href="https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04">documentation</a>. I, however, skipped step 5 on 'Setting Up Server Blocks' as it was not needed in the context of the problem statement. Lastly, as part of configuring Nginx, I wrote a config file, <code>jenkins-config</code>, whose contents are mentioned below:</li>
</ul>
<pre><code class="nginx">server {
listen 80;
server_name &lt;EC2 PUBLIC IP&gt;;
Expand All @@ -894,14 +896,27 @@ <h3 id="installing-jenkins-on-ec2-instance">Installing Jenkins on EC2 Instance</
}
</code></pre>

<ul>
<li>I placed this script under the <code>/etc/nginx/sites-available/</code> directory.</li>
<li>Next, I created a <em>symlink</em> to the config file, <code>jenkins-config</code>, as follows:</li>
</ul>
<pre><code class="bash">sudo ln -s /etc/nginx/sites-available/jenkins-config /etc/nginx/sites-enabled/
</code></pre>

<ul>
<li>Lastly, I reloaded the Nginx service to have it run the latest configuration:</li>
</ul>
<pre><code class="bash">sudo nginx -s reload
</code></pre>

<p>After resolving the issue, I installed the plugins that Jenkins recommends.</p>
<h2 id="configuring-sast-tools">Configuring SAST Tools</h2>
<p>After Jenkins was set up on an EC2 instance on AWS, the next step was to add all the tools required to perform the various tests on the application. Below, I have mentioned how I went about doing the same on AWS with a note about any additional steps I took when I deviate from the setup instructions mentioned in <a href="/static_analysis">Static Analysis</a>.</p>
<p>I, once again, made a <code>reports/</code> directory inside <code>Jenkins Home Directory</code> to store all reports generated in a single location. This time, I also added a <code>tool_scripts/</code> directory in the same location to house the various scripts I was required to use through the entire pipeline.</p>
<h3 id="sonarqube">SonarQube</h3>
<p>The installation for SonarQube is divided into two halves - setting up SonarQube Server and setting up the SonarQube Scanner. I was using docker previously to run the SonarQube Server on the local setup and hence, I had to use available services on AWS to run the SonarQube server as a container. For the second half, I re-used the steps from my <a href="/static_analysis/#sonarqube">documentation</a> on installing SonarQube.</p>
<ul>
<li>To set up a container with SonarQube Server running in it, I followed this <a href="https://itnext.io/run-your-containers-on-aws-fargate-c2d4f6a47fda">tutorial</a> as it explained things in a simpler language as compared to other available articles. I, however, skipped steps <code>1</code> and <code>5</code> as I directly pulled the docker image from docker hub and I did not want to use a domain to point to the container. So, in essence, I started off by creating a cluster (which is a collective of services and tasks), created a new task definition (which is the details about the container) and lastly, created a service to run the task (container). There was one other thing where I deviated from the tutorial I used, in the security group configuration, I added rules to allow access to the container only from the Office IP by selecting the 'My IP' option under the 'Source' field and another rule to allow the Jenkins EC2 instance to access the container via its public IP.</li>
<li>To set up a container with SonarQube Server running in it, I followed this <a href="https://itnext.io/run-your-containers-on-aws-fargate-c2d4f6a47fda">tutorial</a> as it explained things in a simpler language as compared to other available articles. I, however, skipped steps <code>1</code> and <code>5</code> as I directly pulled the docker image from docker hub and I did not want to use a domain to point to the container. So, in essence, I started off by creating a cluster on ECS (which is a collective of services and tasks), created a new task definition (which is the details about the container) and lastly, created a service to run the task (container). There was one other thing where I deviated from the tutorial I used, in the security group configuration, I added rules to allow access to the container only from my current IP by selecting the 'My IP' option under the 'Source' field and another rule to allow the Jenkins EC2 instance to access the container via its public IP.</li>
</ul>
<p><strong>Note</strong>: I had a doubt initially whether or not I could pull images from Docker hub directly. After searching for a bit, it turned that I could. I just had to specify the repository URL structure as <code>docker.io/&lt;docker-image-name&gt;:version</code> while creating the task definition.</p>
<ul>
Expand Down
2 changes: 1 addition & 1 deletion Task_Report/site/search/search_index.json

Large diffs are not rendered by default.

Loading

0 comments on commit e4652cd

Please sign in to comment.