Hashicorp Terraform is an open-source tool for provisioning and managing cloud infrastructure. It codifies infrastructure in configuration files that describe the topology of cloud resources, such as virtual machines, storage accounts, and networking interfaces. Terraform’s command-line interface (CLI) provides a simple mechanism to deploy and version the configuration files to Azure or any other supported cloud.

This article describes the benefits of using Terraform to manage Azure infrastructure.

Automate infrastructure management.

Terraform’s template-based configuration files enable you to define, provision, and configure Azure resources in a repeatable and predictable manner. Automating infrastructure has several benefits:

• Lowers the potential for human errors while deploying and managing infrastructure.
• Deploys the same template multiple times to create identical development, test, and production environments.
• Reduces the cost of development and test environments by creating them on-demand.

Understand infrastructure changes before they are applied

As a resource topology becomes complex, understanding the meaning and impact of infrastructure changes can be difficult.

Terraform provides a command-line interface (CLI) that allows users to validate and preview infrastructure changes before they are deployed. Previewing infrastructure changes in a safe, productive manner has several benefits:

• Team members can collaborate more effectively by quickly understanding proposed changes and their impact.
• Unintended changes can be caught early in the development process

Deploy infrastructure to multiple clouds

Terraform is a popular tool choice for multi-cloud scenarios, where similar infrastructure is deployed to Azure and additional cloud providers or on-premises datacenters. It enables developers to use the same tools and configuration files to manage infrastructure on multiple cloud providers.

Next steps

Now that you have an overview of Terraform and its benefits, here are suggested next steps:

• Get started by installing Terraform and configuring it to use Azure.
• Create an Azure virtual machine using Terraform
• Explore the Azure Resource Manager module for Terraform

Estimated reading time: 2 minutes

If your container needs to use an HTTP, HTTPS, or FTP proxy server, you can configure it in different ways:

• In Docker 17.07 and higher, you can configure the Docker client to pass proxy information to containers automatically.
• In Docker 17.06 and lower, you must set appropriate environment variables within the container. You can do this when you build the image (which makes the image less portable) or when you create or run the container.

Configure the Docker client

1. On the Docker client, create or edit the file ~/.docker/config.json in the home directory of the user which starts containers. Add JSON such as the following, substituting the type of proxy with httpsProxy or ftpProxy if necessary, and substituting the address and port of the proxy server. You can configure multiple proxy servers at the same time.You can optionally exclude hosts or ranges from going through the proxy server by setting a noProxy key to one or more comma-separated IP addresses or hosts. Using the * character as a wildcard is supported, as shown in this example.

   {
    "proxies":
    {
      "default":
      {
        "httpProxy": "http://127.0.0.1:3001",
        "httpsProxy": "http://127.0.0.1:3001",
        "noProxy": "*.test.example.com,.example2.com"
      }
    }
   }
   

   Save the file.

2. When you create or start new containers, the environment variables are set automatically within the container.

Use environment variables

Set the environment variables manually

When you build the image, or using the --env flag when you create or run the container, you can set one or more of the following variables to the appropriate value. This method makes the image less portable, so if you have Docker 17.07 or higher, you should configure the Docker client instead.

Now copy your data recursively from your source volume to your newly created volume using the following command(Assuming your old volume is mounted at /mnt/ebs)

sudo cp -r –preserve=all /mnt/ebs/. Amazon’s Elastic Block Store Volumes are easy to use and expand but notoriously hard to shrink once their size has grown. Here are the steps for Shrinking any mounted EBS volume on EC2 Instances.

For various reasons you may need to expand or shrink the size of your EBS volume. /mnt/ebs1 –verbose
Copying data recursively preserves the file attributes such as mode,ownership,timestamps and security contexts, if possible additional attributes such as links.

5. /mnt/ebs1 –verbose
sudo cp -r –preserve=all /mnt/ebs/. After copying the data detach your old volume and mount your new volume at the same mount point using the following command

sudo umount -l /mnt/ebs
sudo umount -l /mnt/ebs

sudo mount /dev/sdi /mnt/ebs
sudo mount /dev/sdi /mnt/ebs
Automating the process by Shell Script
In order to automate all this process here is the Script which will automate the process

Download the zip from this page https://github.com/cloudthat/ebs-shrink

Extract the zip and navigate to the extracted folder

Step 1: Place the script on the instance with permissions to execute

scp -i /ebsreducescript @:/home/ec2-user/.
scp -i /ebsreducescript @:/home/ec2-user/.
Now SSH into the instance using the following command and Update file permissions

ssh -i ec2-user@ip-address
ssh -i ec2-user@ip-address

sudo chmod 500 ebsreducescript
sudo chmod 500 ebsreducescript
Step 2: Configure AWS Cli

Also before executing the script you need to configure awscli use the following commands to install

sudo yum install python-pip
sudo yum install python-pip

sudo pip install awscli
sudo pip install awscli
Run aws configure at the command line to set up your credentials and settings.

$ aws configure
AWS Access Key ID [None]: ANKITIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: ANKITXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-1
Default output format [None]: json

$ aws configure
AWS Access Key ID [None]: ANKITIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: ANKITXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-1
Default output format [None]: json
This Script Will Create a New Volume of User Defined,Mount it to /mnt/ebs1 Copy all the data from the attached EBS Volume recursively that need to be Migrated preserving all the user permissions,then Unmount it and Mount the Newly Created EBS Volume at the Same Mount Point

Note:The migrated volume is not deleted it is just unmounted So if any error occurs we can reattach it. Because you are only charged for the space currently allocated to your EBS volumes, it is also cost efficient to allocate only your approximate short term need and expand the volume as the need arises (this should be considered during the planning phase of your projects and/or during formulation of growth strategies for your service).

The below steps have been tested on an Amazon Linux instance to resize an EBS Volume using the AWS console

Create a new EBS Voume of your desired size from the console and attach it to your instance
Now login to your instance using ssh and format your new attached volume using the following command (Assuming that new volume is attached at /dev/sdi)

ssh -i ec2-user@ip-address
ssh -i ec2-user@ip-address

sudo mkfs -t ext4 /dev/sdi
sudo mkfs -t ext4 /dev/sdi
3.Then make a directory at /mnt/ebs1 and mount the new volume using following commands

sudo mkdir /mnt/ebs1
sudo mkdir /mnt/ebs1

sudo mount /dev/sdi /mnt/ebs1
sudo mount /dev/sdi /mnt/ebs1
4. This Script can only use for Attached(Mounted) EBS Volumes not Root Volume.

The key value which we need to define in the config file in this context is called Snitch. We shall check the status of the cluster using this command as shown below: cass The owns field above indicates the percentage of data owned by each node. The owns field indicates the percentage of data owned by the node.

Let us perform some tests to make sure the data was replicated intact across multiple Availability Zones.

Test 1:

Node 1 was stopped.
Connection was made to the Cluster on remaining nodes and records were read from the table user.
All records were intact.
Node 1 was started.
On Node 1, ‘nodetool repair -h hostname_of_Node1 repair first’ was run.
Connection was made to the Cluster on Node 1 and records were read from the table user.
All records were intact.
Test 2:

Node 1 and Node 2 were stopped. We can control how nodes are configured within a cluster, including inter-node communication, data partitioning and replica placement etc., in this config file. Using CQLSH, you can execute queries using Cassandra Query Language (CQL).

Next, we shall create a table user with 5 records for tests.

CREATE TABLE user(user_id text,login text,region text,PRIMARY KEY (user_id));
1
CREATE TABLE user(user_id text,login text,region text,PRIMARY KEY (user_id));
Now, let us insert some queries in this table:

insert into user (user_id,login,region) values (‘1′,’test.1,’IN’);
insert into user (user_id,login,region) values (‘2′,’test.2′,’IN’);
insert into user (user_id,login,region) values (‘3′,’test.3′,’IN’);
insert into user (user_id,login,region) values (‘4′,’test.4′,’IN’);
insert into user (user_id,login,region) values (‘5′,’test.5′,’IN’);

insert into user (user_id,login,region) values (‘1′,’test.1,’IN’);
insert into user (user_id,login,region) values (‘2′,’test.2′,’IN’);
insert into user (user_id,login,region) values (‘3′,’test.3′,’IN’);
insert into user (user_id,login,region) values (‘4′,’test.4′,’IN’);
insert into user (user_id,login,region) values (‘5′,’test.5′,’IN’);

cqlsh> select * from user;
1
cqlsh> select * from user;
query

Now that our keyspace/database consists of data, let us check for ownership & effectiveness:

12 (1) As we can see here, the owns field above is NOT nil after defining the keyspace. So, let us go ahead and create a sample keyspace. Apache Cassandra is an open source non-relational/NOSQL database. Basically, a snitch indicates as to which Region and Availability zones does each node in the cluster belongs to. Cassandra nodes use seeds for finding each other and learning the topology of the ring. But, in this case, we shall use EC2Snitch as all of our nodes in the cluster are within a single region.

We shall set the snitch value as shown below: snitch

Also, since we are using multiple nodes, we need to group our nodes. NetworkTopologyStrategy places replicas on distinct racks/AZs as sometimes, nodes in the same rack/AZ might usually fail at the same time due to power, cooling or network issues.

Let us set the replication factor to 3 for our “first” keyspace:

CREATE KEYSPACE “first” WITH REPLICATION ={‘class’ :’NetworkTopologyStrategy’, ‘us-east’ : 3};
1
CREATE KEYSPACE “first” WITH REPLICATION ={‘class’ :’NetworkTopologyStrategy’, ‘us-east’ : 3};
The above CQL command creates a database/keyspace ‘first’ with class as NetworkTopologyStrategy and 3 replicas in us-east (In this case, one replica in AZ/rack 1a, one replica in rack AZ/1b and one replica in rack AZ/1c). The nodetool utility is a command line interface for managing a cluster. It gives information about the network topology so as to the requests are routed efficiently. We shall create a keyspace with data replication strategy & replication factor. We will also learn how to ensure that the data remains intact even when an entire AZ goes down.

The initial setup consists of a Cassandra cluster with 6 nodes with 2 nodes (EC2s) spread across AZ-1a , 2 in AZ-1b and 2 in AZ-1c.

Initial Setup:
Cassandra Cluster with six nodes.

AZ-1a: us-east-1a: Node 1, Node 2
AZ-1b: us-east-1b: Node 3, Node 4
AZ-1c: us-east-1c: Node 5, Node 6
Next, we have to make changes in the Cassandra configuration file. This strategy will also help in case of disaster recovery.

Stay tuned for more blogs!!. Cassandra uses a command prompt called Cassandra Query Language Shell, also known as CQLSH, which acts as an interface for users to communicate with it. We shall use NetworkTopology replication strategy since we have our cluster deployed across multiple availability zones. Additionally, Cassandra has replication strategies which place the replicas based on the information provided by the snitch. cassandra.yaml file is the main configuration file for Cassandra. It is massively scalable and is designed to handle large amounts of data across multiple servers (Here, we shall use Amazon EC2 instances), providing high availability. And, the total number of replicas across the cluster is known as replication factor. It is used during startup to discover the cluster.

Cassandra nodes use this list of hosts to find each other and learn the topology of the ring. There are different types of snitches available. Hence, from the above tests, it is quite clear and is recommended to use 6 node cassandra cluster spread across three availability zones and with minimum replication factor of 3 (1 replica in all the 3 AZs) to make cassandra fault tolerant from one whole Availability Zone going down. In this blog, we shall replicate data across nodes running in multiple Availability Zones (AZs) to ensure reliability and fault tolerance. As we can see, the owns field above is nil as there are no keyspaces/databases created. Replication strategy indicates the nodes where replicas are placed. (Scenario wherein an entire AZ i.e; us-east-1a would go down)
Connection was made to the Cluster on remaining nodes in the other AZs (us-east-1b, us-east-1c) and records were read from the table user.
All records were intact.
Node 1 and Node 2 were started.
‘nodetool repair -h hostname_of_Node1 repair first’ was run on Node 1
‘nodetool repair -h hostname_of_Node2 repair first’ was run on Node 2
Connection was made to the Cluster on Node 1 and Node 2 and records were read from the table user.
All records were intact.
Similar tests were done by shutting down nodes in us-east-1b & us-east-1c AZs to check if the records were intact even when an entire Availability Zone goes down. We shall do so, by defining seeds key in the configuration file (Cassandra.yaml) .

On the whole AWS API Gateway is a beneficial package for the developer. These dedicated servers are explicitly set to handle the API calls for an application. Install the one specific to your Browser. It also has a useful testing platform which can be used to test the calls. It also provides caching and monitoring.

In my next blog I have talked about securing your API’s in API gateway, passing Query string & headers, transforming response type and returning the custom error code.The hands on is included. The increase in the number of API calls, increases the load of the API server which may require auto-scaling, which is cost-consuming.

1

The latest approach of the best architects is to utilize a new AWS service that explicitly replaces the need for a dedicated API Server. Finally a REST Client will be used to call the API.

Step 1: Create a DynamoDB table:
3
Create a DynamoDB table named Employee with the attributes of primary key as emp_id. Use code from the following location, https://s3-ap-southeast-1.amazonaws.com/cloudthatcode/addEmployee.zip. Use Lambda as the integration type and select the Lambda function you created earlier.

Test the API with the following values from the console.
{

“emp_id” : “2”

“emp_Name” : “Snow”

“emp_Salary” : “200000”

}

{

“emp_id” : “2”

“emp_Name” : “Snow”

“emp_Salary” : “200000”

}

You should see the entries in the DynamoDB table.

For creating an API we need to create Resource, create POST method under the resource and utilize the Lambda function as integration type and the Deployment of API is up for launch.

For Detailed Instructions..

Step 4: Call the API using REST Client

Test the API setup. The next part of the blog is a detailed tutorial on how to use AWS API Gateway along with AWS Lambda & DynamoDB.

People who are familiar with DynamoDB, API Gateway and Lambda can proceed with the high-level instructions. There are extensions for Mozilla firefox and Chrome which can be used which can be use to make REST calls. The bottleneck of this setup is that the API server has to be maintained to handle all the API calls. AWS API Gateway provides the ability to act as an interface between application and database which uses AWS Lambda function as the backend.

To get the essence of AWS API Gateway, we need to get hands-on with it. Invoke the API URL with the following values

{

“emp_id” : “3”

“emp_Name” : “Jack”

“emp_Salary” : “200000”

}

{

“emp_id” : “3”

“emp_Name” : “Jack”

“emp_Salary” : “200000”

}

Check the DynamoDB table again for the entries.

The response from the DynamoDB is processed by the Lambda function and routed via the API Gateway.

For Detailed Instructions..

With minimal effort a REST API has been created which accepts data, which in turn gets processed by Lambda function and finally stored in a DynamoDB table. Accept the rest of attributes as default and review the table details.

For Detailed Instructions..
Step 2: Create a Lambda Function:

Create a node.js Lambda function called addEmployee. Next a Lambda function which inserts the data into DynamoDB table. The API gateway provides nifty features where we can create different stages for development. Replace IAM access and secret keys of a user with access to write into the DynamoDB table.

For Detailed Instructions..

Step 3: Create an API in API Gateway :

Next create an API called Employee_API, a resource called employee and create a POST Method. If you have any queries on API gateway or you have trouble getting it setup, feel free to drop a question into the comment section below or you can also ask the questions to https://forum.cloudthat.com

Thank you. In the current world of API, every mobile application and website have to communicate using dedicated API servers. Also for people who are new to these services, there are detailed instructions which can be followed for step-by-step guidance.

The first step will be to create a DynamoDB table which stores the data. This API is ready for use and can be used in any application.

The API gateway frees the developer from maintaining infrastructure for APIs. API servers act as an intermediary between the application and the database. An API Gateway is setup to trigger the Lambda function. For this Install a REST client.