gscale-Global Scale: An automation tool for scaling your content or application over AWS CloudFront

Amazon Web Services AWS has one interesting and very beneficial service for the content delivery network called CloudFront. CDN helps you to scale your application worldwide, now by scaling, I mean the latency of the content will be very less, hence clients won’t see any downtime or delay website. In today's technological world the face or the value of the company or the business is purely based on the websites or how their product is available, authenticate, and scalable.

What is CDN and how it’s being used?

Source

The concept of CDN was introduced and accepted in the 1990s which focuses on reducing the bottleneck created on the internet because of the increase in the volume of traffic started flowing through the internet. In 1990–2000, due to the World Wide Web, opportunities arose in the e-commerce business and to globally scale their business. Services like Amazon and eBay were some of the most notable e-commerce websites to be released in this time period.

CDN is basically systems established all over the globe which are set up only to provide a distribution platform of the application in that area. Doing this, the people around that system will get no or less latency for the application which is allowed to remain on that system. This system is called an edge location server. It caches the application data and when a user hits the application, it will receive the interface from the nearest edge location. So, technically it's defined as the geographical distribution network of the proxy server and their data center for high availability and performance by distributing the service spatially relative to end-users.

Source

Due to this drastic and unpredicted pandemic, the use of the internet to scale the business and the new application coming up every day is hugely investing and using CDN. Even, the predictive analysis went wrong about the online business growth, use of the internet, and the volume of data that is passing through the CDN. The trend in the increment of e-commerce sales worldwide and the volume of data moving through CDN is shown below.

Source

Source

Where gscale comes into play?

This tool will help the industry to scale the application over the globe on top of Amazon Web Services AWS CloudFront(CDN service). This tool automated the use of CloudFront which is in beta version hence there ain’t much flexibility but works fine with Apache HTTPD web server. The primary focus of this tool is to disburden operations or developers to deploy and scale it on CDN. Usually, this takes a lot of time to configure and use. Using this tool you can interactively automate the deployment and CDN of your application.

We have used Amazon Web Services AWS particularly because it’s the global leading public cloud service provider and it’s having a huge network of more than 250+ edge locations for CDN and higher internet speed. This tool is built on top of python using basic libraries. Making this script, I learned a lot about how tools or language line terraform or ansible might work and core ideas about developing automation script for the DevOps world.

How to use gscale?

It’s built on top of python. We have used the command line interface provided by the Amazon Web Services AWS. Internally we have used the command available for a particular service to interact with the API of AWS and use it. Hence, you will require AWS CLI v2, python, and AWS account.

This tool is in its beta version, some more implementation will be made which will allow user for all the customization, flexibility, and more functions. I am making this open to all in my GitHub repo.

Here, I will show how you can build this same tool or how this tool was made. This will also help you to build such tools. So, let’s not waste time on the scripting stuff. I’ll show which command, I have used to make this script and how it’s working dynamic and automated using python.

There are some steps that we have to perform to deploy any application on CDN and even configuring the webserver.

Source: Shubham bhalala

Step 0: Creating a Security Group/Firewall

To create a security group or firewall in AWS webui we have to follow these steps: go to ec2 →go to Security Group →click on create →Add the necessary rules

Now, we are focusing on creating a script which does this thing automated. Whenever we do anything using the webui, behind the scene it’s calling the APIs of AWS. So, when we are using the AWS CLI V2 we have to write the command which will also called the same API. We as teh developer if this script will make this even more user friendly and automated by abstracting this by one more layer of python code, using the subprocess or os module.

aws ec2 create-security-group --description "gscale-security-group" --group-name "gscale"

This will create a security group with name gscale and it’s description will be gscale-security-group. Description is nothing but just some information about the security group we are creating, if you don’t keep it, its completely fine.

Step 1: Adding ingress rules to the firewall created

In webui it’s simple to add the rules which, I stated in Step 0 above. To do it using CLI here is the command.

aws ec2 authorize-security-group-ingress --group-name gscale --protocol tcp --port 22 --cidr 0.0.0.0/0

Here group-name is same as the command in Step 0, it represent the name of the firewall/Security group we created, it tell that the following rule needs to be added to this security group. protocol referns to the service allowed through the firewall. For example we want ssh to be enabled, the protocol used is tcp. port refers to the port number on which this service is listening or running, for ssh it’s 22. cidr is a notation system, which depicts the range of IP address. Here, it means which all IP or system are allowed through, 0.0.0.0/0 means all.

Step 2: Creating a EC2 instance

aws ec2 run-instances --image-id ami-052c08d70def0ac62 --count 1 --instance-type t2.micro --key-name arthkey --security-group-id sg-0f7c9e78ad9cf23a9

image-id refers to which image we want our OS to be, whether redhat, ubuntu, amazon linux etc. count refers to the number of instance you want to launch. instance-type refers to the computing system we want in our instance, by computing it means, CPU, RAM, Hard disk. key-name refers to the authentication key for login purpose. secuirty-group-id refers to the firewall we want on out instance. We can get the secuirty-group-id from the output of Step 0.

Step 3: Creating a EBS volume and attaching it for permanent storage

3.1 Creating EBS Volume

aws ec2 create-volume --volume-type gp2 --size 1 --availability-zone ap-south-1a

volume-type refers to the size and type of hard disk we want to create. size refers to the amount of storage we want. availability-zone refers to the zone in which we want to create this volume, here, we have to be careful about where the instance is been hosted or else the volume won’t be able to connect. We can get this from the output of Step 2.

3.2 Attaching the EBS Volume

aws ec2 attach-volume --volume-id vol-0e708aab6da539244 --instance-id i-0ceb50862eff63a2c --device /dev/sdf

volume-id refers to which volume we want to attach. We will get this from the output of Step 3.1 . instance-id refers to which instance we want to attach this volume. We will get this from the output of Step 2. device refers to the device name we want the volume to be attached to the instance as.

Step 4: Configuring the instance

In this step we have to do multiple things, like installing webserver, configuring it. Then formatting and mounting the device we attached. We will use the ssh command to perform the specific task. To do ssh we need the public IP of the instance which we will get from the output of the Step 1.

1. Listing the devices
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo fdisk -l"
2. Formatting device
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo mkfs.ext4 /dev/xvdf"
3. Installing HTTPD software
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo yum install httpd -y"
4. Mounting the formated drive
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo mount /dev/xvdf /var/www/html"
5. Checking the status of httpd
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo systemctl status httpd"
6. Starting the httpd service
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo systemctl start httpd"
7. Permenantly starting the service
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo systemctl enable httpd"
8. Again checking the status
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo systemctl status httpd"
9. Listing the active partition mounted
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo df -h"
10. Installing git
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo yum install git -y"

Step 5: Creating S3 bucket and adding objects to it

5.1 Creating S3 bucket

aws s3api create-bucket --bucket shubhambhalala --acl public-read-write --region us-east-1

bucket refers to the name of the S3 bucket we want to create, it need to be unique in the whole world. acl refers to the access control list, we have made it public because we have to use it in the website which need to be public for all the clients. region refers to the zone where we want to keep the bucket. AWS CLI V2 doesn’t support ap-south-1 for S3 bucket hence we have used us-east-1.

5.2 Uploading objects into the bucket

aws s3api put-object --acl public-read-write --bucket shubhambhalala --key /image/shubham.png --body F:\shubham.png

Here, only key and body is new parameter or fields we need to understand. key refers to the path we want to save in the S3 bucket and body refers to the object we want to put in the key location.

Step 6: Creating CloudFront Distribution

aws cloudfront create-distribution --origin-domain-name shubhambhalala.s3.amazonaws.com --default-root-object //image/shubham.png

This is the crucial and main command to implement the CDN through edge locations of Amazon. origin-domain-name refers to the place from where we have to pick the data to distribute over the edge locations of amazon. default-root-object refers to the object in S3 bucket we want to create distribution of. This will give an domain name as output for the object which we have to use in the website. Then we have to run the following final step.

Step 7: Pulling git code and hosting it

1. Cloning git repo for website code
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo git clone https://github.com/AnonMrNone/gscale.git"
2. Copying files to document root
ssh -i arthkey.pem ec2-user@65.0.11.243  "sudo cp gscale/index.html /var/www/html/"

This will get the code from the github and copy the files in /var/www/html.

Now, we have to finally visit the IP of the instance followed by the page we want to access.

Thank you for reading this article. I would suggest to go through the video demo and give suggestion on this.

GitHub: https://github.com/AnonMrNone/gscale

LinkedIn: https://www.linkedin.com/in/shubham-bhalala-a5062916b/