Terraform is the most popular Infrastructure as Code (IAC) tool provided by Hashicorp. With the advent of cloud and self-serviced infrastructure, a groundbreaking concept called Infrastructure as Code (IaC) emerged.
The reason for this is simple. Before IaC, teams used to build and configure infrastructure manually. Slowly and gradually, the infrastructure grew in size and complexity, leading to inconsistency, which led to multiple issues such as cost, scalability, and availability.
Infrastructure built manually needs to be maintained manually. If you need to create a similar infrastructure again, such as for scaling and availability, you would need a team of experts to do it.
That was not straightforward in large organizations due to the complexity of their infrastructure and numerous teams involved in the process of managing it. Migration projects took months because there was no single source of truth of infrastructure and configuration.
People used to rely on huge documentations which usually did not reflect the actual state of things. Scaling infrastructure was a problem because of this very reason.
IaC solves all these problems as now you have all your infrastructure defined as code which makes its way to a version control system such as Git. You can then review the code through a pull request, and can maintain a single source of truth because you spin infrastructure from code.
Terraform is a declarative IaC tool which means that you just need to declare what you need as configuration and Terraform will ensure that the infrastructure maintains the desired state.
That provides organizations with an ability to define reusable modules and apply policies and best practices through templates. The advantages of this are multifold.
Terraform makes it much easier to work upon as developers no longer need to learn another programming language or write complex code to spin infrastructure.
It promotes reuse, and the developers who do not have much experience with infrastructure can use modules developed by infrastructure experts and just pass variables into the module.
As a result, Terraform speeds up the whole infrastructure provisioning process, ensures consistency in defining it, brings more accountability as developers check-in code to VCS and maintains the history.
You can also create an approval workflow if you want experts to review the infrastructure code and ensure that it complies with organization policies such as budget and security.
You can do smart things like hooking in GitOps and role-based access control to ensure that people only have the accesses that they need.
That is an excellent enabler for organizations embracing a more DevOps-friendly approach and ways of working. That increases the overall efficiency of the team as now teams can decide on their own what infrastructure they require, code it, deploy it with transparency, and make changes to it based on their learning and feedback.
You no longer need to overprovision infrastructure as you can always make controlled changes later on, and therefore, it is an excellent enabler of agile software development within an organization.
How Does This Work?
Hashicorp proposes the following model for managing infrastructure.
- Infrastructure operators are responsible for creating reusable modules and store them in module registries.
- Policy owners are responsible for writing a Sentinel policy as code which ensures that the Terraform configurations meet organization guidelines such as security, budget, etc.
- Application developers then reuse the modules from the module registries to choose what they need and customize them as per their requirements through variables.
With Terraform, you can run a continuous integration, continuous delivery workflow for infrastructure. Below is a typical workflow:
- A DevOps engineer writes infrastructure code and checks into a VCS repository such as GitHub.
- The VCS repository then triggers a post-commit hook to a CI/CD tool such as Jenkins.
- Jenkins then runs a Terraform plan. The plan contains details on what components of infra it will create, modify, and delete if it were to apply the IaC.
- Jenkins then posts the Terraform plan to Slack so that approvers can review the plan and approve it if it follows all organization policies and does what it intends to do.
- The approver then approves the plan on Slack and Slack triggers the “Apply” job on Jenkins.
- Jenkins then runs “Terraform apply” to apply the changes to the infrastructure.
You need to configure Slack to receive plans and notifications from Jenkins.
In your Slack workspace, create a new channel called terraform-approval.
Install the Jenkins CI app on Slack
Go to [your-workspace].slack.com/apps and search for the Jenkins CI app.
Click on Add to Slack.
Select the terraform-approval channel and “Add Jenkins CI integration.”
On the next page, you would find an Integration Token Credential Id, keep that handy.
Create a file upload app on Slack
You would now need to create a custom app so that Jenkins can use the app to upload the plan file to Slack.
Go to api.slack.com and click on Start Building.
Go to OAuth & Permissions.
Scroll down and add a file:write bot token OAuth scope to the app.
Click on Install App to Workspace and allow the app to perform actions in channels and conversations.
Copy the Bot User OAuth Access Token generated on the next page and keep it handy.
Go to your Slack channel and add the terraform_plan_upload user to your channel.
Click on “Invite to Channel.”
Setting Up Your Cloud Environment
You would now need to configure the cloud environment to allow Terraform to manage infrastructure remotely. I have taken the Google Cloud platform as an example for this story. You would need to create a service account.
Go to IAM and Admin -> Service Accounts.
Create a service account called Terraform.
Grant the project editor role to the service account so that it can spin resources.
Grant the service account “service account user” access and generate a JSON key for Terraform to authenticate with Google Cloud APIs as the service account.
Installing Terraform
Terraform CLI is a binary file and it is simple to set up and use. Go to Terraform’s website and download the appropriate package based on your environment.
Unzip the package and then either set the path pointing to the Terraform binary or move the Terraform binary to your bin directory.
Spin Up Jenkins
Run the following for Terraform to spin up Jenkins.
git clone https://github.com/bharatmicrosystems/terraform-ci-cd.git
cd terraform-ci-cd/
cp terraform.tfvars.example terraform.tfvars
Copy the JSON key you’ve downloaded to terraform-ci-cd/ and rename it to credentials.json.
Use Terraform to spin up Jenkins on Google Cloud.
terraform init
terraform plan
terraform apply
Set Up Jenkins
Access Jenkins on http://External_IP:8080.
SSH into your jenkins-master instance and print the initialAdminPassword.
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Create an admin user account and click on “Save and Continue.”
Install required plugins
You would now need to install the Slack Upload and Slack Notification plugins so that Jenkins can upload plans and send notifications to Slack whenever you run a plan.
Go to the plugin manager ([your_jenkins_url]/pluginManager/available) and install Slack Upload and Slack Notification plugins.
Install the “Build Authorization Token Root” plugin to allow builds to be triggered remotely.
After the plugins are installed, configure the Jenkins Slack notification plugin settings.
Go to Manage Jenkins -> Configure System -> Slack Notification.
Create a secret text credential and paste the Integration Token Credential Id you obtained from the Jenkins CI app on Slack in the secret field.
Add your workspace and channel name, test connection, and save.
Install Terraform on Jenkins
Since Jenkins would run Terraform, we would need to install it there.
su — jenkins
wget https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
yum install -y unzip
unzip terraform_0.12.24_linux_amd64.zip
mv terraform /usr/bin/
chmod +x /usr/bin/terraform
There are a few things that cannot be checked-in into VCS, such as credentials and secrets, project details, etc. For that, Terraform provides a way to store that centrally using *.auto.tfvars files.
Copy the credentails.json file to /var/lib/jenkins/.
$ vim terraform.auto.tfvars
project = "<your_project_name>"
region = "<your_default_region>"
credentials = "/var/lib/jenkins/credentials.json"
Create a freestyle job called terraform-gce-instance with the following configuration:
Select Git as Source Code Management. You can fork this repo as an example of the source code repository.
Generate a random authentication token that you would use for triggering the build add in the below configuration. Use this link to generate one.
Add an Execute Shell build step and add the following script:
cp -a /var/lib/jenkins/terraform.auto.tfvars .
terraform init
if [ $action = ‘plan’ ]; then
terraform plan > plan.out
elif [ $action = ‘apply’ ]; then
terraform apply — auto-approve
elif [ $action = ‘destroy’ ]; then
terraform destroy — auto-approve
fi
Copy the file upload app’s Bot token to add token.
Click on Advanced and add a custom message, and include the approval link ([your_jenkins_url]/buildByToken/buildWithParameters?job=$JOB_NAME&token=[YOUR_JENKINS_BUILD_TOKEN]&action=apply) which will trigger the job with apply action when it is clicked.
Apply and save.
Setup post-commit hooks in your repository
In this example, I would be using GitHub to send a post-commit hook to Jenkins. You can use an equivalent webhook from your repository to Jenkins.
Go to https://github.com/<your_repo>/settings/hooks
Add webhook.
Testing the Configuration
Push configuration to GitHub and that should trigger the Jenkins job which would run the plan.
Jenkins will then update Slack with the plan’s output and an approval link.
Click on the link and see if it triggers the job on Jenkins with terraform apply.
Verifying Results
Go to Google Cloud Console -> Compute Engine, and you should see a new instance, test-instance, created.
Conclusion
Thank you for reading. I hope you enjoyed the article!