Implementing DevOps Practices: Crafting a Powerful CI/CD Pipeline from Scratch

Implementing DevOps Practices: Crafting a Powerful CI/CD Pipeline from Scratch

From Code Commit to Production: A Complete DevOps Roadmap


DevOps is like a team sport for making software better and faster. It's all about teamwork, using cool tools, and working together to make awesome apps and games quickly and with fewer mistakes. In short, DevOps is like teamwork between software developers (who write code) and operations teams (who manage infrastructure). It's a way of working that helps teams build and deliver software more quickly and smoothly. Before starting this project it will be better for us to understand a little some about related terminology and their history.

Before DevOps: ( The Evolution of Application Development )

In the past, developing and delivering software was slow. Teams worked separately, causing delays and communication issues. Everything was done manually, which led to mistakes. Improvements were rare because teams didn't collaborate much, and feedback took a long time to make any changes. This process followed a predefined approach known as SDLC or Software Development Life Cycle.

What is SDLC? SDLC, or Software Development Life Cycle, is like a roadmap or plan that guides how software is created. It's a step-by-step process that helps teams design, develop, test, and release software in an organized and efficient way

DevOps is also a part of the Software Development Life Cycle (SDLC). It focuses on optimizing and improving the processes within the SDLC, especially in terms of collaboration, automation, and continuous delivery. DevOps practices are integrated into different stages of the SDLC to enhance efficiency, quality, and speed of software development, ensuring a more streamlined and effective development process overall.

Before DevOps, several software development models were commonly used within the broader Software Development Life Cycle (SDLC).:

  1. Waterfall Model: A linear and sequential approach where each phase (requirements, design, implementation, testing, deployment) is completed before moving to the next. Progression flows in one direction, making it challenging to accommodate changes once a phase is finished.

    Once the water starts flowing over the edge of the cliff, it starts falling down the mountain and the water cannot go back up.

    Waterfall Model

  2. Spiral Model: An iterative model combining elements of both waterfall and prototyping models. The Spiral Model is a crucial part of the Software Development Life Cycle (SDLC), focusing on managing risks. It resembles a spiral with undefined loops, each loop being a development phase. It involves a series of cycles, each encompassing risk analysis, planning, engineering, and evaluation. It allows for flexibility in accommodating changes during the development process.

    Spiral Model

  3. Agile Model: Agile methodologies, such as Scrum or Kanban, emphasize iterative and incremental development. They focus on adaptability, collaboration, and responding to change throughout the development process, allowing for quicker releases and customer feedback incorporation.

    In more simple definition the Agile Model is like building something piece by piece, quickly and flexibly. It focuses on teamwork, adapting to changes, and delivering small parts of the project often. Unlike the Waterfall or Spiral Models, Agile doesn't wait until the end to show progress; it's all about quick adjustments and regular updates.

  4. V-Model (Verification and Validation Model): Similar to the waterfall model but with a focus on validation and verification at each stage. It pairs each development stage with a testing phase, emphasizing testing and validation alongside development activities.

  5. Rapid Application Development (RAD): Emphasizes rapid prototyping and iterative development. It involves close interaction between developers and end-users to quickly build and refine software based on continuous feedback.

How to choose : ( Different ways to make software )

  1. Step-by-Step Recipe (Waterfall): Good when you know exactly what you want, like following steps in order.

  2. Adjustable Building Blocks (Agile): Like playing with Legos, building bit by bit and changing as needed.

  3. Risk Management Circles (Spiral): Handling risks in loops, especially for uncertain or risky projects.

  4. Prototypes (Prototype): Make a sample first, refining it based on feedback before building the whole thing.

  5. Little by Little (Incremental): Working on small parts, adding and improving gradually.

  6. Multiple Cycles (Iterative): Repeating steps to improve with each cycle.

  7. All-at-Once (Big Bang): Building everything together, best for smaller or vague projects.

Each way suits different projects depending on how clear the plan is and how much it might change.

DevOps Engineer: (The Role and Responsibilities)

A DevOps engineer is like a bridge builder between software development and IT operations. They work to make sure that the processes for building, testing, and delivering software are smooth and efficient. Their job involves automating tasks, managing tools, and improving collaboration among different teams to speed up software delivery while maintaining quality and reliability. They focus on making sure that the software development and deployment run smoothly and that everyone involved works together effectively.

They rely on a range of tools to accomplish these tasks, including Jenkins Continuous Integration/Continuous Deployment (CI/CD) tools, version control systems like Git, configuration management tools such as Ansible or Puppet, containerization tools like Docker or Kubernetes, monitoring and logging solutions like Prometheus or grafana, collaboration platforms like Slack or Jira, cloud services from providers such as AWS or Azure, as well as testing tools like Selenium or JUnit. This expansion includes a mention of various tools commonly used by DevOps engineers to support their responsibilities in software development and deployment.

Key Tools Utilized in this Project


  1. Linux: An open-source operating system used widely due to its stability, security, and flexibility. It's prevalent in server environments and for software development.

  2. Git: A version control system that tracks changes in code, enabling collaboration among developers, and managing codebase versions.

  3. GitHub: A platform built around Git, providing hosting for software development and collaboration using Git repositories.

  4. Docker: Allows the creation, deployment, and running of applications in containers, ensuring consistency across various environments.

  5. Kubernetes: An orchestration tool for managing containerized applications, automating deployment, scaling, and management.

  6. DockerHub: A cloud-based repository where Docker users store and share container images.

  7. Jenkins: An automation server used for Continuous Integration and Continuous Deployment (CI/CD) to automate the software development process.

  8. AWS (Amazon Web Services): A comprehensive cloud computing platform offering various services like computing power, storage, and databases.

  9. MobaXterm: A terminal tool with many features like remote connectivity, X11 server, and tabbed SSH client, enhancing remote working capabilities.

Key Tools Utilized in this Project

Progressing Methodically: ( Our Step-by-Step Approach in this Project )

Throughout this comprehensive DevOps project journey, we will progress through the following steps methodically, one by one.

  1. Set Up GitHub Repository: Create a GitHub account, establish a code repository, and push the code to it.

  2. Provision Jenkins Server: Set up and configure the Jenkins server for continuous integration and deployment.

  3. Deploy Kubernetes Cluster: Create and configure a Kubernetes cluster for application deployment and management.

  4. Build Jenkins CI/CD Pipeline: Develop a comprehensive CI/CD pipeline using Jenkins for seamless code integration and deployment.

  5. Deploy WP-MySQL Galera Multinode: Implement a robust Galera-based MySQL setup ensuring database security and failover, monitored using Grafana.

  6. Configure AWS Load Balancer: Set up an AWS Application Load Balancer to efficiently distribute traffic between application instances.

  7. Domain Name Configuration: Add a domain name and configure access to the application through it for user-friendly access.

  8. Code Verification: Modify the code and verify the changes to ensure proper functionality.

  9. Application Scaling: Scale the application up and down to handle variable loads efficiently.

  10. Rolling Updates & Rollbacks: Perform rolling updates for seamless application upgrades and have a rollback plan in case of issues.

  11. Backup and Restore Strategy: Implement a backup and restoration process to ensure data security and reliability.

Step-1: (Set Up GitHub Repository)

Go to the GitHub account login page and log in with your GitHub account authentication. Navigate to new and create a new repository. Assign a Repository name and Description (optional). Choose between the account options: public repositories, visible to the world, and private repositories, requiring payment and authentication for access.

Create a private GitHub repository that now exists for the free tier. Free private GitHub repositories have some restrictions placed upon them. One is that no more than three contributors can work on a private GitHub repository.

Check to Add a README file That helps write the description for your project. Nevigate to <> Code Local and copy this HTTPS code, actually this link is crucial for pushing new code to your GitHub account repository from git. In my case my repository name is grras and HTTPS link is

Created new repository in GitHub account.

Further, we will set webhook also for integration with Jenkins for auto build facility in a CI/CD pipeline.

Webhooks allow external services to be notified when certain events happen. When the specified events happen, we’ll send a POST request to each of the URLs you provide.

Webhook: Webhooks allow external services to be notified when certain events happen. When the specified events happen, we’ll send a POST request to each of the URLs you provide.

When you navigate to Add webhook option, you will see a page that needs some configuration to enter manually.

Step-2: (Provision Jenkins Server)

Either you can go threw this official Jenkins link to configure Jenkins on Linux or you can set up your own. For this, we have created a Rhel Linux instance and got SSH through MobaXterm on Windows localhost for further configuration and setup. I am using Rhel Linux instance with 4 GB RAM & 2 Core CPU configuration. So let's start.

I am using Azure Instance here.
# hostnamectl set-hostname jenkins
# exec bash
  • Open the 8080 port on the machine security firewall. You can see 8080 port is listened to by JAVA.
# firewall-cmd --permanent --add-port=8080/tcp
# firewall-cmd --reload
# firewall-cmd --list-all
# # netstat -tulpn | grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      3648/java
  • Import a Jenkins repository under /etc/yum.repos.d and key from the Jenkins server repository which helps to authenticate that Jenkins should be downloaded or installed from an authenticated source.
# wget -O /etc/yum.repos.d/jenkins.repo \
# rpm --import
  • Upgrade the system
# sudo yum upgrade
  • Install pre-required Jenkins dependencies packages and java sdk, which ismust for Jenkins. Because Jenkins is developed by Oracle and created in java.
# yum install fontconfig java-17-openjdk
  • Install Jenkins in the system
# yum install jenkins
  • Start and enable Jenkins service.

  •       # systemctl enable --now jenkins
          # systemctl status jenkins
          ● jenkins.service - Jenkins Continuous Integration Server
               Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; vendor preset: disabled)
               Active: active (running) since Tue 2023-12-12 12:07:18 IST; 35min ago
             Main PID: 3648 (java)
                Tasks: 39 (limit: 10756)
               Memory: 359.2M
                  CPU: 1min 25.363s
               CGroup: /system.slice/jenkins.service
                       └─3648 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
          Dec 12 12:07:20 jenkins jenkins[3648]: 2023-12-12 06:37:20.235+0000 [id=49]        INFO        hudson.util.Retrier#start: Performed the action check updates server>
NOTE: If you are working with Jenkins in a local VM, then set domain name entry in /etc/hosts file for accessing Jenkins through the domain name. Remember time by time we have to update host entries with other tools machines' IP shortly.
#  echo "" >> /etc/hosts
  • Now go to the web browser and paste exactly this.
Or if you working with cloud infrastructure, go with Jenkins server public IP address with an 8080 port. Remember & be sure to open the 8080 firewall port in the inbound rule.

  • Unlock Jenkins through Jenkins secret password, which is saved on the location Jenkins home directory "/var/lib/jenkins/secrets/initialAdminPassword" copy it and paste into the browser input bar.

  • Click on Default Install Suggested plugins option, and wait to complete the installation plugins packages.

# cat  /var/lib/jenkins/secrets/initialAdminPassword

  • Congratulations! Your Jenkins is ready to work.

Important interview questions related to Jenkins.

What is Jenkins? Give a brief introduction
Jenkins is an open-source DevOps tool primarily designed for continuous integration/continuous delivery (CI/CD) and deployment. Developed in Java, it serves as automation software used to execute CI/CD workflows commonly referred to as pipelines. Jenkins helps automate various stages of software development, from building and testing code to deploying applications. Jenkins was developed in Java language. and owned by sun microsystem
What does CI/CD mean?
CI/CD represents two distinct work cultures within Jenkins. 'CI' stands for Continuous Integration, while 'CD' encompasses both Continuous Delivery and Continuous Deployment. This definition might seem a bit tricky, but it's crucial to grasp the concepts of Continuous Integration, Continuous Delivery, and Deployment.
what is the actual difference between CI/CD?
Think of DevOps as a way of working that's a lot like the journey software goes through from creation to deployment, just with a strong focus on streamlining and automating tasks. It's like connecting a bunch of tools to a central system (like Jenkins) that work together smoothly, forming a clear path for the software to follow. When we talk about Continuous Integration, it means that whenever we make changes to the code, this special system automatically builds and tests it. So, it's like having a buddy who checks your work every time you make a change. When it's time to share our work with the world, we're really careful about it. Before sending our application out there, we run thorough checks in a special environment that's an exact copy of the real thing, called the staging environment. This helps us ensure that everything works just right before it goes live. That's Continuous Delivery in action. Once everything in the staging area looks good and behaves as expected, we press the button (or sometimes, the system does it for us!) to move our work into the actual environment where everyone can use it. This step is called Continuous Deployment, and it's like the grand finale of the whole process!
What is the Main difference between Build Periodically & Poll SCM in Jenkins?
Poll SCM periodically polls the SCM to check whether changes were made (i.e. new commits) and builds the project if new commits were pushed since the last build. whereas build periodically builds the project periodically even if nothing has changed.

NOTE: Jenkins Related Information
--------> jenkins user is an application user in os check UID:977
# cat /etc/passwd | grep "jenkins"
jenkins:x:977:977:Jenkins Automation Server:/var/lib/jenkins:/bin/false

---------> These following files come with this Jenkins package.
# rpm -ql jenkins

--------> Find all files related to user jenkins and list 
# find / -name jenkins -type d -exec ls -ld {} \;
# find / -name jenkins -type f -exec ls -al {} \;

-------> Check jenkins related files & data
# ls /var/lib/jenkins
config.xml                      jenkins.install.InstallUtil.installingPlugins   jenkins.telemetry.Correlator.xml  plugins                   updates
hudson.model.UpdateCenter.xml   jenkins.install.InstallUtil.lastExecVersion     jobs                              secret.key                userContent
hudson.plugins.git.GitTool.xml  jenkins.install.UpgradeWizard.state             nodeMonitors.xml                  secret.key.not-so-secret  users
identity.key.enc                jenkins.model.JenkinsLocationConfiguration.xml  nodes                             secrets

NOTE: If you want to change Jenkins's default configuration service then you can modify it with the below command. Follow this official link:

# PERM="--permanent"
# SERV="$PERM --service=jenkins"

# firewall-cmd $PERM --new-service=jenkins
# firewall-cmd $SERV --set-short="Jenkins ports"
# firewall-cmd $SERV --set-description="Jenkins port exceptions"
# firewall-cmd $SERV --add-port=$YOURPORT/tcp
# firewall-cmd $PERM --add-service=jenkins
# firewall-cmd --zone=public --add-service=http --permanent
# firewall-cmd --reload

Step-3: (Set Up Kubernetes Cluster Deployment Environment**)**

If you're operating within a Virtual Machine Environment, the rest of the process remains unchanged, with only pre-configuration adjustments required, Follow the article as outlined below
  • Firewall and SELinux should be disabled

  • swap memory should be removed and off

Result of Kubernetes two nodes cluster:

Lots of Congratulations...

Step-4: (Build Jenkins CI/CD Pipeline)

Your cluster is ready, your Jenkins server is running properly. you have created a GitHub repository account and you are ready to push the code so now we are ready to create a Jenkins pipeline for the application.

Step4.1: (Push Code on GitHub & add GitHub Jenkins server address on webhook)

  • A developer writes code using the Windows software application VS Code and pushes it using Git.

  • Open or edit code through VS CODE and push code through git.

push code on github

  • Add config global information to tell who is actually working and push the code on GitHub repository server.
$ git config --global ""
$ git config --global "Rakesh"

$ git config

$ git config
  • Create and move the all data at once using git checkout -b <new_branch>
$ git checkout -b stage
Switched to a new branch 'stage'
  • Add GitHub remote repository and assign with an alias name origin .
$ git status 
$ git remote add origin
  • Check whether remote repository are added or not
$ git remote -v
origin (fetch)
origin (push)
We are creating a password less public-key based GitHub authentication

$ ssh-keygen.exe
$ cat /c/Users/Rakesh-PC/.ssh/
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwOtiIb9Yctu2MJgnS9Avol93dTYGlgPPCaRxTCxz+xxlOb+sZJsz29+4GFkvcZZDbhsmjKRE4ByqlKXBIM+Jv1LlgL8i2hsNFUqdij6Y41kFzbSQIVINNDnQHBOxjMUjz7HRkqteLQQRHRF3roG61sv1kFWC/Xza0DmsHeE6TrRhl+Kj2OEDT7AZ3oQuTp2yGgVTrAk/wOdQnxjvK0eY3EewlVhbJStkCZ6Fac7q05oBh24eY6egf5othVNvUeRq0W5OureB7dyRvobB98rHcmwaLxnzOwZZVH/YHHIgM1PAxCfW3iam+EI9Sm0C++wBZA6/51zzjjekzZaf3Ir04kZe7qZXQoS9hfTDgaW0tanDZvGOl3GdUH9XxnRur5MCNmivcPv8IjlsIleoGM7vLDLXBMTgIEFAZ/++b2MBww59gGQZ6+oCfNgIXBaoGl3FWapbKKx9mbEI1kDxEs11qHvUxTw4kQInacEO6abxvJeI5XFXktvUGokKhBW1P8q8= Rakesh-PC@RakeshPC
  • We will paste this public key to the GitHub for public key-based authentication.

  • Go to and click on "New ssh key" and paste your public key here.

Your attached key should reflected like this output, otherwise check your public key and steps carefully.

Now open your git on windows, and type below command to check whether we are ready to work with GitHub passwordless.

  • Ensure SSH agent is running on your windows machine
$ eval "$(ssh-agent -s)"
Agent pid 1217
  • Add the private key to the SSH agent and check the connection with GitHub.
$  ssh-add ~/.ssh/id_rsa
$ ssh -T

  • Commit the codes with a message, and push them to stage branch on grras GitHub repository. (This time github will not ask the username and password)
$ git status
$ git commit -m "This is my first commit"
$ git push origin stage

Output should be

  • Check the data in GitHub repository stage branch .

  • Now add Jenkins to the GitHub webhook for trigger option.


Step-5: (Deploy WP-MySQL Galera Multinode)

Step-6: (Configure AWS Load Balancer)

Step-7: (Domain Name Configuration)

Step-8: (Code Verification)

Step-9: (Application Scaling)

Step-10: (Rolling Updates & Rollbacks)

Step-11: (Backup and Restore Strategy)