What does DevOps mean to me…
DevOps, the word itself says a lot. To me DevOps defines processes which defy the separation of two worlds – developer and operations. It clearly states that, as a developer you own how and where your code should run. This means that being a developer, your responsibilities are not limited to only development but also extended to implement deployment strategies for your code. You should be aware where and how to deploy the code instead of having any dependencies and the same goes for an operation guy who should also be aware of the purpose of the code which need to be deployed. It is highly important to accept this fact and responsibility since being a developer, I am fully aware of the compatibility, complexities, dependencies and platform where my code can run. DevOps concept and methodologies, emphasize on bridging the gaps between development team and operations team and have one single team capable of full SDLC process, implementing Agile methodologies and providing a consistent robust delivery model for clients. Well, to get more in-depth concepts on DevOps, I would recommend you to refer some of the wonderful training materials and videos that are available online. I found this video which gives some insights about DevOps and I think it is good.
This post of mine emphasize on exploring the capabilities of new Azure DevOps project and how it’s in-build CI/CD feature works building a robust release pipeline. Continuing my .NET Core migration series of post, what can be better than using the capabilities of Azure DevOps project. Just to let you know, following are the stages of this migration journey that I have completed so far.
1. Developing a sample ASP.NET Web API application using .NET Framework 4.6.1. (click here)
2. Converting ASP.NET Web API application to ASP.NET Core Web API (click here)
3. Introduction to Azure Cosmos DB as our data storage (click here)
4. Provisioning Jenkins server from Azure marketplace (click here)
5. Introduction to Docker and enabling Containerization capabilities to our .NET Core API solution (click here)
6. Introduction to Azure Container Registry (ACR) and Azure Kubernetes Service (AKS) in Azure (click here)
Our Agenda For using Azure DevOps
We are going to use Azure DevOps Project in order to accomplish the build and release pipeline for our sample .NET Core application which also include containerization aspect to it. However, since the contents of this whole implementation might be too large to accommodate in a single post, and hence we are going to segregate it into multiple posts.
In this post we are going to go over a sample implementation of Azure DevOps project from the portal for a new project and know how exactly the DevOps project plays a vital role.
Our second post would be to use Azure DevOps Service management tool to create the build and release pipeline for our existing .NET Core project which should finally build docker images, store it to ACR and then deploy the container to AKS.
In our third and final post we are going to see how integration of Jenkins CI/CD tool with Azure DevOps Service works to build and publish image container to AKS
Azure DevOps And its flavors
Azure DevOps has two flavor which contribute towards the same goal of building a robust Release pipeline supporting Continuous Integration and Delivery processes.
Azure DevOps Service, a management tool that facilitates a distinct ecosystem incorporating services that support Build and Release pipeline, associate a platform to incorporate Agile principles using Scrum and Kanban methodologies, integrate Test Automation capabilities on top of providing a distinct platform of CI/CD processes and delivery pipeline. Just to let your know Visual Studio Team Services (VSTS) is also referred as as Azure DevOps.
Azure DevOps Project is part of the Azure DevOps Management Service, which is integrated with portal providing a unique platform to setup CI/CD pipeline in Azure within few minutes.
How to get started
Well there are two ways you can get started with Azure DevOps project. You can directly use Azure DevOps project template from the portal which will help you to create the whole CI/CD process in few minutes or create through Azure DevOps Management Service from https://dev.azure.com. We will use Azure DevOps project template from the portal and then validate the configurations, build definitions and release pipeline using Azure DevOps Management service. We are going to trigger the build from the management service portal.
Let’s get started….
Create a sample project using Azure DevOps Project Template from Azure portal
Login to Azure portal https://portal.azure.com and search for Azure DevOps Project from the marketplace.
Once you have selected the project template, click Create from the template detailed screen.
In the application creation screen, you can either create a new application having basic features using some key technologies like .NET, Node.js, PHP, Java, Python, etc., or select an existing application where you need apply DevOps processes.
I am going to take the option of creating an new .NET application and then click Next. The next screen will give me options to select the runtime that I would like to go for like ASP.NET or ASP.NET Core. It will also let me add a SQL Server database in Azure, if I choose to go for it. I am going to go for ASP.NET Core and leave the database option as it is since I don’t need it here.
In my next screen, I will be provided with options to select the Azure service for deploying my application. The services that we can use are like Kubernetes Service, Service Fabric, Widows Web App, Linux Web App, Web App for Containers and Virtual machine. I am going to go for Kubernetes Service, since I have already planned and configured our .NET Core application for Containerization aspects. For the sample project, we don’t need Kubernetes service but I want to see how the configuration of Kubernetes service is applied in the project.
Next screen will give you the fields to define the Project name, Azure DevOps Service Organization name, Azure Subscription id, Cluster name which will be automatically populated based on the Project name and Location.
In the same creation screen, you will find Additional Settings which on selection opens another window where you can configure the Kubernetes Service and Container Registry.
In Kubernetes Service settings, you can create a new azure devops services organization if you don’t have one, give a resource group name, provide the number of nodes you need for the cluster, select VM size and provide location for Log Analytics.
In Container Registry Settings, you can provide the registry name which will get created, whether it is going to be a basic, standard or premium Registry and the location of the registry.
Once the process is complete, you can see that project has been created in the dashboard. Click Go to resource.
In the project dashboard, you will be able to view the CI/CD pipeline generated.
Till here everything looks good. However, I feel if we can get some insights on what is happening behind the scene of DevOps, that will be great.
Behind the scene of Azure DevOps project
If you look closer there are various resources and components involved in building DevOps project that we can emphasize on. I would like to know how each of these azure resources and services plays their key role for the execution of the DevOps project. And then I would also like to know various steps and processes involved in executing the Build and Release pipeline associated with the DevOps project.
From the above infographic, we can review various resources and components that plays a vital role in this project. Post creation of the project from Azure portal, there are three resource groups that got created –
1. Resource Group for DevOps project itself
2. Resource Group containing the collection of Kubernetes Service, Log Analytics, Application Insights and Container Registry
3. Resource Group containing the compute resources like AKS Node, NIC (Network Interface), DNS zone, Kubernetes Load balancer and NSG (Network Security Group)
Let us start identifying the resource group containing Compute services.
As we have selected only 1 node for our Kubernetes cluster, we have one AKS agent (VM) which is under the Agent pool availability set.
The agent is responsible to process the execution of containers using Kubelet which runs the container runtime (Docker) to connect with resources like VNET, storage, etc.
The network interface connects the agent to virtual network (VNET) by kube-proxy available in each node.
The load balancer provide an external IP address to let traffic connect with the application.
AKS DNS zone helps us to access application in easy mode.
NSG (Network Security Groups) are created automatically when the load balancer is created which filter traffic for agents or nodes.
The container resource group have four major resources deployed as part of the DevOps project –
1. Container Registry (ACR) which store images for container deployments
2. Kubernetes Service (AKS) which manage the hosting environment for Kubernetes in order to deploy and manage containerized applications. Each of these containers are executed by a container runtime like Docker
3. Application Insights which monitors the health of Kubernetes Service
4. Log Analytics which collect data and telemetry from Application Insights
Explore DevOps project features and components
While we went through the various components and resources deployed by Azure DevOps project, it is a good time to know the features of the project. Let us switch to https://dev.azure.com and have a quick look into the DevOps project created.
In the left navigation menu, the most important menu items that we should be considering are Boards, Repos, Pipelines, Test Plans and Artifacts.
1. Boards will be used to track work, Kanban boards, backlogs and sprint planning.
2. Repos will basically connect to Git associated with your azure account which will manage the source codes.
3. Pipelines are the most important aspects of DevOps project as it contains definitions and integration of CI/CD pipeline. This is the only one which we will cover in this article.
4. Test Plan plays a significant role if you are going for Test Automation
5. Artifacts are deployable components for your application through Azure pipeline.
Azure CI/CD Pipeline
Azure pipeline helps to continuously build, test and deploy application to cloud. It has various components –
3. Library (collection of shared build and releases assets for the project)
4. Task Groups (groups which share common actions with multiple build and releases)
5. Deployment Groups (collection of machines each having an azure pipeline agent used to host app).
Azure DevOps Build Pipeline
1. The Build pipeline starts with providing the Build Name and selection of Agent Pool (e.g. Hosted Ubuntu 1604, Hosted VS 2017, etc.).
2. Select the Source Control (e.g. Azure Repos Git, GitHub, BitBucket, etc.), name of the Team Project, Repository selection, Branch selection for manual and scheduled builds and other non-required parameters.
3. Create the Agent job which will have multiple steps or tasks to be completed. The Agent job will inherit from the Agent pool which execute an agent during build process. You can also have multiple agent running same set of tasks by selecting the option of Parallelism.
Just to let you know, that the steps here added for the job is very specific to build of docker images and working with ACR. Hence this might not be the appropriate steps in your case and you can choose any steps that would fit your build strategy.
a. Azure Container Registry deployment step requires information of Azure subscription, creation of ACR, creation of resource group, template mapped to Azure RM template, etc.
b. Building of docker images step which will use the ACR information and Docker file path to build the images.
c. Pushing of docker images step involves of placing the images in ACR.
d. Installation of Helm Tool step involves installing Helm and Kubernetes in Agent machine. In case you are new to Helm, it is a package manager for Kubernetes which helps to deploy apps to Kubernetes. The packaging format of Helm are called Charts, which contains collection of files that provides Kubernetes resources. In our next step we are going package and deploy those Charts.
You might be wondering, how do I get the chart path or how can we create it. Well the charts are available in the project folder and specific to individual project of a solution. However, I am not going to go to the depth of the charts since I will be covering that in my next articles. Here is the project structure containing chart.
e. With ARM templates copy files from the source folder to the destination folder
f. Publish the build artifacts to Azure pipeline
Once the build has been a success, you can view the build status
Azure DevOps Release Pipeline
The release pipeline will contain the execution steps in order to release the application and deploy it to a specific location or resources which in this case is Kubernetes cluster.
You can add as many stages you want like DEV, STAGING and PRODUCTION
Dropping of the artifacts will trigger the CI build
Next phase is how the agent is going to execute the steps to create AKS cluster, extract the application routing zone providing the deployment outputs followed by installing Helm and then packaging and deploying the Helm charts.
In the AKS Cluster creation step, if you have a template for AKS you can specify it along with template parameters. Make sure to make OMS Location to East US or a location where the cluster is applicable. South Central US is not available.
Ensure that you have selected the right Azure subscription endpoint, resource group and location for the cluster creation.
While configuring the Application Routing Zone, provide the script that will be executed in this step.
In Helm package and deployment of charts, ensure you have the right values for your subscription, resource group and AKS Cluster name
Once the configuration is complete, you can create a release.
Error from the Release
After the successful CI build, I did notice that the deployment to Kubernetes has failed.
On clicking the Release-1 link, it redirects me to the Log generated from the release pipeline in azure devops management service portal. The error says that “The VM size of AgentPoolProfile:agentpool is not allowed in your subscription in location ‘centralus’.” . This means that the location centralus that has been selected for Kubernetes service is not available for my subscription.
In order to fix this issue, I have changed the OMS location in the template parameters to East US and made it worked. Initially it was pointing to South Central US.
This post is just an introduction to how we can build CI/CD pipeline using Azure DevOps project which is not covering the actual deployment of our .NET core project. In my next post, instead of using Azure DevOps project which has a limitation of not using existing ACR and AKS, I am going to create the CI/CD pipeline from the scratch. Also, I will modify our existing solution to add Helm Charts to manage Kubernetes. Along with that, we will also see how we can integrate Jenkins build with DevOps release pipeline to have a complete CI/CD process in place for our existing .NET core application.