Building a Production-Grade DevOps Pipeline: From Infrastructure to GitOps

How I automated the deployment of 15+ microservices using Terraform, GitHub Actions, and ArgoCD

---

Introduction

Imagine you're a chef running a restaurant with 15 different kitchens, each specializing in a different cuisine. Now imagine trying to coordinate orders, ingredient deliveries, and quality control across all of them - manually. That's essentially what deploying modern microservices feels like without proper automation.

In this post, I'll walk you through how I built a complete DevOps pipeline for the OpenTelemetry Astronomy Shop - a polyglot e-commerce platform with 15+ microservices written in Go, Python, .NET, Node.js, Java, Rust, and C++.

Note: This project uses the OpenTelemetry Demo application - an open-source project created by the OpenTelemetry community specifically for learning and practicing observability and cloud-native technologies. Huge thanks to the maintainers for providing this excellent educational resource!

By the end, you'll understand:

• Infrastructure as Code (IaC) with Terraform

• CI/CD Pipelines with GitHub Actions

• GitOps with ArgoCD

• Observability with OpenTelemetry, Jaeger, Prometheus, and Grafana

Let's dive in.

---

The Architecture at a Glance

Before we get into the "how," let's understand the "what."

The Application Stack

The DevOps Stack

---

Part 1: Infrastructure as Code with Terraform

The Problem: "It Works on My Cluster"

If you've ever tried to recreate a Kubernetes cluster from memory, you know the pain. Security groups, node pools, networking - one wrong click in the cloud console, and you're debugging for hours.

The analogy: Think of Terraform as a detailed blueprint for a house. Instead of telling a contractor "build me something nice," you hand them exact specifications. Every wall, every outlet, every door - defined in code.

The Solution: Declarative Infrastructure

I chose DigitalOcean Kubernetes Service (DOKS) for its simplicity and cost-effectiveness, but the same principles apply to AWS EKS, GCP GKE, or Azure AKS.

Here's the core Terraform configuration:

# main.tf - DigitalOcean Kubernetes Cluster

resource "digitalocean_kubernetes_cluster" "otel_demo" {
  name    = "otel-demo-cluster"
  region  = "nyc3"
  version = "1.28.2-do.0"

  # Auto-scaling node pool - pay only for what you use
  node_pool {
    name       = "critical-pool"
    size       = "s-4vcpu-8gb"
    auto_scale = true
    min_nodes  = 2
    max_nodes  = 5

    labels = {
      environment = "production"
      managed-by  = "terraform"
    }
  }

  # Network isolation for security
  vpc_uuid = digitalocean_vpc.otel_vpc.id
}

# Private network for cluster communication
resource "digitalocean_vpc" "otel_vpc" {
  name     = "otel-demo-vpc"
  region   = "nyc3"
  ip_range = "10.10.10.0/24"
}

Key Design Decisions

1. Auto-scaling (2-5 nodes)

The cluster scales based on demand. During low traffic, we run 2 nodes. During peak load or deployments, it scales to 5. This saved approximately 40% on infrastructure costs compared to fixed sizing.

2. VPC Isolation

All cluster traffic stays within a private network. External access only through the load balancer.

3. Versioned Infrastructure

Every change goes through pull requests. Want to upgrade Kubernetes? Update the version string and terraform apply.

Deployment Commands

# Initialize Terraform (downloads providers)
terraform init

# Preview changes without applying
terraform plan

# Apply the infrastructure
terraform apply

# Destroy everything (careful!)
terraform destroy

DigitalOcean Kubernetes cluster running with 3/3 nodes in the critical-pool

The Impact

---

Part 2: CI/CD Pipelines with GitHub Actions

The Problem: "I'll Just Deploy It Manually"

With 15+ services written in 7 different languages, manual deployments became a nightmare:

• "Did someone run the tests before deploying?"

• "Which version is in production?"

• "Who built this image and when?"

The analogy: Imagine an assembly line where each worker builds their part differently. One uses metric, another uses imperial. Chaos ensues. CI/CD is like standardizing that assembly line - every product follows the same quality checks.

The Solution: Language-Agnostic Pipelines

I created 7 service-specific pipelines (with plans to add more) that follow a consistent pattern.

Pipeline Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│    BUILD    │────▶│    TEST     │────▶│   DOCKER    │
│  (compile)  │     │ (unit/lint) │     │  (image)    │
└─────────────┘     └─────────────┘     └─────────────┘
                                               │
                                               ▼
                                        ┌─────────────┐
                                        │   DEPLOY    │
                                        │ (manifest)  │
                                        └─────────────┘

Example: Cart Service Pipeline (.NET)

# .github/workflows/cart-ci.yaml
name: Cart Service CI/CD

on:
  push:
    branches: [main]
    paths:
      - 'src/cart/**'  # Only trigger when cart code changes
  pull_request:
    paths:
      - 'src/cart/**'

env:
  SERVICE_NAME: cart
  DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'

      - name: Restore dependencies
        run: dotnet restore src/cart/

      - name: Build
        run: dotnet build src/cart/ --no-restore

      - name: Run tests
        run: dotnet test src/cart/ --no-build --verbosity normal

  docker:
    needs: build  # Only runs if build succeeds
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      # Tag with GitHub Run ID for traceability
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: ./src/cart
          push: true
          tags: ${{ env.DOCKER_USERNAME }}/cart:${{ github.run_id }}

  deploy:
    needs: docker
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
        with:
          token: ${{ secrets.GH_PAT }}

      # Update the Kubernetes manifest with new image tag
      - name: Update manifest
        run: |
          sed -i "s|image: .*/cart:.*|image: ${{ env.DOCKER_USERNAME }}/cart:${{ github.run_id }}|g" \
            kubernetes/cart/deploy.yaml

      # Commit the change (triggers ArgoCD sync)
      - name: Commit and push
        run: |
          git config user.name "cloudenochcsis"
          git config user.email "cloudenochcsis@gmail.com"
          git add kubernetes/cart/deploy.yaml
          git commit -m "[CI]: Update cart image tag to ${{ github.run_id }}"
          git push

Multi-Language Support

Each service has its own pipeline tailored to its language:

Path-Based Triggers: Build Only What Changed

Notice this key section in the workflow:

on:
  push:
    paths:
      - 'src/cart/**'

This means the Cart pipeline only runs when Cart code changes. Push a frontend change? Cart pipeline doesn't run. This optimization reduced our CI minutes by approximately 70%.

Image Tagging Strategy

Instead of latest (which is considered harmful), I use GitHub Run IDs:

cloudenochcsis/cart:21115741490
cloudenochcsis/frontend:21289354234
cloudenochcsis/checkout:21120806887

Benefits:

• Traceability: Every image links back to a specific CI run

• Immutability: Tags never get overwritten

• Rollback: Easy to revert to any previous version

Pull request showing automated CI/CD checks running for multiple services

The Impact

---

Part 3: GitOps with ArgoCD

The Problem: "kubectl apply" is Not a Strategy

Even with CI/CD building and testing our code, deployments still required manual intervention:

# The old way (don't do this)
kubectl apply -f deployment.yaml
kubectl rollout status deployment/cart
# Fingers crossed...

The analogy: GitOps is like having a vigilant security guard who constantly compares what's in your building against the blueprint. If someone moves a wall, the guard moves it back automatically.

The Solution: Declarative, Git-Driven Deployments

ArgoCD watches our Git repository and automatically synchronizes the cluster state with whatever's in the kubernetes/ directory.

ArgoCD login screen - "Let's get stuff deployed!"

ArgoCD showing successful sync status for the opentelemetry-demo application

ArgoCD Application Configuration

# argo-cd/application-local.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: opentelemetry-demo
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/cloudenochcsis/opentelemetry-devops-project.git
    targetRevision: main
    path: kubernetes/
    directory:
      recurse: true  # Include all subdirectories

  destination:
    server: https://kubernetes.default.svc
    namespace: otel-demo

  syncPolicy:
    automated:
      prune: true      # Remove resources deleted from Git
      selfHeal: true   # Auto-recover from manual changes
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

The GitOps Workflow

Here's how a code change flows from commit to production:

┌──────────────────────────────────────────────────────────────┐
│                    THE GITOPS WORKFLOW                       │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│   1. Developer pushes code to main                           │
│              │                                               │
│              ▼                                               │
│   2. GitHub Actions runs tests & builds image                │
│              │                                               │
│              ▼                                               │
│   3. Pipeline updates kubernetes/[service]/deploy.yaml       │
│      with new image tag                                      │
│              │                                               │
│              ▼                                               │
│   4. Pipeline commits & pushes manifest change               │
│              │                                               │
│              ▼                                               │
│   5. ArgoCD detects Git change (polling every 3 min)         │
│              │                                               │
│              ▼                                               │
│   6. ArgoCD syncs manifests to cluster                       │
│              │                                               │
│              ▼                                               │
│   7. Kubernetes performs rolling update                      │
│              │                                               │
│              ▼                                               │
│   8. New version is live! 🚀                                 │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Deploying all services to the cluster using kubectl apply

K9s terminal UI showing all pods running in the otel-demo namespace

ArgoCD namespace showing all control plane components running

Self-Healing in Action

The selfHeal: true setting is where the magic happens. Watch what happens when someone makes an unauthorized change:

# Someone manually scales down the cart service
kubectl scale deployment/opentelemetry-demo-cartservice --replicas=0

# ArgoCD detects the drift within seconds
# ArgoCD automatically scales it back to the desired state (1 replica)
# No human intervention required

Kubernetes Manifest Structure

kubernetes/
├── cart/
│   ├── deploy.yaml        # Deployment + ConfigMap
│   └── svc.yaml           # Service definition
├── checkout/
│   ├── deploy.yaml
│   └── svc.yaml
├── frontend/
│   ├── deploy.yaml
│   └── svc.yaml
├── ... (12 more services)
│
├── # Observability Stack
├── otelcol/               # OpenTelemetry Collector
├── jaeger/                # Distributed tracing
├── prometheus/            # Metrics
├── grafana/               # Dashboards
│
├── # Infrastructure
├── kafka/                 # Message queue
├── valkey/                # Cache (Redis-compatible)
├── postgres/              # Database
├── ingress/               # TLS & routing
│
└── serviceaccount.yaml    # Shared service account

Debugging: Checkout service showing CrashLoopBackOff with 7 restarts

Investigating pod failures - Prometheus container terminated due to OOMKilled

ArgoCD showing all resources healthy and synced after debugging

Example: Cart Service Deployment Manifest

# kubernetes/cart/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opentelemetry-demo-cartservice
  labels:
    app.kubernetes.io/component: cartservice
    app.kubernetes.io/version: "1.12.0"
spec:
  replicas: 1
  selector:
    matchLabels:
      opentelemetry.io/name: opentelemetry-demo-cartservice
  template:
    metadata:
      labels:
        opentelemetry.io/name: opentelemetry-demo-cartservice
    spec:
      serviceAccountName: opentelemetry-demo

      # Wait for cache before starting
      initContainers:
      - name: wait-for-valkey
        image: busybox:latest
        command: ['sh', '-c', 'until nc -z opentelemetry-demo-valkey 6379; do sleep 2; done']

      containers:
      - name: cartservice
        image: akpadetsi/cart:21115741490  # Updated by CI/CD
        ports:
        - containerPort: 8080
          protocol: TCP

        env:
        # Service configuration
        - name: VALKEY_ADDR
          value: "opentelemetry-demo-valkey:6379"
        - name: FLAGD_HOST
          value: "opentelemetry-demo-flagd"

        # OpenTelemetry configuration
        - name: OTEL_SERVICE_NAME
          value: "cartservice"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://opentelemetry-demo-otelcol:4317"

        resources:
          limits:
            memory: 160Mi

The Impact

---

Part 4: Observability with OpenTelemetry

The Problem: "It's Slow, But I Don't Know Why"

With 15 microservices, a single user request might touch 8 different services. When something's slow or broken, finding the culprit is like finding a needle in a haystack - blindfolded.

The analogy: Observability is like having security cameras in every room of a building, plus motion sensors, temperature monitors, and a central control room to view it all.

The Solution: The Three Pillars of Observability

┌─────────────────────────────────────────────────────────────┐
│                  OBSERVABILITY ARCHITECTURE                 │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐        │
│   │ Frontend│  │  Cart   │  │Checkout │  │ Payment │        │
│   └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘        │
│        │            │            │            │             │
│        └────────────┴─────┬──────┴────────────┘             │
│                           │                                 │
│                           ▼                                 │
│              ┌────────────────────────┐                     │
│              │  OpenTelemetry Collector│                    │
│              │  (Receives all signals) │                    │
│              └───────────┬────────────┘                     │
│                          │                                  │
│          ┌───────────────┼───────────────┐                  │
│          │               │               │                  │
│          ▼               ▼               ▼                  │
│    ┌──────────┐   ┌──────────┐   ┌──────────┐               │
│    │  Jaeger  │   │Prometheus│   │OpenSearch│               │
│    │ (Traces) │   │ (Metrics)│   │  (Logs)  │               │
│    └────┬─────┘   └────┬─────┘   └──────────┘               │
│         │              │                                    │
│         └──────┬───────┘                                    │
│                │                                            │
│                ▼                                            │
│         ┌──────────┐                                        │
│         │ Grafana  │ ◀── Unified dashboards                 │
│         └──────────┘                                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

OpenTelemetry Collector Configuration

The OTel Collector is the heart of our observability pipeline:

# Collector configuration (ConfigMap)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}  # Batch telemetry for efficiency
  memory_limiter:
    check_interval: 1s
    limit_percentage: 90

connectors:
  spanmetrics: {}  # Generate RED metrics from traces

exporters:
  otlp:
    endpoint: opentelemetry-demo-jaeger-collector:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://opentelemetry-demo-prometheus-server:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp, spanmetrics]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

Service Instrumentation

Every microservice sends telemetry to the collector:

# Environment variables in each deployment
env:
- name: OTEL_SERVICE_NAME
  value: "cartservice"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://opentelemetry-demo-otelcol:4317"
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "service.namespace=opentelemetry-demo"

Distributed Tracing with Jaeger

When a user places an order, the trace shows the complete journey:

Checkout Request (250ms total)
├── Frontend (15ms)
├── Cart Service (45ms)
│   └── Valkey Cache Lookup (5ms)
├── Product Catalog (30ms)
├── Currency Service (20ms)
├── Shipping Quote (35ms)
├── Payment Service (80ms) ◀── Bottleneck identified!
└── Email Service (25ms)

Metrics with Prometheus + Grafana

Pre-built dashboards provide real-time visibility:

1. Kubernetes Overview - Pod counts, node health, resource utilization

2. Service RED Metrics - Rate, Errors, Duration per service

3. OTel Collector Health - Telemetry pipeline throughput

The Impact

---

Challenges Faced & Lessons Learned

Challenge 1: CrashLoopBackOff - Missing Environment Variables

Problem: The checkout service entered CrashLoopBackOff with 7 restarts. Investigation revealed missing critical environment variables (CHECKOUT_PORT, SHIPPING_PORT, PAYMENT_PORT).

Symptoms:

kubectl get pods -n otel-demo
# opentelemetry-demo-checkoutservice-85688dcfb5-wj5hf   0/1   CrashLoopBackOff   7 (2m17s ago)

Root Cause: Services couldn't communicate because port environment variables weren't defined in the deployment manifests.

Solution:

• Added missing environment variables to deployment manifests for recommendation, payment, shipping, and checkout services

• Standardized port configuration across all services

• Created a checklist for new service deployments to prevent similar issues

# Added to deployment manifests
env:
- name: CHECKOUT_PORT
  value: "8080"
- name: SHIPPING_PORT
  value: "8080"
- name: PAYMENT_PORT
  value: "8080"
- name: PRODUCT_CATALOG_ADDR
  value: "opentelemetry-demo-productcatalogservice:3550"

Challenge 2: OOMKilled - Prometheus Memory Exhaustion

Problem: Prometheus pod kept restarting due to out-of-memory (OOMKilled) errors.

Symptoms:

kubectl describe pod opentelemetry-demo-prometheus-dd9458b6f-hfcpg -n otel-demo
# State:      Waiting
# Reason:     CrashLoopBackOff
# Last State: Terminated
# Reason:     OOMKilled
# Exit Code:  137

Root Cause: Initial resource limits were too conservative. Prometheus was collecting metrics from 15+ services and hitting memory limits during high cardinality metric ingestion.

Solution:

• Increased memory limits from default to accommodate metric volume

• Implemented metric retention policies to control data growth

• Added resource requests and limits to prevent node resource exhaustion

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "2Gi"
    cpu: "1000m"

Challenge 3: Duplicate Port Definitions

Problem: Kubernetes deployment failed with warnings about duplicate port names in service manifests.

Symptoms:

kubectl apply -f complete-deploy.yaml
# Warning: spec.template.spec.containers[1].ports[0]: duplicate port name "service"

Root Cause: Multiple containers in the same pod were trying to use the same port name "service", causing conflicts during deployment.

Solution:

• Reviewed all service manifests and removed duplicate port definitions

• Ensured each port had a unique name within the pod specification

• Standardized port naming convention: http, grpc, metrics instead of generic "service"

Challenge 4: ArgoCD Sync Conflicts and Resource Management

Problem: ArgoCD showed resources as "OutOfSync" even after successful deployments. Some resources had duplicate port definitions causing apply failures.

Symptoms:

Warning: spec.template.spec.containers[1].ports[0]: duplicate port name "service"

Solution:

• Cleaned up duplicate port definitions in service manifests

• Configured ArgoCD with prune: true and selfHeal: true for automatic reconciliation

• Used PruneLast=true sync option to handle dependencies correctly

• Organized manifests into clear directory structures that ArgoCD could recurse through

Challenge 5: Image Tag Management and Rollback

Problem: Using latest tags made it impossible to track which version was deployed or roll back to previous versions.

Solution:

• Implemented GitHub Run ID based tagging: servicename:21115741490

• Each image tag is immutable and traceable back to specific CI run

• Rollback became as simple as updating the manifest with a previous tag and committing

Challenge 6: Debugging Distributed Systems

Problem: With 15 services running across multiple pods, finding the root cause of failures and performance issues was challenging. Traditional log analysis across multiple containers was time-consuming.

Solution:

• Implemented OpenTelemetry instrumentation from day one for all services

• Used Jaeger for distributed tracing to visualize request flows across services

• Leveraged K9s terminal UI for quick pod status checks and log viewing

• Used ArgoCD UI to monitor deployment health and sync status in real-time

Key Lesson: The screenshots show the journey from failure (CrashLoopBackOff, OOMKilled) to success (all pods running, ArgoCD synced). Having proper monitoring tools (K9s, ArgoCD UI, kubectl describe) made debugging these issues possible. Without visibility into pod states and resource usage, these problems would have been nearly impossible to diagnose.

---

Putting It All Together: The Complete Flow

Let's trace a real change from code to production:

09:00 - Developer fixes bug in Cart Service
09:01 - Push to main branch
09:02 - GitHub Actions triggered (cart-ci.yaml)
09:03 - .NET build completes ✓
09:04 - Unit tests pass ✓
09:05 - Docker image built and pushed (tag: 21345678)
09:06 - Pipeline updates kubernetes/cart/deploy.yaml
09:06 - Pipeline commits "Update cart image tag to 21345678"
09:07 - ArgoCD detects Git change
09:07 - ArgoCD syncs to cluster
09:08 - Kubernetes rolling update begins
09:09 - New pods healthy, old pods terminated
09:09 - Fix is live in production ✓

Total time: 9 minutes (zero manual intervention)

---

Open Telemetry Application live in the browser

Key Takeaways

1. Infrastructure as Code is Non-Negotiable

Terraform transforms infrastructure from "click around until it works" to reproducible, version-controlled code. Your cluster becomes as reviewable as your application code.

2. CI/CD Should Be Language-Agnostic in Design

While build steps differ, the pipeline structure (build → test → containerize → deploy) remains consistent. This makes onboarding new services predictable.

3. GitOps Eliminates Configuration Drift

With ArgoCD, Git becomes the single source of truth. No more "what's actually running in production?" questions.

4. Observability is Not Optional

With microservices, you can't debug what you can't see. OpenTelemetry provides the visibility needed to operate complex distributed systems.

---

What's Next?

This project continues to evolve. Future improvements include:

• Progressive Delivery: Implementing canary deployments with Argo Rollouts

• Policy Enforcement: Adding OPA Gatekeeper for security policies

• Cost Optimization: Implementing spot instances for non-critical workloads

• Multi-Cluster: Extending to multi-region deployments

---

Acknowledgments

This project was built using the OpenTelemetry Astronomy Shop Demo - an incredible open-source project maintained by the OpenTelemetry community. The demo application provides a realistic, polyglot microservices architecture specifically designed for learning and practicing cloud-native technologies.

Huge thanks to the OpenTelemetry community for creating and maintaining this educational resource. The demo includes:

• 15+ microservices in 7+ programming languages

• Pre-built observability instrumentation

• Real-world e-commerce scenarios

• Production-grade architecture patterns

This made it possible to focus on learning DevOps practices (Terraform, CI/CD, GitOps, observability) without having to build a complex application from scratch. If you're learning cloud-native technologies, I highly recommend using this demo as your playground.

Project Repository: github.com/open-telemetry/opentelemetry-demo

---

Resources

• OpenTelemetry Demo Repository

• ArgoCD Documentation

• Terraform DigitalOcean Provider

• GitHub Actions Documentation

---

About the Author

I'm a DevOps Engineer passionate about automation, observability, and cloud-native technologies. This project represents my approach to building production-grade systems: automate everything, observe everything, and iterate continuously.

Want to connect?

• GitHub

• LinkedIn

---

Found this helpful? Share it with someone learning DevOps! Questions? Drop a comment below.

---