Airbyte Deployment

Complete example deploying Airbyte with Helm, database, and proxy

Overview

This example shows how to deploy Airbyte on Qovery using Terraform. Airbyte is deployed using Helm with a PostgreSQL database and proxy for authentication. This is a complete, production-ready example.

This is based on the qovery-airbyte repository - a real-world example of deploying a complex application stack with Qovery.

What You’ll Deploy

• PostgreSQL Database: Managed database for Airbyte metadata
• Airbyte (Helm): Complete Airbyte deployment from official Helm chart
• Proxy Application: Authentication proxy for Airbyte web UI
• Deployment Stages: Ordered deployment (Database → App → Proxy)

Prerequisites

Before deploying, gather these IDs from Qovery Console:

# Required variables
export TF_VAR_qovery_access_token="your-api-token"
export TF_VAR_qovery_organization_id="your-org-id"
export TF_VAR_qovery_project_id="your-project-id"
export TF_VAR_qovery_cluster_id="your-cluster-id"
export TF_VAR_qovery_github_token_id="your-github-token-id"

File Structure

qovery-airbyte/
├── main.tf
├── variables.tf
└── airbyte-values.yaml

variables.tf

variable "qovery_access_token" {
  description = "Qovery API token"
  type        = string
  sensitive   = true
}

variable "qovery_organization_id" {
  description = "Qovery Organization ID"
  type        = string
}

variable "qovery_project_id" {
  description = "Qovery Project ID"
  type        = string
}

variable "qovery_cluster_id" {
  description = "Qovery Cluster ID"
  type        = string
}

variable "qovery_github_token_id" {
  description = "GitHub token for Qovery"
  type        = string
}

variable "airbyte_helm_version" {
  description = "Airbyte Helm chart version"
  type        = string
  default     = "1.7.1"
}

variable "airbyte_service_name" {
  description = "Airbyte service name"
  type        = string
  default     = "Airbyte"
}

variable "qovery_airbyte_web_app_proxy_basic_auth" {
  description = "Basic Auth for Airbyte web app proxy (htpasswd format)"
  type        = string
  default     = ""
}

main.tf

terraform {
  required_providers {
    qovery = {
      source  = "qovery/qovery"
      version = "~> 0.48.2"
    }
  }
}

provider "qovery" {
  token = var.qovery_access_token
}

# Read Airbyte Helm values from file
data "local_file" "airbyte_values" {
  filename = "${path.module}/airbyte-values.yaml"
}

# Create environment for Airbyte
resource "qovery_environment" "airbyte" {
  project_id = var.qovery_project_id
  cluster_id = var.qovery_cluster_id
  name       = "airbyte"
  mode       = "PRODUCTION"

  deployment_stage_order = [
    "DATABASE",
    "APP",
    "PROXY"
  ]
}

# Create PostgreSQL database for Airbyte
resource "qovery_database" "airbyte_db" {
  environment_id = qovery_environment.airbyte.id
  name           = "airbyte-db"
  type           = "POSTGRESQL"
  version        = "17"
  mode           = "MANAGED"
  storage        = 20
  accessibility  = "PRIVATE"

  deployment_stage_id = qovery_environment.airbyte.deployment_stage_ids["DATABASE"]
}

# Add Airbyte Helm repository
resource "qovery_helm_repository" "airbyte" {
  organization_id = var.qovery_organization_id
  name            = "airbyte"
  kind            = "OCI_DOCKER_HUB"
  url             = "https://airbytehq.github.io/helm-charts"
}

# Deploy Airbyte using Helm
resource "qovery_helm" "airbyte" {
  environment_id = qovery_environment.airbyte.id
  name           = var.airbyte_service_name

  source = {
    helm_repository = {
      repository_id = qovery_helm_repository.airbyte.id
      chart_name    = "airbyte"
      chart_version = var.airbyte_helm_version
    }
  }

  # Allow Airbyte to create cluster-wide resources
  allow_cluster_wide_resources = true

  # Override default values
  values_override = {
    file = {
      raw = data.local_file.airbyte_values.content
    }
  }

  # Configure database connection using Qovery environment variables
  environment_variables = [
    {
      key   = "DATABASE_HOST"
      value = qovery_database.airbyte_db.internal_host
    },
    {
      key   = "DATABASE_PORT"
      value = tostring(qovery_database.airbyte_db.port)
    },
    {
      key   = "DATABASE_DB"
      value = "airbyte"
    },
    {
      key   = "DATABASE_USER"
      value = qovery_database.airbyte_db.login
    }
  ]

  secrets = [
    {
      key   = "DATABASE_PASSWORD"
      value = qovery_database.airbyte_db.password
    }
  ]

  deployment_stage_id = qovery_environment.airbyte.deployment_stage_ids["APP"]

  depends_on = [
    qovery_database.airbyte_db,
    qovery_helm_repository.airbyte
  ]
}

# Deploy proxy application for Airbyte web app
resource "qovery_application" "airbyte_webapp_proxy" {
  environment_id = qovery_environment.airbyte.id
  name           = "airbyte-webapp-proxy"

  git_repository = {
    url           = "https://github.com/Qovery/simple-example-with-htt-auth"
    branch        = "main"
    root_path     = "/"
    git_token_id  = var.qovery_github_token_id
  }

  build_mode      = "DOCKER"
  dockerfile_path = "Dockerfile"

  cpu    = 100
  memory = 128

  min_running_instances = 1
  max_running_instances = 1

  ports = [{
    internal_port       = 80
    external_port       = 443
    protocol            = "HTTP"
    publicly_accessible = true
    name                = "http"
  }]

  environment_variables = [
    {
      key   = "TARGET"
      value = "http://airbyte-airbyte-webapp-svc.${qovery_environment.airbyte.id}:80"
    },
    {
      key   = "HTPASSWD"
      value = var.qovery_airbyte_web_app_proxy_basic_auth
    }
  ]

  healthchecks = {
    readiness_probe = {
      type = {
        http = {
          path = "/"
          port = 80
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }

    liveness_probe = {
      type = {
        http = {
          path = "/"
          port = 80
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }
  }

  deployment_stage_id = qovery_environment.airbyte.deployment_stage_ids["PROXY"]

  depends_on = [qovery_helm.airbyte]
}

airbyte-values.yaml

Create a airbyte-values.yaml file to customize Airbyte configuration:

global:
  serviceAccountName: airbyte-admin
  deploymentMode: oss

  database:
    type: external
    host: ${DATABASE_HOST}
    port: ${DATABASE_PORT}
    database: ${DATABASE_DB}
    user: ${DATABASE_USER}
    password: ${DATABASE_PASSWORD}

webapp:
  enabled: true
  replicaCount: 1
  service:
    type: ClusterIP
    port: 80

server:
  enabled: true
  replicaCount: 1

worker:
  enabled: true
  replicaCount: 1

airbyte-bootloader:
  enabled: true

Deployment Steps

1. Clone or Create Files

   # Option 1: Clone the repository
   git clone https://github.com/evoxmusic/qovery-airbyte
   cd qovery-airbyte
   
   # Option 2: Create the files manually
   # Create main.tf, variables.tf, and airbyte-values.yaml
   
2. Set Environment Variables

   export TF_VAR_qovery_access_token="your-api-token"
   export TF_VAR_qovery_organization_id="your-org-id"
   export TF_VAR_qovery_project_id="your-project-id"
   export TF_VAR_qovery_cluster_id="your-cluster-id"
   export TF_VAR_qovery_github_token_id="your-github-token-id"
   
3. Optional: Set Basic Authentication

   # Generate basic auth credentials
   export TF_VAR_qovery_airbyte_web_app_proxy_basic_auth=$(htpasswd -nb admin yourpassword)
   
4. Initialize Terraform

   terraform init
   
5. Plan and Apply

   # Review changes
   terraform plan
   
   # Deploy
   terraform apply
   
6. Access Airbyte

   After ~5 minutes, access Airbyte:

   # Option 1: Get URL from Qovery Console
   # Navigate to the airbyte-webapp-proxy application
   
   # Option 2: Use port-forward
   qovery port-forward --application airbyte-webapp-proxy --port 8080:443
   
   # Open http://localhost:8080
   

Cleanup

terraform destroy

Key Takeaways

This example demonstrates several advanced Terraform patterns:

Multi-Service Deployment

Deploying multiple interconnected services:

• Managed PostgreSQL database
• Helm chart application
• Proxy application for authentication

Deployment Stages

Ensuring proper deployment order:

1. DATABASE: PostgreSQL deploys first
2. APP: Airbyte Helm chart deploys after database
3. PROXY: Authentication proxy deploys last

Dynamic Configuration

Using Qovery’s database connection details in Helm values:

• Database host, port, credentials injected as environment variables
• Variables referenced in airbyte-values.yaml

External Helm Charts

Integrating third-party Helm repositories:

• Adding Airbyte’s official Helm repository
• Deploying specific chart versions
• Overriding values with custom configuration

Security

• Optional basic authentication for web access
• Private database accessibility
• Secrets stored securely

Health Checks

Proper liveness and readiness probes:

• Initial delay to allow services to start
• Regular health checks
• Automatic restarts on failure

Resource Management

Setting appropriate CPU/memory limits:

• Minimal resources for proxy (100 CPU, 128 MB)
• Managed database with defined storage
• Cluster-wide resource permissions for Airbyte

Troubleshooting

Deployment Takes Too Long

Airbyte deployment typically takes 5-10 minutes:

• Database must be ready first
• Helm chart pulls multiple images
• Airbyte bootloader initializes the database

Check deployment status:

qovery status --watch

Cannot Access Web UI

If you can’t access the Airbyte UI:

1. Verify all services are deployed
2. Check proxy application is running
3. Ensure basic auth credentials are correct
4. Try port-forwarding locally

Database Connection Issues

If Airbyte can’t connect to the database:

• Verify database is in RUNNING state
• Check environment variables are set correctly
• Review airbyte-values.yaml configuration
• Check Airbyte logs for connection errors

Next Steps

GitHub Repository

Complete source code and additional documentation

Airbyte Documentation

Learn more about Airbyte

Helm Configuration

Deploy more Helm charts

Advanced Patterns

Learn advanced Terraform techniques