Deploy Kingdom on GKE

Deploy Halo Kingdom components to Google Kubernetes Engine

Overview

This guide walks you through deploying the Kingdom to Google Kubernetes Engine (GKE). The dev environment configuration can be used as a basis for your deployment.

This guide provides one approach to deployment. Adjust quotas, names, and resource configurations based on your specific requirements.

Prerequisites

Before starting, complete the deployment prerequisites including:

• Bazel/Bazelisk installation
• Google Cloud SDK configuration
• kubectl installation
• Terraform installation

Kingdom Components

The Kingdom deployment creates the following Kubernetes resources:

Services

• gcp-kingdom-data-server (Cluster IP)
• system-api-server (External LoadBalancer)
• v2alpha-public-api-server (External LoadBalancer)

Deployments

• gcp-kingdom-data-server-deployment
• system-api-server-deployment
• v2alpha-public-api-server-deployment

CronJobs

• completed-measurements-deletion-cronjob
• pending-measurements-cancellation-cronjob
• exchanges-deletion-cronjob

Secrets & ConfigMaps

• certs-and-configs-<hash> (Secret)
• config-files-<hash> (ConfigMap)

Network Policies

• default-deny-network-policy
• kube-dns-network-policy
• gke-network-policy
• API server and data server network policies

Deployment Steps

1. Provision Infrastructure with Terraform

   Use the example Terraform configuration to provision Google Cloud resources:

   cd src/main/terraform/gcloud/examples/kingdom
   

   Create a backend.tf file for state management:

   backend.tf

   terraform {
     backend "gcs" {
       bucket = "my-terraform-state-bucket"
       prefix = "terraform/state/halo-cmms-kingdom"
     }
   }
   

   Initialize and apply the Terraform configuration:

   terraform init
   terraform plan
   terraform apply
   

   The Terraform configuration creates:

   • GKE cluster named kingdom
   • Cloud Spanner instance (1000 processing units, Enterprise edition)
   • KMS key ring for encryption
   • External IP addresses for API servers
2. Get Cluster Credentials

   Configure kubectl to access your new cluster:

   gcloud container clusters get-credentials kingdom --region=us-central1
   
3. Build and Push Container Images (Optional)

   If not using pre-built release images, build and push images from source:

   bazel run -c opt //src/main/docker:push_all_kingdom_gke_images \
     --define container_registry=gcr.io \
     --define image_repo_prefix=halo-kingdom-demo \
     --define image_tag=build-0001
   

   Using Hybrid Development

   For containerized builds, replace commands:

   • bazel build → tools/bazel-container build
   • bazel run → tools/bazel-container-run
4. Generate Kubernetes Kustomization

   Generate the K8s configuration using Bazel:

   bazel build //src/main/k8s/dev:kingdom.tar \
     --define google_cloud_project=halo-kingdom-demo \
     --define spanner_instance=halo-cmms \
     --define kingdom_public_api_address_name=kingdom-v2alpha \
     --define kingdom_system_api_address_name=kingdom-system-v1alpha \
     --define container_registry=ghcr.io \
     --define image_repo_prefix=world-federation-of-advertisers \
     --define image_tag=0.5.2
   

   Extract the generated archive to a secure location:

   mkdir -p ~/kingdom-deployment
   tar -xf bazel-bin/src/main/k8s/dev/kingdom.tar -C ~/kingdom-deployment
   

   Store this directory securely as it will contain sensitive information. Persist it for applying updates.

5. Customize Kubernetes Secret

   Prepare certificate and configuration files in ~/kingdom-deployment/src/main/k8s/dev/kingdom_secret/:

   Required Files:

   1. all_root_certs.pem - TLS trusted root CA store

      • Concatenation of root certificates for all entities:
        • All Duchies
        • All EDPs
        • All MC reporting tools
        • Kingdom itself
      cat *_root.pem > all_root_certs.pem
      

   2. kingdom_root.pem - Kingdom’s root CA certificate

   3. kingdom_tls.pem - Kingdom’s TLS certificate

   4. kingdom_tls.key - Private key for Kingdom’s TLS certificate

   5. duchy_cert_config.textproto - Duchy certificate to ID mapping

      • Example

   6. duchy_id_config.textproto - External to internal Duchy ID mapping

      • Example

   7. Protocol Configurations:

      • llv2_protocol_config_config.textproto - Liquid Legions v2
      • ro_llv2_protocol_config_config.textproto - Reach-Only Liquid Legions v2
      • hmss_protocol_config_config.textproto - Honest Majority Share Shuffle

   Never use testing secret files in production! The repository contains test certificates that must only be used for testing.

   For Testing Only

   bazel build //src/main/k8s/testing/secretfiles:archive
   tar -xf bazel-bin/src/main/k8s/testing/secretfiles/archive.tar \
     -C ~/kingdom-deployment/src/main/k8s/dev/kingdom_secret/
   
6. Customize Kubernetes ConfigMap

   Place configuration files in ~/kingdom-deployment/src/main/k8s/dev/config_files/:

   • authority_key_identifier_to_principal_map.textproto - Principal mapping configuration
   • known_event_group_metadata_type_set.pb - Protobuf FileDescriptorSet for EventGroup metadata types

   See Creating Resources for details.

7. Apply Kubernetes Kustomization

   Deploy all Kingdom components:

   cd ~/kingdom-deployment
   kubectl apply -k src/main/k8s/dev/kingdom
   

   Verify deployment:

   kubectl get deployments
   kubectl get services
   

   Expected output:

   NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
   gcp-kingdom-data-server-deployment     1/1     1            1           1m
   system-api-server-deployment           1/1     1            1           1m
   v2alpha-public-api-server-deployment   1/1     1            1           1m
   

   NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
   gcp-kingdom-data-server     ClusterIP      10.3.245.210   <none>         8443/TCP
   system-api-server           LoadBalancer   10.3.248.13    34.67.15.39    8443:30347/TCP
   v2alpha-public-api-server   LoadBalancer   10.3.255.191   34.132.87.22   8443:31300/TCP
   
8. Configure DNS Records

   Add A records in your DNS configuration mapping domain names to the external IPs:

   Example for halo-cmm.org:

   • v2alpha.kingdom.dev.halo-cmm.org → Public API external IP
   • v1alpha.system.kingdom.dev.halo-cmm.org → System API external IP

   These hostnames enable Duchies, EDPs, and model providers to access Kingdom services.

Certificate Requirements

Generate certificates using Cloud Certificate Authority Service or your preferred CA.

Requirements:

• Support both client and server TLS
• Include DNS hostnames in Subject Alternative Name (SAN):
  • *.kingdom.dev.halo-cmm.org (or your domain)
  • localhost

Encryption Keys:

Generate using Tinkey:

tinkey create-keyset --key-template AES128_GCM --out-format binary --out key.tink

Updating Configuration

To update secrets or configuration:

1. Modify files in the Kustomization directory
2. Re-apply the configuration:

kubectl apply -k src/main/k8s/dev/kingdom

Testing the Deployment

Verify the Kingdom is working properly by:

1. Running ResourceSetup to create resources
2. Completing a multi-cluster correctness test

Terraform Configuration Reference

main.tf

# Key resources created
resource "google_spanner_instance" "spanner_instance" {
  name             = "halo-cmms"
  config           = "regional-us-central1"
  processing_units = 1000
  edition          = "ENTERPRISE"
}

module "kingdom_cluster" {
  source          = "../../modules/cluster"
  name            = "kingdom"
  machine_type    = "e2-custom-2-4096"
  max_node_count  = 2
}

Next Steps

Deploy Duchy

Deploy Duchy components to complete the system

Operations Guide

Learn about creating resources and system operations