Testing

Testing procedures and organization for the Halo measurement system

The Halo measurement system uses Bazel for running all tests, including unit tests, integration tests, and correctness tests.

Running Tests

Run All Tests

Execute the entire test suite:

bazel test //src/test/...

Run Specific Test Targets

Single Package

bazel test //src/test/kotlin/org/wfanet/measurement/common:all

Test Organization

Tests are organized in the src/test/ directory tree, mirroring the structure of the source code:

Kotlin Tests

src/test/kotlin/
├── org/wfanet/measurement/
│   ├── common/          # Common utilities tests
│   ├── kingdom/         # Kingdom service tests
│   ├── reporting/       # Reporting service tests
│   └── integration/     # Integration tests

Test infrastructure that is complex enough to warrant its own tests, or is used by multiple packages, goes into a testing subpackage in src/main/ with targets marked as testonly.

Integration Tests

Integration tests verify that multiple components work together correctly. They are located in:

• src/test/kotlin/org/wfanet/measurement/integration/k8s/ - Kubernetes integration tests
• src/test/kotlin/org/wfanet/measurement/integration/common/ - Common integration utilities

Running Integration Tests

bazel test //src/test/kotlin/org/wfanet/measurement/integration/...

Correctness Tests

Correctness tests validate the accuracy of measurement computations against expected results. These tests typically run against a full CMMS deployment.

Synthetic Generator Correctness Test

This test validates end-to-end measurement accuracy:

bazel test //src/test/kotlin/org/wfanet/measurement/integration/k8s:SyntheticGeneratorCorrectnessTest \
    --test_output=streamed \
    --define=kingdom_public_api_target=v2alpha.kingdom.example.org:8443 \
    --define=mc_name=measurementConsumers/your-mc-id \
    --define=mc_api_key=your-api-key

Use the --test_output=streamed flag to see test output in real-time, which is helpful for long-running integration tests.

Test Configuration with Make Variables

Many integration and correctness tests require configuration via Bazel’s --define flag:

Resource Configuration

--define=mc_name=measurementConsumers/Rcn7fKd25C8
--define=mc_api_key=W9q4zad246g
--define=edp1_name=dataProviders/FeQ5FqAQ5_0

TestContainers

Some tests use TestContainers to spin up Docker containers for dependencies like databases.

1. Ensure Docker is running

   TestContainers requires Docker to be available on your system.

2. Run tests normally

   bazel test //src/test/kotlin/org/wfanet/measurement/kingdom/deploy/gcloud/spanner:...
   
3. Container-based testing

   When running inside a container, use tools/bazel-container which handles the required setup:

   tools/bazel-container test //src/test/...
   

Podman is not supported for TestContainers-based tests. You must use Docker.

Testing Best Practices

Unit Testing Standards

Test the Contract

Test public APIs, not implementation details. Internal functionality should not be exposed just for testing.

Use Test Doubles Wisely

Choose appropriate test doubles:

• Fakes for working implementations
• Mocks for behavior verification
• Stubs for simple responses
• Real dependencies when practical

Smaller Test Cases

Bias toward more, smaller test cases rather than fewer large ones.

Truth Assertions

Use the Truth library for Kotlin test assertions, including ProtoTruth for protocol buffers.

Writing Tests in Kotlin

import com.google.common.truth.Truth.assertThat
import com.google.common.truth.extensions.proto.ProtoTruth.assertThat
import kotlin.test.assertFailsWith
import org.junit.Test

class MyServiceTest {
  @Test
  fun `createResource returns expected result`() {
    val service = MyService()
    val result = service.createResource("test")
    
    assertThat(result.name).isEqualTo("test")
  }
  
  @Test
  fun `createResource with invalid input throws exception`() {
    val service = MyService()
    
    assertFailsWith<IllegalArgumentException> {
      service.createResource("")
    }
  }
}

CI/CD Testing

Continuous integration runs tests automatically on pull requests and commits. The CI pipeline:

1. Runs formatters and linters
2. Executes all unit tests
3. Runs integration tests where applicable
4. Checks for compiler warnings
5. Validates code coverage

All tests must pass before a pull request can be merged. See the Contributing Guide for more information.

Test Output Options

Stream Output

# See output as tests run
bazel test //src/test/... --test_output=streamed

Monitoring Test Execution

For correctness tests running against a deployed system, you can monitor progress through:

• Pod logs in Kubernetes clusters
• Spanner database queries to check measurement states
• Test output with --test_output=streamed

Example test run showing successful completion:

INFO: Created measurement measurementConsumers/ABC/measurements/XYZ.
INFO: Computation not done yet, wait for another 30 seconds.
...
INFO: Got computed result from Kingdom: reach { value: 11542 }
INFO: Expected result: reach { value: 11570 }
INFO: Computed result is equal to the expected result. Correctness Test passes.

Next Steps

Dev Standards

Learn about code review and commit standards

Contributing

How to contribute to the project