AWS has announced the availability of parallel test execution in AWS CodeBuild, a fully managed continuous integration service. According to the company, this new feature significantly reduces build times by allowing test suites to run concurrently across multiple build compute environments.
The announcement highlights the growing challenge of lengthy test execution times in continuous integration (CI) pipelines as projects become increasingly complex. These long cycles can delay feature delivery, hinder developer productivity, and increase costs.
Thomas Fernandez wrote in a Semaphore blog post on parallel testing:
Parallel testing lets us do more while waiting less. It’s an essential tool to keep sharp and ready so we can always establish a fast feedback loop.
With parallel test execution in CodeBuild, developers can now configure their build process to split test suites and run them in parallel across multiple independent build nodes. CodeBuild provides environment variables to identify the current node and the total number of nodes, enabling intelligent test distribution. The feature supports a sharding approach with two main strategies:
- Equal distribution: Sorts test files alphabetically and distributes them evenly across parallel environments.
- Stability: Uses a consistent hashing algorithm to maintain file-to-shard assignments even when test files are added or removed.
To enable parallel testing, developers configure the batch fanout section in their buildspec.xml file, specifying the desired level of parallelism. The pre-installed codebuild-tests-run utility is used in the build step to manage test execution and sharding based on the chosen strategy. A sample of a buildspec.yml that shows parallel test execution with Cucumber on a Linux platform looks like:
version: 0.2
batch:
fast-fail: false
build-fanout:
parallelism: 5
ignore-failure: false
phases:
install:
commands:
- echo 'Installing Ruby dependencies'
- gem install bundler
- bundle install
pre_build:
commands:
- echo 'prebuild'
build:
commands:
- echo 'Running Cucumber Tests'
- cucumber --init
- |
codebuild-tests-run
--test-command "cucumber"
--files-search "codebuild-glob-search '**/*.feature'"
post_build:
commands:
- echo "Test execution completed"
CodeBuild also offers automatic merging of test reports from the parallel executions into a single, consolidated test summary. This simplifies result analysis by providing aggregated pass/fail statuses, test durations, and failure details in the CodeBuild console, via the AWS CLI, or through integration with other reporting tools.
(Source: AWS Documentation)
A demonstration highlighted in an AWS blog post on the feature showed a reduction in total test time from 35 minutes to 6 minutes (including environment provisioning) for a Python project with 1,800 tests when running on ten parallel compute environments.
Sébastien Stormacq, a Principal Developer Advocate at AWS, wrote:
The 1,800 tests of the demo project take one second each to complete. When I run this test suite sequentially, it took 35 minutes to complete. When I run the test suite in parallel on ten compute environments, it took 6 minutes to complete, including the time to provision the environments. The parallel run took 17.9 percent of the time of the sequential run.
This new capability is compatible with all testing frameworks, and the AWS documentation provides examples for popular languages and frameworks like Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec). For frameworks with specific requirements for test file lists, CodeBuild provides the CODEBUILD_CURRENT_SHARD_FILES environment variable, which contains a newline-separated list of test files for the current shard.
Parallel test execution in AWS CodeBuild is available today in all AWS Regions where CodeBuild is offered, across all three compute modes: on-demand, reserved capacity, and AWS Lambda compute, with no additional cost beyond the standard CodeBuild pricing for the resources used.
