GitLab recently discussed a solution to automate the migration of container images from Amazon Elastic Container Registry (ECR) to GitLab’s Container Registry. The team created a CI/CD pipeline to automate the process of discovering, retagging, and transferring container images from Amazon ECR to GitLab’s Container Registry.
Tim Rizzi, Principal Product Manager at GitLab summarised the capability in a blog post. The pipeline makes the migration tasks easy by enabling engineers to configure it and execute the process without manual intervention. Rizzi provided the context of the challenges faced by the Platform Engineers. When modernizing their DevSecOps toolchains with GitLab, Platform Engineers reported issues with the manual and repetitive nature of transferring hundreds of container images across multiple repositories and tags.
The newly created CI/CD pipeline starts by establishing secure access to AWS ECR through a read-only Identity and Access Management (IAM) policy, which provides the necessary permissions for retrieving image metadata and downloading container layers while adhering to security standards.
Once access is configured, the next step is setting up GitLab CI/CD variables. These include AWS credentials (account ID, region, access key, and secret key) and a BULK_MIGRATE flag set to “true” to enable automated migration. With these configurations in place, the pipeline uses Docker-in-Docker service to perform image operations reliably.
The pipeline operates through three interconnected phases, each building on the previous one to ensure a seamless migration process. Discovery involves scanning and identifying all repositories in ECR by retrieving the list of repository names. Once the repositories are identified, Tag Enumeration examines each one to list all associated image tags.
REPOS=$(aws ecr describe-repositories --query 'repositories[*].repositoryName' --output text)
TAGS=$(aws ecr describe-images --repository-name $repo --query 'imageDetails[*].imageTags[]' --output text)
docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${repo}:${tag}
docker tag ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${repo}:${tag} ${CI_REGISTRY_IMAGE}/${repo}:${tag}
docker push ${CI_REGISTRY_IMAGE}/${repo}:${tag}
Finally, Transfer fetches each image from ECR, updates its tag to align with GitLab’s registry, and uploads it to its new destination. The pipeline further incorporates error handling mechanisms and comprehensive logging, allowing teams to track progress and troubleshoot any issues that may arise during execution.
We came across this discussion on Reddit, where the tech community was discussing the challenges related to inefficiencies of manual or slow processes in container image handling. There was also a crosspost to the Kubernetes subreddit, where users were invited to share their thoughts.
However, the above Reddit threads focused on strategies like pre-baked AMIs and alternative tools like Spegel to mitigate delays, while the GitLab solution proposes an automated pipeline for image migration and reduced manual effort.
Regarding the benefits of this automated approach, there will not be a need for manual scripting or oversight during migration, ensuring consistency in image naming between ECR and GitLab. Additionally, the automation solution provides clear logs for monitoring progress and troubleshooting failed transfers.
To get started with this solution, users need to copy the provided .gitlab-ci.yml file into their repository and configure the necessary variables in their GitLab CI/CD settings.
In the blog post, Rizzi has also offered practical advice for successful migrations: running pipelines during off-peak hours to minimize impact on team workflows, monitoring logs closely for potential issues, verifying all images before decommissioning ECR, and implementing rate limiting for very large migrations to avoid network congestion.
