This page outlines three types of Codecov Self-Hosted deployment that can be used to meet a wide variety of user needs.
? For use with 4.4.8+
These deployment strategies assume you're installing v4.4.8 or later. Earlier versions of Codecov Self-Hosted need different considerations when deploying.
Please contact email@example.com if you are deploying an earlier version of Codecov Self-Hosted
Being a containerized service, there are numerous strategies that can be utilized to deploy Codecov Self-Hosted successfully. How Codecov Self-Hosted is deployed ultimately depends on a combination of available resources, uptime needs, and planned usage.
This documentation is not intended to be an all-encompassing guide for deploying Codecov Self-Hosted. It should, however, serve to outline a few popular deployment strategies that can meet many common requirements of modern on-premises deployments.
Generally, deployment strategies fall along a spectrum of very easy to deploy but with low availability, to more challenging to deploy with high availability. Where a particular use case falls along this spectrum can be helpful in determine which deployment strategy should be used.
Unlike other reference pages in this documentation, which can be fairly easily skimmed, it is recommended to read this page in its entirety if you're considering a non-trivial deployment of Codecov Self-Hosted.
This document will outline three deployment strategies, one of each of the following types:
Minimal Configuration, minimal availability -- This is great for trial and proof of concept style deployments where an organization is still attempting to determine if Codecov is right for the organization's needs.
Moderate Configuration, high availability -- This is great for small to midsize deployments that prioritize high availability now, but not necessarily incredibly large scale. This deployment is right for an organization that has decided to use Codecov, wants to ensure high uptime to end users, and desires the ability to move to a larger scale deployment in the future with no data loss and minimal additional effort.
High Configuration, high availability, high scale -- This is great for deployments that have to scale to meet the needs of many users within extremely large organizations. This deployment strategy is the one employed by Codecov's SaaS offering: codecov.io.
The remainder of this document will outline each of these deployment types in detail, as well as provide any reference materials necessary for configuration and setup.
Deployment Strategy I: Minimal Configuration, Minimal Availability
? Trial Purposes Only
For all but the smallest of organizations, it is recommended to use this Deployment Strategy for trial purposes only.
The most straightforward deployment is to deploy Codecov Self-Hosted on a single server using docker-compose with no external services. If your goal is simply to evaluate Codecov Self-Hosted, you have no internal requirement to preserve the data generated by Codecov, and plan to start fresh with a more robust deployment if you purchase Codecov; then this is the recommended Deployment Strategy.
The greatest benefit to this deployment is that a single engineer should be able to produce a usable version of Codecov Self-Hosted in two hours or less. Therefore, it's great for testing out the product.
This Deployment Strategy utilizes Docker volumes for persistence of the database, archive file/object storage, and caching. Therefore, loss of the underlying server would result in total data loss. For this reason, it is recommended to use this deployment only for the trial and proof of concept stage of using Codecov Self-Hosted. However, included with this documentation are recommended modifications to make the data generated by this deployment persistent. It is recommended to follow those instructions if you with to upgrade your deployment in the future without any potential data loss.
How to Deploy
Modifying to Use Cloud Services
? Using Cloud Services does not Guarantee Uptime
The below modification only ensures that data generated by Codecov Self-Hosted will persist past the lifetime of the server on which Codecov Self-Hosted is deployed.
If you require high availability / guarantees on uptime, please do not use this deployment.
This Deployment Strategy can be modified to support smaller deployments and/or to provide data persistence. This is useful if you do not want to allocate a large amount of resources to the installation, or if you would prefer to maintain data generated a trial/proof of concept period in the event that your organization decides to deploy Codecov permanently.
The following modifications will guarantee data persistence in the event that Codecov Self-Hosted's underlying server fails. However, this deployment provides no high availability guarantee, and it is recommended to use a different deployment to support that use case.
It is possible to use this deployment with cloud services in order to guarantee data persistence in the event of a loss of the underlying server running Codecov Self-Hosted. To do this the implementer must:
Create a Postgres 10 database in their cloud provider (e.g., RDS, Google Cloud SQL)
Create a cloud storage bucket (e.g., using S3 or Google Cloud Storage) for uploaded report storage.
Create a redis server for dedicated caching (e.g., using Elasticache)
Update the codecov.yml to use these services.
Remove the postgres and redis blocks from the docker-compose.yml
Example configuration files are provided below. These files assume the user is deploying on a single EC2 instance in Amazon Web Services using GitHub as the repository provider:
1setup: 2 # Replace with the http location of your Codecov 3 # https://docs.codecov.io/docs/configuration#section-codecov-url 4 codecov_url: "<codecov-url>" 5 # Replace with your Codecov Self-Hosted Key 6 # https://docs.codecov.io/docs/configuration#section-enterprise-license 7 enterprise_license: "<license-key>" 8 # Replace with a random string 9 # https://docs.codecov.io/docs/configuration#section-cookie-secret 10 http: 11 cookie_secret: "<your-cookie-secret>" 12github: 13 client_id: "<client-id>" 14 client_secret: "<client-secret>" 15 global_upload_token: "<upload-token>" 16services: 17 database_url: "<postgres-rds-url>" 18 redis_url: "<redis-elasticache-url>" 19 minio: 20 host: s3.amazonaws.com 21 bucket: <bucket-name> 22 region: <bucket-region> 23 verify_ssl: true 24 access_key_id: <aws-iam-access-key> 25 secret_access_key: <aws-iam-secret> 26
Note that if you're using GCP with Google Cloud Storage, your
minioblock should read as follows:
1#for GCS 2services: 3 minio: 4 host: storage.googleapis.com 5 verify_ssl: true 6 bucket: <bucket-name> 7 region: <bucket-region> 8 access_key_id: <gcs-hmac-key> 9 secret_access_key: <gcs-hmac-secret> 10
You will need to do some additional work to setup Google Cloud Storage. You can find those instructions in the v4.4.8 changelog
With this configuration you can completely remove minio from your docker-compose.yml:
1version: "3" 2 3services: 4 traefik: 5 image: traefik:v1.7-alpine 6 command: 7 - --api 8 - --docker 9 - --docker.watch 10 - --docker.constraints=tag==web 11 - --metrics 12 - --metrics.prometheus 13 14 # NOTE: Toggle the lines below to enable SSL 15 - --entryPoints=Name:http Address::80 Compress::true 16 - --defaultEntryPoints=http 17 # - --entrypoints=Name:http Address::80 Redirect.EntryPoint:https Compress::true 18 # - --entryPoints=Name:https Address::443 TLS:/ssl/ssl.crt,/ssl/ssl.key Compress::true 19 # - --defaultentrypoints=http,https 20 21 volumes: 22 - /var/run/docker.sock:/var/run/docker.sock:rw 23 - /dev/null:/traefik.toml:rw 24 # NOTE: Provide the SSL certs in the ./config folder below 25 # - ./config:/ssl 26 ports: 27 - "80:80" 28 - "8080:8080" 29 - "443:443" 30 networks: 31 - codecov 32 depends_on: 33 - web 34 web: 35 image: codecov/enterprise:v4.4.8 #or newer 36 command: web 37 volumes: 38 - ./codecov.yml:/config/codecov.yml:ro 39 ports: 40 - "5000" 41 labels: 42 - "traefik.tags=web" 43 - "traefik.backend=web" 44 - "traefik.port=5000" 45 - "traefik.frontend.rule=PathPrefix: /" 46 environment: 47 - STATSD_HOST=statsd 48 - DATADOG_TRACE_ENABLED=false 49 networks: 50 - codecov 51 depends_on: 52 - statsd 53 54 worker: 55 image: codecov/enterprise:v4.4.8 #or newer 56 command: worker 57 volumes: 58 - ./codecov.yml:/config/codecov.yml:ro 59 environment: 60 - STATSD_HOST=statsd 61 - DATADOG_TRACE_ENABLED=false 62 networks: 63 - codecov 64 depends_on: 65 - statsd 66 67 statsd: 68 image: prom/statsd-exporter:v0.6.0 69 command: -statsd.listen-udp=:8125 -statsd.listen-tcp=:8125 70 ports: 71 - "8125" 72 - "9102" 73 networks: 74 - codecov 75 76networks: 77 codecov: 78 driver: bridge 79
Note that the AWS Access Key and AWS Secret Key must be the identifying credentials for an IAM user that can specify the AWS bucket specified in the minio block of the above codecov.yml.
Scaling on a Single Server
Thanks to the use of Traefik, it is possible to scale both of codecov's services, the web and worker containers, on a single server. You should consider scaling this way if:
There is sufficient headroom on the underlying server
Report processing is lengthier than desired (e.g., status checks are taking longer than usual to complete on Pull Requests)
Web requests are taking longer than desired to complete (e.g., individual pages are taking awhile to load when viewing Codecov's web front end in the browser.)
Scaling is straightforward with:
docker-compose scale <service-name>=<container count>. For example, to add more workers:
docker-compose scale worker=3. Since Traefik is used as a reverse proxy in this deployment, new web containers should be automatically detected and used. Worker containers communicate via Redis, and should automatically begin processing tasks after startup.
❗️ The Following Configuration Changes Should be Avoided
Despite outlining the modifications in detail, it is highly recommended to avoid performing the below steps. These modifications are described because they are both performed often by users who are new to Codecov Enterprise and guaranteed not to work without unknown quantities of additional effort.
Given the simplicity of this deployment, it is tempting to do the following:
Modify the docker-compose.yml and codecov.yml to use cloud services
Spin up multiple EC2 instances using this configuration
Put a load balancer in front of all the instances
Proclaim that you have a high availability deployment.
This approach will not work. The reason being that this Deployment Strategy handles its own proxying between http/https based services using Traefik. Traefik is used in this case because it provides enormous convenience when deploying scalable/multiple services on a single server. However, if you place an external load balancer in front of multiple servers each running Traefik, the reliability of the underlying services is not guaranteed. Unless you're comfortable modifying/removing Traefik, and supplying your own reverse proxying solution for the web and minio containers, it is not recommended to try the above modification.
Deployment Strategy II: Moderate Configuration, High availability
? Suitable for Most Self-Hosted Users
This Deployment Strategy is the one generally recommended by Codecov. It strikes an acceptable balance between implementation difficulty, scalability, and availability.
This Deployment Strategy is very similar to Deploy Strategy I with the cloud services modification. However, it also provides an additional high availability / uptime guarantee by running multiple copies of each Codecov Self-Hosted service and load balancing them.
Use this Deployment Strategy if you require scalability, moderate ease of implementation and maintenance, and high availability to end users. This is the Deployment Strategy that is suitable for most Codecov Enterprise users.
The basic approach is the following:
Decouple the services into the following: web and worker, minio; and deploy each on their own instance
Place a load balancer in front of the instances running web and worker containers
Optionally place a load balancer in front of the instances running minio. This is required if you intended to horizontally scale minio and/or require high availability of minio.
Create the required cloud services, and update the codecov.yml to use those services.
In practical terms, you will need to create a single codecov.yml and two to three different docker-compose.yml's to support this deployment.
An example deployment in AWS would do the following:
Create at least three EC2 instances: two for web and worker containers, one for minio. Though more can be created if desired.
Create an Application Load Balancer for the web and worker containers. Optionally, one can be created for minio.
Create RDS (postgres 10), Elasticache (redis), and an S3 bucket for the deployment.
Split the standard docker-compose.yml file into a web+worker file and a minio file and deploy them to the appropriate instances.
Example configuration files are shown assuming a deployment into Amazon Web Services:
1setup: 2 # Replace with the http location of your Codecov 3 # https://docs.codecov.io/docs/configuration#section-codecov-url 4 codecov_url: "<load balancer path>" 5 # Replace with your Codecov Self-Hosted License key 6 # https://docs.codecov.io/docs/configuration#section-enterprise-license 7 enterprise_license: "<license-key>" 8 # Replace with a random string 9 # https://docs.codecov.io/docs/configuration#section-cookie-secret 10 http: 11 cookie_secret: "<your-cookie-secret>" 12github: 13 client_id: "<client-id>" 14 client_secret: "<client-secret>" 15 global_upload_token: "<upload-token>" 16services: 17 database_url: "<postgres-rds-url>" 18 redis_url: "<redis-elasticache-url>" 19 minio: 20 host: s3.amazonaws.com 21 bucket: <bucket-name> 22 region: <bucket-region> 23 verify_ssl: true 24 access_key_id: <aws-iam-access-key> 25 secret_access_key: <aws-iam-secret> 26
The web and worker containers can be merged into a single docker-compose.yml and deployed. Note that, most importantly, traefik has been removed:
1version: "3" 2 3services: 4 web: 5 image: codecov/enterprise:v4.4.8 #or newer 6 command: web 7 volumes: 8 - ./codecov.yml:/config/codecov.yml:ro 9 ports: 10 - "8000:5000" 11 - "80:5000" 12 environment: 13 - STATSD_HOST=statsd 14 depends_on: 15 - statsd 16 restart: always 17 worker: 18 image: codecov/enterprise:v4.4.8 #or newer 19 command: worker 20 volumes: 21 - ./codecov.yml:/config/codecov.yml:ro 22 environment: 23 - STATSD_HOST=statsd 24 - DATADOG_TRACE_ENABLED=false 25 depends_on: 26 - statsd 27 statsd: 28 image: prom/statsd-exporter:v0.6.0 29 command: -statsd.listen-udp=:8125 -statsd.listen-tcp=:8125 30 ports: 31 - "8125" 32 - "9102" 33
If desired, you can split the web and worker docker-compose into separate docker-compose.yml files and deploy them separately. The worker does not require any sort of HTTP access to function properly, as it only communicates via Redis.
Deployment Strategy III: High Configuration, High Availability, High Scale
? Recommended for High Scale
The following deployment is a near replica of what is used for Codecov's SaaS offering: codecov.io. It is recommended when high availability and high scale are required, and maintainers don't mind taking on the additional burden of Kubernetes and Terraform.
This Deployment Strategy is recommended for the most demanding of Codecov Self-Hosted deployments. It is managed using Terraform and requires Kubernetes. Codecov has provided an terraform configuration files supporting this deployment for all three major cloud providers: Azure, AWS, and Google Cloud Platform. Those resources and supporting documentation can be found on GitHub
A Note on System Requirements and Resource Needs
There are two main components to consider when determining the performance of the underlying server(s) that will support Codecov Self-Hosted: Web traffic to the Codecov Self-Hosted frontend, Report processing demands.
Since every organization is different, it's impossible to make hard and fast recommendations that will meet every deployment's needs.
As a general building block, Codecov uses Google's n1-standard-4 for deployment of web containers and the n1-standard-8 for deployment of worker containers (see: Google Machine Types) .
By far the performance bottleneck on Codecov will be coverage report processing time. Therefore it's helpful to consider resource needs in terms of number of reports processed per minute. If your install will initially process 25 or less reports per minute (i.e., 25 or fewer commits running through a codecov enabled CI/CD per minute), it is recommended to deploy the equivalent of at least 1 n1-standard-8 for all infrastructure when using Deployment Strategy I; and 2 n1-standard-8's for web+worker (to provide high availability) and 1 n1-standard-4 for minio when using Deployment Strategy II. Once deployed, the performance of each instance can be monitored and scaled up or down as needed.
This is generally a good starting point, and is straightforward to scale horizontally when using Deployment Strategy II. If larger deployments are needed, the Codecov team is happy to provide more detailed recommendations depending on specific needs.