CI/CD at Scale: Automating the Path from Code to Production

I have been at HMH long enough to live through three generations of CI/CD: Jenkins, then Concourse CI, and now GitHub Actions with self-hosted runners on Kubernetes. Most of this story happened in the Concourse era. HMH chose Concourse as the deliberate alternative to per-team Jenkins sprawl, one centralized build platform running thousands of builds a day, with each team scoped to its own namespace.

But a CI platform is only as good as its shared building blocks. Without a common catalog of well-built pipeline resources, every team reinvents secrets handling, image publishing, and notifications, and quality varies with whoever wrote it last.

I founded the organization's shared Concourse resource catalog and stayed its largest contributor for four years. The first pull request set the standards every consuming team still follows: development guidelines for authoring a resource, a consistent build convention, a PR template, and semantic versioning. The catalog grew to include:

• A Vault resource that lets pipelines pull dynamic secrets like AWS STS without leaking secret material into build logs
• Adaptive kubectl and helm images that target multiple Kubernetes versions from one image, which quietly eased every fleet-wide EKS upgrade
• Team notifications, artifact repository integration, and code-quality gate resources, adopted from the community and hardened for internal use

One library, one place to patch. When a fleet-wide EKS upgrade needed new kubectl and helm behavior, we changed the resource once and every consuming pipeline inherited it.

For the ML platform, I designed the standardized instanced-pipeline architecture. A meta-pipeline sets itself GitOps-style, then fans out per-tenant with per-environment child pipelines from reusable templates. The templates cover artifact builds, image mirroring and promotion, model approval, and scheduled training runs.

Separation of duties is encoded in the platform itself. Credentials for lower environments can promote builds through development, integration, and certification, but never to production. Production credentials can only make that final hop. No single credential can walk code from development to production.

Security gates live in the pipelines too. Snyk scans IaC and built artifacts, and Orca covers runtime, so a vulnerable image is a blocked pull request instead of a ticket that sits for months. An AMI validation pipeline boots every machine image in an ephemeral instance and verifies its boot sequence before the image can promote.

The newest generation runs GitHub Actions with Actions Runner Controller on the Kubernetes platform. We host Vault and Consul in our own accounts, so we wanted the runners hosted internally as well, next to the systems they talk to. Moving pipeline auth to GitHub OIDC for AWS retired the Vault-brokered credential flows. Tokens are now short-lived, scoped, and per-workflow, and there is one less system whose outage could stop every build.

The same era brought the move off self-hosted GitHub entirely. I led the architecture and planning of the GitHub Enterprise Server to Cloud migration covering 6,000+ repositories, and I built the enterprise foundations myself: SSO and user lifecycle automation managed as code, enterprise-wide policies, and break-glass procedures that were tested and documented. Execution later moved to another team. The patterns and architecture from that plan are what the migration ran on.

The catalog became how teams build pipelines at HMH, and the instanced-pipeline architecture became the CI backbone of the ML platform. The catalog's documentation has held up because I tested every example against the real CLI before publishing.

Paved roads beat mandates. Teams adopted the shared catalog because it was the easiest correct path. Standards live longest when they ship with working examples. The guidelines from that first pull request are still followed years later because they came with a build convention and a template anyone could copy.

Splitting promotion authority across teams and environments was the decision that made the pipeline a security boundary. It cost a little convenience up front and bought a lot of trust. Building it in from the start was far cheaper than retrofitting it later would have been.

Three generations in, the strongest lesson is to not put the tooling I implement on a pedestal. Jenkins was right for its moment, and so was Concourse. The industry always moves, and the best shared service is one you can simplify away.