DIY Private Container Registry

Márk Sági-Kazár

2024-02-03 @ FOSDEM ’24

whoami

Márk Sági-Kazár

Head of Open Source @ OpenMeter

CNCF Ambassador

@sagikazarmark

https://sagikazarmark.hu

hello@sagikazarmark.hu

Let me tell you a story…

• Distribute images to customers
• Design partners

Requirements

• Share container images with design partners
• Target environments
  • developer machine
  • CI
  • container orchestrator
• → Flexible authentication and authorization
• Minimize operational burden (monitoring, backup, etc)

What do people do when they are in need of a Cloud Native solution?

CNCF Landscape

Source: https://landscape.cncf.io/card-mode?category=container-registry&grouping=category

Available solutions

Cloud-hosted

P2P

All-in-one solutions

Plain old registries

Cloud-hosted registries

• Pros
  • Easy to set up
  • No operational burden
• Cons
  • Requires cloud provider account
  • Manual IAM setup

Surprisingly, companies were not eager to register cloud provider accounts.

New requirements

• No cloud provider registration required
• Flexible authorization (eg. entitlement-based)

More design partners and customers More projects with more container images

Back to research

Cloud-hosted

P2P

All-in-one solutions

Plain old registries

All-in-one solutions

JFrog

Quay (wasn’t open source at the time)

Portus (unmaintained)

Harbor

Harbor

• Structures artifacts into projects
• Robot accounts for service-to-service auth
• Simple authorization
• Image replication
• Uses Distribution as a registry

Tons of features

How we used Harbor

Harbor quirks and cons

• Group based access for users, but not for robot accounts
• Cross-project robot account creation requires admin access
• API integration only works with the admin credentials
• Complex software for a specific use case
• Operation is not trivial

But: can use object store due to Distribution

New requirements

• Self-serve portal
• Closer integration with sales and licensing systems

Maybe it’s time to build our own solution.

PLG is everywhere today

How do container registries work?

OCI

OCI

• HTTP interface -> Basic auth (if the client supports it)
• Authz?

Docker registry auth “specification”

aka. docker login

• Token based authorization
• OAuth2 protocol

• Not a formal spec
• Documented under the Distribution project

1. Client attempts to begin an operation with the registry
2. If the registry requires authorization it will return a 401 Unauthorized HTTP response with information on how to authenticate
3. Client makes a request to the authorization service for a token
4. Authorization service returns a token representing the client’s authorized access
5. Client retries the original request with the token embedded in the request’s Authorization header
6. Registry authorizes the client by validating the token and the claim set embedded within it and begins the operation as usual

Let’s put it all together

Plain old registries

• Distribution
• Zot

• CNCF Sandbox projects

Distribution

• Reference Docker registry implementation
• Major providers rely on it

https://github.com/distribution/distribution

• Supports various object stores
• Providers: Docker Hub, GitHub CR, GitLab CR, DigitalOcean CR, Harbor
• Team is focused on v3
• Last minor version is from 2022
• Doesn’t support workload identities (see this issue)

Zot

A production-ready vendor-neutral OCI-native container image registry (purely based on OCI Distribution Specification)

• Registry auth is currently broken (fixed!)

https://zotregistry.io

• Donated by Cisco
• Supports object stores

Portward

• Registry auth library and service
• Build your own with the library
• Use an existing integration with the service
• ⚠️ Work in progress!

https://github.com/portward/registry-auth

https://github.com/portward/portward

Registry auth caveats

• Not a formal specification
• There are several gaps in it
• Competing, but incompatible specs (eg. ChartMuseum Auth)
• Partial implementations

Future: OCI Auth spec

• OCI Auth working group
• Fairly new (first meeting: August 1, 2023)

https://github.com/opencontainers/wg-auth

Thank you

Any questions?

@sagikazarmark

https://sagikazarmark.hu

hello@sagikazarmark.hu