§25 Clever Cloud's Open-Source Stack and the GitOps Deployment

Status: Proposed · 2026-09-29. This is the companion to §24. The owner's direction (2026-09-29): "use rustfs and clever cloud opensource stack to gitops, evaluate all the tools that are we adopt from clever cloud". This document does four things: it inventories every relevant open-source project from Clever Cloud (§2), compares Sōzu with Loam's planned edge (§3), compares Biscuit with the planned auth (§4), and designs the GitOps layout (§5–§6). Its decisions are D185–D188 in §24's table, and its questions are Q-RT-8 … Q-RT-14.

Amended 2026-10-02 by §38 (proposed): "GitOps from Clever Cloud" means Clever's open-source operator and infrastructure tooling (D185, the CKE Terraform and Karpenter providers); Clever publishes no GitOps engine, so Argo CD stays (D186, D453), with a Flux layout for the smallest profile (D454). §6.3 gains waves for CloudNativePG, the Knative Operator, Authentik and Knative (D455); the operator also reconciles Knative tenancy per namespace (D443). Plan MT3.

Markers are the same as in §24. Every license was read from the repository's LICENSE file (or, where there is none, from the crate manifest, as noted). Activity dates come from the GitHub API on 2026-09-29.

License rule applied throughout. A linked dependency must not be AGPL, BSL, SSPL or ELv2. An AGPL service may run only unmodified, as a separate process, and the risk is flagged even then. LGPL linked into a Rust binary is not banned by the rule, but static linking brings relinking obligations. It is flagged wherever it appears.

1. SummaryCopy Anchor Link

• Adopt as linked dependencies: biscuit-auth (Apache-2.0) for sandbox tokens (D188), and clevercloud-sdk (MIT) behind an optional Cellar provider feature.
• Fork: clever-kubernetes-operator (MIT), as the skeleton of loam-operator (D185). What carries over is small, a kube-rs controller registry, finalizers, event recording, metrics and a hardened Helm chart, because its CRDs provision Clever Cloud add-ons through Clever's API rather than running workloads in the cluster.
• Adopt as unmodified tools or services, only when deploying on Clever Cloud: terraform-provider-clevercloud and karpenter-provider-clever-cloud (Apache-2.0), plus biscuit-cli (BSD-3-Clause) as a developer tool.
• Reject as the edge: Sōzu (AGPL-3.0) and sozu-gateway. Envoy stays the edge, and Pingora is the library if a Rust L7 component is ever needed.
• Reference only: clever-tools (CLI UX), clever-components (UI), kawa, nlrs, poule, simple-vmm, cellar-migration, kaniop (AGPL-3.0, not Clever's).
• Not published: Clever Cloud's hypervisor, deployer, log pipeline and Materia KV engine. No open repository was found in CleverCloud, sozu-proxy or eclipse-biscuit; only simple-vmm, a 2021 example VMM, touches virtualization.
• GitOps: Argo CD (D186), with an app-of-apps and sync waves over one umbrella chart. RustFS is the object store (D178), TiDB Operator v2 deploys PD and TiKV (D179), and the forked operator reconciles Loam and Function resources.

2. Inventory (D187)Copy Anchor Link

Searched on 2026-09-29: every public, non-archived, non-fork repository of github.com/CleverCloud (about 250), github.com/sozu-proxy and github.com/eclipse-biscuit (where Biscuit moved from Clever Cloud), and biscuit-auth. About 150 of Clever's repositories are *-example deployment samples, most with no license. They are grouped as one row. Everything with a plausible place in Loam's GitOps, runtime, edge or auth stack is listed individually.

2.1 GitOps, operators and infrastructureCopy Anchor Link

2.2 CLI, SDKs and UICopy Anchor Link

2.3 Edge and proxy (Sōzu family)Copy Anchor Link

2.4 Authorization (Biscuit)Copy Anchor Link

2.5 Messaging, metrics, logs and storage toolingCopy Anchor Link

Not open source. Clever Cloud's hypervisor and VM images, its deployer, its log and metrics pipeline and the Materia KV engine are not published. Only clients and demos for Materia exist (mkv-*, MIT). Nothing from those systems can be adopted.

3. The edge: Sōzu vs Envoy vs Pingora or RiverCopy Anchor Link

Verdict. Envoy stays the edge (D184). Sōzu is well engineered and fast, and its hot reconfiguration is appealing. But D176 puts HTTP/3 and gRPC in phase 1, and Sōzu has neither HTTP/3 nor gRPC routes. It would also bring an AGPL-3.0 binary into Loam's default install. As an unmodified separate process that is allowed by the rule, but every distributor of the self-hosted bundle would then carry the AGPL source-offer duty for it. Pingora is the right tool if Loam ever needs a Rust L7 component of its own (for example, a gateway that terminates HTTP/3 next to the supervisor). River is not an option while it is stalled.

4. Biscuit vs OpenFGA + OIDC vs the sandbox tokens (D182, D188)Copy Anchor Link

These mechanisms answer different questions, so the comparison is about which job each one does.

Recommendation.

1. OpenFGA stays the authority for every decision on Loam data (the Authorizer, D66). Biscuit Datalog is not allowed to become a second policy store. Tokens carry only narrowing facts: tenant, namespace, function, invocation, operations and expiry.
2. JWTs stay for external clients (§19 §5.3). They are universal, and MCP clients and OIDC federation expect them.
3. Biscuit is used inside the runtime (D188), where offline attenuation pays off:
   • The control plane issues each node supervisor a Biscuit whose authority allows minting for the tenants scheduled on that node, with a short TTL and regular renewal.
   • For each invocation, the supervisor appends a block that narrows the token to one tenant, function, version and invocation id and an expiry of at most the invocation's wall limit. It adds a further block per call with the allowed operations (state:get, pubsub:publish, retrieval:query, …). No round trip is needed per invocation.
   • loam-dapr and the gateway verify the chain locally with the control plane's public key, take the namespace from the token, then ask OpenFGA.
   • A function that calls a sub-function passes an attenuated copy. This is §19 §5.2's "vended tokens can only attenuate", enforced cryptographically rather than by the token endpoint.
   • Suspension of an agent (§19 §5.4) adds its revocation ids to a TiKV-backed deny list that verifiers cache.
4. Risk: two token formats. The gateway accepts JWTs from outside and Biscuits from sandboxes, and never the reverse (a Biscuit is not valid on public routes). Q-RT-10 asks whether §19's vending flow should also switch to Biscuit, so that there is one attenuation mechanism.

5. The forked operator and the ObjectStoreProvider trait (D178, D185)Copy Anchor Link

Fork plan. Fork CleverCloud/clever-kubernetes-operator to dina-kar/loam-operator and keep the MIT LICENSE with Clever Cloud's copyright plus a NOTICE. Rename the API group from api.clever-cloud.com to Loam's (placeholder loam.<domain>/v1alpha1, Q-RT-8). Keep crates/core, svc/k8s, svc/http and the Helm chart. Delete svc/clevercloud and the add-on CRDs. Add:

/// Implemented by `rustfs` (default), `s3` (AWS, R2, GCS interop, any S3) and `cellar` (feature `provider-cellar`).
#[async_trait::async_trait]
pub trait ObjectStoreProvider: Send + Sync {
    fn kind(&self) -> &'static str;
    /// What the backend supports; Loam refuses a provider that lacks conditional writes (D1's WAL needs them).
    fn capabilities(&self) -> ProviderCapabilities; // if_none_match_put, if_match_put, max_object, ...
    async fn ensure_bucket(&self, spec: &BucketSpec) -> Result<BucketRef, ProviderError>;
    async fn issue_credentials(&self, bucket: &BucketRef, scope: &CredentialScope) -> Result<S3Credentials, ProviderError>;
    async fn revoke_credentials(&self, bucket: &BucketRef, key_id: &str) -> Result<(), ProviderError>;
    /// The `object_store` configuration Loam's roles receive (endpoint, region, path-style, TLS).
    fn store_config(&self, bucket: &BucketRef) -> ObjectStoreConfig;
}

• rustfs: RustFS is deployed by its upstream Helm chart (rustfs/rustfs, helm/rustfs, Apache-2.0, 1.0.0 on 2026-09-16), and the provider creates buckets and access keys through RustFS's S3 and admin APIs (verify admin API shape).
• s3: static credentials or IRSA / workload identity; no bucket creation unless allowed.
• cellar: buckets and keys through clevercloud-sdk; data over S3. Whether Cellar honours If-None-Match and If-Match on PUT is unverified, and it decides whether Cellar can hold Loam's WAL or only static assets (Q-RT-9).

6. GitOps layout (D186)Copy Anchor Link

6.1 Argo CD or FluxCopy Anchor Link

Both are Apache-2.0 and CNCF-graduated. The latest releases are Argo CD v3.5.3 (2026-09-14; v3.6.0-rc1 on 2026-09-16) and Flux v2.9.5 (2026-08-31). Argo CD is chosen, for these reasons:

1. The draft and this design are written in sync waves. Argo CD's argocd.argoproj.io/sync-wave orders resources inside an Application and, in an app-of-apps, orders the child Applications. Flux expresses the same thing with dependsOn between Kustomizations and HelmReleases. That works, but it is a different model from the one the owner wrote.
2. Custom health checks for CRDs. Waves only wait for healthy resources. Argo CD's Lua resource.customizations.health lets the TiDB Operator v2 groups (PDGroup, TiKVGroup), RustFS, Loam and Function report real readiness.
3. Hub and spoke for Loam cloud and BYOC. One Argo CD with ApplicationSets (cluster generator) can manage many clusters, and it has a UI for operators.

Costs, stated plainly. Argo CD is heavier than Flux: application controller, repo server, Redis, server and optional Dex (verify RSS on k3d). That matters on the dev machine and in small BYOC clusters. And since Argo CD 1.8, an Application's own health is not assessed by default. Without adding the documented argoproj.io/Application health customization, a parent's waves do not wait for child Applications to become healthy (verify against the v3.5 docs). The umbrella chart has no Argo-specific templates, so Flux can consume it unchanged if a BYOC customer requires Flux (Q-RT-11).

6.2 Repository layoutCopy Anchor Link

deploy/
  helm/loam-stack/                 # umbrella chart (Chart.yaml dependencies, all toggleable)
    charts/                        # rustfs (upstream), tidb-operator (upstream), loam-operator (fork),
                                   # resonate, loam-dapr, loam-gateway, envoy-gateway, dapr (long tail, off by default),
                                   # runtime (supervisor DaemonSet, RuntimeClasses)
    values.yaml
  gitops/
    bootstrap/argocd/              # Argo CD install + argocd-cm health customizations (Application, PDGroup, TiKVGroup, Loam, Function)
    root.yaml                      # the root Application (app-of-apps)
    apps/                          # one Application per wave, each rendering loam-stack with one component enabled
    envs/{dev-k3d,selfhosted,clever-cke,cloud}/values.yaml
  infra/clever-cke/                # Terraform (terraform-provider-clevercloud) — outside Argo CD

6.3 Sync wavesCopy Anchor Link

envs/dev-k3d sets one replica everywhere and small TiKV memory (§20's playground peaked at 3.2 GB RSS for PD, TiKV and TiDB). It can use RustFS in single-node mode.

6.4 TiDB Operator v2 check (D179)Copy Anchor Link

• Status. main is v2 ("The main branch is now the default branch and hosts the v2 version", README). v2.0.0 was released on 2025-12-18 and v2.0.1 on 2026-03-26 (not pre-releases). v2.1.0-beta.6 and v2.2.0-alpha.11 (2026-09-16) are pre-releases. v1.6.6 (2026-08-12, release-1.x) is v1's latest. The v2 CRDs are still core.pingcap.com/v1alpha1 even in the GA line, which is an API-stability risk to pin against.
• PD + TiKV only. v2 splits a cluster into a Cluster and one resource per component group. examples/pdms deploys Cluster, PDGroup, TSOGroup, SchedulingGroup and TiKVGroup with no TiDBGroup, and examples/basic puts TiDB in its own file (03-tidb.yaml). So a PD-and-TiKV cluster is a supported shape. In v1, TidbClusterSpec.TiDB is an optional pointer (json:"tidb,omitempty", release-1.6 types.go), but whether v1's controllers run cleanly without TiDB was not checked (verify).
• Still to verify in F0: that storage.api-version = 2 passes through TiKVGroup.spec.template.spec.config; that PD keyspace pre-allocation (D122) can be set in PDGroup config; and Q33's question of whether keyspace is accepted for classic clusters.

7. RisksCopy Anchor Link

8. Open questionsCopy Anchor Link

9. SourcesCopy Anchor Link

Read on 2026-09-29 through the GitHub API (repos/<owner>/<repo>, /license, /commits, /releases) and the files named.

• CleverCloud org listing (all public, non-archived, non-fork repositories); sozu-proxy, eclipse-biscuit and biscuit-auth org listings.
• CleverCloud/clever-kubernetes-operator: LICENSE (MIT), README.md, Cargo.toml (v0.8.0, workspace crates/core, crates/operator), crates/operator/Cargo.toml (kube 3.1, clevercloud-sdk 1.0.1), crates/core/src/{controller,registry,strategy}.rs, crates/operator/src/svc/crd/postgresql.rs (group = "api.clever-cloud.com"), deployments/kubernetes/helm/.
• sozu-proxy/sozu: LICENSE (AGPL-3.0), README.md, lib/Cargo.toml (AGPL-3.0), command/Cargo.toml (LGPL-3.0), bin/Cargo.toml (AGPL-3.0), doc/. CleverCloud/sozu-gateway: README.md, docs/features.md, Cargo.toml (sozu-command-lib = "=2.2.1"). CleverCloud/sozu-client: Cargo.toml.
• eclipse-biscuit/biscuit-rust: README.md, biscuit-auth/Cargo.toml (6.0.0, Apache-2.0), biscuit-auth/LICENSE. eclipse-biscuit/biscuit LICENSE. eclipse-biscuit/biscuit-cli LICENSE (BSD-3-Clause).
• CleverCloud/karpenter-provider-clever-cloud README.md; CleverCloud/clever-tools README.md; CleverCloud/clever-components README.md; LICENSE files of every repository in §2.
• pando85/kaniop LICENSE.md (AGPL-3.0).
• pingcap/tidb-operator: README.md, examples/{basic,pdms}/, api/core/v1alpha1/, releases and tags (v2.0.0, v2.0.1, v2.1.0-beta.6, v2.2.0-alpha.11, v1.6.6), release-1.6:pkg/apis/pingcap/v1alpha1/types.go.
• rustfs/rustfs: LICENSE (Apache-2.0), helm/, release 1.0.0. cloudflare/pingora (Apache-2.0, 0.9.0), memorysafety/river (Apache-2.0/MIT, v0.5.0, last commit 2024-09-06), envoyproxy/envoy (Apache-2.0, v1.39.1), argoproj/argo-cd (Apache-2.0, v3.5.3), fluxcd/flux2 (Apache-2.0, v2.9.5).
• Loam: §10 §1, §18, §19 §5, §20 §9 and §15, §21, §22 §13b; D1, D61, D66, D67, D71, D72, D73, D74, D122, D-SC-16; Q33.