feat(sandbox): add Kubernetes user namespace isolation #983
Open
mrunalp wants to merge 8 commits intoNVIDIA:mainfrom
Open
feat(sandbox): add Kubernetes user namespace isolation #983mrunalp wants to merge 8 commits intoNVIDIA:mainfrom
mrunalp wants to merge 8 commits intoNVIDIA:mainfrom
Conversation
Add opt-in support for Kubernetes user namespace isolation on sandbox pods. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced, providing defense-in-depth for the supervisor process. Configuration is two-layered: a cluster-wide default via OPENSHELL_ENABLE_USER_NAMESPACES (default false) and a per-sandbox override via the new `user_namespaces` field on SandboxTemplate. When user namespaces are active, the pod security context is extended with SETUID, SETGID, and DAC_READ_SEARCH capabilities to match the bounding-set requirements inside a user namespace.
…ation Address review feedback: - Emit a warning when GPU and user namespaces are both active on a sandbox, matching the risk mitigation documented in the architecture. - Add a User Namespace Isolation section to the security best practices doc covering prerequisites, Helm config, and GPU caveats.
Enable OPENSHELL_ENABLE_USER_NAMESPACES on the gateway, create a sandbox, and verify the pod spec has hostUsers: false and the extended capability set (SETUID, SETGID, DAC_READ_SEARCH) required for user namespace operation.
Explain why user namespaces fail in Docker-in-Docker dev clusters: nested overlayfs does not support MOUNT_ATTR_IDMAP. Document where user namespaces work (bare-metal, VM-based, managed K8s) and where they don't (kind, k3s-in-Docker, old kernels, NFS).
Step-by-step reproduction guide for deploying OpenShell with user namespace isolation on OpenShift. Covers TLS setup, image push to the internal registry, supervisor binary distribution via DaemonSet with SELinux labeling, Helm deploy, CLI configuration, and end-to-end verification.
Document that user namespaces can be deployed via Helm on any K8s 1.33+ cluster and link to the OCP testing guide. Update verification section to reflect successful end-to-end validation on OCP 4.22.
Use an explicit sandbox name and deterministic cleanup so the user namespace e2e test cannot inspect the wrong sandbox on a busy cluster. Align the documented Kubernetes requirements with the tested rollout story and save the pre-PR review notes for sharing.
Drop hardcoded binary paths and KUBECONFIG variable assignments. Readers are expected to have kubectl and helm on their PATH.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in Kubernetes user namespace isolation for sandbox pods. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced, providing defense-in-depth against container escape vulnerabilities.
OPENSHELL_ENABLE_USER_NAMESPACES/ Helmserver.enableUserNamespaces(default off), with per-sandbox override viaSandboxTemplate.user_namespacesin the APISETUID,SETGID,DAC_READ_SEARCH) when user namespaces are active, matching the Podman driver's approachDirectoryOrCreatetoDirectoryRelated Issue
Fixes #982
Changes
Proto & config (3 files):
proto/openshell.proto— addoptional bool user_namespacestoSandboxTemplatecrates/openshell-core/src/config.rs— addenable_user_namespacestoConfigcrates/openshell-driver-kubernetes/src/config.rs— add field toKubernetesComputeConfigServer & driver wiring (4 files):
crates/openshell-server/src/cli.rs— add--enable-user-namespacesCLI argcrates/openshell-server/src/lib.rs— pass config to K8s drivercrates/openshell-server/src/compute/mod.rs— translateuser_namespaces→host_usersinbuild_platform_configcrates/openshell-driver-kubernetes/src/main.rs— add CLI arg for standalone driverK8s driver (1 file):
crates/openshell-driver-kubernetes/src/driver.rs— addplatform_config_boolhelper, setspec.hostUsers: false, extend capabilities conditionally, change hostPath type toDirectory, GPU warning, unit testsHelm (2 files):
deploy/helm/openshell/values.yaml— addenableUserNamespaces: falsedeploy/helm/openshell/templates/statefulset.yaml— wire env varDocs (4 files):
docs/security/best-practices.mdx— add User Namespace Isolation sectionarchitecture/kubernetes-user-namespaces.md— design doc with DinD limitation and Helm deployment guidearchitecture/kubernetes-user-namespaces-ocp-testing.md— step-by-step OCP reproduction guideTests (1 file):
e2e/rust/tests/user_namespaces.rs— e2e pod spec verificationTesting
cargo test -p openshell-driver-kubernetes— 26 tests pass (8 new for user namespaces)cargo test -p openshell-server --lib— server-side inversion test passesmise run e2e— 31/32 pass (1 pre-existing flaky failure unrelated to this change)user_namespacestest passes againstmise run cluster(verifies pod spec)0 → 3285581824)0 → 2437873664)MOUNT_ATTR_IDMAPfails on nested overlayfs (documented)Checklist
docs/security/best-practices.mdx)Open Items