The idea is to remove the need for cloning the entire runc binary by
replacing the final execve() call of the container process with an
execve() call to a clone of a small C binary which just does an execve()
of its arguments.
This provides similar protection against CVE-2019-5736 but without
requiring a >10MB binary copy for each "runc init". When compiled with
musl, runc-dmz is 13kB (though unfortunately with glibc, it is 1.1MB
which is still quite large).
It should be noted that there is still a window where the container
processes could get access to the host runc binary, but because we set
ourselves as non-dumpable the container would need CAP_SYS_PTRACE (which
is not enabled by default in Docker) in order to get around the
proc_fd_access_allowed() checks. In addition, since Linux 4.10[1] the
kernel blocks access entirely for user namespaced containers in this
scenario. For those cases we cannot use runc-dmz, but most containers
won't have this issue.
This new runc-dmz binary can be opted out of at compile time by setting
the "runc_nodmz" buildtag, and at runtime by setting the RUNC_DMZ=legacy
environment variable. In both cases, runc will fall back to the classic
/proc/self/exe-based cloning trick. If /proc/self/exe is already a
sealed memfd (namely if the user is using contrib/cmd/memfd-bind to
create a persistent sealed memfd for runc), neither runc-dmz nor
/proc/self/exe cloning will be used because they are not necessary.
[1]: https://github.com/torvalds/linux/commit/bfedb589252c01fa505ac9f6f2a3d5d68d707ef4
Co-authored-by: lifubang <lifubang@acmcoder.com>
Signed-off-by: lifubang <lifubang@acmcoder.com>
[cyphar: address various review nits]
[cyphar: fix runc-dmz cross-compilation]
[cyphar: embed runc-dmz into runc binary and clone in Go code]
[cyphar: make runc-dmz optional, with fallback to /proc/self/exe cloning]
[cyphar: do not use runc-dmz when the container has certain privs]
Co-authored-by: Aleksa Sarai <cyphar@cyphar.com>
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Previously, if /var/run was mounted noexec, our cloned binary logic
would not work if memfd_create(2) was not available because we would try
to exec a binary that is on a noexec filesystem.
We cannot guarantee there will be an executable filesystem on the system
(other than mounting one ourselves, which would cause a bunch of other
headaches) but we can at least try the obvious options (/tmp, /bin, and
/). If none of these work, we will have to fail.
Reported-by: lifubang <lifubang@acmcoder.com>
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
This allow us to remove the amount of C code in runc quite
substantially, as well as removing a whole execve(2) from the nsexec
path because we no longer spawn "runc init" only to re-exec "runc init"
after doing the clone.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
The runtime spec now allows relative mount dst paths, so remove the
comment saying we will switch this to an error later and change the
error messages to reflect that.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
From the Go specification:
"1. For a nil slice, the number of iterations is 0." [1]
Therefore, an additional nil check for before the loop is unnecessary.
[1]: https://go.dev/ref/spec#For_range
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Burstable CFS controller is introduced in Linux 5.14. This helps with
parallel workloads that might be bursty. They can get throttled even
when their average utilization is under quota. And they may be latency
sensitive at the same time so that throttling them is undesired.
This feature borrows time now against the future underrun, at the cost
of increased interference against the other system users, by introducing
cfs_burst_us into CFS bandwidth control to enact the cap on unused
bandwidth accumulation, which will then used additionally for burst.
The patch adds the support/control for CFS bandwidth burst.
runtime-spec: https://github.com/opencontainers/runtime-spec/pull/1120
Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com>
Co-authored-by: Nadeshiko Manju <me@manjusaka.me>
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
This makes libcontainer/userns self-dependent, largely returning to
the original implementation from lxc. The `uiMapInUserNS` is kept as
a separate function for unit-testing and fuzzing.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If the container does not have own mount namespace configured (i.e. it
shares the mount namespace with the host), its "prestart" (obsoleted)
and "createRuntime" hooks are called twice, and its cgroups and Intel
RDT settings are also applied twice.
The code being removed was originally added by commit 2f2764984 ("Move
pre-start hooks after container mounts", Feb 17 2016). At that time,
the syncParentHooks() was called from setupRootfs(), which was only
used when the container config has mount namespace (NEWNS) enabled.
Later, commit 244c9fc426 ("*: console rewrite", Jun 4 2016) spli
the relevant part of setupRootfs() into prepareRootfs(). It was still
called conditionally (only if mount namespace was enabled).
Finally, commit 91ca331474 ("chroot when no mount namespaces is
provided", Jan 25 2018) removed the above condition, meaning
prepareRootfs(), and thus syncParentHooks(), is now called for any
container.
Meaning, the special case for when mount namespace is not enabled is no
longer needed.
Remove it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
When a hook has failed, the error message looks like this:
> error running hook: error running hook #1: exit status 1, stdout: ...
The two problems here are:
1. it is impossible to know what kind of hook it was;
2. "error running hook" stuttering;
Change that to
> error running createContainer hook #1: exit status 1, stdout: ...
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
golangci-lint v1.54.2 comes with errorlint v1.4.4, which contains
the fix [1] whitelisting all errno comparisons for errors coming from
x/sys/unix.
Thus, these annotations are no longer necessary. Hooray!
[1] https://github.com/polyfloyd/go-errorlint/pull/47
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In the runc state JSON we always use snake_case. This is a no-op change,
but it will cause any existing container state files to be incorrectly
parsed. Luckily, commit fbf183c6f8 ("Add uid and gid mappings to
mounts") has never been in a runc release so we can change this before a
1.2.z release.
Fixes: fbf183c6f8 ("Add uid and gid mappings to mounts")
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
The original implementation of cgroupns had additional synchronisation
to "ensure" that the process is in the correct cgroup before unsharing
the cgroupns. This behaviour was actually never necessary, and after
commit 5110bd2fc0 ("nsenter: remove cgroupns sync mechanism") there is
no synchronisation at all, meaning that CLONE_NEWCGROUP should not get
any special treatment.
Fixes: 5110bd2fc0 ("nsenter: remove cgroupns sync mechanism")
Fixes: df3fa115f9 ("Add support for cgroup namespace")
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
The kernel ignores these arguments, and passing them can lead to
confusing error messages (the old source is irrelevant for MS_REMOUNT),
as well as causing issues for a future patch where we switch to
move_mount(2).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
*os.File is correctly tracked by the garbage collector, and there's no
need to use raw file descriptors for this code.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
This includes quite a few cleanups and improvements to the way we do
synchronisation. The core behaviour is unchanged, but switching to
embedding json.RawMessage into the synchronisation structure will allow
us to do more complicated synchronisation operations in future patches.
The file descriptor passing through the synchronisation system feature
will be used as part of the idmapped-mount and bind-mount-source
features when switching that code to use the new mount API outside of
nsexec.c.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The code in this function became quite complicated and not entirely
correct over time. As a result, if an error is returned from parseSync,
it might end up stuck waiting for the child to finish.
1. Let's not wait() for the child twice. We already do it in the
defer statement (call p.terminate()) when we are returning an error.
2. Remove sentResume and sentRun since we do not want to check if
these were sent or not. Instead, introduce and check seenProcReady, as
procReady is always expected from runc init.
3. Eliminate the possibility to wrap nil as an error.
4. Make sure we always call shutdown on the sync socket, and do not let
shutdown error shadow the ierr.
This fixes the issue of stuck `runc runc` with the optimization patch
(sending procSeccompDone earlier) applied.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
These are not exhaustive, but at least confirm that the feature is not
obviously broken (we correctly set the time offsets).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Fix up a few things that were flagged in the review of the original
timens PR, namely around error handling and validation.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
The idea was to make them strict on dest path from the beginning for
idmap mounts, as runc would do that for all mounts in the future. But
that is causing too many problems.
For now, let's just allow relative paths for idmap mounts too. It just
seems safer.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This reverts commit 881e92a3fd and adjust
the code so the idmap validations are strict.
We now only throw a warning and the container is started just fine.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This commit does two things:
1. Consolidate StartInitialization calling logic into Init().
2. Fix init error handling logic.
The main issues at hand are:
- the "unable to convert _LIBCONTAINER_INITPIPE" error from
StartInitialization is never shown;
- errors from WriteSync and WriteJSON are never shown;
- the StartInit calling code is triplicated;
- using panic is questionable.
Generally, our goals are:
- if there's any error, do our best to show it;
- but only show each error once;
- simplify the code, unify init implementations.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Currently, logrus is used from the Go part of runc init, mostly for a
few debug messages (see setns_init_linux.go and standard_init_linux.go),
and a single warning (see rootfs_linux.go).
This means logrus is part of init implementation, and thus, its setup
belongs to StartInitialization().
Move the code there. As a nice side effect, now we don't have to convert
_LIBCONTAINER_LOGPIPE twice.
Note that since this initialization is now also called from libct/int
tests, which do not set _LIBCONTAINER_LOGLEVEL, let's make
_LIBCONTAINER_LOGLEVEL optional.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
With the new vm.memfd_noexec sysctl, we need to make sure we explicitly
request MFD_EXEC, otherwise an admin could inadvertently break
containers in a somewhat-annoying-to-debug fashion.
It should be noted that vm.memfd_noexec=2 is broken on Linux 6.4
(MFD_EXEC works even in the most restrictive mode) and the most severe
breakage is going to be fixed in Linux 6.6[1].
[1]: https://lore.kernel.org/20230705063315.3680666-2-jeffxu@google.com/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
While the ro-bind-mount trick did eliminate the memory overhead of
copying the runc binary for each "runc init" invocation, on machines
with very significant container churn, creating a temporary mount
namespace on every container invocation can trigger severe lock
contention on namespace_sem that makes containers fail to spawn.
The only reason we added bindfd in commit 16612d74de ("nsenter:
cloned_binary: try to ro-bind /proc/self/exe before copying") was due to
a Kubernetes e2e test failure where they had a ridiculously small memory
limit. It seems incredibly unlikely that real workloads are running
without 10MB to spare for the very short time that runc is interacting
with the container.
In addition, since the original cloned_binary implementation, cgroupv2
is now almost universally used on modern systems. Unlike cgroupv1, the
cgroupv2 memcg implementation does not migrate memory usage when
processes change cgroups (even cgroupv1 only did this if you had
memory.move_charge_at_immigrate enabled). In addition, because we do the
/proc/self/exe clone before synchronising the bootstrap data read, we
are guaranteed to do the clone before "runc init" is moved into the
container cgroup -- meaning that the memory used by the /proc/self/exe
clone is charged against the root cgroup, and thus container workloads
should not be affected at all with memfd cloning.
The long-term fix for this problem is to block the /proc/self/exe
re-opening attack entirely in-kernel, which is something I'm working
on[1]. Though it should also be noted that because the memfd is
completely separate to the host binary, even attacks like Dirty COW
against the runc binary can be defended against with the memfd approach.
Of course, once we have in-kernel protection against the /proc/self/exe
re-opening attack, we won't have that protection anymore...
[1]: https://lwn.net/Articles/934460/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
As we now log the log file name in logCriuErrors.
While at it, there is no need to use var.String() with %s as it is done
by the runtime.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
When criu fails, it does not give us much context to understand what
was the cause of an error -- for that, we need to take a look into its
log file.
This is somewhat complicated to do (as you can see in parts of
checkpoint.bats removed by this commit), and not very user-friendly.
Add a function to find and log errors from criu logs, together with some
preceding context, in case either checkpoint or restore has failed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
A few cases relied on the fact that systemd is used, and thus
/sys/fs/cgroup/user.slice is available.
Guess what, in case of "make unittest" it might not be.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Split the test into two -- for fs and systemd cgroup managers, and only
run the second one if systemd is available.
Prevents the following failure during `make unittest`:
> === RUN TestNilResources
> manager_test.go:27: systemd not running on this host, cannot use systemd cgroups manager
> --- FAIL: TestNilResources (0.22s)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
"time" namespace was introduced in Linux v5.6
support new time namespace to set boottime and monotonic time offset
Example runtime spec
"timeOffsets": {
"monotonic": {
"secs": 172800,
"nanosecs": 0
},
"boottime": {
"secs": 604800,
"nanosecs": 0
}
}
Signed-off-by: Chethan Suresh <chethan.suresh@sony.com>
This aligns v2 usage calculations more closely with v1.
Current node-level reporting for v1 vs v2 on the same
machine under similar load may differ by ~250-750Mi.
Also return usage as combined swap + memory usage, aligned
with v1 and non-root v2 cgroups.
`mem_cgroup_usage` in the kernel counts NR_FILE_PAGES
+ NR_ANON_MAPPED + `nr_swap_pages` (if swap enabled) [^0].
Using total - free results in higher "usage" numbers.
This is likely due to various types of reclaimable
memory technically counted as in use (e.g. inactive anon).
See also https://github.com/kubernetes/kubernetes/issues/118916 for more context
[^0]: https://github.com/torvalds/linux/blob/06c2afb862f9da8dc5efa4b6076a0e48c3fbaaa5/mm/memcontrol.c#L3673-L3680
Signed-off-by: Alexander Eldeib <alexeldeib@gmail.com>
close_range(2) is far more efficient than a readdir of /proc/self/fd and
then doing a syscall for each file descriptor.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
While testing this with old kernel versions and kernels that don't
support idmap mounts for some of the filesystems used by a container, I
realized we can throw a more clear errors.
Let's make it clear which syscall we are using, when it is not supported
and when if the fs doesn't support idmap mounts, which path it is.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Since systemd v240 (commit 8e8b5d2e6d91180a), one can use
/dev/{char,block}-MAJOR syntax to specify that all MAJOR:*
devices are allowed.
Use it, if available, since it's more straightforward, plus
we can skip somewhat expensive parsing of /proc/devices.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Currently bind mounts of filesystems with nodev, nosuid, noexec,
noatime, relatime, strictatime, nodiratime options set fail in rootless
mode if the same options are not set for the bind mount.
For ro filesystems this was resolved by #2570 by remounting again
with ro set.
Follow the same approach for nodev, nosuid, noexec, noatime, relatime,
strictatime, nodiratime but allow to revert back to the old behaviour
via the new `--no-mount-fallback` command line option.
Add a testcase to verify that bind mounts of filesystems with nodev,
nosuid, noexec, noatime options set work in rootless mode.
Add a testcase that mounts a nodev, nosuid, noexec, noatime filesystem
with a ro flag.
Add two further testcases that ensure that the above testcases would
fail if the `--no-mount-fallback` command line option is set.
* contrib/completions/bash/runc:
Add `--no-mount-fallback` command line option for bash completion.
* create.go:
Add `--no-mount-fallback` command line option.
* restore.go:
Add `--no-mount-fallback` command line option.
* run.go:
Add `--no-mount-fallback` command line option.
* libcontainer/configs/config.go:
Add `NoMountFallback` field to the `Config` struct to store
the command line option value.
* libcontainer/specconv/spec_linux.go:
Add `NoMountFallback` field to the `CreateOpts` struct to store
the command line option value and store it in the libcontainer
config.
* utils_linux.go:
Store the command line option value in the `CreateOpts` struct.
* libcontainer/rootfs_linux.go:
In case that `--no-mount-fallback` is not set try to remount the
bind filesystem again with the options nodev, nosuid, noexec,
noatime, relatime, strictatime or nodiratime if they are set on
the source filesystem.
* tests/integration/mounts_sshfs.bats:
Add testcases and rework sshfs setup to allow specifying
different mount options depending on the test case.
Signed-off-by: Ruediger Pluem <ruediger.pluem@vodafone.com>
We cannot have both srcFD and idMapFD set at the same time.
So, we can simplify this struct to only have one field which is used a srcFD
most of the time and as idMapFD when we do an id map mount.
Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
Aleksa Sarai