This allow us to remove the amount of C code in runc quite
substantially, as well as removing a whole execve(2) from the nsexec
path because we no longer spawn "runc init" only to re-exec "runc init"
after doing the clone.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
The original implementation of cgroupns had additional synchronisation
to "ensure" that the process is in the correct cgroup before unsharing
the cgroupns. This behaviour was actually never necessary, and after
commit 5110bd2fc0 ("nsenter: remove cgroupns sync mechanism") there is
no synchronisation at all, meaning that CLONE_NEWCGROUP should not get
any special treatment.
Fixes: 5110bd2fc0 ("nsenter: remove cgroupns sync mechanism")
Fixes: df3fa115f9 ("Add support for cgroup namespace")
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Fix up a few things that were flagged in the review of the original
timens PR, namely around error handling and validation.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
With the new vm.memfd_noexec sysctl, we need to make sure we explicitly
request MFD_EXEC, otherwise an admin could inadvertently break
containers in a somewhat-annoying-to-debug fashion.
It should be noted that vm.memfd_noexec=2 is broken on Linux 6.4
(MFD_EXEC works even in the most restrictive mode) and the most severe
breakage is going to be fixed in Linux 6.6[1].
[1]: https://lore.kernel.org/20230705063315.3680666-2-jeffxu@google.com/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
While the ro-bind-mount trick did eliminate the memory overhead of
copying the runc binary for each "runc init" invocation, on machines
with very significant container churn, creating a temporary mount
namespace on every container invocation can trigger severe lock
contention on namespace_sem that makes containers fail to spawn.
The only reason we added bindfd in commit 16612d74de ("nsenter:
cloned_binary: try to ro-bind /proc/self/exe before copying") was due to
a Kubernetes e2e test failure where they had a ridiculously small memory
limit. It seems incredibly unlikely that real workloads are running
without 10MB to spare for the very short time that runc is interacting
with the container.
In addition, since the original cloned_binary implementation, cgroupv2
is now almost universally used on modern systems. Unlike cgroupv1, the
cgroupv2 memcg implementation does not migrate memory usage when
processes change cgroups (even cgroupv1 only did this if you had
memory.move_charge_at_immigrate enabled). In addition, because we do the
/proc/self/exe clone before synchronising the bootstrap data read, we
are guaranteed to do the clone before "runc init" is moved into the
container cgroup -- meaning that the memory used by the /proc/self/exe
clone is charged against the root cgroup, and thus container workloads
should not be affected at all with memfd cloning.
The long-term fix for this problem is to block the /proc/self/exe
re-opening attack entirely in-kernel, which is something I'm working
on[1]. Though it should also be noted that because the memfd is
completely separate to the host binary, even attacks like Dirty COW
against the runc binary can be defended against with the memfd approach.
Of course, once we have in-kernel protection against the /proc/self/exe
re-opening attack, we won't have that protection anymore...
[1]: https://lwn.net/Articles/934460/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
"time" namespace was introduced in Linux v5.6
support new time namespace to set boottime and monotonic time offset
Example runtime spec
"timeOffsets": {
"monotonic": {
"secs": 172800,
"nanosecs": 0
},
"boottime": {
"secs": 604800,
"nanosecs": 0
}
}
Signed-off-by: Chethan Suresh <chethan.suresh@sony.com>
While testing this with old kernel versions and kernels that don't
support idmap mounts for some of the filesystems used by a container, I
realized we can throw a more clear errors.
Let's make it clear which syscall we are using, when it is not supported
and when if the fs doesn't support idmap mounts, which path it is.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This commit adds support for idmap mounts as specified in the runtime-spec.
We open the idmap source paths and call mount_setattr() in runc PARENT,
as we need privileges in the init userns for that, and then sends the
fds to the child process. For this fd passing we use the same mechanism
used in other parts of thecode, the _LIBCONTAINER_ env vars.
The mount is finished (unix.MoveMount) from go code, inside the userns,
so we reuse all the prepareBindMount() security checks and the remount
logic for some flags too.
This commit only supports idmap mounts when userns are used AND the mappings
are the same specified for the userns mapping. This limitation is to
simplify the initial implementation, as all our users so far only need
this, and we can avoid sending over netlink the mappings, creating a
userns with this custom mapping, etc. Future PRs will remove this
limitation.
Co-authored-by: Francis Laniel <flaniel@linux.microsoft.com>
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
We add idmap.h with the needed includes and defines in case the system
headers don't have the definition for the idmap syscalls we need.
Future patches will use these helpers.
Co-authored-by: Francis Laniel <flaniel@linux.microsoft.com>
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Processes can watch /proc/self/mounts or /mountinfo, and the kernel
will notify them whenever the namespace's mount table is modified. The
notified process still needs to read and parse the mountinfo to
determine what changed once notified. Many such processes, including
udisksd and SystemD < v248, make no attempt to rate-limit their
mountinfo notifications. This tends to not be a problem on many systems,
where mount tables are small and mounting and unmounting is uncommon.
Every runC exec which successfully uses the try_bindfd container-escape
mitigation performs two mount()s and one umount() in the host's mount
namespace, causing any mount-watching processes to wake up and parse the
mountinfo file three times in a row. Consequently, using 'exec' health
checks on containers has a larger-than-expected impact on system load
when such mount-watching daemons are running. Furthermore, the size of
the mount table in the host's mount namespace tends to be proportional
to the number of OCI containers as a unique mount is required for the
rootfs of each container. Therefore, on systems with mount-watching
processes, the system load increases *quadratically* with the number of
running containers which use health checks!
Prevent runC from incidentally modifying the host's mount namespace for
container-escape mitigations by setting up the mitigation in a temporary
mount namespace.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Modify receive_fd() and send_fd() so they can be more readily reused in
cloned_binary.c. Change receive_fd() to have a single responsibility:
receiving and returning a single file descriptor over a UNIX domain
socket. Make send_fd() useable in precarious execution contexts such as
a clone(CLONE_VFORK|CLONE_VM) "thread" where allocating heap memory or
calling exit() would be dangerous.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Older kernels may return EINVAL on unshare when a process is reading
runc's /proc/$PID/status or /proc/$PID/maps. This was fixed by kernel
commit 12c641ab8270f ("unshare: Unsharing a thread does not require
unsharing a vm") in Linuxt v4.3.
For CentOS 7, the fix was backported to CentOS 7.7 (kernel 3.10.0-1062).
To work around this kernel bug, let's retry on EINVAL a few times.
Reported-by: zzyyzte <zhang.yu58@zte.com.cn>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
stage_2_pid is not yet assigned, so this kills the PID -1, but as
the sane_kill() wrapper is just a nop in that case. Just remove these
calls to kill stage_2_pid before it is cloned/assigned.
I've checked by executing the error paths that no binary is left by mistake.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
First, check if strdup() fails and error out.
While we are there, the else case was missing brackets, as we only need
to check ret in the else case. Fix that too
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Remove upper bound in integer sanity check
to not restrict the number of socket-activated
sockets passed in.
Closes#3488
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
After adding some debug info to cloned_binary.c I found out that
is_self_cloned() is not working right when runc binary is on tmpfs,
resulting in one extra re-exec of runc.
With some added debug:
$ mkdir bin
$ sudo mount -t tmpfs tmp bin
$ sudo cp runc bin
$ sudo ./bin/runc --debug exec xxx true
DEBU[0000] nsexec[763590]: => is_self_cloned
DEBU[0000] nsexec[763590]: got seals 1 (want 15)
DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 0
DEBU[0000] nsexec[763590]: try_bindfd: 5
DEBU[0000] nsexec[763590]: re-exec itself...
DEBU[0000] nsexec[763590]: => is_self_cloned
DEBU[0000] nsexec[763590]: got seals 1 (want 15)
DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 0
DEBU[0000] nsexec[763590]: try_bindfd: -1
DEBU[0000] nsexec[763590]: fallback to make_execfd: 5
DEBU[0000] nsexec[763590]: re-exec itself...
DEBU[0000] nsexec[763590]: => is_self_cloned
DEBU[0000] nsexec[763590]: got seals 15 (want 15)
DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 1
From the above, it is seen that
- `is_self_cloned` returns 0,
- `try_bindfd` is called and succeeds,
- runc re-execs itself,
- the second call to `is_self_cloned` returns 0 again (because GET_SEALS returns 1),
- runc falls back to `make_execfd`, and re-execs again,
- finally, the third `is_self_cloned` returns 1.
I guess that the code relied on the following (quoting fcntl(2)):
> Currently, file seals can be applied only to a file descriptor
> returned by memfd_create(2) (if the MFD_ALLOW_SEALING was employed).
> On other filesystems, all fcntl() operations that operate on seals
> will return EINVAL.
It looks like in case of a file on tmpfs it returns 1 (F_SEAL_SEAL).
With the fix:
DEBU[0000] nsexec[768367]: => is_self_cloned
DEBU[0000] nsexec[768367]: got seals 1 (want 15)
DEBU[0000] nsexec[768367]: no CLONED_BINARY_ENV
DEBU[0000] nsexec[768367]: <= is_self_cloned, is_cloned = 0
DEBU[0000] nsexec[768367]: try_bindfd: 5
DEBU[0000] nsexec[768367]: re-exec itself...
DEBU[0000] nsexec[768367]: => is_self_cloned
DEBU[0000] nsexec[768367]: got seals 1 (want 15)
DEBU[0000] nsexec[768367]: fstatfs says ro = 1
DEBU[0000] nsexec[768367]: fstat says nlink = 1
DEBU[0000] nsexec[768367]: <= is_self_cloned, is_cloned = 1
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit 4d1d6185ab added this
nsenter_unsupported.go file in order for nsenter to be a valid (but
empty, non-functional) Go package on unsupported platforms.
As a result, runc can be build successfully without CGO, which results
in a non-working and hard-to-debug binary (see issue 3330).
As the functionality of being able to compile a package which is
definitely not working is questionable, and I can't think of any use
cases, let's remove the file.
With this, runc can no longer be build without CGO:
[kir@kir-rhat runc]$ CGO_ENABLED=0 make runc
go build -trimpath "-buildmode=pie" -tags "seccomp" -ldflags "-X main.gitCommit=v1.0.0-452-g00f56786-dirty -X main.version=1.1.0-rc.1+dev " -o runc .
go build github.com/opencontainers/runc/libcontainer/nsenter: build constraints exclude all Go files in /home/kir/go/src/github.com/opencontainers/runc/libcontainer/nsenter
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The source of the bind mount might not be accessible in a different user
namespace because a component of the source path might not be traversed
under the users and groups mapped inside the user namespace. This caused
errors such as the following:
# time="2020-06-22T13:48:26Z" level=error msg="container_linux.go:367:
starting container process caused: process_linux.go:459:
container init caused: rootfs_linux.go:58:
mounting \"/tmp/busyboxtest/source-inaccessible/dir\"
to rootfs at \"/tmp/inaccessible\" caused:
stat /tmp/busyboxtest/source-inaccessible/dir: permission denied"
To solve this problem, this patch performs the following:
1. in nsexec.c, it opens the source path in the host userns (so we have
the right permissions to open it) but in the container mntns (so the
kernel cross mntns mount check let us mount it later:
https://github.com/torvalds/linux/blob/v5.8/fs/namespace.c#L2312).
2. in nsexec.c, it passes the file descriptors of the source to the
child process with SCM_RIGHTS.
3. In runc-init in Golang, it finishes the mounts while inside the
userns even without access to the some components of the source
paths.
Passing the fds with SCM_RIGHTS is necessary because once the child
process is in the container mntns, it is already in the container userns
so it cannot temporarily join the host mntns.
This patch uses the existing mechanism with _LIBCONTAINER_* environment
variables to pass the file descriptors from runc to runc init.
This patch uses the existing mechanism with the Netlink-style bootstrap
to pass information about the list of source mounts to nsexec.c.
Rootless containers don't use this bind mount sources fdpassing
mechanism because we can't setns() to the target mntns in a rootless
container (we don't have the privileges when we are in the host userns).
This patch takes care of using O_CLOEXEC on mount fds, and close them
early.
Fixes: #2484.
Signed-off-by: Alban Crequy <alban@kinvolk.io>
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
Co-authored-by: Rodrigo Campos <rodrigo@kinvolk.io>
Currently, if the log level is not set to e.g. "debug", runc init sends
some debug logs to the parent, which parses and discards it.
It is better to not send those in the first place.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The code already parses an environment variable into an integer twice,
and we're about to add a third one.
Factor it out to getenv_int().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This makes it possible to use bail() even if logging is not set up
(yet), so we don't have to think whether it's OK to use it or not.
In addition, this might help some unit tests that do not set log
forwarding.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
- add missing colons before error message;
- unify error messages after cmd.Start and cmd.Wait, so that they show
context and the error itself.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Instead of reading a single message, do read all the logs from the init,
and use DisallowUnknownFields for stricter checking.
While at it, use reapChildren to reap zombies (and add an extra check).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The test was not working since at least commit 64bb59f592
renamed pid to stage2_pid (or maybe even earlier), so the pid
was never received (i.e. pid.Pid was 0).
The problem was not caught because os.FindProcess never return an error
on Unix.
Factor out and fix pid decode function:
- use DisallowUnknownInput to get error if JSON will be changed;
- check pids to make sure they are valid
- and use unix.Wait4 to reap zombies.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Make sure we close all file descriptors at the end of the test.
2. Make sure we close child fds after the start.
3. Use newPipe for logs as well, for simplicity and uniformity.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Go 1.17 introduce this new (and better) way to specify build tags.
For more info, see https://golang.org/design/draft-gobuild.
As a way to seamlessly switch from old to new build tags, gofmt (and
gopls) from go 1.17 adds the new tags along with the old ones.
Later, when go < 1.17 is no longer supported, the old build tags
can be removed.
Now, as I started to use latest gopls (v0.7.1), it adds these tags
while I edit. Rather than to randomly add new build tags, I guess
it is better to do it once for all files.
Mind that previous commits removed some tags that were useless,
so this one only touches packages that can at least be built
on non-linux.
Brought to you by
go1.17 fmt ./...
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The two exceptions I had to add to codespellrc are:
- CLOS (used by intelrtd);
- creat (syscall name used in tests/integration/testdata/seccomp_*.json).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit 2bab4a5 resulted in a warning from gcc:
nsexec.c: In function ‘write_log’:
nsexec.c:171:2: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
171 | write(logfd, json, ret);
| ^~~~~~~~~~~~~~~~~~~~~~~
As there's nothing we can or want to do in case write fails,
let's just tell the compiler we're not going to use it.
Fixes: 2bab4a5
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The contents of the pointer returned on asprintf() error are undefined
i.e., it can be anything there. We set it to NULL on error so that
free() afterwards won't get a garbage pointer.
This patch applies the above to message and stage as well to be
consistent with what we do for json.
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
According to C standards, `size_t` is always an unsigned integer type.
Thus, checking unsigned expressions to be less than zero is not needed.
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
As reported in issue 3119, there is a race in nsexec logging
that can lead to garbled json received by log forwarder, which
complains about it with a "failed to decode" error.
This happens because dprintf (used since the very beginning of nsexec
logging introduced in commit ba3cabf932) relies on multiple write(2)
calls, and with additional logging added by 64bb59f592 a race is
possible between runc init parent and its children.
The fix is to prepare a string and write it using a single call to
write(2).
[v2: NULLify json on error from asprintf]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As pointed out in TODO item added by commit 64bb59f59, it is not
necessary to have a special sync mechanism for cgroupns, as the parent
adds runc init to cgroup way earlier (before sending nl bootstrap data.
This sync was added by commit df3fa115f9, which was also added a
second cgroup manager.Apply() call, later removed in commit
d1ba8e39f8. It seems the original author had the idea to wait for
that second Apply().
Fixes: df3fa115f9
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Don't ignore close(2) return code, rather bail if there is any
unexpected failures. By checking the close return code we make sure we
don't introduce the same bug (closing an already closed fd) I've fixed
in the previous patch.
As a side note, we are not handling in this patch when close(2) returns
EINTR and the go runtime, since go 1.14, sends SIGURG to preempt
goroutines. This should not happen here though, as nsenter is guaranteed
to be executed before the go runtime starts.
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
This was closed in the child[1], before calling clone_parent (so runc
INIT will have this fd closed too), there is no point closing it again.
This was not causing issues because we ignore the return code of
close(2) and no one was opening a new fd between both calls to close.
However, with the new patches that I'm working on (PR #2576), this
problem is no longer inocuos: we do open a new fd in that PR, sometimes
that fd is allocated between the two close(2) calls and, as the lowest
fd is allocated to the new fd, sometimes the second close ends up
incorrectly closing this new fd.
Before it was not a problem in practice, but it was incorrect
nevertheless.
This seems to be long standing bug, present since at least 2018
(a54316bae), when SYNC_GRANDCHILD was introduced.
[1]: https://github.com/opencontainers/runc/blob/5547b5774f71f75a088e7432fa961778750a0fbd/libcontainer/nsenter/nsexec.c#L888
Co-authored-by: Alban Crequy <alban@kinvolk.io>
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
Aleksa Sarai