(This is a cherry-pick of b999376fb237195265081a8b8ba3fd3bd6ef8c2c.)
While the ro-bind-mount trick did eliminate the memory overhead of
copying the runc binary for each "runc init" invocation, on machines
with very significant container churn, creating a temporary mount
namespace on every container invocation can trigger severe lock
contention on namespace_sem that makes containers fail to spawn.
The only reason we added bindfd in commit 16612d74de ("nsenter:
cloned_binary: try to ro-bind /proc/self/exe before copying") was due to
a Kubernetes e2e test failure where they had a ridiculously small memory
limit. It seems incredibly unlikely that real workloads are running
without 10MB to spare for the very short time that runc is interacting
with the container.
In addition, since the original cloned_binary implementation, cgroupv2
is now almost universally used on modern systems. Unlike cgroupv1, the
cgroupv2 memcg implementation does not migrate memory usage when
processes change cgroups (even cgroupv1 only did this if you had
memory.move_charge_at_immigrate enabled). In addition, because we do the
/proc/self/exe clone before synchronising the bootstrap data read, we
are guaranteed to do the clone before "runc init" is moved into the
container cgroup -- meaning that the memory used by the /proc/self/exe
clone is charged against the root cgroup, and thus container workloads
should not be affected at all with memfd cloning.
The long-term fix for this problem is to block the /proc/self/exe
re-opening attack entirely in-kernel, which is something I'm working
on[1]. Though it should also be noted that because the memfd is
completely separate to the host binary, even attacks like Dirty COW
against the runc binary can be defended against with the memfd approach.
Of course, once we have in-kernel protection against the /proc/self/exe
re-opening attack, we won't have that protection anymore...
[1]: https://lwn.net/Articles/934460/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Older kernels may return EINVAL on unshare when a process is reading
runc's /proc/$PID/status or /proc/$PID/maps. This was fixed by kernel
commit 12c641ab8270f ("unshare: Unsharing a thread does not require
unsharing a vm") in Linux v4.3.
For CentOS 7, the fix was backported to CentOS 7.7 (kernel 3.10.0-1062).
To work around this kernel bug, let's retry on EINVAL a few times.
Reported-by: zzyyzte <zhang.yu58@zte.com.cn>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit cecb039d24)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
First, check if strdup() fails and error out.
While we are there, the else case was missing brackets, as we only need
to check ret in the else case. Fix that too
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
(cherry picked from commit 5ce511d6a6)
Remove upper bound in integer sanity check
to not restrict the number of socket-activated
sockets passed in.
Closes#3488
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
(cherry picked from commit 03a210d0f2)
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
Commit 4d1d6185ab added this
nsenter_unsupported.go file in order for nsenter to be a valid (but
empty, non-functional) Go package on unsupported platforms.
As a result, runc can be build successfully without CGO, which results
in a non-working and hard-to-debug binary (see issue 3330).
As the functionality of being able to compile a package which is
definitely not working is questionable, and I can't think of any use
cases, let's remove the file.
With this, runc can no longer be build without CGO:
[kir@kir-rhat runc]$ CGO_ENABLED=0 make runc
go build -trimpath "-buildmode=pie" -tags "seccomp" -ldflags "-X main.gitCommit=v1.0.0-452-g00f56786-dirty -X main.version=1.1.0-rc.1+dev " -o runc .
go build github.com/opencontainers/runc/libcontainer/nsenter: build constraints exclude all Go files in /home/kir/go/src/github.com/opencontainers/runc/libcontainer/nsenter
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The source of the bind mount might not be accessible in a different user
namespace because a component of the source path might not be traversed
under the users and groups mapped inside the user namespace. This caused
errors such as the following:
# time="2020-06-22T13:48:26Z" level=error msg="container_linux.go:367:
starting container process caused: process_linux.go:459:
container init caused: rootfs_linux.go:58:
mounting \"/tmp/busyboxtest/source-inaccessible/dir\"
to rootfs at \"/tmp/inaccessible\" caused:
stat /tmp/busyboxtest/source-inaccessible/dir: permission denied"
To solve this problem, this patch performs the following:
1. in nsexec.c, it opens the source path in the host userns (so we have
the right permissions to open it) but in the container mntns (so the
kernel cross mntns mount check let us mount it later:
https://github.com/torvalds/linux/blob/v5.8/fs/namespace.c#L2312).
2. in nsexec.c, it passes the file descriptors of the source to the
child process with SCM_RIGHTS.
3. In runc-init in Golang, it finishes the mounts while inside the
userns even without access to the some components of the source
paths.
Passing the fds with SCM_RIGHTS is necessary because once the child
process is in the container mntns, it is already in the container userns
so it cannot temporarily join the host mntns.
This patch uses the existing mechanism with _LIBCONTAINER_* environment
variables to pass the file descriptors from runc to runc init.
This patch uses the existing mechanism with the Netlink-style bootstrap
to pass information about the list of source mounts to nsexec.c.
Rootless containers don't use this bind mount sources fdpassing
mechanism because we can't setns() to the target mntns in a rootless
container (we don't have the privileges when we are in the host userns).
This patch takes care of using O_CLOEXEC on mount fds, and close them
early.
Fixes: #2484.
Signed-off-by: Alban Crequy <alban@kinvolk.io>
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
Co-authored-by: Rodrigo Campos <rodrigo@kinvolk.io>
Currently, if the log level is not set to e.g. "debug", runc init sends
some debug logs to the parent, which parses and discards it.
It is better to not send those in the first place.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The code already parses an environment variable into an integer twice,
and we're about to add a third one.
Factor it out to getenv_int().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This makes it possible to use bail() even if logging is not set up
(yet), so we don't have to think whether it's OK to use it or not.
In addition, this might help some unit tests that do not set log
forwarding.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
- add missing colons before error message;
- unify error messages after cmd.Start and cmd.Wait, so that they show
context and the error itself.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Instead of reading a single message, do read all the logs from the init,
and use DisallowUnknownFields for stricter checking.
While at it, use reapChildren to reap zombies (and add an extra check).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The test was not working since at least commit 64bb59f592
renamed pid to stage2_pid (or maybe even earlier), so the pid
was never received (i.e. pid.Pid was 0).
The problem was not caught because os.FindProcess never return an error
on Unix.
Factor out and fix pid decode function:
- use DisallowUnknownInput to get error if JSON will be changed;
- check pids to make sure they are valid
- and use unix.Wait4 to reap zombies.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Make sure we close all file descriptors at the end of the test.
2. Make sure we close child fds after the start.
3. Use newPipe for logs as well, for simplicity and uniformity.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Go 1.17 introduce this new (and better) way to specify build tags.
For more info, see https://golang.org/design/draft-gobuild.
As a way to seamlessly switch from old to new build tags, gofmt (and
gopls) from go 1.17 adds the new tags along with the old ones.
Later, when go < 1.17 is no longer supported, the old build tags
can be removed.
Now, as I started to use latest gopls (v0.7.1), it adds these tags
while I edit. Rather than to randomly add new build tags, I guess
it is better to do it once for all files.
Mind that previous commits removed some tags that were useless,
so this one only touches packages that can at least be built
on non-linux.
Brought to you by
go1.17 fmt ./...
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The two exceptions I had to add to codespellrc are:
- CLOS (used by intelrtd);
- creat (syscall name used in tests/integration/testdata/seccomp_*.json).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit 2bab4a5 resulted in a warning from gcc:
nsexec.c: In function ‘write_log’:
nsexec.c:171:2: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
171 | write(logfd, json, ret);
| ^~~~~~~~~~~~~~~~~~~~~~~
As there's nothing we can or want to do in case write fails,
let's just tell the compiler we're not going to use it.
Fixes: 2bab4a5
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The contents of the pointer returned on asprintf() error are undefined
i.e., it can be anything there. We set it to NULL on error so that
free() afterwards won't get a garbage pointer.
This patch applies the above to message and stage as well to be
consistent with what we do for json.
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
According to C standards, `size_t` is always an unsigned integer type.
Thus, checking unsigned expressions to be less than zero is not needed.
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
As reported in issue 3119, there is a race in nsexec logging
that can lead to garbled json received by log forwarder, which
complains about it with a "failed to decode" error.
This happens because dprintf (used since the very beginning of nsexec
logging introduced in commit ba3cabf932) relies on multiple write(2)
calls, and with additional logging added by 64bb59f592 a race is
possible between runc init parent and its children.
The fix is to prepare a string and write it using a single call to
write(2).
[v2: NULLify json on error from asprintf]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As pointed out in TODO item added by commit 64bb59f59, it is not
necessary to have a special sync mechanism for cgroupns, as the parent
adds runc init to cgroup way earlier (before sending nl bootstrap data.
This sync was added by commit df3fa115f9, which was also added a
second cgroup manager.Apply() call, later removed in commit
d1ba8e39f8. It seems the original author had the idea to wait for
that second Apply().
Fixes: df3fa115f9
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Don't ignore close(2) return code, rather bail if there is any
unexpected failures. By checking the close return code we make sure we
don't introduce the same bug (closing an already closed fd) I've fixed
in the previous patch.
As a side note, we are not handling in this patch when close(2) returns
EINTR and the go runtime, since go 1.14, sends SIGURG to preempt
goroutines. This should not happen here though, as nsenter is guaranteed
to be executed before the go runtime starts.
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
This was closed in the child[1], before calling clone_parent (so runc
INIT will have this fd closed too), there is no point closing it again.
This was not causing issues because we ignore the return code of
close(2) and no one was opening a new fd between both calls to close.
However, with the new patches that I'm working on (PR #2576), this
problem is no longer inocuos: we do open a new fd in that PR, sometimes
that fd is allocated between the two close(2) calls and, as the lowest
fd is allocated to the new fd, sometimes the second close ends up
incorrectly closing this new fd.
Before it was not a problem in practice, but it was incorrect
nevertheless.
This seems to be long standing bug, present since at least 2018
(a54316bae), when SYNC_GRANDCHILD was introduced.
[1]: https://github.com/opencontainers/runc/blob/5547b5774f71f75a088e7432fa961778750a0fbd/libcontainer/nsenter/nsexec.c#L888
Co-authored-by: Alban Crequy <alban@kinvolk.io>
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
We shouldn't refuse to build on architectures just because we don't know
what the syscall number of memfd_create(2) is. In addition, use the
correct defined(...) macros for ppc64 (these are the ones glibc uses).
Fixes: 3aead32ea2 ("nsenter: hard-code memfd_create(2) syscall numbers")
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Since the previous commit, some strings logged by write_log() contain a
literal newline, which leads to errors like this one:
> # time="2020-06-07T15:41:37Z" level=error msg="failed to decode \"{\\\"level\\\":\\\"debug\\\", \\\"msg\\\": \\\"nsexec-0[2265]: update /proc/2266/uid_map to '0 1000 1\\n\" to json: invalid character '\\n' in string literal"
The fix is to escape such characters.
Add a simple (as much as it can be) routine which implements JSON string
escaping as required by RFC4627, section 2.5, plus escaping of DEL (0x7f)
character (not required, but allowed by the standard, and usually done
by tools such as jq).
As much as I hate to code something like this, I was not able to find
a ready to consume and decent C implementation (not using glib).
Added a test case (and some additional asserts in C code, conditionally
enabled by the test case) to make sure the implementation is correct.
The test case have to live in a separate directory so we can use
different C flags to compile the test, and use C from go test.
[v2: try to simplify the code, add more tests]
[v3: don't do exit(1), try returning an error instead]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In order to make 'runc --debug' actually useful for debugging nsexec
bugs, provide information about all the internal operations when in
debug mode.
[@kolyshkin: rebasing; fix formatting via indent for make validate to pass]
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The stars can be aligned in a way that results in runc to leave a stale
bind mount in container's state directory, which manifests itself later,
while trying to remove the container, in an error like this:
> remove /run/runc/test2: unlinkat /run/runc/test2/runc.W24K2t: device or resource busy
The stale mount happens because runc start/run/exec kills runc init
while it is inside ensure_cloned_binary(). One such scenario is when
a unified cgroup resource is specified for cgroup v1, a cgroup manager's
Apply returns an error (as of commit b006f4a180), and when
(*initProcess).start() kills runc init just after it was started.
One solution is NOT to kill runc init too early. To achieve that,
amend the libcontainer/nsenter code to send a \0 byte to signal
that it is past the initial setup, and make start() (for both
run/start and exec) wait for this byte before proceeding with
kill on an error path.
While at it, improve some error messages.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Apparently, scripts/validate-c is not working in CI (or maybe
maintainers ignored the failures from it) -- current C code
gets some changes if we run indent on it.
This commit fixes this, simplifying things along the way.
In particular:
1. Remove "validate" make target, add "cfmt" target that just runs
indent on all *.c files in the repository (NOTE that *.h files
are not included, as before).
This may help a contributor to fix their code -- they just need
to run "make cfmt" now instead of running "make validate" and
copy-pasting the indent command and options from the hint.
2. Split GHA validate/misc into validate/release and validate/cfmt.
The latter checks that the sources are not changed after "make cfmt".
3. Adds a few more options to indent. This was mostly motivated by
trying to save the existing formatting, minimizing the amount of
changes indent produces.
The new options are:
* -il0: sets the offset for goto labels to 0 (currently all labels
but one are not indented -- let's keep it that way);
* -ppi2: sets the indentation for nested preprocessor directives
to 2 spaces (same as it is done in "SYS_memfd_create" defines);
* -cp1: sets the indentation between #else / #endif and the
following comment to 1 space.
4. Reformat the code using the new indent options.
5. Remove the now-unused script/{.validate,validate-c}.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Remove comments with architectures when defining SYS_memfd_create,
as they are redundant, and indent has a funny way of indenting them.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This block apparently does nothing except for creating
a need for additional indentation. Remove it.
While at it, break a long line in this code.
No functional change.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Some libc versions still in use by distributions (such as SLE) do not
define SYS_memfd_create even though the kernel supports the feature.
Since the syscall numbers are fixed, we can just hard-code them if
__NR_memfd_create is not defined.
We only do this for a handful of architectures, since containers aren't
widely supported on every possible Linux architecture.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
It is obvious that the loop at the first place executes at least
twice, and the close() call after the first time always returns
an EBADF error, so move these operations outside the loop that
do not need to be repeated.
Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Aleksa Sarai