Aleksa Sarai
614ce12f0e
(This is a cherry-pick of b999376fb237195265081a8b8ba3fd3bd6ef8c2c.)
While the ro-bind-mount trick did eliminate the memory overhead of
copying the runc binary for each "runc init" invocation, on machines
with very significant container churn, creating a temporary mount
namespace on every container invocation can trigger severe lock
contention on namespace_sem that makes containers fail to spawn.
The only reason we added bindfd in commit 16612d74de ("nsenter:
cloned_binary: try to ro-bind /proc/self/exe before copying") was due to
a Kubernetes e2e test failure where they had a ridiculously small memory
limit. It seems incredibly unlikely that real workloads are running
without 10MB to spare for the very short time that runc is interacting
with the container.
In addition, since the original cloned_binary implementation, cgroupv2
is now almost universally used on modern systems. Unlike cgroupv1, the
cgroupv2 memcg implementation does not migrate memory usage when
processes change cgroups (even cgroupv1 only did this if you had
memory.move_charge_at_immigrate enabled). In addition, because we do the
/proc/self/exe clone before synchronising the bootstrap data read, we
are guaranteed to do the clone before "runc init" is moved into the
container cgroup -- meaning that the memory used by the /proc/self/exe
clone is charged against the root cgroup, and thus container workloads
should not be affected at all with memfd cloning.
The long-term fix for this problem is to block the /proc/self/exe
re-opening attack entirely in-kernel, which is something I'm working
on[1]. Though it should also be noted that because the memfd is
completely separate to the host binary, even attacks like Dirty COW
against the runc binary can be defended against with the memfd approach.
Of course, once we have in-kernel protection against the /proc/self/exe
re-opening attack, we won't have that protection anymore...
[1]: https://lwn.net/Articles/934460/
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
nsenter
The nsenter package registers a special init constructor that is called before
the Go runtime has a chance to boot. This provides us the ability to setns on
existing namespaces and avoid the issues that the Go runtime has with multiple
threads. This constructor will be called if this package is registered,
imported, in your go application.
The nsenter package will import "C" and it uses cgo
package. In cgo, if the import of "C" is immediately preceded by a comment, that comment,
called the preamble, is used as a header when compiling the C parts of the package.
So every time we import package nsenter, the C code function nsexec() would be
called. And package nsenter is only imported in init.go, so every time the runc
init command is invoked, that C code is run.
Because nsexec() must be run before the Go runtime in order to use the
Linux kernel namespace, you must import this library into a package if
you plan to use libcontainer directly. Otherwise Go will not execute
the nsexec() constructor, which means that the re-exec will not cause
the namespaces to be joined. You can import it like this:
import _ "github.com/opencontainers/runc/libcontainer/nsenter"
nsexec() will first get the file descriptor number for the init pipe
from the environment variable _LIBCONTAINER_INITPIPE (which was opened
by the parent and kept open across the fork-exec of the nsexec() init
process). The init pipe is used to read bootstrap data (namespace paths,
clone flags, uid and gid mappings, and the console path) from the parent
process. nsexec() will then call setns(2) to join the namespaces
provided in the bootstrap data (if available), clone(2) a child process
with the provided clone flags, update the user and group ID mappings, do
some further miscellaneous setup steps, and then send the PID of the
child process to the parent of the nsexec() "caller". Finally,
the parent nsexec() will exit and the child nsexec() process will
return to allow the Go runtime take over.
NOTE: We do both setns(2) and clone(2) even if we don't have any
CLONE_NEW* clone flags because we must fork a new process in order to
enter the PID namespace.