Kir Kolyshkin
18e286261e
After adding some debug info to cloned_binary.c I found out that is_self_cloned() is not working right when runc binary is on tmpfs, resulting in one extra re-exec of runc. With some added debug: $ mkdir bin $ sudo mount -t tmpfs tmp bin $ sudo cp runc bin $ sudo ./bin/runc --debug exec xxx true DEBU[0000] nsexec[763590]: => is_self_cloned DEBU[0000] nsexec[763590]: got seals 1 (want 15) DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 0 DEBU[0000] nsexec[763590]: try_bindfd: 5 DEBU[0000] nsexec[763590]: re-exec itself... DEBU[0000] nsexec[763590]: => is_self_cloned DEBU[0000] nsexec[763590]: got seals 1 (want 15) DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 0 DEBU[0000] nsexec[763590]: try_bindfd: -1 DEBU[0000] nsexec[763590]: fallback to make_execfd: 5 DEBU[0000] nsexec[763590]: re-exec itself... DEBU[0000] nsexec[763590]: => is_self_cloned DEBU[0000] nsexec[763590]: got seals 15 (want 15) DEBU[0000] nsexec[763590]: <= is_self_cloned, is_cloned = 1 From the above, it is seen that - `is_self_cloned` returns 0, - `try_bindfd` is called and succeeds, - runc re-execs itself, - the second call to `is_self_cloned` returns 0 again (because GET_SEALS returns 1), - runc falls back to `make_execfd`, and re-execs again, - finally, the third `is_self_cloned` returns 1. I guess that the code relied on the following (quoting fcntl(2)): > Currently, file seals can be applied only to a file descriptor > returned by memfd_create(2) (if the MFD_ALLOW_SEALING was employed). > On other filesystems, all fcntl() operations that operate on seals > will return EINVAL. It looks like in case of a file on tmpfs it returns 1 (F_SEAL_SEAL). With the fix: DEBU[0000] nsexec[768367]: => is_self_cloned DEBU[0000] nsexec[768367]: got seals 1 (want 15) DEBU[0000] nsexec[768367]: no CLONED_BINARY_ENV DEBU[0000] nsexec[768367]: <= is_self_cloned, is_cloned = 0 DEBU[0000] nsexec[768367]: try_bindfd: 5 DEBU[0000] nsexec[768367]: re-exec itself... DEBU[0000] nsexec[768367]: => is_self_cloned DEBU[0000] nsexec[768367]: got seals 1 (want 15) DEBU[0000] nsexec[768367]: fstatfs says ro = 1 DEBU[0000] nsexec[768367]: fstat says nlink = 1 DEBU[0000] nsexec[768367]: <= is_self_cloned, is_cloned = 1 Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
nsenter
The nsenter package registers a special init constructor that is called before
the Go runtime has a chance to boot. This provides us the ability to setns on
existing namespaces and avoid the issues that the Go runtime has with multiple
threads. This constructor will be called if this package is registered,
imported, in your go application.
The nsenter package will import "C" and it uses cgo
package. In cgo, if the import of "C" is immediately preceded by a comment, that comment,
called the preamble, is used as a header when compiling the C parts of the package.
So every time we import package nsenter, the C code function nsexec() would be
called. And package nsenter is only imported in init.go, so every time the runc
init command is invoked, that C code is run.
Because nsexec() must be run before the Go runtime in order to use the
Linux kernel namespace, you must import this library into a package if
you plan to use libcontainer directly. Otherwise Go will not execute
the nsexec() constructor, which means that the re-exec will not cause
the namespaces to be joined. You can import it like this:
import _ "github.com/opencontainers/runc/libcontainer/nsenter"
nsexec() will first get the file descriptor number for the init pipe
from the environment variable _LIBCONTAINER_INITPIPE (which was opened
by the parent and kept open across the fork-exec of the nsexec() init
process). The init pipe is used to read bootstrap data (namespace paths,
clone flags, uid and gid mappings, and the console path) from the parent
process. nsexec() will then call setns(2) to join the namespaces
provided in the bootstrap data (if available), clone(2) a child process
with the provided clone flags, update the user and group ID mappings, do
some further miscellaneous setup steps, and then send the PID of the
child process to the parent of the nsexec() "caller". Finally,
the parent nsexec() will exit and the child nsexec() process will
return to allow the Go runtime take over.
NOTE: We do both setns(2) and clone(2) even if we don't have any
CLONE_NEW* clone flags because we must fork a new process in order to
enter the PID namespace.