In the past we incorrectly handled eBPF errors in two ways:
1. We would only ignore errors if there was an allow rule in the list
(this doesn't make sense because for security purposes we only care
if a *deny* rule is being ignored). Arguably this is a security flaw
but you would only get an error from bpf(2) in rare cases, and thus
is not a big enough deal to go through security review.
2. If we were in a rootless container we would still return an error
even though bpf(2) is blocked for rootless containers.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
It appears that briefly thawing the cgroup while freezing
greatly increases its chances to freeze successfully.
The test case I used is doing runc exec in a look parallel with runc
pause/resume in another loop, and the failure to freeze rate reduced
from 40 to 0 per minute (tested inside a VM using a busybox container
running sleep 1h, doing about 1500 pause/resumes and 650 execs per
minute), with max retries being 150 (of 1000).
This is still a game of chances, so failures are possible.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Before this commit, Set() used GetState() to check the freezer state
and retry the operation if the actual state still differs from requested.
This should help with the situation when a new process (such as one
added by runc exec) is added to the container's cgroup while it's being
freezed by the kernel, but it's not working as it should.
The problem is, GetState() never returns FREEZING state, looping until
the state is either FROZEN or THAWED, so Set() does not have a chance
to repeate the freeze attempt.
As a result, the container might end up stuck in a FREEZING state,
with GetState() never returning (which in turn blocks some other
operations).
One way to fix this would be to have GetState returning FREEZING state
instead of retrying ad infinitum. It would result in changing the public
API, and no callers of GetState expects it to return this.
To fix, let's not use GetState() from Set(). Instead, read the
freezer.state file directly and act accordingly -- return success
on FROZEN, retry on FREEZING, and error out on any other (unexpected)
value.
While at it, further improve the code:
- limit the number of retries;
- if retries are exceeded, thaw and return an error;
- don't retry (or read the state back) on THAW.
I played a lot with various reproducers for this bug, including
- parallel runc execs and runc pause/resumes
- parallel runc execs and runc --systemd-cgroup update
(the latter performs freeze/unfreeze);
- continuously running /bin/printf inside container
in parallel with runc pause/resume;
- running pthread bomb (from criu test suite) in parallel
with runc pause/resume;
and I was not able to make freeze work 100%, meaning sometimes
runc pause fails, or runc --systemd-cgroup update produces a warning.
With that said, it's still a big improvement over the previous
state of affairs where container is stuck in FREEZING state,
and GetState() (and all its users) are also stuck.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The merge 6eed6e5795 broke the build because ab27e12ceb ("Implement
GetStat for cpuset cgroup.") dropped the errors import which was used by
c85cd2b325 ("libct/cg/fs/cpuset: don't parse mountinfo") and the CI
wasn't retriggered.
Fix this by just importing "github.com/pkg/errors" again.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
A recent commit a35cad3b22 added warnings about systemd being too
old. While those warnings are valid, they break some existing tests,
and also don't add much value to a user (IOW no one is going to upgrade
systemd because runc says it's old).
Demote those to warnings.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It was already explained why we ignore the error, so let's ignore this
deliberately.
This fixes
> name.go:22:7: Error return value of `join` is not checked (errcheck)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Rewrite getPageUsageByNUMA
1. Be less strict to unknown contents, i.e. skip it. This makes the
function more future-proof. Before this commit, if a line like
"a=b" is encountered, the function returns an error, which is
propagated all the way up to and returned by (CgroupManager).GetStats.
2. Be more strict to contents it recognizes, i.e. return an error.
In case the first field in the line is recognized (e.g. "total=123",
the rest of the line should be in format "N<id>=<value> ...".
3. Optimize. Before this commit, addNUMAStatsByType was called for every
item in the line, which is excessive and might even be slow in case
there are many NUMA nodes. It is enough to look up the field once.
4. Remove a bunch of global numaNode* and numaStat* constants. Those
were used by only one function, and it does not make sense to have
them defined globally. Some were moved to the function, some were
eliminated entirely.
5. Improve readability and added code comments.
Finally, add some test cases for good and bad contents.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
strings.SplitN not always return N fields if not staify, sometimes
cgroup interface add some custom fields make parse memory.numa_stat
fails, it will case panic
Signed-off-by: acetang <aceapril@126.com>
This:
> === RUN TestGetHugePageSizeImpl
> utils_test.go:504: (input [hugepages-akB], error strconv.Atoi: parsing "a": invalid syntax)
feels like an error but it's not.
Only log errors.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Add a check to unifiedResToSystemdProps that systemd is recent enough
to support AllowedCPUs/AllowedMemoryNodes unit properties, and skip
setting the property if it is not supported.
Note that this is not an error as the setting is still applied to
the underlying cgroupfs -- it's just systemd unit property that is
being skipped.
2. In all the places we skip an unsupported property, warn about it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Support for systemd properties AllowedCPUs and AllowedMemoryNodes
was added by commit 13afa58d0e, but only for unified resources
of systemd v2 driver.
This adds support for Cpu.Cpus and Cpu.Mems resources to
both systemd v1 and v2 cgroup drivers.
An integration test is added to check that the settings work.
[v2: check for systemd version]
[v3: same in the test]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As the caller of this function just logs the error, it does not make
sense to pass it. Instead, log it (once) and return -1.
This is a preparation for the second user.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Here, we always create a parent directory, so using MkdirAll
is redundant. Use Mkdir instead.
One difference between MkdirAll and Mkdir is the former ignores
EEXIST, and since we sometimes try to create a directory that
already exists, we need to explicitly ignore that.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In cgroup v1, the parent of a cgroup mount is on tmpfs
(or maybe any other fs but definitely not cgroupfs).
Use this to traverse up the tree until we reach non-cgroupfs.
This should work in any setups (nested container, non-standard
cgroup mounts) so issue [1] won't happen.
Theoretically, the only problematic case is when someone mounts
cpuset cgroupfs into a directory on another cgroupfs. Let's
assume people don't do that -- if they do, they will get other
error (e.g. inability to read cpuset.cpus file).
[1] https://github.com/opencontainers/runc/issues/1367
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Drop the custom mountinfo parser, reuse the cgroups.GetCgroupMounts()
to get the cgroup root. Faster, and much less code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Apparently it is inevitable that we have to read mountinfo multiple
times when dealing with cgroup v1. It seems we can only do it once
and reuse the data, without major modifications to the code.
This commit does a few things.
1. Drop our custom mountinfo parser implementation in favor of
moby/sys/mountinfo. While the custom parser is faster
(about 2x according to benchmark) for this particular case,
the one from the package is more correct and future-proof.
2. Read mountinfo only once, caching all the entries with fstype of
cgroup. With this, there's no need to worry about performance
degradation introduced above.
3. Drop "isSubsystemAvailable" optimization (introduced by commit
2a1a6cdf44) because now with the cache it is probably slowing
things down.
4. Modify the tests accordingly.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
> libcontainer/cgroups/utils.go:282:4: SA4006: this value of `paths` is never used (staticcheck)
> paths = make(map[string]string)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case we get ENOSYS from openat2(2), this is expected, so log that
we're falling back to using securejoin as debug.
Otherwise, log it as a warning (as the error is unexpected, but we're
still good to go).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case openat2 is available, it will be used to guarantee
that we're not accessing anything other than cgroupfs[2] files.
In cases when openat2 is not available, or when cgroup has a
non-standard prefix (not "/sys/fs/cgroup", which might theoretically
be the case on some very old installs and/or very custom systems),
fall back to using securejoin + os.Open like we did before.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The `all` argument was introduced by commit f557996401 specifically
for use by cAdvisor (see [1]), but there were no test cases added,
so it was later broken by 5ee0648bfb which started incrementing
numFound unconditionally.
Fix this (by not checking numFound in case all is true), and add a
simple test case to avoid future regressions.
[1] https://github.com/google/cadvisor/pull/1476
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Move the Device-related types to libcontainer/devices, so that
the package can be used in isolation. Aliases have been created
in libcontainer/configs for backward compatibility.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* cpuset.cpus -> AllowedCPUs
* cpuset.mems -> AllowedMemoryNodes
No test for cgroup v2 resources.unified override, as this requires a
separate test case, and all the unified resources are handled uniformly
so there's little sense to test all parameters.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case systemd is used as cgroups manager, and a user sets some
resources using unified resource map (as per [1]), systemd is not
aware of any parameters, so there will be a discrepancy between
the cgroupfs state and systemd unit state.
Let's try to fix that by converting known unified resources to systemd
properties.
Currently, this is only implemented for pids.max as a POC.
Some other parameters (that might or might not have systemd unit
property equivalents) are:
$ ls -l | grep w-
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.freeze
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.max.depth
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.max.descendants
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.procs
-rw-r--r--. 1 root root 0 Oct 21 09:43 cgroup.subtree_control
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.threads
-rw-r--r--. 1 root root 0 Oct 10 13:57 cgroup.type
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpu.max
-rw-r--r--. 1 root root 0 Oct 10 13:57 cpu.pressure
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpuset.cpus
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpuset.cpus.partition
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpuset.mems
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpu.weight
-rw-r--r--. 1 root root 0 Oct 22 10:30 cpu.weight.nice
-rw-r--r--. 1 root root 0 Oct 22 10:30 hugetlb.1GB.max
-rw-r--r--. 1 root root 0 Oct 22 10:30 hugetlb.1GB.rsvd.max
-rw-r--r--. 1 root root 0 Oct 22 10:30 hugetlb.2MB.max
-rw-r--r--. 1 root root 0 Oct 22 10:30 hugetlb.2MB.rsvd.max
-rw-r--r--. 1 root root 0 Oct 22 10:30 io.bfq.weight
-rw-r--r--. 1 root root 0 Oct 22 10:30 io.latency
-rw-r--r--. 1 root root 0 Oct 22 10:30 io.max
-rw-r--r--. 1 root root 0 Oct 10 13:57 io.pressure
-rw-r--r--. 1 root root 0 Oct 22 10:30 io.weight
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.high
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.low
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.max
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.min
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.oom.group
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.pressure
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.swap.high
-rw-r--r--. 1 root root 0 Oct 10 13:57 memory.swap.max
Surely, it is a manual conversion for every such case...
[1] https://github.com/opencontainers/runtime-spec/pull/1040
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Mrunal Patel