Burstable CFS controller is introduced in Linux 5.14. This helps with
parallel workloads that might be bursty. They can get throttled even
when their average utilization is under quota. And they may be latency
sensitive at the same time so that throttling them is undesired.
This feature borrows time now against the future underrun, at the cost
of increased interference against the other system users, by introducing
cfs_burst_us into CFS bandwidth control to enact the cap on unused
bandwidth accumulation, which will then used additionally for burst.
The patch adds the support/control for CFS bandwidth burst.
runtime-spec: https://github.com/opencontainers/runtime-spec/pull/1120
Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com>
Co-authored-by: Nadeshiko Manju <me@manjusaka.me>
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
These are not exhaustive, but at least confirm that the feature is not
obviously broken (we correctly set the time offsets).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
We read output from the following files if they exists:
- cpu.pressure
- memory.pressure
- io.pressure
Each are in format:
```
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
```
Signed-off-by: Daniel Dao <dqminh89@gmail.com>
Co-authored-by: Sandor Szücs <sandor.szuecs@zalando.de>
Co-authored-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Rename CGROUP_PATH to CGROUP_V2_PATH so it is more clear that it can
only be used for CGROUP_V2, and to resolve ambiguity with CGROUP_PATH
variable used in tests/rootless.sh.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Apparently, bash with set -e deliberately ignores non-zero return codes
from ! cmd, unless this is the last command. The workaround is to either
use "! cmd || false', "or run ! cmd". Choose the latter, and require
bash-core 1.5.0 (since this is when "run !" was added), replacing the
older check.
Alas I only learned this recently from the bash-core documentation.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This test checks that the container is restored into a different cgroup.
To do so, a user should
- use --manage-cgroups-mode ignore on both checkpoint and restore;
- change the cgroupsPath value in config.json before restoring.
The test does some checks to ensure that its logic is correct, and that
after the restore the old (original) cgroup does not exist, the new one
exists, and the container's init is in that new cgroup.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Add a test case for an issue fixed by the previous commit.
The env should has more than 8 core CPU to meet the test requirement.
Signed-off-by: Chengen, Du <chengen.du@canonical.com>
We use awk in other 9 or so places, and here it's gawk.
Since this is on Linux, most probably awk is gawk.
So s/gawk/awk/.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The situation when /sys/fs/cgroup/unified is not present normal and
should not result in anything on stderr. Suppress it.
Fixes: cc15b887a0
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Currently only amd64 and arm64v8 tarball have been checked in testdata,
while busybox bundle is downloaded on fly, and supports multiple architectures.
To enable integration tests for more architectures, the hello world
bundle is replaced by busybox one.
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
The variable $ROOTLESS, as set by helpers.bash and used in many places,
provides the same value as $EUID which is always set by bash. Since we
are using bash, we can rely on $EUID being omnipresent.
Modify all uses accordingly, and since the value is known to be a
number, omit the quoting.
Similarly, replace all uses of $(id -u) to $EUID.
Do some trivial cleanups along the way, such as
- simplify some if A; then B; to A && B;
- do not use [[ instead of [ where not necessary.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This makes it work similar to all the other variables we use as binary
flags.
The new 'shellcheck disable' is due to a bug in shellcheck (basically,
it does not track the scope of variables or execution order, assuming
everything is executed as soon as it is seen).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Strictly speaking, == is for [[ only, not for [ / test,
and, unlike =, the right side is a pattern.
To avoid confusion, use =. In cases where we compare with empty string,
use -z instead.
Keep using [[ in some cases since it does not require quoting the left
and right side of comparison (I trust shellcheck on that one).
This should have no effect (other than the code being a tad more
strict).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This is a way to prevent the code doing something really bad when a
variable it uses is not set. Good to have since it helps to catch some
logical errors etc.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Audit all checks for non-empty variables (i.e. ' -z ', ' -n ',
' != ""' and '= ""'), and fix those cases where a variable might be
unset. Those variables (that might not be set) are
- RUNC_USE_SYSTEMD
- BATS_RUN_TMPDIR
- AUX_UID
- AUX_DIR
- SD_PARENT_NAME
- REL_PARENT_PATH
- ROOT
- HAVE_CRIU
- ROOTLESS_FEATURES
- and a few test-specific or file-specific variables
This should allow us to enable set -u.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Some tests (those in help.bats and version.bats) do not use setup_bundle
(as they do not need to start any containers), and thus they do not set
$ROOT. As a consequence, these tests now call "runc --root /state" which
is not nice.
Make adding --root conditional (only if $ROOT is set).
Amazingly, this change breaks help.bats tests under rootless, because
"sudo rootless" does not change the value of XDG_RUNTIME_DIR which still
points to root-owned directory, and as a result we have this:
> runc foo -h (status=1):
> the path in $XDG_RUNTIME_DIR must be writable by the user
> time="2022-02-08T07:04:57Z" level=error msg="mkdir /run/user/0/runc: permission denied"
This could be fixed by adding proper $ROOT, but it's easier just to skip
those tests under non-root.
NOTE that version.bats is not broken because -v is handled by urfave/cli
very early, so app.Before function is not run.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Older bash versions treats variable as unset if nothing has been
assigned to it. Here is an example from CentOS 7 system:
[kir@localhost ~]$ bash -u -c 'x() { local args=(); echo "${args[@]}"; }; x'
bash: args[@]: unbound variable
[kir@localhost ~]$ echo $BASH_VERSION
4.2.46(2)-release
Rewrite to work around this.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. This valid warning is reported by shellcheck v0.8.0:
In tests/integration/helpers.bash line 38:
KERNEL_MINOR="${KERNEL_VERSION#$KERNEL_MAJOR.}"
^-----------^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.
Did you mean:
KERNEL_MINOR="${KERNEL_VERSION#"$KERNEL_MAJOR".}"
Fix this.
2. These (invalid) warnings are also reported by the new version:
In tests/integration/events.bats line 13:
@test "events --stats" {
^-- SC2030 (info): Modification of status is local (to subshell caused by @bats test).
In tests/integration/events.bats line 41:
[ "$status" -eq 0 ]
^-----^ SC2031 (info): status was modified in a subshell. That change might be lost.
Basically, this is happening because shellcheck do not really track
the call tree and/or local variables. This is a known (and reported)
deficiency, and the alternative to disabling these warnings is moving
the code around, which is worse due to more changes in git history.
So we have to silence/disable these.
3. Update shellcheck to 0.8.0.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This was introduced in an initial commit, back in the day when criu was
a highly experimental thing. Today it's not; most users who need it have
it packaged by their distro vendor.
The usual way to run a binary is to look it up in directories listed in
$PATH. This is flexible enough and allows for multiple scenarios (custom
binaries, extra binaries, etc.). This is the way criu should be run.
Make --criu a hidden option (thus removing it from help). Remove the
option from man pages, integration tests, etc. Remove all traces of
CriuPath from data structures.
Add a warning that --criu is ignored and will be removed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Using "$@" instead of $1 in update_config() allows us to use it from
hooks.bats, where jq is used with more options than usual.
We need to disable SC2016 as otherwise shellcheck sees $something inside
single quotes and think we are losing the shell expansion (we are not).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This can be used to specify a different runc binary, for example:
sudo -E RUNC=$PWD/runc.mine tests/integration/cwd.bats
A different (but compatible enough) runtime also works:
sudo -E RUNC=/usr/local/bin/crun tests/integration/cwd.bats
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Some test directories are created using mktemp -d, and so they have
permissions set to 0700 and are thus inaccessible to a user inside
userns. This was workarounded for $ROOT in userns.bats before.
Now, when we have updated Cirrus CI config to use Fedora 35 (rather than
34), userns tests fail:
> runc run failed: unable to start container process: error during
> container init: error preparing rootfs: mount
> /tmp/bats-run-4pCERd/runc.f66gCC/bundle/rootfs:/tmp/bats-run-4pCERd/runc.f66gCC/bundle/rootfs,
> flags: 0x5000: permission denied
Fedora 34 image used kernel v5.11, while Fedora 35 has v5.15.
Apparently, the newer kernel also checks that the parent directories
are accessible by the user before doing mount.
Move the old workaround from userns.bats to helpers.bats, drop the r bit
(not needed), and add $BATS_RUN_TMPDIR (also created by mktemp -d) to
fix userns.bats test failures under Fedora 35.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Check that runc run and runc exec put the process on the same cgroups v2
when using hybrid mode.
Signed-off-by: Mauricio Vásquez <mauricio@kinvolk.io>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Add functional test to check seccomp notify end-to-end. This test uses the
sample seccomp agent from the contrib/cmd folder.
Signed-off-by: Mauricio Vásquez <mauricio@kinvolk.io>
Signed-off-by: Rodrigo Campos <rodrigo@kinvolk.io>
Co-authored-by: Rodrigo Campos <rodrigo@kinvolk.io>
Add a test case for an issue fixed by the previous commit.
Unfortunately, this is somewhat complicated as there's no easy way to
create a transient unit, so a binary, sd-helper, had to be added. On top
of that, an ability to create a parent/pod cgroup is added to
helpers.bash, which might be useful for future integration tests.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
... and add the file to be checked by shellcheck.
The warnings fixed are:
In tests/integration/helpers.bash line 10:
INTEGRATION_ROOT=$(dirname "$(readlink -f "$BASH_SOURCE")")
^----------^ SC2128: Expanding an array without an index only gives the first element.
In tests/integration/helpers.bash line 22:
TESTDATA="${INTEGRATION_ROOT}/testdata"
^------^ SC2034: TESTDATA appears unused. Verify use (or export if used externally).
In tests/integration/helpers.bash line 42:
echo "runc $@ (status=$status):" >&2
^-- SC2145: Argument mixes string and array. Use * or separate argument.
^-----^ SC2154: status is referenced but not assigned.
In tests/integration/helpers.bash line 43:
echo "$output" >&2
^-----^ SC2154: output is referenced but not assigned.
In tests/integration/helpers.bash line 77:
| .linux.gidMappings += [{"hostID": '"$(($ROOTLESS_GIDMAP_START + 10))"', "containerID": 1, "size": 20}]
^--------------------^ SC2004: $/${} is unnecessary on arithmetic variables.
In tests/integration/helpers.bash line 78:
| .linux.gidMappings += [{"hostID": '"$(($ROOTLESS_GIDMAP_START + 100))"', "containerID": 1000, "size": '"$(($ROOTLESS_GIDMAP_LENGTH - 1000))"'}]'
^--------------------^ SC2004: $/${} is unnecessary on arithmetic variables.
^---------------------^ SC2004: $/${} is unnecessary on arithmetic variables.
In tests/integration/helpers.bash line 125:
base_path=$(gawk '$(NF-2) == "cgroup" && $NF ~ /\<'${g}'\>/ { print $5; exit }' /proc/self/mountinfo)
^--^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
base_path=$(gawk '$(NF-2) == "cgroup" && $NF ~ /\<'"${g}"'\>/ { print $5; exit }' /proc/self/mountinfo)
In tests/integration/helpers.bash line 127:
eval CGROUP_${g^^}_BASE_PATH="${base_path}"
^----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
eval CGROUP_"${g^^}"_BASE_PATH="${base_path}"
In tests/integration/helpers.bash line 229:
if [ "x$CGROUP_UNIFIED" = "xyes" ]; then
^----------------^ SC2268: Avoid x-prefix in comparisons as it no longer serves a purpose.
Did you mean:
if [ "$CGROUP_UNIFIED" = "yes" ]; then
In tests/integration/helpers.bash line 234:
eval cgroup=\$${var}${REL_CGROUPS_PATH}
^----^ SC2086: Double quote to prevent globbing and word splitting.
^-----------------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
eval cgroup=\$"${var}""${REL_CGROUPS_PATH}"
In tests/integration/helpers.bash line 236:
cat $cgroup/$source
^-----^ SC2086: Double quote to prevent globbing and word splitting.
^-----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
cat "$cgroup"/"$source"
In tests/integration/helpers.bash line 242:
current="$(get_cgroup_value $1)"
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
current="$(get_cgroup_value "$1")"
In tests/integration/helpers.bash line 245:
echo "current" $current "!?" "$expected"
^------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
echo "current" "$current" "!?" "$expected"
In tests/integration/helpers.bash line 257:
[ $(id -u) != "0" ] && user="--user"
^------^ SC2046: Quote this to prevent word splitting.
In tests/integration/helpers.bash line 259:
current=$(systemctl show $user --property $source $SD_UNIT_NAME | awk -F= '{print $2}')
^-----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
current=$(systemctl show $user --property "$source" $SD_UNIT_NAME | awk -F= '{print $2}')
In tests/integration/helpers.bash line 261:
[ "$current" = "$expected" ] || [ -n "$expected2" -a "$current" = "$expected2" ]
^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.
In tests/integration/helpers.bash line 309:
check_cgroup_value "cpu.weight" $weight
^-----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
check_cgroup_value "cpu.weight" "$weight"
In tests/integration/helpers.bash line 310:
check_systemd_value "CPUWeight" $weight
^-----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
check_systemd_value "CPUWeight" "$weight"
In tests/integration/helpers.bash line 383:
if [ $CGROUP_UNIFIED = "no" -a ! -e "${CGROUP_MEMORY_BASE_PATH}/memory.memsw.limit_in_bytes" ]; then
^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.
In tests/integration/helpers.bash line 412:
local cpu_count=$(grep -c '^processor' /proc/cpuinfo)
^-------^ SC2155: Declare and assign separately to avoid masking return values.
In tests/integration/helpers.bash line 450:
sleep $delay
^----^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
sleep "$delay"
In tests/integration/helpers.bash line 453:
echo "Command \"$@\" failed $attempts times. Output: $output"
^-- SC2145: Argument mixes string and array. Use * or separate argument.
In tests/integration/helpers.bash line 471:
runc state $1
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
runc state "$1"
In tests/integration/helpers.bash line 472:
if [ $2 == "checkpointed" ]; then
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
if [ "$2" == "checkpointed" ]; then
In tests/integration/helpers.bash line 484:
mkdir $dir
^--^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
mkdir "$dir"
In tests/integration/helpers.bash line 497:
kill -9 $(cat "$dir/pid")
^---------------^ SC2046: Quote this to prevent word splitting.
In tests/integration/helpers.bash line 508:
export ROOT=$(mktemp -d "$BATS_RUN_TMPDIR/runc.XXXXXX")
^--^ SC2155: Declare and assign separately to avoid masking return values.
In tests/integration/helpers.bash line 512:
cd "$ROOT/bundle"
^---------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
Did you mean:
cd "$ROOT/bundle" || exit
In tests/integration/helpers.bash line 535:
cd "$INTEGRATION_ROOT"
^--------------------^ SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
Did you mean:
cd "$INTEGRATION_ROOT" || exit
For more information:
https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
https://www.shellcheck.net/wiki/SC2034 -- TESTDATA appears unused. Verify u...
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It is not used since PR 2757, as all tests are run with cd to bundle
directory.
runc_spec argument count checking is removed since otherwise shellcheck
complains:
> SC2120: runc_spec references arguments, but none are ever passed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Before this commit, "require rootless_cgroup" feature check required "cgroup"
to be present in ROOTLESS_FEATURES. The idea of the requirement, though, is
to ensure that rootless runc can manage cgroups.
In case of systemd + cgroup v2, rootless runc can manage cgroups,
thanks to systemd delegation, so modify the feature check accordingly.
Next, convert (simplify) some of the existing users to the modified check.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. In case of cgroup v2 + systemd + rootless, get the list of controllers from
the own cgroup, rather than from the root one (as systemd cgroup delegation
is required in this case, and it might not be set or fully working).
2. Use "requires cgroups_<controller>" in tests that need those
controllers.
Tested on Fedora 31 (which does not support delegation of cpuset controller).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This was added by commit ca4f427af, together with set_resources_limit,
and was needed for the latter, since sed was used to update resources.
Now when we switched to jq in commit 79fe41d3c1, this kludge is no
longer needed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This is somewhat radical approach to deal with kernel memory.
Per-cgroup kernel memory limiting was always problematic. A few
examples:
- older kernels had bugs and were even oopsing sometimes (best example
is RHEL7 kernel);
- kernel is unable to reclaim the kernel memory so once the limit is
hit a cgroup is toasted;
- some kernel memory allocations don't allow failing.
In addition to that,
- users don't have a clue about how to set kernel memory limits
(as the concept is much more complicated than e.g. [user] memory);
- different kernels might have different kernel memory usage,
which is sort of unexpected;
- cgroup v2 do not have a [dedicated] kmem limit knob, and thus
runc silently ignores kernel memory limits for v2;
- kernel v5.4 made cgroup v1 kmem.limit obsoleted (see
https://github.com/torvalds/linux/commit/0158115f702b).
In view of all this, and as the runtime-spec lists memory.kernel
and memory.kernelTCP as OPTIONAL, let's ignore kernel memory
limits (for cgroup v1, same as we're already doing for v2).
This should result in less bugs and better user experience.
The only bad side effect from it might be that stat can show kernel
memory usage as 0 (since the accounting is not enabled).
[v2: add a warning in specconv that limits are ignored]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit 41670e21f0 added some randomization to cgroup paths
and (if systemd cgroup driver is used) systemd unit names,
but the randomization was per bats instance, not per test.
Fix this by refactoring init_cgroups_path/set_cgroups_path
(moving variable/random part to set_cgroups_path).
NOTE though that the randomization is only performed for those tests
that explicitly call set_cgroups_path.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Helper function init_cgroup_paths sets two sets of cgroup path variables
for cgroup v1 case (below XXX is cgroup controller name, e.g. MEMORY):
1. CGROUP_XXX_BASE_PATH -- path to XXX controller mount point
(e.g. CGROUP_MEMORY_BASE_PATH=/sys/fs/cgroup/memory);
2. CGROUP_XXX -- path to the particular container XXX controller cgroup
(e.g. CGROUP_MEMORY=/sys/fs/cgroup/memory/runc-cgroups-integration-test/test-cgroup).
The second set of variables is mostly used by check_cgroup_value(),
with only two exceptions:
- CGROUP_CPU in @test "update rt period and runtime";
- few CGROUP_XXX in @test "runc delete --force in cgroupv1 with
subcgroups".
Remove these variables, as their values are not used much
and are easy to get (as can be seen in modified test cases).
While at it, mark some variables as local.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Get rid of fixed ROOT, *_BUNDLE, and CONSOLE_SOCKET dirs.
Now they are temporary directories created in setup_bundle.
2. Automate containers cleanup: instead of having to specify all
containers to be removed, list and destroy everything (which is
now possible since every test case has its own unique root).
3. Randomize cgroup paths so two tests running in parallel won't
use the same cgroup.
Now it's theoretically possible to execute tests in parallel.
Practically it's not possible yet because bats uses GNU parallel,
which do not provide a terminal for whatever it executes, and
many runc tests (all those that run containers with terminal:
true) needs a tty. This may possibly be addressed later.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It's a little known fact, but BATS_TMPDIR is just /tmp, and it is not
being cleaned by bats after the test.
BATS_RUN_TMPDIR (available since bats 1.2.1) is $BATS_TMPDIR/bats-run-$$
(i.e. it is per bats run and includes PID), and it is cleaned up after
the test (unless --no-tmpdir-cleanup is given).
Use BATS_RUN_TMPDIR for better isolation and cleanup.
[v2: check for BATS_RUN_TMPDIR, effectively requiring bats >= 1.2.1]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
... by using retry, to avoid code duplication.
While at it, make it check the number of arguments and error out on
invalid usage.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It is only used in a couple of places and is easy to replace
with ROOT=xxx teardown_running_container.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Aleksa Sarai