runc/config-linux.md

Linux Container Configuration

This document describes the schema for the Linux-specific section of the container configuration. The Linux container specification uses various kernel features like namespaces, cgroups, capabilities, LSM, and filesystem jails to fulfill the spec.

Default Filesystems

The Linux ABI includes both syscalls and several special file paths. Applications expecting a Linux environment will very likely expect these file paths to be setup correctly.

The following filesystems SHOULD be made available in each container's filesystem:

Path	Type
/proc	procfs
/sys	sysfs
/dev/pts	devpts
/dev/shm	tmpfs

Namespaces

A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. For more information, see the namespaces(7) man page.

Namespaces are specified as an array of entries inside the namespaces root field. The following parameters can be specified to setup namespaces:

• type (string, REQUIRED) - namespace type. The following namespace types are supported:

  • pid processes inside the container will only be able to see other processes inside the same container.
  • network the container will have its own network stack.
  • mount the container will have an isolated mount table.
  • ipc processes inside the container will only be able to communicate to other processes inside the same container via system level IPC.
  • uts the container will be able to have its own hostname and domain name.
  • user the container will be able to remap user and group IDs from the host to local users and groups within the container.
  • cgroup the container will have an isolated view of the cgroup hierarchy.

• path (string, OPTIONAL) - an absolute path to namespace file in the runtime mount namespace

If a path is specified, that particular file is used to join that type of namespace. If a namespace type is not specified in the namespaces array, the container MUST inherit the runtime namespace of that type. If a namespaces field contains duplicated namespaces with same type, the runtime MUST error out.

Example

    "namespaces": [
        {
            "type": "pid",
            "path": "/proc/1234/ns/pid"
        },
        {
            "type": "network",
            "path": "/var/run/netns/neta"
        },
        {
            "type": "mount"
        },
        {
            "type": "ipc"
        },
        {
            "type": "uts"
        },
        {
            "type": "user"
        },
        {
            "type": "cgroup"
        }
    ]

User namespace mappings

uidMappings (array of objects, OPTIONAL) describes the user namespace uid mappings from the host to the container. gidMappings (array of objects, OPTIONAL) describes the user namespace gid mappings from the host to the container.

Each entry has the following structure:

• hostID (uint32, REQUIRED) - is the starting uid/gid on the host to be mapped to containerID.
• containerID (uint32, REQUIRED) - is the starting uid/gid in the container.
• size (uint32, REQUIRED) - is the number of ids to be mapped.

The runtime SHOULD NOT modify the ownership of referenced filesystems to realize the mapping. Note that the number of mapping entries MAY be limited by the kernel.

Example

    "uidMappings": [
        {
            "hostID": 1000,
            "containerID": 0,
            "size": 32000
        }
    ],
    "gidMappings": [
        {
            "hostID": 1000,
            "containerID": 0,
            "size": 32000
        }
    ]

Devices

devices (array of objects, OPTIONAL) lists devices that MUST be available in the container. The runtime may supply them however it likes (with mknod, by bind mounting from the runtime mount namespace, etc.).

Each entry has the following structure:

• type (string, REQUIRED) - type of device: c, b, u or p. More info in mknod(1).
• path (string, REQUIRED) - full path to device inside container. If a [file][file.1] already exists at path that does not match the requested device, the runtime MUST generate an error.
• major, minor (int64, REQUIRED unless type is p) - major, minor numbers for the device.
• fileMode (uint32, OPTIONAL) - file mode for the device. You can also control access to devices with cgroups.
• uid (uint32, OPTIONAL) - id of device owner.
• gid (uint32, OPTIONAL) - id of device group.

The same type, major and minor SHOULD NOT be used for multiple devices.

Example

   "devices": [
        {
            "path": "/dev/fuse",
            "type": "c",
            "major": 10,
            "minor": 229,
            "fileMode": 438,
            "uid": 0,
            "gid": 0
        },
        {
            "path": "/dev/sda",
            "type": "b",
            "major": 8,
            "minor": 0,
            "fileMode": 432,
            "uid": 0,
            "gid": 0
        }
    ]

Default Devices

In addition to any devices configured with this setting, the runtime MUST also supply:

• /dev/null
• /dev/zero
• /dev/full
• /dev/random
• /dev/urandom
• /dev/tty
• /dev/console is setup if terminal is enabled in the config by bind mounting the pseudoterminal slave to /dev/console.
• /dev/ptmx. A bind-mount or symlink of the container's /dev/pts/ptmx.

Control groups

Also known as cgroups, they are used to restrict resource usage for a container and handle device access. cgroups provide controls (through controllers) to restrict cpu, memory, IO, pids and network for the container. For more information, see the kernel cgroups documentation.

The path to the cgroups can be specified in the Spec via cgroupsPath. cgroupsPath can be used to either control the cgroup hierarchy for containers or to run a new process in an existing container. If cgroupsPath is:

• ... an absolute path (starting with /), the runtime MUST take the path to be relative to the cgroup mount point.
• ... a relative path (not starting with /), the runtime MAY interpret the path relative to a runtime-determined location in the cgroup hierarchy.
• ... not specified, the runtime MAY define the default cgroup path. Runtimes MAY consider certain cgroupsPath values to be invalid, and MUST generate an error if this is the case. If a cgroupsPath value is specified, the runtime MUST consistently attach to the same place in the cgroup hierarchy given the same value of cgroupsPath.

Implementations of the Spec can choose to name cgroups in any manner. The Spec does not include naming schema for cgroups. The Spec does not support per-controller paths for the reasons discussed in the cgroupv2 documentation. The cgroups will be created if they don't exist.

You can configure a container's cgroups via the resources field of the Linux configuration. Do not specify resources unless limits have to be updated. For example, to run a new process in an existing container without updating limits, resources need not be specified.

A runtime MUST at least use the minimum set of cgroup controllers required to fulfill the resources settings. However, a runtime MAY attach the container process to additional cgroup controllers supported by the system.

Example

   "cgroupsPath": "/myRuntime/myContainer",
   "resources": {
      "memory": {
         "limit": 100000,
         "reservation": 200000
      },
      "devices": [
         {
            "allow": false,
            "access": "rwm"
         }
      ]
   }

Device whitelist

devices (array of objects, OPTIONAL) configures the device whitelist. The runtime MUST apply entries in the listed order.

Each entry has the following structure:

• allow (boolean, REQUIRED) - whether the entry is allowed or denied.
• type (string, OPTIONAL) - type of device: a (all), c (char), or b (block). null or unset values mean "all", mapping to a.
• major, minor (int64, OPTIONAL) - major, minor numbers for the device. null or unset values mean "all", mapping to * in the filesystem API.
• access (string, OPTIONAL) - cgroup permissions for device. A composition of r (read), w (write), and m (mknod).

Example

   "devices": [
        {
            "allow": false,
            "access": "rwm"
        },
        {
            "allow": true,
            "type": "c",
            "major": 10,
            "minor": 229,
            "access": "rw"
        },
        {
            "allow": true,
            "type": "b",
            "major": 8,
            "minor": 0,
            "access": "r"
        }
    ]

Disable out-of-memory killer

disableOOMKiller contains a boolean (true or false) that enables or disables the Out of Memory killer for a cgroup. If enabled (false), tasks that attempt to consume more memory than they are allowed are immediately killed by the OOM killer. The OOM killer is enabled by default in every cgroup using the memory subsystem. To disable it, specify a value of true. For more information, see the memory cgroup man page.

• disableOOMKiller (bool, OPTIONAL) - enables or disables the OOM killer

Example

    "disableOOMKiller": false

Set oom_score_adj

oomScoreAdj sets heuristic regarding how the process is evaluated by the kernel during memory pressure. For more information, see the proc filesystem documentation section 3.1. This is a kernel/system level setting, where as disableOOMKiller is scoped for a memory cgroup. For more information on how these two settings work together, see the memory cgroup documentation section 10. OOM Contol.

• oomScoreAdj (int, OPTIONAL) - adjust the oom-killer score

Example

    "oomScoreAdj": 100

Memory

memory (object, OPTIONAL) represents the cgroup subsystem memory and it's used to set limits on the container's memory usage. For more information, see the memory cgroup man page.

The following parameters can be specified to setup the controller:

• limit (uint64, OPTIONAL) - sets limit of memory usage in bytes

• reservation (uint64, OPTIONAL) - sets soft limit of memory usage in bytes

• swap (uint64, OPTIONAL) - sets limit of memory+Swap usage

• kernel (uint64, OPTIONAL) - sets hard limit for kernel memory

• kernelTCP (uint64, OPTIONAL) - sets hard limit in bytes for kernel TCP buffer memory

• swappiness (uint64, OPTIONAL) - sets swappiness parameter of vmscan (See sysctl's vm.swappiness)

Example

    "memory": {
        "limit": 536870912,
        "reservation": 536870912,
        "swap": 536870912,
        "kernel": 0,
        "kernelTCP": 0,
        "swappiness": 0
    }

CPU

cpu (object, OPTIONAL) represents the cgroup subsystems cpu and cpusets. For more information, see the cpusets cgroup man page.

The following parameters can be specified to setup the controller:

• shares (uint64, OPTIONAL) - specifies a relative share of CPU time available to the tasks in a cgroup

• quota (int64, OPTIONAL) - specifies the total amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by period below)

• period (uint64, OPTIONAL) - specifies a period of time in microseconds for how regularly a cgroup's access to CPU resources should be reallocated (CFS scheduler only)

• realtimeRuntime (int64, OPTIONAL) - specifies a period of time in microseconds for the longest continuous period in which the tasks in a cgroup have access to CPU resources

• realtimePeriod (uint64, OPTIONAL) - same as period but applies to realtime scheduler only

• cpus (string, OPTIONAL) - list of CPUs the container will run in

• mems (string, OPTIONAL) - list of Memory Nodes the container will run in

Example

    "cpu": {
        "shares": 1024,
        "quota": 1000000,
        "period": 500000,
        "realtimeRuntime": 950000,
        "realtimePeriod": 1000000,
        "cpus": "2-3",
        "mems": "0-7"
    }

Block IO

blockIO (object, OPTIONAL) represents the cgroup subsystem blkio which implements the block IO controller. For more information, see the kernel cgroups documentation about blkio.

The following parameters can be specified to setup the controller:

• blkioWeight (uint16, OPTIONAL) - specifies per-cgroup weight. This is default weight of the group on all devices until and unless overridden by per-device rules. The range is from 10 to 1000.

• blkioLeafWeight (uint16, OPTIONAL) - equivalents of blkioWeight for the purpose of deciding how much weight tasks in the given cgroup has while competing with the cgroup's child cgroups. The range is from 10 to 1000.

• blkioWeightDevice (array, OPTIONAL) - specifies the list of devices which will be bandwidth rate limited. The following parameters can be specified per-device:

  • major, minor (int64, REQUIRED) - major, minor numbers for device. More info in man mknod.
  • weight (uint16, OPTIONAL) - bandwidth rate for the device, range is from 10 to 1000
  • leafWeight (uint16, OPTIONAL) - bandwidth rate for the device while competing with the cgroup's child cgroups, range is from 10 to 1000, CFQ scheduler only

  You must specify at least one of weight or leafWeight in a given entry, and can specify both.

• blkioThrottleReadBpsDevice, blkioThrottleWriteBpsDevice, blkioThrottleReadIOPSDevice, blkioThrottleWriteIOPSDevice (array, OPTIONAL) - specify the list of devices which will be IO rate limited. The following parameters can be specified per-device:

  • major, minor (int64, REQUIRED) - major, minor numbers for device. More info in man mknod.
  • rate (uint64, REQUIRED) - IO rate limit for the device

Example

    "blockIO": {
        "blkioWeight": 10,
        "blkioLeafWeight": 10,
        "blkioWeightDevice": [
            {
                "major": 8,
                "minor": 0,
                "weight": 500,
                "leafWeight": 300
            },
            {
                "major": 8,
                "minor": 16,
                "weight": 500
            }
        ],
        "blkioThrottleReadBpsDevice": [
            {
                "major": 8,
                "minor": 0,
                "rate": 600
            }
        ],
        "blkioThrottleWriteIOPSDevice": [
            {
                "major": 8,
                "minor": 16,
                "rate": 300
            }
        ]
    }

Huge page limits

hugepageLimits (array of objects, OPTIONAL) represents the hugetlb controller which allows to limit the HugeTLB usage per control group and enforces the controller limit during page fault. For more information, see the kernel cgroups documentation about HugeTLB.

Each entry has the following structure:

• pageSize (string, REQUIRED) - hugepage size

• limit (uint64, REQUIRED) - limit in bytes of hugepagesize HugeTLB usage

Example

   "hugepageLimits": [
        {
            "pageSize": "2MB",
            "limit": 209715200
        }
   ]

Network

network (object, OPTIONAL) represents the cgroup subsystems net_cls and net_prio. For more information, see the net_cls cgroup man page and the net_prio cgroup man page.

The following parameters can be specified to setup the controller:

• classID (uint32, OPTIONAL) - is the network class identifier the cgroup's network packets will be tagged with

• priorities (array, OPTIONAL) - specifies a list of objects of the priorities assigned to traffic originating from processes in the group and egressing the system on various interfaces. The following parameters can be specified per-priority:

  • name (string, REQUIRED) - interface name
  • priority (uint32, REQUIRED) - priority applied to the interface

Example

   "network": {
        "classID": 1048577,
        "priorities": [
            {
                "name": "eth0",
                "priority": 500
            },
            {
                "name": "eth1",
                "priority": 1000
            }
        ]
   }

PIDs

pids (object, OPTIONAL) represents the cgroup subsystem pids. For more information, see the pids cgroup man page.

The following parameters can be specified to setup the controller:

• limit (int64, REQUIRED) - specifies the maximum number of tasks in the cgroup

Example

   "pids": {
        "limit": 32771
   }

IntelRdt

Intel platforms with new Xeon CPU support Intel Resource Director Technology (RDT). Cache Allocation Technology (CAT) is a sub-feature of RDT, which currently supports L3 cache resource allocation.

This feature provides a way for the software to restrict cache allocation to a defined 'subset' of L3 cache which may be overlapping with other 'subsets'. The different subsets are identified by class of service (CLOS) and each CLOS has a capacity bitmask (CBM).

In Linux kernel, it is exposed via "resource control" filesystem, which is a "cgroup-like" interface.

Comparing with cgroups, it has similar process management lifecycle and interfaces in a container. But unlike cgroups' hierarchy, it has single level filesystem layout.

Intel RDT "resource control" filesystem hierarchy:

mount -t resctrl resctrl /sys/fs/resctrl
tree /sys/fs/resctrl
/sys/fs/resctrl/
|-- info
|   |-- L3
|       |-- cbm_mask
|       |-- min_cbm_bits
|       |-- num_closids
|-- cpus
|-- schemata
|-- tasks
|-- <container_id>
    |-- cpus
    |-- schemata
    |-- tasks

For containers, we can make use of tasks and schemata configuration for L3 cache resource constraints if hardware and kernel support Intel RDT/CAT.

The file tasks has a list of tasks that belongs to this group (e.g., <container_id>" group). Tasks can be added to a group by writing the task ID to the "tasks" file (which will automatically remove them from the previous group to which they belonged). New tasks created by fork(2) and clone(2) are added to the same group as their parent. If a pid is not in any sub group, it is in root group.

The file schemata has allocation masks/values for L3 cache on each socket, which contains L3 cache id and capacity bitmask (CBM).

	Format: "L3:<cache_id0>=<cbm0>;<cache_id1>=<cbm1>;..."

For example, on a two-socket machine, L3's schema line could be L3:0=ff;1=c0 Which means L3 cache id 0's CBM is 0xff, and L3 cache id 1's CBM is 0xc0.

The valid L3 cache CBM is a contiguous bits set and number of bits that can be set is less than the max bit. The max bits in the CBM is varied among supported Intel Xeon platforms. In Intel RDT "resource control" filesystem layout, the CBM in a group should be a subset of the CBM in root. Kernel will check if it is valid when writing. e.g., 0xfffff in root indicates the max bits of CBM is 20 bits, which mapping to entire L3 cache capacity. Some valid CBM values to set in a group: 0xf, 0xf0, 0x3ff, 0x1f00 and etc.

intelRdt (object, OPTIONAL) represents the L3 cache resource constraints in Intel Xeon platforms.

For more information, see Intel RDT/CAT kernel interface.

The following parameters can be specified for the container:

• l3CacheSchema (string, OPTIONAL) - specifies the schema for L3 cache id and capacity bitmask (CBM)

Example

There are two L3 caches in the two-socket machine, the default CBM is 0xfffff
and the max CBM length is 20 bits. This configuration assigns 4/5 of L3 cache
id 0 and the whole L3 cache id 1 for the container:

"linux": {
	"intelRdt": {
		"l3CacheSchema": "L3:0=ffff0;1=fffff"
	}
}

Sysctl

sysctl (object, OPTIONAL) allows kernel parameters to be modified at runtime for the container. For more information, see the sysctl(8) man page.

Example

   "sysctl": {
        "net.ipv4.ip_forward": "1",
        "net.core.somaxconn": "256"
   }

Seccomp

Seccomp provides application sandboxing mechanism in the Linux kernel. Seccomp configuration allows one to configure actions to take for matched syscalls and furthermore also allows matching on values passed as arguments to syscalls. For more information about Seccomp, see Seccomp kernel documentation. The actions, architectures, and operators are strings that match the definitions in seccomp.h from libseccomp and are translated to corresponding values.

seccomp (object, OPTIONAL)

The following parameters can be specified to setup seccomp:

• defaultAction (string, REQUIRED) - the default action for seccomp. Allowed values are the same as syscalls[].action.

• architectures (array of strings, OPTIONAL) - the architecture used for system calls. A valid list of constants as of libseccomp v2.3.2 is shown below.

  • SCMP_ARCH_X86
  • SCMP_ARCH_X86_64
  • SCMP_ARCH_X32
  • SCMP_ARCH_ARM
  • SCMP_ARCH_AARCH64
  • SCMP_ARCH_MIPS
  • SCMP_ARCH_MIPS64
  • SCMP_ARCH_MIPS64N32
  • SCMP_ARCH_MIPSEL
  • SCMP_ARCH_MIPSEL64
  • SCMP_ARCH_MIPSEL64N32
  • SCMP_ARCH_PPC
  • SCMP_ARCH_PPC64
  • SCMP_ARCH_PPC64LE
  • SCMP_ARCH_S390
  • SCMP_ARCH_S390X
  • SCMP_ARCH_PARISC
  • SCMP_ARCH_PARISC64

• syscalls (array of objects, REQUIRED) - match a syscall in seccomp.

  Each entry has the following structure:

  • names (array of strings, REQUIRED) - the names of the syscalls.

  • action (string, REQUIRED) - the action for seccomp rules. A valid list of constants as of libseccomp v2.3.2 is shown below.

    • SCMP_ACT_KILL
    • SCMP_ACT_TRAP
    • SCMP_ACT_ERRNO
    • SCMP_ACT_TRACE
    • SCMP_ACT_ALLOW

  • args (array of objects, OPTIONAL) - the specific syscall in seccomp.

    Each entry has the following structure:

    • index (uint, REQUIRED) - the index for syscall arguments in seccomp.

    • value (uint64, REQUIRED) - the value for syscall arguments in seccomp.

    • valueTwo (uint64, REQUIRED) - the value for syscall arguments in seccomp.

    • op (string, REQUIRED) - the operator for syscall arguments in seccomp. A valid list of constants as of libseccomp v2.3.2 is shown below.

      • SCMP_CMP_NE
      • SCMP_CMP_LT
      • SCMP_CMP_LE
      • SCMP_CMP_EQ
      • SCMP_CMP_GE
      • SCMP_CMP_GT
      • SCMP_CMP_MASKED_EQ

Example

   "seccomp": {
       "defaultAction": "SCMP_ACT_ALLOW",
       "architectures": [
           "SCMP_ARCH_X86",
           "SCMP_ARCH_X32"
       ],
       "syscalls": [
           {
               "names": [
                   "getcwd",
                   "chmod"
               ],
               "action": "SCMP_ACT_ERRNO"
           }
       ]
   }

Rootfs Mount Propagation

rootfsPropagation (string, OPTIONAL) sets the rootfs's mount propagation. Its value is either slave, private, or shared. The Shared Subtrees article in the kernel documentation has more information about mount propagation.

Example

    "rootfsPropagation": "slave",

Masked Paths

maskedPaths (array of strings, OPTIONAL) will mask over the provided paths inside the container so that they cannot be read. The values MUST be absolute paths in the container namespace.

Example

    "maskedPaths": [
        "/proc/kcore"
    ]

Readonly Paths

readonlyPaths (array of strings, OPTIONAL) will set the provided paths as readonly inside the container. The values MUST be absolute paths in the container namespace.

Example

    "readonlyPaths": [
        "/proc/sys"
    ]

Mount Label

mountLabel (string, OPTIONAL) will set the Selinux context for the mounts in the container.

Example

    "mountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c715,c811"