Skip to content

Latest commit

 

History

History
239 lines (207 loc) · 11.7 KB

jailer.md

File metadata and controls

239 lines (207 loc) · 11.7 KB
Raw

The Firecracker Jailer

Jailer Usage

The jailer is invoked in this manner:

jailer --id <id> \
       --node <numa_node>\
       --exec-file <exec_file> \
       --uid <uid> \
       --gid <gid>
       [--chroot-base-dir <chroot_base>]
       [--netns <netns>]
       [--daemonize]
       [--seccomp-level <level>]
       [--...extra arguments for Firecracker]
  • id is the unique VM identification string, which may contain alphanumeric characters and hyphens. The maximum id length is currently 64 characters.
  • numa_node represents the NUMA node the process gets assigned to. More details are available below.
  • exec_file is the path to the Firecracker binary that will be exec-ed by the jailer. The user can provide a path to any binary, but the interaction with the jailer is mostly Firecracker specific.
  • uid and gid are the uid and gid the jailer switches to as it execs the target binary.
  • chroot_base represents the base folder where chroot jails are built. The default is /srv/jailer.
  • netns represents the path to a network namespace handle. If present, the jailer will use this to join the associated network namespace.
  • When present, the --daemonize flag causes the jailer to cal setsid() and redirect all three standard I/O file descriptors to /dev/null.
  • --seccomp-level specifies whether seccomp filters should be installed and how restrictive they should be. Possible values are:
    • 0 : disabled.
    • 1 : basic filtering. This prohibits syscalls not whitelisted by Firecracker.
    • 2 (default): advanced filtering. This adds further checks on some of the parameters of the allowed syscalls.
  • The jailer adheres to the "end of command options" convention, meaning all parameters specified after -- are forwarded to Firecracker. For example, this can be paired with the --config-file Firecracker argument to specify a configuration file when starting Firecracker via the jailer (the file path and the resources referenced within must be valid relative to a jailed Firecracker). Please note the jailer already passes the following parameters to the Firecracker process: --api-sock, --seccomp-level and --id.

Jailer Operation

After starting, the Jailer goes through the following operations:

  • Validate all provided paths and the VM id.
  • Close all open file descriptors based on /proc/<jailer-pid>/fd except input, output and error.
  • Create the <chroot_base>/<exec_file_name>/<id>/root folder, which will be henceforth referred to as chroot_dir. exec_file_name is the last path component of exec_file (for example, that would be firecracker for /usr/bin/firecracker). Nothing is done if the path already exists (it should not, since id is supposed to be unique).
  • Copy exec_file to <chroot_base>/<exec_file_name>/<id>/root/<exec_file_name>.
  • Create the cgroup sub-folders. At the moment, the jailer uses three cgroup v1 controllers: cpu, cpuset, and pids. On most systems, these (along with others) are mounted by default somewhere in /sys/fs/cgroup (they should be mounted by the user otherwise). The jailer will parse /proc/mounts to detect where each of the three controllers can be found (multiple controllers may share the same path). For each identified location (referred to as <cgroup_base>), the jailer creates the <cgroup_base>/<exec_file_name>/<id> subfolder, and writes the current pid to <cgroup_base>/<exec_file_name>/<id>/tasks. Also, the value of numa_node is written to the appropriate cpuset.mems file.
  • Call unshare() into a new mount namespace, use pivot_root() to switch the old system root mount point with a new one base in chroot_dir, switch the current working directory to the new root, unmount the old root mount point, and call chroot into the current directory.
  • Use mknod to create a /dev/net/tun equivalent inside the jail.
  • Use mknod to create a /dev/kvm equivalent inside the jail.
  • Use chown to change ownership of the chroot_dir (root path / as seen by the jailed firecracker), /dev/net/tun, /dev/kvm. The ownership is changed to the provided uid:gid.
  • If --netns <netns> is present, attempt to join the specified network namespace.
  • If --daemonize is specified, call setsid() and redirect STDIN, STDOUT, and STDERR to /dev/null.
  • Drop privileges via setting the provided uid and gid.
  • Exec into <exec_file_name> --id=<id> --api-sock=/api.socket --seccomp-level=<level> --start-time-us=<opaque> --start-time-cpu-us=<opaque> (and also forward any extra arguments provided to the jailer after --, as mentioned in the Jailer Usage section), where:
    • id: (string) - The id argument provided to jailer.
    • level: (number) - the --seccomp-level argument provided to jailer.
    • opaque: (number) time calculated by the jailer that it spent doing its work.

Example Run and Notes

Let’s assume Firecracker is available as /usr/bin/firecracker, and the jailer can be found at /usr/bin/jailer. We pick the unique id 551e7604-e35c-42b3-b825-416853441234, and we choose to run on NUMA node 0, using uid 123, and gid 100. For this example, we are content with the default /srv/jailer chroot base dir.

We start by running:

/usr/bin/jailer --id 551e7604-e35c-42b3-b825-416853441234 --node 0 \
--exec-file /usr/bin/firecracker --uid 123 --gid 100 \
--netns /var/run/netns/my_netns --daemonize

After opening the file descriptors mentioned in the previous section, the jailer will create the following resources (and all their prerequisites, such as the path which contains them):

  • /srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root/firecracker (copied from /usr/bin/firecracker)

Firecracker will create its API socket at /srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root/api.socket

We are going to refer to /srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root as <chroot_dir>.

Let’s also assume the cpu, cpuset, and pids cgroups are mounted at /sys/fs/cgroup/cpu, /sys/fs/cgroup/cpuset, and /sys/fs/cgroup/pids, respectively. The jailer will create the following subfolders (which will inherit settings from the parent cgroup):

  • /sys/fs/cgroup/cpu/firecracker/551e7604-e35c-42b3-b825-416853441234
  • /sys/fs/cgroup/cpuset/firecracker/551e7604-e35c-42b3-b825-416853441234
  • /sys/fs/cgroup/pids/firecracker/551e7604-e35c-42b3-b825-416853441234

It’s worth noting that, whenever a folder already exists, nothing will be done, and we move on to the next directory that needs to be created. This should only happen for the common firecracker subfolder (but, as for creating the chroot path before, we do not issue an error if folders directly associated with the supposedly unique id already exist).

The jailer then writes the current pid to /sys/fs/cgroup/cpu/firecracker/551e7604-e35c-42b3-b825-416853441234/tasks, /sys/fs/cgroup/cpuset/firecracker/551e7604-e35c-42b3-b825-416853441234/tasks, and /sys/fs/cgroup/pids/firecracker/551e7604-e35c-42b3-b825-416853441234/tasks. It also writes 0 to /sys/fs/cgroup/cpuset/firecracker/551e7604-e35c-42b3-b825-416853441234/cpuset.mems.

Since the --netns parameter is specified in our example, the jailer opens /var/run/netns/my_netns to get a file descriptor fd, uses setns(fd, CLONE_NEWNET) to join the associated network namespace, and then closes fd.

The --daemonize flag is also present, so the jailers opens /dev/null as RW and keeps the associate file descriptor as dev_null_fd (we do this before going inside the jail), to be used later.

Build the chroot jail. First, the jailer uses unshare() to enter a new mount namespace, and changes the propagation of all mount points in the new namespace to private using mount(NULL, “/”, NULL, MS_PRIVATE | MS_REC, NULL), as a prerequisite to pivot_root(). Another required operation is to bind mount <chroot_dir> on top of itself using mount(<chroot_dir>, <chroot_dir>, NULL, MS_BIND, NULL). At this point, the jailer creates the folder <chroot_dir>/old_root, changes the current directory to <chroot_dir>, and calls syscall(SYS_pivot_root, “.”, “old_root”). The final steps of building the jail are unmounting old_root using umount2(“old_root”, MNT_DETACH), deleting old_root with rmdir, and finally calling chroot(“.”) for good measure. From now, the process is jailed in <chroot_dir>.

Create the special file /dev/net/tun, using mknod(“/dev/net/tun”, S_IFCHR | S_IRUSR | S_IWUSR, makedev(10, 200)), and then call chown(“/dev/net/tun”, 123, 100), so Firecracker can use it after dropping privileges. This is required to use multiple TAP interfaces when running jailed. Do the same for /dev/kvm.

Change ownership of <chroot_dir> to uid:gid so that Firecracker can create its API socket there.

Since the --daemonize flag is present, call setsid() to join a new session, a new process group, and to detach from the controlling terminal. Then, redirect standard file descriptors to /dev/null by calling dup2(dev_null_fd, STDIN), dup2(dev_null_fd, STDOUT), and dup2(dev_null_fd, STDERR). Close dev_null_fd, because it is no longer necessary.

Finally, the jailer switches the uid to 123, and gid to 100, and execs

./firecracker \
  --id="551e7604-e35c-42b3-b825-416853441234" \
  --api-sock=/api.socket \
  --seccomp-level=2 \
  --start-time-us=<opaque> \
  --start-time-cpu-us=<opaque>

We can now use the socket at /srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root/api.socket to interact with the VM.

Observations

  • The user must create hard links for (or copy) any resources which will be provided to the VM via the API (disk images, kernel images, named pipes, etc) inside the jailed root folder. Also, permissions must be properly managed for these resources; for example the user which Firecracker runs as must have both read and write permissions to the backing file for a RW block device.
  • It’s up to the user to load balance VM placement among multiple NUMA nodes (if present), using the --node command line argument.
  • The user must also manage any further fine tuning of resource partitioning via cgroups (most likely the ones created by the jailer), or any other means.
  • It’s up to the user to handle cleanup after running the jailer. One way to do this involves registering handlers with the cgroup notify_on_release mechanism, while being wary about potential race conditions (the instance crashing before the subscription process is complete, for example).
  • For extra resilience, the jailer expects to be spawned by the user in a new PID namespace, most likely via a combination of clone() with the CLONE_NEWPID flag and exec(). A process must be created in a new PID namespace in order to become a pseudo-init process, and the other option is to use a clone() in the jailer, which seems unnecessary.
  • When running with --daemonize, the jailer will fail to start if it's a process group leader, because setsid() returns an error in this case. Spawning the jailer via clone() and exec() also ensures it cannot be a process group leader.
  • We run the jailer as the root user; it actually requires a more restricted set of capabilities, but that's to be determined as features stabilize.
  • The jailer can only log messages to stdout/err for now, which is why the logic associated with --daemonize runs towards the end, instead of the very beginning. We are working on adding better logging capabilities.

Caveats

  • If all the cgroup controllers are bunched up on a single mount point using the "all" option, our current program logic will complain it cannot detect individual controller mount points.