[LibOS] Add secure implementation of eventfd

The previous implementation of eventfd was insecure and disabled by
default (enabled via `sys.insecure__allow_eventfd` manifest option). The
new implementation of eventfd is secure but currently restricted to a
single process: eventfd objects created in parent process become invalid
in children.

However, sometimes it is acceptable for applications to use host-based
insecure eventfd implementation. Because of these cases, we keep the old
``sys.insecure__allow_eventfd`` manifest syntax.

LibOS regression tests are added to test the secure implementation of
eventfd. Three LTP tests are enabled. Memcached and Rust examples also
use the secure eventfd now.

Signed-off-by: Dmitrii Kuvaiskii <[email protected]>

CI-Examples/memcached/memcached.manifest.template

@@ -18,10 +18,6 @@ loader.env.LD_LIBRARY_PATH = "/lib:/usr/{{ arch_libdir }}"
 loader.uid = 1000
 loader.gid = 1000
 
-# Memcached requires `eventfd` for worker thread notifications, starting from v1.6.11. If you have a
-# Memcached version older than that, we recommend to remove this insecure manifest option.
-sys.insecure__allow_eventfd = true
-
 sys.enable_sigterm_injection = true
 
 fs.mounts = [

CI-Examples/nginx/Makefile

@@ -29,8 +29,10 @@ ifeq ($(SGX),1)
 all: nginx.manifest.sgx nginx.sig
 endif
 
-# Note that Gramine doesn't support eventfd() and PR_SET_DUMPABLE, so we manually
-# overwrite these macros in the autogenerated configuration header of Nginx.
+# Note that Gramine doesn't support PR_SET_DUMPABLE, so we manually overwrite this macro in the
+# autogenerated configuration header of Nginx. Additionally, we disable support for eventfd, so that
+# Nginx falls back to alternative Inter Process Communication (IPC) means; recall that Gramine's
+# multi-process support for eventfd is insecure, thus we prefer not to use it here.
 $(INSTALL_DIR)/sbin/nginx: $(NGINX_SRC)/configure
 	cd $(NGINX_SRC) && ./configure --prefix=$(abspath $(INSTALL_DIR)) \
 		--without-http_rewrite_module --with-http_ssl_module

CI-Examples/rust/rust-hyper-http-server.manifest.template

@@ -30,12 +30,6 @@ sgx.trusted_files = [
   "file:{{ arch_libdir }}/",
 ]
 
-# The Tokio runtime requires eventfd, and the Gramine implementation
-# currently relies on the host in an insecure manner. This setting isn't
-# suitable for production deployment, but works well as a stopgap during
-# development while a proper implementation in Gramine is being worked on.
-sys.insecure__allow_eventfd = true
-
 # The maximum number of threads in a single process needs to be declared in advance.
 # You need to account for:
 # - one main thread

Documentation/devel/features.md

@@ -2657,21 +2657,35 @@ Gramine and is supported.
 
 ### Event notifications (eventfd)
 
-Gramine currently implements an *insecure* version of the `eventfd()` system call. It is considered
-insecure in the context of SGX backend because it relies on the host OS, which could for example
-maliciously drop an event or inject a random one. To enable this `eventfd()` implementation, the
-manifest file must contain `sys.insecure__allow_eventfd = true` ({ref}`see manifest documentation
-<allowing-eventfd>`).
+There are two modes of eventfd:
 
-Gramine supports polling on eventfd via `poll()`, `ppoll()`, `select()`, `epoll_*()` system calls.
+1. Secure "emulate-in-Gramine" -- the eventfd object is created inside Gramine, and all operations
+   are resolved entirely inside Gramine. A dummy eventfd object is created on the host, purely to
+   trigger read/write notifications (e.g., in epoll); eventfd values are verified inside Gramine and
+   are never exposed to the host. Since the host is used purely for notifications, a malicious host
+   can only induce Denial of Service (DoS) attacks; thus this implementation is secure and enabled
+   by default. This implementation is automatically disabled if `sys.insecure__allow_eventfd`
+   {ref}`manifest option <allowing-eventfd>` is enabled.
 
-Gramine may implement a secure version of `eventfd()` for communication between Gramine processes in
-the future. Such secure version will *not* be able to receive events from the host OS.
+   The emulation is currently implemented at the level of a single process. The emulation *may* work
+   for multi-process applications, e.g., if the child process inherits the eventfd object but
+   doesn't use it. However, all eventfds created in the parent process are marked as invalid in
+   child processes, i.e. inter-process communication via eventfds is not allowed.
+
+   Note that this secure version is *not* able to receive events from the host OS.
+
+2. Insecure "passthrough-to-host" -- the eventfd object is created on the host, and all operations
+   are delegated to the host. Since this implementation is insecure, it is disallowed by default. To
+   use this implementation, it must be explicitly allowed via the `sys.insecure__allow_eventfd`
+   {ref}`manifest option <allowing-eventfd>`.
+
+Gramine supports polling on eventfd via `poll()`, `ppoll()`, `select()`, `epoll_*()` system calls,
+in both secure and insecure modes.
 
 <details><summary>Related system calls</summary>
 
-- ▣ `eventfd()`: insecure implementation
-- ▣ `eventfd2()`: insecure implementation
+- ☑ `eventfd()`: see notes above
+- ☑ `eventfd2()`: see notes above
 - ☑ `close()`
 
 - ☑ `read()`

Documentation/manifest-syntax.rst

@@ -327,17 +327,28 @@ a 1 |~| MiB brk size.
 
 .. _allowing-eventfd:
 
-Allowing eventfd
-^^^^^^^^^^^^^^^^
+Allowing host-based insecure eventfd
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
     sys.insecure__allow_eventfd = [true|false]
     (Default: false)
 
-This specifies whether to allow system calls `eventfd()` and `eventfd2()`. Since
-eventfd emulation currently relies on the host, these system calls are
-disallowed by default due to security concerns.
+By default, Gramine implements eventfd in a secure but restricted way: currently
+this secure implementation only works when eventfd usage is confined to a single
+process (note that application may still be multi-process and spawn child
+processes, but eventfds created in parent will be invalid in children).
+
+However, sometimes it is acceptable for applications to use host-based insecure
+eventfd implementation. This implementation works without the above-mentioned
+restriction in multi-process applications. Use ``sys.insecure__allow_eventfd``
+manifest syntax to switch to this insecure implementation.
+
+.. note ::
+   ``sys.insecure__allow_eventfd`` is pass-through and thus potentially insecure
+   in e.g. SGX environments. It is the responsibility of the app developer to
+   analyze the app usage of eventfd, with security implications in mind.
 
 .. _external-sigterm-injection:
 

libos/include/libos_fs.h

@@ -182,6 +182,10 @@ struct libos_fs_ops {
     /* Poll a single handle. Must not block. */
     int (*poll)(struct libos_handle* hdl, int in_events, int* out_events);
 
+    /* Verify a single handle after poll. Must update `pal_ret_events` in-place with only allowed
+     * ones. Used in e.g. secure eventfd FS to verify if the host is not lying to us. */
+    void (*post_poll)(struct libos_handle* hdl, pal_wait_flags_t* pal_ret_events);
+
     /* checkpoint/migrate the file system */
     ssize_t (*checkpoint)(void** checkpoint, void* mount_data);
     int (*migrate)(void* checkpoint, void** mount_data);

libos/include/libos_handle.h

@@ -135,7 +135,11 @@ struct libos_epoll_handle {
 };
 
 struct libos_eventfd_handle {
+    bool broken_in_child;
     bool is_semaphore;
+    spinlock_t lock; /* protects below fields */
+    uint64_t val;
+    uint64_t dummy_host_val;
 };
 
 struct libos_handle {