Replace Sigar system stats backend with OSHI (Graylog2#9840)

This commit adds a new Operating System and Hardware Information library (OSHI) that does not require the cumbersome packaging that Sigar currently needs. This library has a very active community and a pretty good platform options. A new config option ("disable_native_system_stats_collector") is added, that defaults to false.

Based on Graylog2#6056 by @desiderantes. Thank you!

Co-authored-by: Mario Daniel Ruiz Saavedra <[email protected]>

UPGRADING.rst

@@ -1,175 +1,42 @@
 **************************
-Upgrading to Graylog 4.0.x
+Upgrading to Graylog 4.1.x
 **************************
 
-.. _upgrade-from-33-to-40:
+.. _upgrade-from-40-to-41:
 
 .. contents:: Overview
    :depth: 3
    :backlinks: top
 
-.. warning:: Please make sure to create a MongoDB database backup before starting the upgrade to Graylog 4.0!
+.. warning:: Please make sure to create a MongoDB database backup before starting the upgrade to Graylog 4.1!
 
 Breaking Changes
 ================
 
-Changes to the Elasticsearch Support
-------------------------------------
+Configuration file changes
+--------------------------
 
-Starting with Graylog v4.0, bigger changes to the Elasticsearch versions supported are happening:
+The system stats collector has been reimplemented using OSHI instead of SIGAR.
+The configuration option `disable_sigar` has been renamed to `disable_native_system_stats_collector`.
 
-  - Support for Elasticsearch versions prior to v6.8.0 is dropped.
-  - Support for Elasticsearch v7.x is now included.
-
-This means that you can upgrade to Graylog v4.0 without an Elasticsearch update only if you have been on at least Elasticsearch v6.8.0 before.
-Additionally, due to the fact that Elasticsearch supports only indices created by the last two major versions (i.e. ES6.8.0+ reads indices created by ES5 & ES6, while ES7 reads indices created by ES6 & ES7), you can change to Graylog v4.0 with an Elasticsearch update without reindexing only if you have been on at least Elasticsearch v6.0.0 before.
-If you have been on any older Elasticsearch version you need to reindex every index you want to keep once for every two major versions of Elasticsearch up to at least ES6.
-
-When upgrading Elasticsearch from one major version to another, please read the upgrade guides provided by elastic:
-
-  - `To 6.8.0 <https://www.elastic.co/guide/en/elasticsearch/reference/6.8/setup-upgrade.html>`_
-  - `To 7.9.0 <https://www.elastic.co/guide/en/elasticsearch/reference/7.9/setup-upgrade.html>`_
-
-And our :ref:`Elasticsearch Upgrade Notes <upgrading-elasticsearch>`.
-
-Please do notice that Graylog does not support rolling upgrades between major versions, while Elasticsearch does. If you are upgrading from one major version of Elasticsearch to another, you need to restart Graylog in order to reinitialize the storage module. A procedure which allows a rolling upgrade of Elasticsearch between two major versions of a multi-node Graylog cluster is outlined :ref:`here <es_rolling_upgrade>`.
-
-LDAP and Active Directory configuration changes
------------------------------------------------
-
-The LDAP and Active Directory authentication and authorization support has been rewritten in Graylog 4.0. Existing LDAP settings will automatically be migrated to the new backend.
-
-Because the new backends require more configuration settings than the old one, migrated backends are **disabled by default**! Please make sure to review the migrated backends under "System / Authentication" and enable one backend to make it possible to login with LDAP or Active Directory users again.
-
-LDAP and Active Directory group mapping removed
------------------------------------------------
-
-The old LDAP and Active Directory group mapping has been replaced by teams support in Graylog Enterprise.
-
-Existing custom roles will be migrated to the new entity sharing system on the first server startup.
-
-Fixing certificate validation for LDAP servers used for authentication
-----------------------------------------------------------------------
-
-Prior to v3.3.3, the certificates of LDAP servers which are connected to using a secure connection (SSL or TLS) were not validated, even if the "Allow self-signed certificates" option was unchecked. Starting with v3.3.3, certificates are validated against the local default keystore. This might introduce a breaking change, depending on your local LDAP settings and the validity of the certificates used (if any). Please ensure that all certificates used are valid, their common name matches the host part of your configured LDAP server and your local keystore contains all CA/intermediate certs required for validation.
-
-A `CVE <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15813>`_ is tracked for this issue.
 
 Change of API endpoint for user retrieval and modification
 ----------------------------------------------------------
 
-In 4.0 we changed the following user API endpoints to expect a user ID parameter instead of the username:
-
 +-----------------------------------------------+-----------------------------+
 | Endpoint                                      | Description                 |
 +===============================================+=============================+
-| ``PUT /users/{userId}``                       | Update user account         |
-+-----------------------------------------------+-----------------------------+
-| ``PUT /users/{userId}/password``              | Update user password        |
-+-----------------------------------------------+-----------------------------+
-| ``GET /users/{userId}/tokens``                | Get user tokens             |
-+-----------------------------------------------+-----------------------------+
-| ``POST /users/{userId}/tokens/{name}``        | Generate new user API token |
+| ``PUT /example/placeholder``                  | TODO placeholder comment    |
 +-----------------------------------------------+-----------------------------+
-| ``DELETE /users/{userId}/tokens/{idOrToken}`` | Delete user API token       |
-+-----------------------------------------------+-----------------------------+
-
-Disabled Cross-Origin Requests by Default
------------------------------------------
-
-For improved security, Cross-Origin requests towards the API server are now disallowed by default.
-In the rare case, that your setup is serving the frontend assets from a different
-origin than the server, you can re-enable this with ``http_enable_cors = true`` in ``graylog.conf``.
-
-Removal of pluggable authentication realm Java APIs
----------------------------------------------------
-
-The Java API to implement custom authentication realms has been removed and got replaced with with the ``AuthServiceBackend`` Java API.
-
-SSO Authentication Plugin
--------------------------
-
-Due to the aforementioned removal of the pluggable authentication realm Java APIS, the `SSO Authentication Plugin <https://github.com/Graylog2/graylog-plugin-auth-sso>`_ doesn't work with Graylog 4.0 anymore.
-
-The core feature of the old SSO plugin (trusted HTTP header authentication) got integrated in the server.
-
-The old SSO plugin **must be removed** from the plugin folder before starting a Graylog 4.0 server.
 
 
 API Endpoint Deprecations
 =========================
 
-The following API endpoints are deprecated beginning with 4.0.
-
-Deprecation of cluster stats endpoints
---------------------------------------
-
-Starting with v4.0, the cluster stats endpoints are deprecated and will be removed in a future version. Those include:
-
-- ``/system/cluster/stats``
-- ``/system/cluster/stats/elasticsearch``
-- ``/system/cluster/stats/mongo``
-
-Deprecation of API endpoint for unpaginated listing of grok patterns
---------------------------------------------------------------------
-
-In 3.0 we introduced a new API endpoint to retrieve grok patterns from the backend: ``/system/grok/paginated`` which allows
-to pass pagination parameters.
-We therefore mark ``/system/grok`` as deprecated. Users who use this endpoint for scripting purpose should change
-their scripts to the format of the new endpoint, so they only need to to change the URL when ``/system/grok/paginated`` will become
-``/system/grok``.
-
-Deprecation of API endpoint for unpaginated listing of streams
---------------------------------------------------------------
-
-In 4.0 we introduce a new API endpoint to retrieve streams from the backend: ``/streams/paginated`` which allows
-to pass pagination parameters.
-We therefore mark ``/streams`` as deprecated. Users who use this endpoint for scripting purpose should change
-their scripts to the format of the new endpoint, so they only need to to change the URL when ``/streams/paginated`` will become
-``/streams``.
-
-Deprecation of API endpoint for unpaginated listing of users
-------------------------------------------------------------
-
-In 4.0 we introduce a new API endpoint to retrieve users from the backend: ``/users/paginated`` which allows
-to pass pagination parameters.
-We therefore mark ``/users`` as deprecated. Users who use this endpoint for scripting purpose should change
-their scripts to the format of the new endpoint, so they only need to to change the URL when ``/users/paginated`` will become
-``/users``.
+The following API endpoints are deprecated beginning with 4.1.
 
 API Endpoint Removals
 =====================
 
-The following API endpoints have been removed in 4.0.
-
-Removal of legacy Dashboard API
--------------------------------
-
-Starting with 3.3, the previous Dashboard API was replaced by the views API. Therefore it was moved to a ``/legacy`` prefix
-and marked for deprecation. Now it is removed altogether.
-
-Removal of legacy Saved Searches API
-------------------------------------
-
-Starting with 3.3, the previous Saved Searches API was replaced by the views API. Therefore it was moved to a ``/legacy`` prefix
-and marked for deprecation. Now it is removed altogether.
-
-Removal of legacy redirects for Dashboards & Saved Search API
--------------------------------------------------------------
-
-For 3.3, the pre-views Dashboards & Saved Searches APIs were moved to a ``/legacy`` prefix. The new APIs were moved to ``/dashboards`` & ``/search/saved`` and legacy redirects were created for the previous routes (``/views/dashboards`` & ``/views/savedSearches``).
-
-With 4.0, the legacy redirects (which were marked as being deprecated in 3.3) are removed.
-
-Removal of legacy LDAP API endpoints
-------------------------------------
-
-The following API endpoints for LDAP configuration management have been removed. They have been replaced with the new ``/system/authentication/services/backends`` API endpoints.
+The following API endpoints have been removed in 4.1.
 
-- ``GET /system/ldap/settings``
-- ``PUT /system/ldap/settings``
-- ``DELETE /system/ldap/settings``
-- ``GET /system/ldap/settings/groups``
-- ``PUT /system/ldap/settings/groups``
-- ``GET /system/ldap/groups``
-- ``POST /system/ldap/test``

bin/graylogctl

@@ -51,7 +51,7 @@ GRAYLOG_CONF=${GRAYLOG_CONF:=/etc/graylog/server/server.conf}
 GRAYLOG_PID=${GRAYLOG_PID:=/tmp/graylog.pid}
 LOG_FILE=${LOG_FILE:=log/graylog-server.log}
 LOG4J=${LOG4J:=}
-DEFAULT_JAVA_OPTS="-Djava.library.path=${GRAYLOGCTL_DIR}/../lib/sigar -Djdk.tls.acknowledgeCloseNotify=true -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow"
+DEFAULT_JAVA_OPTS="-Djdk.tls.acknowledgeCloseNotify=true -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow"
 if $JAVA_CMD -XX:+PrintFlagsFinal 2>&1 |grep -q UseParNewGC; then
 	DEFAULT_JAVA_OPTS="${DEFAULT_JAVA_OPTS} -XX:+UseParNewGC"
 fi

distribution/src/main/assembly/graylog.xml

@@ -17,13 +17,6 @@
                 <include>UPGRADING.rst</include>
             </includes>
         </fileSet>
-        <fileSet>
-            <directory>${project.basedir}/../lib/sigar-${sigar.version}</directory>
-            <outputDirectory>lib/sigar</outputDirectory>
-            <includes>
-                <include>**/*</include>
-            </includes>
-        </fileSet>
 
         <!-- empty directories -->
         <fileSet>

full-backend-tests/src/test/java/org/graylog/shared/system/stats/SystemStatsIT.java

@@ -0,0 +1,56 @@
+/*
+ * Copyright (C) 2020 Graylog, Inc.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the Server Side Public License, version 1,
+ * as published by MongoDB, Inc.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * Server Side Public License for more details.
+ *
+ * You should have received a copy of the Server Side Public License
+ * along with this program. If not, see
+ * <http://www.mongodb.com/licensing/server-side-public-license>.
+ */
+package org.graylog.shared.system.stats;
+
+import io.restassured.specification.RequestSpecification;
+import org.graylog.storage.elasticsearch6.ElasticsearchInstanceES6Factory;
+import org.graylog.testing.completebackend.ApiIntegrationTest;
+import org.graylog.testing.completebackend.GraylogBackend;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static io.restassured.RestAssured.given;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.graylog.testing.completebackend.Lifecycle.CLASS;
+
+@ApiIntegrationTest(serverLifecycle = CLASS, elasticsearchFactory = ElasticsearchInstanceES6Factory.class)
+public class SystemStatsIT {
+    private final GraylogBackend sut;
+    private final RequestSpecification requestSpec;
+
+    public SystemStatsIT(GraylogBackend sut, RequestSpecification requestSpec) {
+        this.sut = sut;
+        this.requestSpec = requestSpec;
+    }
+
+    @Test
+    void filesystemStats() {
+        final Map<Object, Object> filesystems = given()
+                .spec(requestSpec)
+                .when()
+                .get("/system/stats")
+                .then()
+                .statusCode(200)
+                .extract().jsonPath().getMap("fs.filesystems");
+
+        assertThat(filesystems).isNotEmpty();
+        assertThat(filesystems.get("/usr/share/graylog/data/journal")).satisfies(entry ->
+                assertThat(((HashMap) entry).get("mount")).isEqualTo("/"));
+    }
+}

graylog-project-parent/pom.xml

@@ -453,15 +453,9 @@
                 <scope>provided</scope>
             </dependency>
             <dependency>
-                <groupId>org.fusesource</groupId>
-                <artifactId>sigar</artifactId>
-                <version>${sigar.version}</version>
-                <exclusions>
-                    <exclusion>
-                        <groupId>log4j</groupId>
-                        <artifactId>log4j</artifactId>
-                    </exclusion>
-                </exclusions>
+                <groupId>com.github.oshi</groupId>
+                <artifactId>oshi-core</artifactId>
+                <version>${oshi.version}</version>
             </dependency>
             <dependency>
                 <groupId>org.graylog2.repackaged</groupId>
@@ -706,7 +700,7 @@
                 <configuration>
                     <forkCount>2</forkCount>
                     <reuseForks>true</reuseForks>
-                    <argLine>-Djava.library.path=${project.basedir}/../lib/sigar-${sigar.version} -Dio.netty.leakDetectionLevel=paranoid -Djava.awt.headless=true</argLine>
+                    <argLine>-Dio.netty.leakDetectionLevel=paranoid -Djava.awt.headless=true</argLine>
                     <excludes>
                         <exclude>**/*IntegrationTest.java</exclude>
                         <exclude>**/*IT.java</exclude>

graylog2-server/pom.xml

@@ -496,8 +496,8 @@
             <artifactId>HdrHistogram</artifactId>
         </dependency>
         <dependency>
-            <groupId>org.fusesource</groupId>
-            <artifactId>sigar</artifactId>
+            <groupId>com.github.oshi</groupId>
+            <artifactId>oshi-core</artifactId>
         </dependency>
         <dependency>
             <groupId>com.rabbitmq</groupId>

graylog2-server/src/main/java/org/graylog2/bootstrap/ServerBootstrap.java

@@ -211,7 +211,7 @@ protected List<Module> getSharedBindingsModules() {
         result.add(new SharedPeriodicalBindings());
         result.add(new SchedulerBindings());
         result.add(new GenericInitializerBindings());
-        result.add(new SystemStatsModule(configuration.isDisableSigar()));
+        result.add(new SystemStatsModule(configuration.isDisableNativeSystemStatsCollector()));
 
         return result;
     }

graylog2-server/src/main/java/org/graylog2/plugin/BaseConfiguration.java

@@ -71,8 +71,8 @@ public abstract class BaseConfiguration extends PathConfiguration {
     @Parameter("message_recordings_enable")
     private boolean messageRecordingsEnable = false;
 
-    @Parameter("disable_sigar")
-    private boolean disableSigar = false;
+    @Parameter("disable_native_system_stats_collector")
+    private boolean disableNativeSystemStatsCollector = false;
 
     @Parameter(value = "http_proxy_uri")
     private URI httpProxyUri;
@@ -162,8 +162,8 @@ public boolean isMessageRecordingsEnabled() {
         return messageRecordingsEnable;
     }
 
-    public boolean isDisableSigar() {
-        return disableSigar;
+    public boolean isDisableNativeSystemStatsCollector() {
+        return disableNativeSystemStatsCollector;
     }
 
     public URI getHttpProxyUri() {

graylog2-server/src/main/java/org/graylog2/shared/system/stats/OshiService.java

@@ -0,0 +1,61 @@
+/*
+ * Copyright (C) 2020 Graylog, Inc.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the Server Side Public License, version 1,
+ * as published by MongoDB, Inc.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * Server Side Public License for more details.
+ *
+ * You should have received a copy of the Server Side Public License
+ * along with this program. If not, see
+ * <http://www.mongodb.com/licensing/server-side-public-license>.
+ */
+package org.graylog2.shared.system.stats;
+
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import oshi.SystemInfo;
+import oshi.hardware.HardwareAbstractionLayer;
+import oshi.software.common.AbstractFileSystem;
+import oshi.software.os.OperatingSystem;
+import oshi.util.GlobalConfig;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import java.util.Arrays;
+import java.util.stream.Collectors;
+
+@Singleton
+public class OshiService {
+    private static final Logger LOG = LoggerFactory.getLogger(OshiService.class);
+    private final HardwareAbstractionLayer hal;
+    private final OperatingSystem os;
+
+
+    @Inject
+    public OshiService() {
+        // Don't let OSHI filter out "overlay" filesystems.
+        // Otherwise we cannot get proper disk statistics from within Docker.
+        final String fsTypes = GlobalConfig.get(AbstractFileSystem.OSHI_PSEUDO_FILESYSTEM_TYPES, "");
+        GlobalConfig.set(AbstractFileSystem.OSHI_PSEUDO_FILESYSTEM_TYPES,
+                Arrays.stream(fsTypes.split(",")).filter(fs -> !fs.equals("overlay")).collect(Collectors.joining(",")));
+        SystemInfo systemInfo = new SystemInfo();
+        hal = systemInfo.getHardware();
+        os = systemInfo.getOperatingSystem();
+        LOG.debug("Successfully loaded OSHI");
+
+    }
+
+    public HardwareAbstractionLayer getHal() {
+        return hal;
+    }
+
+    public OperatingSystem getOs() {
+        return os;
+    }
+}