Recursively merge mqtt_websockets (netdata#16755)

* initial version

* basic GitHub Actions CI

* Create README.md

* run GitHub Action at least once weekly
to check test pass if distro was updated but project doesn't get
any change for long time

* add test for rbuf_find_bytes

* minor - add .vscode to .gitignore

* minor - add CFLAGS to makefile

* minor - silence unused var warning

* minor - update readme

* add submodules

* initial commit

* add README

* allow setting MQTT LWT

* allow setting LWT QOS

* handle MQTT keep-alives properly

* allow choosing keep alive time

* handle WS_OP_CONNECTION_CLOSE

* allow sending other frames than WS_OP_BINARY_FRAME

* minor readability improvements

* work on graceful disconnect

* reset ws_client state on subsequent connections

* implement mqtt_wss_destroy

* more descriptive RC for mqtt_wss_service

* properly free/destroy SSL

* work on reconnect

* set return code for WS disconnect

* Less logging under normal operation

* readme point to `test.c` as how-to for now

* OpenSSL certificate checking by default

* reset poll_fds on reconnect

* fix older SSL versions

* empty install and dist targets

* fix LGTM warning

* add rbuf_get_capacity

* gh actions test

* ws_ping impl

* test.c - port as cmd line param

* properly handle WebSocket disconnect packet

* initial HTTPS proxy support

* remove base64 submodule, use OpenSSL

* change to urandom

* ws_client_process WS_RAW retval fix

* coverity fixes

* CID 1448838

* CID 1448836

* CID 1448829

* test check for error on init

* make it play nicer with automake projects

* MQTT-C coverity fixes

* Create codeql-analysis.yml

* Make typedef introduce a new name for struct. (netdata#2)

* add autotools related files to gitignore

* minor - silence -Wmaybe-uninitialized warning

* reinit on MQTT clean session connect

* fix apple endianness functions

* add libcrypto for macos

* MacOS compatibility

* fix FreeBSD build

* use TLS SNI

* propagate buffer full EC to app layer

* store buffer sizes in mqtt_wss_client

* allow auto buffer growth on buffer too small

* quicker connection drop on BUFFER FULL

* always clear last_ec on connect

* parse all HTTP headers

* minor - rename constant for clarity

* rename idx and idx2 for code readability

* limit response header field count

* flush not needed, replace with descriptive err msg

* bump MQTT-C

* Blocking publish and inflight MQTT-C buffer growth

* update obsolete comment

* add missing unlock in error case

* fix EINTR

* initial MQTT 5 implementation

Implements minimal MQTT 5 features. Up to QoS1

* fixes for bugs in initial MQTT5 implementation

* initial statistics support

* fix vbi parser

* initial base http proxy auth support

* fix base64 helper for longer credentials

* minor - create with single allocation

* allow defining custom alloc functions for user

* Allow custom memory fncs and macros by user (netdata#5)

* update crbuf module
* allow custom malloc, free, strdup, calloc ...

* use long long for till_next_keep_alive

* dont send PUBACK on QOS0 (netdata#6)

* fix publish parser (netdata#7)

* mark QoS0 as GC on send (netdata#8)

* log extra info in case of OpenSSL error (netdata#9)

* log extra info in case of SSL error

* fix build error with older SSL (netdata#10)

* adds possibility to decrypt traffic with wireshark for debugging (netdata#13)

* initial commit

* add fncs key:uint64 and data:opaque ptr

* add uint64_t key iterator

* add github test runner

* start working on proper tests

* Initial support for topic aliases (netdata#12)

Add support for topic alias functionality for PUBLISH packets
also adds support for parsing all MQTT properties as opposed to just skipping and ignoring them (what we did previously)

* implement c_rhash_iter_str_keys + tests

* Fixes of Topic Alias implementation (netdata#14)

* initial removal of mqtt-c support (netdata#15)

* honor max msglen for server (netdata#19)

* Update README.md (netdata#20)

* memory align fragments (netdata#21)

* Remove mqtt_websocket submodule

* Remove c-rbuf and c_rhash submodules

* Exclude mqtt_websockets from Codacy

It seems that it was excluded before merging
the mqtt_websockets submodule.

---------

Co-authored-by: Timotej Šiškovič <[email protected]>
Co-authored-by: Timotej S <[email protected]>
Co-authored-by: Emmanuel Vasilakis <[email protected]>

.codacy.yml

@@ -22,3 +22,4 @@ exclude_paths:
   - build/**
   - build_external/**
   - packaging/**
+  - mqtt_websockets/**

.gitmodules

@@ -1,6 +1,3 @@
-[submodule "mqtt_websockets"]
-	path = mqtt_websockets
-	url = https://github.com/underhood/mqtt_websockets.git
 [submodule "aclk/aclk-schemas"]
 	path = aclk/aclk-schemas
 	url = https://github.com/netdata/aclk-schemas.git

mqtt_websockets/.github/workflows/run-tests.yaml

@@ -0,0 +1,14 @@
+name: run-tests
+on:
+  push:
+  schedule:
+      - cron: '5 3 * * 0'
+  pull_request:
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install ruby and deps
+        run: sudo apt-get install ruby ruby-dev mosquitto
+      - name: Checkout
+        uses: actions/checkout@v2

mqtt_websockets/.gitignore

@@ -0,0 +1,10 @@
+build/*
+!build/.keep
+test
+.vscode
+mqtt/mqtt.c
+mqtt/include/mqtt.h
+libmqttwebsockets.*
+*.o
+.dirstamp
+.deps

mqtt_websockets/Makefile

@@ -0,0 +1,61 @@
+# Copyright (C) 2020 Timotej Šiškovič
+# SPDX-License-Identifier: GPL-3.0-only
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
+# without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with this program.
+# If not, see <https://www.gnu.org/licenses/>.
+
+CC = gcc -std=gnu99
+CFLAGS = -Wextra -Wall `pkg-config --cflags openssl` `pkg-config --cflags libcrypto`
+BUILD_DIR = build
+
+INCLUDES = -Isrc/include -Ic-rbuf/include -Ic_rhash/include -Imqtt/include
+
+all: test
+
+$(BUILD_DIR)/c_rhash.o: c_rhash/src/c_rhash.c c_rhash/src/c_rhash_internal.h c_rhash/include/c_rhash.h
+	$(CC) -o $(BUILD_DIR)/c_rhash.o -c c_rhash/src/c_rhash.c $(CFLAGS) $(INCLUDES)
+
+c-rbuf/build/ringbuffer.o:
+	cd c-rbuf && $(MAKE) build/ringbuffer.o
+
+$(BUILD_DIR)/ws_client.o: src/ws_client.c src/include/ws_client.h src/include/common_internal.h
+	$(CC) -o $(BUILD_DIR)/ws_client.o -c src/ws_client.c $(CFLAGS) $(INCLUDES)
+
+$(BUILD_DIR)/test.o: src/test.c src/include/ws_client.h libmqttwebsockets.a
+	$(CC) -o $(BUILD_DIR)/test.o -c src/test.c $(CFLAGS) $(INCLUDES)
+
+$(BUILD_DIR)/mqtt_wss_log.o: src/mqtt_wss_log.c src/include/mqtt_wss_log.h
+	$(CC) -o $(BUILD_DIR)/mqtt_wss_log.o -c src/mqtt_wss_log.c $(CFLAGS) $(INCLUDES)
+
+$(BUILD_DIR)/mqtt_wss_client.o: src/mqtt_wss_client.c src/include/mqtt_wss_client.h src/include/ws_client.h src/include/common_internal.h $(BUILD_DIR)/common_public.o
+	$(CC) -o $(BUILD_DIR)/mqtt_wss_client.o -c src/mqtt_wss_client.c $(CFLAGS) $(INCLUDES)
+
+$(BUILD_DIR)/mqtt_ng.o: src/mqtt_ng.c src/include/mqtt_ng.h src/include/common_internal.h $(BUILD_DIR)/common_public.o
+	$(CC) -o $(BUILD_DIR)/mqtt_ng.o -c src/mqtt_ng.c $(CFLAGS) $(INCLUDES)
+
+$(BUILD_DIR)/common_public.o: src/common_public.c src/include/common_public.h
+	$(CC) -o $(BUILD_DIR)/common_public.o -c src/common_public.c $(CFLAGS) $(INCLUDES)
+
+libmqttwebsockets.a: $(BUILD_DIR)/mqtt_wss_client.o $(BUILD_DIR)/ws_client.o c-rbuf/build/ringbuffer.o $(BUILD_DIR)/c_rhash.o $(BUILD_DIR)/mqtt_wss_log.o $(BUILD_DIR)/mqtt_ng.o $(BUILD_DIR)/common_public.o
+	ar rcs libmqttwebsockets.a $(BUILD_DIR)/mqtt_wss_client.o $(BUILD_DIR)/ws_client.o c-rbuf/build/ringbuffer.o $(BUILD_DIR)/c_rhash.o $(BUILD_DIR)/mqtt_wss_log.o $(BUILD_DIR)/mqtt_ng.o $(BUILD_DIR)/common_public.o
+
+test: $(BUILD_DIR)/test.o libmqttwebsockets.a
+	$(CC) -o test $(BUILD_DIR)/test.o libmqttwebsockets.a `pkg-config --libs openssl` -lpthread $(CFLAGS)
+
+clean:
+	rm -f $(BUILD_DIR)/*
+	cd c-rbuf && $(MAKE) clean
+	rm -f test libmqttwebsockets.a
+
+install:
+
+dist:
+
+distdir:

mqtt_websockets/README.md

@@ -0,0 +1,12 @@
+# mqtt_websockets
+
+Library to connect MQTT client over Websockets Secure (WSS).
+Documentation is pending. Best way to figure out how to use the library it to look at `src\test.c`.
+
+## License
+
+The Project is released under GPL v3 license. See [License](LICENSE)
+
+Uses following git submodules:
+- **c-rbuf**: under LGPL 3 by underhood
+- **c_rhash**: GPL 3 by underhood

mqtt_websockets/c-rbuf/.github/workflows/codeql-analysis.yml

@@ -0,0 +1,67 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+  schedule:
+    - cron: '19 3 * * 1'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'cpp' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
+        # Learn more:
+        # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

mqtt_websockets/c-rbuf/.github/workflows/run-tests.yaml

@@ -0,0 +1,13 @@
+name: run-tests
+on:
+  push:
+  schedule:
+      - cron: '5 3 * * 0'
+  pull_request:
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - run: make
+      - run: ./test

mqtt_websockets/c-rbuf/.gitignore

@@ -0,0 +1,8 @@
+build/*
+!build/.keep
+libringbuffer.*
+test
+.vscode
+*.o
+.deps
+.dirstamp

mqtt_websockets/c-rbuf/LICENSE

@@ -0,0 +1,165 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

mqtt_websockets/c-rbuf/Makefile

@@ -0,0 +1,24 @@
+CFLAGS = -Wall -Wextra -std=c99
+
+all: libringbuffer.a libringbuffer.so build/ringbuffer_test.o
+	gcc -o test build/ringbuffer_test.o libringbuffer.a $(CFLAGS)
+
+libringbuffer.so: build/ringbuffer.o
+	gcc -shared -o libringbuffer.so build/ringbuffer.o
+
+libringbuffer.a: build/ringbuffer.o
+	ar rcs libringbuffer.a build/ringbuffer.o
+
+build/ringbuffer.o: src/ringbuffer.c src/ringbuffer_internal.h include/ringbuffer.h
+	gcc -o build/ringbuffer.o -c src/ringbuffer.c -Iinclude -fPIC $(CFLAGS)
+
+build/ringbuffer_test.o: libringbuffer.a tests/ringbuffer_test.c
+	gcc -o build/ringbuffer_test.o -c tests/ringbuffer_test.c -Iinclude $(CFLAGS)
+
+clean:
+	rm -f build/*
+	rm -f libringbuffer.a
+	rm -f libringbuffer.so
+	rm -f test
+
+distclean: clean