Merge pull request zowe#77 from zowe/zebra_dev

Update main branch with latest code

.github/workflows/pipeline.yml

@@ -0,0 +1,73 @@
+name: Build and Test Workflow
+on:
+  push:
+    branches:
+      - main 
+  pull_request:
+    types: [opened, synchronize]
+  workflow_dispatch:
+    inputs:
+      PERFORM_RELEASE:
+        description: '[Release] perform release'
+        required: false
+        default: 'false'
+        type: boolean
+
+
+jobs:
+  check-permission:
+    runs-on: ubuntu-latest
+    steps:
+      # this action will fail the whole workflow if permission check fails
+      - name: check permission
+        uses: zowe-actions/shared-actions/permission-check@main
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+  build-test:
+    runs-on: ubuntu-latest
+    steps: 
+      - name: '[Prep 1] Checkout'
+        uses: actions/checkout@v2
+
+      - name: '[Prep 2] Cache node modules'
+        uses: actions/cache@v2
+        with:
+          # npm cache files are stored in `~/.npm` on Linux/macOS
+          path: |
+            ~/.npm
+            ~/.nvm/.cache
+            ~/.nvm/versions
+          key: ${{ runner.os }}-build-cache-node-modules-${{ hashFiles('**/package-lock.json') }}
+          restore-keys: |
+            ${{ runner.os }}-build-cache-node-modules-
+          
+      - name: '[Prep 3] Setup Node'
+        uses: actions/setup-node@v2
+        with:
+          node-version: 8.11.4
+
+      - name: '[Build] Nodejs project build'
+        run: npm --prefix src run build
+
+      - name: '[Test] Nodejs project unit test'
+        run: npm --prefix src run test
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: [check-permission, build-test]
+    if: ${{ success() && github.event_name == 'push' }}
+    steps:
+      - name: '[Deploy] Deploy to demo server'
+        uses: appleboy/ssh-action@master
+        if: ${{ success() && github.event_name == 'push' }}
+        with:
+          host: ${{ secrets.DEMO_SERVER_HOST }}
+          username: ${{ secrets.DEMO_SERVER_USERNAME }}
+          key: ${{ secrets.DEMO_SERVER_KEY }}
+          script: |
+            cd /root/zebra
+            git pull origin main
+            
+
+

Documentation/Contributor Documentation/1. Zebra Design.md

@@ -0,0 +1,58 @@
+## Design Approach
+  The design approach used in the project is based on the following:  
+  - ***Data Flow Design***  
+    SMF/RMF data is retrieved over the internet in XML format. The XML is then passed to the parsing engine and an output in JSON/CSV format is produced. The outputs for Realtime RMF reports are saved into a NoSQL database. Performance metrics from Realtime reports are exposed through an endpoint and Scraped Using Prometheus.
+  - ***Architecture Design***  
+    The project will follow a Three Layer Architecture so that the objects in the system as a whole can be organized to best separate concerns and prepare for distribution and reuse.
+  - ***Graphical Tool***  
+    Prometheus serves as a data source for building Realtime dashboards using a graphical tool. The project makes use of an open source graphical tool (Grafana) for creating real time monitoring dashboards from parsed RMF data.
+
+## Design Pattern:
+  The Application Classes were factored into the following 3 layers:  
+  - ***The App-Server Layer***  
+    This layer consist of the following components:  
+      - ***HTTP GET Functions***  
+        These functions send a HTTP GET Request to RMF DDS server for RMF Monitor III or RMF Post Processor data.  
+      - ***Parser***  
+        This consist of functions for parsing RMF Monitor III and RMF Post Processor data.  
+      - ***Model***  
+        This consist of Schema definitions for data to be saved into MongoDB.  
+  - ***The data Layer***  
+    This layer contains the two data warehouse for the project:  
+      - ***Prometheus***  
+        Prometheus saves Realtime Performance metrics exposed by the parsing Engine.  
+      - ***MongoDB***  
+        MongoDB saves Realtime data output from the parsing Engine.  
+  - ***The Presentation Layer***  
+    This layer consist of:  
+      - ***Zowe API Catalog/Browser***  
+        This tool displays JSON output from the parsing Engine.  
+      - ***Grafana***  
+        This tool displays dashboard from real time data output from the parsing Engine.  
+
+## System Design Consideration
+  1. ***Directories***  
+      - ***Root Directory***  
+        This directory consist of:  
+          - ***App.js file***  
+            This is the file contain app/server code.    
+          - ***PromMetricsv1.js***  
+            This file contains function that expose real time CPU utilization metrics using prom-client library.  
+          - ***Mongov1.js***  
+            This file contains functions that save real time parsing engine output to MongoDB.  
+      - ***App Server Directory***  
+        This directory Consist of:  
+          - ***v1_Controllers folder***  
+            This folder is made up of files that contain functions controlling the Events/Actions of the parsing Engine.  
+          - ***Parser folder***  
+            This folder is made up of files containing functions for parsing real time and post processor data by the parsing Engine.  
+          - ***Models folder***  
+            This folder contains Schema files for saving data to MongoDB as well as db.js file which contains functions for connecting to MongoDB.  
+          - ***Routes folder***  
+            This folder contain files for mapping URL Endpoints to controller functions of the parsing Engine.  
+      - ***Config***
+        This directory consist of:  
+          - ***Zconfig.json***  
+            This file contains the configurations of the parsing engine in JSON format.
+  2. ***Exception Handling***  
+    Exception Handlers occur at the application level. Errors are displayed in JSON format/Error Pages.  

Documentation/Contributor Documentation/2. Architecture Design.md

@@ -0,0 +1,10 @@
+## Architecture
+  1. ***User (request)***: User send a request to the zebra App using any of its recognized URL(s).  
+  2. ***GetRequest***: zebra app send a get request to RMF DDS server for post processor or Monitor III data depending on the URL specified by the user. This happens through the use of DDS HTTP API. DDS server returns an XML file.  
+  3. ***Parser***: The XML file returned is feed to RMF post processor parser, RMF monitor III parser or CPU utilization parser depending on the URL specified by the user. The parser returned a JSON.  
+  4. ***User (response)***: User can view parsed RMF report using a browser or Zowe API Catalog.  
+  5. ***Prom-client***: The JSON returned by CPU utilization parser is used to create custom Prometheus metrics using prom-client library. The custom Prometheus metrics are exposed via /prommetric endpoint by the zebra app.  
+  6. ***Prometheus server***: this server scrape custom Prometheus metrics from zebra app /prommetric endpoint  
+  7. ***Grafana***: Grafana dashboards are built by connecting Grafana to Prometheus server. This dashboard shows CPU utilization chart I Realtime.  
+  8. ***MongoDB***: Realtime data output from the parsing engine is saved to MongoDB database.  
+  ![](/Documentation/Images/ZebraArchitecture.png)

Documentation/Contributor Documentation/3. Class Diagram.md

@@ -0,0 +1,14 @@
+## Class Diagram
+![](/Documentation/Images/classdiagram.png)  
+The class diagram shows directional association between the project classes. ***Classes*** are represented in the project by files with ***“.js”*** as file extension.  
+***Routers*** classes are the starting point of communication with other classes as they map incoming API request from users to controller and parser classes. The Router class names contains the word “Router” at the end. These classes are responsible for recognizing the app’s endpoint.  
+***Controller*** class names contains the word ***“Controller”*** at the end and consist of functions for: 
+- Sending GET Request to DDS server
+- Sending XML to Parser
+- Filtering parsed JSON based on user specified parameters in the URL
+- Displaying the JSON response to User
+- Adding Configuration settings to the App  
+
+***Parser*** classes consist of functions for parsing XML to JSON. RMF monitor III parser has a single function which can parse all RMF III XML reports due to their format consistency. Monitor I parser (RMFPPparser) has two functions for parsing CPU post processor and Workload Postprocessor data.  
+***Mongov1*** and ***PromMetricsV1*** classes have a ***setInterval()*** function which makes them run continuously at an interval specified by the user in the app configuration.   They both interact with monitor III router to retrieve a JSON.  
+***MongoV1*** class is responsible for retrieving real time ***CPC, PROC and USAGE*** monitor III reports and fed them into ***MongoDB*** while ***PromMetricsV1*** class is responsible for retrieving real time CPC report JSON, filtering the JSON for Total utilization, MSU value and Effective Utilization values. It then use prom client library to create custom gauge metric. The custom metrics are saved into a register. The metrics in the register are exposed via an Endpoint in the mainRouter class. The exposed metrics can then be scraped by ***Prometheus*** and real time charts can be plotted using ***Grafana***.  

Documentation/Contributor Documentation/4. Activity Diagram.md

@@ -0,0 +1,16 @@
+## Activity Diagram  
+  1. ***RMF Monitor III Activity Diagram***  
+    ![](/Documentation/Images/rmf3ad.png)  
+    RMF Monitor III Activities starts when a User trigger a request using the /v1/{lpar}/rmf3 Endpoint, This leads to a series of activities involving v1Router, RMF3Controller and RMF3Parser. A Condition exist to check is the Request to DDS server is successful, this condition determines the data that get returned to RMF3Controller and finally to the User.  In the End, the user receives a JSON Response containing parsed RMF III report or Error Message in case of a failed request to DDS.  
+  2. ***RMF Post Processor Activity Diagram***  
+    ![](/Documentation/Images/rmf1ad.png)  
+    RMF Monitor I Activities starts when a User trigger a request using the /v1/{lpar}/rmfpp Endpoint, This leads to a series of activities involving v1Router, RMFPPController and RMFPPParser. Two conditions exist in the activity flow of RMF post Processor. First is the condition that checks if the request to DDS server is successful. The Second one checks the value of the Report parameter specified by the user during a request. The value of the report parameter is used to determine the parser for the returned by the request to DDS.  
+  3. ***Static XML File Activity Diagram***
+    ![](/Documentation/Images/rmfstaticad.png)  
+    Static XML File to JSON Activities starts when a User trigger a request using the /static Endpoint, This leads to a series of activities involving staticXMLRouter, staticXMLFileController and RMFPPParser. A re-use of the RMFPPParser occurs here. Two conditions exist in the activity flow of static File to JSON as well. First is the condition that checks if reading the static file specified by the user in the URL’s file (takes file path as value) parameter is successful. The Second one checks the value of the Report parameter specified by the user during a request. The value of the report parameter is used to determine the parser for the returned by the request to DDS.  
+  4. ***MongoV1 Activity Diagram***  
+    ![](/Documentation/Images/mongoad.png)  
+    Activities of the Mongo class do not require user intervention. A setInterval function runs continuously at an interval specified in the Apps Configuration. This function serves as a trigger and make use of RMF III Activities to retrieve and store JSON into MongoDB.  
+  5. ***Prometheus Metrics Activity Diagram***  
+    ![](/Documentation/Images/cpuad.png)  
+    Just like in Mongo Class, Activities of the prometheus Realtime Metrics (PromMetricsv1) class do not need user intervention. The setInterval function triggers RMF III Activities which returns a JSON. Through a call back function, Prometheus Realtime Metrics class filters the JSON and create Prometheus custom gauge metrics for Effective utilization, MSU and Total utilization values. These Metrics are then saved to a prom-client library register.