Skip to content

Latest commit

 

History

History
614 lines (577 loc) · 25.7 KB

TODO.md

File metadata and controls

614 lines (577 loc) · 25.7 KB
Raw

TODO - Frontend

Target: v1.2

TO DO

  • LogRhythm Beats mapping for Collection Configuration
    • Gmail Message Tracking Beat
    • GSuite Beat
    • Okta Beat
    • Qualys FIM Beat
    • Sophos Central Beat
  • Error reporting card to deal with many errors
    • Deduplicate error messages
  • Admin - MS SQL - Add Test button

TO FIX

  • Translations
    • Title for Pipeline Properties
    • Title for Pipeline Collection Builder
    • Breadcrumbs
  • Fix issue #40 - [FR] Offer to extract the .log field the same way the .message can be
  • Fix issue #45 - JSON key names including dots and empty JSON key names are not quoted in JQ as they should
  • Fix issue #48 - [ERROR] [failed to process log: input must be a string Filter: ascii_downcase/0 (RequiredOpCount:0)
  • Fix issue #51 - JQ Filter bug introduced in fix to issue #48

TO TEST

Target: v1.1

TO DO

  • Incorporate Axon style UI
    • Tables
    • Headers
      • Header on all pages
      • Logo in Header
        • for Dark mode
        • for Light mode
      • Breadcrumbs
      • Remove unecessary "Return" buttons
    • Action buttons style
    • Marketplace Template listing as Cards
  • LogRhythm Beats mapping for Collection Configuration
    • Prisma Cloud
    • Symantec WSS Beat
    • Microsoft Graph API Beat
    • Carbon Black Cloud Beat
    • Cisco AMP Beat
    • Duo Authentication Security Beat
    • Proofpoint Beat
    • Gmail Message Tracking Beat
    • GSuite Beat
    • Okta Beat
    • Qualys FIM Beat
    • Sophos Central Beat
  • Add support for new LogRhythm Beats via logrhythmBeat in pipeline.options.identificationStyle[]
    • In Tail
    • In Deploy
  • Error reporting card to deal with many errors
    • Deduplicate error messages
  • Admin - MS SQL - Add Test button
  • Open Collector - Stats and troubleshooting
    • Create embryo Manage Open Collector page
    • Import UI/UX work
    • Get Open Collector information (stats, containers, ...)
    • Implement managing tasks
      • Start Container
      • Stop Container
      • Restart Container
      • Export Configuration from LR container to file
      • Import Configuration from LR container from file
      • View Configuration from LR container
        • Save to file from Config Viewer
      • Export Logs from Container to file
      • View Logs from Container in real time
  • Landing page
    • Add a simple Wizard (👈 low value, low priority. Might resurect if users demand)
      • Ask for name for Log Source (👈 low value, low priority. Might resurect if users demand)
      • Create collection configuation (👈 low value, low priority. Might resurect if users demand)
        • Pick Shipper (👈 low value, low priority. Might resurect if users demand)
        • Pick type of collection (Flat File, Syslog, REST, etc...) (👈 low value, low priority. Might resurect if users demand)
        • Configure it (👈 low value, low priority. Might resurect if users demand)
      • Pick up Open Collector (👈 low value, low priority. Might resurect if users demand)
        • Add one if none in the list (👈 low value, low priority. Might resurect if users demand)
        • Offer to deploy required Shipper if not already on OC (👈 low value, low priority. Might resurect if users demand)
      • Create field mapping (👈 low value, low priority. Might resurect if users demand)
        • Run pre-configured Tail (👈 low value, low priority. Might resurect if users demand)
      • Recap page (👈 low value, low priority. Might resurect if users demand)
        • Ability to rename Log Source (👈 low value, low priority. Might resurect if users demand)
        • Ability to assign Log Source to other Open Collectors (👈 low value, low priority. Might resurect if users demand)
        • Ability to Enable / Commit the Log Source creation (👈 low value, low priority. Might resurect if users demand)

TO FIX

  • Translations
    • Title for Pipeline Properties
    • Title for Pipeline Collection Builder
    • Breadcrumbs
  • Shippers icons - Replace old LogRhythm logo with current one
  • Fix issue #40 - [FR] Offer to extract the .log field the same way the .message can be
  • Fix the height of Field editor in Mapping Builder (the breadcrumbs are taking more space than the old header)

TO TEST

Target: v1.0

TO DO

  • Incorporate Axon style UI
    • Left hand side navigation bar
    • Login page
    • Logout page
    • Cards
    • Fields
    • Dialogs
      • Standard Confirm dialog
      • OpenCollector properties
      • Pipeline properties
      • Account properties
      • Role properties
    • Tables
    • Headers
      • Breadcrumbs
    • Large buttons style
    • Action buttons style
    • Settings
  • LogRhythm Beats mapping for Collection Configuration
    • Azure Event Hub
      • With custom Mapping
      • Without custom Mapping
        • Implement skipping OC Pipeline deployment
        • Implement pick Azure Eventhub LS Type in SIEM
          • skip LST creation
          • skip MPE rule creation
          • skip Log Processing Policy creation
        • Implement disable Field Mapping edit
    • S3
    • WebHook
      • https
    • PubSub
    • Kafka
  • Create proper Beats Collection Configuration object, expanding from the notted notation in field names
  • Provide Beats's configuration to Backend when starting Tail
  • Add File drop field type to Collection Configuration
    • Internal template
    • Collection Configuration Editor
    • Pass file information to Backend during Tail
    • Pass file information to Backend during Deployment
  • Error reporting card to deal with many errors
    • Add sticky title and X close button
    • Make bottom panel sticky, with [Close] button
  • Landing page
    • News from EZ Market Place
    • Common tasks
    • Recent items
  • Provide Beats's configuration to Backend when starting Tail
  • Update link to the Wiki
    • Move to docs.logrhythm.com
    • Use Staging Doc during Dev

TO FIX

  • JSON to Yaml Collection config fails with TypeError: jsonConfig.forEach is not a function
  • Mapping Field Editor - @ sign in field name is not escaped and cause JQ to fail import

TO TEST

Target: v0.9

TO DO

  • LogRhythm Beats mapping for Collection Configuration
    • Azure Event Hub
      • With custom Mapping
      • Without custom Mapping
    • S3
    • Azure Event Hub
      • With custom Mapping
      • Without custom Mapping
    • WebHook
      • https
    • PubSub
    • Kafka
    • ...
  • Add a simple Wizard from landing page
    • Ask for name for Log Source
    • Create collection configuation
      • Pick Shipper
      • Pick type of collection (Flat File, Syslog, REST, etc...)
      • Configure it
    • Pick up Open Collector
      • Add one if none in the list
      • Offer to deploy required Shipper if not already on OC
    • Create field mapping
      • Run pre-configured Tail
    • Recap page
      • Ability to rename Log Source
      • Ability to assign Log Source to other Open Collectors
      • Ability to Enable / Commit the Log Source creation
  • MS SQL Admin
    • MS SQL Connection
      • Add MS SQL Connection admin page
      • Collect MS SQL credentials
      • Push to API
      • Disable fields while loading
      • Remind user to go and update DB after change
    • Update EMDB
      • Add Update EMDB admin page
      • Collect MS SQL, EZ DB, EZ_Get_Versions view and EZ DB's content versions
      • Display SQL, EZ DB, EZ_Get_Versions view and EZ DB's content versions
      • Gather provoleged credentials
      • Call API to Update EMDB
  • Missing MS SQL configuration
    • Check MS SQL needs configuration right after Login in
    • Add warning in navigation bar
    • Add notification on Admin menu icon
    • Add notification on Admin/SIEM/Manage MS SQL Connection button
  • Internationalisation
    • Make all text internationalised
    • Add Language switch in Settings
    • Add Language switch on Login page
  • MarketPlace notification indicator to show on Notification sub-menu too
  • Get and show status of server components on Login page
  • Disable Logon button if no SQL server is available

TO FIX

  • Identifiers to EZ Market seems incorect/misformatted/ignored by EZ Market
  • Width of Advanced and Settings menus. Too narrow for some languages.
  • Erroneous "SIEM Not Connected" alert message on Windows deployment
  • Issue #12 - [BUG] Pipeline import from EZ Market Place - Wrong icon colour on "Import Both" button in Day mode
  • Issue #11 - [BUG] Fields Mapping - Incorrect LR MPE tags for Outbound bytes type fields
  • Top bar not stuck to top left of screen in Admin : SIEM : Update EMDB

TO TEST

  • Translated message in deployment steps roll over status (especially Skipped steps)

Target: v0.8

TO DO

  • RBAC
    • Add User Admin page
    • Add Roles Admin page
    • Add landing Admin page
    • Add navigation in Users and Roles Admin pages
  • General
    • Settings - Only show Backend settings in Dev Mode
    • Error messages
      • Add pop up with error details and pointer to Wiki
      • Write Wiki with error explanation
  • Mapping Editor
    • Warn user if Socket is not connected to Backend (as Tail feature needs it)
    • Automatically start Parsing when starting Tail
      • Revert this change, as better optimistion now in place
    • Optimise Parsing Start/Stop
      • Automatically start Parsing when entries are added to the Queue
      • Automatically stop Parsing when the Queue is fully processed
    • Change scheduling method for the Background Processing to allow changes in the Settings to be picked up once already running
    • Change 'Start/Stop background processing' button to a simple indicator, or remove
    • Handle Sub Rules (👈 low value, low priority. Might resurect if users demand)
    • Resurface the .message feature
      • In UI (menu Settings)
      • Send to API when Saving
      • When generating JQ Transform
        • In Field Mapping Editor
        • In deployment
      • Allow for different holder (👈 low value, low priority. Might resurect if users demand)
        • .message
        • .response
        • Configurable in the Collection Template
    • Externalise Collection Templates
    • Add a way to import log samples
  • Provide feedback if failing to connect over SSH
    • In Open Collectors list page
      • On version check
      • On Shipper deployment
    • In Tail
    • In Deployment
  • Warn user if Socket is not connected to Backend (as some features needs it)
    • On Shipper deployment
  • Market Place
    • Find a Free-tiered public database (👈 using journey.logrhythm.com)
    • Find a Free-tiered public hosting (for Admin and API) (👈 using journey.logrhythm.com)
    • Find / register domain (👈 using journey.logrhythm.com)
    • Get notifications from the Market Place
    • List available Pipeline Templates
    • Import Pipeline Template
      • Collection
        • In new Pipeline
        • In existing Pipeline
      • Mapping
        • In new Pipeline
        • In existing Pipeline
      • From the target Pipeline itself
        • Collection
        • Mapping
    • Publisher profile's properties
      • Create Publisher profile with user's Pseudo-name
      • Edit Publisher profile
    • Export to Market Place
      • Deal with non-existing Publisher Profile
      • Pipeline name
      • Icon / Visual
      • Readme
      • Collection
        • Sanitization of secrets
      • Fields Mapping
        • Sanitization of export
    • Market Place Admin
      • Approval / Review / Moderation of submissions
      • Change Status of submission
        • Under review
        • Rejected
        • Approved
      • Change Visibility of submission
        • Visible
        • Hidden
      • Delete submission
  • Import/Export of Collection
    • Import Collection from file
    • Import Collection from Copy/Paste
    • Export Collection to file
    • Export Collection to Copy/Paste (👈 low value, low priority. Might resurect if users demand)
  • Import/Export of Fields Mapping
    • Import Mapping from file
    • Import Mapping from Copy/Paste (👈 low value, low priority. Might resurect if users demand)
    • Export Mapping to file
    • Export Mapping to Copy/Paste (👈 low value, low priority. Might resurect if users demand)
    • Update Mapping import from File to re-use new importFromEZImportableConfig function
  • LogRhythm Beats mapping for Collection Configuration
    • WebHook
      • http (Contribution from @Jt3kt)
    • ...
  • Open Collector Properties
    • SUDO requirement (👈 we are using a non-interactive TTY, so we can't provide any extra creds for sudo)
      • Collect optional SUDO Username (👈 we are using a non-interactive TTY, so we can't provide any extra creds for sudo)
      • Collect optional SUDO Password (👈 we are using a non-interactive TTY, so we can't provide any extra creds for sudo)
  • Pipeline - Deployment - Change Pipeline status to Ready when deployed successfully
  • Shipper Installation
    • Update Shippers list with jsBeat v1.1.5

TO FIX

  • Fan Out feature - OC refuses to import JQ Transform (👈 not an issue. Was only caused on Dev system)
  • Admin - Roles - Creates new duplicated Role instead of updating existing one when changing the Priviledge
  • Ability to collect data from external site (Github, ...)
  • JQ small nags
    • .message is commented in Transform template
    • Filter uses un-cleaned Stream Name for device_type
  • Change "priviledged" to "privileged", through out
    • Actions (Store)
    • UI
  • Change typical message ('Error updating persistance layer') when API returns an error to a more generic, or more specific if necessary
  • Shipper installer - Content Security Policy prevents download of shippers_url.json from GitHub
  • Square avatars for navigation bar

TO TEST

  • Fan Out feature
  • .message feature

Target: v0.7

TO DO

  • OpenCollectors list
    • Load SSH Token from file instead of copy/paste in field (👈 Keep it as is for now, as LRCloud users won't have access to their local files)
    • Use alternative to Filebeat
      • LR Rest Beat
        • Install LR Rest
        • Communicate each step to Frontend
        • Display Install progress in Frontend
        • Create LR Rest Beat configuration
        • Use LR Rest Beat for Tail
      • jsBeat
    • Move OS version to the left of Open Collector Version
    • Change Filebeat version to Shippers version
    • Collect different Shippers versions from Backend
      • Collect active OC Beats versions
        • Use merged API (CheckOpenCollectorAndBeatsVersions) intead of CheckOCVersion
        • Create icons for the LogRhythm Beats
      • Collect jsBeat version
    • Load and display different Shippers and versions
    • Add action button to Shippers version column, when empty
      • Install
    • Add action buttons to Actions column
      • Install Shipper
      • Upgrade Shipper
      • Uninstall Shipper
  • Pipeline list
    • Ability to rename Pipeline in Pipeline list
    • Ability to change Status Pipeline in Pipeline list
    • Ability to deploy Log Source to Open Collectors for Production
      • Add Deployed status
  • Pipeline properties page
    • List Deployments
    • Delete Deployment
    • Ability to assign Log Source to Open Collectors for Production
  • Assign Log Source to Open Collectors for Production page
    • Create new Edit Deployment page
    • Get list of OC Log Sources from Backend
    • Try to map EZ OCs with OC Log Sources
    • Flag EZ OCs that do not have the right Shipper for the Pipeline
    • Capture user selection of OC Log Source
    • Kick off deployment of Stream on OC and creation of LogSource and all the chain in EMDB
      • Deployment of Stream on OC
        • Finish rigging Front and Back (openCollector, beat and stream parameters)
      • Creation of LogSource and all the chain in EMDB
      • Add Skipped step status
      • Better icon colours
      • Skipped and Error information panel
      • Handle Sub Rules (👈 irrelevant until FieldMappingEditor allow for better Sub Rule design)
      • Add deployment to list and persist
      • Add indicator to the Deployment list of already deployed Stream(s)
      • Offer action to un-deploy
        • Button (-) and action
  • Hide Settings page for Prod
  • Create Logging shared Lib to push logs to Console and Windows Journal
  • Day mode
    • Polish CSS / Styling to get good color scheme in Day mode too
      • Header bar background
      • Roll over line in Mapping
      • Frequency line graphs in Mapping
      • Drop down list background
      • Text colour
      • Navigation icons colour
    • Add Day/Dark mode switch on Logon page/card
  • Highlight the current active page in the navigation bar
  • Pipeline Mapping Edit
    • Use the right beat name, instead of pipeline name in beatName ()
    • Use the real Beat name in the JQ
    • Rename the JQ Filter area to match what the backend names it
    • Persist extractMessageFieldOnly (and use it in DeploymentEdit when ready)
  • Collection Builder Edit
    • Add way to encrypt password
    • Add way to encrypt normal text and multi-lines

TO FIX

  • Open Collector List - Installation progress bar showing always full, even when progress is not yet 100%
  • Field Mapping - JQ Transform - Put the Pipeline name in .output.device_name, and the Beat name in .output.beatname, so to comply with LS Virtualisation templates
  • Rename "New Pipeline Details" popup title to "Pipeline Details"
  • NPM modules with vulnerabilities
  • Do not assume .message is always present (LR Generic Beat puts data in .response) when generating JQ Transform
  • Incomplete JQ Transform once deployed
  • Pipeline Properties - Deployment shows an item for brand new (not yet deployed) Pipeline
  • Collection Editor - Drop down menu background colour wrong when in Light mode
  • Shipper installer - Content Security Policy prevents download of shippers_url.json from GitHub
    • app.ab179f5f.js:1 Refused to connect to 'https://raw.githubusercontent.com/TonyMasse/EZ-Cloud/main/resources/shippers_url.json' because it violates the following Content Security Policy directive: "default-src 'self'". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

TO TEST

  • Trying to deploy on dead/non-existent OC
  • Trying to deploy on OC with jsBeat

Target: v0.6

TO DO

  • OpenCollectors list
    • Load SSH Token from file instead of copy/paste in field
    • Use alternative to Filebeat
      • jsBeat
        • Install NodeJS
          • Store NodeJS download URL on GitHub
        • Install jsBeat
          • Store jsBeat download URL on GitHub
        • Communicate each step to Frontend
        • Display Install progress in Frontend
        • Create jsBeat configuration
          • Create configuration template
          • Update function that create fresh config
          • Update function that create output config (ready for Beat to use)
        • Use jsBeat / FlatFile for Tail
      • LR Rest Beat
        • Install LR Rest
        • Communicate each step to Frontend
        • Display Install progress in Frontend
        • Create LR Rest Beat configuration
        • Use LR Rest Beat for Tail
      • Update UI to not be Filebeat centric
        • Download button and function
        • Copy to Clipboard button and function
        • Add icons for Shippers
  • Pipeline list
    • Ability to rename Pipeline in Pipeline list
    • Add Collection Shipper in Pipeline list
    • Add Collection Method in Pipeline list
    • Add Mapping stats in Pipeline list
    • Change status of Pipeline when adding Collection or Mapping
      • when adding Collection
      • when adding Mapping
    • Make Sorting by Status to make sense (Ready > Dev > New)
  • Hide Settings page for Prod
  • Create Logging shared Lib to push logs to Console and Windows Journal
  • Prevent user from using Tail for Pipeline set to HTTP-JSON collection (👈 irrelevant as we are moving away from Filebeat)
  • Day mode
    • Polish CSS / Styling to get good color scheme in Day mode too
      • Header bar background
      • Roll over line in Mapping
      • Drop down list background
      • Text colour
      • Navigation icons colour
  • Add a simple Wizard from landing page
    • Ask for name for Log Source
    • Create collection configuation
      • Pick Shipper
      • Pick type of collection (Flat File, Syslog, REST, etc...)
      • Configure it
    • Pick up Open Collector
      • Add one if none in the list
    • Create field mapping
      • Run pre-configured Tail
    • Recap page
      • Ability to rename Log Source
      • Ability to assign Log Source to other Open Collectors
      • Ability to Enable / Commit the Log Source creation
  • Field Mapping
    • Show popup for ERROR messages
    • Add "Console" view, and display STDERR messages there
    • Add "Console" view, and display STDOUT messages there

TO FIX

  • Damn Tedious saving NULL as 'null' in OC list
  • Open Collector List - Installation progress bar showing always full, even when progress is not yet 100%
  • Dark / Day mode - Not loading from Local Storage correctly

TO TEST

  • Tail with jsBeat
  • Collection with jsBeat

Target: v0.5-rc2

TO DO

  • OpenCollectors list
    • Get API to check OC host for OC / OS / FB versions, and display it in OC List (Frontend and Backend)
      • Get SSH config for a given OC UID (BACKEND)
    • Hide LS columns in OC list
    • Hide Open OC button in OC list
    • Load SSH Token from file instead of copy/paste in field
    • Install Filebeat when click on INSTALL button (BACKEND)
      • Store Filebeat download URL on GitHub
      • Gather Filebeat URL from GitHub
      • Offer choice to user to select package/version, fall back to URL to 7.13
      • Download Filebeat
      • Run checksum
      • Install
      • Communicate each step to Frontend
      • Display Install progress in Frontend
  • Pipeline list
    • Ability to rename Pipeline in Pipeline list
    • Add Collection Type in Pipeline list
    • Add Mapping stats in Pipeline list
    • Change status of Pipeline when adding Collection or Mapping
  • Use OC IP when tailing in Mapping
  • Hide Settings page for Prod
  • Create Logging shared Lib to push logs to Console and Windows Journal
  • Prevent user from using Tail for Pipeline set to HTTP-JSON collection
  • Hide empty dashboard (or put something in it)
  • Day mode
    • Add Dark/Day mode switch under Settings
    • Save user preference in web browser's localStorage
    • Polish CSS / Styling to get good color scheme in Day mode too

TO FIX

  • OC password not updated in Store
  • OC Load tries to update TableLoading read only computed
  • Damn Tedious saving NULL as 'null' in OC list
  • v0.5-rc1 limitations
    • Split Syslog collection into Syslog over TCP and Syslog over UDP
    • Stop using SSH details from /src/config/ssh.json on tail, and use the specified Pipeline default Collector instead
  • Clear collection config after clicking on Delete Configuration (Pipeline Property page)

TO TEST

  • SSH via Token
  • Filebeat on-demand deployment

Target: v0.5-rc1

TO DO

  • OpenCollectors list
    • Get API to check OC host for OC / OS / FB versions, and display it in OC List (Frontend and Backend)
      • Get SSH config for a given OC UID (BACKEND)
    • Hide LS columns in OC list
    • Hide Open OC button in OC list
    • Load SSH Token from file instead of copy/paste in field
    • Install Filebeat when click on INSTALL button (BACKEND)
      • Store Filebeat download URL on GitHub
      • Gather Filebeat URL from GitHub
      • Offer choice to user to select package/version, fall back to URL to 7.13
      • Download Filebeat
      • Run checksum
      • Install
      • Communicate each step to Frontend
      • Display Install progress in Frontend
  • Pipeline list
    • Ability to rename Pipeline in Pipeline list
    • Add Collection Type in Pipeline list
    • Add Mapping stats in Pipeline list
    • Change status of Pipeline when adding Collection or Mapping
  • Use OC IP when tailing in Mapping
  • Hide Settings page for Prod
  • Create Logging shared Lib to push logs to Console and Windows Journal
  • Prevent user from using Tail for Pipeline set to HTTP-JSON collection
  • Hide empty dashboard (or put something in it)

TO FIX

  • OC password not updated in Store
  • OC Load tries to update TableLoading read only computed
  • Damn Tedious saving NULL as 'null' in OC list

TO TEST

  • SSH via Token
  • Filebeat on-demand deployment