Merging upstream updates

.github/workflows/sigma-test.yml

@@ -8,7 +8,7 @@ on:
     branches:
       - "*"
   pull_request:
-    branches: [ master ]
+    branches: [ master, oscd ]
 
 jobs:
   test-sigma:
@@ -22,10 +22,12 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -r tools/requirements.txt -r tools/requirements-devel.txt
+        pip install pipenv
+        pipenv lock
+        pipenv install --dev --deploy
     - name: Test Sigma Tools and Rules
       run: |
-        make test
+        pipenv run make test
     - name: Test SQL(ite) Backend
       run: |
-        make test-backend-sql
+        pipenv run make test-backend-sql

CHANGELOG.md

@@ -6,6 +6,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 from version 0.14.0.
 
+## 0.19.1 - 2021-02-28
+
+### Changed
+
+* Added LGPL license to distribution
+
+## 0.19 - 2021-02-23
+
+### Added
+
+* New parameters for Elastic backends
+* Various field mappings
+* FireEye Helix backend
+* Generic log source image_load
+* Kibana NDJSON backend
+* uberAgent ESA backend
+* SumoLogic CSE backend
+
+### Changed
+
+* Updated mdatp backend fields
+* QRadar query generation optimized
+* MDATP: case insensitive search
+
+### Fixed
+
+* Fixing Qradar implementation for create valid AQL queries
+* Nested conditions
+* Various minor bug fixes
+
 ## 0.18.1 - 2020-08-25
 
 Release created for technical reasons (issues with extended README and PyPI), no real changes done.

Makefile

@@ -57,8 +57,9 @@ test-sigmac:
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t arcsight -c tools/config/arcsight.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t arcsight-esm -c tools/config/arcsight.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t qradar -c tools/config/qradar.yml rules/ > /dev/null
-	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t stix -c tools/config/stix.yml -c tools/config/stix-qradar.yml -c tools/config/stix-windows.yml rules/ > /dev/null
+	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t stix -c tools/config/stix-custom.yml -c tools/config/stix-shifter.yml -c tools/config/stix2.0.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t limacharlie -c tools/config/limacharlie.yml rules/ > /dev/null
+	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t chronicle -c tools/config/chronicle.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t carbonblack -c tools/config/carbon-black.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t qualys -c tools/config/qualys.yml rules/ > /dev/null
 	$(COVERAGE) run -a --include=$(COVSCOPE) tools/sigmac -rvdI -t netwitness -c tools/config/netwitness.yml rules/ > /dev/null

Pipfile

@@ -10,13 +10,16 @@ elasticsearch = "~=7.6"
 elasticsearch-async = "~=6.2"
 pytest = "~=5.4"
 colorama = "*"
+setuptools = "*"
+stix2 = "*"
+attackcti = "*"
 
 [packages]
-requests = "~=2.23"
-urllib3 = "~=1.25"
+requests = "~=2.25"
+urllib3 = "~=1.26"
 progressbar2 = "~=3.47"
 pymisp = "~=2.4.123"
 PyYAML = "~=5.1"
 
 [requires]
-python_version = "3.6"
+python_version = "3.8"

README.md

@@ -32,17 +32,17 @@ The SANS webcast on Sigma contains a very good 20 min introduction to the projec
 
 # Use Cases
 
-* Describe your detection method in Sigma to make it sharable
+* Describe your detection method in Sigma to make it shareable
 * Write your SIEM searches in Sigma to avoid a vendor lock-in
 * Share the signature in the appendix of your analysis along with IOCs and YARA rules
 * Share the signature in threat intel communities - e.g. via MISP
 * Provide Sigma signatures for malicious behaviour in your own application
 
 # Why Sigma
 
-Today, everyone collects log data for analysis. People start working on their own, processing numerous white papers, blog posts and log analysis guidelines, extracting the necessary information and build their own searches and dashboard. Some of their searches and correlations are great and very useful but they lack a standardized format in which they can share their work with others. 
+Today, everyone collects log data for analysis. People start working on their own, processing numerous white papers, blog posts and log analysis guidelines, extracting the necessary information and build their own searches and dashboard. Some of their searches and correlations are great and very useful but they lack a standardized format in which they can share their work with others.
 
-Others provide excellent analyses, include IOCs and YARA rules to detect the malicious files and network connections, but have no way to describe a specific or generic detection method in log events. Sigma is meant to be an open standard in which such detection mechanisms can be defined, shared and collected in order to improve the detection capabilities for everyone. 
+Others provide excellent analyses, include IOCs and YARA rules to detect the malicious files and network connections, but have no way to describe a specific or generic detection method in log events. Sigma is meant to be an open standard in which such detection mechanisms can be defined, shared and collected in order to improve the detection capabilities for everyone.
 
 ## Slides
 
@@ -52,7 +52,7 @@ See the first slide deck that I prepared for a private conference in mid January
 
 # Specification
 
-The specifications can be found in the [Wiki](https://github.com/Neo23x0/sigma/wiki/Specification). 
+The specifications can be found in the [Wiki](https://github.com/Neo23x0/sigma/wiki/Specification).
 
 The current specification is a proposal. Feedback is requested.
 
@@ -62,9 +62,9 @@ The current specification is a proposal. Feedback is requested.
 
 Florian wrote a short [rule creation tutorial](https://www.nextron-systems.com/2018/02/10/write-sigma-rules/) that can help you getting started.
 
-## Rule Usage 
+## Rule Usage
 
-1. Download or clone the respository
+1. Download or clone the repository
 2. Check the `./rules` sub directory for an overview on the rule base
 3. Run `python sigmac --help` in folder `./tools` to get a help on the rule converter
 4. Convert a rule of your choice with `sigmac` like `./sigmac -t splunk -c tools/config/generic/sysmon.yml ./rules/windows/process_creation/win_susp_whoami.yml`
@@ -106,7 +106,7 @@ merges multiple YAML documents of a Sigma rule collection into simple Sigma rule
 
 ```bash
 usage: sigmac [-h] [--recurse] [--filter FILTER]
-              [--target {arcsight,es-qs,es-dsl,kibana,xpack-watcher,elastalert,graylog,limacharlie,logpoint,grep,netwitness,powershell,qradar,qualys,splunk,splunkxml,sumologic,fieldlist,mdatp,ee-outliers}]
+              [--target {sqlite,netwitness-epl,logpoint,graylog,netwitness,arcsight,carbonblack,es-rule,ala,elastalert-dsl,splunkxml,fieldlist,sysmon,arcsight-esm,kibana,csharp,qualys,powershell,es-qs,mdatp,humio,grep,qradar,logiq,sql,sumologic,ala-rule,limacharlie,elastalert,splunk,stix,xpack-watcher,crowdstrike,es-dsl,ee-outliers}]
               [--target-list] [--config CONFIG] [--output OUTPUT]
               [--backend-option BACKEND_OPTION] [--defer-abort]
               [--ignore-backend-errors] [--verbose] [--debug]
@@ -131,13 +131,13 @@ optional arguments:
                         tag that must appear in the rules tag list, case-
                         insensitive matching. Multiple log source
                         specifications are AND linked.
-  --target {arcsight,es-qs,es-dsl,kibana,xpack-watcher,elastalert,graylog,limacharlie,logpoint,grep,netwitness,powershell,qradar,qualys,splunk,splunkxml,sumologic,fieldlist,mdatp}, -t {arcsight,es-qs,es-dsl,kibana,xpack-watcher,elastalert,graylog,limacharlie,logpoint,grep,netwitness,powershell,qradar,qualys,splunk,splunkxml,sumologic,fieldlist,mdatp}
+  --target {arcsight,es-qs,es-dsl,kibana,xpack-watcher,elastalert,graylog,limacharlie,logpoint,grep,netwitness,powershell,qradar,qualys,splunk,splunkxml,sumologic,fieldlist,mdatp,devo}, -t {arcsight,es-qs,es-dsl,kibana,xpack-watcher,elastalert,graylog,limacharlie,logpoint,grep,netwitness,powershell,qradar,qualys,splunk,splunkxml,sumologic,fieldlist,mdatp,devo}
                         Output target format
   --target-list, -l     List available output target formats
   --config CONFIG, -c CONFIG
                         Configurations with field name and index mapping for
                         target environment. Multiple configurations are merged
-                        into one. Last config is authorative in case of
+                        into one. Last config is authoritative in case of
                         conflicts.
   --output OUTPUT, -o OUTPUT
                         Output file or filename prefix if multiple files are
@@ -172,13 +172,13 @@ Translate a whole rule directory and ignore backend errors (`-I`) in rule conver
 ```
 tools/sigmac -I -t splunk -c splunk-windows -f 'level>=high' -r rules/windows/sysmon/
 ```
-#### Rule Set Translation with Custom Config 
+#### Rule Set Translation with Custom Config
 Apply your own config file (`-c ~/my-elk-winlogbeat.yml`) during conversion, which can contain you custom field and source mappings
 ```
 tools/sigmac -t es-qs -c ~/my-elk-winlogbeat.yml -r rules/windows/sysmon
 ```
 #### Generic Rule Set Translation
-Use a config file for `process_creation` rules (`-r rules/windows/process_creation`) that instructs sigmac to create queries for a Sysmon log source (`-c tools/config/generic/sysmon.yml`) and the ElasticSearch target backend (`-t es-qs`) 
+Use a config file for `process_creation` rules (`-r rules/windows/process_creation`) that instructs sigmac to create queries for a Sysmon log source (`-c tools/config/generic/sysmon.yml`) and the ElasticSearch target backend (`-t es-qs`)
 ```
 tools/sigmac -t es-qs -c tools/config/generic/sysmon.yml -r rules/windows/process_creation
 ```
@@ -209,7 +209,9 @@ tools/sigmac -t splunk -c ~/my-splunk-mapping.yml -c tools/config/generic/window
 * [LimaCharlie](https://limacharlie.io)
 * [ee-outliers](https://github.com/NVISO-BE/ee-outliers)
 * [Structured Threat Information Expression (STIX)](https://oasis-open.github.io/cti-documentation/stix/intro.html)
+* [LOGIQ](https://www.logiq.ai)
 * [uberAgent ESA](https://uberagent.com/)
+* [Devo](https://devo.com)
 
 Current work-in-progress
 * [Splunk Data Models](https://docs.splunk.com/Documentation/Splunk/7.1.0/Knowledge/Aboutdatamodels)
@@ -228,16 +230,18 @@ It's available on PyPI. Install with:
 pip3 install sigmatools
 ```
 
-Alternatively, if used from the Sigma Github repository, the Python dependencies can be installed with:
+Alternatively, if used from the Sigma Github repository, the Python dependencies can be installed with [Pipenv](https://pypi.org/project/pipenv/).
+Run the following command to get a shell with the installed requirements:
 
 ```bash
-pip3 install -r tools/requirements.txt
+pipenv shell
 ```
 
 For development (e.g. execution of integration tests with `make` and packaging), further dependencies are required and can be installed with:
 
 ```bash
-pip3 install -r tools/requirements-devel.txt
+pipenv install --dev
+pipenv shell
 ```
 
 ## Sigma2MISP
@@ -251,7 +255,7 @@ Example:
 *misp.conf*:
 ```
 url https://host
-key foobarfoobarfoobarfoobarfoobarfoobarfoo 
+key foobarfoobarfoobarfoobarfoobarfoobarfoo
 ```
 
 Load Sigma rule into MISP event 1234:
@@ -266,7 +270,7 @@ sigma2misp @misp.conf --same-event --info "Test Event" -r sigma_rules/
 
 ## Evt2Sigma
 
-[Evt2Sigma](https://github.com/Neo23x0/evt2sigma) helps you with the rule creation. It generates a Sigma rule from a log entry. 
+[Evt2Sigma](https://github.com/Neo23x0/evt2sigma) helps you with the rule creation. It generates a Sigma rule from a log entry.
 
 ## Sigma2attack
 
@@ -291,7 +295,7 @@ Result once imported in the MITRE ATT&CK® Navigator ([online version](https://m
 
 ## S2AN
 
-Similar to **Sigma2attack**, [S2AN](https://github.com/3CORESec/S2AN) is a pre-compiled binary for both Windows and GNU/Linux that generates [MITRE ATT&CK® Navigator](https://github.com/mitre/attack-navigator/) layers from a directory of Sigma rules. 
+Similar to **Sigma2attack**, [S2AN](https://github.com/3CORESec/S2AN) is a pre-compiled binary for both Windows and GNU/Linux that generates [MITRE ATT&CK® Navigator](https://github.com/mitre/attack-navigator/) layers from a directory of Sigma rules.
 
 S2AN was developed to be used as a standalone tool or as part of a CI/CD pipeline where it can be quickly downloaded and executed without external dependencies.
 
@@ -317,22 +321,26 @@ These tools are not part of the main toolchain and maintained separately by thei
 * [uncoder.io](https://uncoder.io/) - Online Translator for SIEM Searches
 * [THOR](https://www.nextron-systems.com/2018/06/28/spark-applies-sigma-rules-in-eventlog-scan/) - Scan with Sigma rules on endpoints
 * [Joe Sandbox](https://www.joesecurity.org/)
-* [ypsilon](https://github.com/P4T12ICK/ypsilon) - Automated Use Case Testing 
+* [ypsilon](https://github.com/P4T12ICK/ypsilon) - Automated Use Case Testing
 * [RANK VASA](https://globenewswire.com/news-release/2019/03/04/1745907/0/en/RANK-Software-to-Help-MSSPs-Scale-Cybersecurity-Offerings.html)
 * [TA-Sigma-Searches](https://github.com/dstaulcu/TA-Sigma-Searches) (Splunk App)
 * [TimeSketch](https://github.com/google/timesketch/commit/0c6c4b65a6c0f2051d074e87bbb2da2424fa6c35)
-* [SIΣGMA](https://github.com/3CORESec/SIEGMA) - SIEM consumable generator that utilizes Sigma for query conversion 
+* [SIΣGMA](https://github.com/3CORESec/SIEGMA) - SIEM consumable generator that utilizes Sigma for query conversion
+
+Sigma is available in some Linux distribution repositories:
+
+[![Packaging status](https://repology.org/badge/vertical-allrepos/sigma.svg)](https://repology.org/project/sigma/versions)
 
 # Contribution
 
 If you want to contribute, you are more then welcome. There are numerous ways to help this project.
 
 ## Use it and provide feedback
 
-If you use it, let us know what works and what does not work. 
+If you use it, let us know what works and what does not work.
 
 E.g.
-- Tell us about false positives (issues section) 
+- Tell us about false positives (issues section)
 - Try to provide an improved rule (new filter) via [pull request](https://help.github.com/en/articles/editing-files-in-another-users-repository) on that rule
 
 ## Work on open issues
@@ -341,7 +349,7 @@ The github issue tracker is a good place to start tackling some issues others ra
 
 ## Provide Backends / Backend Features / Bugfixes
 
-Various requests for sigmac (sigma converter) backends exist. Some backends are very limited and need features. We are working on a documentation on how to write new backends but our time for this project is currently mostly spent for issue resolutions. 
+Various requests for sigmac (sigma converter) backends exist. Some backends are very limited and need features. We are working on a documentation on how to write new backends but our time for this project is currently mostly spent for issue resolutions.
 
 ## Spread the word
 

contrib/sigma2sumologic.py

@@ -218,7 +218,7 @@ def get_rule_as_sumologic(file):
     except Exception as e:
         if args.debug:
             traceback.print_exc()
-        logger.exception("error seaching sumo  " + str(file) + "----" + str(e))
+        logger.exception("error searching sumo  " + str(file) + "----" + str(e))
         with open(os.path.join(args.outdir, "sigma-" + file_basename + '-error.txt'), "w") as f:
             # f.write(json.dumps(r, indent=4, sort_keys=True) + " ERROR: %s\n\nQUERY: %s" % (e, sumo_query))
             f.write(" ERROR: %s\n\nQUERY: %s" % (e, sumo_query))

rules-unsupported/sysmon_always_install_elevated_parent_child_correlated.yml

@@ -0,0 +1,42 @@
+title: Always Install Elevated Parent Child Correlated
+id: 078235c5-6ec5-48e7-94b2-f8b5474379ea
+description: This rule will looks any process with low privilege launching Windows Installer service (msiexec.exe) that tries to install MSI packages with SYSTEM privilege 
+#look for MSI start by low privilege user, write the process guid to the suspicious_guid variable
+#look for child process from the suspicious_guid, alert if it's Windows Installer trying to install package with SYSTEM privilege
+status: experimental
+author: Teymur Kheirkhabarov (idea), Mangatas Tondang (rule), oscd.community
+date: 2020/10/13
+references: 
+    - https://image.slidesharecdn.com/kheirkhabarovoffzonefinal-181117201458/95/hunting-for-privilege-escalation-in-windows-environment-48-638.jpg
+tags:
+    - attack.privilege_escalation
+    - attack.t1548.002
+logsource:
+    product: windows
+    category: process_creation
+detection:
+    system_integrity:
+        IntegrityLevel: 'System'
+    system_user:
+        User: 'NT AUTHORITY\SYSTEM'
+    image_1:
+        Image|contains|all: 
+            - '\Windows\Installer\'
+            - 'msi'
+        Image|endswith: 
+            - 'tmp'
+    image_2:
+        Image|endswith: '\msiexec.exe'
+    child_of_suspicious_guid:  
+        ParentProcessGuid: '%suspicious_guid%'
+    condition: write ProcessGuid from (event_id and image_2 and not system_user) to %suspicious_guid%; then if (child_of_suspicious_guid and event_id and image_1 and system_user) or (suspicious_guid and event_id and image_2 and system_user and integrity_level) -> alert
+fields:
+    - EventID
+    - IntegrityLevel
+    - User
+    - Image
+    ParentProcessGuid
+falsepositives:
+    - System administrator usage
+    - Penetration test 
+level: high

rules-unsupported/win_access_fake_files_with_stored_credentials.yml

@@ -0,0 +1,29 @@
+title: Stored Credentials in Fake Files
+id: 692b979c-f747-41dc-ad72-1f11c01b110e
+description: Search for accessing of fake files with stored credentials
+status: experimental
+author: Teymur Kheirkhabarov (idea), Ryan Plas (rule), oscd.community
+date: 2020/10/05
+references: 
+    - https://image.slidesharecdn.com/kheirkhabarovoffzonefinal-181117201458/95/hunting-for-privilege-escalation-in-windows-environment-13-638.jpg
+tags:
+    - attack.credential_access
+    - attack.t1555
+logsource:
+    product: windows
+    service: security
+detection:
+    selection:
+        EventID: 4663
+        AccessList|contains: '%%4416'
+        ObjectName|endswith:
+            - '\%POLICY_ID%\Machine\Preferences\Groups\Groups.xml'
+            - '\%FOLDER_NAME%\Unattend.xml'
+    condition: selection
+fields:
+    - EventID
+    - AccessList
+    - ObjectName
+falsepositives:
+    - Unknown
+level: high