Merges 2.2.0 (airbnb#929)

* Adds Alert Publisher framework (airbnb#900)

* Squashed commit of AlertPublishers (Provisional)

i

aefoiajwoef

initial draft to publisher crap

awefla

aewlfkjhawiehgsv

Add some alerts

aweklrgjhsakf

fadsuhiawef

Adds some basic publisher framework

pylint

Compoiste publisher

Assemble_alert_publisher_for_output

more code

fix tests

Replace with publishers

WORKING COMMIT

This commit is proven to work on Stage.

TODOs:
- publishers are hard to code, and the class system seems redundant?
- merging needs to be tested
- required outputs will get published if blanket publication is enabled

Fixes a bug where publishers are assigned to required outputs

Tidies up docblocks

Flesh out publisher documentation

* Extends SlackOutput to support custom messages and attachments

* Refactor publishers into new directory

Functional commit

* Continues to fumble with classes and namespaces and stuff

* cyclial python module dependency

* Missing mock_s3

* workign commit fumbling with more namespaces and directories

* pylint

* Publishers working?

* w

* Fixups

* Fix comment

* fix

* Consolidate some functions, rename some Classes to be more compact

* more touchiups

* i

* Update documentation

* Refactor to move DefaultPublisher into the core

* Add test coverage

* Address unused-argument

* Remove some extraneous import module things

* Move core.py publisher code into shared

* fix comment

* Catch, log, and reraise keyerror in composite publisher

* Fix buggy docuemntationt

* Fix docvumentation

* @rule -> @rule

* fix documentation string

* Reverses the order specific and unspecific output publishers are executed to be more intuitive

* Move import_folders to new module

* Remove deepcopy, delegating it to CompositePublisher

* Clean up docblocks

* Remove extraneous return

* Write a test for chained inheritance

* Move test files

* Move incorrectly placed comment

* Raise exception when failing to register publisher

* Raise exception on invalid arg output

* Touch up tests

* Gets rid of ugly python \

* Fix some bad list code plus DRY out DefaultPublisher

* Renames publish_alert to compose_alert to be less imperative

* enforce ssl access only on all S3 buckets (airbnb#905)

* enforcing ssl only access on all streamalert s3 buckets

* updating unit tests for s3 bucket resource creation

* fixing duplicate policy bug with cloudtrail bucket (airbnb#907)

* [Alert Publishers] Add some community Slack Publishers (airbnb#904)

* Adds some base Slack publishers for the community

Remove stuff

 Fix up

fix merge

Touchups

Remove deepcopy from slack publishers

Fix bug and missing test inclusion due to missing init file

fix licenses

Capitalize fields and remove redundant

fix tests

Fix a... test?

* Convert map() to array syntax. Fix timezone problem

* [Alert Publishers] Standardizes magic fields with @-sign prefix (airbnb#917)

* Prefixes all magic publisher fields with @

* Fix documentation

* [Fixup] Alert Publisher PR Feedback (airbnb#918)

* PR Fixups

* I did not end up figuring out how to uncouple this cyclic dependency. optimzied instead

* ?

* wtf why is consider-using-ternary a mandatory pylint condition?

* [Alert Publishers] Rebuilds PagerDuty integration + Adds some community pagerduty publishers (airbnb#911)

* WIP: PagerDuty publishers

fixup

wip but maybe working commit of refactoring all this pagerduty crap

Fixing unit tests

unit tests

Draft

ya

wrfg

fixed

wip

Fix ssl verification

Working commit; need add more test coverage

Fix documentation

Add publishers

Test for enumerate_fields + fix bug

Add publisher that strips out "streamalert:normalization" from the publication

Ef

* Upgrades integration paths

* successfully deployed and tested v1 on staging

* Finishes draft

* fixup

* Yeah... yeah..

* Expand docblocks

* WIP

* Fixups documents

* Fix bug in pd publisher

* fix

* Fix bug and some documentation

* remove

* Fix some bugs

* Fix some error messages

* Fixes a bug causing alerts not to merge correctly

* Fix tests

* alphabetize enumerate_fields

* Add new remove_fields publisher

* Add more test for work

* y

* Pr feedback and add test

* Fix some bugs introduced due to "default behavior" in pagerduty integration (airbnb#920)

* Fix a bug regarding defaults in the pagerduty integration refactor

* PR fixup

* fix silly bug (airbnb#921)

* Adds a new publisher, improves to description parser (airbnb#922)

* Adds a publisher to bubble deep dict fields to top of publication

* Improves the rule parser to accommodate for some weird cases

* pylint

* Fix comment

* bubble_fields -> populate_fields

* More PR feedback

* Adds new publisher for converting array to string.

* naisu

* Moves some directories to be more consistent

* pr feedback

* [Publishers] Adds publisher error detection to rule_test.sh (airbnb#923)

* tmp wip commit

* First attempt at adding publishers to rule_tst

* Improved format

* fixps

* Consolidates test logic a little

* Improves format

* PR Fixup: compose_alert now requires output

* PR feedback

* Improvements to Slack Publishers (airbnb#924)

* Adds new publisher to slack

* Improves description parser

* Add tests

* [Publishers] Adds support for images on Pagerduty v2 (airbnb#925)

* Support images on pagerduty v2 API

* Tests, and adds a new pagerduty publisher to attach an image

* bumping version to 2.2.0

docs/source/index.rst

@@ -77,6 +77,7 @@ Table of Contents
    rules
    testing
    outputs
+   publishers
    metrics
    troubleshooting
    faq

docs/source/outputs.rst

@@ -15,6 +15,7 @@ Out of the box, StreamAlert supports:
 * **AWS SNS**
 * **AWS SQS**
 * **CarbonBlack**
+* **Demisto**
 * **GitHub**
 * **Jira**
 * **Komand**
@@ -74,6 +75,9 @@ Adding support for a new service involves five steps:
 
 .. code-block:: python
 
+  from stream_alert.alert_processor.helpers import compose_alert
+
+
   def get_user_defined_properties(self):
     """Returns any properties for this output that must be provided by the user
     At a minimum, this method should prompt the user for a 'descriptor' value to
@@ -88,12 +92,29 @@ Adding support for a new service involves five steps:
                                     '(ie: name of integration/channel/service/etc)'))
     ])
 
-  def _dispatch(self, **kwargs):
+  def _dispatch(self, alert, descriptor):
     """Handles the actual sending of alerts to the configured service.
     Any external API calls for this service should be added here.
     This method should return a boolean where True means the alert was successfully sent.
+
+    In general, use the compose_alert() method defined in stream_alert.alert_processor.helpers
+    when presenting the alert in a generic polymorphic format to be rendered on the chosen output
+    integration. This is so specialized Publishers can modify how the alert is represented on the
+    output.
+
+    In addition, adding output-specific fields can be useful to offer more fine-grained control
+    of the look and feel of an alert.
+
+    For example, an optional field that directly controls a PagerDuty incident's title:
+    - '@pagerduty.incident_title'
+
+
+    When referencing an alert's attributes, reference the alert's field directly (e.g.
+    alert.alert_id). Do not rely on the published alert.
     """
-    ...
+
+    publication = compose_alert(alert, self, descriptor)
+    # ...
     return True
 
 **Note**: The ``OutputProperty`` object used in ``get_user_defined_properties`` is a namedtuple consisting of a few properties:

docs/source/publishers.rst

@@ -0,0 +1,276 @@
+Publishers
+==========
+
+Overview
+--------
+
+Publishers are a framework for transforming alerts prior to dispatching to outputs, on a per-rule basis.
+This allows users to customize the look and feel of alerts.
+
+
+How do Publishers work?
+-----------------------
+
+Publishers are blocks of code that are run during alert processing, immediately prior to dispatching
+an alert to an output.
+
+
+
+Implementing new Publishers
+---------------------------
+
+All publishers must be added to the ``publishers`` directory. Publishers have two valid syntaxes:
+
+
+**Function**
+
+Implement a top-level function with that accepts two arguments: An Alert and a dict. Decorate this function
+with the ``@Register`` decorator.
+
+.. code-block:: python
+
+  from stream_alert.shared.publisher import Register
+
+  @Register
+  def my_publisher(alert: Alert, publication: dict) -> dict:
+    # ...
+    return {}
+
+
+**Class**
+
+Implement a class that inherits from the ``AlertPublisher`` and fill in the implementations for ``publish()``.
+Decorate the class with the ``@Register`` decorator.
+
+.. code-block:: python
+
+  from stream_alert.shared.publisher import AlertPublisher, Register
+
+  @Register
+  class MyPublisherClass(AlertPublisher):
+
+    def publish(alert: Alert, publication: dict) -> dict:
+      # ...
+      return {}
+
+
+**Recommended Implementation**
+
+Publishers should always return dicts containing only simple types (str, int, list, dict).
+
+Publishers are executed in series, each passing its published ``Alert`` to the next publisher. The ``publication``
+arg is the result of the previous publisher (or ``{}`` if it is the first publisher in the series). Publishers
+should freely add, modify, or delete fields from previous publications. However, publishers should avoid
+doing in-place modifications of the publications, and should prefer to copy-and-modify:
+
+.. code-block:: python
+
+  from stream_alert.shared.publisher import Register
+
+  @Register
+  def sample_publisher(alert, publication):
+    publication['new_field'] = 'new_value']
+    publication.pop('old_field', None)
+
+    return publication
+
+
+Preparing Outputs
+-----------------
+
+In order to take advantage of Publishers, all outputs must be implemented with the following guidelines:
+
+**Use compose_alert()**
+
+When presenting unstructured or miscellaneous data to an output (e.g. an email body, incident details),
+outputs should be implemented to use the ``compose_alert(alert: Alert, output: OutputDispatcher, descriptor: str) -> dict``
+method.
+
+``compose_alert()`` loads all publishers relevant to the given ``Alert`` and executes these publishers in series,
+returning the result of the final publisher.
+
+All data returned by ``compose_alert()`` should be assumed as optional.
+
+.. code-block:: python
+
+  from stream_alert.alert_processor.helpers import compose_alert
+
+  def _dispatch(self, alert, descriptor):
+    # ...
+    publication = compose_alert(alert, self, descriptor)
+    make_api_call(misc_data=publication)
+
+
+**"Default" Implementations**
+
+For output-specific fields that are mandatory (such as an incident Title or assignee), each output
+should offer a default implementation:
+
+.. code-block:: python
+
+  def _dispatch(self, alert, descriptor):
+    default_title = 'Incident Title: #{}'.format(alert.alert_id)
+    default_html = '<html><body>Rule: {}</body></html>'.format(alert.rule_description)
+    # ...
+
+
+**Custom fields**
+
+Outputs can be implemented to offer custom fields that can be filled in by Publishers. This (optionally)
+grants fine-grained control of outputs to Publishers. Such fields should adhere to the following conventions:
+
+* They are top level keys on the final publication dictionary
+* Keys are strings, following the format: ``@{output_service}.{field_name}``
+* Keys MUST begin with an at-sign
+* The ``output_service`` should match the current outputs ``cls.__service__`` value
+* The ``field_name`` should describe its function
+* Example: ``@slack.attachments``
+
+Below is an example of how you could implement an output:
+
+.. code-block:: python
+
+  def _dispatch(self, alert, descriptor):
+    # ...
+    publication = compose_alert(alert, self, descriptor)
+
+    default_title = 'Incident Title: #{}'.format(alert.alert_id)
+    default_html = '<html><body>Rule: {}</body></html>'.format(alert.rule_description)
+
+    title = publication.get('@pagerduty.title', default_title)
+    body_html = publication.get('@pagerduty.body_html', default_html)
+
+    make_api_call(title, body_html, data=publication)
+
+
+**Alert Fields**
+
+When outputs require mandatory fields that are not subject to publishers, they should reference the ``alert``
+fields directly:
+
+.. code-block:: python
+
+  def _dispatch(self, alert, descriptor):
+    rule_description = alert.rule_description
+    # ...
+
+
+Registering Publishers
+----------------------
+
+Register publishers on a rule using the ``publisher`` argument on the ``@rule`` decorator:
+
+.. code-block:: python
+
+  from publishers import publisher_1, publisher_2
+  from stream_alert.shared.rule import Rule
+
+  @rule(
+    logs=['stuff'],
+    outputs=['pagerduty', 'slack'],
+    publishers=[publisher_1, publisher_2]
+  )
+  def my_rule(rec):
+    # ...
+
+The ``publishers`` argument is a structure containing references to **Publishers** and can follow any of the
+following structures:
+
+**Single Publisher**
+
+.. code-block:: python
+
+  publishers=publisher_1
+
+When using this syntax, the given publisher will be applied to all outputs.
+
+
+**List of Publishers**
+
+.. code-block:: python
+
+  publishers=[publisher_1, publisher_2, publisher_3]
+
+When using this syntax, all given publishers will be applied to all outputs.
+
+
+**Dict mapping Output strings to Publisher**
+
+.. code-block:: python
+
+  publishers={
+    'pagerduty:analyst': [publisher_1, publisher_2],
+    'pagerduty': [publisher_3, publisher_4],
+    'demisto': other_publisher,
+  }
+
+When using this syntax, publishers under each key will be applied to their matching outputs. Publisher keys
+with generic outputs (e.g. ``pagerduty``) are loaded first, before publisher keys that pertain to more
+specific outputs (e.g. ``pagerduty:analyst``).
+
+The order in which publishers are loaded will dictate the order in which they are executed.
+
+
+DefaultPublisher
+----------------
+
+When the ``publishers`` argument is omitted from a ``@rule``, a ``DefaultPublisher`` is loaded and used. This
+also occurs when the ``publishers`` are misconfigured.
+
+The ``DefaultPublisher`` is reverse-compatible with old implementations of ``alert.output_dict()``.
+
+
+Putting It All Together...
+--------------------------
+
+Here's a real-world example of how to effectively use Publishers and Outputs:
+
+PagerDuty requires all Incidents be created with an `Incident Summary`, which appears at as the title of every
+incident in its UI. Additionally, you can optionally supply `custom details` which appear below as a large,
+unstructured body.
+
+By default, the PagerDuty integration sends ``"StreamAlert Rule Triggered - rule_name"`` as the `Incident Summary`,
+along with the entire Alert record in the `custom details`.
+
+However, the entire record can contain mostly irrelevant or redundant data, which can pollute the PagerDuty UI
+and make triage slower, as responders must filter through a large record to find the relevant pieces of
+information, this is especially true for alerts of very limited scope and well-understood remediation steps.
+
+Consider an example where informational alerts are triggered upon login into a machine. Responders only care
+about the **time** of login, **source IP address**, and the **username** of the login.
+
+You can implement a publisher that only returns those three fields and strips out the rest from the alert.
+The publisher can also simplify the PagerDuty title:
+
+.. code-block:: python
+
+  from stream_alert.shared.publisher import Register
+
+  @Register
+  def simplify_pagerduty_output(alert, publication):
+    return {
+      '@pagerduty.record': {
+          'source_ip': alert.record['source_ip'],
+          'time': alert.record['timestamp'],
+          'username': alert.record['user'],
+      },
+      '@pagerduty.summary': 'Machine SSH: {}'.format(alert.record['user']),
+    }
+
+Suppose this rule is being output to both PagerDuty and Slack, but you only wish to simplify the PagerDuty
+integration, leaving the Slack integration the same. Registering the publisher can be done as such:
+
+.. code-block:: python
+
+  from publishers.pagerduty import simplify_pagerduty_output
+  from stream_alert.shared.rule import Rule
+
+  @rule(
+    logs=['ssh'],
+    output=['slack:engineering', 'pagerduty:engineering'],
+    publishers={
+      'pagerduty:engineering': simplify_pagerduty_output,
+    }
+  )
+  def machine_ssh_login(rec):
+    # ...