Skip to content

Commit

Permalink
release 2.3.0 (airbnb#976)
Browse files Browse the repository at this point in the history
* [apps] add Intercom app integration (airbnb#937)

* [apps] Add Intercom app integration

* [tests] Add unit tests for Intercom app

* [docs] Add instructions in README for Intercom app

* [PR review] address comments

1. Fix RST header
2. Use proper syntax for linking in RST
3. Default to 0 for sleep seconds
4. Ensure timestamp is UTC
5. Add Intercom app set up instructions to app-configuration.rst

* [PR review] Update unit tests to mock `calendar.timegm`

* [PR review] Update token validation to look for a base64 encoded string with prefix

* Upgrade aliyun api to retry timeout (airbnb#970)

* [dep] upgrade dependencies and rebuild aliyun-python-sdk

* [conf] Update aliyun actiontrail schema

* [bandit] fix bandit failures

* [bandit] Address picky Ryan's comments

* bumping version to 2.3.0
  • Loading branch information
ryandeivert authored Aug 19, 2019
1 parent 21bafa9 commit a078b4a
Show file tree
Hide file tree
Showing 20 changed files with 433 additions and 56 deletions.
7 changes: 4 additions & 3 deletions conf/logs.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
"acsRegion",
"referencedResources",
"recipientAccountId",
"requestParameters",
"userAgent",
"additionalEventData"
]
Expand Down Expand Up @@ -1811,9 +1812,9 @@
]
}
},
"fleet:results":{
"fleet:results": {
"schema": {
"name":"string",
"name": "string",
"hostIdentifier": "string",
"calendarTime": "string",
"unixTime": "integer",
Expand Down Expand Up @@ -2094,4 +2095,4 @@
]
}
}
}
}
12 changes: 12 additions & 0 deletions docs/source/app-configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ Supported Services
- Access Logs
- Integrations Logs

* `Intercom <https://developers.intercom.com/intercom-api-reference/reference#view-admin-activity-logs>`_

- Admin Activity Logs

* *More to come*


Expand Down Expand Up @@ -163,3 +167,11 @@ The Aliyun API requires an access key and access key secret for an authorized us
To obtain the access key and access key secret, an authorized user of the Aliyun account should follow their directions to `Create an Access Key <https://www.alibabacloud.com/help/doc-detail/53045.htm>`_.
Additionly, the user for whom the access key was created must have sufficient privileges to make use of ActionTrail; follow the directions on the `Grant ActionTrail permissions to RAM users <https://www.alibabacloud.com/help/doc-detail/28818.htm>`_ page.
How to set up the Intercom App
------------------------------
The Intercom API requires an access token. Get an access token by following these `instructions <https://developers.intercom.com/building-apps/docs/authorization#section-how-to-get-an-access-token>`_.
To specify an API version, follow `these instructions <https://developers.intercom.com/building-apps/docs/api-versioning>`_ to do so through Intercom's Developer Hub.
The default will be the latest stable version. The Intercom app works on versions 1.2 or later.
2 changes: 1 addition & 1 deletion requirements-top-level.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
aliyun-python-sdk-core==2.8.5
aliyun-python-sdk-core==2.13.5
aliyun-python-sdk-actiontrail==2.0.0
autoflake
autopep8
Expand Down
83 changes: 48 additions & 35 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,78 +1,79 @@
aliyun-python-sdk-core==2.8.5
aliyun-python-sdk-actiontrail==2.0.0
autoflake==1.2
autopep8==1.4.3
backoff==1.7.0
bandit==1.5.1
boto3==1.9.50
boxsdk==2.0.0a11
cbapi==1.3.6
coverage==4.5.2
coveralls==1.5.1
google-api-python-client==1.6.4
jmespath==0.9.3
jsonlines==1.2.0
mock==2.0.0
moto==1.3.7
netaddr==0.7.19
nose==1.3.7
nose-timer==0.7.3
pathlib2==2.3.2
policyuniverse==1.3.2.0
pyfakefs==3.5.3
pylint==1.9.3
requests==2.20.1
Sphinx==1.8.2
sphinx-rtd-theme==0.4.2
ushlex==0.99.1
yapf==0.25.0
## The following requirements were added by pip freeze:
alabaster==0.7.12
aliyun-python-sdk-actiontrail==2.0.0
aliyun-python-sdk-core==2.13.5
asn1crypto==0.24.0
astroid==1.6.5
attrdict==2.0.0
attrs==19.1.0
autoflake==1.3.1
autopep8==1.4.4
aws-sam-translator==1.13.1
aws-xray-sdk==0.95
Babel==2.6.0
backoff==1.8.0
backports.functools-lru-cache==1.5
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
bandit==1.6.2
boto==2.49.0
botocore==1.12.50
boto3==1.9.208
botocore==1.12.208
boxsdk==2.0.0a11
cachetools==3.0.0
cbapi==1.5.1
certifi==2018.10.15
cffi==1.11.5
cfn-lint==0.23.3
chainmap==1.0.2
chardet==3.0.4
configparser==3.5.0
cookies==2.2.1
coverage==4.5.4
coveralls==1.8.2
cryptography==2.4.2
DateTime==4.3
decorator==4.4.0
docker==3.5.1
docker-pycreds==0.3.0
docopt==0.6.2
docutils==0.14
ecdsa==0.13
enum34==1.1.6
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.17.1
futures==3.2.0
gitdb2==2.0.5
GitPython==2.1.11
google-api-python-client==1.6.4
httplib2==0.12.0
idna==2.7
imagesize==1.1.0
ipaddress==1.0.22
isort==4.3.4
Jinja2==2.10
jsondiff==1.1.1
Jinja2==2.10.1
jmespath==0.9.4
jsondiff==1.1.2
jsonlines==1.2.0
jsonpatch==1.24
jsonpickle==1.0
jsonpointer==2.0
jsonschema==3.0.2
lazy-object-proxy==1.3.1
MarkupSafe==1.1.0
mccabe==0.6.1
mock==3.0.5
moto==1.3.13
netaddr==0.7.19
nose==1.3.7
nose-timer==0.7.5
oauth2client==4.1.3
packaging==18.0
pathlib2==2.3.4
pbr==5.1.1
pika==0.12.0
policyuniverse==1.3.2.0
prompt-toolkit==2.0.7
protobuf==3.6.1
pyaml==18.11.0
Expand All @@ -81,31 +82,43 @@ pyasn1-modules==0.2.2
pycodestyle==2.4.0
pycparser==2.19
pycryptodome==3.7.1
pyfakefs==3.6
pyflakes==2.0.0
Pygments==2.3.0
PyJWT==1.6.4
pylint==1.9.5
pyOpenSSL==18.0.0
pyparsing==2.3.0
pyrsistent==0.15.4
python-dateutil==2.7.5
python-jose==2.0.2
pytz==2018.7
PyYAML==3.13
PyYAML==5.1.2
requests==2.22.0
requests-toolbelt==0.8.0
responses==0.10.4
rsa==4.0
s3transfer==0.1.13
s3transfer==0.2.1
scandir==1.9.0
singledispatch==3.4.0.3
six==1.11.0
smmap2==2.0.5
snowballstemmer==1.2.1
solrq==1.1.1
Sphinx==1.8.5
sphinx-rtd-theme==0.4.3
sphinxcontrib-websupport==1.1.0
sshpubkeys==3.1.0
stevedore==1.30.0
typing==3.6.6
uritemplate==3.0.0
urllib3==1.24.1
urllib3==1.24.2
ushlex==0.99.1
validators==0.13.0
wcwidth==0.1.7
websocket-client==0.54.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0
yapf==0.28.0
zope.interface==4.6.0
2 changes: 1 addition & 1 deletion stream_alert/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
"""StreamAlert version."""
__version__ = '2.2.1'
__version__ = '2.3.0'
8 changes: 8 additions & 0 deletions stream_alert/apps/_apps/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ To obtain the access key and access key secret, an authorized user of the Aliyun

Additionly, the user for whom the access key was created must have sufficient privileges to make use of ActionTrail; follow the directions on the `Grant ActionTrail permissions to RAM users <https://www.alibabacloud.com/help/doc-detail/28818.htm>`_ page.

How to set up the intercom app
##############################

The Intercom API requires an access token. Get an access token by following `these instructions <https://developers.intercom.com/building-apps/docs/authorization#section-how-to-get-an-access-token>`_.

To specify an API version, follow `these instructions <https://developers.intercom.com/building-apps/docs/api-versioning>`_ to do so through Intercom's Developer Hub.
The default will be the latest stable version. The Intercom app works on versions 1.2 or later.

How to set up the slack app
###########################

Expand Down
Binary file not shown.
105 changes: 105 additions & 0 deletions stream_alert/apps/_apps/intercom.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
import calendar
import time
import re

import requests

from . import AppIntegration, StreamAlertApp, get_logger


LOGGER = get_logger(__name__)


@StreamAlertApp
class IntercomApp(AppIntegration):
"""Intercom StreamAlert app"""
_INTERCOM_LOGS_URL = 'https://api.intercom.io/admins/activity_logs'

def __init__(self, event, config):
super(IntercomApp, self).__init__(event, config)
self._next_page = None

@classmethod
def _type(cls):
return 'admin_activity_logs'

@classmethod
def service(cls):
return 'intercom'

@classmethod
def _required_auth_info(cls):
return {
'token': {
'description': 'the access token for this Intercom app',
'format': re.compile(r'^dG9r([0-9A-Za-z+\/=]*)$')
}
}

def _sleep_seconds(self):
"""Return the number of seconds this polling function should sleep for
between requests to avoid failed requests. Intercom API allows for a default of 500 requests
every minute, distributed over 10 seconds periods. This means for a default rate limit of
500 per minute, you can send a maximum of 83 operations per 10 second period. It's unlikely
we will hit that limit, so this can default to 0.
Resource(s):
https://developers.intercom.com/intercom-api-reference/reference#rate-limiting
Returns:
int: Number of seconds that this function should sleep for between requests
"""
return 0

def _gather_logs(self):
# Generate headers
headers = {'Authorization': "Bearer %s" % self._config.auth['token'],
'Accept': 'application/json'}

# Results are paginated with a page url field provided that is used in subsequent queries.
# If this field exists, make a a query to the page url, and if not, make a fresh query to
# the default _INTERCOM_LOGS_URL with the required created_at_before and created_at_after
# parameters
if self._next_page:
params = None
url = self._next_page
else:
params = {'created_at_before': int(calendar.timegm(time.gmtime())),
'created_at_after': self._last_timestamp}
url = self._INTERCOM_LOGS_URL

LOGGER.info("Requesting events from: %s params: %s", url, params)

try:
result, response = self._make_get_request(url, headers=headers, params=params)
except requests.exceptions.ConnectionError:
LOGGER.exception('Received bad response from Intercom')
return False

if not result:
return False

activities = [
activity
for activity
in response['activity_logs']
if int(activity['created_at']) > self._last_timestamp]

if not activities:
return False

# Save next page url if any for paginated results
if response['pages']['next'] is not None:
self._more_to_poll = True
self._next_page = response['pages']['next']
else:
self._more_to_poll = False
self._next_page = None

most_recent_timestamp = max(int(activity['created_at']) for activity in activities)

# Save most recent time stamp for the next query
if most_recent_timestamp > self._last_timestamp:
self._last_timestamp = most_recent_timestamp

return activities
2 changes: 1 addition & 1 deletion stream_alert/apps/_apps/onelogin.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
class OneLoginApp(AppIntegration):
"""OneLogin StreamAlert App"""
_ONELOGIN_EVENTS_URL = 'https://api.{}.onelogin.com/api/1/events'
_ONELOGIN_TOKEN_URL = 'https://api.{}.onelogin.com/auth/oauth2/v2/token'
_ONELOGIN_TOKEN_URL = 'https://api.{}.onelogin.com/auth/oauth2/v2/token' # nosec
_ONELOGIN_RATE_LIMIT_URL = 'https://api.{}.onelogin.com/auth/rate_limit'
# OneLogin API returns 50 events per page
_MAX_EVENTS_LIMIT = 50
Expand Down
2 changes: 1 addition & 1 deletion stream_alert/apps/_apps/salesforce.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ class SalesforceApp(AppIntegration):
ReportExport:
events contain details about reports that a user exported.
"""
_SALESFORCE_TOKEN_URL = 'https://login.salesforce.com/services/oauth2/token'
_SALESFORCE_TOKEN_URL = 'https://login.salesforce.com/services/oauth2/token' # nosec
_SALESFORCE_QUERY_URL = ('{instance_url}/services/data/v{api_version}/'
'{query}{start_time}{event_type}')
# Use the Query resource to retrieve log files.
Expand Down
10 changes: 5 additions & 5 deletions stream_alert_cli/manage_lambda/package.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,17 +43,17 @@ class LambdaPackage(object):
# Define a package dict to support pinning versions across all subclasses
PACKAGE_LIBS = {
'aliyun-python-sdk-actiontrail': 'aliyun-python-sdk-actiontrail==2.0.0',
'backoff': 'backoff==1.7.0',
'boto3': 'boto3==1.9.50',
'backoff': 'backoff==1.8.0',
'boto3': 'boto3==1.9.208',
'boxsdk[jwt]': 'boxsdk[jwt]==2.0.0a11',
'cbapi': 'cbapi==1.3.6',
'cbapi': 'cbapi==1.5.1',
'google-api-python-client': 'google-api-python-client==1.6.4',
'jmespath': 'jmespath==0.9.3',
'jmespath': 'jmespath==0.9.4',
'jsonlines': 'jsonlines==1.2.0',
'netaddr': 'netaddr==0.7.19',
'oauth2client': 'oauth2client==4.1.3',
'policyuniverse': 'policyuniverse==1.3.2.0',
'requests': 'requests==2.20.1',
'requests': 'requests==2.22.0',
}

def __init__(self, config):
Expand Down
2 changes: 1 addition & 1 deletion stream_alert_cli/test/results.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ class TestResult(object):
_PASS_STRING = format_green('Pass')
_FAIL_STRING = format_red('Fail')
_SIMPLE_TEMPLATE = '{header}:'
_PASS_TEMPLATE = '{header}: {pass}'
_PASS_TEMPLATE = '{header}: {pass}' # nosec
_DESCRIPTION_LINE = (
'''
Description: {description}'''
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/stream_alert_apps/test_apps/test_aliyun.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,5 +99,5 @@ def test_gather_logs_entries(self, client_mock):
'"StartTime":"2018-06-23T19:28:30Z"}'
logs = self._app._gather_logs()
assert_equal(2, len(logs))
assert self._app._more_to_poll
assert self._app._more_to_poll # nosec
assert_equal(self._app.request.get_NextToken(), "20")
Loading

0 comments on commit a078b4a

Please sign in to comment.