Updated user documentation with initial content. (microsoft#40)

* Updated user documentation with initial content.

* Fixing some formatting and links

* Redirects needed for readthedocs

* Documenation and minor changes

* typos

* Updated documentaion

* Additional tidy of docs - added FoliumMap.rst

Fixed a number of links
Updated GeoIPLookups.rst
Updated PackageStructure.rst with more complete list and added doc and module links
Added test for test_auditdextract (read_file and clustering)
Change pylint to alert on dup code > 8 lines
Some changes to geoip.py to read dbfolder from config and prevent test errors
Changes to test_linuxsyslog and test_pkg_config to prevent errors due to missing config
Added notebook test to test_process_tree_utils
Warning suppression for test_tiproviders to prevent expected test warnings

* Updating the version number

Co-authored-by: Pete Bryan <[email protected]>

Removing azure-mgmt (microsoft#42)

* Removing azure-mgmt and adding azure-mgmt-subscription and azure-mgmt-resource

* GeoLite2  url and archive extraction changes

* typo fix for printing mising packages

* replace dummy key for Maxmind geolitelookup

* black formatting

* Cleaned redundent elements

* Changed ti_provider_settings to generic provider_settings module

Added config support for GeoIP providers
GeoIP classes try to obtain API key from config if not supplied
Prevented GeoLiteLookup from instantiating on package load
Added missing pytz and pyyaml packages to requirements.txt and setup.py.

Note: syslog_utils test still failing due to missing geoip database. Will fix by adding secret to piplelines.

* Added config options for geoip.py classes. Will try to read config by default.

Refactored pkg_config and provider_settings to consolidate global functionality in the former.
Fixed tests to copy with missing GeoIP database
Added msticpyconfig-test.yaml for build-time tests (requires env vars to be set)

* Fxing some linting errors

Co-authored-by: Ashwin Patil <[email protected]>
Co-authored-by: Pete Bryan <[email protected]>

Merge pull request microsoft#39 from microsoft/bug/timeformatting

Allow query parameters to be ISO datetime string

Iputils (microsoft#34)

* ip utils and heartbeat queries

* fix pylint and import errors

* fix pylint errors and kql heartbeat queries

* missing project columns in heartbeat queries

* adding KQL time series queries

* fix yaml parsing error in timeseries kql

* add scoreanomolies query

* fixes in time series kql queries

* refactor timeseries kql queries

* changes to mv-expand in kql query

* replace queryproject values

* Miscellaneous fixes from notebook testing:

- Query templates
- Doc updates (new doc page on msticpyconfig.yaml)
- Changed param_extractor to always prefer supplied params over defaults
- Several linter/mypy errors
- wsconfig throws meaningful error if config values are not found
- tilookup fix - exception thrown if an empty IoCs list sent to it
- geoip - fixed multiple problems with the DF lookup version of the API

* pkg and whois function addition

* Typo in wsconfig.py

* logic change to check for missing packages

* added tqdm dependency

* fix black formatting and add ipwhois dependency

* pylint warning fix

* fixing more pylint warnings

* user option for missing package installation

* docstring update

* Updated Pandas requirement

* Updates to version and requirements/setup.py

* Merge fixes

* Linting warnings in ip_utils

* ip_utils linting fixes (post-black)

Multi timeline plot (microsoft#26)

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Updated version numbers

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added Linux Queries and made minor updates

* Addded demo notebook for Data

* Import YAML Files with package

* testing if init required

* update file path for queries

* manifest update

* update manifest

* update manifest again

* Including an exmple query file to manually import

* query update

* query updates

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added defauly location in case geoip unsucessful

* Added intial linuxsyslog library

* Added defauly location in case geoip unsucessful

* Added defauly location in case geoip unsucessful

* Layout fix

* Updated README with linuxsyslog details

* Added defauly location in case geoip unsucessful

* added cluster_syslog_logons

* added cluster_syslog_logons

* fixed merge issue

* Update variable names

* updated cluster_syslog_logons

* improved user_logon query

* Fixed merge issue

* Added doc_strings

* query updates

* replaced ' with " in query

* Added additional queries

* Black formatted

* formatting test files with black

* updated logon clusters

* reverting black test formatting

* updated queries

* update queries

* running python black formatting against python36

* running python black formatting against python36

* Adding python black line length of 90 characters to match flake8

* update get host data with data lib

* update imports

* updated with new data packages

* added comments

* added host record function and unit test

* Added sudo evet detections

* aded comments to track work

* Include detections doc

* fixed typo

* typo fixes

* query updates

* typo fix

* typo_fix

* added new test

* Added clustering of sudo sessions

* Added application detection in host record

* parse datetime for test data correctly

* removed un-needed functions

* added risk session detection

* renamed linuxsyslog

* update tests with new packge name

* Updated risky cmd detection to include syslog

* query_update

* query update

* query update

* query_udpate

* query update

* added network queries

* query updates

* Query Updates

* syslog_utils

* merges

* merge

* fixes

* fixes

* custom overlay color

* fixes

* format changes

* fixed test failure

* Fixes and feature adds

* minor fixes

* folium fixes

* fixes

* More broken doclinks. Updated README to include TIProviders summary.

* Added notebook tests for nbdisplay and nbwidgets.

Updated Base64Unpack, EventClustering, NotebookWidgets and TIProviders notebooks.
Minor change to base64unpack.py to prevent pandas warning
Updates to documentation/README.md

* Initial update of timeline

* fixes

* Removed Linux elements

* linux events add

* formatting

* Adding checks for no providers or missing keys.

Updating TIProvider docs for this.
Adding missing data query yaml - kql_sent_winevent.yaml

* Initial merge of timeline display.

Adding WinSecurityEvent.json events file

* Adding pre-commit hooks including download_tlds.py

* Moved timeline to timeline.py

Added support for legacy usage, dict usage, and grouped DF usage through single API
Fixed some things with range control:
- time formatted axis labels
- increasing min/max range by 10% (so first/last events are not on the edge of the graph)
Implemented positioning of legend - inline, left, right
Tooltip columns (for dict usage) now taken from all data sources
Correcting some mypy and pylint errors.

* Fixes from testing notebook development:

-entityschema:  fixing __repr__ in
- nbwidgets: added filtering to all select widgets
- security_base: removing broken and deprecated properties adding __repr__
- timeline: setting bigger default range, bug fix not setting tooltip columns
- ti_lookup - remove unneeded import
- ti_provider_base: added severity to LookupResult

* nbwidgets - bug in restoring current index in selected items list

* Fixed error with legend parameter

* Fixing foliummap error to display in notebook.

* More bugs in timeline and eventcluster.

* Missing some changed files

* Black formatting for utility.py

* Fixing some linting warnings.

* Documentation for Event Timeline

* Fixing bandit issue with urlopen

Fixing doc warnings with TIProviders.rst

* 2nd attempt at suppressing Bandit warning. This is using a a fixed http url

* Changing RangeTool title and adding small font help string

* Refactored and consolidated code. Implemented review comments from Pete.

* Bug in getting ref_time value

TI Providers - Az Sentinel BYOTI (microsoft#23)

* Initial code for BYOTI provider. Plus tidying module references.

* kql_base.py provider for BYOTI

* BYOTI provider with unit tests.

Fixes for mypy warnings

* Black formatting

* Re-wrote test_tiproviders to use mocked http requests.  Simplified tests to make it easy to add new types.

Now also tests lookup_iocs for http providers.

* Bug fixes and TIProviders.ipynb Usage notebook

* Documentation updates

* Renaming AzureSentinelByoti to AzSTI.

Fixing a couple of bugs foundin network queries and geoip.py

* Few more bugs and typos found in testing

Update version to 0.1.9 (microsoft#13)

Documentation build errors (microsoft#11)

Merge pull request microsoft#8 from Microsoft/readme-urls

Readme urls

Adding Sphinx documentation for Read the docs (microsoft#6)

* Adding Sphinx documentation for Read the docs

* Updating requirements.txt

* Fixing some more line-length warnings

* Removing intake (breaking build on pip install)

Fixing some errors in iocextract

* Updating JupyterAndSecurity doc.

Adding function to execute simple kql string query.

* Warning fixes for flake8 (mostly line length)

* A couple more pylint warning fixes/suppressions

* Adding Linux Auditd collection document.

* Removing  license parameter from setuptools.setup

This seems to cause the license text to be concatenated to the project description and makes a mess of the PyPi description