Cyberowl is a daily updated summary of the most frequent types of security incidents currently being reported from different sources. The primary objective of Cyberowl is to assist analysts and security professionals in quickly identifying different types of security incidents affecting their organization's assets. This is the quickest and most effective way to always stay current on the latest types of incidents.
In this section, we will outline the various steps Cyberowl takes to collect data and generate reports.
Spiders are classes which define how a certain site will be scraped, including how to perform the crawl and how to extract structured data from their pages. In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site.
Cyberowl uses both Scrapy spiders and Selenium web drivers to collect data from different sources.
- Scrapy is used for static content.
- Selenium is used for dynamically generated content.
Check out cyberwol spiders here.
The Item Pipeline is the place where you define the custom behaviour for processing items after they have been scraped.
In other words, the Item Pipeline is where cleaning and processing of data is done as well as storing it in a markdown file.
Check out cyberwol pipelines here.
For report generation, we have implemented a custom MDTemplate class which is used to format the data into a markdown file.
And with that, we generate the reports using the functions in utils.py.
Check out MDTemplate here.
Check out utils.py here.
Api Docs are generated using mkdocs. Check out Contributing for more details on how to generate the Api Docs.
To setup Cyberowl, you need to install the following dependencies:
First install poetry:
pip install poetryClone the project, and install the dependencies:
poetry installAnd finally, run the main script:
poetry run python src/cyberowl/main.pyContributing to Cyberowl is a two step process. Please check out CONTRIBUTING.md for more details.
Cyberowl is licensed under the MIT License.
Cyberowl is a community project and we welcome all contributors to join us. Please check out our Code of Conduct for more details.