Proxy support, updated readme

INSTALL.md

@@ -6,6 +6,11 @@ Clone repo locally and use `prepare.py` script to prepare repo code for using in
 
 You should run the script to rename dirs and variables in py-files.
 
+Also you can clean all useless files this way:
+```sh
+$ make clean
+```
+
 ## Update main code
 
 You should edit file `osint_cli_tool_skeleton/core.py` to use you OSINT methods / calls / etc. instead of template method (get title and status code from main page of site).

Makefile

@@ -23,7 +23,7 @@ format:
 	black --skip-string-normalization ${LINT_FILES}
 
 clean:
-	rm -rf reports htmcov dist build *.egg-info
+	rm -rf reports htmcov dist build *.egg-info *.txt *.csv *.pdf
 
 install:
 	pip3 install .

README.md

@@ -19,52 +19,29 @@
 
 Template for new OSINT command-line tools.
 
-**Press button "Use this template" to generate your own tool repository.** See [INSTALL.md](INSTALL.md) for further setup.
+**Press button "[Use this template](https://github.com/soxoj/osint-cli-tool-skeleton/generate)" to generate your own tool repository.** See [INSTALL.md](INSTALL.md) for further setup.
 
 ## Features
 
 - Detailed readme
+- Process N targets from args, text files, stdin
+- Make TXT, CSV reports
+- Proxy support
 - Ready to publish Python package
 
-## Installation
-
-Make sure you have Python3 and pip installed.
-
-### Manually
-
-1. Clone or [download](https://github.com/soxoj/osint-cli-tool-skeleton/archive/refs/heads/main.zip) respository
-```sh
-$ git clone https://github.com/soxoj/osint-cli-tool-skeleton
-```
-
-2. Install dependencies
-```sh
-$ pip3 install -r requirements.txt
-```
-
-### As a the package
-
-You can clone/download repo and install it from the directory to use as a Python package.
-```sh
-$ pip3 install .
-```
-
-Also you can install it from the PyPI registry:
-```sh
-$ pip3 install https://github.com/soxoj/osint-cli-tool-skeleton
-```
-
 ## Usage
 
-You can run this tool as a Python module:
 ```sh
-$ python3 -m osint-cli-tool-skeleton
+$ python3 -m osint-cli-tool-skeleton <target>
 
 # or simply
 
-$ osint_cli_tool_skeleton
+$ osint_cli_tool_skeleton <target>
 ```
 
+<details>
+<summary>Targets</summary>
+
 Specify targets one or more times:
 ```sh
 $ osint_cli_tool_skeleton www.google.com reddit.com patreon.com
@@ -99,6 +76,10 @@ Or combine tool with other through input/output pipelining:
 ```sh
 $ cat list.txt | osint_cli_tool_skeleton --targets-from-stdin
 ```
+</details>
+
+<details>
+<summary>Reports</summary>
 
 The skeleton implements CSV reports:
 ```sh
@@ -123,3 +104,46 @@ Results found: 1
 1) Value: Google
 Code: 200
 ```
+</details>
+
+<details>
+<summary>Proxy</summary>
+
+The tool supports proxy:
+```sh
+$ osint_cli_tool_skeleton www.google.com --proxy http://localhost:8080
+```
+</details>
+
+## Installation
+
+Make sure you have Python3 and pip installed.
+
+
+<details>
+<summary>Manually</summary>
+
+1. Clone or [download](https://github.com/soxoj/osint-cli-tool-skeleton/archive/refs/heads/main.zip) respository
+```sh
+$ git clone https://github.com/soxoj/osint-cli-tool-skeleton
+```
+
+2. Install dependencies
+```sh
+$ pip3 install -r requirements.txt
+```
+</details>
+
+<details>
+<summary>As a the package</summary>
+
+You can clone/download repo and install it from the directory to use as a Python package.
+```sh
+$ pip3 install .
+```
+
+Also you can install it from the PyPI registry:
+```sh
+$ pip3 install https://github.com/soxoj/osint-cli-tool-skeleton
+```
+</details>

osint_cli_tool_skeleton/_version.py

@@ -1 +1 @@
-__version__ = '0.0.1'
+__version__ = '0.0.3'

osint_cli_tool_skeleton/cli.py

@@ -200,7 +200,11 @@ async def main():
         sys.exit(1)
 
     # convert input to output
-    processor = Processor(no_progressbar=args.no_progressbar)
+    processor = Processor(
+        no_progressbar=args.no_progressbar,
+        proxy=args.proxy,
+    )
+
     output_data = await processor.process(input_data)
 
     # console output

osint_cli_tool_skeleton/core.py

@@ -2,7 +2,6 @@
 from typing import List, Any
 
 from aiohttp import TCPConnector, ClientSession
-from bs4 import BeautifulSoup as bs
 
 from .executor import AsyncioProgressbarQueueExecutor, AsyncioSimpleExecutor
 
@@ -59,7 +58,16 @@ def __repr__(self):
 
 class Processor:
     def __init__(self, *args, **kwargs):
-        connector = TCPConnector(ssl=False)
+        from aiohttp_socks import ProxyConnector
+
+        # make http client session
+        proxy = kwargs.get('proxy')
+        self.proxy = proxy
+        if proxy:
+            connector = ProxyConnector.from_url(proxy, ssl=False)
+        else:
+            connector = TCPConnector(ssl=False)
+
         self.session = ClientSession(
             connector=connector, trust_env=True
         )
@@ -73,6 +81,7 @@ async def close(self):
 
 
     async def request(self, input_data: InputData) -> OutputDataList:
+        from bs4 import BeautifulSoup as bs
         status = 0
         result = None
         error = None