Code cleanup (tableau#435)

* added anvoa to supported pre-deployed models in tabpy (tableau#350) * added anvoa to supported pre-deployed models in tabpy * fixed pep8 issue * fixed md * Add Ctrl+C handler (tableau#348) * Add Ctrl+C handler * Fix unit tests warnings for genson * Add test to increase code coverage * Add * Change default from 10Mb to 100Mb for request size * Increase code coverage * Increase code coverage * Convert buffer size to int * Add Ctrl+C test * Delete test added to the wrong folder * Update CHANGELOG * Update test_app.py * Remove dead code * Don't count coverage for multiline expressions * Add test case for invalid protocol * Add test case for _check_endpoint_name * Remove dead code * Fix vulnerabilities found by LGTM (tableau#361) * Fix vulnerabilities found by LGTM * Fix test failures * Dev improvements (tableau#384) * Fix flake8 warnings * Merge from master * Fix pycodestyle * Fix more flake8 warnings * Fix tests to pass again * Create test_coveralls_codestyle.yml (tableau#382) * Use github actions * Documentation improvements (tableau#385) * Delete .travis.yml * Fix Ctrl+C failing on extra parameter in signal handler * Remove outdated documentation for how to configure connection * tabpy.py to use docopt * Update tabpy-user with docopt * Update CHANGELOG * Fix code style * Change regex, add remove method, and edit qeury_timeout (tableau#375) * Added Client.remove method to delete deployed model * Fix bug for query_timeout types * Update CHANGELOG * Fix missing script result/return bug * Fix github workflow for push * Code improvement: app._parse_config (tableau#391) * Update app.py * v0.8.10 (tableau#392) * Collect test coverage with scrutinizer instead of coveralls * Restore coverage collecting with coveralls * Update scrutinizer settings * Add support for Python 3.8 * Fix static page and add unit test for it * Delete obsolete test * Dev cov (tableau#394) * Collect test coverage with scrutinizer instead of coveralls * Restore coverage collecting with coveralls * Update scrutinizer settings * Add support for Python 3.8 * Fix static page and add unit test for it * Delete obsolete test * Restore scrutinizer configuration * Linting as separate build step * Restore scrutinizer configuration * Update .scrutinizer.yml * Update .scrutinizer.yml * Restore scrutinizer configuration * Update pull_request.yml * Code style improvements * Code style improvements * Code style improvements * Add coverall workflow * Initial checkin, working tests. * Made common base class for server info tests. * pep8 checks. * Added documentation for TABPY_AUTH_INFO * Fix spacing. * Refactor config parsing to allow custom parsers. (tableau#412) * Refactor config parsing to allow custom parsers. * Fix pep8 * Update version and changelog. * Changed default for tabpy_auth_info to a boolean. * Secure info (tableau#414) * Update README.md * Doc update (tableau#402) * Fixed broken link * Linked to install doc. * Dev fix spelling (tableau#408) * Add spelling fix workflow * Refactor config parsing to allow custom parsers. * Fix pep8 * Update version and changelog. * Changed default for tabpy_auth_info to a boolean. * Remove configuration for securing the info API and make it secure by default. * Revert "Merge branch 'master' into secureInfo" This reverts commit 3688561, reversing changes made to a4acc65. * Removed auth config from uni test. Co-authored-by: nmannheimer <[email protected]> Co-authored-by: Oleksandr Golovatyi <[email protected]> * Secure info (tableau#417) * Update README.md * Doc update (tableau#402) * Fixed broken link * Linked to install doc. * Dev fix spelling (tableau#408) * Add spelling fix workflow * Refactor config parsing to allow custom parsers. * Fix pep8 * Update version and changelog. * Changed default for tabpy_auth_info to a boolean. * Remove configuration for securing the info API and make it secure by default. * Revert "Merge branch 'master' into secureInfo" This reverts commit 3688561, reversing changes made to a4acc65. * Removed auth config from uni test. * Removed example for removed setting. * Remove unused example config. Co-authored-by: nmannheimer <[email protected]> Co-authored-by: Oleksandr Golovatyi <[email protected]> * Clean up API documentation (tableau#420) * Update README.md * Doc update (tableau#402) * Fixed broken link * Linked to install doc. * Dev fix spelling (tableau#408) * Add spelling fix workflow * Update LICENSE update to Tableau Software LLC * v 1.1.0: Secure /info with auth (tableau#415) - Authorization is now required for the /info API method. This method did not check authentication previously. This change is backwards compatible with Tableau clients. - Improved config parsing flexibility. Previously the TABPY_EVALUATE_TIMEOUT setting would be set to a default if tabpy couldn't parse the value. Now it will throw an exception at startup. * Clean up API documentation * Clean up API documentation * Clean up API documentation Co-authored-by: nmannheimer <[email protected]> Co-authored-by: lriggs <[email protected]> Co-authored-by: Olek Golovatyi <[email protected]> * Code improvements (tableau#431) * Rename tabpy_server and tabpy_tools to server and tools (breaking change) * fix flake8 warnings * Clean up code to reduce number of conditions * Remove pypi publishing instructions - those are Tableau specific * Restore tabpy_tools and tabpy_server names * Restore tabpy_tools and tabpy_server names * Restore tabpy_tools and tabpy_server names * Unit and integration tests passing * Update .gitignore * do not track settings.json for VSCode * Fix server -> tabpy_server * more cleaning for tabpy_server names * make "python setup.py test" work * add coverage module as required * delete tests node for scrutinizer run Co-authored-by: sbabayan <[email protected]> Co-authored-by: ogolovatyi <[email protected]> Co-authored-by: Brennan Bugbee <[email protected]> Co-authored-by: Logan Riggs <[email protected]> Co-authored-by: Olek Golovatyi <[email protected]> Co-authored-by: nmannheimer <[email protected]>
rishirelan · Jul 29, 2020 · ff09541 · ff09541
1 parent 94b1f7e
commit ff09541
Show file tree

Hide file tree

Showing 45 changed files with 217 additions and 267 deletions.
diff --git a/.gitignore b/.gitignore
@@ -123,8 +123,9 @@ tabpy/tabpy_server/staging
 
 # VS Code
 *.code-workspace
-.vscode
+.vscode/
 
 # etc
 setup.bat
 *~
+tabpy_log.log.1
diff --git a/.scrutinizer.yml b/.scrutinizer.yml
@@ -25,7 +25,6 @@ build:
           -
             command: pylint-run
             use_website_config: true
-    tests: true
 checks:
   python:
     code_rating: true

diff --git a/.vscode/settings.json b/.vscode/settings.json
diff --git a/CHANGELOG b/CHANGELOG
@@ -1,5 +1,11 @@
 # Changelog
 
+## v1.2.0
+
+### Improvements
+
+- Minor code cleanup.
+
 ## v1.1.0
 
 ### Improvements

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -15,7 +15,6 @@
 - [Documentation Updates](#documentation-updates)
 - [TabPy with Swagger](#tabpy-with-swagger)
 - [Code styling](#code-styling)
-- [Publishing TabPy Package](#publishing-tabpy-package)
 
 <!-- tocstop -->
 
@@ -31,7 +30,7 @@ and run it locally.
 These are prerequisites for an environment required for a contributor to
 be able to work on TabPy changes:
 
-- Python 3.6 or 3.7:
+- Python 3.6, 3.7 or 3.8:
   - To see which version of Python you have installed, run `python --version`.
 - git
 - Node.js for npm packages - install from <https://nodejs.org>.
@@ -146,25 +145,3 @@ Access-Control-Allow-Methods = GET, OPTIONS, POST
 ```sh
 flake8 .
 ```
-
-## Publishing TabPy Package
-
-Execute the following commands to build and publish a new version of
-TabPy package:
-
-```sh
-python setup.py sdist bdist_wheel
-python -m twine upload dist/*
-```
-
-To publish test version of the package use the following command:
-
-```sh
-python -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
-```
-
-To install package from TestPyPi use the command:
-
-```sh
-pip install --upgrade -i https://test.pypi.org/simple/ tabpy
-```
diff --git a/docs/security.md b/docs/security.md
@@ -6,7 +6,7 @@ you may want to consider the following as you use TabPy:
 - The REST server and Python execution share the same Python session,
   meaning that HTTP requests and user scripts are evaluated in the
   same addressable memory and processor threads.
-- The tabpy_tools client does not perform client-side validation of the
+- The tabpy.tabpy_tools client does not perform client-side validation of the
   SSL certificate on TabPy Server.
 - Python scripts can contain code which can harm security on the server
   where the TabPy is running. For example, Python scripts can:

diff --git a/docs/tabpy-tools.md b/docs/tabpy-tools.md
@@ -351,7 +351,7 @@ method provided in this tools package:
 
 ```python
 
-from tabpy_tools.schema import generate_schema
+from tabpy.tabpy_tools.schema import generate_schema
 
 schema = generate_schema(
   input={'x': 3, 'y': 2},
@@ -368,7 +368,7 @@ To describe more complex input, like arrays, you would use the following syntax:
 
 ```python
 
-from tabpy_tools.schema import generate_schema
+from tabpy.tabpy_tools.schema import generate_schema
 
 schema = generate_schema(
   input={'x': [6.35, 6.40, 6.65, 8.60],
@@ -413,10 +413,10 @@ Response:
 
 ```json
 {
-  'model': 'clustering',
-  'response': [0, 0, 0, 1, 1, 1, 1],
-  'uuid': '1ca01e46-733c-4a77-b3da-3ded84dff4cd',
-  'version': 2
+  "model": "clustering",
+  "response": [0, 0, 0, 1, 1, 1, 1],
+  "uuid": "1ca01e46-733c-4a77-b3da-3ded84dff4cd",
+  "version": 2
 }
 
 ```

diff --git a/setup.py b/setup.py
@@ -7,6 +7,7 @@
 
 import os
 from setuptools import setup, find_packages
+import unittest
 
 
 DOCLINES = (__doc__ or "").split("\n")
@@ -97,16 +98,20 @@ def read(fname):
         },
         setup_requires=["pytest-runner"],
         tests_require=[
+            "coverage",
+            "coveralls",
+            "hypothesis",
             "mock",
             "nltk",
             "numpy",
             "pandas",
             "pytest",
+            "pytest-cov",
             "scipy",
             "sklearn",
             "textblob",
         ],
-        test_suite="pytest",
+        test_suite="tests",
     )
 
 

diff --git a/tabpy/VERSION b/tabpy/VERSION
@@ -1 +1 @@
-1.1.0
+1.2.0
diff --git a/tabpy/models/deploy_models.py b/tabpy/models/deploy_models.py
@@ -1,6 +1,5 @@
 import os
 from pathlib import Path
-import pip
 import platform
 import subprocess
 import sys
@@ -9,26 +8,18 @@
 
 def main():
     # Determine if we run python or python3
-    if platform.system() == "Windows":
-        py = "python"
-    else:
-        py = "python3"
+    py = "python" if platform.system() == "Windows" else "python3"
 
-    if len(sys.argv) > 1:
-        config_file_path = sys.argv[1]
-    else:
-        config_file_path = setup_utils.get_default_config_file_path()
-    print(f"Using config file at {config_file_path}")
-    port, auth_on, prefix = setup_utils.parse_config(config_file_path)
-    if auth_on:
-        auth_args = setup_utils.get_creds()
-    else:
-        auth_args = []
+    file_path = sys.argv[1] if len(sys.argv) > 1 else setup_utils.get_default_config_file_path()
+    print(f"Using config file at {file_path}")
+
+    port, auth_on, prefix = setup_utils.parse_config(file_path)
+    auth_args = setup_utils.get_creds() if auth_on else []
 
     directory = str(Path(__file__).resolve().parent / "scripts")
     # Deploy each model in the scripts directory
     for filename in os.listdir(directory):
-        subprocess.run([py, f"{directory}/{filename}", config_file_path] + auth_args)
+        subprocess.run([py, f"{directory}/{filename}", file_path] + auth_args)
 
 
 if __name__ == "__main__":

diff --git a/tabpy/models/utils/setup_utils.py b/tabpy/models/utils/setup_utils.py
@@ -44,21 +44,14 @@ def get_creds():
 
 def deploy_model(funcName, func, funcDescription):
     # running from deploy_models.py
-    if len(sys.argv) > 1:
-        config_file_path = sys.argv[1]
-    else:
-        config_file_path = get_default_config_file_path()
+    config_file_path = sys.argv[1] if len(sys.argv) > 1 else get_default_config_file_path()
     port, auth_on, prefix = parse_config(config_file_path)
 
     connection = Client(f"{prefix}://localhost:{port}/")
 
     if auth_on:
         # credentials are passed in from setup.py
-        if len(sys.argv) == 4:
-            user, passwd = sys.argv[2], sys.argv[3]
-        # running Sentiment Analysis independently
-        else:
-            user, passwd = get_creds()
+        user, passwd = sys.argv[2], sys.argv[3] if len(sys.argv) == 4 else get_creds()
         connection.set_credentials(user, passwd)
 
     connection.deploy(funcName, func, funcDescription, override=True)

diff --git a/tabpy/tabpy_server/app/SettingsParameters.py b/tabpy/tabpy_server/app/SettingsParameters.py
diff --git a/tabpy/tabpy_server/app/app.py b/tabpy/tabpy_server/app/app.py
@@ -1,13 +1,12 @@
 import concurrent.futures
 import configparser
 import logging
-from logging import config
 import multiprocessing
 import os
 import shutil
 import signal
 import sys
-import tabpy.tabpy_server
+import tabpy
 from tabpy.tabpy import __version__
 from tabpy.tabpy_server.app.ConfigParameters import ConfigParameters
 from tabpy.tabpy_server.app.SettingsParameters import SettingsParameters
@@ -61,15 +60,16 @@ class TabPyApp:
     python_service = None
     credentials = {}
 
-    def __init__(self, config_file=None):
+    def __init__(self, config_file):
         if config_file is None:
             config_file = os.path.join(
                 os.path.dirname(__file__), os.path.pardir, "common", "default.conf"
             )
 
         if os.path.isfile(config_file):
             try:
-                logging.config.fileConfig(config_file, disable_existing_loggers=False)
+                from logging import config
+                config.fileConfig(config_file, disable_existing_loggers=False)
             except KeyError:
                 logging.basicConfig(level=logging.DEBUG)
 
@@ -243,13 +243,20 @@ def _parse_config(self, config_file):
         pkg_path = os.path.dirname(tabpy.__file__)
 
         parser = configparser.ConfigParser(os.environ)
+        logger.info(f"Parsing config file {config_file}")
 
+        file_exists = False
         if os.path.isfile(config_file):
-            with open(config_file) as f:
-                parser.read_string(f.read())
-        else:
+            try:
+                with open(config_file, 'r') as f:
+                    parser.read_string(f.read())
+                    file_exists = True
+            except Exception:
+                pass
+
+        if not file_exists:
             logger.warning(
-                f"Unable to find config file at {config_file}, "
+                f"Unable to open config file {config_file}, "
                 "using default settings."
             )
 

diff --git a/tabpy/tabpy_server/common/default.conf b/tabpy/tabpy_server/common/default.conf
diff --git a/tabpy/tabpy_server/handlers/base_handler.py b/tabpy/tabpy_server/handlers/base_handler.py
@@ -38,15 +38,8 @@ def set_request(self, request: tornado.httputil.HTTPServerRequest):
         self.method = request.method
         self.url = request.full_url()
 
-        if "TabPy-Client" in request.headers:
-            self.client = request.headers["TabPy-Client"]
-        else:
-            self.client = None
-
-        if "TabPy-User" in request.headers:
-            self.tableau_username = request.headers["TabPy-User"]
-        else:
-            self.tableau_username = None
+        self.client = request.headers.get("TabPy-Client", None)
+        self.tableau_username = request.headers.get("TabPy-User", None)
 
     def set_tabpy_username(self, tabpy_username: str):
         self.tabpy_username = tabpy_username

diff --git a/tabpy/tabpy_server/handlers/evaluation_plane_handler.py b/tabpy/tabpy_server/handlers/evaluation_plane_handler.py
@@ -134,12 +134,9 @@ def _call_subprocess(self, function_to_evaluate, arguments):
         # name - the name is actually defined with user script being wrapped
         # in _user_script function (constructed as a striong) and then executed
         # with exec() call above.
-        if arguments is None:
-            future = self.executor.submit(_user_script,  # noqa: F821
-                                          restricted_tabpy)
-        else:
-            future = self.executor.submit(_user_script,  # noqa: F821
-                                          restricted_tabpy, **arguments)
+        future = self.executor.submit(_user_script,  # noqa: F821
+                                      restricted_tabpy,
+                                      **arguments if arguments is not None else None)
 
         ret = yield gen.with_timeout(timedelta(seconds=self.eval_timeout), future)
         raise gen.Return(ret)