docs(new docs site): removes old sphinx doc site with gatsby (apache#…

…10784) * Remove old apache sphinx doc site with refreshed gatsby * remove rando * add licenses * excluing .mdx files from license checks * fixes * Clean up sphinx references * Addressed comments * colors and tweaks * more fixes * add dummy docs/requirements.txt to satisfy stupid fossa * cp .prettierrc * more licenses * minor touchups * reqs * removing old videos * add github buttons * prettier * tweaks * Eugenia/Fix some of the images and modified some of database pages * add subheadernav * add side headers nav and more responsive design for docs * update resources page and other styling updates * linting * tweaks * removing windows and align:center * update resources * remove links and and card hearder * lots of styling tweaks * Tweaks and minor improvements * lint * fix CI * trigger pre-comimt * rererefix CI Co-authored-by: Maxime Beauchemin <[email protected]> Co-authored-by: Eugenia Moreno <[email protected]> Co-authored-by: Evan Rusackas <[email protected]>
zhaowufu · Sep 12, 2020 · 85a9160 · 85a9160
1 parent 7cd96ed
commit 85a9160
Show file tree

Hide file tree

Showing 244 changed files with 31,487 additions and 4,857 deletions.
diff --git a/.fossa.yml b/.fossa.yml
@@ -32,8 +32,6 @@ analyze:
     type: pip
     target: docs
     path: docs
-    options:
-      requirements: ./requirements/documentation.txt
   - name: .
     type: pip
     target: .

diff --git a/.github/workflows/superset-docs.yml b/.github/workflows/superset-docs.yml
diff --git a/.github/workflows/superset-frontend.yml b/.github/workflows/superset-frontend.yml
@@ -25,3 +25,19 @@ jobs:
       working-directory: ./superset-frontend
       run: |
         bash <(curl -s https://codecov.io/bash) -cF javascript
+
+  docs:
+    name: build
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Install dependencies
+    - name: npm install
+      working-directory: ./superset-frontend
+      run: |
+        npm install
+    - name: gatsby build
+      working-directory: ./superset-frontend
+      run: |
+        npm run build
diff --git a/.github/workflows/superset-python.yml b/.github/workflows/superset-python.yml
@@ -4,6 +4,190 @@ name: Python
 on: [push, pull_request]
 
 jobs:
+  lint:
+    runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        python-version: [3.7]
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Setup Python
+      uses: actions/[email protected]
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      uses: apache-superset/cached-dependencies@b90713b
+      with:
+        run: |
+          apt-get-install
+          pip-upgrade
+          pip install -r requirements/testing.txt
+    - name: pylint
+      # `-j 0` run Pylint in parallel
+      run: pylint -j 0 superset
+
+  pre-commit:
+    runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        python-version: [3.7]
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Setup Python
+      uses: actions/[email protected]
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      uses: apache-superset/cached-dependencies@b90713b
+      with:
+        run: |
+          apt-get-install
+          pip-upgrade
+          pip install -r requirements/integration.txt
+    - name: pre-commit
+      run: pre-commit run --all-files
+
+  babel-extract:
+    runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        python-version: [3.7]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/[email protected]
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        uses: apache-superset/cached-dependencies@b90713b
+        with:
+          run: |
+            apt-get-install
+            pip-upgrade
+            pip install -r requirements/base.txt
+      - name: Test babel extraction
+        run: flask fab babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _,__,t,tn,tct
+
+  test-postgres-presto:
+      runs-on: ubuntu-18.04
+      strategy:
+        matrix:
+          # run unit tests in multiple version just for fun
+          python-version: [3.8]
+      env:
+        PYTHONPATH: ${{ github.workspace }}
+        SUPERSET_CONFIG: tests.superset_test_config
+        REDIS_PORT: 16379
+        SUPERSET__SQLALCHEMY_DATABASE_URI:
+          postgresql+psycopg2://superset:[email protected]:15432/superset
+        SUPERSET__SQLALCHEMY_EXAMPLES_URI:
+          presto://localhost:15433/memory/default
+      services:
+        postgres:
+          image: postgres:10-alpine
+          env:
+            POSTGRES_USER: superset
+            POSTGRES_PASSWORD: superset
+          ports:
+            # Use custom ports for services to avoid accidentally connecting to
+            # GitHub action runner's default installations
+            - 15432:5432
+        presto:
+          image: prestosql/presto:339
+          env:
+            POSTGRES_USER: superset
+            POSTGRES_PASSWORD: superset
+          ports:
+            # Use custom ports for services to avoid accidentally connecting to
+            # GitHub action runner's default installations
+            - 15433:8080
+        redis:
+          image: redis:5-alpine
+          ports:
+            - 16379:6379
+      steps:
+      - uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/[email protected]
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        uses: apache-superset/cached-dependencies@b90713b
+        with:
+          run: |
+            apt-get-install
+            pip-upgrade
+            pip install -r requirements/testing.txt
+            setup-postgres
+      - name: Run celery
+        run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
+      - name: Python unit tests (PostgreSQL)
+        run: |
+          ./scripts/python_tests.sh
+      - name: Upload code coverage
+        run: |
+          bash <(curl -s https://codecov.io/bash) -cF python
+
+  test-postgres-hive:
+    runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        # run unit tests in multiple version just for fun
+        python-version: [3.7, 3.8]
+    env:
+      PYTHONPATH: ${{ github.workspace }}
+      SUPERSET_CONFIG: tests.superset_test_config
+      REDIS_PORT: 16379
+      SUPERSET__SQLALCHEMY_DATABASE_URI:
+        postgresql+psycopg2://superset:[email protected]:15432/superset
+      SUPERSET__SQLALCHEMY_EXAMPLES_URI: hive://localhost:10000/default
+      UPLOAD_FOLDER: /tmp/.superset/uploads/
+    services:
+      postgres:
+        image: postgres:10-alpine
+        env:
+          POSTGRES_USER: superset
+          POSTGRES_PASSWORD: superset
+        ports:
+          # Use custom ports for services to avoid accidentally connecting to
+          # GitHub action runner's default installations
+          - 15432:5432
+      redis:
+        image: redis:5-alpine
+        ports:
+          - 16379:6379
+    steps:
+    - uses: actions/checkout@v2
+    - name: Create csv upload directory
+      run: sudo mkdir -p /tmp/.superset/uploads
+    - name: Give write access to the csv upload directory
+      run: sudo chown -R $USER:$USER /tmp/.superset
+    - name: Start hadoop and hive
+      run: docker-compose -f scripts/databases/hive/docker-compose.yml up -d
+    - name: Setup Python
+      uses: actions/[email protected]
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      uses: apache-superset/cached-dependencies@b90713b
+      with:
+        run: |
+          apt-get-install
+          pip-upgrade
+          pip install -r requirements/testing.txt
+          setup-postgres
+    - name: Run celery
+      run: celery worker --app=superset.tasks.celery_app:app -Ofair -c 2 &
+    - name: Python unit tests (PostgreSQL)
+      run: |
+        ./scripts/python_tests.sh
+    - name: Upload code coverage
+      run: |
+        bash <(curl -s https://codecov.io/bash) -cF python
+
   test-postgres:
     runs-on: ubuntu-18.04
     strategy:

diff --git a/.github/workflows/superset-translations.yml b/.github/workflows/superset-translations.yml
@@ -35,7 +35,7 @@ jobs:
           run: |
             apt-get-install
             pip-upgrade
-            pip install -r requirements/documentation.txt
+            pip install -r requirements/base.txt
       - name: Test babel extraction
         run: |
           flask fab babel-extract --target superset/translations \

diff --git a/.rat-excludes b/.rat-excludes
@@ -46,3 +46,4 @@ requirements/*
 vendor/*
 # github configuration
 .github/*
+.*mdx
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -205,7 +205,7 @@ Finally, never submit a PR that will put master branch in broken state. If the P
   - For Python, include it in `setup.py` denoting any specific restrictions and in `requirements.txt` pinned to a specific version which ensures that the application build is deterministic.
   - For TypeScript/JavaScript, include new libraries in `package.json`
 - **Tests:** The pull request should include tests, either as doctests, unit tests, or both. Make sure to resolve all errors and test failures. See [Testing](#testing) for how to run tests.
-- **Documentation:** If the pull request adds functionality, the docs should be updated as part of the same PR. Doc string are often sufficient, make sure to follow the sphinx compatible standards.
+- **Documentation:** If the pull request adds functionality, the docs should be updated as part of the same PR.
 - **CI:** Reviewers will not review the code until all CI tests are passed. Sometimes there can be flaky tests. You can close and open PR to re-run CI test. Please report if the issue persists. After the CI fix has been deployed to `master`, please rebase your PR.
 - **Code coverage:** Please ensure that code coverage does not decrease.
 - Remove `[WIP]` when ready for review. Please note that it may be merged soon after approved so please make sure the PR is ready to merge and do not expect more time for post-approval edits.
@@ -299,68 +299,10 @@ cd incubator-superset
 
 ### Documentation
 
-The latest documentation and tutorial are available at https://superset.incubator.apache.org/.
+The latest documentation and tutorial are available at https://superset.apache.org/.
 
-Contributing to the official documentation is relatively easy, once you've setup
-your environment and done an edit end-to-end. The docs can be found in the
-`docs/` subdirectory of the repository, and are written in the
-[reStructuredText format](https://en.wikipedia.org/wiki/ReStructuredText) (.rst).
-If you've written Markdown before, you'll find the reStructuredText format familiar.
-
-Superset uses [Sphinx](http://www.sphinx-doc.org/en/1.5.1/) to convert the rst files
-in `docs/` to the final HTML output users see.
-
-Finally, to make changes to the rst files and build the docs using Sphinx,
-you'll need to install a handful of dependencies from the repo you cloned:
-
-```bash
-pip install -r requirements/documentation.txt
-```
-
-To get the feel for how to edit and build the docs, let's edit a file, build
-the docs and see our changes in action. First, you'll want to
-[create a new branch](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging)
-to work on your changes:
-
-```bash
-git checkout -b changes-to-docs
-```
-
-Now, go ahead and edit one of the files under `docs/`, say `docs/tutorial.rst` - change
-it however you want. Check out the
-[ReStructuredText Primer](http://docutils.sourceforge.net/docs/user/rst/quickstart.html)
-for a reference on the formatting of the rst files.
-
-Once you've made your changes, run this command to convert the docs into HTML:
-
-```bash
-make html
-```
-
-You'll see a lot of output as Sphinx handles the conversion. After it's done, the
-HTML Sphinx generated should be in `docs/_build/html`. Navigate there
-and start a simple web server so we can check out the docs in a browser:
-
-```bash
-cd docs/_build/html
-python -m http.server # Python2 users should use SimpleHTTPServer
-
-```
-
-This will start a small Python web server listening on port 8000. Point your
-browser to http://localhost:8000, find the file
-you edited earlier, and check out your changes!
-
-If you've made a change you'd like to contribute to the actual docs, just commit
-your code, push your new branch to Github:
-
-```bash
-git add docs/tutorial.rst
-git commit -m 'Awesome new change to tutorial'
-git push origin changes-to-docs
-```
-
-Then, [open a pull request](https://help.github.com/articles/about-pull-requests/).
+The site is written using the Gatsby framework and docz for the
+documentation subsection. Find out more about it in `docs/README.md`
 
 #### Images
 
@@ -377,15 +319,6 @@ in the docs.
 
 For example, the image referenced above actually lives in `superset-frontend/images/tutorial`. Since the image is moved during the documentation build process, the docs reference the image in `_static/images/tutorial` instead.
 
-#### API documentation
-
-Generate the API documentation with:
-
-```bash
-pip install -r requirements/documentation.txt
-python setup.py build_sphinx
-```
-
 ### Flask server
 
 #### OS Dependencies
@@ -604,7 +537,7 @@ or similar as the later will cause typing issues. The former is of type `List[Ca
 
 To ensure clarity, consistency, all readability, _all_ new functions should use
 [type hints](https://docs.python.org/3/library/typing.html) and include a
-docstring using Sphinx documentation.
+docstring.
 
 Note per [PEP-484](https://www.python.org/dev/peps/pep-0484/#exceptions) no
 syntax for listing explicitly raised exceptions is proposed and thus the

diff --git a/RELEASING/Dockerfile.make_docs b/RELEASING/Dockerfile.make_docs
@@ -20,7 +20,7 @@ ARG VERSION
 RUN git clone --depth 1 --branch ${VERSION} https://github.com/apache/incubator-superset.git /superset
 WORKDIR /superset
 # install doc dependencies
-RUN pip install -r requirements/documentation.txt
+RUN pip install -r requirements/base.txt
 # build the docs
 RUN python setup.py build_sphinx