[AIRFLOW-5206] Common licence in all .md files, TOC + removed TODO.md (…

…apache#5809)
george79diakos · Aug 22, 2019 · e090744 · e090744
1 parent cd4ab7b
commit e090744
Show file tree

Hide file tree

Showing 10 changed files with 182 additions and 197 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -77,6 +77,24 @@ repos:
           - "<!--||-->"
           - --license-filepath
           - license-templates/LICENSE.txt
+      - id: insert-license
+        name: Add licence for all md files
+        files: \.md$
+        exclude: ^\.github/.*$|^airflow/_vendor/.*$
+        args:
+          - --comment-style
+          - "<!--|| -->"
+          - --license-filepath
+          - license-templates/LICENSE.txt
+  - repo: https://github.com/thlorenz/doctoc.git
+    rev: v1.4.0
+    hooks:
+      - id: doctoc
+        name: Add TOC for md files
+        files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README.md$
+        args:
+          - "--maxlevel"
+          - "2"
   - repo: meta
     hooks:
       - id: check-hooks-apply

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,22 +1,21 @@
 <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements.  See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership.  The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License.  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied.  See the License for the
-specific language governing permissions and limitations
-under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
 -->
-
 # Contributing
 
 Contributions are welcome and are greatly appreciated! Every
@@ -40,26 +39,17 @@ little bit helps, and credit will always be given.
 - [Integration test development environment](#integration-test-development-environment)
   - [Prerequisites](#prerequisites)
   - [Using the Docker Compose environment](#using-the-docker-compose-environment)
-    - [Entering bash shell in Docker Compose environment](#entering-bash-shell-in-docker-compose-environment)
-    - [Running individual tests within the container](#running-individual-tests-within-the-container)
   - [Running static code analysis](#running-static-code-analysis)
-      - [Running static code analysis from the host](#running-static-code-analysis-from-the-host)
-      - [Running static code analysis in the docker compose environment](#running-static-code-analysis-in-the-docker-compose-environment)
-      - [Running static code analysis on selected files/modules](#running-static-code-analysis-on-selected-filesmodules)
   - [Docker images](#docker-images)
   - [Default behaviour for user interaction](#default-behaviour-for-user-interaction)
   - [Local Docker Compose scripts](#local-docker-compose-scripts)
-    - [Running the whole suite of tests](#running-the-whole-suite-of-tests)
-    - [Stopping the environment](#stopping-the-environment)
-    - [Fixing file/directory ownership](#fixing-filedirectory-ownership)
-    - [Building the images](#building-the-images)
-    - [Force pulling the images](#force-pulling-the-images)
   - [Cleaning up cached Docker images/containers](#cleaning-up-cached-docker-imagescontainers)
   - [Troubleshooting](#troubleshooting)
 - [Pylint checks](#pylint-checks)
 - [Pre-commit hooks](#pre-commit-hooks)
   - [Installing pre-commit hooks](#installing-pre-commit-hooks)
   - [Docker images for pre-commit hooks](#docker-images-for-pre-commit-hooks)
+  - [Prerequisites for pre-commit hooks](#prerequisites-for-pre-commit-hooks)
   - [Pre-commit hooks installed](#pre-commit-hooks-installed)
   - [Using pre-commit hooks](#using-pre-commit-hooks)
   - [Skipping pre-commit hooks](#skipping-pre-commit-hooks)
@@ -73,10 +63,6 @@ little bit helps, and credit will always be given.
 - [Setting up the node / npm javascript environment](#setting-up-the-node--npm-javascript-environment)
   - [Node/npm versions](#nodenpm-versions)
   - [Using npm to generate bundled files](#using-npm-to-generate-bundled-files)
-    - [npm](#npm)
-    - [npm packages](#npm-packages)
-    - [Upgrading npm packages](#upgrading-npm-packages)
-    - [Javascript Style Guide](#javascript-style-guide)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -712,6 +698,7 @@ check-hooks-apply                Check hooks apply to the repository
 check-merge-conflict             Checks if merge conflict is being committed
 check-executables-have-shebangs  Check that executables have shebang
 check-xml                        Check XML files with xmllint
+doctoc                           Refresh table-of-contents for md files
 detect-private-key               Detects if private key is added to the repository
 end-of-file-fixer                Make sure that there is an empty line at the end
 flake8                           Run flake8

diff --git a/README.md b/README.md
@@ -1,22 +1,21 @@
 <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements.  See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership.  The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License.  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied.  See the License for the
-specific language governing permissions and limitations
-under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
 -->
-
 # Apache Airflow
 
 [![PyPI version](https://badge.fury.io/py/apache-airflow.svg)](https://badge.fury.io/py/apache-airflow)
@@ -35,6 +34,21 @@ versionable, testable, and collaborative.
 
 Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of contents**
+
+- [Getting started](#getting-started)
+- [Beyond the Horizon](#beyond-the-horizon)
+- [Principles](#principles)
+- [User Interface](#user-interface)
+- [Contributing](#contributing)
+- [Who uses Apache Airflow?](#who-uses-apache-airflow)
+- [Who Maintains Apache Airflow?](#who-maintains-apache-airflow)
+- [Links](#links)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
 ## Getting started
 
 Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html).

diff --git a/TODO.md b/TODO.md
diff --git a/UPDATING.md b/UPDATING.md
@@ -1,26 +1,42 @@
 <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements.  See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership.  The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License.  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied.  See the License for the
-specific language governing permissions and limitations
-under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
 -->
 # Updating Airflow
 
 This file documents any backwards-incompatible changes in Airflow and
 assists users migrating to a new version.
 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of contents**
+
+- [Airflow Master](#airflow-master)
+- [Airflow 1.10.4](#airflow-1104)
+- [Airflow 1.10.3](#airflow-1103)
+- [Airflow 1.10.2](#airflow-1102)
+- [Airflow 1.10.1](#airflow-1101)
+- [Airflow 1.10](#airflow-110)
+- [Airflow 1.9](#airflow-19)
+- [Airflow 1.8.1](#airflow-181)
+- [Airflow 1.8](#airflow-18)
+- [Airflow 1.7.1.2](#airflow-1712)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 ## Airflow Master
 
 ### Changes to ImapHook, ImapAttachmentSensor and ImapAttachmentToS3Operator

diff --git a/airflow/contrib/example_dags/example_twitter_README.md b/airflow/contrib/example_dags/example_twitter_README.md
@@ -1,22 +1,21 @@
 <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements.  See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership.  The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License.  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied.  See the License for the
-specific language governing permissions and limitations
-under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
 -->
-
 # Example Twitter DAG
 
 ***Introduction:*** This example dag depicts a typical ETL process and is a perfect use case automation scenario for Airflow. Please note that the main scripts associated with the tasks are returning None. The purpose of this DAG is to demonstrate how to write a functional DAG within Airflow.

diff --git a/dev/README.md b/dev/README.md
@@ -1,24 +1,32 @@
 <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements.  See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership.  The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License.  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied.  See the License for the
-specific language governing permissions and limitations
-under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
 -->
-
 # Development Tools
 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of contents**
+
+- [Airflow Pull Request Tool](#airflow-pull-request-tool)
+- [Airflow release signing tool](#airflow-release-signing-tool)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
 ## Airflow Pull Request Tool
 
 The `airflow-pr` tool interactively guides committers through the process of merging GitHub PRs into Airflow and closing associated JIRA issues.