Merge branch 'master' into hivemeta_sasl

.coveralls.yml

@@ -1 +1 @@
-repo_token: ic8IH7CrUrtweVbmY3VZQ7ncEGe1XJA5E
+service_name: travis-ci

.gitignore

@@ -23,3 +23,4 @@ sftp-config.json
 unittests.cfg
 error.log
 unittests.db
+rat-results.txt

.landscape.yml

@@ -0,0 +1,13 @@
+max-line-length: 90
+strictness: medium
+pep8:
+  full: true
+uses:
+  - flask
+pylint:
+  disable:
+    - cyclic-import
+    - invalid-name
+    - super-on-old-class
+  options:
+    docstring-min-length: 10

.rat-excludes

@@ -0,0 +1,20 @@
+.gitignore
+.gitattributes
+.coverage
+.coveragerc
+.coveralls.yml
+.rat-excludes
+requirements.txt
+.*log
+.travis.yml
+.*pyc
+docs
+.*md
+dist
+build
+airflow.egg-info
+.idea
+metastore_db
+.*sql
+.*csv
+

.travis.yml

@@ -70,6 +70,7 @@ install:
 before_script:
   - mysql -e 'drop database if exists airflow; create database airflow' -u root
   - psql -c 'create database airflow;' -U postgres
+  - export PATH=${PATH}:/tmp/hive/bin
 script:
   - pip --version
   - ls -l $HOME/.wheelhouse

CONTRIBUTING.md

@@ -3,20 +3,31 @@
 Contributions are welcome and are greatly appreciated! Every
 little bit helps, and credit will always be given.
 
-You can contribute in many ways:
+
+# Table of Contents
+  * [TOC](#table-of-contents)
+  * [Types of Contributions](#types-of-contribution)
+      - [Report Bugs](#report-bugs)
+      - [Fix Bugs](#fix-bugs)
+      - [Implement Features](#implement-features)
+      - [Improve Documentation](#improve-documentation)
+      - [Submit Feedback](#submit-feedback)
+  * [Documentation](#documentation)
+  * [Development and Testing](#development-and-testing)
+      - [Setting up a development environment](#setting-up-a-development-environment)
+      - [Pull requests guidelines](#pull-requests-guidelines)
+      - [Testing Locally](#testing-locally)
+  * [Changing the Metadata Database](#changing-the-metadata-database)
+
 
 ## Types of Contributions
 
 ### Report Bugs
 
 Report bugs through Github
 
-If you are reporting a bug, please include:
+We now provide a [templated issue form](https://github.com/airbnb/airflow/blob/master/ISSUE_TEMPLATE.md). You must provide information for all required fields else the bug will be closed.
 
--   Your operating system name and version.
--   Any details about your local setup that might be helpful in
-    troubleshooting.
--   Detailed steps to reproduce the bug.
 
 ### Fix Bugs
 
@@ -32,7 +43,7 @@ We've created the operators, hooks, macros and executors we needed, but we
 made sure that this part of Airflow is extensible. New operators,
 hooks and operators are very welcomed!
 
-### Documentation
+### Improve Documentation
 
 Airflow could always use better documentation,
 whether as part of the official Airflow docs,
@@ -51,54 +62,52 @@ If you are proposing a feature:
 -   Remember that this is a volunteer-driven project, and that
     contributions are welcome :)
 
-## Latests Documentation
+## Documentation
+
+The latest API documentation is usually available [here](http://pythonhosted.org/airflow). To generate a local version, you need to have installed airflow with the `doc` extra. In that case you can generate the doc by running:
 
-[API Documentation](http://pythonhosted.com/airflow)
+    cd docs && ./build.sh
+
+## Development and Testing
 
-## Testing
+### Setting up a development environment
 
-Assuming you are working in a virtualenv. Install development requirements:
+It is usually best to work in a virtualenv. Install development requirements:
 
     cd $AIRFLOW_HOME
     virtualenv env
     source env/bin/activate
-    pip install -r requirements.txt
-    python setup.py develop
-
-Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):
-
-    ./run_unit_tests.sh
-
-Lint the project with:
-
-    flake8 changes tests
+    pip install -e .[devel]
 
-## API documentation
+Feel free to customize based on the extras available in [setup.py](./setup.py)
 
-Generate the documentation with:
-
-    cd docs && ./build.sh
-
-
-## Pull Request Guidelines
+### Pull Request Guidelines
 
 Before you submit a pull request from your forked repo, check that it
 meets these guidelines:
 
 1. The pull request should include tests, either as doctests, unit tests, or
-both.
+both. The airflow repo uses [Travis CI](https://travis-ci.org/airbnb/airflow) to run the tests and [coveralls](https://coveralls.io/github/airbnb/airflow) to track coverage. You can set up both for free on your fork. It will help you making sure you do not break the build with your PR and that you help increase coverage.
 2. If the pull request adds functionality, the docs should be updated as part
 of the same PR. Doc string are often sufficient.  Make sure to follow the
 sphinx compatible standards.
-3. The pull request should work for Python 2.6, 2.7, and 3.3. If you need help
+3. The pull request should work for Python 2.7 and 3.4. If you need help
 writing code that works in both Python 2 and 3, see the documentation at the
 [Python-Future project](http://python-future.org) (the future package is an
 Airflow requirement and should be used where possible).
-4. Code will be reviewed by re running the unittests and `flake8`. Syntax should
-be as rigorous as the core Python project.
+4. As Airflow grows as a project, we try to enforce a more consistent style and try to follow the Python community guidelines. We track this using [landscape.io](https://landscape.io/github/airbnb/airflow/), which you can setup on your fork as well to check before you submit your PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/) and a few other linting rules. It is usually a good idea to lint locally as well using [flake8](https://flake8.readthedocs.org/en/latest/) using `flake8 airflow tests`
 5. Please rebase and resolve all conflicts before submitting.
 
-## Running unit tests
+### Testing locally
+
+#### TL;DR 
+Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):
+
+    ./run_unit_tests.sh
+
+#### Running unit tests
+
+We *highly* recommend setting up [Travis CI](https://travis-ci.org/) on your repo to automate this. It is free for open source projects. If for some reason you cannot, you can use the steps below to run tests.
 
 Here are loose guidelines on how to get your environment to run the unit tests.
 We do understand that no one out there can run the full test suite since
@@ -137,7 +146,7 @@ nosetests docs.
 
 See also the the list of test classes and methods in `tests/code.py`.
 
-## Changing Metadata Database
+### Changing the Metadata Database
 
 When developing features the need may arise to persist information to the the
 metadata database. Airflow has [Alembic](https://bitbucket.org/zzzeek/alembic)

ISSUE_TEMPLATE.md

@@ -0,0 +1,28 @@
+Dear Airflow Maintainers,
+
+Before I tell you about my issue, let me describe my environment:
+
+# Environment
+
+* Version of Airflow (e.g. a release version, running your own fork, running off master -- provide a git log snippet) :
+* Example code to reproduce the bug (as a code snippet in markdown)
+* Screen shots of your DAG's graph and tree views:
+* Stack trace if applicable:
+* Operating System: (Windows Version or `$ uname -a`)
+* Python Version: `$ python --version`
+* (Optional) Python packages: `$ pip freeze` or `$ conda list`
+
+Now that you know a little about me, let me tell you about the issue I am having:
+
+# Description of Issue
+
+* What did you expect to happen?
+* What happened instead?
+* Here is how you can reproduce this issue on your machine:
+
+## Reproduction Steps
+
+1.
+2.
+3.
+

README.md

@@ -1,26 +1,29 @@
 # Airflow
 
-[![Join the chat at https://gitter.im/airbnb/airflow](https://badges.gitter.im/airbnb/airflow.svg)](https://gitter.im/airbnb/airflow?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![PyPI version](https://badge.fury.io/py/airflow.svg)](https://badge.fury.io/py/airflow)
 [![Build Status](https://travis-ci.org/airbnb/airflow.svg)](https://travis-ci.org/airbnb/airflow)
 [![Coverage Status](https://coveralls.io/repos/airbnb/airflow/badge.svg?service=github)](https://coveralls.io/github/airbnb/airflow)
 [![Code Health](https://landscape.io/github/airbnb/airflow/master/landscape.svg?style=flat)](https://landscape.io/github/airbnb/airflow/master)
+[![Requirements Status](https://requires.io/github/airbnb/airflow/requirements.svg?branch=master)](https://requires.io/github/airbnb/airflow/requirements/?branch=master)
 [![pypi downloads](https://img.shields.io/pypi/dm/airflow.svg)](https://pypi.python.org/pypi/airflow/)
+[![Documentation](https://img.shields.io/badge/docs-pythonhosted-blue.svg)](http://pythonhosted.org/airflow/)
+[![Join the chat at https://gitter.im/airbnb/airflow](https://badges.gitter.im/airbnb/airflow.svg)](https://gitter.im/airbnb/airflow?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 Airflow is a platform to programmatically author, schedule and monitor
 workflows.
 
 When workflows are defined as code, they become more maintainable,
 versionable, testable, and collaborative.
 
-![img] (http://i.imgur.com/6Gs4hxT.gif)
-
 Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks.
 The airflow scheduler executes your tasks on an array of workers while
 following the specified dependencies. Rich command line utilities make
 performing complex surgeries on DAGs a snap. The rich user interface
 makes it easy to visualize pipelines running in production,
 monitor progress, and troubleshoot issues when needed.
 
+For more information and documentation, please visit the [Airflow Wiki](https://github.com/airbnb/airflow/wiki).
+
 ## Beyond the Horizon
 
 Airflow **is not** a data streaming solution. Tasks do not move data from
@@ -43,12 +46,36 @@ unit of work and continuity.
 - **Elegant**:  Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
 - **Scalable**:  Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.
 
+## User Interface
+
+- **DAGs**: Overview of all DAGs in your environment.
+![](/docs/img/dags.png)
+
+- **Tree View**: Tree representation of a DAG that spans across time.
+![](/docs/img/tree.png)
+
+- **Graph View**: Visualization of a DAG's dependencies and their current status for a specific run.
+![](/docs/img/graph.png)
+
+- **Task Duration**: Total time spent on different tasks over time.
+![](/docs/img/duration.png)
+
+- **Gantt View**: Duration and overlap of a DAG.
+![](/docs/img/gantt.png)
+
+- **Code View**:  Quick way to view source code of a DAG.
+![](/docs/img/code.png)
+
 ## Who uses Airflow?
 
 As the Airflow community grows, we'd like to keep track of who is using
 the platform. Please send a PR with your company name and @githubhandle
 if you may.
 
+Committers:
+
+* Refer to [Committers](https://github.com/airbnb/airflow/wiki/Committers)
+
 Currently **officially** using Airflow:
 
 * [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)]
@@ -61,10 +88,11 @@ Currently **officially** using Airflow:
 * [Cotap](https://github.com/cotap/) [[@maraca](https://github.com/maraca) & [@richardchew](https://github.com/richardchew)]
 * Easy Taxi [[@caique-lima](https://github.com/caique-lima)]
 * [FreshBooks](https://github.com/freshbooks) [[@DinoCow](https://github.com/DinoCow)]
+* [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk)]
 * [Handy](http://www.handy.com/careers/73115?gh_jid=73115&gh_src=o5qcxn) [[@marcintustin](https://github.com/marcintustin) / [@mtustin-handy](https://github.com/mtustin-handy)]
 * [Holimetrix](http://holimetrix.com/) [[@thibault-ketterer](https://github.com/thibault-ketterer)]
 * [Hootsuite](https://github.com/hootsuite)
-* ING
+* [ING](http://www.ing.com/)
 * [Jampp](https://github.com/jampp)
 * [Kogan.com](https://github.com/kogan) [[@geeknam](https://github.com/geeknam)]
 * [LendUp](https://www.lendup.com/) [[@lendup](https://github.com/lendup)]
@@ -79,14 +107,15 @@ Currently **officially** using Airflow:
 * [WeTransfer](https://github.com/WeTransfer) [[@jochem](https://github.com/jochem)]
 * Wooga
 * Xoom [[@gepser](https://github.com/gepser) & [@omarvides](https://github.com/omarvides)]
+* [WePay](http://www.wepay.com) [[@criccomini](https://github.com/criccomini) & [@mtagle](https://github.com/mtagle)]
 * Yahoo!
 
 ## Links
 
-* [Full documentation on pythonhosted.com](http://pythonhosted.org/airflow/)
+* [Full documentation on pythonhosted.org](http://pythonhosted.org/airflow/)
 * [Airflow Google Group (mailing list / forum)](https://groups.google.com/forum/#!forum/airbnb_airflow)
 * [Airbnb Blog Post about Airflow](http://nerds.airbnb.com/airflow/)
-* [Airflow Common Pitfalls](https://github.com/airbnb/airflow/wiki/Airflow-Common-Pitfalls)
+* [Airflow Common Pitfalls](https://github.com/airbnb/airflow/wiki/Common-Pitfalls)
 * [Hadoop Summit Airflow Video](https://www.youtube.com/watch?v=oYp49mBwH60)
 * [Airflow at Agari Blog Post](http://agari.com/blog/airflow-agari)
 * [Best practices with Airflow (Max) nov 2015](https://youtu.be/dgaoqOZlvEA)

airflow/__init__.py

@@ -1,3 +1,18 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 """
 Authentication is implemented using flask_login and different environments can
 implement their own login mechanisms by providing an `airflow_login` module

airflow/bin/airflow

@@ -2,14 +2,14 @@
 import logging
 import os
 from airflow import configuration
-from airflow.bin.cli import get_parser
+from airflow.bin.cli import CLIFactory
 
 if __name__ == '__main__':
 
     if configuration.get("core", "security") == 'kerberos':
         os.environ['KRB5CCNAME'] = configuration.get('kerberos', 'ccache')
         os.environ['KRB5_KTNAME'] = configuration.get('kerberos', 'keytab')
 
-    parser = get_parser()
+    parser = CLIFactory.get_parser()
     args = parser.parse_args()
     args.func(args)