Skip to content

Commit

Permalink
Merge pull request apache#1184 from airbnb/changes_to_contributing
Browse files Browse the repository at this point in the history 
Changes to Contributing to reflect more closely the current state of dev on Airflow
  • Loading branch information
@artwr
artwr committed Mar 21, 2016
2 parents e96d2d3 + b44dbd4 commit 7dca663
Show file tree
Hide file tree
Showing 3 changed files with 104 additions and 80 deletions.
71 changes: 42 additions & 29 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,22 @@
Contributions are welcome and are greatly appreciated! Every
little bit helps, and credit will always be given.

You can contribute in many ways:

# Table of Contents
* [TOC](#table-of-contents)
* [Types of Contributions](#types-of-contribution)
- [Report Bugs](#report-bugs)
- [Fix Bugs](#fix-bugs)
- [Implement Features](#implement-features)
- [Improve Documentation](#improve-documentation)
- [Submit Feedback](#submit-feedback)
* [Documentation](#documentation)
* [Development and Testing](#development-and-testing)
- [Setting up a development environment](#setting-up-a-development-environment)
- [Pull requests guidelines](#pull-requests-guidelines)
- [Testing Locally](#testing-locally)
* [Changing the Metadata Database](#changing-the-metadata-database)


## Types of Contributions

Expand Down Expand Up @@ -32,7 +47,7 @@ We've created the operators, hooks, macros and executors we needed, but we
made sure that this part of Airflow is extensible. New operators,
hooks and operators are very welcomed!

### Documentation
### Improve Documentation

Airflow could always use better documentation,
whether as part of the official Airflow docs,
Expand All @@ -51,54 +66,52 @@ If you are proposing a feature:
- Remember that this is a volunteer-driven project, and that
contributions are welcome :)

## Latests Documentation
## Documentation

The latest API documentation is usually available [here](http://pythonhosted.com/airflow). To generate a local version, you need to have installed airflow with the `doc` extra. In that case you can generate the doc by running:

[API Documentation](http://pythonhosted.com/airflow)
cd docs && ./build.sh

## Development and Testing

## Testing
### Setting up a development environment

Assuming you are working in a virtualenv. Install development requirements:
It is usually best to work in a virtualenv. Install development requirements:

cd $AIRFLOW_HOME
virtualenv env
source env/bin/activate
pip install -r requirements.txt
python setup.py develop

Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):

./run_unit_tests.sh

Lint the project with:

flake8 changes tests
pip install -e .[devel]

## API documentation
Feel free to customize based on the extras available in [setup.py](./setup.py)

Generate the documentation with:

cd docs && ./build.sh


## Pull Request Guidelines
### Pull Request Guidelines

Before you submit a pull request from your forked repo, check that it
meets these guidelines:

1. The pull request should include tests, either as doctests, unit tests, or
both.
both. The airflow repo uses [Travis CI](https://travis-ci.org/airbnb/airflow) to run the tests and [coveralls](https://coveralls.io/github/airbnb/airflow) to track coverage. You can set up both for free on your fork. It will help you making sure you do not break the build with your PR and that you help increase coverage.
2. If the pull request adds functionality, the docs should be updated as part
of the same PR. Doc string are often sufficient. Make sure to follow the
sphinx compatible standards.
3. The pull request should work for Python 2.6, 2.7, and 3.3. If you need help
3. The pull request should work for Python 2.7 and 3.4. If you need help
writing code that works in both Python 2 and 3, see the documentation at the
[Python-Future project](http://python-future.org) (the future package is an
Airflow requirement and should be used where possible).
4. Code will be reviewed by re running the unittests and `flake8`. Syntax should
be as rigorous as the core Python project.
4. As Airflow grows as a project, we try to enforce a more consistent style and try to follow the Python community guidelines. We track this using [landscape.io](https://landscape.io/github/airbnb/airflow/), which you can setup on your fork as well to check before you submit your PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/) and a few other linting rules. It is usually a good idea to lint locally as well using [flake8](https://flake8.readthedocs.org/en/latest/) using `flake8 airflow tests`
5. Please rebase and resolve all conflicts before submitting.

## Running unit tests
### Testing locally

#### TL;DR
Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):

./run_unit_tests.sh

#### Running unit tests

We *highly* recommend setting up [Travis CI](https://travis-ci.org/) on your repo to automate this. It is free for open source projects. If for some reason you cannot, you can use the steps below to run tests.

Here are loose guidelines on how to get your environment to run the unit tests.
We do understand that no one out there can run the full test suite since
Expand Down Expand Up @@ -137,7 +150,7 @@ nosetests docs.

See also the the list of test classes and methods in `tests/code.py`.

## Changing Metadata Database
### Changing the Metadata Database

When developing features the need may arise to persist information to the the
metadata database. Airflow has [Alembic](https://bitbucket.org/zzzeek/alembic)
Expand Down
100 changes: 53 additions & 47 deletions docs/installation.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,53 +20,59 @@ these extra dependencies.

Here's the list of the subpackages and what they enable:

+-------------+------------------------------------+------------------------------------------------+
| subpackage | install command | enables |
+=============+====================================+================================================+
| mysql | ``pip install airflow[mysql]`` | MySQL operators and hook, support as |
| | | an Airflow backend |
+-------------+------------------------------------+------------------------------------------------+
| postgres | ``pip install airflow[postgres]`` | Postgres operators and hook, support |
| | | as an Airflow backend |
+-------------+------------------------------------+------------------------------------------------+
| samba | ``pip install airflow[samba]`` | ``Hive2SambaOperator`` |
+-------------+------------------------------------+------------------------------------------------+
| hive | ``pip install airflow[hive]`` | All Hive related operators |
+-------------+------------------------------------+------------------------------------------------+
| jdbc | ``pip install airflow[jdbc]`` | JDBC hooks and operators |
+-------------+------------------------------------+------------------------------------------------+
| hdfs | ``pip install airflow[hdfs]`` | HDFS hooks and operators |
+-------------+------------------------------------+------------------------------------------------+
| s3 | ``pip install airflow[s3]`` | ``S3KeySensor``, ``S3PrefixSensor`` |
+-------------+------------------------------------+------------------------------------------------+
| druid | ``pip install airflow[druid]`` | Druid.io related operators & hooks |
+-------------+------------------------------------+------------------------------------------------+
| mssql | ``pip install airflow[mssql]`` | Microsoft SQL operators and hook, |
| | | support as an Airflow backend |
+-------------+------------------------------------+------------------------------------------------+
| vertica | ``pip install airflow[vertica]`` | Vertica hook |
| | | support as an Airflow backend |
+-------------+------------------------------------+------------------------------------------------+
| slack | ``pip install airflow[slack]`` | ``SlackAPIPostOperator`` |
+-------------+------------------------------------+------------------------------------------------+
| all | ``pip install airflow[all]`` | All Airflow features known to man |
+-------------+------------------------------------+------------------------------------------------+
| devel | ``pip install airflow[devel]`` | All Airflow features + useful dev tools |
+-------------+------------------------------------+------------------------------------------------+
| crypto | ``pip install airflow[crypto]`` | Encrypt connection passwords in metadata db |
+-------------+------------------------------------+------------------------------------------------+
| celery | ``pip install airflow[celery]`` | CeleryExecutor |
+-------------+------------------------------------+------------------------------------------------+
| async | ``pip install airflow[async]`` | Async worker classes for gunicorn |
+-------------+------------------------------------+------------------------------------------------+
| ldap | ``pip install airflow[ldap]`` | ldap authentication for users |
+-------------+------------------------------------+------------------------------------------------+
| kerberos | ``pip install airflow[kerberos]`` | kerberos integration for kerberized hadoop |
+-------------+------------------------------------+------------------------------------------------+
| password | ``pip install airflow[password]`` | Password Authentication for users |
+-------------+------------------------------------+------------------------------------------------+
| qds | ``pip install airflow[qds]`` | Enable QDS (qubole data services) support |
+-------------+------------------------------------+------------------------------------------------+
+---------------+-------------------------------------+-------------------------------------------------+
| subpackage | install command | enables |
+===============+=====================================+=================================================+
| mysql | ``pip install airflow[mysql]`` | MySQL operators and hook, support as |
| | | an Airflow backend |
+---------------+-------------------------------------+-------------------------------------------------+
| postgres | ``pip install airflow[postgres]`` | Postgres operators and hook, support |
| | | as an Airflow backend |
+---------------+-------------------------------------+-------------------------------------------------+
| samba | ``pip install airflow[samba]`` | ``Hive2SambaOperator`` |
+---------------+-------------------------------------+-------------------------------------------------+
| hive | ``pip install airflow[hive]`` | All Hive related operators |
+---------------+-------------------------------------+-------------------------------------------------+
| jdbc | ``pip install airflow[jdbc]`` | JDBC hooks and operators |
+---------------+-------------------------------------+-------------------------------------------------+
| hdfs | ``pip install airflow[hdfs]`` | HDFS hooks and operators |
+---------------+-------------------------------------+-------------------------------------------------+
| s3 | ``pip install airflow[s3]`` | ``S3KeySensor``, ``S3PrefixSensor`` |
+---------------+-------------------------------------+-------------------------------------------------+
| druid | ``pip install airflow[druid]`` | Druid.io related operators & hooks |
+---------------+-------------------------------------+-------------------------------------------------+
| mssql | ``pip install airflow[mssql]`` | Microsoft SQL operators and hook, |
| | | support as an Airflow backend |
+---------------+-------------------------------------+-------------------------------------------------+
| vertica | ``pip install airflow[vertica]`` | Vertica hook |
| | | support as an Airflow backend |
+---------------+-------------------------------------+-------------------------------------------------+
| slack | ``pip install airflow[slack]`` | ``SlackAPIPostOperator`` |
+---------------+-------------------------------------+-------------------------------------------------+
| all | ``pip install airflow[all]`` | All Airflow features known to man |
+---------------+-------------------------------------+-------------------------------------------------+
| all_dbs | ``pip install airflow[all_dbs]`` | All databases integrations |
+---------------+-------------------------------------+-------------------------------------------------+
| devel | ``pip install airflow[devel]`` | Minimum dev tools requirements |
+---------------+-------------------------------------+-------------------------------------------------+
| devel_hadoop |``pip install airflow[devel_hadoop]``| Airflow + dependencies on the Hadoop stack |
+---------------+-------------------------------------+-------------------------------------------------+
| crypto | ``pip install airflow[crypto]`` | Encrypt connection passwords in metadata db |
+---------------+-------------------------------------+-------------------------------------------------+
| celery | ``pip install airflow[celery]`` | CeleryExecutor |
+---------------+-------------------------------------+-------------------------------------------------+
| async | ``pip install airflow[async]`` | Async worker classes for gunicorn |
+---------------+-------------------------------------+-------------------------------------------------+
| ldap | ``pip install airflow[ldap]`` | ldap authentication for users |
+---------------+-------------------------------------+-------------------------------------------------+
| kerberos | ``pip install airflow[kerberos]`` | kerberos integration for kerberized hadoop |
+---------------+-------------------------------------+-------------------------------------------------+
| password | ``pip install airflow[password]`` | Password Authentication for users |
+---------------+-------------------------------------+-------------------------------------------------+
| rabbitmq | ``pip install airflow[rabbitmq]`` | Rabbitmq support as a Celery backend |
+---------------+-------------------------------------+-------------------------------------------------+
| qds | ``pip install airflow[qds]`` | Enable QDS (qubole data services) support |
+---------------+-------------------------------------+-------------------------------------------------+

Configuration
'''''''''''''
Expand Down
13 changes: 9 additions & 4 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def run(self):
jdbc = ['jaydebeapi>=0.2.0']
mssql = ['pymssql>=2.1.1', 'unicodecsv>=0.14.1']
mysql = ['mysqlclient>=1.3.6']
optional = ['librabbitmq>=1.6.1']
rabbitmq = ['librabbitmq>=1.6.1']
oracle = ['cx_Oracle>=5.1.2']
postgres = ['psycopg2>=2.6']
s3 = [
Expand All @@ -97,7 +97,10 @@ def run(self):

all_dbs = postgres + mysql + hive + mssql + hdfs + vertica
devel = ['lxml>=3.3.4', 'nose', 'mock']
devel += all_dbs + doc + samba + s3 + slack + crypto + oracle + docker
devel_minreq = devel + mysql + doc + password + s3
devel_hadoop = devel_minreq + hive + hdfs + webhdfs + kerberos
devel_all = devel + all_dbs + doc + samba + s3 + slack + crypto + oracle + docker


setup(
name='airflow',
Expand Down Expand Up @@ -132,12 +135,13 @@ def run(self):
'Flask-WTF==0.12'
],
extras_require={
'all': devel + optional,
'all': devel_all,
'all_dbs': all_dbs,
'async': async,
'celery': celery,
'crypto': crypto,
'devel': devel,
'devel': devel_minreq,
'devel_hadoop': devel_hadoop,
'doc': doc,
'docker': docker,
'druid': druid,
Expand All @@ -150,6 +154,7 @@ def run(self):
'mysql': mysql,
'oracle': oracle,
'postgres': postgres,
'rabbitmq': rabbitmq,
's3': s3,
'samba': samba,
'slack': slack,
Expand Down

0 comments on commit 7dca663

Please sign in to comment.