Skip to content

Commit

Permalink
[AIRFLOW-3934] Increase standard Dataproc PD size (apache#4749)
Browse files Browse the repository at this point in the history
  • Loading branch information
Fokko authored and kaxil committed Apr 15, 2019
1 parent 04e4e4e commit c63ddcc
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 4 deletions.
11 changes: 9 additions & 2 deletions UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ assists users migrating to a new version.

## Airflow Master

### Increase standard Dataproc disk sizes

It is highly recommended to have 1TB+ disk size for Dataproc to have sufficient throughput:
https://cloud.google.com/compute/docs/disks/performance

Hence, the default value for `master_disk_size` in DataprocClusterCreateOperator has beeen changes from 500GB to 1TB.

### Changes to SalesforceHook

* renamed `sign_in` function to `get_conn`
Expand Down Expand Up @@ -86,14 +93,14 @@ compatibility, this option is enabled by default.

The deprecated import mechanism has been removed so the import of modules becomes more consistent and explicit.

For example: `from airflow.operators import BashOperator`
For example: `from airflow.operators import BashOperator`
becomes `from airflow.operators.bash_operator import BashOperator`

### Changes to sensor imports

Sensors are now accessible via `airflow.sensors` and no longer via `airflow.operators.sensors`.

For example: `from airflow.operators.sensors import BaseSensorOperator`
For example: `from airflow.operators.sensors import BaseSensorOperator`
becomes `from airflow.sensors.base_sensor_operator import BaseSensorOperator`

### Renamed "extra" requirements for cloud providers
Expand Down
4 changes: 2 additions & 2 deletions airflow/contrib/operators/dataproc_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,10 @@ def __init__(self,
properties=None,
master_machine_type='n1-standard-4',
master_disk_type='pd-standard',
master_disk_size=500,
master_disk_size=1024,
worker_machine_type='n1-standard-4',
worker_disk_type='pd-standard',
worker_disk_size=500,
worker_disk_size=1024,
num_preemptible_workers=0,
labels=None,
region='global',
Expand Down

0 comments on commit c63ddcc

Please sign in to comment.