Skip to content

Commit

Permalink
Feature ap 243/v3 client (quandl#37)
Browse files Browse the repository at this point in the history
* new quandl client using v3 api

* send up request source and version

* allow passing in file name, and allow passing in download type as query param

* Update readme docs for upgrading and data tables.

* Increase the versions of python we test against on CI and tox. Remove .get from models to avoid 2x call to the api.

Fix some issues where _raw_data was not named correctly.

Remove version 3.2 from travis as it fails even though it should not.

* data tables for developers

* Fix a few flake8 issues and run it locally with tox.

* Improve the connection error handling code based on suggestion from community member femtotrader.

* fix file format

* Improve upgrade documentation to take into account pickled versions of api_key.

* Notes based on feedback.

* analysts get and bulkdownload methods

* refactoring get_table

* using python bool and remove returns options

* remove user warning

* Update README.md

* Reduce memory usage and processing time by not using ordered-dictionaries for the data.

Update the way nested dates are detected and converted into real dates. Also update tests to match.

Add jsondate to tox dependencies.

Ensure that tox installs same dependencies as setup.py

* Updates based on feedback.

* MergedDataset: dont make metadata calls when getting data

* Update FOR_ANALYSTS.md

add bulkdownload description

some updates

* Missing package in setup.

* release setup

* messages

* Update 2_SERIES_UPGRADE.md

* ready and message updates

* Update README.md

minor change to test branch

* add changelog
  • Loading branch information
ccleung authored and junos committed Apr 22, 2016
1 parent 2118659 commit a0fa903
Show file tree
Hide file tree
Showing 70 changed files with 3,308 additions and 457 deletions.
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@

*.pyc

/.tox/
/.eggs
/quandl.egg-info
/build
/dist
/tmp/
14 changes: 14 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
sudo: false
language: python
python:
- "2.7"
- "3.3"
- "3.4"
- "3.5"
install:
- pip install flake8
- pip install -r requirements.txt
script:
- flake8
- python -W always setup.py -q test
cache: pip
49 changes: 49 additions & 0 deletions 2_SERIES_UPGRADE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# 2.x Series package Notes

With the release of version 3 of our API we are officially deprecating version 2 of the Quandl Python package. We have re-written the package and will be moving forward with a 3.x.x package under the new namespace of `quandl` that will rely on version 3 of our RESTful API.

## Upgrading

There are numerous advantages to upgrade from the older 2.x series package including improved performance and stability. The upgrade process is fairly simple.

Upgrade your package using pip and running: `pip install --upgrade quandl`

Wherever you have:

```python
import Quandl
```

change this to:

```python
import quandl as Quandl
```

Additionally if you were relying on a saved version of your `api key` _(authtoken)_ in your scripts please note that this functionality has been dropped from the existing package. To continue to load your api key from the filesystem we recommend you import the key and set it via the api config `quandl.ApiConfig.api_key`. Below is a example of importing your previously pickled key if it exists:

```python
import quandl as Quandl
import pickle
import os.path
if os.path.isfile('authtoken.p'):
Quandl.ApiConfig.api_key = pickle.load(open('authtoken.p', 'rb'))
```

That's it! The new package should be ready to use.

## Continuing with the Old Release

If you wish to continue using the old package during this transitional period please follow the instructions below:

https://github.com/quandl/quandl-python/tree/v2.8.9

To continue using Quandl API version 2, do the following:

1. Ensure you have [pip installed](https://pip.pypa.io/en/latest/installing.html)

2. In your Python program's directory, execute `pip freeze > requirements.txt`. Alternatively, create the `requirements.txt` file and enter the desired Quandl package version, e.g., `Quandl==2.8.8`.

3. Execute `pip install -r requirements.txt` to ensure the desired Quandl package version is installed.

Note: for the sample code to work you must have access to read the file with the auth token. The sample script also assumes that file is stored in the same directory from where the script is run. If you wish to use a key in another directory you will need to match the path used in the sample.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
### 3.0.0 - 2016-04-22

* Datatables for developers
* Datatables for analysts
* Backwards compatibility
* Documentation updates
159 changes: 159 additions & 0 deletions FOR_ANALYSTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# Quick Method Guide - Quandl-Python

This quick guide offers convienent ways to retrieve individual datasets or datatables with the Python package without the need for complex commands.

## Retrieving Data

Retrieving data can be achieved easily using the two methods `quandl.get` for datasets and `quandl.get_table` for datatables. In both cases we strongly recommend that you set your api key via:

```python
import quandl
quandl.ApiConfig.api_key = 'tEsTkEy123456789'
```

### Datasets

To retrieve a Quandl dataset you can make the following call:

```python
import quandl
data = quandl.get('NSE/OIL')
```

This finds all data points for the dataset `NSE/OIL` and stores them in a pandas dataframe. You can then view the dataframe with:

```python
data.head()
```

However we recommend applying filters to streamline the results. To do this you may want to specify some additional filters and transformations.

```python
import quandl
data = quandl.get('NSE/OIL', start_date='2010-01-01', end_date='2014-01-01',
collapse='annual', transformation='rdiff',
rows=4)
```

This revised query will find all data points annually for the dataset `NSE/OIL` between the year 2010 and 2014 and transform them using the `rdiff` transformation. The query parameters used are documented in our [api docs](https://www.quandl.com/docs/api#data). Since the `data` was retrieved in the pandas format you can then view the dataframe with data.head().

#### Available parameters:

The following additional parameters can be specified for a dataset call:

| Option | Explanation | Example | Description |
|---|---|---|---|
| api_key | Your access key | `api_key='tEsTkEy123456789'` | Used to identify who you are and provide more access. Only required if not set via `quandl.ApiConfig.api_key=` |
| \<filter / transformation parameter\> | A parameter which filters or transforms the resulting data | `start_date='2010-01-01` | For a full list see our [api docs](https://www.quandl.com/docs/api#data) |

For more information on how to use and manipulate the resulting data see the [pandas documentation](http://pandas.pydata.org/).

#### Download Entire Database (Bulk Download)

You can download all the data in a database in a single call. The following will download the entire EOD database as a zip file to your current working directory:

```python
import quandl
quandl.bulkdownload('EOD')
```

After the download is finished, the `quandl.bulkdownload` will return the filename of the downloaded zip file.

To download database data from the previous day, use the download_type option:

```python
import quandl
quandl.bulkdownload('EOD', download_type='partial')
```

You can also change the filename of the downloaded zip file by using the filename option:

```python
import quandl
quandl.bulkdownload('EOD', filename='/my/path/EOD_DB.zip')
```

#### Download Multiple Codes

Sometimes you want to compare two codes. For example if you wanted to compare the closing prices for Apple and Microsoft, you would obtain the two Quandl codes:

`GOOG/NASDAQ_AAPL` vs. `GOOG/NASDAQ_MSFT`

Append the column you wish to get with a `.`, and put them into an array.

`['GOOG/NASDAQ_AAPL.4','GOOG/NASDAQ_MSFT.4']`

Just make a normal get call with the array passed to get, and your multiset will be returned.

#### Download Multiple Codes (Multiset)

If you want to get multiple codes at once, delimit the codes with ',', and put them into an array. This will return a multiset.

For example:
```python
data = quandl.get(['GOOG/NASDAQ_AAPL.4','GOOG/NASDAQ_MSFT.4'])
```

Which outputs:

```
See www.quandl.com/docs/api for more information.
Returning Dataframe for ['GOOG.NASDAQ_AAPL.4', 'GOOG.NASDAQ_MSFT.4']
GOOG.NASDAQ_AAPL - Close GOOG.NASDAQ_MSFT - Close
Date
1997-08-20 6.16 17.57
1997-08-21 6.00 17.23
1997-08-22 5.91 17.16
1997-08-25 5.77 17.06
1997-08-26 5.56 16.88
```

### Datatables

Datatables work similarly to datasets but provide more flexibility when it comes to filtering. For example a simple way to retrieve datatable information would be:

```python
import quandl
data = quandl.get_table('ZACKS/FC')
```

Given the volume of data stored in datatables, this call will retrieve the first page of the `ZACKS/FC` datatable. You may turn on pagination to return more data by using:

```python
import quandl
data = quandl.get_table('ZACKS/FC', paginate=True)
```

This will retrieve multiple pages of data and merge them together as if they were one large page. In some cases, however, you will still exceed the request limit. In this case we recommend you filter your data using the available query parameters, as in the following example:

```python
import quandl
data = quandl.get_table('ZACKS/FC', paginate=True, m_ticker=['AAPL', 'MSFT'], per_end_date={'gte': '2015-01-01'}}, qopts={'columns':['m_ticker', 'per_end_date']})
```

In this query we are asking for more pages of data, `m_ticker` values of either `AAPL` or `MSFT` and a `per_end_date` that is greater than or equal to `2015-01-01`. We are also filtering the returned columns on `m_ticker`, `per_end_date` and `comp_name` rather than all available columns. The output format is `pandas`.

#### Available parameters:

The following additional parameters can be specified for a datatable call:

| Option | Explanation | Example | Description |
|---|---|---|---|
| api_key | Your access key | `api_key='tEsTkEy123456789'` | Used to identify who you are and provide more access. Only required if not set via `quandl.ApiConfig.api_key=` |
| \<filter / transformation parameter\> | A parameter which filters or transforms the resulting data | `start_date='2010-01-01'` | For a full list see our [api docs](https://www.quandl.com/docs/api#datatables) |
| paginate | Wether to autoamtically paginate data | `paginate=True` | Will paginate through the first few pages of data automatically and merge them together in a larger output format. |

For more information on how to use and manipulate the resulting data see the [pandas documentation](http://pandas.pydata.org/).

#### Things to note

* Some datatables will return `sample` data if a valid api key is not used. If you are not recieving all of the expected data please double check your API key.
* When using the paginate=True option depending on the total number of rows in the result set you may receive an error indicating that there are more pages that have not been downloaded. This is due to a very large result sets that would be too large to send via the analyst method. If this happens we recommend you take one of two approaches:
* *(recommended)* Refine your filter parameters to retrieve a smaller results set
* Use the the [Detailed](./FOR_DEVELOPERS.md) method to iterate through more of the data.

## More usages

For even more advanced usage please see our [Detailed Method Guide] (./FOR_DEVELOPERS.md).
Loading

0 comments on commit a0fa903

Please sign in to comment.