Skip to content

Commit

Permalink
edits to clarify backtesting analysis
Browse files Browse the repository at this point in the history
  • Loading branch information
@jraviotta
jraviotta committed Aug 9, 2019
1 parent 2bc67b4 commit ccf3c69
Show file tree
Hide file tree
Showing 3 changed files with 151 additions and 76 deletions.
66 changes: 40 additions & 26 deletions docs/data-analysis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,33 @@

You can analyze the results of backtests and trading history easily using Jupyter notebooks. A sample notebook is located at `user_data/notebooks/analysis_example.ipynb`. For usage instructions, see [jupyter.org](https://jupyter.org/documentation).

## Example snippets

### Load backtest results into a pandas dataframe

```python
# Load backtest results
df = load_backtest_data("user_data/backtest_data/backtest-result.json")

# Show value-counts per pair
df.groupby("pair")["sell_reason"].value_counts()
```

### Load live trading results into a pandas dataframe

``` python
# Fetch trades from database
df = load_trades_from_db("sqlite:///tradesv3.sqlite")

# Display results
df.groupby("pair")["sell_reason"].value_counts()
```

## Strategy debugging example

Debugging a strategy can be time-consuming. FreqTrade offers helper functions to visualize raw data.

### Import requirements and define variables used in the script
### Import requirements and define variables used in analyses

```python
# Imports
Expand Down Expand Up @@ -47,46 +69,38 @@ print("Loaded " + str(len(bt_data)) + f" rows of data for {pair} from {data_loca
### Load and run strategy

* Rerun each time the strategy file is changed
* Display the trade details. Note that using `data.head()` would also work, however most indicators have some "startup" data at the top of the dataframe.

Some possible problems:

* Columns with NaN values at the end of the dataframe
* Columns used in `crossed*()` functions with completely different units

```python
# Load strategy using values set above
strategy = StrategyResolver({'strategy': strategy_name,
'user_data_dir': user_data_dir,
'strategy_path': strategy_location}).strategy

# Run strategy (just like in backtesting)
# Generate buy/sell signals using strategy
df = strategy.analyze_ticker(bt_data, {'pair': pair})

# Report results
print(f"Generated {df['buy'].sum()} buy signals")
data = df.set_index('date', drop=True)
data.tail()
```

### Load backtest results into a pandas dataframe
### Display the trade details

```python
# Load backtest results
df = load_backtest_data("user_data/backtest_data/backtest-result.json")
* Note that using `data.head()` would also work, however most indicators have some "startup" data at the top of the dataframe.

# Show value-counts per pair
df.groupby("pair")["sell_reason"].value_counts()
```
#### Some possible problems

### Load live trading results into a pandas dataframe
* Columns with NaN values at the end of the dataframe
* Columns used in `crossed*()` functions with completely different units

``` python
# Fetch trades from database
df = load_trades_from_db("sqlite:///tradesv3.sqlite")
#### Comparison with full backtest

# Display results
df.groupby("pair")["sell_reason"].value_counts()
having 200 buy signals as output for one pair from `analyze_ticker()` does not necessarily mean that 200 trades will be made during backtesting.

Assuming you use only one condition such as, `df['rsi'] < 30` as buy condition, this will generate multiple "buy" signals for each pair in sequence (until rsi returns > 29).
The bot will only buy on the first of these signals (and also only if a trade-slot ("max_open_trades") is still available), or on one of the middle signals, as soon as a "slot" becomes available.

```python
# Report results
print(f"Generated {df['buy'].sum()} buy signals")
data = df.set_index('date', drop=True)
data.tail()
```

Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data.
12 changes: 10 additions & 2 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,15 @@
'pytest-random-order',
]

all_extra = api + plot + develop
jupyter = [
'jupyter',
'nbstripout',
'ipykernel',
'isort',
'yapf',
]

all_extra = api + plot + develop + jupyter

setup(name='freqtrade',
version=__version__,
Expand Down Expand Up @@ -68,7 +76,7 @@
'dev': all_extra,
'plot': plot,
'all': all_extra,
'jupyter': [],
'jupyter': jupyter,

},
include_package_data=True,
Expand Down
149 changes: 101 additions & 48 deletions user_data/notebooks/analysis_example.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,24 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Strategy debugging example"
"# Analyzing bot data\n",
"\n",
"You can analyze the results of backtests and trading history easily using Jupyter notebooks. A sample notebook is located at `user_data/notebooks/analysis_example.ipynb`. For usage instructions, see [jupyter.org](https://jupyter.org/documentation)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Imports\n",
"from pathlib import Path\n",
"import os\n",
"from freqtrade.data.history import load_pair_history\n",
"from freqtrade.resolvers import StrategyResolver\n",
"from freqtrade.data.btanalysis import load_backtest_data\n",
"from freqtrade.data.btanalysis import load_trades_from_db"
]
},
{
Expand All @@ -15,11 +32,9 @@
"source": [
"# Change directory\n",
"# Define all paths relative to the project root shown in the cell output\n",
"import os\n",
"from pathlib import Path\n",
"try:\n",
"\tos.chdir(Path(os.getcwd(), '../..'))\n",
"\tprint(os.getcwd())\n",
"\tos.chdir(Path(Path.cwd(), '../..'))\n",
"\tprint(Path.cwd())\n",
"except:\n",
"\tpass"
]
Expand All @@ -28,7 +43,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import requirements and define variables used in the script"
"## Example snippets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load backtest results into a pandas dataframe"
]
},
{
Expand All @@ -37,23 +59,65 @@
"metadata": {},
"outputs": [],
"source": [
"# Imports\n",
"from pathlib import Path\n",
"from freqtrade.data.history import load_pair_history\n",
"from freqtrade.resolvers import StrategyResolver\n",
"from freqtrade.data.btanalysis import load_backtest_data\n",
"from freqtrade.data.btanalysis import load_trades_from_db\n",
"# Load backtest results\n",
"df = load_backtest_data(\"user_data/backtest_data/backtest-result.json\")\n",
"\n",
"# Show value-counts per pair\n",
"df.groupby(\"pair\")[\"sell_reason\"].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load live trading results into a pandas dataframe"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Fetch trades from database\n",
"df = load_trades_from_db(\"sqlite:///tradesv3.sqlite\")\n",
"\n",
"# Display results\n",
"df.groupby(\"pair\")[\"sell_reason\"].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Strategy debugging example\n",
"\n",
"Debugging a strategy can be time-consuming. FreqTrade offers helper functions to visualize raw data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import requirements and define variables used in analyses"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Define some constants\n",
"ticker_interval = \"1m\"\n",
"# Name of the strategy class\n",
"strategy_name = 'NewStrategy'\n",
"# Path to user data\n",
"user_data_dir = 'user_data'\n",
"# Location of the strategy\n",
"strategy_location = Path(user_data_dir, 'strategies')\n",
"strategy_location = os.path.join(user_data_dir, 'strategies')\n",
"# Location of the data\n",
"data_location = Path(user_data_dir, 'data', 'binance')\n",
"data_location = os.path.join(user_data_dir, 'data', 'binance')\n",
"# Pair to analyze \n",
"# Only use one pair here\n",
"pair = \"BTC_USDT\""
Expand Down Expand Up @@ -85,15 +149,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load and run strategy \n",
"\n",
"* Rerun each time the strategy file is changed\n",
"* Display the trade details. Note that using `data.head()` would also work, however most indicators have some \"startup\" data at the top of the dataframe.\n",
"\n",
"Some possible problems:\n",
"\n",
"* Columns with NaN values at the end of the dataframe\n",
"* Columns used in `crossed*()` functions with completely different units"
"### Load and run strategy\n",
"* Rerun each time the strategy file is changed"
]
},
{
Expand All @@ -107,20 +164,28 @@
" 'user_data_dir': user_data_dir,\n",
" 'strategy_path': strategy_location}).strategy\n",
"\n",
"# Run strategy (just like in backtesting)\n",
"df = strategy.analyze_ticker(bt_data, {'pair': pair})\n",
"\n",
"# Report results\n",
"print(f\"Generated {df['buy'].sum()} buy signals\")\n",
"data = df.set_index('date', drop=True)\n",
"data.tail()"
"# Generate buy/sell signals using strategy\n",
"df = strategy.analyze_ticker(bt_data, {'pair': pair})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load backtest results into a pandas dataframe"
"### Display the trade details\n",
"* Note that using `data.head()` would also work, however most indicators have some \"startup\" data at the top of the dataframe.\n",
"\n",
"#### Some possible problems\n",
"\n",
"* Columns with NaN values at the end of the dataframe\n",
"* Columns used in `crossed*()` functions with completely different units\n",
"\n",
"#### Comparison with full backtest\n",
"\n",
"having 200 buy signals as output for one pair from `analyze_ticker()` does not necessarily mean that 200 trades will be made during backtesting.\n",
"\n",
"Assuming you use only one condition such as, `df['rsi'] < 30` as buy condition, this will generate multiple \"buy\" signals for each pair in sequence (until rsi returns > 29).\n",
"The bot will only buy on the first of these signals (and also only if a trade-slot (\"max_open_trades\") is still available), or on one of the middle signals, as soon as a \"slot\" becomes available.\n"
]
},
{
Expand All @@ -129,18 +194,10 @@
"metadata": {},
"outputs": [],
"source": [
"# Load backtest results\n",
"df = load_backtest_data(\"user_data/backtest_data/backtest-result.json\")\n",
"\n",
"# Show value-counts per pair\n",
"df.groupby(\"pair\")[\"sell_reason\"].value_counts()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load live trading results into a pandas dataframe"
"# Report results\n",
"print(f\"Generated {df['buy'].sum()} buy signals\")\n",
"data = df.set_index('date', drop=True)\n",
"data.tail()"
]
},
{
Expand All @@ -149,11 +206,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Fetch trades from database\n",
"df = load_trades_from_db(\"sqlite:///tradesv3.sqlite\")\n",
"\n",
"# Display results\n",
"df.groupby(\"pair\")[\"sell_reason\"].value_counts()"
"Feel free to submit an issue or Pull Request enhancing this document if you would like to share ideas on how to best analyze the data."
]
}
],
Expand Down

0 comments on commit ccf3c69

Please sign in to comment.