Backtester: Live trading upgrades (thrasher-corp#1023)

* Modifications for a smoother live run

* Fixes data appending

* Successfully allows multi-currency live trading. Adds multiple currencies to live DCA strategy

* Attempting to get cash and carry working

* Poor attempts at sorting out data and appending it properly with USD in mind

* =designs new live data handler

* Updates cash and carry strat to work

* adds test coverage. begins closeallpositions function

* Updates cash and carry to work live

* New kline.Event type. Cancels orders on close. Rn types

* =Fixes USD funding issue

* =fixes tests

* fixes tests AGAIN

* adds coverage to close all orders

* crummy tests, should override

* more tests

* more tests

* more coverage

* removes scourge of currency.Pair maps. More tests

* missed currency stuff

* Fixes USD data issue & collateral issue. Needs to close ALL orders

* Now triggers updates on the very first data entry

* All my problems are solved now????

* fixes tests, extends coverage

* there is some really funky candle stuff going on

* my brain is melting

* better shutdown management, fixes freezing bug

* fixes data duplication issues, adds retries to requests

* reduces logging, adds verbose options

* expands coverage over all new functionality

* fixes fun bug from curr == curr to curr.Equal(curr)

* fixes setup issues and tests

* starts adding external wallet amounts for funding

* more setup for assets

* setup live fund calcs and placing orders

* successfully performs automated cash and carry

* merge fixes

* funding properly set at all times

* fixes some bugs, need to address currencystatistics still

* adds 'appeneded' trait, attempts to fix some stats

* fixes stat bugs, adds cool new fetchfees feature

* fixes terrible processing bugs

* tightens realorder stats, sadly loses some live stats

* this actually sets everything correctly for bothcd ..cd ..cd ..cd ..cd ..!

* fix tests

* coverage

* beautiful new test coverage

* docs

* adds new fee getter delayer

* commits from the correct directory

* Lint

* adds verbose to fund manager

* Fix bug in t2b2 strat. Update dca live config. Docs

* go mod tidy

* update buf

* buf + test improvement

* Post merge fixes

* fixes surprise offset bug

* fix sizing restrictions for cash and carry

* fix server lints

* merge fixes

* test fixesss

* lintle fixles

* slowloris

* rn run to task, bug fixes, close all on close

* rpc lint and fixes

* bugfix: order manager not processing orders properly

* somewhat addresses nits

* absolutely broken end of day commit

* absolutely massive knockon effects from nits

* massive knockon effects continue

* fixes things

* address remaining nits

* jk now fixes things

* addresses the easier nits

* more nit fixers

* more niterinos addressederinos

* refactors holdings and does some nits

* so buf

* addresses some nits, fixes holdings bugs

* cleanup

* attempts to fix alert chans to prevent many chans waiting?

* terrible code, will revert

* to be reviewed in detail tomorrow

* Fixes up channel system

* smashes those nits

* fixes extra candles, fixes collateral bug, tests

* fixes data races, introduces reflection

* more checks n tests

* Fixes cash and carry issues. Fixes more cool bugs

* fixes ~typer~ typo

* replace spot strats from ftx to binance

* fixes all the tests I just destroyed

* removes example path, rm verbose

* 1) what 2) removes FTX references from the Backtester

* renamed, non-working strategies

* Removes FTX references almost as fast as sbf removes funds

* regen docs, add contrib names,sort contrib names

* fixes merge renamings

* Addresses nits. Fixes setting API credentials. Fixes Binance limit retrieval

* Fixes live order bugs with real orders and without

* Apply suggestions from code review

Co-authored-by: Adrian Gallagher <[email protected]>

* Update backtester/engine/live.go

Co-authored-by: Adrian Gallagher <[email protected]>

* Update backtester/engine/live.go

Co-authored-by: Adrian Gallagher <[email protected]>

* Update backtester/config/strategyconfigbuilder/main.go

Co-authored-by: Adrian Gallagher <[email protected]>

* updates docs

* even better docs

Co-authored-by: Adrian Gallagher <[email protected]>

.gitignore

@@ -15,6 +15,7 @@ vendor/
 
 # Binaries for programs and plugins
 gocryptotrader
+/backtester/backtester
 cmd/gctcli/gctcli
 backtester/backtester
 backtester/btcli/btcli

CONTRIBUTORS

@@ -22,6 +22,7 @@ khcchiu | https://github.com/khcchiu
 woshidama323 | https://github.com/woshidama323
 yangrq1018 | https://github.com/yangrq1018
 TaltaM | https://github.com/TaltaM
+samuael | https://github.com/samuael
 crackcomm | https://github.com/crackcomm
 azhang | https://github.com/azhang
 andreygrehov | https://github.com/andreygrehov
@@ -30,27 +31,26 @@ Christian-Achilli | https://github.com/Christian-Achilli
 MarkDzulko | https://github.com/MarkDzulko
 gam-phon | https://github.com/gam-phon
 cornelk | https://github.com/cornelk
-if1live | https://github.com/if1live
 herenow | https://github.com/herenow
+if1live | https://github.com/if1live
+lozdog245 | https://github.com/lozdog245
 mshogin | https://github.com/mshogin
 soxipy | https://github.com/soxipy
 tk42 | https://github.com/tk42
+blombard | https://github.com/blombard
+cavapoo2 | https://github.com/cavapoo2
+CodeLingoTeam | https://github.com/CodeLingoTeam
+CodeLingoBot | https://github.com/CodeLingoBot
+Daanikus | https://github.com/Daanikus
 daniel-cohen | https://github.com/daniel-cohen
 DirectX | https://github.com/DirectX
 frankzougc | https://github.com/frankzougc
 idoall | https://github.com/idoall
-mattkanwisher | https://github.com/mattkanwisher
-mKurrels | https://github.com/mKurrels
-m1kola | https://github.com/m1kola
-cavapoo2 | https://github.com/cavapoo2
-zeldrinn | https://github.com/zeldrinn
-starit | https://github.com/starit
 Jimexist | https://github.com/Jimexist
 lookfirst | https://github.com/lookfirst
+m1kola | https://github.com/m1kola
+mattkanwisher | https://github.com/mattkanwisher
 merkeld | https://github.com/merkeld
-CodeLingoTeam | https://github.com/CodeLingoTeam
-Daanikus | https://github.com/Daanikus
-CodeLingoBot | https://github.com/CodeLingoBot
-blombard | https://github.com/blombard
-soxipy | https://github.com/soxipy
-lozdog245 | https://github.com/lozdog245
+mKurrels | https://github.com/mKurrels
+starit | https://github.com/starit
+zeldrinn | https://github.com/zeldrinn

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2014-2022 The GoCryptoTrader Developers
+Copyright (c) 2014-2023 The GoCryptoTrader Developers
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -146,9 +146,9 @@ Binaries will be published once the codebase reaches a stable condition.
 |User|Contribution Amount|
 |--|--|
 | [thrasher-](https://github.com/thrasher-) | 670 |
-| [shazbert](https://github.com/shazbert) | 268 |
-| [gloriousCode](https://github.com/gloriousCode) | 202 |
-| [dependabot[bot]](https://github.com/apps/dependabot) | 130 |
+| [shazbert](https://github.com/shazbert) | 269 |
+| [gloriousCode](https://github.com/gloriousCode) | 205 |
+| [dependabot[bot]](https://github.com/apps/dependabot) | 139 |
 | [dependabot-preview[bot]](https://github.com/apps/dependabot-preview) | 88 |
 | [xtda](https://github.com/xtda) | 47 |
 | [lrascao](https://github.com/lrascao) | 27 |
@@ -160,13 +160,14 @@ Binaries will be published once the codebase reaches a stable condition.
 | [vadimzhukck](https://github.com/vadimzhukck) | 10 |
 | [140am](https://github.com/140am) | 8 |
 | [marcofranssen](https://github.com/marcofranssen) | 8 |
-| [geseq](https://github.com/geseq) | 7 |
+| [geseq](https://github.com/geseq) | 8 |
 | [dackroyd](https://github.com/dackroyd) | 5 |
 | [cranktakular](https://github.com/cranktakular) | 5 |
 | [khcchiu](https://github.com/khcchiu) | 5 |
 | [woshidama323](https://github.com/woshidama323) | 3 |
 | [yangrq1018](https://github.com/yangrq1018) | 3 |
 | [TaltaM](https://github.com/TaltaM) | 3 |
+| [samuael](https://github.com/samuael) | 3 |
 | [crackcomm](https://github.com/crackcomm) | 3 |
 | [azhang](https://github.com/azhang) | 2 |
 | [andreygrehov](https://github.com/andreygrehov) | 2 |
@@ -175,27 +176,26 @@ Binaries will be published once the codebase reaches a stable condition.
 | [MarkDzulko](https://github.com/MarkDzulko) | 2 |
 | [gam-phon](https://github.com/gam-phon) | 2 |
 | [cornelk](https://github.com/cornelk) | 2 |
-| [if1live](https://github.com/if1live) | 2 |
 | [herenow](https://github.com/herenow) | 2 |
+| [if1live](https://github.com/if1live) | 2 |
+| [lozdog245](https://github.com/lozdog245) | 2 |
 | [mshogin](https://github.com/mshogin) | 2 |
 | [soxipy](https://github.com/soxipy) | 2 |
 | [tk42](https://github.com/tk42) | 2 |
+| [blombard](https://github.com/blombard) | 1 |
+| [cavapoo2](https://github.com/cavapoo2) | 1 |
+| [CodeLingoTeam](https://github.com/CodeLingoTeam) | 1 |
+| [CodeLingoBot](https://github.com/CodeLingoBot) | 1 |
+| [Daanikus](https://github.com/Daanikus) | 1 |
 | [daniel-cohen](https://github.com/daniel-cohen) | 1 |
 | [DirectX](https://github.com/DirectX) | 1 |
 | [frankzougc](https://github.com/frankzougc) | 1 |
 | [idoall](https://github.com/idoall) | 1 |
-| [mattkanwisher](https://github.com/mattkanwisher) | 1 |
-| [mKurrels](https://github.com/mKurrels) | 1 |
-| [m1kola](https://github.com/m1kola) | 1 |
-| [cavapoo2](https://github.com/cavapoo2) | 1 |
-| [zeldrinn](https://github.com/zeldrinn) | 1 |
-| [starit](https://github.com/starit) | 1 |
 | [Jimexist](https://github.com/Jimexist) | 1 |
 | [lookfirst](https://github.com/lookfirst) | 1 |
+| [m1kola](https://github.com/m1kola) | 1 |
+| [mattkanwisher](https://github.com/mattkanwisher) | 1 |
 | [merkeld](https://github.com/merkeld) | 1 |
-| [CodeLingoTeam](https://github.com/CodeLingoTeam) | 1 |
-| [Daanikus](https://github.com/Daanikus) | 1 |
-| [CodeLingoBot](https://github.com/CodeLingoBot) | 1 |
-| [blombard](https://github.com/blombard) | 1 |
-| [soxipy](https://github.com/soxipy) | 2 |
-| [lozdog245](https://github.com/lozdog245) | 2 |
+| [mKurrels](https://github.com/mKurrels) | 1 |
+| [starit](https://github.com/starit) | 1 |
+| [zeldrinn](https://github.com/zeldrinn) | 1 |

backtester/README.md

@@ -45,20 +45,23 @@ An event-driven backtesting tool to test and iterate trading strategies using hi
 - Fund transfer. At a strategy level, transfer funds between exchanges to allow for complex strategy design
 - Backtesting support for futures asset types
 - Example cash and carry spot futures strategy
-- Long-running application
-- GRPC server implementation
+- Long-running application as a GRPC server
+- Custom strategy plugins
+- Live data source trading. Traders can move their back tested strategies and use them against current live data
 
 ## Planned Features
 We welcome pull requests on any feature for the Backtester! We will be especially appreciative of any contribution towards the following planned features:
 
 | Feature | Description |
 |---------|-------------|
+| Perpetual futures support | Accounting for hourly funding rates in user's overall positions allows for much greater strategic depth |
+| Margin borrowing support | Allowing strategies to utilise margin borrowing to have larger positions and handling borrow rate payments |
 | Leverage support | Leverage is a good way to enhance profit and loss and is important to include in strategies |
+| Live ticker data | A potential feature as live trading works off candle data which is only processed at intervals. Adding ticker data as a strategic source allows for faster decision making |
+| Live orderbook data | Processing orders based off the latest orderbook data allows for much more accurate order placement and reduces surprise slippage |
 | Enhance config-builder | Create an application that can create strategy configs in a more visual manner and execute them via GRPC to allow for faster customisation of strategies |
 | Save Backtester results to database | This will allow for easier comparison of results over time |
 | Backtester result comparison report | Providing an executive summary of Backtester database results |
-| Currency correlation | Compare multiple exchange, asset, currencies for a candle interval against indicators to highlight correlated pairs for use in pairs trading |
-| Improve live trading functionality | Live trading is currently only a proof Of concept. Adding live support for running multiple currencies and running off orderbook data will allow for esteemed traders to use their backtested strategies |
 
 
 ## How does it work?