Improvement: Subsystem separation (thrasher-corp#664)

* Initial codes for a trade tracker

* Moving everything in a broken fashion

* Removes tradetracker. Removes some errors for subsystems

* Cleans up some subsystems, renames stuttering types. Removes some global Bot usage

* More basic subsystem renaming and file moving

* Removes engine dependency from events,ntpserver,ordermanager,comms manager

* Exports eventManager, fixes rpcserver. puts rpcserver back for now

* Removes redundant error message, further removes engine dependencies

* experimental end of day interface usage

* adds ability to build the application

* Withdraw and event manager handling

* cleans up apiserver and communications manager

* Cleans up some start/setup processes. Though should separate

* More consistency with Setup Start Stop IsRunning funcs

* Final consistency pass before testing phase

* Fixes engine tests. Fixes stop nil issue

* api server tests

* Communications manager testing

* Connection manager tests and nilsubsystem error

* End of day currencypairsyncer tests

* Adds databaseconnection/databaseconnection_test.go

* Adds withdrawal manager tests

* Deposit address testing. Moved orderbook sync first as its more important

* Adds test for event manager

* More full eventmanager testing

* Adds testfile. Enables skipped test.

* ntp manager tests

* Adds ordermanager tests, Extracts a whole new subsystem from engine and fanangles import cycles

* Adds websocket routine manager tests

* Basic portfolio manager testing

* Fixes issue with currency pair sync startup

* Fixes issue with event manager startup

* Starts the order manager before backtester starts

* Fixes fee tests. Expands testing. Doesnt fix races

* Fixes most test races

* Resolves data races

* Fixes subsystem test issues

* currency pair syncer coverage tests

* Refactors portfolio. Fixes tests. Withdraw validation

Portfolio didn't need to exist with a portfolio manager. Now the porfolio manager
is in charge how the portfolio is handled and all portfolio functions are attached
to the base instead of just exported at the package level

Withdrawal validation occurred at the exchange level when it can just be run at the
withdrawal manager level. All withdrawal requests go through that endpoint

* lint -fix

* golang lint fixes

* lints and comments everything

* Updates GCT logo, adds documentation for some subsystems

* More documentation and more logo updates

* Fixes backtesting and apiserver errors encountered

* Fixes errors and typos from reviewing

* More minor fixes

* Changes %h verb to %w

* reverbs to %s

* Humbly begins reverting to more flat engine package

The main reasoning for this is that the subsystem split doesn't make sense
in a golang environment. The subsystems are only meant to be used with engine
and so by placing them in a non-engine area, it does not work and is
inconsistent with the rest of the application's package layout.

This will begin salvaging the changes made by reverting to a flat
engine package, but maintaining the consistent designs introduced.
Further, I will look to remove any TestMains and decrease the scope
of testing to be more local and decrease the issues that have been
caused from our style of testing.

* Manages to re-flatten things. Everything is within its own file

* mini fixes

* Fixes tests and data races and lints

* Updates docs tool for engine to create filename readmes

* os -> ioutil

* remove err

* Appveyor version increase test

* Removes tCleanup as its unsupported on appveyor

* Adds stuff that I thought was in previous merge master commit

* Removes cancel from test

* Fixes really fun test-exclusive data race

* minor nit fixes

* niterinos

* docs gen

* rm;rf test

* Remove typoline. expands startstop helper. Splits apiserver

* Removes accidental folder

* Uses update instead of replace for order upsert

* addresses nits. Renames files. Regenerates documentation.

* lint and removal of comments

* Add new test for default scenario

* Fixes typo

* regen docs

.appveyor.yml

@@ -26,7 +26,7 @@ environment:
   PSQL_SSLMODE: disable
   PSQL_SKIPSQLCMD: true
   PSQL_TESTDBNAME: gct_dev_ci
-stack: go 1.15.x
+stack: go 1.16.x
 
 services:
   - postgresql96

CONTRIBUTORS

@@ -10,6 +10,7 @@ vazha | https://github.com/vazha
 ermalguni | https://github.com/ermalguni
 MadCozBadd | https://github.com/MadCozBadd
 vadimzhukck | https://github.com/vadimzhukck
+dependabot[bot] | https://github.com/apps/dependabot
 140am | https://github.com/140am
 marcofranssen | https://github.com/marcofranssen
 dackroyd | https://github.com/dackroyd
@@ -22,7 +23,6 @@ bretep | https://github.com/bretep
 Christian-Achilli | https://github.com/Christian-Achilli
 gam-phon | https://github.com/gam-phon
 cornelk | https://github.com/cornelk
-dependabot[bot] | https://github.com/apps/dependabot
 if1live | https://github.com/if1live
 lozdog245 | https://github.com/lozdog245
 soxipy | https://github.com/soxipy

README.md

@@ -1,4 +1,4 @@
-<img src="https://github.com/thrasher-corp/gocryptotrader/blob/master/web/src/assets/page-logo.png?raw=true" width="350px" height="350px" hspace="70">
+<img src="/common/gctlogo.png?raw=true" width="350px" height="350px" hspace="70">
 
 [![Build Status](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/thrasher-corp/gocryptotrader/actions/workflows/tests.yml)
 [![Software License](https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square)](https://github.com/thrasher-corp/gocryptotrader/blob/master/LICENSE)
@@ -143,16 +143,17 @@ Binaries will be published once the codebase reaches a stable condition.
 
 |User|Contribution Amount|
 |--|--|
-| [thrasher-](https://github.com/thrasher-) | 650 |
-| [shazbert](https://github.com/shazbert) | 202 |
-| [gloriousCode](https://github.com/gloriousCode) | 176 |
-| [dependabot-preview[bot]](https://github.com/apps/dependabot-preview) | 87 |
+| [thrasher-](https://github.com/thrasher-) | 654 |
+| [shazbert](https://github.com/shazbert) | 207 |
+| [gloriousCode](https://github.com/gloriousCode) | 179 |
+| [dependabot-preview[bot]](https://github.com/apps/dependabot-preview) | 88 |
 | [xtda](https://github.com/xtda) | 47 |
 | [Rots](https://github.com/Rots) | 15 |
 | [vazha](https://github.com/vazha) | 15 |
 | [ermalguni](https://github.com/ermalguni) | 14 |
-| [MadCozBadd](https://github.com/MadCozBadd) | 10 |
+| [MadCozBadd](https://github.com/MadCozBadd) | 12 |
 | [vadimzhukck](https://github.com/vadimzhukck) | 10 |
+| [dependabot[bot]](https://github.com/apps/dependabot) | 10 |
 | [140am](https://github.com/140am) | 8 |
 | [marcofranssen](https://github.com/marcofranssen) | 8 |
 | [dackroyd](https://github.com/dackroyd) | 5 |
@@ -165,7 +166,6 @@ Binaries will be published once the codebase reaches a stable condition.
 | [Christian-Achilli](https://github.com/Christian-Achilli) | 2 |
 | [gam-phon](https://github.com/gam-phon) | 2 |
 | [cornelk](https://github.com/cornelk) | 2 |
-| [dependabot[bot]](https://github.com/apps/dependabot) | 2 |
 | [if1live](https://github.com/if1live) | 2 |
 | [lozdog245](https://github.com/lozdog245) | 2 |
 | [soxipy](https://github.com/soxipy) | 2 |

backtester/backtest/backtest.go

@@ -40,6 +40,7 @@ import (
 	gctexchange "github.com/thrasher-corp/gocryptotrader/exchanges"
 	"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
 	gctkline "github.com/thrasher-corp/gocryptotrader/exchanges/kline"
+	gctorder "github.com/thrasher-corp/gocryptotrader/exchanges/order"
 	"github.com/thrasher-corp/gocryptotrader/log"
 )
 
@@ -281,7 +282,7 @@ func (bt *BackTest) setupExchangeSettings(cfg *config.Config) (exchange.Exchange
 		sellRule.Validate()
 
 		limits, err := exch.GetOrderExecutionLimits(a, pair)
-		if err != nil {
+		if err != nil && !errors.Is(err, gctorder.ErrExchangeLimitNotLoaded) {
 			return resp, err
 		}
 
@@ -361,15 +362,26 @@ func (bt *BackTest) setupBot(cfg *config.Config, bot *engine.Engine) error {
 	if err != nil {
 		return err
 	}
-
+	bt.Bot.ExchangeManager = engine.SetupExchangeManager()
 	for i := range cfg.CurrencySettings {
 		err = bt.Bot.LoadExchange(cfg.CurrencySettings[i].ExchangeName, false, nil)
 		if err != nil && !errors.Is(err, engine.ErrExchangeAlreadyLoaded) {
 			return err
 		}
 	}
-	if !bt.Bot.OrderManager.Started() {
-		return bt.Bot.OrderManager.Start(bt.Bot)
+	if !bt.Bot.OrderManager.IsRunning() {
+		bt.Bot.OrderManager, err = engine.SetupOrderManager(
+			bt.Bot.ExchangeManager,
+			bt.Bot.CommunicationsManager,
+			&bt.Bot.ServicesWG,
+			bot.Settings.Verbose)
+		if err != nil {
+			return err
+		}
+		err = bt.Bot.OrderManager.Start()
+		if err != nil {
+			return err
+		}
 	}
 
 	return nil
@@ -470,18 +482,26 @@ func (bt *BackTest) loadData(cfg *config.Config, exch gctexchange.IBotExchange,
 		if cfg.DataSettings.DatabaseData.ConfigOverride != nil {
 			bt.Bot.Config.Database = *cfg.DataSettings.DatabaseData.ConfigOverride
 			gctdatabase.DB.DataPath = filepath.Join(gctcommon.GetDefaultDataDir(runtime.GOOS), "database")
-			gctdatabase.DB.Config = cfg.DataSettings.DatabaseData.ConfigOverride
-			err = bt.Bot.DatabaseManager.Start(bt.Bot)
+			err = gctdatabase.DB.SetConfig(cfg.DataSettings.DatabaseData.ConfigOverride)
 			if err != nil {
 				return nil, err
 			}
-			defer func() {
-				err = bt.Bot.DatabaseManager.Stop()
-				if err != nil {
-					log.Error(log.BackTester, err)
-				}
-			}()
 		}
+		bt.Bot.DatabaseManager, err = engine.SetupDatabaseConnectionManager(gctdatabase.DB.GetConfig())
+		if err != nil {
+			return nil, err
+		}
+
+		err = bt.Bot.DatabaseManager.Start(&bt.Bot.ServicesWG)
+		if err != nil {
+			return nil, err
+		}
+		defer func() {
+			stopErr := bt.Bot.DatabaseManager.Stop()
+			if stopErr != nil {
+				log.Error(log.BackTester, stopErr)
+			}
+		}()
 		resp, err = loadDatabaseData(cfg, exch.GetName(), fPair, a, dataType)
 		if err != nil {
 			return nil, fmt.Errorf("unable to retrieve data from GoCryptoTrader database. Error: %v. Please ensure the database is setup correctly and has data before use", err)
@@ -768,7 +788,7 @@ func (bt *BackTest) processDataEvent(e common.DataEventHandler) error {
 // updateStatsForDataEvent makes various systems aware of price movements from
 // data events
 func (bt *BackTest) updateStatsForDataEvent(e common.DataEventHandler) {
-	// update portfolio with latest price
+	// update portfoliomanager with latest price
 	err := bt.Portfolio.Update(e)
 	if err != nil {
 		log.Error(log.BackTester, err)

backtester/backtest/backtest_test.go

@@ -30,7 +30,7 @@ import (
 	gctkline "github.com/thrasher-corp/gocryptotrader/exchanges/kline"
 )
 
-const testExchange = "binance"
+const testExchange = "Bitstamp"
 
 func newBotWithExchange() (*engine.Engine, gctexchange.IBotExchange) {
 	bot, err := engine.NewFromSettings(&engine.Settings{
@@ -40,6 +40,7 @@ func newBotWithExchange() (*engine.Engine, gctexchange.IBotExchange) {
 	if err != nil {
 		log.Fatal(err)
 	}
+	bot.ExchangeManager = engine.SetupExchangeManager()
 	err = bot.LoadExchange(testExchange, false, nil)
 	if err != nil {
 		log.Fatal(err)
@@ -103,7 +104,7 @@ func TestNewFromConfig(t *testing.T) {
 	}
 
 	cfg.CurrencySettings[0].Base = "BTC"
-	cfg.CurrencySettings[0].Quote = "USDT"
+	cfg.CurrencySettings[0].Quote = "USD"
 
 	cfg.DataSettings.APIData = &config.APIData{
 		StartDate: time.Time{},
@@ -162,7 +163,7 @@ func TestLoadData(t *testing.T) {
 			Quote:        "test",
 		},
 	}
-	cfg.CurrencySettings[0].ExchangeName = testExchange
+	cfg.CurrencySettings[0].ExchangeName = "binance"
 	cfg.CurrencySettings[0].Asset = asset.Spot.String()
 	cfg.CurrencySettings[0].Base = "BTC"
 	cfg.CurrencySettings[0].Quote = "USDT"
@@ -193,7 +194,7 @@ func TestLoadData(t *testing.T) {
 		Reports: &report.Data{},
 	}
 
-	cp := currency.NewPair(currency.BTC, currency.USDT)
+	cp := currency.NewPair(currency.BTC, currency.USD)
 	_, err = bt.loadData(cfg, exch, cp, asset.Spot)
 	if err != nil {
 		t.Error(err)
@@ -241,7 +242,7 @@ func TestLoadData(t *testing.T) {
 
 func TestLoadDatabaseData(t *testing.T) {
 	t.Parallel()
-	cp := currency.NewPair(currency.BTC, currency.USDT)
+	cp := currency.NewPair(currency.BTC, currency.USD)
 	_, err := loadDatabaseData(nil, "", cp, "", -1)
 	if err != nil && !strings.Contains(err.Error(), "nil config data received") {
 		t.Error(err)

backtester/common/common_types.go

@@ -41,6 +41,8 @@ var (
 	ErrNilArguments = errors.New("received nil argument(s)")
 	// ErrNilEvent is a common error for whenever a nil event occurs when it shouldn't have
 	ErrNilEvent = errors.New("nil event received")
+	// ErrInvalidDataType occurs when an invalid data type is defined in the config
+	ErrInvalidDataType = errors.New("invalid datatype received")
 )
 
 // EventHandler interface implements required GetTime() & Pair() return

backtester/config/configbuilder/main.go

@@ -395,7 +395,10 @@ func parseDatabase(reader *bufio.Reader, cfg *config.Config) error {
 			}
 		}
 		cfg.DataSettings.DatabaseData.ConfigOverride.Port = uint16(port)
-		database.DB.Config = cfg.DataSettings.DatabaseData.ConfigOverride
+		err = database.DB.SetConfig(cfg.DataSettings.DatabaseData.ConfigOverride)
+		if err != nil {
+			return fmt.Errorf("database failed to set config: %w", err)
+		}
 		if cfg.DataSettings.DatabaseData.ConfigOverride.Driver == database.DBPostgreSQL {
 			_, err = dbPSQL.Connect()
 			if err != nil {

backtester/data/kline/api/api.go

@@ -44,7 +44,7 @@ func LoadData(dataType int64, startDate, endDate time.Time, interval time.Durati
 			return nil, fmt.Errorf("could not convert trade data to candles for %v %v %v, %v", exch.GetName(), a, fPair, err)
 		}
 	default:
-		return nil, fmt.Errorf("could not retrieve data for %v %v %v, invalid data type received", exch.GetName(), a, fPair)
+		return nil, fmt.Errorf("could not retrieve data for %v %v %v, %w", exch.GetName(), a, fPair, common.ErrInvalidDataType)
 	}
 	candles.Exchange = strings.ToLower(candles.Exchange)
 

backtester/data/kline/api/api_test.go

@@ -1,10 +1,10 @@
 package api
 
 import (
+	"errors"
 	"log"
 	"os"
 	"path/filepath"
-	"strings"
 	"testing"
 	"time"
 
@@ -33,6 +33,7 @@ func TestMain(m *testing.M) {
 		log.Fatal(err)
 	}
 
+	bot.ExchangeManager = engine.SetupExchangeManager()
 	err = bot.LoadExchange(testExchange, false, nil)
 	if err != nil {
 		log.Fatal(err)
@@ -62,15 +63,15 @@ func TestLoadCandles(t *testing.T) {
 	}
 
 	_, err = LoadData(-1, tt1, tt2, interval.Duration(), exch, p, a)
-	if err != nil && !strings.Contains(err.Error(), "could not retrieve data for Binance spot BTCUSDT, invalid data type received") {
-		t.Error(err)
+	if !errors.Is(err, common.ErrInvalidDataType) {
+		t.Errorf("expected '%v' received '%v'", err, common.ErrInvalidDataType)
 	}
 }
 
 func TestLoadTrades(t *testing.T) {
 	t.Parallel()
 	interval := gctkline.FifteenMin
-	tt1 := time.Now().Add(-time.Minute * 60).Round(interval.Duration())
+	tt1 := time.Now().Add(-time.Minute * 15).Round(interval.Duration())
 	tt2 := time.Now().Round(interval.Duration())
 	a := asset.Spot
 	p := currency.NewPair(currency.BTC, currency.USDT)

backtester/data/kline/csv/csv.go

@@ -149,7 +149,7 @@ func LoadData(dataType int64, filepath, exchangeName string, interval time.Durat
 			return nil, fmt.Errorf("could not read csv trade data for %v %v %v, %v", exchangeName, a, fPair, err)
 		}
 	default:
-		return nil, fmt.Errorf("could not process csv data for %v %v %v, invalid data type received", exchangeName, a, fPair)
+		return nil, fmt.Errorf("could not process csv data for %v %v %v, %w", exchangeName, a, fPair, common.ErrInvalidDataType)
 	}
 	resp.Item.Exchange = strings.ToLower(exchangeName)
 	resp.Item.Pair = fPair

backtester/data/kline/csv/csv_test.go

@@ -1,8 +1,8 @@
 package csv
 
 import (
+	"errors"
 	"path/filepath"
-	"strings"
 	"testing"
 
 	"github.com/thrasher-corp/gocryptotrader/backtester/common"
@@ -56,7 +56,7 @@ func TestLoadDataInvalid(t *testing.T) {
 		gctkline.FifteenMin.Duration(),
 		p,
 		a)
-	if err != nil && !strings.Contains(err.Error(), "could not process csv data for binance spot BTCUSDT, invalid data type received") {
-		t.Error(err)
+	if !errors.Is(err, common.ErrInvalidDataType) {
+		t.Errorf("expected '%v' received '%v'", err, common.ErrInvalidDataType)
 	}
 }

backtester/data/kline/database/database.go

@@ -48,7 +48,7 @@ func LoadData(startDate, endDate time.Time, interval time.Duration, exchangeName
 		}
 		resp.Item = klineItem
 	default:
-		return nil, fmt.Errorf("could not retrieve database data for %v %v %v, invalid data type received", exchangeName, a, fPair)
+		return nil, fmt.Errorf("could not retrieve database data for %v %v %v, %w", exchangeName, a, fPair, common.ErrInvalidDataType)
 	}
 	resp.Item.Exchange = strings.ToLower(resp.Item.Exchange)
 

backtester/data/kline/database/database_test.go

@@ -1,11 +1,11 @@
 package database
 
 import (
+	"errors"
 	"fmt"
 	"io/ioutil"
 	"os"
 	"path/filepath"
-	"strings"
 	"testing"
 	"time"
 
@@ -86,7 +86,11 @@ func TestLoadDataCandles(t *testing.T) {
 		t.Error(err)
 	}
 
-	err = bot.DatabaseManager.Start(bot)
+	bot.DatabaseManager, err = engine.SetupDatabaseConnectionManager(&bot.Config.Database)
+	if err != nil {
+		t.Error(err)
+	}
+	err = bot.DatabaseManager.Start(&bot.ServicesWG)
 	if err != nil {
 		t.Error(err)
 	}
@@ -161,7 +165,11 @@ func TestLoadDataTrades(t *testing.T) {
 		t.Error(err)
 	}
 
-	err = bot.DatabaseManager.Start(bot)
+	bot.DatabaseManager, err = engine.SetupDatabaseConnectionManager(&bot.Config.Database)
+	if err != nil {
+		t.Error(err)
+	}
+	err = bot.DatabaseManager.Start(&bot.ServicesWG)
 	if err != nil {
 		t.Error(err)
 	}
@@ -202,7 +210,7 @@ func TestLoadDataInvalid(t *testing.T) {
 	dStart := time.Date(2020, 1, 0, 0, 0, 0, 0, time.UTC)
 	dEnd := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
 	_, err := LoadData(dStart, dEnd, gctkline.FifteenMin.Duration(), exch, -1, p, a)
-	if err != nil && !strings.Contains(err.Error(), "could not retrieve database data for binance spot BTCUSDT, invalid data type received") {
-		t.Error(err)
+	if !errors.Is(err, common.ErrInvalidDataType) {
+		t.Errorf("expected '%v' received '%v'", err, common.ErrInvalidDataType)
 	}
 }

backtester/data/kline/live/live.go

@@ -58,7 +58,7 @@ func LoadData(exch exchange.IBotExchange, dataType int64, interval time.Duration
 			}
 		}
 	default:
-		return nil, fmt.Errorf("could not retrieve live data for %v %v %v, invalid data type received", exch.GetName(), a, fPair)
+		return nil, fmt.Errorf("could not retrieve live data for %v %v %v, %w", exch.GetName(), a, fPair, common.ErrInvalidDataType)
 	}
 	candles.Exchange = strings.ToLower(exch.GetName())
 	return &candles, nil