Feature: Candle conversion & Candle validation (thrasher-corp#716)

* Remove old concept. Introduce new job types and candle scaling

* Adds extra processing, commands

* new concept for queued jobs. Jobs can pause. New commands to manage status

* =End of day commit designing tables and implementing prerequisites further.

* Adds postgres data history relations

* Fixes table design for sqlite. Fixes all issues from merge

* Fixes craziness of database design. Adds some functions to get related jobs

* Fixes errors

* Updates some documentation, manages prerequisite jobs a little better, adds rpc funcs

* Fixes database design and adjust repo functions

* Tests database relationship

* Test coverage of new job functions

* Finishes coverage of new functions

* Commands and RPC coverage

* New database modifications for new job types

* Adds db support of new columns. Adds conversion validation. lint

* command blurb changes

* Allows websocket test to pass consistently

* Fixes merge issue preventing datahistorymanager from starting via config

* Minor fixes for different job type processing

* Fixes rangeholder issue, fixes validation, does not address jobs not starting or wrong status

* Fixes database tests, but at what cost. Fixes dhm tests

* Fixes dhj completion issue. Adds prerequisite by nickname

* Fixes validation processing. Adds db tests and validation

* Fixes validation job processing range

* Fixes trade sql. Reduces defaults. Validation processing and errors

* Updates cli job commands. adds validation decimal. fix job validation

* Expands run job handling and tests

* Validation work

* Fixes validation processing

* candle relations. new job type. updating database design

* Adds secondary exchange support. Sets stage for candle override

* Re adds accidentally deleted relationship

* Updates loading and saving candles to have relationship data when relevant

* Now validates and replaces candle data appropriately

* Fixes getting and setting datahistory data. Neatens DHM

* Test coverage

* Updates proto for new db types. New test coverage. Secondary exchange work

* Investigation into never-ending validation jobs. Now that intervals are ruled out, now need to complete the job....

* Fixes issues with validation job completion. Fixes validation volume issue for secondary exchange

* Adds candle warning support to the backtester

* Fixes warnings

* lint and begin docs

* Documentation updates. Final testing changes

* Minor fixes

* docs, prerequisite checks, more testing

* Fixes binance trade test. Rename err

* Documentation fixes. Figure fixes

* documentation update

* Fixes remote PSQL tests

* Fix binance mock test

* Remove unnecessary JSON

* regen proto

* Some minor nit fixes

* Var usage, query sorting, log improving, sql mirroring

* Extra coverage

* Experimental removal of m.jobs and mutex. Fix messaging

* Fixes error

* Lint fixes, command description improvements. More isRunning gates

* description improvements

* Lint

* BUFF regenerate

* Rough concept to fix insertions taking up long periods of time

* New calculation for trade data. Adds batch saving

This also adds an experimental request feature to shut down lingering requests. However, its uncertain whether or not this is having any impact. Initially thought it was the trades that was taking time and not SQL. Will investigate further

* Removes experimental requester. Adds documentation. Fixes typo

* rm unused error

* re-adds more forgotten contributors

* Now with proper commit count

CONTRIBUTORS

@@ -14,23 +14,25 @@ vadimzhukck | https://github.com/vadimzhukck
 140am | https://github.com/140am
 marcofranssen | https://github.com/marcofranssen
 dackroyd | https://github.com/dackroyd
+ydm | https://github.com/ydm
 cranktakular | https://github.com/cranktakular
 woshidama323 | https://github.com/woshidama323
 crackcomm | https://github.com/crackcomm
 azhang | https://github.com/azhang
 andreygrehov | https://github.com/andreygrehov
 bretep | https://github.com/bretep
 Christian-Achilli | https://github.com/Christian-Achilli
+lrascao | https://github.com/lrascao
 MarkDzulko | https://github.com/MarkDzulko
 yangrq1018 | https://github.com/yangrq1018
 gam-phon | https://github.com/gam-phon
 cornelk | https://github.com/cornelk
 if1live | https://github.com/if1live
 lozdog245 | https://github.com/lozdog245
 soxipy | https://github.com/soxipy
+tk42 | https://github.com/tk42
 mshogin | https://github.com/mshogin
 herenow | https://github.com/herenow
-tk42 | https://github.com/tk42
 daniel-cohen | https://github.com/daniel-cohen
 DirectX | https://github.com/DirectX
 frankzougc | https://github.com/frankzougc

README.md

@@ -143,8 +143,8 @@ Binaries will be published once the codebase reaches a stable condition.
 |User|Contribution Amount|
 |--|--|
 | [thrasher-](https://github.com/thrasher-) | 656 |
-| [shazbert](https://github.com/shazbert) | 209 |
-| [gloriousCode](https://github.com/gloriousCode) | 184 |
+| [shazbert](https://github.com/shazbert) | 212 |
+| [gloriousCode](https://github.com/gloriousCode) | 186 |
 | [dependabot-preview[bot]](https://github.com/apps/dependabot-preview) | 88 |
 | [xtda](https://github.com/xtda) | 47 |
 | [dependabot[bot]](https://github.com/apps/dependabot) | 18 |
@@ -156,23 +156,25 @@ Binaries will be published once the codebase reaches a stable condition.
 | [140am](https://github.com/140am) | 8 |
 | [marcofranssen](https://github.com/marcofranssen) | 8 |
 | [dackroyd](https://github.com/dackroyd) | 5 |
+| [ydm](https://github.com/ydm) | 5 |
 | [cranktakular](https://github.com/cranktakular) | 5 |
 | [woshidama323](https://github.com/woshidama323) | 3 |
 | [crackcomm](https://github.com/crackcomm) | 3 |
 | [azhang](https://github.com/azhang) | 2 |
 | [andreygrehov](https://github.com/andreygrehov) | 2 |
 | [bretep](https://github.com/bretep) | 2 |
 | [Christian-Achilli](https://github.com/Christian-Achilli) | 2 |
+| [lrascao](https://github.com/lrascao) | 2 |
 | [MarkDzulko](https://github.com/MarkDzulko) | 2 |
 | [yangrq1018](https://github.com/yangrq1018) | 2 |
 | [gam-phon](https://github.com/gam-phon) | 2 |
 | [cornelk](https://github.com/cornelk) | 2 |
 | [if1live](https://github.com/if1live) | 2 |
 | [lozdog245](https://github.com/lozdog245) | 2 |
 | [soxipy](https://github.com/soxipy) | 2 |
+| [tk42](https://github.com/tk42) | 2 |
 | [mshogin](https://github.com/mshogin) | 2 |
 | [herenow](https://github.com/herenow) | 2 |
-| [tk42](https://github.com/tk42) | 2 |
 | [daniel-cohen](https://github.com/daniel-cohen) | 1 |
 | [DirectX](https://github.com/DirectX) | 1 |
 | [frankzougc](https://github.com/frankzougc) | 1 |

backtester/data/kline/database/database.go

@@ -11,6 +11,7 @@ import (
 	"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
 	gctkline "github.com/thrasher-corp/gocryptotrader/exchanges/kline"
 	"github.com/thrasher-corp/gocryptotrader/exchanges/trade"
+	"github.com/thrasher-corp/gocryptotrader/log"
 )
 
 // LoadData retrieves data from an existing database using GoCryptoTrader's database handling implementation
@@ -29,6 +30,11 @@ func LoadData(startDate, endDate time.Time, interval time.Duration, exchangeName
 			return nil, fmt.Errorf("could not retrieve database candle data for %v %v %v, %v", exchangeName, a, fPair, err)
 		}
 		resp.Item = klineItem
+		for i := range klineItem.Candles {
+			if klineItem.Candles[i].ValidationIssues != "" {
+				log.Warnf(log.BackTester, "candle validation issue for %v %v %v: %v", klineItem.Exchange, klineItem.Asset, klineItem.Pair, klineItem.Candles[i].ValidationIssues)
+			}
+		}
 	case common.DataTrade:
 		trades, err := trade.GetTradesInRange(
 			exchangeName,

backtester/data/kline/database/database_test.go

@@ -110,12 +110,13 @@ func TestLoadDataCandles(t *testing.T) {
 		Interval: gctkline.FifteenMin,
 		Candles: []gctkline.Candle{
 			{
-				Time:   dInsert,
-				Open:   1337,
-				High:   1337,
-				Low:    1337,
-				Close:  1337,
-				Volume: 1337,
+				Time:             dInsert,
+				Open:             1337,
+				High:             1337,
+				Low:              1337,
+				Close:            1337,
+				Volume:           1337,
+				ValidationIssues: "hello world",
 			},
 		},
 	}

backtester/data/kline/kline.go

@@ -37,11 +37,12 @@ func (d *DataFromKline) Load() error {
 				CurrencyPair: d.Item.Pair,
 				AssetType:    d.Item.Asset,
 			},
-			Open:   d.Item.Candles[i].Open,
-			High:   d.Item.Candles[i].High,
-			Low:    d.Item.Candles[i].Low,
-			Close:  d.Item.Candles[i].Close,
-			Volume: d.Item.Candles[i].Volume,
+			Open:             d.Item.Candles[i].Open,
+			High:             d.Item.Candles[i].High,
+			Low:              d.Item.Candles[i].Low,
+			Close:            d.Item.Candles[i].Close,
+			Volume:           d.Item.Candles[i].Volume,
+			ValidationIssues: d.Item.Candles[i].ValidationIssues,
 		}
 		d.addedTimes[d.Item.Candles[i].Time] = true
 	}
@@ -75,11 +76,12 @@ func (d *DataFromKline) Append(ki *gctkline.Item) {
 				CurrencyPair: ki.Pair,
 				AssetType:    ki.Asset,
 			},
-			Open:   gctCandles[i].Open,
-			High:   gctCandles[i].High,
-			Low:    gctCandles[i].Low,
-			Close:  gctCandles[i].Close,
-			Volume: gctCandles[i].Volume,
+			Open:             gctCandles[i].Open,
+			High:             gctCandles[i].High,
+			Low:              gctCandles[i].Low,
+			Close:            gctCandles[i].Close,
+			Volume:           gctCandles[i].Volume,
+			ValidationIssues: gctCandles[i].ValidationIssues,
 		})
 		candleTimes = append(candleTimes, gctCandles[i].Time)
 	}

backtester/eventhandlers/portfolio/portfolio.go

@@ -50,8 +50,8 @@ func (p *Portfolio) Reset() {
 // on buy/sell, the portfolio manager will size the order and assess the risk of the order
 // if successful, it will pass on an order.Order to be used by the exchange event handler to place an order based on
 // the portfolio manager's recommendations
-func (p *Portfolio) OnSignal(signal signal.Event, cs *exchange.Settings) (*order.Order, error) {
-	if signal == nil || cs == nil {
+func (p *Portfolio) OnSignal(s signal.Event, cs *exchange.Settings) (*order.Order, error) {
+	if s == nil || cs == nil {
 		return nil, common.ErrNilArguments
 	}
 	if p.sizeManager == nil {
@@ -63,69 +63,69 @@ func (p *Portfolio) OnSignal(signal signal.Event, cs *exchange.Settings) (*order
 
 	o := &order.Order{
 		Base: event.Base{
-			Offset:       signal.GetOffset(),
-			Exchange:     signal.GetExchange(),
-			Time:         signal.GetTime(),
-			CurrencyPair: signal.Pair(),
-			AssetType:    signal.GetAssetType(),
-			Interval:     signal.GetInterval(),
-			Reason:       signal.GetReason(),
+			Offset:       s.GetOffset(),
+			Exchange:     s.GetExchange(),
+			Time:         s.GetTime(),
+			CurrencyPair: s.Pair(),
+			AssetType:    s.GetAssetType(),
+			Interval:     s.GetInterval(),
+			Reason:       s.GetReason(),
 		},
-		Direction: signal.GetDirection(),
+		Direction: s.GetDirection(),
 	}
-	if signal.GetDirection() == "" {
+	if s.GetDirection() == "" {
 		return o, errInvalidDirection
 	}
 
-	lookup := p.exchangeAssetPairSettings[signal.GetExchange()][signal.GetAssetType()][signal.Pair()]
+	lookup := p.exchangeAssetPairSettings[s.GetExchange()][s.GetAssetType()][s.Pair()]
 	if lookup == nil {
 		return nil, fmt.Errorf("%w for %v %v %v",
 			errNoPortfolioSettings,
-			signal.GetExchange(),
-			signal.GetAssetType(),
-			signal.Pair())
+			s.GetExchange(),
+			s.GetAssetType(),
+			s.Pair())
 	}
 	prevHolding := lookup.GetLatestHoldings()
 	if p.iteration == 0 {
 		prevHolding.InitialFunds = lookup.InitialFunds
 		prevHolding.RemainingFunds = lookup.InitialFunds
-		prevHolding.Exchange = signal.GetExchange()
-		prevHolding.Pair = signal.Pair()
-		prevHolding.Asset = signal.GetAssetType()
-		prevHolding.Timestamp = signal.GetTime()
+		prevHolding.Exchange = s.GetExchange()
+		prevHolding.Pair = s.Pair()
+		prevHolding.Asset = s.GetAssetType()
+		prevHolding.Timestamp = s.GetTime()
 	}
 	p.iteration++
 
-	if signal.GetDirection() == common.DoNothing || signal.GetDirection() == common.MissingData || signal.GetDirection() == "" {
+	if s.GetDirection() == common.DoNothing || s.GetDirection() == common.MissingData || s.GetDirection() == "" {
 		return o, nil
 	}
 
-	if signal.GetDirection() == gctorder.Sell && prevHolding.PositionsSize == 0 {
+	if s.GetDirection() == gctorder.Sell && prevHolding.PositionsSize == 0 {
 		o.AppendReason("no holdings to sell")
 		o.SetDirection(common.CouldNotSell)
-		signal.SetDirection(o.Direction)
+		s.SetDirection(o.Direction)
 		return o, nil
 	}
 
 	// for simplicity, the backtester will round to 8 decimal places
 	remainingFundsRounded := math.Floor(prevHolding.RemainingFunds*100000000) / 100000000
-	if signal.GetDirection() == gctorder.Buy && remainingFundsRounded <= 0 {
+	if s.GetDirection() == gctorder.Buy && remainingFundsRounded <= 0 {
 		o.AppendReason("not enough funds to buy")
 		o.SetDirection(common.CouldNotBuy)
-		signal.SetDirection(o.Direction)
+		s.SetDirection(o.Direction)
 		return o, nil
 	}
 
-	o.Price = signal.GetPrice()
+	o.Price = s.GetPrice()
 	o.OrderType = gctorder.Market
-	o.BuyLimit = signal.GetBuyLimit()
-	o.SellLimit = signal.GetSellLimit()
+	o.BuyLimit = s.GetBuyLimit()
+	o.SellLimit = s.GetSellLimit()
 	sizingFunds := prevHolding.RemainingFunds
-	if signal.GetDirection() == gctorder.Sell {
+	if s.GetDirection() == gctorder.Sell {
 		sizingFunds = prevHolding.PositionsSize
 	}
 
-	sizedOrder := p.sizeOrder(signal, cs, o, sizingFunds)
+	sizedOrder := p.sizeOrder(s, cs, o, sizingFunds)
 	o.Funds = sizingFunds
 	sizedAmountRounded := math.Floor(sizedOrder.Amount*100000000) / 100000000
 	if sizedAmountRounded <= 0 {
@@ -138,7 +138,7 @@ func (p *Portfolio) OnSignal(signal signal.Event, cs *exchange.Settings) (*order
 		return o, nil
 	}
 
-	return p.evaluateOrder(signal, o, sizedOrder)
+	return p.evaluateOrder(s, o, sizedOrder)
 }
 
 func (p *Portfolio) evaluateOrder(d common.Directioner, originalOrderSignal, sizedOrder *order.Order) (*order.Order, error) {

backtester/eventtypes/kline/kline_types.go

@@ -8,9 +8,10 @@ import (
 // a common.DataEventHandler type
 type Kline struct {
 	event.Base
-	Open   float64
-	Close  float64
-	Low    float64
-	High   float64
-	Volume float64
+	Open             float64
+	Close            float64
+	Low              float64
+	High             float64
+	Volume           float64
+	ValidationIssues string
 }

backtester/report/report.go

@@ -23,6 +23,19 @@ func (d *Data) GenerateReport() error {
 		return err
 	}
 
+	for i := range d.OriginalCandles {
+		for j := range d.OriginalCandles[i].Candles {
+			if d.OriginalCandles[i].Candles[j].ValidationIssues == "" {
+				continue
+			}
+			d.Warnings = append(d.Warnings, Warning{
+				Exchange: d.OriginalCandles[i].Exchange,
+				Asset:    d.OriginalCandles[i].Asset,
+				Pair:     d.OriginalCandles[i].Pair,
+				Message:  fmt.Sprintf("candle data %v", d.OriginalCandles[i].Candles[j].ValidationIssues),
+			})
+		}
+	}
 	for i := range d.EnhancedCandles {
 		if len(d.EnhancedCandles[i].Candles) >= maxChartLimit {
 			d.EnhancedCandles[i].IsOverLimit = true

backtester/report/report_test.go

@@ -37,7 +37,15 @@ func TestGenerateReport(t *testing.T) {
 		OriginalCandles: []*gctkline.Item{
 			{
 				Candles: []gctkline.Candle{
-					{},
+					{
+						Time:             time.Now(),
+						Open:             1337,
+						High:             1337,
+						Low:              1337,
+						Close:            1337,
+						Volume:           1337,
+						ValidationIssues: "hello world!",
+					},
 				},
 			},
 		},

backtester/report/report_types.go

@@ -33,6 +33,15 @@ type Data struct {
 	Config          *config.Config
 	TemplatePath    string
 	OutputPath      string
+	Warnings        []Warning
+}
+
+// Warning holds any candle warnings
+type Warning struct {
+	Exchange string
+	Asset    asset.Item
+	Pair     currency.Pair
+	Message  string
 }
 
 // DetailedKline enhances kline details for the purpose of rich reporting results

backtester/report/tpl.gohtml

@@ -278,6 +278,15 @@
 							</tr>
 						{{end}}
                     {{end}}
+					{{ range .Warnings}}
+					<tr>
+						<td>{{.Exchange}}</td>
+						<td>{{.Asset}}</td>
+						<td>{{.Pair.Base}}</td>
+						<td>{{.Pair.Quote}}</td>
+						<td>{{.Message}}</td>
+					</tr>
+					{{end}}
 					</tbody>
 				</table>
 			</div>
@@ -742,7 +751,7 @@
 												<td>${{ printf "%f" $ev.SignalEvent.Price}} {{$pair.Quote}}</td>
 												<td>{{$ev.SignalEvent.GetDirection}}</td>
 												<td>{{$ev.SignalEvent.GetReason}}</td>
-                                            {{ end }}
+											{{ end }}
 											<td>${{printf "%.8f" $ev.Holdings.CommittedFunds}} {{$pair.Quote}}</td>
 										</tr>
                                     {{end}}

cmd/documentation/documentation.go

@@ -175,6 +175,16 @@ func main() {
 				URL:           "https://github.com/tk42",
 				Contributions: 2,
 			},
+			{
+				Login:         "mshogin",
+				URL:           "https://github.com/mshogin",
+				Contributions: 2,
+			},
+			{
+				Login:         "herenow",
+				URL:           "https://github.com/herenow",
+				Contributions: 2,
+			},
 			{
 				Login:         "daniel-cohen",
 				URL:           "https://github.com/daniel-cohen",