Skip to content

Commit

Permalink
Websocket request-response correlation (thrasher-corp#328)
Browse files Browse the repository at this point in the history 
* Establishes new websocket functionality. Begins the creation of the websocket request

* Adding a wrapper over gorilla websocket connect,send,receive to handle ID messages. Doesn't work

* Successfully moved exchange_websocket into its own wshandler namespace. oof

* Sets up ZB to use a round trip WS request system

* Adds Kraken ID support to subscriptions. Renames duplicate func name UnsubscribeToChannels to RemoveSubscribedChannels. Adds some helper methods in the WebsocketConn to reduce duplicate code. Cleans up ZB implementation

* Fixes double locking which caused no websocket data to be read. Fixes requestid for kraken subscriptions

* Completes Huobi and Hadax implementation. Extends ZB error handling. Adds GZip support for reading messages

* Adds HitBTC support. Adds GetCurrencies, GetSymbols, GetTrades WS funcs. Adds super fun new parameter to GenerateMessageID for Unix and UnixNano

* Adds GateIO id support

* Adds Coinut support. Prevents nil reference error in constatus when there isnt one

* Standardises all Exchange websockets to use the wshandler websocket. Removes the wsRequestMtx as wshandler handles that now. Makes the Dialer a dialer, its not externally referenced that I can see.

* Fixes issue with coinut implementation. Updates bitmex currencies. Removes redundant log messages which are used to log messages

* Starts testing. Renames files

* Adds tests for websocket connection

* Reverts request.go change

* Linting everything

* Fixes rebase issue

* Final changes. Fixes variable names, removes log.Debug, removes lines, rearranges test types, removes order correlation websocket type

* Final final commit, fixing ZB issues.

* Adds traffic alerts where missed. Changes empty struct pointer addresses to nil instead. Removes empty lines

* Fixed string conversion

* Fixes issue with ZB not sending success codes

* Fixes issue with coinut processing due to nonce handling with subscriptions

* Fixes issue where ZB test failure was not caught. Removes unnecessary error handling from other ZB tests

* Removes unused interface

* Renames wshandler.Init() to wshandler.Run()

* Updates template file

* Capitalises cryptocurrencies in struct. Moves websocketResponseCheckTimeout and websocketResponseMaxLimit into config options. Moves connection configuration to main exchange Setup (where appropriate). Reverts currencylastupdated ticks. Improves reader close error checking

* Fixes two inconsistent websocket delay times

* Creates a default variable for websocket ResponseMaxLimit and ResponseCheckTimeout, then applies it to setdefaults and all tests

* Updates exchange template to set and use default websocket response limits
  • Loading branch information
@gloriousCode @thrasher-
gloriousCode authored and thrasher- committed Aug 7, 2019
1 parent 6e70f06 commit e209d85
Show file tree
Hide file tree
Showing 113 changed files with 3,221 additions and 2,546 deletions.
4 changes: 2 additions & 2 deletions communications/slack/slack.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,15 +175,15 @@ func (s *Slack) NewConnection() error {
// WebsocketConnect creates a websocket dialer amd initiates a websocket
// connection
func (s *Slack) WebsocketConnect() error {
var Dialer websocket.Dialer
var dialer websocket.Dialer
var err error

websocketURL := s.Details.URL
if s.ReconnectURL != "" {
websocketURL = s.ReconnectURL
}

s.WebsocketConn, _, err = Dialer.Dial(websocketURL, http.Header{})
s.WebsocketConn, _, err = dialer.Dial(websocketURL, http.Header{})
if err != nil {
return err
}
Expand Down
4 changes: 3 additions & 1 deletion config/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@ have multiple deposit accounts for different FIAT deposit currencies.
"Websocket": false,
"UseSandbox": false,
"RESTPollingDelay": 10,
"HTTPTimeout": 15000000000,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"AuthenticatedAPISupport": false,
"APIKey": "Key",
"APISecret": "Secret",
Expand Down
38 changes: 26 additions & 12 deletions config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,20 @@ import (

// Constants declared here are filename strings and test strings
const (
FXProviderFixer = "fixer"
EncryptedConfigFile = "config.dat"
ConfigFile = "config.json"
ConfigTestFile = "../testdata/configtest.json"
configFileEncryptionPrompt = 0
configFileEncryptionEnabled = 1
configFileEncryptionDisabled = -1
configPairsLastUpdatedWarningThreshold = 30 // 30 days
configDefaultHTTPTimeout = time.Second * 15
configMaxAuthFailres = 3
defaultNTPAllowedDifference = 50000000
defaultNTPAllowedNegativeDifference = 50000000
FXProviderFixer = "fixer"
EncryptedConfigFile = "config.dat"
ConfigFile = "config.json"
ConfigTestFile = "../testdata/configtest.json"
configFileEncryptionPrompt = 0
configFileEncryptionEnabled = 1
configFileEncryptionDisabled = -1
configPairsLastUpdatedWarningThreshold = 30 // 30 days
configDefaultHTTPTimeout = time.Second * 15
configDefaultWebsocketResponseCheckTimeout = time.Millisecond * 30
configDefaultWebsocketResponseMaxLimit = time.Second * 7
configMaxAuthFailres = 3
defaultNTPAllowedDifference = 50000000
defaultNTPAllowedNegativeDifference = 50000000
)

// Constants here hold some messages
Expand Down Expand Up @@ -157,6 +159,8 @@ type ExchangeConfig struct {
UseSandbox bool `json:"useSandbox"`
RESTPollingDelay time.Duration `json:"restPollingDelay"`
HTTPTimeout time.Duration `json:"httpTimeout"`
WebsocketResponseCheckTimeout time.Duration `json:"websocketResponseCheckTimeout"`
WebsocketResponseMaxLimit time.Duration `json:"websocketResponseMaxLimit"`
HTTPUserAgent string `json:"httpUserAgent"`
HTTPDebugging bool `json:"httpDebugging"`
AuthenticatedAPISupport bool `json:"authenticatedApiSupport"`
Expand Down Expand Up @@ -824,6 +828,16 @@ func (c *Config) CheckExchangeConfigValues() error {
c.Exchanges[i].HTTPTimeout = configDefaultHTTPTimeout
}

if c.Exchanges[i].WebsocketResponseCheckTimeout <= 0 {
log.Warnf("Exchange %s Websocket response check timeout value not set, defaulting to %v.", c.Exchanges[i].Name, configDefaultWebsocketResponseCheckTimeout)
c.Exchanges[i].WebsocketResponseCheckTimeout = configDefaultWebsocketResponseCheckTimeout
}

if c.Exchanges[i].WebsocketResponseMaxLimit <= 0 {
log.Warnf("Exchange %s Websocket response max limit value not set, defaulting to %v.", c.Exchanges[i].Name, configDefaultWebsocketResponseMaxLimit)
c.Exchanges[i].WebsocketResponseMaxLimit = configDefaultWebsocketResponseMaxLimit
}

err := c.CheckPairConsistency(c.Exchanges[i].Name)
if err != nil {
log.Errorf("Exchange %s: CheckPairConsistency error: %s", c.Exchanges[i].Name, err)
Expand Down
19 changes: 17 additions & 2 deletions config/config_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -650,7 +650,6 @@ func TestUpdateExchangeConfig(t *testing.T) {
// TestCheckExchangeConfigValues logic test
func TestCheckExchangeConfigValues(t *testing.T) {
checkExchangeConfigValues := Config{}

err := checkExchangeConfigValues.LoadConfig(ConfigTestFile)
if err != nil {
t.Errorf(
Expand All @@ -665,12 +664,28 @@ func TestCheckExchangeConfigValues(t *testing.T) {
)
}

checkExchangeConfigValues.Exchanges[0].WebsocketResponseMaxLimit = 0
checkExchangeConfigValues.Exchanges[0].WebsocketResponseCheckTimeout = 0
checkExchangeConfigValues.Exchanges[0].HTTPTimeout = 0
checkExchangeConfigValues.CheckExchangeConfigValues()
err = checkExchangeConfigValues.CheckExchangeConfigValues()
if err != nil {
t.Errorf("Test failed. checkExchangeConfigValues.CheckExchangeConfigValues: %s",
err.Error(),
)
}

if checkExchangeConfigValues.Exchanges[0].HTTPTimeout == 0 {
t.Fatalf("Test failed. Expected exchange %s to have updated HTTPTimeout value", checkExchangeConfigValues.Exchanges[0].Name)
}

if checkExchangeConfigValues.Exchanges[0].WebsocketResponseMaxLimit == 0 {
t.Fatalf("Test failed. Expected exchange %s to have updated WebsocketResponseMaxLimit value", checkExchangeConfigValues.Exchanges[0].Name)
}

if checkExchangeConfigValues.Exchanges[0].WebsocketResponseCheckTimeout == 0 {
t.Fatalf("Test failed. Expected exchange %s to have updated WebsocketResponseCheckTimeout value", checkExchangeConfigValues.Exchanges[0].Name)
}

checkExchangeConfigValues.Exchanges[0].APIKey = "Key"
checkExchangeConfigValues.Exchanges[0].APISecret = "Secret"
checkExchangeConfigValues.Exchanges[0].AuthenticatedAPISupport = true
Expand Down
56 changes: 55 additions & 1 deletion config_example.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -209,6 +211,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -250,6 +254,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -291,6 +297,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -334,6 +342,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -376,6 +386,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand All @@ -387,7 +399,7 @@
"apiUrlSecondary": "NON_DEFAULT_HTTP_LINK_TO_EXCHANGE_API",
"proxyAddress": "",
"websocketUrl": "NON_DEFAULT_HTTP_LINK_TO_WEBSOCKET_EXCHANGE_API",
"availablePairs": "XRPM19,BCHM19,ADAM19,EOSM19,TRXM19,XBTUSD,XBT7D_U105,XBT7D_D95,XBTM19,XBTU19,ETHUSD,ETHM19,LTCM19",
"availablePairs": "XRPU19,BCHU19,ADAU19,ADAU19,TRXU19,XBTUSD,XBT7D_U105,XBT7D_D95,XBTU19,XBTZ19,ETHUSD,ETHU19,LTCU19",
"enabledPairs": "XBTUSD",
"baseCurrencies": "USD",
"assetTypes": "SPOT",
Expand Down Expand Up @@ -417,6 +429,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -458,6 +472,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -500,6 +516,8 @@
"websocket": true,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -542,6 +560,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -583,6 +603,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -624,6 +646,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -667,6 +691,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -710,6 +736,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -753,6 +781,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -794,6 +824,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -836,6 +868,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -879,6 +913,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -922,6 +958,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -964,6 +1002,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1006,6 +1046,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1046,6 +1088,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1086,6 +1130,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1129,6 +1175,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1172,6 +1220,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1215,6 +1265,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down Expand Up @@ -1259,6 +1311,8 @@
"websocket": false,
"useSandbox": false,
"restPollingDelay": 10,
"websocketResponseCheckTimeout": 30000000,
"websocketResponseMaxLimit": 7000000000,
"httpTimeout": 15000000000,
"httpUserAgent": "",
"httpDebugging": false,
Expand Down
4 changes: 2 additions & 2 deletions exchanges/alphapoint/alphapoint_websocket.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ const (
// WebsocketClient starts a new webstocket connection
func (a *Alphapoint) WebsocketClient() {
for a.Enabled {
var Dialer websocket.Dialer
var dialer websocket.Dialer
var err error
a.WebsocketConn, _, err = Dialer.Dial(a.WebsocketURL, http.Header{})
a.WebsocketConn, _, err = dialer.Dial(a.WebsocketURL, http.Header{})

if err != nil {
log.Errorf("%s Unable to connect to Websocket. Error: %s\n", a.Name, err)
Expand Down
Loading

0 comments on commit e209d85

Please sign in to comment.