Skip to content

Commit

Permalink
Added discover.find() timers.
Browse files Browse the repository at this point in the history 
Added 'stats.timers.find.ms'.
Added 'stats.timers.find.request.ms'.
Added 'stats.timers.find.round.ms'.
  • Loading branch information
@tristanls
tristanls committed Jan 20, 2014
1 parent fd16220 commit 4dfda8c
Show file tree
Hide file tree
Showing 6 changed files with 341 additions and 48 deletions.
17 changes: 16 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ _For more detailed documentation including private methods see [Discover doc](do
* [discover.register(contact)](#discoverregistercontact)
* [discover.unreachable(contact)](#discoverunreachablecontact)
* [discover.unregister(contact)](#discoverunregistercontact)
* [Event 'stats.timers.find.ms'](#event-statstimersfindms)
* [Event 'stats.timers.find.request.ms'](#event-statstimersfindrequestms)
* [Event 'stats.timers.find.round.ms'](#event-statstimersfindroundms)

### new Discover(options)

Expand Down Expand Up @@ -247,6 +250,18 @@ discover.find("Zm9v", function (error, contact) {

Unregisters previously registered `contact` (identified by `contact.id` and `contact.vectorClock`) from the network.

#### Event: `stats.timers.find.ms`

* `latency`: _Number_ Latency of `discover.find()` in milliseconds.

#### Event: `stats.timers.find.request.ms`

* `latency`: _Number_ Latency of a single request to another DHT server as part of a round of `discover.find()` DHT lookups.

#### Event: `stats.timers.find.round.ms`

* `latency`: _Number_ Latency of a single round of `discover.find()` DHT lookups in milliseconds.

### Transport Interface

Modules implementing the transport mechanism for Discover shall conform to the following interface. A `transport` is a JavaScript object.
Expand Down Expand Up @@ -443,4 +458,4 @@ An elaboration would be to distribute known contacts to other k-buckets when a `

The implementation has been sourced from:

- [A formal specification of the Kademlia distributed hash table](http://maude.sip.ucm.es/kademlia/files/pita_kademlia.pdf)
- [A formal specification of the Kademlia distributed hash table](http://maude.sip.ucm.es/kademlia/files/pita_kademlia.pdf)
38 changes: 37 additions & 1 deletion docs/Discover.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,14 @@ Node ids in Discover are represented as base64 encoded Strings. This is because
* [discover.getClosestKBuckets(nodeId)](#discovergetclosestkbucketsnodeid)
* [discover.queryCompletionCheck(query, callback)](#discoverquerycompletioncheckquery-callback)
* [discover.register(contact)](#discoverregistercontact)
* [discover.timerEndInMilliseconds(type, key)](#discovertimerendinmillisecondstype-key)
* [discover.timerStart(type, key)](#discovertimerstarttype-key)
* [discover.trace(message)](#discovertracemessage)
* [discover.unreachable(contact)](#discoverunreachablecontact)
* [discover.unregister(contact)](#discoverunregistercontact)
* [Event 'stats.timers.find.ms'](#event-statstimersfindms)
* [Event 'stats.timers.find.request.ms'](#event-statstimersfindrequestms)
* [Event 'stats.timers.find.round.ms'](#event-statstimersfindroundms)

#### new Discover(options)

Expand Down Expand Up @@ -147,6 +152,25 @@ discover.register({

_NOTE: Current implementation creates a new k-bucket for every registered node id. It is important to remember that a k-bucket could store up to k*lg(n) contacts, where lg is log base 2, n is the number of registered node ids on the network, and k is the size of each k-bucket (by default 20). For 1 billion registered nodes on the network, each k-bucket could store around 20 * lg (1,000,000,000) = ~ 598 contacts. This isn't bad, until you have 1 million local entities for a total of 598,000,000 contacts plus k-bucket overhead, which starts to put real pressure on Node.js/V8 memory limit._

#### discover.timerEndInMilliseconds(type, key)

_**CAUTION: reserved for internal use**_

* `type`: _String_ Timer type.
* `key`: _String_ Timer key.
* Return: _Number_ Milliseconds since the first time in the timer.

Calculates a millisecond interval between now and the first timer that was stored at `type` and `key`.

#### discover.timerStart(type, key)

_**CAUTION: reserved for internal use**_

* `type`: _String_ Timer type.
* `key`: _String_ Timer key.

Starts a new timer indexed by `type` and `key`. Multiple starts will result in start times being stored in an array for use by `discover.timerEndInMilliseconds()` later.

#### discover.trace(message)

* `message`: _String_ Message to trace.
Expand Down Expand Up @@ -181,4 +205,16 @@ discover.find("Zm9v", function (error, contact) {
* `id`: _String (base64)_ The previously registered contact id, base 64 encoded.
* `vectorClock`: _Integer_ _(Default: 0)_ Vector clock of contact to unregister.

Unregisters previously registered `contact` (identified by `contact.id` and `contact.vectorClock`) from the network.
Unregisters previously registered `contact` (identified by `contact.id` and `contact.vectorClock`) from the network.

#### Event: `stats.timers.find.ms`

* `latency`: _Number_ Latency of `discover.find()` in milliseconds.

#### Event: `stats.timers.find.request.ms`

* `latency`: _Number_ Latency of a single request to another DHT server as part of a round of `discover.find()` DHT lookups.

#### Event: `stats.timers.find.round.ms`

* `latency`: _Number_ Latency of a single round of `discover.find()` DHT lookups in milliseconds.
95 changes: 85 additions & 10 deletions index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,15 @@ var Discover = module.exports = function Discover (options) {
// register a listener to update our k-buckets with nodes that we manage
// contact with
self.transport.on('node', function (error, contact, nodeId, response) {
if (error)
var latency = self.timerEndInMilliseconds('find.request.ms', contact.id + nodeId);
if (error) {
self.emit('stats.timers.find.request.ms', latency);
return; // failed contact
}

// we successfully contacted the "contact", add it
self.add(contact);
self.emit('stats.timers.find.request.ms', latency);
});

// register a listener to handle transport 'findNode' events
Expand Down Expand Up @@ -240,6 +244,11 @@ var Discover = module.exports = function Discover (options) {
});

self.kBuckets = {};
self.timers = {
'find.ms': {},
'find.request.ms': {},
'find.round.ms': {}
};
};

util.inherits(Discover, events.EventEmitter);
Expand Down Expand Up @@ -295,6 +304,7 @@ Discover.prototype.add = function add (remoteContact) {
*/
Discover.prototype.executeQuery = function executeQuery (query, callback) {
var self = this;
self.timerStart('find.round.ms', query.nodeId);

if (self.tracing)
self.trace('executeQuery(' + util.inspect(query, false, null) + ')');
Expand All @@ -304,8 +314,14 @@ Discover.prototype.executeQuery = function executeQuery (query, callback) {
return;

// if we have no nodes, we can't query anything
if (!query.nodes || query.nodes.length == 0)
return callback(new Error("No known nodes to query"));
if (!query.nodes || query.nodes.length == 0) {
var latency = self.timerEndInMilliseconds('find.ms', query.nodeId);
var roundLatency = self.timerEndInMilliseconds('find.round.ms', query.nodeId);
callback(new Error("No known nodes to query"));
self.emit('stats.timers.find.ms', latency);
self.emit('stats.timers.find.round.ms', roundLatency);
return;
}

if (query.index === undefined)
query.index = 0;
Expand Down Expand Up @@ -422,9 +438,13 @@ Discover.prototype.executeQuery = function executeQuery (query, callback) {
}

// return the response and stop querying
var latency = self.timerEndInMilliseconds('find.ms', nodeId);
var roundLatency = self.timerEndInMilliseconds('find.round.ms', nodeId);
callback(null, response);
query.done = true;
self.transport.removeListener('node', query.listener);
self.emit('stats.timers.find.ms', latency);
self.emit('stats.timers.find.round.ms', roundLatency);
return;
};
self.transport.on('node', query.listener);
Expand All @@ -436,6 +456,7 @@ Discover.prototype.executeQuery = function executeQuery (query, callback) {
query.index++) {

query.ongoingRequests++;
self.timerStart('find.request.ms', query.nodes[query.index].id + query.nodeId);
self.transport.findNode(query.nodes[query.index], query.nodeId, query.sender);
}

Expand All @@ -456,11 +477,20 @@ Discover.prototype.executeQuery = function executeQuery (query, callback) {
*/
Discover.prototype.find = function find (nodeId, callback, announce) {
var self = this;
var traceHeader = "find(" + nodeId + "): ";
self.timerStart('find.ms', nodeId);

var traceHeader;

if (self.tracing) {
traceHeader = "find(" + nodeId + "): ";
}

// see if we have a local match, and return it if not announcing
if (!announce && self.kBuckets[nodeId]) {
return callback(null, self.kBuckets[nodeId].contact);
var latency = self.timerEndInMilliseconds('find.ms', nodeId);
callback(null, self.kBuckets[nodeId].contact);
self.emit('stats.timers.find.ms', latency);
return;
}

// if we have no kBuckets, that means we haven't registered any nodes yet
Expand Down Expand Up @@ -496,8 +526,11 @@ Discover.prototype.find = function find (nodeId, callback, announce) {
var contact = clone(closestContacts[0]);
contact.id = contact.id.toString("base64");
// hide internal implementation details
delete contact.distance;
return callback(null, contact);
delete contact.distance;
var latency = self.timerEndInMilliseconds('find.ms', nodeId);
callback(null, contact);
self.emit('stats.timers.find.ms', latency);
return;
}

// closestContacts will contain contacts with id as a Buffer, we clone
Expand Down Expand Up @@ -535,9 +568,13 @@ Discover.prototype.findViaSeeds = function findViaSeeds (nodeId, callback, annou

// if we have no seeds, that means we don't know of any other nodes to query
if (!self.seeds || self.seeds.length == 0) {
if (self.tracing)
if (self.tracing) {
self.trace(traceHeader + 'No known seeds to query');
return callback(new Error("No known seeds to query"));
}
var latency = self.timerEndInMilliseconds('find.ms', nodeId);
callback(new Error("No known seeds to query"));
self.emit('stats.timers.find.ms', latency);
return;
}

var closestNodes = [];
Expand Down Expand Up @@ -651,6 +688,8 @@ Discover.prototype.queryCompletionCheck = function queryCompletionCheck (query,
// find out if any new nodes are closer than the closest
// node in order to determine if we should keep going or
// stop
self.emit('stats.timers.find.round.ms',
self.timerEndInMilliseconds('find.round.ms', query.nodeId));

// console.log('sorting new nodes');
// sort the new nodes according to distance
Expand Down Expand Up @@ -686,7 +725,10 @@ Discover.prototype.queryCompletionCheck = function queryCompletionCheck (query,
return callback(null, query.closest);
} else {
// console.log("returning not found error");
return callback(new Error("not found"));
var latency = self.timerEndInMilliseconds('find.ms', query.nodeId);
callback(new Error("not found"));
self.emit('stats.timers.find.ms', latency);
return;
}
}

Expand Down Expand Up @@ -800,6 +842,39 @@ Discover.prototype.register = function register (contact) {
return clone(contact); // don't leak internal implementation
};

/*
* `type`: _String_ Timer type.
* `key`: _String_ Timer key.
* Return: _Number_ Milliseconds since the first time in the timer.
*/
Discover.prototype.timerEndInMilliseconds = function timerEndInMilliseconds(type, key) {
var self = this;

if (!self.timers[type] || !self.timers[type][key]) {
return 0;
}

var diff = process.hrtime(self.timers[type][key].shift());
if (self.timers[type][key].length == 0) {
delete self.timers[type][key];
}
// diff[0] : seconds
// diff[1] : nanoseconds
// 1 second = 1e9 nanoseconds
// 1 millisecond = 1e6 nanoseconds
return Math.floor((diff[0] * 1e9 + diff[1]) / 1e6);
};

/*
* `type`: _String_ Timer type.
* `key`: _String_ Timer key.
*/
Discover.prototype.timerStart = function timerStart(type, key) {
var self = this;
self.timers[type][key] = self.timers[type][key] || [];
self.timers[type][key].push(process.hrtime());
};

/*
* `message`: _String_ Message to trace.
*/
Expand Down
14 changes: 14 additions & 0 deletions scripts/localtest.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,20 @@ function startLocalTest() {
discover4.register(node4);
discover5.register(node5);

var discoverNodes = [discover1, discover2, discover3, discover4, discover5];

discoverNodes.forEach(function (discoverNode) {
discoverNode.on('stats.timers.find.ms', function (latency) {
console.log('~stats: stats.timers.find.ms ' + latency);
});
discoverNode.on('stats.timers.find.round.ms', function (latency) {
console.log('~stats: stats.timers.find.round.ms ' + latency);
});
discoverNode.on('stats.timers.find.request.ms', function (latency) {
console.log('~stats: stats.timers.find.request.ms ' + latency);
});
});

console.log('~script self-registrations complete');
console.log('~script allowing nodes to communicate and settle');

Expand Down
Loading

0 comments on commit 4dfda8c

Please sign in to comment.