Avalanche is a decentralized p2p (peer-to-peer) network of nodes that work together to run the Avalanche blockchain protocol.
The network package implements the networking layer of the protocol which allows a node to discover, connect to, and communicate with other peers.
Peers are defined as members of the network that communicate with one another to participate in the Avalanche protocol.
Peers communicate by enqueuing messages between one another. Each peer on either side of the connection asynchronously reads and writes messages to and from the remote peer. Messages include both application-level messages used to support the Avalanche protocol, as well as networking-level messages used to implement the peer-to-peer communication layer.
sequenceDiagram
loop
Peer-1->>Peer-2: Write outbound messages
Peer-2->>Peer-1: Read incoming messages
end
loop
Peer-2->>Peer-1: Write outbound messages
Peer-1->>Peer-2: Read incoming messages
end
When starting an Avalanche node, a node needs to be able to initiate some process that eventually allows itself to become a participating member of the network. In traditional web2 systems, it's common to use a web service by hitting the service's DNS and being routed to an available server behind a load balancer. In decentralized p2p systems however, connecting to a node is more complex as no single entity owns the network. Avalanche consensus requires a node to repeatedly sample peers in the network, so each node needs some way of discovering and connecting to every other peer to participate in the protocol.
In Avalanche, nodes connect to an initial set of bootstrapper nodes known as beacons (this is user-configurable). Once connected to a set of beacons, a node is able to discover other nodes in the network. Over time, a node eventually discovers other peers in the network through PeerList messages it receives through:
- The handshake initiated between two peers when attempting to connect to a peer (see Connecting).
- Periodic
PeerListgossip messages that every peer sends to the peers it's connected to (see Connected).
Upon connection to any peer, a handshake is performed between the node attempting to establish the outbound connection to the peer and the peer receiving the inbound connection.
When attempting to establish the connection, the first message that the node attempting to connect to the peer in the network is a Version message describing compatibility of the candidate node with the peer. As an example, nodes that are attempting to connect with an incompatible version of AvalancheGo or a significantly skewed local clock are rejected by the peer.
sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Version message
Node->>Peer: AvalancheGo v1.0.0
Note right of Peer: My version v1.9.4 is incompatible with your version v1.0.0.
Peer-xNode: Connection dropped
Note over Node,Peer: Handshake Failed
If the Version message is successfully received and the peer decides that it wants a connection with this node, it replies with a PeerList message that contains metadata about other peers that allows a node to connect to them. Upon reception of a PeerList message, a node will attempt to connect to any peers that the node is not already connected to to allow the node to discover more peers in the network.
sequenceDiagram
Note over Node,Peer: Initiate Handshake
Note left of Node: I want to connect to you!
Note over Node,Peer: Version message
Node->>Peer: AvalancheGo v1.9.4
Note right of Peer: LGTM!
Note over Node,Peer: PeerList message
Peer->>Node: Peer-X, Peer-Y, Peer-Z
Note over Node,Peer: Handshake Complete
Node->>Peer: ACK Peer-X, Peer-Y, Peer-Z
Once the node attempting to join the network receives this PeerList message, the handshake is complete and the node is now connected to the peer. The node attempts to connect to the new peers discovered in the PeerList message. Each connection results in another peer handshake, which results in the node incrementally discovering more and more peers in the network as more and more PeerList messages are exchanged.
Some peers aren't discovered through the PeerList messages exchanged through peer handshakes. This can happen if a peer is either not randomly sampled, or if a new peer joins the network after the node has already connected to the network.
sequenceDiagram
Node ->> Peer-1: Version - v1.9.5
Peer-1 ->> Node: PeerList - Peer-2
Node ->> Peer-1: ACK - Peer-2
Note left of Node: Node is connected to Peer-1 and now tries to connect to Peer-2.
Node ->> Peer-2: Version - v1.9.5
Peer-2 ->> Node: PeerList - Peer-1
Node ->> Peer-2: ACK - Peer-1
Note left of Node: Peer-3 was never sampled, so we haven't connected yet!
Node --> Peer-3: No connection
To guarantee that a node can discover all peers, each node periodically gossips a sample of the peers it knows about to other peers.
A PeerList is the message that is used to communicate the presence of peers in the network. Each PeerList message contains networking-level metadata about the peer that provides the necessary information to connect to it, alongside the corresponding transaction id that added that peer to the validator set. Transaction ids are unique hashes that only add a single validator, so it is guaranteed that there is a 1:1 mapping between a validator and its associated transaction id.
PeerListAck messages are sent in response to PeerList messages to allow a peer to confirm which peers it will actually attempt to connect to. Because nodes only gossip peers they believe another peer doesn't already know about to optimize bandwidth, PeerListAck messages are important to confirm that a peer will attempt to connect to someone. Without this, a node might gossip a peer to another peer and assume a connection between the two is being established, and not re-gossip the peer in future gossip cycles. If the connection was never actually wanted by the peer being gossiped to due to a transient reason, that peer would never be able to re-discover the gossiped peer and could be isolated from a subset of the network.
Once a PeerListAck message is received from a peer, the node that sent the original PeerList message marks the corresponding acknowledged validators as already having been transmitted to the peer, so that it's excluded from subsequent iterations of PeerList gossip.
Handshake messages provide a node with some knowledge of peers in the network, but offers no guarantee that learning about a subset of peers from each peer the node connects with will result in the node learning about every peer in the network.
In order to provide a probabilistic guarantee that all peers in the network will eventually learn of one another, each node periodically gossips a sample of the peers that they're aware of to a sample of the peers that they're connected to. Over time, this probabilistically guarantees that every peer will eventually learn of every other peer.
To optimize bandwidth usage, each node tracks which peers are guaranteed to know of which peers. A node learns this information by tracking both inbound and outbound PeerList gossip.
- Inbound
- If a node ever receives
PeerListfrom a peer, that peer must have known about the peers in thatPeerListmessage in order to have gossiped them.
- If a node ever receives
- Outbound
- If a node sends a
PeerListto a peer and the peer replies with anPeerListAckmessage, then all peers in thePeerListAckmust be known by the peer.
- If a node sends a
To efficiently track which peers know of which peers, the peers that each peer is aware of is represented in a bit set. A peer is represented by either a 0 if it isn't known by the peer yet, or a 1 if it is known by the peer.
An node follows the following steps for every cycle of PeerList gossip:
- Get a sample of peers in the network that the node is connected to
- For each peer:
- Figure out which peers the node hasn't gossiped to them yet.
- Take a random sample of these unknown peers.
- Send a message describing these peers to the peer.
sequenceDiagram
Note left of Node: Initialize gossip bit set for Peer-123
Note left of Node: Peer-123: [0, 0, 0]
Node->>Peer-123: PeerList - Peer-1
Peer-123->>Node: PeerListAck - Peer-1
Note left of Node: Peer-123: [1, 0, 0]
Node->>Peer-123: PeerList - Peer-3
Peer-123->>Node: PeerListAck - Peer-3
Note left of Node: Peer-123: [1, 0, 1]
Node->>Peer-123: PeerList - Peer-2
Peer-123->>Node: PeerListAck - Peer-2
Note left of Node: Peer-123: [1, 1, 1]
Note left of Node: No more gossip left to send to Peer-123!
Because network state is generally expected to be stable (i.e nodes are not continuously flickering online/offline), as more and more gossip messages are exchanged nodes eventually realize that the peers that they are connected to have learned about every other peer.
A node eventually stops gossiping peers when there's no more new peers to gossip about. PeerList gossip only resumes once:
- a new peer joins
- a peer disconnects and reconnects
- a new validator joins the network
- a validator's IP is updated