Wire Protocol
Richard T. Carback III
David Stainton
Abstract
This document describes the Elixxir wire protocol which is the lowest level protocol in the Elixxir protocol stack and allows the Elixxir network components to communicate with one another.
Introduction
As mentioned in the architectural overview the Elixxir mix network functions as an overlay network. This means that Elixxir protocol layers are built on top of existing Internet protocols. The Elixxir wire protocol is built on top of TCP/IPv4 (IPv6 is not disabled but also not tested at this time) and consists of TLS transporting gRPC payloads.
TLS X.509 certs are derived from the Network Definition File (NDF). The NDF acts as a sort of DNS zone file for the entire network but includes port numbers and cryptographic keying material necessary for the xx protocol stack, like the consensus document (generated by the Tor network's Directory Authority system) in Tor.
Currently, a permissioning server, which will be replaced by blockchain consensus, generates and signs the NDF. The cMix nodes poll the permissioning server for the "Full" NDF, which consists of all network connectivity information. The cMix gateways poll their local cMix node for both a "Partial" and "Full" NDF. A Partial NDF does not contain connectivity information for other nodes, and clients are able to poll any cMix gateways for the Partial NDF.
Diagram: Permissioning (generates NDF) -> Nodes -> Gateways -> Clients
- Gateways don't talk to permissioning
- Clients don't talk to nodes or permissioning
Similar to how cacheing DNS resolvers prevent clients from having to talk directly to the authoritative servers, xx clients don't need to talk to PKI system directly because the NDF and mix rounds information is cached and made available by the Gateways.
The NDF contains a list of "Nodes" (cMix nodes) and "Gateways" (cMix gateways) that detail the address and the PEM encoded TLS X.509 public keys for each entity. An example of this data structure from a Partial NDF is as follows:
{
"...": "...",
"Gateways": [
{
"Id": "6ZTH8Y01DHrRFahBtLjF4uRTPed/JuM1R12lr2A2hv0B",
"Address": "161.35.228.41:22840",
"Tls_certificate": "-----BEGIN CERTIFICATE-----\nMIIFuzCCA6OgAwIBAgIUNCSGHmWhGjJ8Y60J6hgegyRbUo8wDQYJKoZIhvcNAQEL\nBQAwgYkxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJQ2xhcmVt\nb250MRIwEAYDVQQKDAl4eG5ldHdvcmsxDzANBgNVBAsMBnh4bm9kZTETMBEGA1UE\nAwwKeHgubmV0d29yazEfMB0GCSqGSIb3DQEJARYQYWRtaW5AeHgubmV0d29yazAe\nFw0yMTA4MDkyMTU3NTFaFw0yMzA4MDkyMTU3NTFaMIGJMQswCQYDVQQGEwJVUzEL\nMAkGA1UECAwCQ0ExEjAQBgNVBAcMCUNsYXJlbW9udDESMBAGA1UECgwJeHhuZXR3\nb3JrMQ8wDQYDVQQLDAZ4eG5vZGUxEzARBgNVBAMMCnh4Lm5ldHdvcmsxHzAdBgkq\nhkiG9w0BCQEWEGFkbWluQHh4Lm5ldHdvcmswggIiMA0GCSqGSIb3DQEBAQUAA4IC\nDwAwggIKAoICAQC+6/dn4Ke3Isg3lW8LAi05MIFOARzrENzz8/7gQq2rzDkLXKSU\nUyPB/GVAP4wgW4K9F9Op/AKO/bhEgrNVNiu4H8hk55iYS+P1ptWcYUxssypT8T2c\n1q0SK5DCAikXYdtgIuU3SsxrPP1vVWQDLCScPVmK3ChlaTwSzXGba/W1tOJzB0om\ngKsv6vNohyDX0nVVoqxrNwYcpXiOsnYbVBNzxeutuP2QkcLNyb+L62obXrYVhUDl\nl2r6y6GZCSmc6X86f6kB/ahyyrzqaiyeM8u6H7ZU0WZXtPBatgeQBV2JtvbiamdT\n0kAuhqS395hDsLS/JAuhwZZ5hirDl9YtELFkYpVvux+kDS6umU7PE++Fu65r3O1S\nuCg5d59E3udWlpfFg4nXjncYs2MQKkbna1Otj5JHzIEc/x6HckAuxf7mSTcp4mb3\n9i9PviPbKxvQ/CAL2MWt3feqb3OW8EBp1J6bNbpmQjwQbXenPnvgwyHoh+3HSCMm\nr5XwTy7+MCikOpP+4JisPUcsLx+120qudVpbFWMmL1ju+rolnawAY9khuIjjiTou\nWs9e4yuq2nr35w2b+Ok7nBDZNDUK29rXy/inbCOrjCKFyF6xYTa1qJ3t1XnLY82x\nKkWRBTdmrlwAGVIK0xhJ2Q/JhBeU9C2nT5FrZYlNBrZAOgXGD1scaYk38wIDAQAB\noxkwFzAVBgNVHREEDjAMggp4eC5uZXR3b3JrMA0GCSqGSIb3DQEBCwUAA4ICAQAY\nZn5TBTlkEqI8Yx/OTne1n50aqerUc76hn4Y/ghkVZKErMtJ3YJVWxcLJDWiqxX3T\nkBgaR5Yfv1yczZZa8g9E5mtnQyzZRRrcZR0eR/mhcoeUL0j+kpTaYzbR77nakyER\nGu7WBDYL4EKK2rrGpvwr0kPJSuyAVIpmk1Nv2wV8kaoQhAa6BzAcmQswq2oxbggG\nmNnutd2toZgieanicJFiFW9N/O2llVwUDdzVh2MxWt6kDp8VXCNz0uhtgofr0Quk\np6/eZDGu6z5FY4ap845BNArICvcdc5aw3h9W0QHD82Jw/ijzOzbWDG7KpAo2oIbK\nfPrPA3FNnSmN88mXBd+ve4McdgyfzKi7T8GJrxeIKpBNqHH1c9aQkMDIfo7wr5n9\nb03BbPk3PEbhCpoch2gCNWonwoYLplhpyH8rmrGlY7cPMFKBefy2MbK++nmzZYQE\nqgk6GhGF+KwiRsZqDxtU+jA7Dk5xHLMpYJjCDicrJo+s3PeSjxPRSoDnbRhs8S2y\ntN4N32gmkM5vuAuD4hY1UdeEU1PlUdEBxLIe9PP8sgd0KERmyO5nri933aD0TN/L\nq9yPgGl/JbGGMNii+j9bho9qzR01cCMaaoxvs7RU6CR5TFbwwAkNiMSYhs3NuZlm\nMg6k4jkhaoACInTAvu3EJx5GM5mZSEk/P7oD5/ddAw==\n-----END CERTIFICATE-----\n",
"Bin": "NorthAmerica"
},
"...": "...",
],
"Nodes": [
{
"Id": "6ZTH8Y01DHrRFahBtLjF4uRTPed/JuM1R12lr2A2hv0C",
"Address": "",
"Tls_certificate": "",
"Status": 0
},
{},
],
"...": "...",
}
Every connection made by our wire protocol is protected by TLS. All public keys for TLS are in this (signed) NDF. To make a connection, the initator (Dialer) provides the TLS certificate and the connection string ("Address" field in NDF) and the TLS library uses this to check that the certificate matches. This is similar to certificate pinning, except certificates can change as hosts are added and removed on the network through the PKI (permissioning server, TBD consensus mechanism) via changes in the NDF. The PKI generates many cascades and the topology of these cascades is enforced with TLS authentication. For more details about the public key infrastructure, see the PKI section.
TLS Ciphersuite and Parameterizations
At this time, ciphersuite selections are the defaults for the go tls library.
Our current desired selection is TLS_CHACHA20_POLY1305_SHA256 with the X25519
curve proposed by Daniel J. Bernstein.
We plan to move to a quantum resistant scheme in the future. Our intent is to exchange both quantum resistant and classical keys, then use the derivation of both to derive a symmetric cipher key.
FIXME: The source code does not specify the precise TLS ciphersuite and any other relavant parameters. Then specify the ciphersuite selection here in this document.
Implementation
TODO: Pull out code samples, link via footnote to actual code (as above). Explain intent behind code.
-
how members of thenetwork handle the NDF: https://git.xx.network/elixxir/comms/-/tree/release/network
-
wire protocol implementation https://git.xx.network/xx_network/comms/-/tree/release/connect
-
how members of thenetwork handle the NDF https://git.xx.network/elixxir/registration/-/blob/release/storage/state.go#L338
gRPC protocol messages used by each network component
The gRPC protocols are split across 3 proto files:
- xx_network:comms/messages/messages.proto: Defines generic service and signature structures for requesting and authenticating tokens.
- xx_network:comms/gossip/gossip.proto: Defines a service and structures for the gossip protocol used by gateways.
- elixxir:comms/mixmessages/mixmessages.proto: Defines all of services and structures for the permissioning (registration), server, gateway, and clients in the cMix protocol.
xx_network:comms/messages/messages.proto
// Generic service definition
service Generic {
// Authenticate a token with the server
rpc AuthenticateToken (AuthenticatedMessage) returns (Ack) {
}
// Request a token from the server
rpc RequestToken (Ping) returns (AssignToken) {
}
}
// Generic response message providing an error message from remote servers
message Ack {
string Error = 1;
}
// Empty message for requesting action from any type of server
message Ping {
}
// Wrapper for authenticated messages that also ensure integrity
message AuthenticatedMessage {
bytes ID = 1;
bytes Signature = 2;
bytes Token = 3;
ClientID Client = 4;
google.protobuf.Any Message = 5;
}
// Message used for assembly of Client IDs in the system
message ClientID {
bytes Salt = 1;
string PublicKey = 2;
}
// Provides a token to establish reverse identity to any type of client
message AssignToken {
bytes Token = 1;
}
// RSASignature is a digital signature for the RSA algorithm
message RSASignature {
bytes Nonce = 1;
bytes Signature = 2;
}
// ECCSignature is a digital signature for the ECC algorithm
message ECCSignature {
bytes Nonce = 1;
bytes Signature = 2;
}
Authenticated messaging is used between gateways and servers and between servers.
xx_network:comms/gossip/gossip.proto
// RPC for handling generic reception of Gossip messages
service Gossip {
rpc Endpoint (GossipMsg) returns (Ack);
rpc Stream (stream GossipMsg) returns (Ack);
}
// Generic response message providing an error message from remote servers
message Ack {
string Error = 1;
}
// Generic message used for a variety of Gossip protocols
message GossipMsg {
string Tag = 1;
bytes Origin = 2;
bytes Payload = 3;
bytes Signature = 4;
int64 timestamp = 5;
}
Gateways use gossip messages to transmit information between each other. Specifically, gatways gossip bloom filters which say what ephemeral recipient IDs received messages in each round. Gossip is also used by the consensus/blockchain information.
elixxir:comms/messages/messages.proto
In lieu of the size of the file in https://git.xx.network/elixxir/comms/-/blob/dcd58978e446738d0fbbacbbaed9e5f74b0b6e46/mixmessages/mixmessages.proto, we instead discuss each service and summarize each function here. Instead of a direct implementation, the services often use a generalized interface that is implemented in the corresponding repository. We reference each as a footnote below.
a. AskOnline (messages.AuthenticatedMessage) returns (messages.Ack) -- Polls a
server to determine if it is active for running rounds in the cMix protocol.
b. CreateNewRound (messages.AuthenticatedMessage) returns (messages.Ack) --
makes a new round with a certain it.
c. UploadUnmixedBatch(server mixmessages.Node_UploadUnmixedBatchServer, auth
*connect.Auth) error -- Server interface for sending a new batch
d. DownloadMixedBatch(...,
batchInfo *mixmessages.BatchReady, auth *connect.Auth) error -- Server
interface for handling a mixed batch request
e. FinishRealtime(message *mixmessages.RoundInfo, ..., auth *connect.Auth) error --
Server interface for broadcasting when realtime is complete
f. GetRoundBufferInfo(auth *connect.Auth) (int, error) -- GetRoundBufferInfo
returns # of available precomputations
g. PrecompTestBatch(stream mixmessages.Node_PrecompTestBatchServer,
info *mixmessages.RoundInfo, auth *connect.Auth) error -- PrecompTestBatch is a
server to server streaming broadcast. It simulates sending the completed batch
of FinishRealtime, testing for connectivity.
h. GetMeasure(message *mixmessages.RoundInfo, auth *connect.Auth) (*mixmessages.
RoundMetrics, error) -- Returns round metrics for the given roundinfo object.
j. PostPhase(message *mixmessages.Batch, auth *connect.Auth) error -- Server
Interface for all Internode Comms. Each "phase" processes a batch on a different
operation of the cMix protocol (precomputation and realtime).
**k. StreamPostPhase(server mixmessages.Node_StreamPostPhaseServer,
auth *connect.Auth) error -- streaming version of the PostPhase function.**
**l. PostPrecompResult(roundID uint64, numSlots uint32, auth *connect.Auth) error --
PostPrecompResult interface to finalize both payloads' precomp**
**m. Poll(msg *mixmessages.ServerPoll, auth *connect.Auth) (
*mixmessages.ServerPollResponse, error) -- Gateway -> Server unified polling for
network definition file and round information.**
n. SendRoundTripPing(ping *mixmessages.RoundTripPing, auth *connect.Auth) error --
Given a specific network topology (path), collect statistics about how fast
packets can travel through it.
o. RoundError(error *mixmessages.RoundError, auth *connect.Auth) error -- receive a
report/broadcast of an error to all nodes in a round.
p. GetNDF() (*interconnect.NDF, error) -- Consensus node -> cMix node NDF request
NOTE: For now cMix nodes serve the NDF to the consensus nodes, but this will be
reversed once consensus generates the NDF
q. GetPermissioningAddress() (string, error) -- GetPermissioningAddress gets
gateway the permissioning server's address from server.
r. StartSharePhase(ri *mixmessages.RoundInfo, auth *connect.Auth) error -- Server
-> Server initiating multi-party round DH key generation
s. SharePhaseRound(sharedPiece *mixmessages.SharePiece, auth *connect.Auth) error
-- Server -> Server passing state of multi-party round DH key generation
t. ShareFinalKey(sharedPiece *mixmessages.SharePiece, auth *connect.Auth) error --
Server -> Server sending multi-party round DH key
u. RequestClientKey(nonceRequest *mixmessages.SignedClientKeyRequest,
auth *connect.Auth) (*mixmessages.SignedKeyResponse, error) -- Server interface
for RequestNonceMessage
The bolded elements are the key functions used in the cMix protocol. The rest are peripheral/utility functions or could be deprecated in the future.
a. PutMessage(message *pb.GatewaySlot, ipAddr string) (*pb.GatewaySlotResponse,
error) -- Clients upload a message to the cMix Gateway
b. PutManyMessages(msgs *pb.GatewaySlots, ipAddr string) (*pb.GatewaySlotResponse,
error) -- Clients upload many messages to the cMix Gateway
c. Poll(msg *pb.GatewayPoll) (*pb.GatewayPollResponse, error) -- Client -> Gateway
unified polling for network information
d. RequestHistoricalRounds(msg *pb.HistoricalRounds) (*pb.HistoricalRoundsResponse,
error) -- Client -> Client function for getting historical round data (to figure
out if the gateway has messages for this client.)
e. RequestMessages(msg *pb.GetMessages) (*pb.GetMessagesResponse, error) -- Client
-> Gateway message retrieval.
f. RequestClientKey(message *pb.SignedClientKeyRequest) (*pb.SignedKeyResponse,
error) -- Passthrough from client to server for establishing keys with the cMix
nodes.
Gateways are the interface between nodes and clients. Clients cannot contact nodes directly and this is enforced via firewall/network configuration.
a. RegisterUser(msg *pb.ClientRegistration) (confirmation
*pb.SignedClientRegistrationConfirmations, err error) -- User registration for
client devices. This is not accessible by clients per firewall rules and is
instead implemented by Client Registrar.
b. RegisterNode(salt []byte, serverAddr, serverTlsCert, gatewayAddr,
gatewayTlsCert, registrationCode string) error -- Node registration for the cMix
protocol
c. PollNdf(ndfHash []byte) (*pb.NDF, error) -- utility function for retrieving the
NDF from the network. Limited to local services by firewall rules. Not meant to
be used by clients.
d. Poll(msg *pb.PermissioningPoll, auth *connect.Auth) (*pb.
PermissionPollResponse, error) -- Network information polling function. Returns
NDF, RoundInfo, and other information and events.
e. CheckRegistration(msg *pb.RegisteredNodeCheck) (*pb.RegisteredNodeConfirmation,
error) -- Helper function to determine if node is successfully registered on the
permissioning server. If not registration is rerun.
a. RegisterUser(msg *pb.ClientRegistration) (confirmation
*pb.SignedClientRegistrationConfirmations, err error) -- Signs a public key
for the client to enable key negotiation with cMix nodes (through their
gateways) on the network. Used to limit # of new clients that can access
the network.
There exist other auxilliary services (Authorizor, UDB, and NotificationsBot). These have wire protocol components but are not necessary as part of the core protocol.
Security Consideration
Defense in depth implies using multiple layers of encryption protocols, each of which covers a different segment of the overall end to end path. Without compromising or operating one of the xx network nodes it will not be possible for the adversary to gain access to cMix ciphertexts flowing through the network. It might be the case that this makes certain attacks more difficult, for example attacking the cMix protocol would first require attacking the wire protocol. See the threat model document for further discussion about Compulsion Attacks.