Wire Protocol

Richard T. Carback III
David Stainton

Abstract

This document describes the Elixxir wire protocol which is the lowest level protocol in the Elixxir protocol stack and allows the Elixxir network components to communicate with one another.

Introduction

As mentioned in the architectural overview the Elixxir mix network functions as an overlay network. This means that Elixxir protocol layers are built on top of existing Internet protocols. The Elixxir wire protocol is built on top of TCP/IPv4 (IPv6 is not disabled but also not tested at this time) and consists of TLS transporting gRPC payloads.

TLS X.509 certs are derived from the Network Definition File (NDF). The NDF acts as a sort of DNS zone file for the entire network but includes port numbers and cryptographic keying material necessary for the xx protocol stack, like the consensus document (generated by the Tor network's Directory Authority system) in Tor.

Currently, a permissioning server, which will be replaced by blockchain consensus, generates and signs the NDF. The cMix nodes poll the permissioning server for the "Full" NDF, which consists of all network connectivity information. The cMix gateways poll their local cMix node for both a "Partial" and "Full" NDF. A Partial NDF does not contain connectivity information for other nodes, and clients are able to poll any cMix gateways for the Partial NDF.

Diagram: Permissioning (generates NDF) -> Nodes -> Gateways -> Clients

Gateways don't talk to permissioning
Clients don't talk to nodes or permissioning

Similar to how cacheing DNS resolvers prevent clients from having to talk directly to the authoritative servers, xx clients don't need to talk to PKI system directly because the NDF and mix rounds information is cached and made available by the Gateways.

The NDF contains a list of "Nodes" (cMix nodes) and "Gateways" (cMix gateways) that detail the address and the PEM encoded TLS X.509 public keys for each entity. An example of this data structure from a Partial NDF is as follows:

{
  "...": "...",
  "Gateways": [
    {
      "Id": "6ZTH8Y01DHrRFahBtLjF4uRTPed/JuM1R12lr2A2hv0B",
      "Address": "161.35.228.41:22840",
      "Tls_certificate": "-----BEGIN CERTIFICATE-----\nMIIFuzCCA6OgAwIBAgIUNCSGHmWhGjJ8Y60J6hgegyRbUo8wDQYJKoZIhvcNAQEL\nBQAwgYkxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJQ2xhcmVt\nb250MRIwEAYDVQQKDAl4eG5ldHdvcmsxDzANBgNVBAsMBnh4bm9kZTETMBEGA1UE\nAwwKeHgubmV0d29yazEfMB0GCSqGSIb3DQEJARYQYWRtaW5AeHgubmV0d29yazAe\nFw0yMTA4MDkyMTU3NTFaFw0yMzA4MDkyMTU3NTFaMIGJMQswCQYDVQQGEwJVUzEL\nMAkGA1UECAwCQ0ExEjAQBgNVBAcMCUNsYXJlbW9udDESMBAGA1UECgwJeHhuZXR3\nb3JrMQ8wDQYDVQQLDAZ4eG5vZGUxEzARBgNVBAMMCnh4Lm5ldHdvcmsxHzAdBgkq\nhkiG9w0BCQEWEGFkbWluQHh4Lm5ldHdvcmswggIiMA0GCSqGSIb3DQEBAQUAA4IC\nDwAwggIKAoICAQC+6/dn4Ke3Isg3lW8LAi05MIFOARzrENzz8/7gQq2rzDkLXKSU\nUyPB/GVAP4wgW4K9F9Op/AKO/bhEgrNVNiu4H8hk55iYS+P1ptWcYUxssypT8T2c\n1q0SK5DCAikXYdtgIuU3SsxrPP1vVWQDLCScPVmK3ChlaTwSzXGba/W1tOJzB0om\ngKsv6vNohyDX0nVVoqxrNwYcpXiOsnYbVBNzxeutuP2QkcLNyb+L62obXrYVhUDl\nl2r6y6GZCSmc6X86f6kB/ahyyrzqaiyeM8u6H7ZU0WZXtPBatgeQBV2JtvbiamdT\n0kAuhqS395hDsLS/JAuhwZZ5hirDl9YtELFkYpVvux+kDS6umU7PE++Fu65r3O1S\nuCg5d59E3udWlpfFg4nXjncYs2MQKkbna1Otj5JHzIEc/x6HckAuxf7mSTcp4mb3\n9i9PviPbKxvQ/CAL2MWt3feqb3OW8EBp1J6bNbpmQjwQbXenPnvgwyHoh+3HSCMm\nr5XwTy7+MCikOpP+4JisPUcsLx+120qudVpbFWMmL1ju+rolnawAY9khuIjjiTou\nWs9e4yuq2nr35w2b+Ok7nBDZNDUK29rXy/inbCOrjCKFyF6xYTa1qJ3t1XnLY82x\nKkWRBTdmrlwAGVIK0xhJ2Q/JhBeU9C2nT5FrZYlNBrZAOgXGD1scaYk38wIDAQAB\noxkwFzAVBgNVHREEDjAMggp4eC5uZXR3b3JrMA0GCSqGSIb3DQEBCwUAA4ICAQAY\nZn5TBTlkEqI8Yx/OTne1n50aqerUc76hn4Y/ghkVZKErMtJ3YJVWxcLJDWiqxX3T\nkBgaR5Yfv1yczZZa8g9E5mtnQyzZRRrcZR0eR/mhcoeUL0j+kpTaYzbR77nakyER\nGu7WBDYL4EKK2rrGpvwr0kPJSuyAVIpmk1Nv2wV8kaoQhAa6BzAcmQswq2oxbggG\nmNnutd2toZgieanicJFiFW9N/O2llVwUDdzVh2MxWt6kDp8VXCNz0uhtgofr0Quk\np6/eZDGu6z5FY4ap845BNArICvcdc5aw3h9W0QHD82Jw/ijzOzbWDG7KpAo2oIbK\nfPrPA3FNnSmN88mXBd+ve4McdgyfzKi7T8GJrxeIKpBNqHH1c9aQkMDIfo7wr5n9\nb03BbPk3PEbhCpoch2gCNWonwoYLplhpyH8rmrGlY7cPMFKBefy2MbK++nmzZYQE\nqgk6GhGF+KwiRsZqDxtU+jA7Dk5xHLMpYJjCDicrJo+s3PeSjxPRSoDnbRhs8S2y\ntN4N32gmkM5vuAuD4hY1UdeEU1PlUdEBxLIe9PP8sgd0KERmyO5nri933aD0TN/L\nq9yPgGl/JbGGMNii+j9bho9qzR01cCMaaoxvs7RU6CR5TFbwwAkNiMSYhs3NuZlm\nMg6k4jkhaoACInTAvu3EJx5GM5mZSEk/P7oD5/ddAw==\n-----END CERTIFICATE-----\n",
      "Bin": "NorthAmerica"
    },
    "...": "...", 
  ],
  "Nodes": [
    {
      "Id": "6ZTH8Y01DHrRFahBtLjF4uRTPed/JuM1R12lr2A2hv0C",
      "Address": "",
      "Tls_certificate": "",
      "Status": 0
    },
    {},
  ],
  "...": "...",
  
}

Every connection made by our wire protocol is protected by TLS. All public keys for TLS are in this (signed) NDF. To make a connection, the initator (Dialer) provides the TLS certificate and the connection string ("Address" field in NDF) and the TLS library uses this to check that the certificate matches. This is similar to certificate pinning, except certificates can change as hosts are added and removed on the network through the PKI (permissioning server, TBD consensus mechanism) via changes in the NDF. The PKI generates many cascades and the topology of these cascades is enforced with TLS authentication. For more details about the public key infrastructure, see the PKI section.

TLS Ciphersuite and Parameterizations

At this time, ciphersuite selections are the defaults for the go tls library. Our current desired selection is TLS_CHACHA20_POLY1305_SHA256 with the X25519 curve proposed by Daniel J. Bernstein.

We plan to move to a quantum resistant scheme in the future. Our intent is to exchange both quantum resistant and classical keys, then use the derivation of both to derive a symmetric cipher key.

FIXME: The source code does not specify the precise TLS ciphersuite and any other relavant parameters. Then specify the ciphersuite selection here in this document.

Implementation

TODO: Pull out code samples, link via footnote to actual code (as above). Explain intent behind code.

how members of thenetwork handle the NDF: https://git.xx.network/elixxir/comms/-/tree/release/network
wire protocol implementation https://git.xx.network/xx_network/comms/-/tree/release/connect
how members of thenetwork handle the NDF https://git.xx.network/elixxir/registration/-/blob/release/storage/state.go#L338

gRPC protocol messages used by each network component

The gRPC protocols are split across 3 proto files:

xx_network:comms/messages/messages.proto: Defines generic service and signature structures for requesting and authenticating tokens.
xx_network:comms/gossip/gossip.proto: Defines a service and structures for the gossip protocol used by gateways.
elixxir:comms/mixmessages/mixmessages.proto: Defines all of services and structures for the permissioning (registration), server, gateway, and clients in the cMix protocol.

xx_network:comms/messages/messages.proto

// Generic service definition
service Generic {
    // Authenticate a token with the server
    rpc AuthenticateToken (AuthenticatedMessage) returns (Ack) {
    }

    // Request a token from the server
    rpc RequestToken (Ping) returns (AssignToken) {
    }
}

// Generic response message providing an error message from remote servers
message Ack {
    string Error = 1;
}

// Empty message for requesting action from any type of server
message Ping {
}

// Wrapper for authenticated messages that also ensure integrity
message AuthenticatedMessage {
    bytes ID = 1;
    bytes Signature = 2;
    bytes Token = 3;
    ClientID Client = 4;
    google.protobuf.Any Message = 5;
}

// Message used for assembly of Client IDs in the system
message ClientID {
    bytes Salt = 1;
    string PublicKey = 2;
}

// Provides a token to establish reverse identity to any type of client
message AssignToken {
    bytes Token = 1;
}

// RSASignature is a digital signature for the RSA algorithm
message RSASignature {
    bytes Nonce = 1;
    bytes Signature = 2;
}

// ECCSignature is a digital signature for the ECC algorithm
message ECCSignature {
    bytes Nonce = 1;
    bytes Signature = 2;
}

Authenticated messaging is used between gateways and servers and between servers.

xx_network:comms/gossip/gossip.proto

// RPC for handling generic reception of Gossip messages
service Gossip {
    rpc Endpoint (GossipMsg) returns (Ack);
    rpc Stream (stream GossipMsg) returns (Ack);
}

// Generic response message providing an error message from remote servers
message Ack {
    string Error = 1;
}

// Generic message used for a variety of Gossip protocols
message GossipMsg {
    string Tag = 1;
    bytes  Origin = 2;
    bytes  Payload = 3;
    bytes  Signature = 4;
    int64 timestamp = 5;
}

Gateways use gossip messages to transmit information between each other. Specifically, gatways gossip bloom filters which say what ephemeral recipient IDs received messages in each round. Gossip is also used by the consensus/blockchain information.

elixxir:comms/messages/messages.proto

In lieu of the size of the file in https://git.xx.network/elixxir/comms/-/blob/dcd58978e446738d0fbbacbbaed9e5f74b0b6e46/mixmessages/mixmessages.proto, we instead discuss each service and summarize each function here. Instead of a direct implementation, the services often use a generalized interface that is implemented in the corresponding repository. We reference each as a footnote below.

Server

  a. AskOnline (messages.AuthenticatedMessage) returns (messages.Ack) -- Polls a 
     server to determine if it is active for running rounds in the cMix protocol. 
  b. CreateNewRound (messages.AuthenticatedMessage) returns (messages.Ack) --
     makes a new round with a certain it. 
  c. UploadUnmixedBatch(server mixmessages.Node_UploadUnmixedBatchServer, auth
     *connect.Auth) error -- Server interface for sending a new batch
  d. DownloadMixedBatch(..., 
     batchInfo *mixmessages.BatchReady, auth *connect.Auth) error -- Server 
     interface for handling a mixed batch request
  e. FinishRealtime(message *mixmessages.RoundInfo, ..., auth *connect.Auth) error -- 
     Server interface for broadcasting when realtime is complete
  f. GetRoundBufferInfo(auth *connect.Auth) (int, error) -- GetRoundBufferInfo  
     returns # of available precomputations
  g. PrecompTestBatch(stream mixmessages.Node_PrecompTestBatchServer, 
     info *mixmessages.RoundInfo, auth *connect.Auth) error -- PrecompTestBatch is a 
     server to server streaming broadcast. It simulates sending the completed batch 
     of FinishRealtime, testing for connectivity.
  h. GetMeasure(message *mixmessages.RoundInfo, auth *connect.Auth) (*mixmessages.
     RoundMetrics, error) -- Returns round metrics for the given roundinfo object.
  j. PostPhase(message *mixmessages.Batch, auth *connect.Auth) error -- Server 
     Interface for all Internode Comms. Each "phase" processes a batch on a different 
     operation of the cMix protocol (precomputation and realtime).
  **k. StreamPostPhase(server mixmessages.Node_StreamPostPhaseServer, 
     auth *connect.Auth) error -- streaming version of the PostPhase function.**
  **l. PostPrecompResult(roundID uint64, numSlots uint32, auth *connect.Auth) error -- 
     PostPrecompResult interface to finalize both payloads' precomp**
  **m. Poll(msg *mixmessages.ServerPoll, auth *connect.Auth) (
     *mixmessages.ServerPollResponse, error) -- Gateway -> Server unified polling for 
     network definition file and round information.**
  n. SendRoundTripPing(ping *mixmessages.RoundTripPing, auth *connect.Auth) error -- 
     Given a specific network topology (path), collect statistics about how fast 
     packets can travel through it.
  o. RoundError(error *mixmessages.RoundError, auth *connect.Auth) error -- receive a 
     report/broadcast of an error to all nodes in a round. 
  p. GetNDF() (*interconnect.NDF, error) -- 	Consensus node -> cMix node NDF request
     NOTE: For now cMix nodes serve the NDF to the consensus nodes, but this will be 
     reversed once consensus generates the NDF
  q. GetPermissioningAddress() (string, error) -- GetPermissioningAddress gets 
     gateway the permissioning server's address from server.
  r. StartSharePhase(ri *mixmessages.RoundInfo, auth *connect.Auth) error -- Server 
     -> Server initiating multi-party round DH key generation
  s. SharePhaseRound(sharedPiece *mixmessages.SharePiece, auth *connect.Auth) error 
     -- Server -> Server passing state of multi-party round DH key generation
  t. ShareFinalKey(sharedPiece *mixmessages.SharePiece, auth *connect.Auth) error -- 
     Server -> Server sending multi-party round DH key
  u. RequestClientKey(nonceRequest *mixmessages.SignedClientKeyRequest,
     auth *connect.Auth) (*mixmessages.SignedKeyResponse, error) -- Server interface 
     for RequestNonceMessage

The bolded elements are the key functions used in the cMix protocol. The rest are peripheral/utility functions or could be deprecated in the future.

Gateway

  a. PutMessage(message *pb.GatewaySlot, ipAddr string) (*pb.GatewaySlotResponse, 
     error) -- Clients upload a message to the cMix Gateway
  b. PutManyMessages(msgs *pb.GatewaySlots, ipAddr string) (*pb.GatewaySlotResponse,
     error) --  Clients upload many messages to the cMix Gateway
  c. Poll(msg *pb.GatewayPoll) (*pb.GatewayPollResponse, error) -- Client -> Gateway 
     unified polling for network information
  d. RequestHistoricalRounds(msg *pb.HistoricalRounds) (*pb.HistoricalRoundsResponse,
     error) -- Client -> Client function for getting historical round data (to figure 
     out if the gateway has messages for this client.)
  e. RequestMessages(msg *pb.GetMessages) (*pb.GetMessagesResponse, error) -- Client 
     -> Gateway message retrieval.
  f. RequestClientKey(message *pb.SignedClientKeyRequest) (*pb.SignedKeyResponse, 
     error) -- Passthrough from client to server for establishing keys with the cMix
     nodes.

Gateways are the interface between nodes and clients. Clients cannot contact nodes directly and this is enforced via firewall/network configuration.

Permissioning (Registrar)

  a. RegisterUser(msg *pb.ClientRegistration) (confirmation 
     *pb.SignedClientRegistrationConfirmations, err error) -- User registration for 
     client devices. This is not accessible by clients per firewall rules and is 
     instead implemented by Client Registrar. 
  b. RegisterNode(salt []byte, serverAddr, serverTlsCert, gatewayAddr,
     gatewayTlsCert, registrationCode string) error -- Node registration for the cMix 
     protocol
  c. PollNdf(ndfHash []byte) (*pb.NDF, error) -- utility function for retrieving the 
     NDF from the network. Limited to local services by firewall rules. Not meant to
     be used by clients.
  d. Poll(msg *pb.PermissioningPoll, auth *connect.Auth) (*pb.
     PermissionPollResponse, error) -- Network information polling function. Returns 
     NDF, RoundInfo, and other information and events.
  e. CheckRegistration(msg *pb.RegisteredNodeCheck) (*pb.RegisteredNodeConfirmation, 
     error) -- Helper function to determine if node is successfully registered on the 
     permissioning server. If not registration is rerun.

Client Registrar

  a. 	RegisterUser(msg *pb.ClientRegistration) (confirmation
      *pb.SignedClientRegistrationConfirmations, err error) -- Signs a public key 
      for the client to enable key negotiation with cMix nodes (through their 
      gateways) on the network. Used to limit # of new clients that can access 
      the network.

There exist other auxilliary services (Authorizor, UDB, and NotificationsBot). These have wire protocol components but are not necessary as part of the core protocol.

Security Consideration

Defense in depth implies using multiple layers of encryption protocols, each of which covers a different segment of the overall end to end path. Without compromising or operating one of the xx network nodes it will not be possible for the adversary to gain access to cMix ciphertexts flowing through the network. It might be the case that this makes certain attacks more difficult, for example attacking the cMix protocol would first require attacking the wire protocol. See the threat model document for further discussion about Compulsion Attacks.

Abstract​

Introduction​

TLS Ciphersuite and Parameterizations​

Implementation​

gRPC protocol messages used by each network component​

xx_network:comms/messages/messages.proto​

xx_network:comms/gossip/gossip.proto​

elixxir:comms/messages/messages.proto​

Security Consideration​