Transport protocol

TAG || ENCRYPTED || NONCE --> PACKET
 ^         ^          ^
 |         |          |
 |         |          +-------------+
 |         |                        |
 |         +-------------+          |
 |                       |          |
 +--< AUTH(AUTH_KEY, ENCRYPTED || NONCE)
                         ^          ^
                         |          |
+------------------------+          |
|                                   |
|                   +---------------+
|                   |
+--< ENCRYPT(KEY, NONCE, PAYLOAD)
                    ^       ^
                    |       |
                    |       +--< DATA || PAD [|| ZEROS]
                    |
                    +--< PRP(PRP_KEY, SERIAL)

SERIAL is message’s serial number. Odds are reserved for client (to server) messages, evens for server (to client) messages.

PRP is XTEA block cipher algorithm used here as PRP (pseudo random permutation function) to obfuscate SERIAL. Plaintext SERIAL state is kept in peers internal state, but encrypted before transmission.

XTEA’s encryption key PRP_KEY is the first 128-bit of Salsa20’s output with established common key and zero nonce (message nonces start from 1).

PRP_KEY = 128bit(ENCRYPT(KEY, 0))

ENCRYPT is Salsa20 stream cipher, with established session KEY and obfuscated SERIAL used as a nonce. 512 bit of Salsa20’s output is ignored and only remaining is XORed with ther data, encrypting it.

DATA is padded with PAD (0x80 byte). Optional ZEROS may follow, to fill up packet to conceal payload packet length.

AUTH is Poly1305 authentication function. First 256 bits of Salsa20’s output are used as a one-time key for AUTH.

AUTH_KEY = 256bit(ENCRYPT(KEY, NONCE))

To prevent replay attacks we must remember received SERIALs and drop when receiving duplicate ones.

In encryptionless mode this scheme is slightly different:

 PACKET = ENCODED || NONCE
ENCODED = ENCLESS(DATA || PAD || ZEROS)
  NONCE = PRP(PRP_KEY, SERIAL)

ENCLESS is AONT and chaffing function. There is no need in explicit separate authentication.