blob: 4ff8c9afc48dd94fa7ad80941ddcc18cded3030c [file] [log] [blame]
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<rfc category="info" docName="sp-zerotier-mapping-01">
<title abbrev="ZeroTier mapping for SPs">
ZeroTier Mapping for Scalability Protocols
<author fullname="Garrett D'Amore" initials="G." role="editor"
<date month="October" year="2016" />
<workgroup>Internet Engineering Task Force</workgroup>
<t>This document defines the ZeroTier mapping for scalability protocols.</t>
<section title="Underlying protocol">
<t>ZeroTier expresses an 802.3 style layer 2, where frames
maybe exchanged as if they were Ethernet frames. Virtual broadcast
domains are created within a numbered "network", and frames may then
be exchanged with any peers on that network.</t>
<t>Frames may arrive in any order, or be lost, just a with Ethernet
(best effort delivery), but they are strongly protected by a
cryptographic checksum, so frames that do arrive will be uncorrupted.
Furthermore, ZeroTier guarantees that a given frame will
be received at most once.</t>
<t>Each application on a ZeroTier network has its own address, called
a ZeroTier ID (ZTID), which is globally unique -- this is generated from
a hash of the public key associated with the application.</t>
<t>A given application may participate in multiple ZeroTier networks.</t>
<t>We assume each nanomsg application will have it's own ZeroTier ID,
and will not use more than one. Management of these IDs, as well as
the underlying key pairs, is out of scope of this document.</t>
<t>ZeroTier networks have a maximum payload MTU of 2800 bytes.
However they also have an "optimum" MTU, based upon the underlying
networks (typically UDP) and overheads that are used to exchange
such packets. For our purposes we will assume this to be approximately
1400 bytes. These values can change on different networks.</t>
<section title="Packet layout">
<t>Each nanomsg message sent over ZeroTier will be comprised of one
or more fragments, where each fragment is mapped to a single
underlying ZeroTier L2 frame. We use the EtherType field
of 0901 to indicate nanomsg over ZeroTier protocol (number to
registered with IEEE).</t>
<t>Each frame shall be prepended with the following header:</t>
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
| destination mac address (bytes 0-3) |
| destination mac (bytes 4-5) | source mac (bytes 0-1) |
| source mac address (bytes 2-5) |
| 0x90 | 0x01 | op | flags | version |
| fragment offset | fragment length |
| destination port | source port |
| type | message ID |
| payload...
<t>All numeric fields are in big-endian byte order.</t>
<t>As above, the start of each frame is just as a normal Ethernet
frame, with destination, source, and type of 0x901.</t>
<t>The op is a field that indicates the type of message
being sent. The following values are defined: DATA (0),
CONN (1), DISC (2), PING (3), and ERR (4). These are discussed
further below. Implementations MUST discard messages where the
op is not one of these.</t>
<t>There are two flags defined. The first is MF (1), which
indicates that a message is fragmented, and more fragments follow.
This flag may only be set on DATA messages. The last fragment
of a message will not have this flag set.</t>
<t>The second flag is AK (2), which indicates that a
given message is a reply to an earlier message. This is
only valid for the CONN, DISC, and PING message types.</t>
<t>Note that the MF and the AK flag bits are mutually exclusive.</t>
<t>The version byte MUST be set to 0x1. Implementations MUST
discard any messages received for any other version.</t>
<t>The fragment length and offset are given in terms of octets,
and only include the payload. For example, the first fragment
of a message bearing a 2000 byte payload, itself only carrying
1400 bytes of that payload would have the MF bit set in flags,
offset 0, and length 1400. The second fragment would have the MF
bit clear, length 600, and offset 1400.</t>
<t>As a single fragment cannot exceed the size of a ZeroTier frame,
the high order six bits of the fragment length MUST be zero, and
the value encoded MUST be less than 2800.</t>
<t>Each fragment for a given message must carry the same
message ID. Implementations MUST initialize this to a random
value, and MUST increment this each time a new message is sent.</t>
<t>The port fields are used to discriminate different uses, allowing
one application to have multiple connections or sockets open.
The purpose is analogous to TCP port numbers, except that instead
of the operating system performing the discrimination the application
or library code must do so.</t>
<t>The type field is the numeric SP protocol ID, in big-endian
form. When receiving a message for a port, if the SP protocol ID
does not match the SP protocol expected on that port, the
implementation MUST discard this message.</t>
<t>The maximum payload size, and hence the maximum SP message that
may be transmitted using this transport, is 65535 octets.
Implementations are encouraged to restrict this further.</t>
<t>Note that it is not by accident that the payload is 32-bit
aligned in this message format.</t>
<t>Source and destination MAC addresses shall be constructed
algorithmically from the relevant ZeroTier IDs.</t>
<t>Note that at this time, broadcast and multicast is not supported
by this mapping. (A future update may resolve this.)</t>
<section title = "DATA messages">
<t>DATA messages carry SP protocol payload data. They can only
be sent on an established session (see CONN messages below),
and are never acknowledged.</t>
<section title = "CONN messages">
<t>CONN frames represent a session establishment. They allow
a peer to advertise its port number to a remote peer, and to verify
that a peer is responsive. The payload for the CONN frame is a 4
byte (big-endian) value, consisting of the SP protocol ID of the
<t>The connection is initiated by the initiator sending this
message, with its own SP protocol ID. The AK flag will in this
case be clear.</t>
<t>The responder will acknowledge this by replying with its
SP protocol ID in the 4-byte payload, with the AK flag set.</t>
<t>Alternatively, a responder may reject the connection attempt by
sending a suitably formed ERR message (see below).</t>
<t>If a sender does not receive a reply, it SHOULD retry this
message before giving up and reporting an error to the user.</t>
<t>If a CONN frame is received for a session that already
exists, the receiver MUST reply. The CONN request is idempotent.</t>
<section title = "DISC messages">
<t>DISC messages are used to request a session be terminated.
This notifies the remote sender that no more data will be
sent or accepted, and the session resources may be released.
There is no payload. The party closing the session sends this
with the AK flag clear. There is no acknowledgement.</t>
<section title = "PING messages">
<t>In order to keep session state, implementations will generally
store data for each session. In order to prevent a stale session
from consuming these resources forever, and in order to keep
underlying ZeroTier sessions alive, a PING message may be sent.
This message has no payload.</t>
<t>The sender MUST leave the AK bit clear. If the PING is
is successful, then the responder MUST reply with a PING message
with the AK bit set.</t>
<t>In the event of an error, an implemenation MAY reply with an
ERR message.</t>
<t>Implementations MUST not initiate PING messages if they have
either received or sent other session messages recently.</t>
<t>Implemenations shall use a timeout T1 seconds of be used
before initiating a message the first time, and that in the absence
of a reply, up to N further attempts be made, separated by T2 seconds.
If no reply to the Nth attempt is received after T2 seconds have
passed, then the remote peer should be assumed offline or dead, and
the session closed.</t>
<t>It is recommended that T1, T2, and N all be configurable, with
recommended default values of 60, 10, and 5. With these values,
sessions that appear dead after 2 minutes will be closed, and their
resources reclaimed.</t>
<section title = "ERR messages">
<t>ERR messages indicate a failure in the session, and abruptly
terminates the session. The payload for these messages consists
of a single byte error code, followed by an ASCII message describing
the error (not terminated by zero). This message shall not be
more than 128 bytes in length.</t>
<t>The following error codes are defined:
<t>0x01 No party listening at that address or port.</t>
<t>0x02 No such session found.</t>
<t>0x03 SP protocol ID invalid.</t>
<t>0x04 Generic protocol error.</t>
<t>0x05 Message size too big.</t>
<t>0xff Other uncategorized error.</t>
<t>Implemenations MUST discard any session state upon receiving an
ERR message. These messages are not acknowledged.</t>
<section title = "Reassembly Guidelines">
<t>Implementations MUST accept and reassemble fragmented DATA messages.
Implementations MUST discard fragmented messages of other types.</t>
<t>Messages larger than the ZeroTier MTU (2800) MUST be fragmented.</t>
<t>Implementations SHOULD limit the number of unassembled messages
retained for reassembly, to minimize the likelihood of intentional
abuse. It is suggested that at most 2 unassembled messages be
retained. It is further suggested that if 2 or more unfragmented
messages arrive before a message is reassembled, or more than 5
seconds pass before the reassembly is complete, that the unassembled
fragments be discarded.</t>
<section title="Ports">
<t>The port numbers are 16-bit fields, allowing a single ZT ID to
service multiple application layer protocols, which could be
treated as seperate end points, or as separate sockets in the
application. The implementation is responsible for discriminating
on these and delivering to the appropriate consumer.</t>
<t>As with UDP or TCP, it is intended that each party have its own
port number, and that a pair of ports (combined with ZeroTier IDs)
be used to identify a single conversation.</t>
<t>An SP server should allocate a port for number advertisement.
It is expected cliets will generate ephemeral port numbers.</t>
<t>Implementations are free to choose how to allocate port numbers,
but it is recommended manually configured port numbers are small,
with the high order bit clear, and that numbers > 32768 (high order
bit set) be used for ephemeral allocations.</t>
<t>It is recommended that separate short queues (perhaps just one
or two messages long) be kept per local port in implementations, to
prevent head-of-line blocking issues where backpressure on one
consumer (perhaps just a single thread or socket) blocks others.</t>
<section anchor="URI" title="URI Format">
<t>The URI scheme used to represent ZeroTier addresses makes
use of ZeroTier IDs, ZeroTier network IDs, and our own 16-bit
<t>The format shall be nnzt://&lt;nwid>/&lt;ztid>:&lt;port>,
where the &lt;nwid>
component represents the 16-digit hexadecimal ZeroTier network ID, the
&lt;ztid> represents the 10-digit hexadecimal ZeroTier Device ID, and the
&lt;port> is the 16-bit port number previously described.</t>
<section anchor="IANA" title="IANA Considerations">
<t>This memo includes no request to IANA.</t>
<section anchor="Security" title="Security Considerations">
<t>The mapping isn't intended to provide any additional security in
addition to what ZeroTier does.</t>