blob: d5add5c46909322a3e6c122ae39ccb36da9f828c [file] [log] [blame]
Internet Engineering Task Force G. D'Amore, Ed.
Intended status: Informational October 26, 2016
Expires: April 29, 2017
ZeroTier Mapping for Scalability Protocols
This document defines the ZeroTier mapping for scalability protocols.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 29, 2017.
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
( in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
D'Amore Expires April 29, 2017 [Page 1]
Internet-Draft ZeroTier mapping for SPs October 2016
1. Underlying protocol
ZeroTier expresses an 802.3 style layer 2, where frames maybe
excchanged as if they were Ethernet frames. Virtual broadcast
domains are created within a numbered "network", and frames may then
be exchanged with any peers on that network.
Frames may arrive in any order, or be lost, just a with Ethernet
(best effort delivery), but they are strongly protected by a
cryptographic checksum, so frames that do arrive will be uncorrupted.
Furthermore, ZeroTier guarantees that a given frame will be received
at most once.
Each application on a ZeroTier network has its own address, called a
ZeroTier ID (ZTID), which is globally unique -- this is generated
from a hash of the public key associated with the application.
A given application may participate in multiple ZeroTier networks.
We assume each nanomsg application will have it's own ZeroTier ID,
and will not use more than one. Management of these IDs, as well as
the underlying key pairs, is out of scope of this document.
ZeroTier networks have a maximum payload MTU of 2800 bytes. However
they also have an "optimum" MTU, based upon the underlying networks
(typically UDP) and overheads that are used to exchange such packets.
For our purposes we will assume this to be approximately 1400 bytes.
These values can change on different networks.
2. Packet layout
Each nanomsg message sent over ZeroTier will be comprised of one or
more fragments, where each fragment is mapped to a single underlying
ZeroTier L2 frame. We use the EtherType field of 0901 to indicate
nanomsg over ZeroTier protocol (number to registered with IEEE).
Each frame shall be prepended with the following header:
D'Amore Expires April 29, 2017 [Page 2]
Internet-Draft ZeroTier mapping for SPs October 2016
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
| destination mac address (bytes 0-3) |
| destination mac (bytes 4-5) | source mac (bytes 0-1) |
| source mac address (bytes 2-5) |
| 0x90 | 0x01 | op | flags | version |
| fragment offset | fragment length |
| destination port | source port |
| type | message ID |
| payload...
All numeric fields are in big-endian byte order.
As above, the start of each frame is just as a normal Ethernet frame,
with destination, source, and type of 0x901.
The op is a field that indicates the type of message being sent. The
following values are defined: DATA (0), CONN (1), DISC (2), PING (3),
and ERR (4). These are discussed further below. Implementations
MUST discard messages where the op is not one of these.
There are two flags defined. The first is MF (1), which indicates
that a message is fragmented, and more fragments follow. This flag
may only be set on DATA messages. The last fragment of a message
will not have this flag set.
The second flag is AK (2), which indicates that a given message is a
reply to an earlier message. This is only valid for the CONN, DISC,
and PING message types.
Note that the MF and the AK flag bits are mutually exclusive.
The version byte MUST be set to 0x1. Implementations MUST discard
any messages received for any other version.
The fragment length and offset are given in terms of octets, and only
include the payload. For example, the first fragment of a message
bearing a 2000 byte payload, itself only carrying 1400 bytes of that
D'Amore Expires April 29, 2017 [Page 3]
Internet-Draft ZeroTier mapping for SPs October 2016
payload would have the MF bit set in flags, offset 0, and length
1400. The second fragment would have the MF bit clear, length 600,
and offset 1400.
As a single fragment cannot exceed the size of a ZeroTier frame, the
high order six bits of the fragment length MUST be zero, and the
value encoded MUST be less than 2800.
Each fragment for a given message must carry the same message ID.
Implementations MUST initialize this to a random value, and MUST
increment this each time a new message is sent.
The port fields are used to discriminate different uses, allowing one
application to have multiple connections or sockets open. The
purpose is analogous to TCP port numbers, except that instead of the
operating system performing the discrimination the application or
library code must do so.
The type field is the numeric SP protocol ID, in big-endian form.
When receiving a message for a port, if the SP protocol ID does not
match the SP protocol expected on that port, the implementation MUST
discard this message.
The maximum payload size, and hence the maximum SP message that may
be transmitted using this transport, is 65535 octets.
Implementations are encouraged to restrict this further.
Note that it is not by accident that the payload is 32-bit aligned in
this message format.
Source and destination MAC addresses shall be constructed
algorithmically from the relevant ZeroTier IDs.
Note that at this time, broadcast and multicast is not supported by
this mapping. (A future update may resolve this.)
3. DATA messages
DATA messages carry SP protocol payload data. They can only be sent
on an established session (see CONN messages below), and are never
4. CONN messages
CONN frames represent a session establishment. They allow a peer to
advertise its port number to a remote peer, and to verify that a peer
is responsive. The payload for the CONN frame is a 4 byte (big-
endian) value, consisting of the SP protocol ID of the sender.
D'Amore Expires April 29, 2017 [Page 4]
Internet-Draft ZeroTier mapping for SPs October 2016
The connection is initiated by the initiator sending this message,
with its own SP protocol ID. The AK flag will in this case be clear.
The responder will acknowledge this by replying with its SP protocol
ID in the 4-byte payload, with the AK flag set.
Alternatively, a responder may reject the connection attempt by
sending a suitably formed ERR message (see below).
If a sender does not receive a reply, it SHOULD retry this message
before giving up and reporting an error to the user.
If a CONN frame is received for a session that already exists, the
receiver MUST reply. The CONN request is idempotent.
5. DISC messages
DISC messages are used to request a session be terminated. This
notifies the remote sender that no more data will be sent or
accepted, and the session resources may be released. There is no
payload. The party closing the session sends this with the AK flag
clear. There is no acknowledgement.
6. PING messages
In order to keep session state, implementations will generally store
data for each session. In order to prevent a stale session from
consuming these resources forever, and in order to keep underlying
ZeroTier sessions alive, a PING message may be sent. This message
has no payload.
The sender MUST leave the AK bit clear. If the PING is is
successful, then the responder MUST reply with a PING message with
the AK bit set.
In the event of an error, an implemenation MAY reply with an ERR
Implementations MUST not initiate PING messages if they have either
received or sent other session messages recently.
Implemenations shall use a timeout T1 seconds of be used before
initiating a message the first time, and that in the absence of a
reply, up to N further attempts be made, separated by T2 seconds. If
no reply to the Nth attempt is received after T2 seconds have passed,
then the remote peer should be assumed offline or dead, and the
session closed.
D'Amore Expires April 29, 2017 [Page 5]
Internet-Draft ZeroTier mapping for SPs October 2016
It is recommended that T1, T2, and N all be configurable, with
recommended default values of 60, 10, and 5. With these values,
sessions that appear dead after 2 minutes will be closed, and their
resources reclaimed.
7. ERR messages
ERR messages indicate a failure in the session, and abruptly
terminates the session. The payload for these messages consists of a
single byte error code, followed by an ASCII message describing the
error (not terminated by zero). This message shall not be more than
128 bytes in length.
The following error codes are defined:
0x01 No party listening at that address or port.
0x02 No such session found.
0x03 SP protocol ID invalid.
0x04 Generic protocol error.
0x05 Message size too big.
0xff Other uncategorized error.
Implemenations MUST discard any session state upon receiving an ERR
message. These messages are not acknowledged.
8. Reassembly Guidelines
Implementations MUST accept and reassemble fragmented DATA messages.
Implementations MUST discard fragmented messages of other types.
Messages larger than the ZeroTier MTU (2800) MUST be fragmented.
Implementations SHOULD limit the number of unassembled messages
retained for reassembly, to minimize the likelihood of intentional
abuse. It is suggested that at most 2 unassembled messages be
retained. It is further suggested that if 2 or more unfragmented
messages arrive before a message is reassembled, or more than 5
seconds pass before the reassembly is complete, that the unassembled
fragments be discarded.
D'Amore Expires April 29, 2017 [Page 6]
Internet-Draft ZeroTier mapping for SPs October 2016
9. Ports
The port numbers are 16-bit fields, allowing a single ZT ID to
service multiple application layer protocols, which could be treated
as seperate end points, or as separate sockets in the application.
The implementation is responsible for discriminating on these and
delivering to the appropriate consumer.
As with UDP or TCP, it is intended that each party have its own port
number, and that a pair of ports (combined with ZeroTier IDs) be used
to identify a single conversation.
An SP server should allocate a port for number advertisement. It is
expected cliets will generate ephemeral port numbers.
Implementations are free to choose how to allocate port numbers, but
it is recommended manually configured port numbers are small, with
the high order bit clear, and that numbers > 32768 (high order bit
set) be used for ephemeral allocations.
It is recommended that separate short queues (perhaps just one or two
messages long) be kept per local port in implementations, to prevent
head-of-line blocking issues where backpressure on one consumer
(perhaps just a single thread or socket) blocks others.
10. URI Format
The URI scheme used to represent ZeroTier addresses makes use of
ZeroTier IDs, ZeroTier network IDs, and our own 16-bit ports.
The format shall be nnzt://<nwid>/<ztid>:<port>, where the <nwid>
component represents the 16-digit hexadecimal ZeroTier network ID,
the <ztid> represents the 10-digit hexadecimal ZeroTier Device ID,
and the <port> is the 16-bit port number previously described.
11. IANA Considerations
This memo includes no request to IANA.
12. Security Considerations
The mapping isn't intended to provide any additional security in
addition to what ZeroTier does.
D'Amore Expires April 29, 2017 [Page 7]
Internet-Draft ZeroTier mapping for SPs October 2016
Author's Address
Garrett D'Amore (editor)
D'Amore Expires April 29, 2017 [Page 8]