Zero Tier

Manual – ZeroTier https://zerotier.
com/manual/
ZeroTier
Global Area Networking
Menu
Search … Search
1. Introduction DOWNLOAD
PRICING
ZeroTier is a smart Ethernet switch for planet Earth. USERS
DOWNLOAD
KNOWLEDGE BASE
It’s a distributed network hypervisor built atop a cryptographically secure global peer to peer SUPPORT
network. It provides advanced network virtualization and management capabilities on par with an COMMUNITY
enterprise SDN switch, but across both local and wide area networks and connecting almost any MANUAL
kind of app or device. DEVELOPERS
DOWNLOAD
GITHUB
This manual describes the design and operation of ZeroTier and its associated services, apps, and COMMUNITY
libraries. Its intended audience includes IT professionals, network administrators, information MANUAL
security experts, and developers. NEWS
ABOUT
LOGIN
The �rst section (2) of this guide explains ZeroTier’s design and operation at a high level and is
SKIP TO MENU TOGGLE BUTTON
written for those with at least an intermediate knowledge of topics like TCP/IP and Ethernet
networking. It’s not required reading for most users, but understanding how things work in detail
helps clarify everything else and helps tremendously with troubleshooting should anything go
wrong.
The remaining sections deal more concretely with deployment and administration.
Table of Contents
1. Introduction
2. Network Hypervisor Overview
1. VL1: The ZeroTier Peer to Peer Network
1. Network Topology and Peer Discovery
2. Addressing
3. Cryptography
4. Trusted Paths for Fast Local SDN
5. Multipath
2. VL2: The Ethernet Virtualization Layer
1. Network Identi�ers and Controllers
2. Certi�cates and Other Credentials
3. Multicast, ARP, NDP, and Special Addressing Modes
4. Ethernet Bridging
5. Public Networks
6. Ad-Hoc Networks
7. Quality of Service (QoS)
3. The Network Rules Engine
1. Rule Sets and Rule Evaluation
1. Actions and Match Conditions
2. Capabilities
3. Tags
4. Rule De�nition Language
1. An Introductory Example
2. Rule De�nition Language Syntax
3. Actions, Matches, Operators, and Constants
5. Useful Design Patterns
1. TCP Whitelisting
2. Locking Down UDP
3. Traf�c Observation and Interception
4. The Classi�ed System Pattern
4. ZeroTier One
1. JSON API
2. Local Con�guration Options
3. Network Controller Con�guration
4. Creating Your Own Roots (a.k.a. “Moons”)
5. SDK
1. Socket API (libzt)
1. Starting the Service
2. Joining a Network ?
3. Communicating with peers
1 de 24 27/11/19 10:51
Manual – ZeroTier https://zerotier.com/manual/
4. Handling events
1. Node Events
2. Network Events
3. Peer Events
4. Path Events
5. Route Events
6. Address Events
7. Network Stack Events (debugging)
8. Netif Events (debugging)
5. Errors
6. Thread Model
7. Statistics
8. Network Controller Mode
2. Frame-based API (libztcore)
3. Downloads
4. Examples
1. C Example
2. Java Example
2. Network Hypervisor Overview

The ZeroTier network hypervisor (currently found in the node/ subfolder of the ZeroTierOne git
repository) is a self-contained network virtualization engine that implements an Ethernet
virtualization layer similar to VXLAN on top of a global encrypted peer to peer network.
The ZeroTier protocol is original, though aspects of it are similar to VXLAN and IPSec. It has two
conceptually separate but closely coupled layers in the OSI model sense: VL1 and VL2. VL1 is the
underlying peer to peer transport layer, the “virtual wire,” while VL2 is an emulated Ethernet layer
that provides operating systems and apps with a familiar communication medium.
2.1. VL1: The ZeroTier Peer to Peer Network
A global data center requires a global wire closet.
In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6 cables or wireless radio
channels over which data is carried and the physical transceiver chips that modulate and
demodulate it. VL1 is a peer to peer network that does the same thing by using encryption,
authentication, and a lot of networking tricks to create virtual wires on a dynamic as-needed
basis.
2.1.1. Network Topology and Peer Discovery
VL1 is designed to be zero-con�guration. A user can start a new ZeroTier node without having to
write con�guration �les or provide the IP addresses of other nodes. It’s also designed to be fast.
Any two devices in the world should be able to locate each other and communicate almost
instantly.
To achieve this VL1 is organized like DNS. At the base of the network is a collection of always-
present root servers whose role is similar to that of DNS root name servers. Roots run the same
software as regular endpoints but reside at fast stable locations on the network and are
designated as such by a world de�nition. World de�nitions come in two forms: the planet and one
or more moons. The protocol includes a secure mechanism allowing world de�nitions to be
updated in-band if root servers’ IP addresses or ZeroTier addresses change.
There is only one planet. Earth’s root servers are operated by ZeroTier, Inc. as a free service. There
are currently twelve root servers organized into two six-member clusters distributed across every
major continent and multiple network providers. Almost everyone in the world has one within less
than 100ms network latency from their location.
A node can “orbit” any number of moons. A moon is just a convenient way to add user-de�ned
root servers to the pool. Users can create moons to reduce dependency on ZeroTier, Inc.
infrastructure or to locate root servers closer for better performance. For on-premise SDN use a
cluster of root servers can be located inside a building or data center so that ZeroTier can continue
to operate normally if Internet connectivity is lost.
Nodes start with no direct links to one another, only upstream to roots (planet and moons). Every
peer on VL1 possesses a globally unique 40-bit (10 hex digit) ZeroTier address, but unlike IP
addresses these are opaque cryptographic identi�ers that encode no routing information. To
communicate peers �rst send packets “up” the tree, and as these packets traverse the network
they trigger the opportunistic creation of direct links along the way. The tree is constantly trying to
“collapse itself” to optimize itself to the pattern of traf�c it is carrying.
Peer to peer connection setup goes like this:
1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a
root).
2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet
upstream until planetary roots are reached. Planetary roots know about all nodes, so
eventually the packet will reach B if B is online.
3. R also sends a message called rendezvous to A containing hints about how it might reach B.
2 de 24 27/11/19 10:51
Meanwhile the root that forwards the packet to B sends rendezvous informing B how it
might reach A.
4. A and B get their rendezvous messages and attempt to send test messages to each other,
possibly accomplishing hole punching of any NATs or stateful �rewalls that happen to be in
the way. If this works a direct link is established and packets no longer need to take the
scenic route.
Since roots forward packets, A and B can reach each other instantly. A and B then begin
attempting to make a direct peer to peer connection. If this succeeds it results in a faster lower
latency link. We call this transport triggered link provisioning since it’s the forwarding of the packet
itself that triggers the peer to peer network to attempt direct connection.
VL1 never gives up. If a direct path can’t be established, communication can continue through
(slower) relaying. Direct connection attempts continue forever on a periodic basis. VL1 also has
other features for establishing direct connectivity including LAN peer discovery, port prediction for
traversal of symmetric IPv4 NATs, and explicit port mapping using uPnP and/or NAT-PMP if these
are available on the local physical LAN.
A blog post from 2014 by ZeroTier’s original author explains some of the reasoning behind VL1’s
design.
2.1.2. Addressing
Every node is uniquely identi�ed on VL1 by a 40-bit (10 hex digit) ZeroTier address. This address
is computed from the public portion of a public/private key pair. A node’s address, public key, and
private key together form its identity.
On devices running ZeroTier One the node identity is stored in identity.public and identity.secret
in the service’s home directory.
When ZeroTier starts for the �rst time it generates a new identity. It then attempts to advertise it
upstream to the network. In the very unlikely event that the identity’s 40-bit unique address is
taken, it discards it and generates another.
Identities are claimed on a �rst come �rst serve basis and currently expire from planetary roots
after 60 days of inactivity. If a long-dormant device returns it may re-claim its identity unless its
address has been taken in the meantime (again, highly unlikely).
The address derivation algorithm used to compute addresses from public keys imposes a
computational cost barrier against the intentional generation of a collision. Currently it would take
approximately 10,000 CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensive but
not impossible, but it’s only the �rst line of defense. After generating a collision an attacker would
then have to compromise all upstream nodes, network controllers, and anything else that has
recently communicated with the target node and replace their cached identities.
ZeroTier addresses are, once advertised and claimed, a very secure method of unique
identi�cation.
When a node attempts to send a message to another node whose identity is not cached, it sends
a whois query upstream to a root. Roots provide an authoritative identity cache.
2.1.3. Cryptography
If you don’t know much about cryptography you can safely skip this section. TL;DR: packets are
end-to-end encrypted and can’t be read by roots or anyone else, and we use modern 256-bit
crypto in ways recommended by the professional cryptographers that created it.
Asymmetric public key encryption is Curve25519/Ed25519, a 256-bit elliptic curve variant.
Every VL1 packet is encrypted end to end using (as of the current version) 256-bit Salsa20 and
authenticated using the Poly1305 message authentication (MAC) algorithm. MAC is computed
after encryption (encrypt-then-MAC) and the cipher/MAC composition used is identical to the NaCl
reference implementation.
As of today we do not implement forward secrecy or other stateful cryptographic features in VL1.
We don’t do this for the sake of simplicity, reliability, and code footprint, and because frequently
changing state makes features like clustering and fail-over much harder to implement. See our
discussion on GitHub.
We may implement forward secrecy in the future. For those who want this level of security today,
we recommend using other cryptographic protocols such as SSL or SSH over ZeroTier. These
protocols typically implement forward secrecy, but using them over ZeroTier also provides the
secondary bene�t of defense in depth. Most cryptography is compromised not by a �aw in
encryption but through bugs in the implementation. If you’re using two secure transports, the odds
of a critical bug being discovered in both at the same time is very low. The CPU overhead of
double-encryption is not signi�cant for most work loads.
2.1.4. Trusted Paths for Fast Local SDN
To support the use of ZeroTier as a high performance SDN/NFV protocol over physically secure
3 de 24 27/11/19 10:51
networks the protocol supports a feature called trusted paths. It is possible to con�gure all
ZeroTier devices on a given network to skip encryption and authentication for traf�c over a
designated physical path. This can cut CPU use noticeably in high traf�c scenarios but at the cost
of losing virtually all transport security.
Trusted paths do not prevent communication with devices elsewhere, since traf�c over other
paths will be encrypted and authenticated normally.
We don’t recommend the use of this feature unless you really need the performance and you
know what you’re doing. We also recommend thinking carefully before disabling transport
security on a cloud private network. Larger cloud providers such as Amazon and Azure tend to
provide good network segregation but many less costly providers offer private networks that are
“party lines” and are not much more secure than the open Internet.
2.1.5. Multipath
As of 1.4.0 (coming soon) ZeroTier will allow peers to communicate over multiple physical paths
simultaneously and will automatically load balance according to path strength. A peer supporting
multipath logic will fall back to classical non-multipath behavior when communicating with peers
which do not support it or that do not have it enabled. This feature is accomplished via passive
and active measurement techniques and all additional traf�c overhead will cease when user
traf�c ceases, there is no ambient traf�c overhead when not in use.
Modes
Currently two modes are supported: random and proportionally balanced. The former will send
traf�c over any paths that are detected and while no balancing takes place it will stop sending
traf�c over a path if it expires. The latter will continuously measure the quality of each path and
allocate traf�c across them in proportion to their observed stability and performance.
Notion of link quality
Quality (speci�cally quality relative to other paths) is the value we use to determine how to
allocate traf�c across paths. It is composed of two sub-quantities: stability and performance .
These separate quantities are important for future quality of service (QoS) efforts. For instance we
may wish to map certain types of traf�c to paths with high stability when it might not require high
performance (or vice versa). These quantities are de�ned as follows:
Stability: This is essentially a measure of how well behaved a path is. Namely how its
latency varies over time, how throughput varies over time, packet loss ratio, and packet error
ratio.
Performance: This is simply a measure of how much traf�c this link is known to be able to
carry.
NOTE: The above quantities are the sum of a series of weighted contributions by more
fundamental metrics:
Latency milliseconds, [0, 2^16]: How long it takes a packet to travel from one node to
another.
Packet Delay Variance (PDV) milliseconds, [0, 2^16]: How the latency between two nodes
varies over time. This is very similar to the concept of jitter but not exactly.
Throughput Disturbance Coef�cient (TDC) unitless, [0, 1]: How the observed throughput of a
path varies with time. It could be the result of traf�c shaping or congestion.
Packet Loss Ratio (PLR) unitless, [0, 1]: The percentage of packets which are lost.
Packet Error Ratio (PER) unitless, [0, 1]: The percentage of packets which failed
authentication/CRC checks.
Max Throughput bytes per second, [0, 2^64]: Maximum number of bytes observed to have
been sent over a path within one second. This value will slowly decrease over time so that it
acts like a rubber band around a tube.
New protocol verbs
While understanding the following isn’t important for one to use this feature, it’s here in case
anyone is curious:
VERB_ACK: An acknowledgment of receipt of a series of recent packets from another peer.

This is used to calculate relative throughput values and to detect packet loss. Every packet
type with the exception of VERB_ACK and VERB_QOS_MEASUREMENT are counted toward this value.
<[4] 32-bit number of bytes received since last ACK>
Upon receipt of this packet, the local peer will verify that the correct number of bytes were
received by the remote peer. If these values do not agree it could be an indicator of packet
loss. Additionally, the local peer knows the interval of time that has elapsed since the last
received ACK. With this information it can compute a rough estimate of the current
throughput. This is sent at a rate of once every 250 ms .
VERBQOSMEASUREMENT: A packet containing timing measurements useful for estimating

path quality. Composed of a list of pairs for an arbitrary set of recent packets. This is used to
sample for latency and packet delay variance.
<[8] 64-bit packet ID of previously-received packet>
4 de 24 27/11/19 10:51
<[1] 8-bit packet sojourn time>

<...repeat until end of max 1400 byte packet...>
The number of possible records per QoS packet is: (1400 * 8) / 72 = 155 . This packet should
be sent very rarely as it can be somewhat large if the connection is saturated. Future
versions might use a bloom table to probabilistically determine these values in a vastly more
space-ef�cient manner.
Note: The ‘internal packet sojourn time’ is a slight misnomer as it is a measure of the amount
of time between when a packet was received and the egress time of its tracking QoS packet.
This is sent at a rate of once every second .
Enabling Multipath
Communication over multiple paths is enabled by setting multipathMode in local.conf . Restart

ZeroTier and it will automatically detect available physical interfaces and begin allocating across
all paths.
{
"settings": {
"multipathMode": 2
}
}
Where:
0 = ZT_MULTIPATH_NONE : No active multipath. Traf�c is merely sent over the strongest path. This
mode will automatically failover to the next-strongest path in the event that a path goes
down.
1 = ZT_MULTIPATH_RANDOM : Traf�c is randomly distributed among all active paths.
2 = ZT_MULTIPATH_PROPORTIONALLY_BALANCED : Traf�c is allocated across all active paths in
proportion to their observed stability and performance (quality). Will cease sending traf�c
over links that appear to be stale.
Monitoring a multipath link
Using the raw JSON output format of the status command zerotier-cli -j status will yield
something like the following which contains an entry for each peer, aggregate link properties, and
individual path properties. This example shows traf�c split roughly evenly across the paths on
separate physical interfaces.
"settings": {
"multipathMode": 2
}
...
"multipath": {
"9f47c03a5b": {
"aggregateLinkLatency": 25,
"aggregateLinkPacketDelayVariance": 6,
"paths": [
{
"address": "172.116.130.93/43032",
"allocation": 0.512172758579254,
"ifname": "phy0",
"latency": 13.0,
"stability": 0.820895612239838
},
{
"address": "2605:e000:1600:c315:0000:ff0b:eca2:452c/9993",
"allocation": 0.487827271223068,
"ifname": "wlan0",
"latency": 30.0,
"stability": 0.772978186607361
}
]
}
}
If one builds with the ZT_DEBUG=1 �ag, a series of traces will be emitted which detail the current
state of aggregate links to each peer, for instance:
link to peer 9f47c03a5b is composed of (2) physical paths (en0, ipv4, 0.512), (en4, ipv4, 0.487), has packet delay variance (6 ms), mean latency (25 ms)
link to peer 9f47c03a5b is fully redundant
link to peer 9f47c03a5b is no longer redundant
Performance and security considerations
To minimize performance overhead the following decisions were made:
To reduce the speed at which ACK/QoS counters and tables are populated, only packets with
IDs divisible by 16 are included in measurements, this amounts to about 6.25% of all packets.
To reduce computational overhead all non-trivial measurements or those measurements
resulting from statistical analysis are cached and their update frequency is not a function of
user traf�c.
Rate limits are imposed on the number of VERB_ACK and VERB_QOS_MEASUREMENT packets that will
be accepted for processing by a peer.
5 de 24 27/11/19 10:51
2.2. VL2: The Ethernet Virtualization Layer
VL2 is a VXLAN-like network virtualization protocol with SDN management features. It

implements secure VLAN boundaries, multicast, rules, capability based security, and certi�cate
based access control.
VL2 is built atop and carried by VL1, and in so doing it inherits VL1’s encryption and endpoint
authentication and can use VL1 asymmetric keys to sign and verify credentials. VL1 also allows us
to implement VL2 entirely free of concern for underlying physical network topology. Connectivity
and routing ef�ciency issues are VL1 concerns. It’s important to understand that there is no
relationship between VL2 virtual networks and VL1 paths. Much like VLAN multiplexing on a
wired LAN, two nodes that share multiple network memberships in common will still only have
one VL1 path (virtual wire) between them.
2.2.1. Network Identi�ers and Controllers
Each VL2 network (VLAN) is identi�ed by a 64-bit (16 hex digit) ZeroTier network ID that contains
the 40-bit ZeroTier address of the network’s controller and a 24-bit number identifying the
network on the controller.
Network ID: 8056c2e21c123456

| |
| Network number on controller
|
ZeroTier address of controller
When a node joins a network or requests a network con�guration update, it sends a network
con�g query message (via VL1) to the network’s controller. The controller can then use the node’s
VL1 address to look it up on the network and send it the appropriate certi�cates, credentials, and
con�guration information. From the perspective of VL2 virtual networks, VL1 ZeroTier addresses
can be thought of as port numbers on an enormous global-scale virtual switch.
A common misunderstanding is to con�ate network controllers with root servers (planet and
moons). Root servers are connection facilitators that operate at the VL1 level. Network controllers
are con�guration managers and certi�cate authorities that belong to VL2. Generally root servers
don’t join or control virtual networks and network controllers are not root servers, though it is
possible to have a node do both.
Controller Security Considerations
Network controllers serve as certi�cate authorities for ZeroTier virtual networks. As such, their
identity.secret �les should be guarded closely and backed up securely. Compromise of a
controller’s secret key would allow an attacker to issue fraudulent network con�gurations or admit
unauthorized members, while loss of the secret key results in loss of ability to control the network
in any way or issue con�guration updates and effectively renders the network unusable.
It is important that controllers’ system clocks remain relatively accurate (to within 30-60 seconds)
and that they are secure against remote tampering. Many cloud providers provide secure time
sources either directly via the hypervisor or via NTP servers within their networks.
2.2.2. Certi�cates and Other Credentials
All credentials issued by network controllers to member nodes in a given network are signed by
the controller’s secret key to allow all network members to verify them. Credentials have
timestamp �elds populated by the controller, allowing relative comparison without the need to
trust the node’s local system clock.
Credentials are issued only to their owners and are then pushed peer to peer by nodes that wish
to communicate with other nodes on the network. This allows networks to grow to enormous
sizes without requiring nodes to cache large numbers of credentials or to constantly consult the
controller.
Credential Types
Certi�cates of Membership: a certi�cate that a node presents to obtain the right to

communicate on a given network. Certi�cates of membership are accepted if they agree,
meaning that the submitting member’s certi�cate’s timestamp differs from the recipient’s
certi�cate’s timestamp by no more than the recipient certi�cate’s maximum timestamp delta
value. This creates a decentralized moving-window scheme for certi�cate expiration without
requiring node clock synchronization or constant checking with the controller.
Revocations: a revocation instantaneously revokes a given credential by setting a hard
timestamp limit before which it will not be accepted. Revocations are rapidly propagated
peer to peer among members of a network using a rumor mill algorithm, allowing a controller
to revoke a member credential across the entire network even if its connection to some
members is unreliable.
Capabilities: a capability is a bundle of network rules that is signed by the controller and can
be presented to other members of a network to grant the presenter elevated privileges within
the framework of the network’s base rule set. More on this in the section on rules.
Tags: a tag is a key/value pair signed by the controller that is automatically presented by
members to one another and can be matched on in base or capability network rules. Tags
can be used to categorize members by role, department, classi�cation, etc.
Certi�cates of Ownership: these certify that a given network member owns something, such
6 de 24 27/11/19 10:51
as an IP address. These are currently only used to lock down networks against IP address
spoo�ng but could be used in the future to certify ownership of other network-level entities
that can be matched in a �lter.
2.2.3. Multicast, ARP, NDP, and Special Addressing Modes
ZeroTier networks support multicast via a simple publish/subscribe system.
When a node wishes to receive multicasts for a given multicast group, it advertises membership in
this group to other members of the network with which it is communicating and to the network
controller. When a node wishes to send a multicast it both consults its cache of recent
advertisements and periodically solicits additional advertisements.
Broadcast (Ethernet ff:ff:ff:ff:ff:ff) is treated as a multicast group to which all members subscribe. It
can be disabled at the network level to reduce traf�c if it is not needed. IPv4 ARP receives special
handling (see below) and will still work if normal broadcast is disabled.
Multicasts are propagated using simple sender-side replication. This places the full outbound
bandwidth load for multicast on the sender and minimizes multicast latency. Network
con�gurations contain a network-wide multicast limit con�gurable at the network controller. This
speci�es the maximum number of other nodes to which any node will send a multicast. If the
number of known recipients in a given multicast group exceeds the multicast limit, the sender
chooses a random subset.
There is no global limit on multicast recipients, but setting the multicast limit very high on very
large networks could result in signi�cant bandwidth overhead.
Special Handling of IPv4 ARP Broadcasts
IPv4 ARP is built on simple Ethernet broadcast and scales poorly on large or distributed networks.
To improve ARP’s scalability ZeroTier generates a unique multicast group for each IPv4 address
detected on its system and then transparently intercepts ARP queries and sends them only to the
correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6
NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth
consumption. A similar strategy is implemented under the hood by a number of enterprise
switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation
mode is transparent to the OS and application layers, but it does mean that packet sniffers will not
see all ARP queries on a virtual network the way they typically can on smaller wired LANs.
Multicast-Free IPv6 Addressing Modes
IPv6 uses a protocol called NDP in place of ARP. It is similar in role and design but uses narrow
multicast in place of broadcast for superior scalability on large networks. This protocol
nevertheless still imposes the latency of an additional multicast lookup whenever a new address
is contacted. This can add hundreds of milliseconds over a wide area network, or more if latencies
associated with pub/sub recipient lookup are signi�cant.
IPv6 addresses are large enough to easily encode ZeroTier addresses. For faster operation and
better scaling we’ve implemented several special IPv6 addressing modes that allow the local node
to emulate NDP. These are ZeroTier’s rfc4193 and 6plane IPv6 address assignment schemes. If
these addressing schemes are enabled on a network, nodes locally intercept outbound NDP
queries for matching addresses and then locally generate spoofed NDP replies.
Both modes dramatically reduce initial connection latency between network members. 6plane
additionally exploits NDP emulation to transparently assign an entire IPv6 /80 pre�x to every node
without requiring any node to possess additional routing table entries. This is designed for virtual
machine and container hosts that wish to auto-assign IPv6 addresses to guests and is very useful
on microservice architecture backplane networks.
Finally there is a security bene�t to NDP emulation. ZeroTier addresses are cryptographically
authenticated, and since Ethernet MAC addresses on networks are computed from ZeroTier
addresses these are also secure. NDP emulated IPv6 addressing modes are therefore not
vulnerable to NDP reply spoo�ng.
Normal non-NDP-emulated IPv6 addresses (including link-local addresses) can coexist with NDP-
emulated addressing schemes. Any NDP queries that do not match NDP-emulated addresses are
sent via normal multicast.
2.2.4. Ethernet Bridging
ZeroTier emulates a true Ethernet switch. This includes the ability to L2 bridge other Ethernet
networks (wired LAN, WiFi, virtual backplanes, etc.) to virtual networks using conventional
Ethernet bridging.
To act as a bridge a network member must be designated as such by the controller. This is for
security reasons as normal network members are not permitted to send traf�c from any origin
other than their MAC address. Designated bridges also receive special treatment from the
multicast algorithm, which more aggressively and directly queries them for group subscriptions
and replicates all broadcast traf�c and ARP requests to them. As a result bridge nodes experience
a slightly higher amount of multicast bandwidth overhead.
7 de 24 27/11/19 10:51
Bridging has been tested extensively on Linux using the Linux kernel native bridge, which cleanly
handles network MTU mismatch. There are third party reports of bridging working on other
platforms. The details of setting up bridging, including how to selectively block traf�c like DHCP
that may not be wanted across the bridge, are beyond the scope of this manual.
2.2.5. Public Networks
It is possible to disable access control on a ZeroTier network. A public network’s members do not
check certi�cates of membership, and new members to a public network are automatically
marked as authorized by their host controller. It is not possible to de-authorize a member from a
public network.
Rules on the other hand are enforced, so it’s possible to implement a special purpose public
network that only allows access to a few things or that only allows a restricted subset of traf�c.
Public networks are useful for testing and for peer to peer “party lines” for gaming, chat, and other
applications. Participants in public networks are warned to pay special attention to security. If
joining a public network be careful not to expose vulnerable services or accidentally share private
�les via open network shares or HTTP servers. Make sure your operating system, applications,
and services are fully up to date.
ZeroTier, Inc. operates a public network called Earth (no relation to the root server planet de�nition
of the same name) with the network ID 8056c2e21c000001 . Earth issues IPv4 addresses in the
unused IPv4 space 28.0.0.0/7 and rfc4193 IPv6 addresses and allows multicast for service
discovery. It’s essentially a global LAN party.
2.2.6. Ad-Hoc Networks
A special kind of public network called an ad-hoc network may be accessed by joining a network
ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead
their con�guration and other credentials are generated locally. Ad-hoc networks permit only IPv6
UDP and TCP unicast traf�c (no multicast or broadcast) using 6plane format NDP-emulated IPv6
addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN
(connection open) packets are only allowed to desintation ports within the encoded range.
For example ff00160016000000 is an ad-hoc network allowing only SSH, while ff0000ffff000000 is
an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care
must be taken to avoid exposing vulnerable services or sharing unwanted �les or other resources.
2.2.7. Quality of Service (QoS)
In a near future release (not yet implemented as of 1.4.2), ZeroTier will support new traf�c
prioritization rules that build on the link measuring features described in Section 2.1.5. Nine
buckets will be available for traf�c classi�cation: [0, 1, 2, 3, (4), 5, 6, 7, 8] , each
geometrically-increasing in priority where bucket 4 is the default in the absence of a classi�cation
rule for a given type of traf�c.
For instance, say you’d like to prioritize your VoIP traf�c over standard web traf�c:
priority 6
ipprotocol udp
and dport 5060-5065
and sport 5060-5065
;
priority 3
ipprotocol tcp
and dport 80
and sport 80
;
This would place VoIP traf�c on ports 5060 to 5065 at a higher priority 6 than the standard port 80
web traf�c in bucket 3 .
3. The Network Rules Engine

Traf�c on ZeroTier networks can be observed and controlled with a system of globally applied
network rules. These are enforced in a distributed fashion by both the senders and the receivers of
packets. To escape the rules engine a malicious attacker would need to fully compromise both
sides of any conversation.
8 de 24 27/11/19 10:51
The ZeroTier VL2 rules engine differs from most other �rewalls and SDN rules engines in several
ways. The most immediately relevant of these is that the ZeroTier rules engine is stateless,
meaning it lacks connection tracking. This means that bidirectional whitelisting can’t be
accomplished by simply whitelisting reply packets to established connections. Instead some
thought must be put into how to allow both sides of a desired �ow. Rule patterns to achieve the
most common desired objectives are included in this manual.
The decision to make our rules engine stateless was a design trade-off driven by several
concerns. First we wanted to keep complexity, code footprint, and memory use very low to
support small embedded devices. The second and more fundamental reason is that distributed
stateful �ltering requires distributed state synchronization. This would have added a large volume
of additional sync traf�c as well as introducing inescapablenew sources of instability and failure
and a lot of surface area for security vulnerabilities.
While ZeroTier lacks state tracking, its rules engine includes something not found anywhere else
in the enterprise networking space: capability-based security and device tagging. Capabilities and
tags allow extremely complex micro-segmented network rule schemes to be implemented in a
sane, conceptual way that is both easier for human beings to understand and more ef�cient for
machines to handle.
This section assumes some level of familiarity with network rules as they’re commonly used on
�rewalls and routers, etc. While the rules engine is part of VL2, it’s been given its own section in
this manual due to the depth and cross-cutting nature of the topic.
3.1. Rule Sets and Rule Evaluation
Rule sets are ordered lists of one or more rules, with each rule consisting of one or more match
conditions followed by one action. As a rule set is evaluated, each match is tested in order and is
then ANDed or ORed with the previous match result state. When an action is encountered it is
taken if the result of the preceding matches is true. An action with no preceding matches is
always taken. If no permissive actions are taken by any rule set the packet is discarded.
Here is a simple rule set that constrains Ethernet traf�c on a network to only IPv4, ARP, or IPv6 as
it would appear in the raw JSON format used by ZeroTier One’s built-in network controller
implementation. (Don’t worry if this seems verbose and dif�cult. We have a more human-friendly
way of writing rule sets, but before we introduce it it’s important to understand what is really
happening.)
[
{
"etherType": 2048,
"not": true,
"or": false,
"type": "MATCH_ETHERTYPE"
},
{
"etherType": 2054,
"not": true,
"or": false,
},
{
"etherType": 34525,
"not": true,
"or": false,
},
{
"type": "ACTION_DROP"
},
{
"type": "ACTION_ACCEPT"
}
]
This checks whether an Ethernet level packet is not IPv4 (ethertype 2048) and not IPv4 ARP
(ethertype 2054) and not IPv6 (ethertype 34525). If all three matches evaluate to true (meaning
the ethertype is none of these) then the drop action is taken. Otherwise the accept action is taken.
Networks have one base rule set that is applied to all traf�c. Its size is constrained to 1024 entries
(each match or action is an entry). It should be used to set the overall policies for all members of
the network, and for most common use cases it’s all you’ll need. For more complex scenarios, both
capabilities and tags provide methods of both managing complexity and scaling the overall size of
a network’s rule system.
3.1.1. Actions and Match Conditions
These are the available matches and actions in raw form. The rule de�nition language outlined in
section 3.4 provides a friendlier way for human beings to specify rules.
Action Argument(s) Description

Drop packet and terminate all rule
ACTION_DROP
evaluation, including capabilities.
Terminate evaluation of this rule
ACTION_BREAK set but continue evaluating
capabilities.
9 de 24 27/11/19 10:51

Accept packet and terminate
ACTION_ACCEPT
further evaluation.
Send a copy of up to the �rst length
ACTION_TEE length,address bytes (-1 for all) to a ZeroTier
address.
Transparently redirect this packet
ACTION_REDIRECT address
to a ZeroTier address without
changing its headers.
Match Condition Argument(s) Description
MATCH_SOURCE_ZEROTIER_ADDRESS zt VL1 source address.
MATCH_DEST_ZEROTIER_ADDRESS zt VL1 destination address.
MATCH_MAC_SOURCE mac Ethernet source address.
MATCH_MAC_DEST mac Ethernet destination address.
MATCH_IPV4_SOURCE ip IPv4 source address.
MATCH_IPV4_DEST ip IPv4 destination address.
MATCH_IPV6_SOURCE ip IPv6 source address.
MATCH_IPV6_DEST ip IPv6 destination address.
IP TOS �eld bitwise-ANDed with
MATCH_IP_TOS mask,start,end
mask is within range.
MATCH_IP_PROTOCOL ipProtocol IP protocol number.
MATCH_ETHERTYPE etherType Ethernet frame type.
ICMP type and code, if applicable.
MATCH_ICMP icmpType,icmpCode
(V4 or V6)
IP (V4 or V6) source port range
MATCH_IP_SOURCE_PORT_RANGE start,end
(inclusive).
IP (V4 or V6) destination port range
MATCH_IP_DEST_PORT_RANGE start,end
(inclusive).
Bitwise AND characteristic bits
MATCH_CHARACTERISTICS mask
with mask, true if result is nonzero.
Ethernet frame size range
MATCH_FRAME_SIZE_RANGE start,end
(inclusive).
Match if a random 32-bit number is
MATCH_RANDOM probability less than or equal to the
probability.
Difference between tags with this
MATCH_TAGS_DIFFERENCE id,value
ID is less than or equal to the value.
MATCH_TAGS_BITWISE_AND id,value Tags ANDed together equal value.
MATCH_TAGS_BITWISE_OR id,value Tags ORed together equal value.
MATCH_TAGS_BITWISE_XOR id,value Tags XORed together equal value.
MATCH_TAGS_EQUAL id,value Both tags equal the same value.
Sending side’s tag equals this
MATCH_TAG_SENDER id,value
value.
Receiving side’s tag equals this
MATCH_TAG_RECEIVER id,value
value.
3.2. Capabilities
A capability is a small rule set that is bundled into a credential object, signed by the network
controller, and issued to only those member(s) permitted to exercise it. When a member detects
that outgoing traf�c does not match the base rule set but is allowed by one of its capabilities, it
periodically pushes the matching capability credential to the recipient ahead of the packet(s) in
question. Peer to peer capability distribution is automatic and is triggered by capability match.
When the recipient receives the capability it authenticates it by checking its signature and
timestamp and, provided the capability is valid, adds it to the set of capabilities to apply to
incoming traf�c from the capability’s owner. The sender has effectively told the recipient “I can too
send this packet! Teacher says so!”
Capabilities allow large systems of rules to be broken down into functional aspects and then
distributed intelligently only to those members with a need to know. This avoids the bandwidth
and storage overhead of distributing huge monolithic rule sets and organizes rules conceptually to
make them easier for administrators to understand.
There are three terminating actions that can be taken in a rule set: accept, break, and drop. The
accept action terminates rule evaluation and accepts the packet. The break action terminates the
evaluation of the current rule set but permits the further evaluation of capabilities. The drop action
terminates rule evaluation and drops the packet without checking capabilities in the base rule set,
but is equivalent to break in capability rule sets. In most cases break should be used unless certain
traf�c must be absolutely prohibited under any circumstance.
In the simple base rule set example in section 3.1 the drop action is taken in the unapproved case.
This means that ethernet whitelisting cannot be overridden by a capability. If we change
ACTION_DROP in our example to ACTION_BREAK , then it becomes possible to issue the following
capability:
10 de 24 27/11/19 10:51
{
"etherType": 2114,
"not": false,
"or": false,
},
{
}
]
Ethertype 2114 is wake-on-LAN, a special packet that can cause some systems to wake from
sleep mode. If we place the above tiny rule set into a capability and issue it to a device, this device
but no others will now be permitted to send wake-on-LAN magic packets. (Wake-on-LAN
requires hardware support so it would only work to target devices plugged into a physical
network bridged to a ZeroTier network, but don’t worry about that here. It’s just an example of
special traf�c.)
Capability rule sets are limited to only 64 entries. The idea is to keep them small and simple. A
capability should grant one thing or one small set of conceptually related things.
3.3. Tags
ZeroTier provides a second mechanism to control rule set complexity. Tags are 32-bit numeric key-
value pair credentials that are issued to network members and signed by the controller. They are
then distributed peer to peer on a need to know basis in a similar manner to capabilities.
Tags provide a way to conditionally drop or allow traf�c between members by member
classi�cation. They allow very detailed network micro-segmentation by member role, permission,
function, etc. without resulting in a combinatorial explosion in rules table size.
Let’s say we want to permit traf�c on TCP ports 139 and 445 (netbios/CIFS �le sharing) only
between systems that belong to the same department. Our company has 12,000 devices and 10
departments. Without tags this would require 144,000,000 rules, but with tags it can be
accomplished by only a few.
First a tag is created to represent the department. Let’s give it tag ID 100. Each member system
receives the tag with a value from 1 to 10 indicating which department it belongs to. We can then
add the following rules to our network’s base rule set (or to a capability if so desired):
[
{
"type": "MATCH_IP_DEST_PORT_RANGE",
"not": false,
"or": false,
"start": 139,
"end": 139
},
{
"type": "MATCH_IP_DEST_PORT_RANGE",
"not": false,
"or": true,
"start": 445,
"end": 445
},
{
"type": "MATCH_IP_PROTOCOL",
"not": false,
"or": false,
"ipProtocol": 6
},
{
"type": "MATCH_TAGS_DIFFERENCE",
"not": false,
"or": false,
"id": 10,
"value": 0
},
{
}
]
This tells members in our network to accept TCP packets on ports 139 or 445 if the difference
between tags with tag ID 10 is zero, meaning they match. (If a member does not have a value for
this tag, it does not match.) Now all members of the same department can access CIFS �le shares,
but CIFS sharing between departments could still be prohibited. (TCP whitelisting requires some
additional rules due to the stateless nature of our rules engine. See the section below on rule
design patterns.)
Tags can be compared on numeric value or as bit �elds via several different bit mask operations
allowing many different systems of member classi�cation to be implemented.
3.4. Rule De�nition Language
Raw rule sets are verbose and dif�cult to write, so we created a minimal rule de�nition langugage
that’s easier for human beings.
The parser for this language can be found in the rule-compiler subfolder of the ZeroTierOne
11 de 24 27/11/19 10:51
project and as zerotier-rule-compiler on NPM. It’s written in JavaScript and is the same code that
powers the in-browser editor in ZeroTier Central.
3.4.1. An Introductory Example

# Whitelist only IPv4 (/ARP) and IPv6 traffic and allow only ZeroTier-assigned IP addresses
drop # drop cannot be overridden by capabilities
not ethertype ipv4 # frame is not ipv4
and not ethertype arp # AND is not ARP
and not ethertype ipv6 # AND is not ipv6
or not chr ipauth # OR IP addresses are not authenticated (1.2.0+ only!)
;
# Allow SSH, HTTP, and HTTPS by allowing all TCP packets (including SYN/!ACK) to these ports
accept
ipprotocol tcp
and dport 22 or dport 80 or dport 443
;
# Create a tag for which department someone is in

tag department
id 1000 # arbitrary, but must be unique
enum 100 sales # has no meaning to filter, but used in UI to offer a selection
enum 200 engineering
enum 300 support
enum 400 manufacturing
;
# Allow Windows CIFS and netbios between computers in the same department using a tag
accept
ipprotocol tcp
and tdiff department 0 # difference between department tags is 0, meaning they match
and dport 139 or dport 445
;
# Drop TCP SYN,!ACK packets (new connections) not explicitly whitelisted above
break # break can be overridden by a capability
chr tcp_syn # TCP SYN (TCP flags will never match non-TCP packets)
and not chr tcp_ack # AND not TCP ACK
;
# Create a capability called "superuser" that lets its holders override all but the initial "drop"
cap superuser
id 1000 # arbitrary, but must be unique
accept; # allow with no match conditions means allow anything and everything
;
# Accept other packets

accept;
This creates a network that can pass IPv4 (and ARP) and IPv6 traf�c but no other Ethernet frame
types. In addition the not chr ipauth condition drops traf�c between IP addresses that have not
been assigned by ZeroTier to their respective sources or destinations, blocking all IP spoo�ng.
These are enforced with a hard drop , preventing them from being overridden by any capability.
All UDP is allowed, but all non-whitelisted new TCP connections (SYN/!ACK packets) are blocked.
This is done with a soft break , allowing it to be overridden by capabilities. New TCP connections
are allowed on ports 22, 80, and 443 for everyone. Windows �le sharing is allowed only between
computers in the same department by way of a tag. A super-user capability that can be assigned
to administrative nodes allows the sender to initiate any kind of connection.
See section 3.5.1 for a discussion of how we accomplish TCP whitelisting here.
3.4.2. Rule De�nition Language Syntax

# The remainder of this line is a comment.
action [ ... args ... ]

[and|or] [not] match [... args ...]
[ ... additional matches ... ]
;
tag <name>
id <id> # arbitrary unique 32-bit ID
[default <value>] # default tag value to assign to members
[enum <value> <name>] # value can be any 32-bit unsigned integer
[flag <bit> <name>] # bit can be 0 to 31
[ ... additional enums or flags ... ]
;
cap <name>
id <id> # arbitrary unique 32-bit ID
[ ... ]
;
[ ... additional action blocks ... ]
;
macro <name[($var1,...)]>
[ ... ]
;
;
include <name(value,...)>
Each action, tag, capability, or macro block ends with a semicolon. White space separates things.
12 de 24 27/11/19 10:51
Indentation is not signi�cant. Hash symbols indicate that the remainder of a line is a comment.
As described in sections 3.1 through 3.3, a rule set is composed of one or more sequences of
match,[match],…,action in which the action is taken if the chain of matches evaluates to true.
Capabilities are small bundles of rules that can be assigned to nodes to give them special abilities,
and tags are key/value pairs that can be assigned to nodes to allow rules to be selectively applied.
Macros are just a convenient way of de�ning more complex actions that can then be applied in
multiple places. An example of a macro might be:
macro allowtcp($port)
accept
ipprotocol tcp
and dport $port
;
;
You could then allow TCP ports in a standard TCP whitelisting scheme by just saying:
include allowtcp(80)
Tag enum and flag directives have no meaning to the actual ZeroTier rules engine, but they can be
used by user interfaces like ZeroTier Central to make it easier to assign and search tags.
3.4.3. Actions, Matches, Operators, and Constants

Drop packet and terminate all rule evaluation, including
drop
capabilities.
Terminate evaluation of this rule set but continue evaluating
break
capabilities.
accept Accept packet and terminate further evaluation.
Send a copy of up to the �rst length bytes (-1 for all) to a
tee <length> <address>
ZeroTier address.
Transparently redirect this packet to a ZeroTier address
redirect <address>
without changing its headers.
Match
Argument(s) Description
Condition
ztsrc <address> VL1 source address.
ztdest <address> VL1 destination address.
macsrc <address> Ethernet source address.
macdest <address> Ethernet destination address.
ipsrc <address/pre�x> Source IP address or network (V4 or V6 auto-detected).
ipdest <address/pre�x> Destination IP address or network (V4 or V6 auto-detected).
iptos <mask> <start[-end]> IP TOS �eld bitwise-ANDed with mask is within range.
ipprotocol <protocol> IP protocol number.
ethertype <type> Ethernet frame type.
ICMP type (V4 or V6) and code. Use -1 for code if not
icmp <type> <code>
applicable.
sport <start[-end]> IP (V4 or V6) source port range (inclusive).
dport <start[-end]> IP (V4 or V6) destination port range (inclusive).
Bitwise AND characteristic bits with mask, true if result is
chr <mask/name>
nonzero.
framesize <start[-end]> Ethernet frame size range (inclusive).
random <probability> Match randomly with probability 0.0 (0%) to 1.0 (100%).
Difference between tags with this ID is less than or equal to
tdiff <id> <value>
the value.
tand <id> <value> Tags ANDed together equal value.
tor <id> <value> Tags ORed together equal value.
txor <id> <value> Tags XORed together equal value.
teq <id> <value> Both tags equal the same value.
tseq <id> <value> Sending side’s tag equals this value.
treq <id> <value> Receiving side’s tag equals this value.
Match conditions may be joined by and (default if none speci�ed) or or and may be modi�ed by
not.
For convenience the following symbols can be used when matching on certain packet attributes:
IP protocols: icmp (for IPv4), igmp, ipip, tcp, egp, igp, udp, rdp, esp, ah, icmp6 (for IPv6), l2tp,
sctp, and udplite.
Ethernet frame types: ipv4, arp, ipv6, wol (wake on LAN), rarp, atalk, aarp, ipxa, ipxb.
Packet characteristics (bit masks for chr):
inbound: packet is being �ltered on the receiving side (use not inbound for sending side)
multicast: destination is a multicast or broadcast MAC
broadcast: destination is the broadcast MAC
13 de 24 27/11/19 10:51
ipauth: sender IP is assigned by ZeroTier to the sending node

tcp_�n: packet is TCP with FIN �ag set
tcp_syn: packet is TCP with SYN �ag set
tcp_rst: packet is TCP with RST �ag set
tcp_psh: packet is TCP with PSH �ag set
tcp_ack: packet is TCP with ACK �ag set
tcp_urg: packet is TCP with URG �ag set
tcp_ece: packet is TCP with ECE �ag set
tcp_cwr: packet is TCP with CWR �ag set
tcp_ns: packet is TCP with NS �ag set
tcp_rs2: packet is TCP with RS2 (reserved bit 2) �ag set
3.5. Useful Design Patterns
3.5.1. TCP Whitelisting
First, add this at or near the bottom of your rules:
# Block TCP SYN,!ACK to prevent new non-whitelisted TCP connections from being initiated
# unless previously whitelisted or allowed by a capability.
break chr tcp_syn and not tcp_ack;
Then above the SYN,!ACK break (or in a capability) add rules to allow TCP packets with permitted
destination ports:
# Allow TCP port 80 (HTTP)

accept ipprotocol tcp and dport 80;
Section 3.4.1 shows a complete example.
ZeroTier’s �lter is stateless. If we block all TCP packets except those with the correct destination
port, this will prevent reply packets from returning to their senders.
Allowing reply packets with the correct (inverse) source port seems like a simple �x, but this is
insecure as it allows anyone to initiate any TCP connection if they bind to whitelisted source ports.
The best solution is to block all non-whitelisted TCP packets with �ags SYN,!ACK (SYN and not
ACK) and and then whitelist desired destination ports. This way TCP replies and control traf�c are
allowed, but new connections cannot be opened unless they are permitted.
3.5.2. Locking Down UDP
UDP is tougher to deal with in a stateless paradigm. It’s connectionless so there is no way to
speci�cally select a new session vs. an existing session. The best way to lock down UDP on a
network is to use tags to allow it to and from things like DNS servers that need to speak it.
tag udpserver
id 1000
default 0
;
accept
ipprotocol udp
and tor udpserver 1
or chr multicast
;
break ipprotocol udp;
First we de�ne a tag called udpserver with a default value of 0. We don’t set any enums or �ags
for this tag since it will be used as a boolean. For servers that need to respond to DNS queries, set
the udpserver to 1 .
Then we accept UDP traf�c if the value of the udpserver tag is 1 when both sender and receiver
tags are ORed together, or if UDP traf�c is multicast. This allows multicast mDNS and Netbios
announcements and allows UDP traf�c to and from UDP servers, but prohibits other horizontal
UDP traf�c.
3.5.3. Traf�c Observation and Interception
Here’s a simple rule to monitor everything:
# Send a copy of EVERY packet on both sender and receiver side to ZeroTier address "deadbeef11".
tee -1 deadbeef11;
That’s going to �ood deadbeef11 with two full copies of every single packet, since it will match on
both the sender and the recipient side. A less bandwidth-intensive security monitor setup might
look like this:
tee 128 deadbeef11

chr inbound
and chr tcp_syn or chr tcp_rst or chr tcp_fin
14 de 24 27/11/19 10:51
or random 0.1
;
tee 128 deadbeef22

not chr inbound
and chr tcp_syn or chr tcp_rst or chr tcp_fin
or random 0.1
;
This is a bit more clever. It sends the �rst 128 bytes of every TCP SYN, RST, or FIN packet (TCP
connection open and close) to one observer on the inbound side and another observer on the
outbound side. It also sends the �rst 128 bytes of other packets with a probability of one in ten.
The �rst 128 bytes of a packet will be enough to see the Ethernet and IP headers as well as layer
4+ information about many protocols.
This would allow observers to watch every new TCP connection on the network and also
passively monitor other traf�c in a “fuzzy” probabilistic fashion without using very much
bandwidth. We split sending and receiving observers to prevent duplicate packets and also to
allow us to detect cases where one side is failing to tee traf�c more frequently than would be
easily explained by packet loss.
There are many, many variations on the above that are possible but hopefully these will be
enough to get you started.
Passive out-of-line monitoring with tee is not perfect. If bandwidth between two parties exceeds
the bandwidth of one or both parties to the observer, packets will get dropped. It also provides no
mechanism to �lter traf�c in depth since the observer cannot directly intervene (though it could
de-authorize a member via the controller API).
Active in-line monitoring can be achieved with redirect:
redirect deadbeef11
ipprotocol tcp
and dport 80 or sport 80
;
This will pipe all HTTP traf�c through deadbeef11 . The target node will receive each Ethernet frame
intact including its original Ethernet source and destination MAC address. To forward the packet
on to its �nal destination, simply re-send it unmodi�ed via the same interface on which it was
received. The intermediate node could scan, modify, and interrupt HTTP traf�c across the network.
To regular network participants this is completely invisible as it occurs at layer 2 (Ethernet),
though some performance degradation may be noticed especially if the link is running across
WAN.
3.5.4. The Classi�ed System Pattern
One useful pattern for access control resembles the way military organizations classify data.
Information is deemed classi�ed, and only those who have the required level of classi�cation are
allowed to access it.
A model resembling this can be implemented in ZeroTier network rules as follows:
# Is this member classified?

tag classified
id 2
enum 0 no
enum 1 secret
enum 2 top
default no
;
# Clearance flags (a bit like groups)

tag clearance
id 1
default 0
flag 0 staging
flag 1 production
flag 2 financial
flag 3 security
flag 4 executive
;
# If one party is classified, require at least one overlapping clearance bit

break
not tor classified 0
and tand clearance 0
;
Initially members will be assigned a default classi�cation of 0 (“no”). These can freely
communicate since the bitwise OR of their classi�ed tags will be zero. Neither member possesses
a classi�cation requirement.
To restrict access to a member, set its classi�ed tag to secret or top. (In this example there is no
difference, but two levels are included in case you want to implement some kind of more detailed
segmentation based on these.) Now the �rst match ( not tor classified 0 ) will be true and the
packet will be dropped unless the two communicating members have at least once clearance bit
in common ( tand clearance 0 ).
15 de 24 27/11/19 10:51
4. ZeroTier One
ZeroTier One is a service that can run on laptops, desktops, servers, virtual machines, and
containers to provide virtual network connectivity through a virtual network port much like a VPN
client. It can also act as a network controller and as a federated root server.
Binary packages are available on the ZeroTier site and source code is found on GitHub.
After the service is installed and started, networks can be joined using their 16-digit network IDs.
Each network appears as a virtual “tap” network port on your system that behaves just like an
ordinary Ethernet port.
The ZeroTier One service keeps its con�guration and state information in its working directory. It’s
found by default at the following locations:
Windows: C:\ProgramData\ZeroTier\One
Macintosh: /Library/Application Support/ZeroTier/One
Linux: /var/lib/zerotier-one
FreeBSD/OpenBSD: /var/db/zerotier-one
4.1. JSON API
Macintosh and Windows versions of ZeroTier One come with a graphical UI that runs as a tray /
task bar app, and all three versions include a command line interface called zerotier-cli . These
interfaces control the service using a local JSON API available via http on ZeroTier’s primary port
(9993 by default) and authenticated using a token stored as authtoken.secret in the service’s
working directory.
The API may be accessed directly via HTTP GET and POST requests that include the X-ZT1-Auth
header whose value must be the contents of authtoken.secret . Most objects and �elds are read-
only. Networks and moons can be added by POSTing and removed by DELETEing. Bold �elds are
read/write (rw) or write-only (ro) and can be modi�ed by POSTing a JSON object. Types are strict
in POSTs, e.g. an integer �eld must be 1 and not "1" .
GET /status
address string 10-digit (40 bit) ZeroTier address of this node
clock integer Current system clock at time of this request
cluster object|null Cluster status if clustering is enabled (usually null)
con�g object Contents of local.conf con�guration �le (see section 4.2)
online boolean True if node can communicate with at least one root
World ID of current planet (always 149604618 except in
planetWorldId integer
testing scenarios)
planetWorldTimestamp integer Timestamp of current planet
publicIdentity string Public identity of this node (address and public key)
tcpFallbackActive boolean If true, node is tunneling through a ZeroTier TCP relay (slow)
version string ZeroTier One version
versionBuild integer Build version
versionMajor integer Major version number
versionMinor integer Minor version number
versionRev integer Revision portion of version number
GET /peer
[ … array of peers … ]
GET /peer/##########
address string 10-digit ZeroTier address of peer
latency integer Last measured latency in milliseconds
role string LEAF for normal nodes, MOON or PLANET for roots
version string Version of node, if known (learned via direct paths)
versionMajor integer Node major version
versionMinor integer Node minor version
versionRev integer Node revision
paths [object] Array of path objects
address string Physical address of other end of path
active boolean Is path currently considered active?
expired boolean Is path expired?
preferred boolean Is path preferred?
lastReceive integer Timestamp of last packet received
lastSend integer Timestamp of last packet sent
linkQuality �oat Link quality, 0.0 to 1.0
trustedPathId integer Trusted path ID or 0 if not a trusted path
GET /network
[ … array of networks … ]
GET,POST,DELETE /network/################
id string 16-digit network ID
nwid string 16-digit network ID (deprecated)
16 de 24 27/11/19 10:51
GET /status
mac string Ethernet MAC address of this node on this network
mtu integer Port MTU
portDeviceName string Operating system device name or null if not applicable
portError integer OS-speci�c error code if a port error occurred or 0 if no error
broadcastEnabled boolean True if Ethernet broadcast (ff:ff:ff:ff:ff:ff) is enabled
bridge boolean True if this port can be bridged (L2 bridging)
dhcp boolean Should DHCP be used? (currently unused, always false)
name string Short name of network as con�gured at controller
status string OK, NOT_FOUND, ACCESS_DENIED, or PORT_ERROR
type string Network authentication mode: PUBLIC or PRIVATE
allowManaged (rw) boolean If true, ZT-managed IPs and routes are assigned
If true, ZT-managed IPs and routes can overlap public IP
allowGlobal (rw) boolean
space
allowDefault (rw) boolean If true, network can override system default route (full tunnel)
assignedAddresses [string] Array of ZT-managed IP assignments (IP/bits)
routes [object] Array of ZT-managed routes
target string Network (IP/bits) of target for this route
via string|null Next hop or null if this is a local (same LAN) network
�ags integer Currently unused, always 0
metric integer Route metric, currently unused, always 0
GET /moon
[ … array of moons … ]
GET,POST,DELETE /moon/################
id string 16-digit hex moon ID
timestamp integer Most recent timestamp of moon’s world de�nition
signature string Hex signature
updatesMustBeSignedBy string Public key that must be used to sign the next world update
waiting boolean If true, we are waiting for the seed to respond
10-digit ZT address of any member of moon, sent in initial
seed (ro) string
POST
roots [object] Array of root de�nitions for this moon’s world
identity string Identity of root (address and public key)
stableEndpoints [string] Array of root’s physical addresses
4.2. Local Con�guration Options
A �le called local.conf in the ZeroTier home folder contains con�guration options that apply to the
local node. It can be used to set up trusted paths, blacklist physical paths, set up physical path
hints for certain nodes, and de�ne trusted upstream devices (federated roots). In a large
deployment it can be deployed using a tool like Puppet, Chef, SaltStack, etc. to set a uniform
con�guration across systems. It’s a JSON format �le that can also be edited and rewritten by
ZeroTier One itself, so ensure that proper JSON formatting is used.
Settings available in local.conf (this is not valid JSON, and JSON does not allow comments):
{
"physical": { /* Settings that apply to physical L2/L3 network paths. */
"NETWORK/bits": { /* Network e.g. 10.0.0.0/24 or fd00::/32 */
"blacklist": true|false, /* If true, blacklist this path for all ZeroTier traffic */
"trustedPathId": 0|!0 /* If present and nonzero, define this as a trusted path (see below) */
} /* ,... additional networks */
},
"virtual": { /* Settings applied to ZeroTier virtual network devices (VL1) */
"##########": { /* 10-digit ZeroTier address */
"try": [ "IP/port"/*,...*/ ], /* Hints on where to reach this peer if no upstreams/roots are online */
"blacklist": [ "NETWORK/bits"/*,...*/ ] /* Blacklist a physical path for only this peer. */
}
},
"settings": { /* Other global settings */
"primaryPort": 0-65535, /* If set, override default port of 9993 and any command line port */
"portMappingEnabled": true|false, /* If true (the default), try to use uPnP or NAT-PMP to map ports */
"softwareUpdate": "apply"|"download"|"disable", /* Automatically apply updates, just download, or disable built-in software updates */
"softwareUpdateChannel": "release"|"beta", /* Software update channel */
"softwareUpdateDist": true|false, /* If true, distribute software updates (only really useful to ZeroTier, Inc. itself, default is false) */
"interfacePrefixBlacklist": [ "XXX",... ], /* Array of interface name prefixes (e.g. eth for eth#) to blacklist for ZT traffic */
"allowManagementFrom": "NETWORK/bits"|null, /* If non-NULL, allow JSON/HTTP management from this IP network. Default is 127.0.0.1 only. */
"allowTcpFallbackRelay": true|false /* Allow or disallow establishment of TCP relay connections (true by default) */
}
}
trustedPathId: A trusted path is a physical network over which encryption and

authentication are not required. This provides a performance boost but sacri�ces all
ZeroTier’s security features when communicating over this path. Only use this feature if you
know what you are doing and really need the performance! To set up a trusted path, all
devices on the same trusted physical network must have the same trusted path ID. Trusted
path IDs are arbitrary unsigned 64-bit integers. These are not secrets. The security of a
trusted path depends on its physical con�guration. Take special care that any �rewalls at its
boundaries do not allow traf�c in our out with IPs overlapping the trusted network range.
An example local.conf :
17 de 24 27/11/19 10:51
{
"physical": {
"10.0.0.0/24": {
"blacklist": true
},
"10.10.10.0/24": {
"trustedPathId": 101010024
},
},
"virtual": {
"feedbeef12": {
"role": "UPSTREAM",
"try": [ "10.10.20.1/9993" ],
"blacklist": [ "192.168.0.0/24" ]
}
},
"settings": {
"softwareUpdate": "apply",
"softwraeUpdateChannel": "release"
}
}
4.3. Network Controller Con�guration
See the network controller microservice docs on GitHub. By default the controller microservice is
included in all ZeroTier One builds for Windows, Macintosh, Linux, and BSD (as of version 1.2.0).
4.4. Creating Your Own Roots (a.k.a. Moons)
As of version 1.2.2 this feature is still considered somewhat experimental. Please let us know if
you experience problems. We’ll remove this notice when it’s been in the wild for a while.
As described in section 2.1.1 all ZeroTier nodes on a planet effectively inhabit a single data center.
This makes it easy to directly connect devices anywhere, but it has the disadvantage of not
working without an Internet connection. Network connections are far from perfectly reliable, and
sometimes for security reasons a user may wish to “air gap” a set of nodes from the rest of the
Internet entirely.
In 1.2.0 we introduced the ability to add your own user-de�ned roots. Since the data center we
inhabit is the planet, a user-de�ned set of roots is called a moon. When a node “orbits” a moon, it
adds the moon’s roots to its root server set. Nodes orbiting moons will still use planetary roots, but
they’ll use the moon’s roots if they look faster or if nothing else is available.
The �rst step in creating a moon is to deploy a set of root servers. In most cases we recommend
two. These are regular ZeroTier nodes, but ones that are always on and have static (physical) IP
addresses. These static IPs could be global Internet IPs or physical intranet IPs that are only
reachable internally. In the latter case your moon’s roots won’t work outside your of�ce, but that
doesn’t matter. Roaming nodes will just use planetary roots instead.
We recommend that root servers do not act as network controllers, join networks, or perform any
other overlapping functions. They need good reliable network connections but otherwise require
very little RAM, storage, or CPU. A root could be a small VM, VPS, or cloud instance, or a small
device like a Raspberry Pi. If you provision your roots as VMs, take care that they do not all reside
on the same physical hardware. This would defeat the purpose of having two.
The next step is to create a world de�nition using zerotier-idtool . You will need the
identity.public �les from each of your root servers. Pick one root (doesn’t matter which) and run
zerotier-idtool initmoon <identity.public of one root> >>moon.json . The zerotier-idtool command
will output a JSON version of your world de�nition to stdout, so we redirect it to moon.json .
Examine this �le. It will contain something like:
{
"id": "deadbeef00",
"objtype": "world",
"roots": [
{
"identity": "deadbeef00:0:34031483094...",
"stableEndpoints": []
}
],
"signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
"signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
"updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
"worldType": "moon"
}
The world ID is technically arbitrary and could be any random 64-bit value. By convention we use
the address of one of the roots.
The world de�nition JSON �le contains secrets, so it’s important to keep it in a safe place. The
signingKey_SECRET is what will allow you to update your world de�nition automatically in the future.
Now add your other root(s) and de�ne their stable endpoints. You’ll end up with something that
looks like:
{
"id": "deadbeef00",
18 de 24 27/11/19 10:51
"objtype": "world",
"roots": [
{
"identity": "deadbeef00:0:34031483094...",
"stableEndpoints": [ "10.0.0.2/9993","2001:abcd:abcd::1/9993" ]
},
{
"identity": "feedbeef11:0:83588158384...",
"stableEndpoints": [ "10.0.0.3/9993","2001:abcd:abcd::3/9993" ]
}
],
"signingKey": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
"signingKey_SECRET": "ffc5dd0b2baf1c9b220d1c9cb39633f9e2151cf350a6d0e67c913f8952bafaf3671d2226388e1406e7670dc645851bf7d3643da701fd4599fedb9914c3918db3",
"updatesMustBeSignedBy": "b324d84cec708d1b51d5ac03e75afba501a12e2124705ec34a614bf8f9b2c800f44d9824ad3ab2e3da1ac52ecb39ac052ce3f54e58d8944b52632eb6d671d0e0",
"worldType": "moon"
}
The static IP addresses you use must be reachable from all the places you want these roots to
serve. If you’re deploying these for use at a physical location, use internal IPs. If you want them to
be usable off-site, use public IPs from your DMZ or host them at a cloud provider with a data
center presence close to you. Low-cost cloud hosts that provide simple static direct IP addressing
and dual-stack IPv4/IPv6 support like Digital Ocean, Vultr, and Linode make ideal places to host
roots. The lowest priced instances at these providers are more than suf�cient in most cases.
The third step is to generate the actual signed world with zerotier-idtool genmoon moon.json . In this
case this will generate a �le called 000000deadbeef00.moon . This does not contain secret keys but is
signed by the secret from the JSON �le.
Now go to your roots, create (if it does not exist) a subdirectory of their working directories
(usually /var/lib/zerotier-one on Linux) called moons.d , and copy the signed moon �le there. Now
restart the roots and they should be ready.
You can add these roots to regular nodes in one of two ways: by placing the same world de�nition
�le in their moons.d directories or by using the zerotier-cli orbit command: zerotier-cli orbit
deadbeef00 deadbeef00 . The �rst argument is the world ID (which we can shorten by removing the
two leading zeroes) and the second is the address of any of its roots. This will contact the root and
obtain the full world de�nition from it if it’s online and reachable.
Once you’ve “orbited” your moon, try zerotier-cli listpeers . You should see the roots you’ve
created listed as MOON instead of LEAF . They will now be used as alternative root servers.
5. SDK – Embedding ZeroTier Into an Application

The ZeroTier SDK can be used in two very different ways: libztcore which is the platform-
agnostic network virtualization engine, and libzt which is the engine paired with a lightweight
userspace network stack. libzt is a superset of libztcore and its usage is distinguished by the
fact that it exposes a standard socket API and simple network/service control API. The ZeroTier
source code is open source and is licensed under the GNU GPL v3 (not LGPL). If you’d like to
embed it in a closed-source commercial product or appliance, please e-mail contact@zerotier.com
to discuss commercial licensing. Otherwise it can be used for free. Before we dive into the
technicals, the �rst thing to understand are the differences between the two APIs as each is
intended for a very different purpose:
libzt is intended for convenience and simplicity:

socket API: zts_socket(), zts_connect(), zts_bind(), ...
control API: zts_start(), zts_join(), zts_leave(), ....
libztcore is intended for raw performance. If your goal is simply moving frames as quickly as
possible and you’re willing to put in some extra work, this is what you’re looking for. The API
is described in include/ZeroTierOne.h. For an example of how this API is used, see the living
documentation that is service/OneService.cpp.
core API: ZT_VirtualNetworkFrameFunction(), ZT_WirePacketSendFunction(), ...
5.1 Socket API (libzt)
The libzt project combines our network virtualization engine with a lightweight user-space TCP/IP
stack and a Posix-compliant network API. The result can be built into an application as a library,
allowing it to access virtual networks without elevated permissions or special OS access to create
tun/tap ports.
5.1.1 Starting the Service
The next few sections explain how to use the control API. These functions are non-blocking and
will return an error code speci�ed in include/ZeroTierConstants.h and will result in the generation of
callback events. It is your responsibility to handle these events.
To start the service, simply call:
zts_start(char *path, void (*userCallbackFunc)(struct zts_callback_msg*), int port)
At this stage, if a cryptographic identity for this node does not already exist, it will generate a new
one and store it on disk, the node’s address (commonly referred to as nodeId ) will be derived from
this identity and will be presented to you upon receiving the ZTS_EVENT_NODE_ONLINE shown below.
19 de 24 27/11/19 10:51
NOTE: The �rst argument path is a path where you will allow ZeroTier to store its automatically-
generated cryptographic identity �les ( identity.public and identity.secret ), these �les are your
keys to communicating on the network. Keep them safe and keep them unique. If any two nodes
are online using the same identities you will have a bad time. The second argument
userCallbackFunc is a function that you specify to handle all generated events for the life of your
program (see below):
void myZeroTierEventCallback(struct zts_callback_msg *msg)

{
if (msg->eventCode == ZTS_EVENT_NODE_ONLINE) {
printf("ZTS_EVENT_NODE_ONLINE, nodeId=%llx\n", msg->node->address);
// You can join networks now!
}
// ...
}
After calling zts_start() you will receive one or more of the following events:
ZTS_EVENT_NODE_OFFLINE
ZTS_EVENT_NODE_ONLINE
ZTS_EVENT_NODE_DOWN
ZTS_EVENT_NODE_IDENTITY_COLLISION
ZTS_EVENT_NODE_UNRECOVERABLE_ERROR
ZTS_EVENT_NODE_NORMAL_TERMINATION
After receiving ZTS_EVENT_NODE_ONLINE you will be allowed to join or leave networks.
At the end of your program or when no more network activity is anticipated, the user application
can shut down the service with zts_stop() . However, it is safe to leave the service running in the
background inde�nitely as it doesn’t consume much memory or CPU while at idle. zts_stop() is a
non-blocking call and will itself issue a series of events indicating that various aspects of the
ZeroTier service have successfully shut down.
It is worth noting that while zts_stop() will stop the service, but the user-space network stack will
continue operating in a headless hibernation mode. This is intended behavior due to the fact that
the network stack we’ve chosen doesn’t currently support the notion of shutdown since it was
initially designed for embedded applications that are simply switched off. If you do need a way to
shut everything down and free all resources you can call zts_free() , but please note that calling
this function will prevent all subsequent zts_start() calls from succeeding and will require a full
application restart if you want to run the service again. The events ZTS_EVENT_NODE_ONLINE and
ZTS_EVENT_NODE_OFFLINE can be seen periodically throughout the lifetime of your application
depending on the reliability of your underlying network link, these events are lagging indicators
and are typically only triggered every thirty (30) seconds.
Lastly, the function zts_restart() is provided as a way to restart the ZeroTier service along with all
of its virtual interfaces. The network stack will remain online and undisturbed during this call. Note
that this call will temporarily block until the service has fully shut down, then will return and you
may then watch for the appropriate startup callbacks mentioned above.
5.1.2 Joining a Network
Joining a ZeroTier virtual network is as easy as calling zts_join(uint64_t networkId) . Similarly there
is a zts_leave(uint64_t networkId) . Note that zts_start() must be called and a
ZTS_EVENT_NODE_ONLINE event must be received before these calls will succeed. After calling
zts_join() any one of the following events may be generated:
ZTS_EVENT_NETWORK_NOT_FOUND
ZTS_EVENT_NETWORK_CLIENT_TOO_OLD
ZTS_EVENT_NETWORK_REQUESTING_CONFIG
ZTS_EVENT_NETWORK_OK
ZTS_EVENT_NETWORK_ACCESS_DENIED
ZTS_EVENT_NETWORK_READY_IP4
ZTS_EVENT_NETWORK_DOWN
ZTS_EVENT_NETWORK_READY_IP4 and ZTS_EVENT_NETWORK_READY_IP6 are combinations of a few different

events. They signal that the network was found, joined successfully, an IP address was assigned
and the network stack’s interface is ready to process traf�c of the indicated type. After this point
you should be able to communicate with peers on the network.
5.1.3 Communicating with peers
After successfully starting the service and joining a network, communicating with other nodes
(peers) on that network is as easy as it would ordinarily be without ZeroTier. However, one thing
to be aware of is the difference between relay and P2P modes. In the event that a direct
connection cannot be established between your nodes, ZeroTier offers a free relaying service, this
means that your nodes are reachable almost instantaneously but at a temporary performance
cost. One should wait to send large amounts of traf�c until a ZTS_EVENT_PEER_P2P is received for the
node that you’re interested in talking to. This event usually only takes a few seconds to appear
after data has initially been sent. Similarly if after some time ZeroTier determines that a previously
known path to one of your nodes is no longer available you will see a ZTS_EVENT_PEER_RELAY event.
One can use zts_get_peer_status(uint64_t peerId) to query the current reachability state of
another node. This function will actually return the previously mentioned event values, plus an
additional one called ZTS_EVENT_PEER_UNREACHABLE if no known direct path exists between the calling
20 de 24 27/11/19 10:51
node and the remote node.
5.1.4 Handling events
As mentioned in previous sections, the control API works by use of non-blocking calls and the
generation of a few dozen different event types. Depending on the type of event there may be
additional contextual information attached to the zts_callback_msg object that you can use. This
contextual information will be housed in one of the following structures which are de�ned in
include/ZeroTier.h :
struct zts_callback_msg
{
int eventCode;
struct zts_node_details *node;
struct zts_network_details *network;
struct zts_netif_details *netif;
struct zts_virtual_network_route *route;
struct zts_physical_path *path;
struct zts_peer_details *peer;
struct zts_addr_details *addr;
};
Here’s an example of a callback function:

{
if (msg->eventCode == ZTS_EVENT_NODE_ONLINE) {
printf("ZTS_EVENT_NODE_ONLINE, node=%llx\n", msg->node->address);
// You can join networks now!
}
}
In this callback function you can perform additional non-blocking API calls or other work. While
not returning control to the service isn’t forbidden (the event messages are generated by a
separate thread) it is recommended that you return control as soon as possible as not returning
will prevent the user application from receiving additional callback event messages which may be
time-sensitive.
A typical ordering of messages may look like the following:
ZTS_EVENT_NETIF_UP --- network=a09acf023be465c1, mac=73b7abcfc207, mtu=10000

ZTS_EVENT_ADDR_NEW_IP4 --- addr=11.7.7.184 (on network=a09acf023be465c1)
ZTS_EVENT_ADDR_NEW_IP6 --- addr=fda0:9acf:233:e4b0:7099:9309:4c9b:c3c7 (on network=a09acf023be465c1)
ZTS_EVENT_NODE_ONLINE, node=c4c7ba3cf
ZTS_EVENT_NETWORK_READY_IP4 --- network=a09acf023be465c1
ZTS_EVENT_NETWORK_READY_IP6 --- network=a09acf023be465c1
ZTS_EVENT_PEER_P2P --- node=74d0f5e89d
ZTS_EVENT_PEER_P2P --- node=9d219039f3
ZTS_EVENT_PEER_P2P --- node=a09acf0233
5.1.4.1 Node Events
These events pertain to the state of the current node. This message type will arrive with a
zts_node_details object accessible via msg->node . Additionally, one can query the status of the node
with zts_get_node_status() , this will return the status as an integer value only.
ZTS_EVENT_NODE_OFFLINE
ZTS_EVENT_NODE_ONLINE
ZTS_EVENT_NODE_DOWN
ZTS_EVENT_NODE_IDENTITY_COLLISION
ZTS_EVENT_NODE_UNRECOVERABLE_ERROR
ZTS_EVENT_NODE_NORMAL_TERMINATION
5.1.4.2 Network Events
These events pertain to the state of the indicated network. This event type will arrive with a
zts_network_details object accessible via msg->network . If for example you want to know the
number of assigned routes for your network you can use msg->network->num_routes . Similarly for the
MTU, use msg->network->mtu . Additionally, one can query the status of the network with
zts_get_network_status(uint64_t networkId) , this will return the status as an integer value only.
ZTS_EVENT_NETWORK_NOT_FOUND
ZTS_EVENT_NETWORK_CLIENT_TOO_OLD
ZTS_EVENT_NETWORK_REQUESTING_CONFIG
ZTS_EVENT_NETWORK_OK
ZTS_EVENT_NETWORK_ACCESS_DENIED
ZTS_EVENT_NETWORK_DOWN
5.1.4.3 Peer Events
These events are triggered when the reachability status of a peer has changed, this can happen
at any time. This event type will arrive with a zts_peer_details object for additional context.
Additionally, one can query the status of the network with zts_get_peer_status(uint64_t peerId) ,
this will return the status as an integer value only.
ZTS_EVENT_PEER_P2P
ZTS_EVENT_PEER_RELAY
21 de 24 27/11/19 10:51
ZTS_EVENT_PEER_UNREACHABLE
5.1.4.4 Path Events
These events are triggered when a direct path to a peer has been discovered or is now considered
too old to be used. You will see these in conjunction with peer events. This event type will arrive
with a zts_physical_path object for additional context.
ZTS_EVENT_PATH_DISCOVERED
ZTS_EVENT_PATH_ALIVE
ZTS_EVENT_PATH_DEAD
5.1.4.5 Route Events
This event type will arrive with a zts_virtual_network_route object for additional context.
ZTS_EVENT_ROUTE_ADDED
ZTS_EVENT_ROUTE_REMOVED
5.1.4.6 Address Events
These events are triggered when new addresses are assigned to the node on a particular virtual
network. This event type will arrive with a zts_addr_details object for additional context.
ZTS_EVENT_ADDR_ADDED_IP4
ZTS_EVENT_ADDR_REMOVED_IP4
ZTS_EVENT_ADDR_ADDED_IP6
ZTS_EVENT_ADDR_REMOVED_IP6
5.1.4.7 Network Stack Events (debugging)
These events aren’t very important to the application developer but are important for debugging.
These signal whether the userspace networking stack was brought up successfully. You can
ignore these in most cases. This event type will arrive with no additional contextual information.
ZTS_EVENT_STACK_UP
ZTS_EVENT_STACK_DOWN
5.1.4.8 Netif Events (debugging)
These events aren’t very important to the application developer but are important for debugging.
These signal whether the userspace networking stack was brought up successfully. You can
ignore these in most cases. This event type will arrive with a zts_netif_details object for
additional context.
ZTS_EVENT_NETIF_UP
ZTS_EVENT_NETIF_DOWN
ZTS_EVENT_NETIF_REMOVED
ZTS_EVENT_NETIF_LINK_UP
ZTS_EVENT_NETIF_LINK_DOWN
5.1.5 Errors
Just as there are two APIs (socket and control), there are two sets of error codes. The control API
( zts_start() , zts_join() , etc) errors de�ned in include/ZeroTierConstants.h are:
ZTS_ERR_OK :Everything is ok
ZTS_ERR_INVALID_ARG : An argument provided by the user application is invalid (e.g. out of
range, NULL, etc)
ZTS_ERR_SERVICE : The service isn’t initialized or is for some reason currently unavailable. Try
again.
ZTS_ERR_INVALID_OP : For some reason this API operation is not permitted or doesn’t make
sense at this time.
ZTS_ERR_NO_RESULT : The call succeeded, but no object or relevant result was available
ZTS_ERR_GENERAL : General internal failure (memory allocation, null reference, etc)
The socket API error codes are de�ned in doc/errno.h
NOTE: For Android/Java (or similar) which use JNI, the socket API’s error codes are negative values
NOTE: For protocol-level errors such as dropped packets or internal network stack errors, see the
section Statistics
5.1.6 Thread Model
The control API for libzt is thread safe and can be called at any time from any thread. There is a
single internal lock guarding access to this API. The socket API is similar in this regard. Callback
events are generated by a separate thread and are independent from the rest of the API’s internal
locking mechanism. Not returning from a callback event won’t impact the rest of the API but it will
prevent your application from receiving future events so it is in your application’s best interest to
perform as little work as possible in the callback function and promptly return control back to
ZeroTier.
22 de 24 27/11/19 10:51
Note: Internally, libzt will spawn a number of threads for various purposes: a thread for the core
service, a thread for the network stack, a low priority thread to process callback events, and a
thread for each network joined. However, the vast majority of work is performed by the core
service and stack threads.
5.1.7 Statistics
Protocol and service statistics are available in debug builds of libzt . These statistics are detailed
fully in the section of include/ZeroTier.h that is guarded by LWIP_STATS . The protocol constants are
de�ned in include/ZeroTierConstants.h . An example usage is as follows:
C++ example:
struct zts_stats_proto stats;
// Get count of received pings

if (zts_get_protocol_stats(ZTS_STATS_PROTOCOL_ICMP, &stats) == ZTS_ERR_OK) {
printf("icmp.recv=%d\n", stats.recv);
}
// Get count of dropped TCP packets

if (zts_get_protocol_stats(ZTS_STATS_PROTOCOL_TCP, &stats) == ZTS_ERR_OK) {
printf("tcp.drop=%d\n", stats.drop);
}
5.1.8 Network Controller Mode
The library form of ZeroTier can act as a network controller and in libzt this is controlled via the
zts_controller_* API calls speci�ed in include/ZeroTier.h . Currently controller mode is not available
in the iOS and macOS framework builds.
5.2 Frame-based API (libztcore)

For more information on the libztcore API see include/ZeroTierOne.h
5.3 Downloads
SDK bundles for common architectures and platforms can be found here: http://zerotier.com
/download.shtml
5.4 Examples
5.4.1 C Example
#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include "ZeroTier.h"
bool node_ready = false;

bool network_ready = false;

{
switch (msg->eventCode)
{
case ZTS_EVENT_NODE_ONLINE:
printf("ZTS_EVENT_NODE_ONLINE, nodeId=%llx\n", msg->node->address);
node_ready = true;
break;
case ZTS_EVENT_NODE_OFFLINE:
printf("ZTS_EVENT_NODE_OFFLINE\n");
node_ready = false;
break;
case ZTS_EVENT_NETWORK_READY_IP4:
printf("ZTS_EVENT_NETWORK_READY_IP4, networkId=%llx\n", msg->network->nwid);
network_ready = true;
break;
case ZTS_EVENT_PEER_P2P:
printf("ZTS_EVENT_PEER_P2P, nodeId=%llx\n", msg->peer->address);
break;
case ZTS_EVENT_PEER_RELAY:
printf("ZTS_EVENT_PEER_RELAY, nodeId=%llx\n", msg->peer->address);
break;
default:
break;
}
}
int main()
{
char *str = "welcome to the machine";
char *remoteIp = "11.7.7.223";
int remotePort = 8082;
int fd, err = 0;
struct zts_sockaddr_in addr;
addr.sin_family = ZTS_AF_INET;
addr.sin_addr.s_addr = inet_addr(remoteIp);
addr.sin_port = htons(remotePort);
23 de 24 27/11/19 10:51
// Set up ZeroTier service and wait for callbacks

int port = 9994;
int nwid = 0x0123456789abcdef;
zts_start("test/path", &myZeroTierEventCallback, port);
printf("Waiting for node to come online...\n");
while (!node_ready) { sleep(1); }
zts_join(nwid);
printf("Joined virtual network. Requesting configuration...\n");
while (!network_ready) { sleep(1); }
// Socket API example

if ((fd = zts_socket(AF_INET, SOCK_STREAM, 0)) < 0) {
printf("error creating socket\n");
}
if ((err = zts_connect(fd, (const struct sockaddr *)&addr, sizeof(addr))) < 0) {
printf("error connecting to remote host\n");
}
if ((err = zts_write(fd, str, strlen(str))) < 0) {
printf("error writing to socket\n");
}
zts_close(fd);
zts_stop();
return 0;
}
5.3.2 Java Example
Starting ZeroTier:
MyZeroTierEventListener listener = new MyZeroTierEventListener();

ZeroTier.start(getApplicationContext().getFilesDir() + "/zerotier", listener, myPort);
// Wait for EVENT_NODE_ONLINE
while (listener.isOnline == false) {
try {
Thread.sleep(interval);
} catch (Exception e) { }
}
ZeroTier.join(myNetworkId);
// Wait for EVENT_NETWORK_READY_IP4/6
while (listener.isNetworkReady == false) {
try {
Thread.sleep(interval);
} catch (Exception e) { }
}
// Now you can use the socket API!
An example event listener:
import com.zerotier.libzt.ZeroTier;
import com.zerotier.libzt.ZeroTierEventListener;
import com.zerotier.libzt.ZeroTierPeerDetails;
public class MyZeroTierEventListener implements ZeroTierEventListener

{
public void onZeroTierEvent(long id, int eventCode)
{
if (eventCode == ZeroTier.EVENT_NODE_ONLINE) {
System.out.println("EVENT_NODE_ONLINE: nodeId=" + Long.toHexString(ZeroTier.get_node_id()));
isOnline = true;
}
if (eventCode == ZeroTier.EVENT_NODE_OFFLINE) {
System.out.println("EVENT_NODE_OFFLINE");
}
if (eventCode == ZeroTier.EVENT_NETWORK_READY_IP4) {
System.out.println("ZTS_EVENT_NETWORK_READY_IP4: nwid=" + Long.toHexString(id));
if (id == myNetworkId) {
isNetworkReady = true;
}
}
if (eventCode == ZeroTier.EVENT_PEER_P2P) {
System.out.println("EVENT_PEER_P2P: id=" + Long.toHexString(id));
}
if (eventCode == ZeroTier.EVENT_PEER_RELAY) {
System.out.println("EVENT_PEER_RELAY: id=" + Long.toHexString(id));
}
// ...
}
}
Connect Users
News Download
Github Login
Twitter Manual
E-Mail Knowledge Base
Support
Privacy
24 de 24 27/11/19 10:51

Zero Tier

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Zero Tier

Încărcat de

Drepturi de autor:

Formate disponibile

Manual – ZeroTier https://zerotier.

Global Area Networking

2. Network Hypervisor Overview

2.1. VL1: The ZeroTier Peer to Peer Network

A global data center requires a global wire closet.

2.1.1. Network Topology and Peer Discovery

Peer to peer connection setup goes like this:

Asymmetric public key encryption is Curve25519/Ed25519, a 256-bit elliptic curve variant.

2.1.4. Trusted Paths for Fast Local SDN

Notion of link quality

New protocol verbs

VERB_ACK: An acknowledgment of receipt of a series of recent packets from another peer.

VERBQOSMEASUREMENT: A packet containing timing measurements useful for estimating

<[1] 8-bit packet sojourn time>

Communication over multiple paths is enabled by setting multipathMode in local.conf . Restart

Monitoring a multipath link

Performance and security considerations

To minimize performance overhead the following decisions were made:

2.2. VL2: The Ethernet Virtualization Layer

VL2 is a VXLAN-like network virtualization protocol with SDN management features. It

2.2.1. Network Identi�ers and Controllers

Network ID: 8056c2e21c123456

Controller Security Considerations

2.2.2. Certi�cates and Other Credentials

Certi�cates of Membership: a certi�cate that a node presents to obtain the right to

2.2.3. Multicast, ARP, NDP, and Special Addressing Modes

ZeroTier networks support multicast via a simple publish/subscribe system.

Special Handling of IPv4 ARP Broadcasts

Multicast-Free IPv6 Addressing Modes

2.2.4. Ethernet Bridging

2.2.5. Public Networks

2.2.6. Ad-Hoc Networks

2.2.7. Quality of Service (QoS)

3. The Network Rules Engine

3.1. Rule Sets and Rule Evaluation

3.1.1. Actions and Match Conditions

Action Argument(s) Description

Action Argument(s) Description

3.4. Rule De�nition Language

3.4.1. An Introductory Example

# Create a tag for which department someone is in

# Accept other packets

3.4.2. Rule De�nition Language Syntax

action [ ... args ... ]

3.4.3. Actions, Matches, Operators, and Constants

Action Argument(s) Description

ipauth: sender IP is assigned by ZeroTier to the sending node

3.5. Useful Design Patterns

3.5.1. TCP Whitelisting

First, add this at or near the bottom of your rules:

# Allow TCP port 80 (HTTP)

Section 3.4.1 shows a complete example.

3.5.2. Locking Down UDP

break ipprotocol udp;

3.5.3. Traf�c Observation and Interception

Here’s a simple rule to monitor everything:

tee 128 deadbeef11

tee 128 deadbeef22

Active in-line monitoring can be achieved with redirect:

3.5.4. The Classi�ed System Pattern

A model resembling this can be implemented in ZeroTier network rules as follows:

# Is this member classified?

# Clearance flags (a bit like groups)

# If one party is classified, require at least one overlapping clearance bit

zts_start(char path, void (userCallbackFunc)(struct zts_callback_msg*), int port)