Tunnel Broker System Using IPv4 Anycast

Xin Liu
Department of Electronic Engineering Tsinghua Univ.

Xing Li
Department of Electronic Engineering Tsinghua Univ.

lx@ns.6test.edu.cn

xing@cernet.edu.cn

ABSTRACT
Tunnel Broker model, as dened in RFC3053, is a solution to the automation of IPv6 address assignment and IPv6 over IPv4 tunnel management. Tunnel server selection and fault tolerance are two big problems for a Tunnel Broker system. Anycast makes it possible to solve the tunnel server selection problem and improve the systems fault tolerance. This paper proposes an architecture of the Tunnel Broker system using IPv4 anycast. In the proposed architecture a user could hook up to the IPv6 network via the nearest tunnel server, and some failures in the Tunnel Broker system could be recovered in some circumstances.

are located near each other, in a LAN for instance, but they dont work well when the Tunnel Broker system covers a large area and the tunnel servers are distributed distantly. Fault tolerance is another problem in addition to tunnel server selection. Now that there are several tunnel servers in a Tunnel Broker system, it is natural to expect that if one fails, the others would be able to take over its users. However, this is extremely dicult to achieve when we have the assumption that the users have no special software installed for the Tunnel Broker system, because in this situation there is no way to change a users tunnel parameters when his upstream tunnel server fails. This assumption is essential because we couldnt write a client software for every operating system that supports IPv6. IPv4 anycast gives us a way to solve the tunnel server selection problem and improve the systems fault tolerance. If the tunnel servers are congured with an anycast IPv4 address and each user tunnels to this address, the users IPv6 trac will go through the nearest tunnel server, and if one tunnel server fails no changes have to be made on the users side when other tunnel servers take over them. This paper proposes an architecture of the Tunnel Broker system that makes use of IPv4 anycast. The design principles are: 1. No special software should be required on the users side. Following this principle would guarantee the usefulness of the system it could accommodate every operating system that supports IPv6. 2. Each user could attach to the tunnel server nearest to it. 3. Topology adjustments of the tunnel servers do not require changes on the user routers. This is essential for the systems fault tolerance as long as the rst principle applies. 4. IPv6 trac to and from the user IPv6 networks goes inside the Tunnel Broker system as much as possible. This promises some kind of QoS because the whole Tunnel Broker system is supposed to be under full control and measures could be taken to adjust and improve its transmission behaviors. The proposed architecture works best with IPv4 pseudo-

Keywords
IPv6, anycast, tunnel broker

1.

INTRODUCTION

IP version 6 ( IPv6 ) is the next generation Internet Protocol that is the successor of the widely used IP version 4 ( IPv4 ) in the current Internet. However, IPv6 could not intercooperate with IPv4. In the deployment of IPv6, it is a well adopted practice that IPv6 networks are interconnected via IPv6 over IPv4 ( IPv6/IPv4 ) tunnels. Tunnel Broker [1] is one of the solutions to automate the management tasks of assigning IPv6 addresses and maintaining IPv6/IPv4 tunnels. In the Tunnel Broker model there is a component called Tunnel Server. Tunnel servers act as access points tunnels to the users are setup on them. In a typical Tunnel Broker system there will be one tunnel broker and several tunnel servers. It is a problem to optimally select a tunnel server for a particular user to tunnel with. To the best knowledge of the authors the available solutions are quite simple. The implementation made by CERNET IPv6 Testbed [9] chooses the tunnel server randomly. The [5] makes the choice on the basis of a simple number-of-tunnels balancing criteria, namely always selecting the tunnel server with the fewest tunnels. These solutions are ecient when the tunnel servers

anycast as categorized in [6]. This kind of anycast is implemented by assigning a single unicast address to many hosts located in dierent places and announcing the address from these places in the routing system simultaneously. This way when an IP datagram is sent to the unicast address it will be routed to the nearest host that is assigned the address, and the unicast address becomes an anycast one. This anycast address could be used the same as a unicast address, for example appearing as the source address of IP datagrams, because the routing system is expected to be relatively stable and the IP datagrams to the address wont be delivered randomly to more than one of the hosts assigned the address. If the routing system is constantly changing, or the true anycast [3, 8] is implemented, the systems behavior will decline dramatically because the packet loss ratio is likely to exceed some tenths.

down, other tunnel servers in the same group will try to take over all of its attached users. 4. DNS System ( DNS ) The DNS system is responsible for providing mappings from IPv6 addresses to domain names. Typically each user will get a domain name that will resolve to one of his IPv6 addresses. Some other policies not specied in [1] but adopted in the proposed architecture are listed below. A user has to register rst to use the Tunnel Broker system. Registration makes access control relatively easy. A user gets a permanent IPv6 address block when registering. Each user could only have a single IPv6 address block and a single IPv6/IPv4 tunnel. Communication between the users and the tunnel broker is via HTTP. A user could interact with the tunnel broker with a WWW browser. Other ways of communication may be implemented to ease the users operation.

2. 2.1

ARCHITECTURE Overview

This section presents the proposed architecture.

The architecture is demonstrated in Figure 1. It conforms to the one dened in [1]. Each part of the system is explained below. 1. Tunnel Broker ( TB ) Tunnel Broker is the organizer of the Tunnel Broker system. It interacts with the users and sends instructions to the DNS system and the tunnel servers. It monitors the tunnel servers so that if a tunnel server is down it could have other tunnel servers in the same tunnel server group take over the failed tunnel servers attached users. 2. Tunnel Server ( TS ) Tunnel Server is the point where the users get connected into the IPv6 network. Tunnels are setup between the tunnel servers and the users. Typically each tunnel server will have at least three addresses: an IPv6 unicast address for IPv6 routing purpose, an IPv4 unicast address for IPv4 accessibility, and an IPv4 anycast address as the endpoint for the user tunnels. For each of the attached users, the tunnel server will have a tunnel to it with the IPv4 anycast address set as the tunnels local IPv4 address and a routing entry specifying that the users IPv6 network is reachable through the tunnel. Each tunnel server monitors its attached users so that if a user is not reachable it will report to the tunnel server. 3. Tunnel Server Group ( TSG ) Tunnel servers are divided into tunnel server groups by IPv4 anycast addresses all the tunnel servers sharing a single IPv4 anycast address forms a tunnel server group. Whenever a user sends a tunnel activation request to the tunnel broker, it is associated with a tunnel server group and nally it would be attached to the nearest tunnel server in the group. If a user could not reach his upstream tunnel server some time after the tunnel is established, other tunnel servers in the group will try to take over him. If a tunnel server is

2.2

Tunnel Server Group Selection

On receiving a users tunnel activation request, the tunnel broker has to select an associated tunnel server group for the user. In the proposed architecture the decision is made based on the users IPv6 address block. This is for simplicity of both the selection process and the construction of the underlying routing system. How the routing system is constructed will be explained later.

2.3

IPv4 Anycast

It is a problem to properly use IPv4 anycast in the Tunnel Broker system. In [7] it is said that IPv4 anycast address could be used as tunnel endpoint IPv4 address. An IPv6/IPv4 tunnel has at least two parameters, that is, the two IPv4 addresses of its two ends. If the two addresses are both unicast addresses, it is an ordinary tunnel and it is usually regarded as a point-to-point link. If one of the addresses is an anycast address, it seems that the tunnel becomes an one-to-many link. This might be true if congured as depicted in Figure 2. The user has a tunnel from 1.1.1.1 to 2.2.2.2, and each tunnel server in TSG1 has a tunnel from 2.2.2.2 to 1.1.1.1. This way IPv6 datagrams from the user could go out through any of the tunnel servers in TSG1, and if any tunnel server in TSG1 gets an IPv6 datagram whose destination address is in the users IPv6 network it just forwards the datagram to the user. When a tunnel server in TSG1 is down, TS1 for instance, everything will go smooth if adjustment is taken on the routing system so that IPv4 datagrams sent to 2.2.2.2 would no longer reach TS1 and IPv6 datagrams sent to the users IPv6 network would not be passed to TS1. Robustness is achieved this way. However, this way of using IPv4 anycast is not adopted in the proposed architecture. There are several reasons.

Figure 1: Architecture It is unscalable. In a tunnel server group, each tunnel server has to keep tunnels and routing table entries to all the users that are associated with the group. It is OK if the tunnel servers are there for redundancy purpose only, because in this situation there are supposed to be only a small number of tunnel servers in a group and they are located near each other, typically in a LAN. But if the tunnel servers are widely spread out in a large area and are supposed to serve many users, there will be a large number of tunnel servers in a group and each is meant to serve only a small number of users that are near it. In this case it is intolerable and absolutely unnecessary to congure each tunnel server to serve all the users. Moreover, the user information synchronization between the tunnel servers are quite dicult if there are many tunnel servers. The quality of the IPv6 datagram transmission service is more dicult to control. IPv6 datagrams sent out from a user will be optimally delivered to the nearest tunnel server and then into the IPv6 network due to the use of IPv4 anycast. However, IPv6 datagrams sent to a user will probably not take the optimal way when a tunnel server receives an IPv6 datagram to a user, it just delivers it to the user through the tunnel regardless of its distance to the user. The IPv4 network between a user and a tunnel server is supposed to be completely out of the control of the Tunnel Broker system, and nothing could be done by the system to improve the IPv6 datagram transmission service. High IPv4 connectivity requirement. The IPv4 connectivity between the user and each tunnel server in a group is required, but this could not be guaranteed in some circumstances due to IPv4 packet lters set on the intermediate routers. Even if the connectivity is excellent between the user and its nearest tunnel server, there will still be packet loss if some other tunnel server could not reach the user, because IPv6 datagrams may be forwarded to the user by that tunnel server. How the IPv4 anycast is used in the proposed architecture is depicted in Figure 3. A tunnel from 1.1.1.1 to 2.2.2.2 is congured on the users side, but only the tunnel server nearest to the user will have a tunnel setup from 2.2.2.2 to 1.1.1.1 TS3 in case of Figure 3. Other tunnel servers in the group will not know the existence of the user, and because they dont have tunnels to the user, theyll even drop the IPv4 datagrams that come from the user with encapsulated IPv6 datagrams. This is why the proposed architecture works best with IPv4 pseudo-anycast. This way of using IPv4 anycast has several advantages. Scalability. Each tunnel server will only care for the users attached to it. One does not even have to know all the other tunnel servers in the same group. Fault tolerance is not lost because measures will be taken to alter the attachment relationship if something goes wrong. Some QoS could be obtained. IPv6 datagrams sent to a user will rst enter the Tunnel Broker system through any tunnel server, then be transferred to the tunnel server that is nearest to the user, and nally be delivered to the user via the tunnel. The IPv6 datagram transmission within the Tunnel Broker system is usually under full control of the organization that runs the Tunnel Broker system and measures could be taken to provide better transmission service. It also has some disadvantages. As mentioned earlier, it only works well with IPv4 pseudo-anycast. Moreover, since a tunnel server is only aware of the users attached to it, extra help is required in order to make its attached users to hook

Figure 2: One-to-Many Tunnel

entry in its routing table for each of its attached users. Since the users are dynamically attached, the corresponding routing entries will also be dynamic. A dynamic routing software is required to have the tunnel servers within a group exchanging user reachability information. The implementation software of the Tunnel Broker system on a tunnel server will only manipulate its own routing entries; the routing information exchange task is left for standard routing software such as zebra. In theory any routing protocol could do the work, but for simplicity and ease of control it is recommended that BGP is used and all the tunnel servers in a group are either partially or fully meshed with IPv6/IPv4 tunnels. This way the intermediate routers between the tunnel servers could be hidden away from the system and thus do not need manipulation.

2.5

Failure Detection and Recovery

Failure recovery is very important for the systems fault tolerance. In the proposed architecture two kinds of failure could be recovered in some circumstances: Tunnel server failure. A tunnel server is down. User failure. A users IPv6 network is not reachable. The reason might be the users router is down, or the IPv4 connectivity is broken between the user and its upstream tunnel server. Because once a tunnel is setup the conguration on the users side could no longer be changed by the Tunnel Broker system, these failures could only be recoverable if they are reected on the routing system. For instance, if a tunnel server is down but the IPv4 routing system takes no changes, its attached users will still send IPv6 datagrams to it by its IPv4 anycast address, and the failure is unrecoverable. Another example is that if a user could no longer reach its upstream tunnel server due to some IPv4 lters on the intermediate routers, it is unrecoverable because this failure could not result in routing system changes and nothing could be done by the Tunnel Broker system. So it is best that a Tunnel Broker system could interact with the underlying IPv4 routing system. However, this paper does not discuss such interaction because it is too complex and tends to be implementation specic. Some failures are recoverable even if there is no interaction between the Tunnel Broker system and the underlying IPv4 routing system such as when a users upstream tunnel server is no longer the nearest one due to IPv4 routing changes. The failures could be detected by using some kind of keep alive protocol. In the proposed architecture, the tunnel broker monitors all the tunnel servers and each tunnel server monitors its attached users. The monitoring is done in an echo mechanism. Software running on each tunnel server listens on a specic IPv4 UDP port and provides an echo service. The tunnel broker periodically sends echo requests to each tunnel server, and it will get echo replies if the tunnel servers are alive.

Figure 3: One-to-One Tunnel up to other tunnel servers when something is wrong such as a user is no longer reachable by the tunnel server due to routing system changes. In the proposed architecture, whenever something goes wrong it is the tunnel broker that chooses another tunnel server for the aected users. This is designed for both simplicity and security. This lowers the scalability of the system, but since the routing system and the tunnel servers are supposed to be stable, such kind of failure should be seldom and its eect will be insignicant.

2.4

IPv6 Routing

In the IPv6 routing system the Tunnel Broker system could be regarded as a relatively independent entity that acts as the default gateway for all of its users, but IPv6 routing within the Tunnel Broker system is also a problem. It is the tunnel servers that are concerned with IPv6 routing; the tunnel broker and the DNS system has nothing to do with it. The routing architecture of the tunnel servers could be constructed as described below. 1. Each tunnel server group is statically congured to take charge of a list of IPv6 address blocks. This is feasible because as mentioned earlier a users associated tunnel server group is decided according to his IPv6 address block and this actually binds each user to a specic tunnel server group. Only static routing is required between the tunnel server groups and the external IPv6 routers and the impact to the external routing system will be kept minimum. 2. Within a tunnel server group dynamic routing is required. As described before, a tunnel server keeps an

 Each users router should turn on its ICMPv6 echo service. A tunnel server periodically sends ICMPv6 echo requests to each attached user, and it will get ICMPv6 echo replies if the users are alive. Active polling is adopted for scalability and ease of control. The users could not be monitored passively because there is supposed to be no special software installed on the users side. The tunnel broker could not monitor by passively receiving reports of aliveness from the tunnel servers because it could hardly control the interval between two successive reports from a tunnel server and it will be exhausted by the reports if there are many tunnel servers.

5. The tunnel broker records the users current status and then sends the tunnels information back to the user, including the tunnel server groups IPv4 anycast address. It may also give the user instructions on how to setup an IPv6/IPv4 tunnel on the users side. 6. The user setups a tunnel on his router and makes his upstream tunnel server his IPv6 default gateway. The tunnels local IPv4 address should be the IPv4 address oered to the tunnel broker, and its remote IPv4 address should be the tunnel server groups IPv4 anycast address. Because the tunnel server has to monitor the users status, the users router has to be congured an IPv6 address that is known to the tunnel server. It is recommended that the routers tunnel interface is congured the rst IPv6 address in the users IPv6 address block. This way the user does not need to inform the tunnel server of its IPv6 address, and the tunnel server does not need to assign extra IPv6 addresses to the user for the monitoring task.

3. 3.1

PROCEDURES Registration and Unregistration

Procedures taken in various conditions are described below.

These procedures are trivial. When registering, a user sends a registration request to the tunnel broker with information such as his name, email address and the expected size of the IPv6 address block to be allocated to him. The tunnel broker assigns an IPv6 address block to the user, adds a DNS resource entry for him, and returns the information to him with his password. When unregistering, a user just sends an unregistration request to the tunnel broker with his name and password, and the tunnel broker takes back the IPv6 address block assigned to him, deletes everything that is related to him in the Tunnel Broker system, and returns a good-bye message.

3.3

Tunnel Deactivation

This procedure is relatively simple. It has 4 steps: 1. The tunnel deactivation is triggered either by the users tunnel deactivation request or by some other events such as described in Step 2 of the tunnel activation procedure. 2. The tunnel broker informs the users associated tunnel server of his removal. The tunnel server deletes the users tunnel and the corresponding routing entry. The routing table change should be noticed by the routing software running on the tunnel server and get announced into the routing system. 3. The tunnel broker records the users current status. If the tunnel deactivation is triggered by the users request, the tunnel broker then sends a message to the user conrming the tunnels removal. 4. If the tunnel deactivation is triggered by the users request, the user will receive a reply message from the tunnel broker and then clean up the conguration on his side.

3.2

Tunnel Activation

There are 6 steps in the procedure. 1. A user sends a tunnel activation request to the tunnel broker. His IPv4 address may be provided explicitly or obtained from the communication channel such as a TCP connection. 2. The tunnel broker checks the users current status. If the users tunnel is already active and the parameters are the same as those in the users current request, the tunnel broker just goes to Step 5; otherwise the users tunnel is deactivated. 3. The tunnel broker determines the users associated tunnel server group according to his IPv6 address block, and then invokes the tunnel server selection procedure to choose his upstream tunnel server in the group. 4. The tunnel broker informs the selected tunnel server that a new user has come. The tunnel server then setups a tunnel for the user and adds an entry in its IPv6 routing table saying that IPv6 datagrams sent to the users IPv6 address block should go through the newly created tunnel. The tunnels local IPv4 address should be the groups IPv4 anycast address and its remote IPv4 address should be the users IPv4 address. The routing table change should be noticed by the standard routing software running on the tunnel server and announced into the routing system.

3.4

Failure Recovery

When a user failure occurs: 1. The tunnel server deletes the users tunnel tunnel and the corresponding routing entry, and then sends a report to the tunnel broker. 2. The tunnel broker performs Step 3 of the tunnel activation procedure again to select a new upstream tunnel server for the user. If the selection fails, or the selected tunnel server is the users original upstream tunnel server, the tunnel broker modies the users recorded status as if a tunnel deactivation procedure has been performed because the user is unreachable either in IPv4 or in IPv6.

When a tunnel server failure occurs, for each of its attached users the tunnel broker will perform Step 3 of the tunnel activation procedure to select a new upstream tunnel server. If the tunnel broker gets a new upstream tunnel server for a user, it will perform Step 4 of the tunnel activation procedure to attach the user and then update the users status information. Those users who can not get new upstream tunnel servers will be marked as temporarily unreachable and tried periodically later. A tunnel sever may be temporarily down and then up again after some time. When it recovers, it has to inform the tunnel broker of its existence. The tunnel broker will then inform it of the removal of all the users that used to be attached to it but are now attached to other tunnel servers and all of its attached users that are marked as temporarily unreachable will be marked normal again.

Step 7 is there to ensure there is no problem on the IPv4 connectivity between the user and its upstream tunnel server. If the user could reach a tunnel server but the tunnel server could not reach the user, the tunnel server will not be selected as the users upstream tunnel server. If the IPv4 routing system is not stable and the IPv4 datagrams sent to the IPv4 anycast address are delivered to some of the tunnel servers by chance, the reported tunnel server could hardly be the same as the one originating the ICMPv4 echo requests.

4.

CONCLUSION

3.5

Tunnel Server Selection

This is a key procedure in the proposed architecture. It is invoked by the tunnel broker to nd a tunnel server nearest to a user in a tunnel server group. It goes like this: 1. The tunnel broker chooses a tunnel server randomly in the group. 2. The tunnel broker sends a tunnel server selection request to the tunnel server. 3. The tunnel server sends a number of ICMPv4 echo requests to the user with the groups IPv4 anycast address as the source address. These ICMPv4 echo requests should contain the tunnel servers identication such as its IPv4 unicast address. 4. Upon receiving ICMPv4 echo requests, the user will send ICMPv4 echo replies back to the IPv4 anycast address. These replies will contain the information included in the requests according to the ICMPv4 specication. They will be possibly delivered to many actual tunnel servers, but the one nearest to the user should get the most. 5. On receiving such an ICMPv4 echo reply a tunnel server checks the originating tunnel servers identication contained in it. If the originator is not itself, the tunnel server sends a report to the originator; otherwise it just records the reception of the reply. 6. Until now the tunnel server sending out ICMPv4 echo requests has got to know all the receivers of the triggered echo replies. It reports to the tunnel broker that who has received the most replies. 7. The tunnel broker then chooses the reported tunnel server and starts from Step 2 again. The procedure from Step 2 to Step 6 could be performed at most 3 times. Whenever the reported tunnel server is the one to which the tunnel server selection request is sent, it is chosen as the users upstream tunnel server and the whole tunnel server selection procedure is terminated. If no such a tunnel server could be obtained, the selection fails.

In this paper we have proposed an architecture of a Tunnel Broker system using IPv4 anycast. It fully conforms to the model proposed in [1]. We have presented the design principles, the system components and the key procedures. We have shown that by adopting this architecture the users could hook up to the IPv6 network via its nearest tunnel server and some failures could be recovered in some circumstances. The implementation of the proposed architecture has been under construction at the due date of this paper. The rst version is expected to be nished at the beginning of August 2003. It will be deployed on CERNET IPv6 Testbed, and the source codes are expected to be released in GPL or BSD license and put onto Sourceforge.

5.

REFERENCES

[1] A. Durand, P. Fasano, I. Guardini, D. Lento, IPv6 Tunnel Broker ( RFC3053 ), 2001 [2] Bill Fenner, David Meyer, Multicast Source Discovery Protocol (MSDP), ftp://ftp.ietf.org/internetdrafts/draft-ietf-msdp-spec-19.txt, 2003 [3] C. Partridge, T. Mendez, W. Milliken, Host Anycasting Service ( RFC1546 ), 1993 [4] D. Kim, D. Meyer, H. Kilmer, D. Farinacci, Anycast Rendevous Point (RP) mechanism using Protocol Independent Multicast (PIM) and Multicast Source Discovery Protocol (MSDP) ( RFC3446 ), 2003 [5] Domenico Lento, Ivano Guardini, Paolo Fasano, ipv6tb: an implementation of the IPv6 Tunnel Broker, http://carmen.ipv6.tilab.com/ipv6/tools/ipv6tb/, 2002 [6] Jun-ichiro itojun Hagino, K. Ettikan, An analysis of IPv6 anycast, ftp://ftp.ietf.org/internet-drafts/draftietf-ipngwg-ipv6-anycast-analysis-01.txt, 2002 [7] R. Gilligan, E. Nordmark, Transition Mechanisms for IPv6 Hosts and Routers ( RFC2893 ), 2000 [8] R. Hinden, S. Deering, Internet Protocol Version 6 (IPv6) Addressing Architecture ( RFC3513 ), 2003 [9] Xin Liu, Cheng Yan, Xing Li, IPv6/IPv4 Tunnel Broker Implementation, CERNET2000, 2000