Tethered Linux CPE for IP Service Delivery

Motivations, Model and Implementation Considerations

Fernando Snchez David Brazewell

Principal Systems Engineer Head of Access and Service Development
PLUMgrid, Inc. Sky Plc.
Sunnyvale, CA, USA London, United Kingdom
fernando@plumgrid.com david.brazewell@bskyb.com

Abstract This paper presents a new delivery model for IP

communication services based on using lightweight Customer
Premises Equipment (CPE) that is remotely controlled by
virtualized software images running on a provider cloud.
By removing the complex pieces of the service logic from the
equipment in the customer premises, and by placing them in a
virtualized software instance running inside the Service
Providers datacenter, relevant Operational and Capital
Expenditure (CAPEX and OPEX) savings can be obtained.
CAPEX savings are achieved by reducing the complexity of
the piece of hardware to be deployed at the customer premises,
or by extending the life cycle of existing devices already deployed.
Potential OPEX savings are even larger: the complex logic of
the service implementation (the software complexity) is under
more manageable control of the service provider, thus enabling
simpler troubleshooting and software upgrades, and reducing
technical support calls and truck rolls.
These devices are standalone, relatively complex,
intelligent and somewhat in-expensive black boxes that
perform networking functions such as routing, switching,
firewalling and local IP Address Management (IPAM,
generally in the form of DHCP server), amongst several others.
As opposed to mobile terminals, these fixed service
terminals are generally not perceived by the end customer as
highly valued pieces of the service, but rather as service
enablers. The customer usually values the quality of the
network service itself (the VPN connection, the IPTV service,
the internet connectivity), while the CPE is the box in the
corner with blinking lights.
Fig. 1. IP service delivery model with an integrated stack CPE

Hardware development is simplified as the service complexity

is implemented in software with a continuous development
model, thus reducing the time to market for new services whilst
maintaining consistency of service regardless of hardware in the
home.

This model is enabled by the latest advances in the

networking stack of the Linux kernel (which allow to implement
network functions as programs running in kernel space), coupled
with the use of Software-Defined Networking (SDN) and
Network Functions Virtualization (NFV). The combination of
these technologies allows controlling the broadband service in the
cloud, while maintaining local, on-premises service enforcement,
security and address management.
KeywordsCPE, tethered, SDN, Linux, BPF
I. THE IP CPE PROBLEM LOW PERCEIVED VALUE, HIGH
COMPLEXITY.
Most fixed line services offered by Communication
Service Providers (CSPs) to both the residential and business
market segments are transported on top of the IP protocol.
These are generally implemented through the installation of
Customer Premises Equipment (CPE) devices such as home
gateways, cable modems, Digital Subscriber Line (xDSL)
routers or Passive Optical Network (xPON) Network
Termination Units (NTUs). (Fig.1)
A. CPEs are a key enabler for IP services
However, the CPE is a key piece of the service and often
overlooked in its importance. It is the only real bit of the
network the customer sees, it also is the presence of the service
in the home, and yet often its derided (turn it off and turn it
on again!) when it should be front and center (or functionally
discrete if preferred), fully operational, working whenever a
customer needs it.
It is the also the ideal place to run certain services as it is
the gateway to the many things the customer needs the
broadband service for. It should be used for their protection
and security, it is the portal into the home for other Over-the-

Sponsors Alexei Starovoitov Distinguished Engineer at PLUMgrid

Inc., Linux Plumber, and key contributor to the BPF subsystem in the Linux
kernel; Pere Moncls CTO at PLUMgrid Inc.
top (OTT) delivered services and many operators demand it be
service aware to ensure best delivery of those things they
provide to the customer.
Additionally, as more complex IP services are introduced
(e.g. Corporate Security, Parental Controls or IPTV), more
demands are made of this legacy hardware and in many cases
this demand may be beyond the devices capability, requiring a
new device or such a change to the existing software that it
usually takes many months to develop and distribute an
upgrade. Likely at great expense to the Operator, and with the
risk of being potentially restrictive to the launch of a new
service.
As an example, up to 2014 the UK market has seen 5
different models of the home hub that British Telecom installs
for their customers broadband service. Sky Broadband has
reportedly used 8 different CPE models between 2006 and
2013.
This high-importance/low-perceived-value conundrum has
been limiting the profitability of operator-managed networking
services ever since broadband services were first introduced.
B. IP CPEs are complex but need to be cost-effective
IP-based communication services are by design
decentralized. Any node in an IP network (including the
CPE) must intelligently peer with the adjacent network nodes,
and process, multiplex, encapsulate and de-encapsulate
information across a number of layers in a stack of networking
protocols.
Additionally, the need for tight security (derived from the
any-to-any connectivity principle of the IP protocol) and
private addressing (derived from its nature and addressing
limitations), add up complexity to the CPE in the form of
features such as IPAM/DHCP, Network Address Translation,
stateful firewalling and many others.
On top of that, IP now transports dozens of different
services with diverging requirements. Voice, Video, Internet,
and even Cellular Communications (through femtocells) are
now transported on top of these CPEs. This means that the
device must be able to classify each and every one of the
packets flows travelling across it, and intelligently provide
differentiated treatment to each one of them.
Finally, the CPE must also provide visibility into the
services traffic flows, and must perform metering and
sometimes even complex authentication and accounting
features on them, so that the Service Provider can properly run
a profitable business out of the network.
All these features must be performed or enforced locally
at the customer premises, and build up into a complex list of
features that translate into an elaborate and large amount of
logical functions that have to be implemented in large amounts
of complex software.
But, on the other hand, the device itself into which these
functions are implemented has to be built into consumer-grade,
cost-effective and commoditized hardware, so as not to eat up
the small margins Operators receive through subscriptions.
This leads to failure-prone and loosely maintained devices,
unavoidably leading to numerous failures and service
disruptions.
Service Providers have been looking for ways to effectively
centralize the logic of these services in a location under their
control (move the CPE to the cloud), and simplify the device
located in the customer premises.
This would lead to a model where the service intelligence is
hosted and controlled in the network, while the CPE equiment
becomes passive and simple. It could be an enabler for these
devices to be looked at the in the future in the same way we
view old style terminal devices today.
C. Moving the CPE to the cloud
This paper suggests a way to solve this issue by taking
advantage of modern efficiencies in software-defined
networking and cloud based computing. It is to *shrink* the
device, make it even smaller and cheaper. It is also to remove
the software complexity from the consumer-grade piece of
hardware sitting in the customer premises (far from the Service
Provider control), and put it in a virtualized container that can
be hosted in carrier-class, highly-available, Service Providers
premises.
This model allows for remote control of a service
implementation point that sits in the customer premises. The
CPE becomes a passive device that is tethered to a virtualized
software image running in the Service Providers datacenter.
(Fig.2)
Fig. 2. High-level view of the Tethered CPE Service Delivery architecture.
This new model provides a win for cost and service
performance for now and into the future. It enables rapid
change via agile development on services hosted in a
centralized fashion, allowing for upgrades and adoption of new
standards as they come out.
Also, the advantages of reducing the complexity of
customer premises equipment are numerous:
Dramatically reduced cost/economies of scale
Lower failure rate
Lower dependency on end-customer interaction
(power, location, maintenance)
Software is easier to customize and can be
provided by a different vendor than hardware
 (software innovators with
manufacturers)
Software bugs can be easily fixed if the software
runs in the telco premises (Central Office or
Datacenter)
Reduce the code base in the device, meaning
minimal changes required over time, reducing
bugs and extending the lifetime of the code base in
the router.
By keeping the service implementation complexity in their
centralized location, operators can have tight control of the
service and perform any maintenance required without the
need of making any changes in the customer premises, which
would mean logistic problems in the form of technician visits
or device swap-outs.
Crucially this model also shows how Operators can extend
the reach of their IP services to those customers with legacy
CPE hardware who may have required an upgrade or be
exempt otherwise.
II. CHALLENGES IN THE CURRENT CPE MODEL
A. Challenges along the entire CPE lifecycle
Large numbers of CPE (sometimes in the order of
magnitude of millions) require features to be locked down 12
months in advance of being required. Large scale or iterative
changes are currently not really possible in real time. CPE
firmware rollouts can take weeks or months.
As Table 1 shows, hardware development is usually started
in advance of the software development. However, software
development has a longer lead time so extends the overall
length of the time to market. Removing the complexity of the
software could mean overall CPE development timescales
could be reduced by as much as 12 months.
TABLE I. CPE LIFECYCLE CHART
CPE Lifecycle Charta.
CPE Component Lifecycle Min
Hardware Lifecycle (day 0) 6 mo.
Software Lifecycle (+6 mo.) 12 mo.
Total timeline 18 mo.
a.
From procurement and development to deployment; estimated data from UK Broadband provider
The current integrated stack CPE model (where the same
low-end device performs the control plane and the data plane
functions), presents challenges along the entire supply chain:
The Procurement phase is long and complex, with
service providers in the need to contact vendors
with long lists of features, issuing RFx processes,
and performing in depth testing and certification
processes due to the high failure rate that these
consumer-grade devices could suffer otherwise.
hardware The Staging phase needs to be very carefully
performed, as the configuration finally pushed to
the customer premises will later be difficult to
change. Additionally, they have to deal with
customized firmware loads from all these vendors,
due to the high complexity of the software
functions to be implemented in the device, and the
difficulty of later upgrading a device that is
running in the customer premises.
The Deployment of these devices is often quite
complex, as existing standards such as TR-069 are
not always fully supported on these devices, and
sometimes are not able to perform every
configuration task required for these complex
services. This could result in an installation
process that requires either a technician visit, or
with a long technical support call that could end
up as a replacement.
In the Operation phase, Service providers are
forced to support and maintain a device that sits in
the customer premises under control of a
potentially unskilled end user. It will be
commonly misconfigured, unplugged, hacked, or
even swapped for other equipment that the user
buys directly, and that will be probably untested.
Remote reconfiguration capabilities are limited or
non-existing. Efficient remote management,
telemetry or metering are difficult and unreliable.
And upgrades or bug fixes are often not possible
or are limited.
All this has turned it cheaper to just toss a problematic
device in the trash bin and send a new one to a customer in
trouble, thus restarting the entire process. This is far from
being an efficient solution to the problem and shows a clear
need for a better model.
B. Current CPEs feature network stacks that cant be
effectively upgraded
Max The speed at which communications technology changes
seems to be exponentially increasing. New protocols and
12 mo. technologies have become enablers of innovation and value
18 mo. creation, so new standard versions keep constantly appearing
into the market.
24 mo.
In the residential services market, the introduction of
(2014) protocols such as IPv6, SIP, CGN or PCP have demonstrated
the need for updateable network protocol stacks and periodic
driver changes. You dont want to lock an IPv6 stack down
then find out 12 months later when it is ready that the network
its connecting to has changed, as have the standards.
In the business services segment, the increased control
plane complexity of protocols such as BGP, IS-IS, OSPF, SIP
and many others drive the price of these CPEs to higher orders
of magnitude.
The same needs are also applicable to the devices used in
public wireless access points. Newer protocols and the need for
features such as real multitenancy (enabling the virtual
wireless provider business model on public Access Points),
show a clear need for devices that can be remotely controlled
from the Service Provider cloud, and easily upgraded.
Nevertheless, all of the control plane functions can be
handled with general-purpose CPUs, without the need for
specialized hardware. These functions can be virtualized and
moved as workloads into the Service Provider cloud, while the
data plane can be running in a much simpler (and cost-
effective) device in the customer premises.
III. TETHERED CPE MODEL IMPROVEMENTS
The tethered CPE proposed in this paper leverages an
SDN model by splitting the CPE device into two entities: a
control plane representing the software service logic and
complexity (the brain of the device), and a data plane
entity performing the actual traffic manipulation and the local
enforcement of the service logic. (Fig.3)
Fig. 3. Split-plane model for network elements.
The control plane logic (the complex software) is removed
from the device sitting in the customer premises and
implemented in a virtual machine in the Service Provider
datacenter that can be easily managed, upgraded or troubleshot
by the Service Provider. The Data Plane then is remotely
controlled by the virtualized control plane through a
messaging protocol.
The device sitting in the customer premises acts as a
passive enforcement point of the logic controlled in the cloud.
Essentially it is turned into an NTU-like device, somewhat
like an evolved version of and old phone jack. But it is able
to provide locally in the customer premises all data plane
functions required for the service, such as firewalling, NAT,
QoS, DHCP, DNS, etc. It is local enforcement of service
logic that sits in the cloud.
Additionally, this model must allow for headless
operation of the Tethered CPE: if the connection between the
control plane running in the datacenter and the data plane
running in the customer premises is temporarily lost, the
tethered CPE will keep working without service interruption,
according to the last set of instructions received until the
connectivity is restored.
Implementing the data plane with standard tools available
in the Linux kernel (leveraging the recent additions to its
networking stack that are discussed later in this paper)
significantly reduces the set of features required in the device.
As long as it can run a Linux kernel, it will be able to behave as
a Tethered CPE.
This presents numerous advantages across all phases of the
CPE lifecycle:
The Procurement phase is greatly simplified, as
the list of requirements on the device is reduced
and standardized, thus making them more cost-
effective and stable. This also means that these
features are less dependant on the performance of
the hardware, and dont require large amounts of
flash or running memory.
As customers migrate between services, the
hardware that needs to be changed to support new
features is minimal. Current market trends of
Home Gateways vs simple CPE means that home
network costs are increasing., and this model
reduces that need.
It also allows for customer features to be managed
independently of Wide Area Network (WAN)
interfaces (Fibre, xDSL or Cable Modems) and
allows customers to maintain settings between
WAN upgrade or migration as everything is held
in the cloud.
And even more importantly, investment protection
is guaranteed, as the short list of features required
remains unchanged over time.
The cloud image that implements the service
complexity runs on the service providers
datacenter, so it can easily be managed, patched
and upgraded. This allows switching from long
development cycles to an agile development
model able to change every month and not every
every year for new features and bug resolutions,
while significantly reducing the time to market of
new features.
The Staging is reduced as the set of features to be
tested is standardized. The tethered device now
has a lower risk of failure and a reduced numbers
of bugs. This helps reduce avoidable
replacements and associated ticket escalations or
technician site visits. Moreover, the configuration
initially pushed to the device in the customer
premises can be easily updated remotely, so the
risk associated with closing down a configuration
in the Staging phase is removed.
Deployment of these tethered devices turns into an
simple and error-free task, as the device is
 configured as a virtualized software image
running in the Service Provider cloud.
Additionally, and as discussed later in this paper, a
large group of tethered CPEs can be handled from
a single control plane software instance so
configurations and changes are simplified. The
service can be configured once in the cloud, and
implemented in and number of tethered CPEs
automatically.
The Operations phase improves by limiting access
to the device by the end users. They can be
provided with an online service portal that enables
changes to a limited set of features (for example,
NAPT and port forwarding). From a management
perspective, the service provider now has full
control and easy access to the software running
the service, so all monitoring, metering, telemetry
and support functions are simplified.
IV. : ENABLING THE TETHERED CPE MODEL THROUGH SDN,
NFV AND THE LATEST CHANGES TO THE LINUX KERNELS BPF
SUBSYSTEM.
The concept of separating the control plane and data plane
of networking appliances has been periodically popular in the
networking industry. Early attemps to implement this
separation in firewalls and network address translators were
articulated under the Middlebox Communication (midcom)
IETF working group that published an architecture and
framework RFC in 2002 [1]. Some authors, though, consider
that the arrival of the Openflow protocol in 2008 was the
starting point of the Software-Defined Networking (SDN)
model [2]. SDN was adopted first in academic environments
around 2009, and has been making a big impact in the
networking industry since 2011. Early applications of this
model appeared in datacenter and transport networks, but its
applicability to the CPE model presents plenty of advantages as
discussed earlier in this paper.
This split-plane model requires an implementation
mechanism able to efficiently run a tethered or remotely
controlled networking data plane inside consumer-grade
networking equipment, and under a well-known and standard
technology framework, with standard Application
Programming Interfaces (APIs), interoperable software and,
ultimately, enabling commoditization of the hardware used to
implement it. The data plane should be flexible enough to
implement a variety of networking functions (not only
switching and routing), and be programmable to allow for new
feature implementation. Additionally, this model also requires
the capability to re-package the control plane logic as an
independent software instance running inside a virtual machine
(following the Networks Function Virtualization, or NFV,
model).
The protocol used for communication between control
plane and data plane should also be flexible enough to allow
for configuration of a variety of features, and extensible so that
these features can be augmented over time.
A. The Linux kernel as the right framework to implement a
networking data plane
When looking for a technology framework that would
guarantee flexibility and extensibility in the feature set that a
data plane can implement, interoperability between vendors
(hardware manufacturers and software developers), wide
support across different hardware architectures, and
widespread knowledge of the technology (thus allowing for
abundant potential development resources), use of the Linux
kernel presents a fantastic option. Implementing the data plane
using standard tools available in Linux enables interoperability
between manufacturers, portability to different hardware
platforms, and widely available talent for software
development and operations.
Moreover, using the Linux kernel directly without adding
any vendor-specific library or API framework on top of it
guarantees that this architecture is not limited to work with any
specific vendors hardware, or bound by the capabilities that a
vendors API may support.
The Linux kernel has traditionally supported several
frameworks providing pieces of what could be turned into fully
capable networking functions. There are low-level filtering
tools available (perf, ftrace, stap, ktap), traffic analyzers and
monitors (tcpdump, iptraf), tools for monitoring the networking
stack at different layers (ss, iptraf, netstat, nicstat, ip), tight
control of the network interfaces and the link layer (ethtool,
lldptool) and powerful hierarchical queuing and prioritization
mechanisms (qdisc).
These tools offer access to different pieces of information
about the running networking subsystem. However, in order to
build a fully capable networking data plane, the Linux kernel
should have the ability to execute atomic networking functions
and serialize them to form a complex data plane composed of
several of those networking functions. This would enable
building these atomic networking functions as simple
programs, and serializing them to build a full networking
topology (the full data plane of the service) in kernel space.
Recently, interesting improvements have been included in
the kernels networking subsystem, adding the ability to
implement all the functions required to build all pieces that
compose the data plane of virtually any networking function.
This ability has been introduced recently starting with the
introduction of kernel version 3.15, and are included under the
extended Berkeley Packet Filter (BPF) subsystem.
B. Evolution of the Berkeley Packet Filter (BPF) subsystem
The Berkeley Packet Filter (BPF) subsystem traditionally
allowed an application to quickly filter packets out of a stream.
An early BPF user was the tcpdump application, which used
BPF to implement the filtering behind its complex command-
line syntax. Support for BPF in Linux was added in version
2.1.75 of the kernel in 1997. The BPF interpreter stayed
without changing much for some time (except for some
performance tweaks and minor changes), but it has later seen a
number of extensions through the years. Around the 3.0 kernel
release, other features such as a JIT (just-in-time) compiler
(allowing for real-time updates of the filters and associated
actions over a running system) were added [3].
Recently in versions 3.15 - 3.19, the Linux kernels BPF
subsystem has received a major overhaul which drastically
expands its applicability, adding a number of capabilities and
performance improvements. BPF is now available for direct
use from user space. Additionally, BPF programs can now be
attached to sockets. Once the BPF program is loaded, it will be
run on every packet that shows up on the given socket. This
allows to launch programs that process traffic showing up on a
socket, thus opening up the capability to implement traffic
processing applications as the traffic is received.
Consequently, the BPF subsystem is able to filter packets
out of a stream, and launch user-supplied BPF programs that
would be able to process that traffic and perform any actions
on it.
These recent additions to the kernel add a new system call
named, simply, bpf(); this new call acts as a sort of
connector for other external operations that can be executed
as external programs inside the BPF in-kernel virtual
machine. So, for example, user space can load a BPF program
into the kernel with a call to [4]:
int bpf(BPF_PROG_LOAD, union bpf_attr *attr,
unsigned int size);
The relevant sub-structure of union bpf_attr in this case is:
struct { /* used by BPF_PROG_LOAD command */
__u32 prog_type; /* program type */
__u32 insn_cnt;
__aligned_u64 insns; /* 'struct bpf_insn *' */
__aligned_u64 license; /* 'const char *' */
__u32 log_level; /* verbosity level */
__u32 log_size;
__aligned_u64 log_buf;
};
Here, prog_type describes the context in which a program is
expected to be used; it controls which data and helper
functions will be available to the program when it runs.
BPF_PROG_TYPE_SOCKET is used for programs that will be
attached to sockets, while BPF_PROG_TYPE_TRACING is for
tracing filters. The size of the program (in instructions) is
provided in insn_cnt, while insns points to the program
itself. The license field points to a description of the license
for the program; it may be used in the future to restrict some
functionality to GPL-compatible programs.
Some security questions could arise given that this provides
the ability to run programs within the kernel. So a key part of
the code ensures that BPF programs cannot harm the running
system. Should something suspect turn up, the program will
not be loaded.[5]
Another significant addition in the latest set of kernel
patches set is the addition of "maps." A map is a simple
key/value data store that can be shared between user space and
BPF scripts and is persistent within the kernel. As an example
of their use, we could consider a program that creates a map
with two entries, indexed by IP protocol type, and adds a BPF
script that inspects passing packets and increments the
appropriate entry for each. The program in user space can
query those entries and take action depending on the values
stored in them.
Maps can only be created or deleted from user space; BPF
programs do not have that capability. Maps are created and
deleted with:
int bpf_create_map(enum bpf_map_type map_type,
int key_size, int value_size,
int max_entries);
close(map_fd);
To store values into and retrieve values from maps, a user
space program can call:
int bpf_update_elem(int map_fd,
void *key, void *value);
int bpf_lookup_elem(int map_fd,
void *key, void *value);
int bpf_delete_elem(int map_fd, void *key);
int bpf_get_next_key(int map_fd,
void *key, void *next_key);
But how are these BPF programs actually run? There is
little point in running an eBPF program on demand from user
space; there is not much that it could do that couldn't be more
easily accomplished directly. Instead, eBPF programs are
meant to respond to events within the kernel.
One common event, of course, is the reception of a packet
from the network. BPF now includes a new form of access to
the socket filtering mechanism, allowing a program to directly
attach an BPF program to an open socket:
setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER_EBPF,
&prog_id, sizeof(prog_id));
Here, prog_id must be the ID number of a program previously
loaded into the kernel with the bpf() system call.
Summarizing, the linux kernel now has the ability to match
traffic upon arrival on a socket, filter it and, on a match
condition, launch a small in-kernel program from userspace
that can process the traffic comparing against a key/value table,
modify it and send it to the next stage.
These elements allow implementation of virtually any
atomic networking function that is available today on a
network appliance, as the scheme of parsing packets, looking
them up inside a table, updating the system according to the
information contained within them and modifying and queuing
the packets in the output, comprises nearly all actions that the
data plane of a network appliance performs. Moreover, the
ability to add these filters in a Just In Time (JIT) fashion,
allows to dynamically update these network functions inside a
running kernel.
The model presented in this paper is based on writing
atomic networking functions (like switching, routing,
firewalling, etc.), as BPF programs launched inside the
Linux kernel that process the traffic. The mapping feature of
BPF adds the ability to implement tables against which the
packets can be matched. (Fig. 4)
Fig. 4. Traffic being processed through several BPF programs.
These in-kernel data plane programs performing each
networking stage can be complemented with user space
helper programs that are able to perform the communication
between the in-kernel, on-premises, tethered CPE data plane,
and the centralized, virtualized, control plane instance running
in the Service Providers premises.
Moreover, an adequate overall system architecture design
would allow to define topologies across an entire domain (with
a domain being formed by an arbitrarily large number of data
planes), and to dynamically attach endpoints connected to any
of those data plane instances to the topologies defined centrally
for that domain. Topologies can be defined in the control
plane, including information about the identifier of the
endpoints that will belong to each topology. For example,
parameters related with the endpoints MAC address, the
physical port on which they appear (or the data plane they
appear on), or even a generic pre-configured user-defined tag,
can be used by the control plane to place each endpoint in the
right place of the right topology, dynamically enabling it in the
data plane where and when the endpoint is detected (enabling
any changes just in time). This opens the possibility of
defining a service as a simple topology just once inside the
control plane entity, and dynamically attaching an arbitrary
number of tethered CPEs to it, just when the endpoints to be
serviced are detected in the CPE in question. To simplify the
concept, a single service control plane can control an
arbitrary number of data plane tethered CPEs, so a change in
the service can be implemented as change once, apply to all
(Fig.5)
Finally, on the simplicity, portability and interoperability, a
key advantage of this proposal is the fact that support of the
Linux kernel itself is the only requirement for the tethered
CPE. The Linux kernel has been ported to pretty much every
hardware architecture available, including Intel and ARM
amongs dozens of others, so portability, interoperability and
API simplicity are guaranteed.
Fig. 5. Several data planes in residential gateways tethered to a single control
plane instance.
V. : CONSIDERATIONS FOR A TETHERED CPE
IMPLEMENTATION
A. Data plane of a Broadband Internet service
The IP service provided by the CPE in question will be
formed by a limited set of atomic networking functions
running inside the BPF subsystem. As an example,
considering the network topology used to provide most of
todays broadband services, they can be broken down into a
limited number of networking functions. Many of todays
residential Internet access services are formed by a
combination of:
SWITCH function, interconnecting all terminals
inside the customer premises amongst them and into
the exit towards the Internet.
ROUTER function, separating the customer premises
network from the external network.
NAT function, masquerading the numerous private
addresses used at the customer premises for all
terminals, into a single (or limited set of) publicly
routable address(es). Additionally, this function will
likely be performing some sort of Network Address
Port Translation (NAPT) so that certain layer-4 ports
are mapped across the NAT connection to/from
terminals with private addresses.
FIREWALL function, limiting the traffic into and out
of the private side of the connection, and preventing
malicious traffic from reaching the inside of the
customer premises.
ENCAPSULATION/DECAPSULATION function,
Allowing for isolation of each customer or service
domain by including the traffic inside tunnels such as
 GRE/IPSEC/MPLS/VxLAN/etc. This also enables
connectivity with existing VPN services.
As packets arrive at the entry socket, they will be filtered
and then processed by the SWITCH function (or program)
which will parse the packets, and look them up against the
local switch table (map) to find their destination. The packet
may then be forwarded to the ROUTER function, where once
again they will be matched against the local router table/map
to decide whether they should be passed on to the next
function. Along the way, packets will be modified by the
program in order to perform functions like decreasing their
Time-to-live (TTL), calculating checksums, or modifying their
destination MAC address, for example. This scheme will be
repeated as required until the entire set of functions forming the
data plane is completed.
B. Communications channel between data plane and control
plane
Given the aforementioned capacity of the Linux kernel to
behave as a complex data plane that comprises a number of
atomic network functions, it is now possible to design a full
distributed system that links that data plane instance with a
control plane software image able to inject the desired
topology into the data plane running in the Linux kernel.
Each of these atomic networking functions will require
communication with the centralized control plane to receive
the topology configuration, update the local and centralized
tables (in the case that an incoming packet does not match a
previously existing entry in the table), or may decide that a
particular packet needs to be passed on to a control plane
function (if the packet belongs to a control plane protocol).
This means that an efficient data interchange method needs to
exist for each networking function towards the control plane,
so that the control plane can configure entries into each atomic
data plane program, and each program can inform the
control plane of the events happening in the local network.
This channel can be established by helper programs
residing in user space, through the use of a dedicated
communications socket towards the control plane. The
mechanism used for this socket is once again implementation
dependent. Given that there could be a potentially high number
of endpoints (data plane entities) in a domain that would be
sending information to, and receiving information from the
control plane, the use of a communications mechanism that is
able to handle a potentially high number of endpoints is an
important design decision. Potentially, there could be hundreds
or even thousands of data plane endpoints (CPEs) in a single
control plane domain, so the choice of an adequate messaging
mechanism between them will be a relevant factor determining
the overall system performance and scalability.
Finally, it is relevant to mention that these communications
channels established between control plane and data plane
entities need to be protected by a security model based on some
sort of trusted computing framework. Discussing this
framework is considered out of the scope of this paper.
C. Control Plane considerations
Most of the complexity of the networking system is moved
in this model towards a virtualized software image running in
the providers cloud. This software image will have to
implement all software functions that were previously hosted
inside the control plane entity of the CPE, including:
Dynamic routing protocol processing (BGP, OSPF,
etc.)
Dynamic link-layer protocol processing
Address resolution processing (ARP, etc.)
Management protocol processing (SNMP, etc.)
Configuration management
And, in general, all functions that are typically hosted
in the control/supervisor module of a distributed
switch or router
In addition to these functions, a complete system will have
features typically belonging to higher layers such as topology
management, system state maintenance, event correlation, and
all interactions with other OSS/BSS systems performing
network management, authentication, accounting or service
orchestration.
Whether these features are implemented in the same
software image or a in a separate one, and the way the system
would interact with the rest of the Service Provider ecosystem,
is completely implementation-dependent. There are several
models possible and discussing the advantages and
disadvantages of each one of them is considered out of the
scope of this paper.
VI. : SUMMARY, CONCLUSIONS AND FUTURE WORK
This paper presented the reasons why the current IP CPE
deployment model (based on deploying intelligent
independent equipment in the customer premises) has
important challenges that have been limiting the profitability of
IP based services for telecommunication service providers.
It also presented an alternative service delivery architecture
based on a tethered CPE. This model leverages SDN and
NFV concepts by splitting the control plane and data plane
entities traditionally built in networking appliances. It is
formed by a remotelly controlled data plane instance which
sits at the customer premises and provides local enforcement of
service policies (firewall, IPAM, routing, QoS, multicast, etc.),
while moving most of the software complexity to a virtualized
software instance running in a carrier-class location that is
under more manageable control of the service provider.
Finally, this paper presented an implementation of the data
plane for this architecture exclusively based on using the Linux
kernel. In particular, the latest advances in the BPF subsystem
and its capability to implement in-kernel virtual machines
allow for a very complete, efficient and portable
implementation of this model on a varied range of hardware
platforms.
The authors believe that this architecture solves some of the
issues currently outstanding in most of the virtual CPE
models in the market. Therefore, we view this paper as an IP Internet Protocol
initial step towards a real implementation in the environment of
a telecommunications service provider. IPAM IP Address Management
IPTV IP Television
Efforts are currently ongoing towards completing a real
implementation showing the feasibility of this model, NAT Network Address Translation
combining the technlogies described in this paper with
additional pieces provided by hardware and software vendors. NAPT Network Address Port Translation
These pieces include cost-effective hardware platforms in NFV Network Functions Virtualization
which to implement this model, applying an efficient
communications protocol and a complete object model NTU Network Termination Unit
between data plane and control plane, or providing OPEX Operational Expense
orchestration and management tools for controlling and
provisioning the potentially large number of Tethered CPEs OSPF Open Shortest Path First
and services that a real implementation would require. OTT Over the top
This model uses software and hardware pieces that are PCP Port Control Protocol
well-known and already being deployed at a large scale (even
if they were conceived for other areas of the network or other PON Passive Optical Networking
services and applications), so we expect that reusing them for a
QOS Quality of Service
tethered CPE model will be significantly faster than
rebuilding the technology from scratch. We consider it should SIP Session Initiation Protocol
be possible to have a full working model built with the pieces
currently available (either as open source tools or as tools SDN Software-defined Networking
provided by the vendors involved in this paper) in a very short VPN Virtual Private Network
timeframe.
WAN Wide Area Network

ABBREVIATIONS AND ACRONYMS

ACKNOWLEDGMENT
AAA Authentication, authorization and accounting
The authors would like to acknowledge Alexei Starovoitov
API Application Programming Interface (Distinguished Engineer at PLUMgrid Inc. and key contributor
BGP Border Gateway Protocol to the BPF subsystem used to implement the data plane in the
model presented in this paper), and Pere Monclus (CTO at
BPF Berkeley Packet Filter PLUMgrid Inc.) for sharing a piece of their vast knowledge,
and also for their time and useful suggestions while reviewing
CAPEX Capital Expense
this paper.
CLI Command-line interface
REFERENCES
CO Central Office
CPE Customer Premises Equipment [1] P. Srisuresh et al, RFC 3303: Middlebox communication architecture
and framework, IETF Midcom WG, 2002 (http://www.rfc-
CSP Communication Service Provider editor.org/rfc/rfc3303.txt)
[2] P. Goransson, C. Black, Software Defined Networks: A
CGN Carrier-grade NAT Comprehensive Approach. Morgan Kauffman, 2014, p. 50
DC - Datacenter [3] J. Corbet, BPF: the universal in-kernel virtual machine. LWN.net,
2014 (https://lwn.net/Articles/599755/)
DHCP Dynamic Host Configuration Protocol [4] J. Schulist, D. Borkmann, A. Starovoitov. Linux Socket Filtering aka
Berkeley Packet Filer (BPF)
DNS Domain Name Service (https://www.kernel.org/doc/Documentation/networking/filter.txt)
DSL Digital Subscriber Line [5] J. Corbet, Extending Extended BPF. LWN.net, 2014
(https://lwn.net/Articles/603983/)
IS-IS Intermediate System to Intermediate System