The Road to SDN: An Intellectual History of Programmable Networks
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The Road to SDN: An Intellectual History
of Programmable Networks
Nick Feamster Jennifer Rexford Ellen Zegura
Georgia Tech Princeton University Georgia Tech
feamster@cc.gatech.edu jrex@cs.princeton.edu ewz@cc.gatech.edu
ABSTRACT network’s data-plane elements (i.e., routers, switches, and
Software Defined Networking (SDN) is an exciting tech- other middleboxes) via a well-defined Application Pro-
nology that enables innovation in how we design and man- gramming Interface (API). OpenFlow [51] is a prominent
age networks. Although this technology seems to have ap- example of such an API. An OpenFlow switch has one or
peared suddenly, SDN is part of a long history of efforts to more tables of packet-handling rules. Each rule matches a
make computer networks more programmable. In this pa- subset of traffic and performs certain actions on the traffic
per, we trace the intellectual history of programmable net- that matches a rule; actions include dropping, forwarding,
works, including active networks, early efforts to separate or flooding. Depending on the rules installed by a con-
the control and data plane, and more recent work on Open- troller application, an OpenFlow switch can behave like
Flow and network operating systems. We highlight key a router, switch, firewall, network address translator, or
concepts, as well as the technology pushes and application something in between.
pulls that spurred each innovation. Along the way, we de- Over the past few years, SDN has gained significant
bunk common myths and misconceptions about the tech- traction in industry. Many commercial switches support
nologies and clarify the relationship between SDN and re- the OpenFlow API. Initial vendors that supported Open-
lated technologies such as network virtualization. Flow included HP, NEC, and Pronto; this list has since
expanded dramatically. Many different controller plat-
1. Introduction forms have emerged [23, 30, 37, 46, 56, 64, 82]. Program-
mers have used these platforms to create many applica-
Computer networks are complex and difficult to man-
tions, such as dynamic access control [16, 52], server load
age. These networks have many kinds of equipment, from
balancing [39,83], network virtualization [53,68], energy-
routers and switches to middleboxes such as firewalls, net-
efficient networking [42], and seamless virtual-machine
work address translators, server load balancers, and intru-
migration and user mobility [24]. Early commercial suc-
sion detection systems. Routers and switches run com-
cesses, such as Google’s wide-area traffic-management
plex, distributed control software that is typically closed
system [44] and Nicira’s Network Virtualization Plat-
and proprietary. The software implements network pro-
form [53], have garnered significant industry attention.
tocols that undergo years of standardization and interop-
Many of the world’s largest information-technology com-
erability testing. Network administrators typically config-
panies (e.g., cloud providers, carriers, equipment vendors,
ure individual network devices using configuration inter-
and financial-services firms) have joined SDN industry
faces that vary across vendors—and even across different
consortia like the Open Networking Foundation [55] and
products from the same vendor. Although some network-
the Open Daylight initiative [58].
management tools offer a central vantage point for config-
Although the excitement about SDN has become more
uring the network, these systems still operate at the level
palpable during the past few years, many of the ideas un-
of individual protocols, mechanisms, and configuration
derlying SDN have evolved over the past twenty years
interfaces. This mode of operation has slowed innovation,
(or more!). In some ways, SDN revisits ideas from early
increased complexity, and inflated both the capital and op-
telephony networks, which used a clear separation of con-
erational costs of running a network.
trol and data planes to simplify network management and
Software Defined Networking (SDN) is changing the
the deployment of new services. Yet, open interfaces like
way we design and manage networks. SDN has two defin-
OpenFlow enable more innovation in controller platforms
ing characteristics. First, an SDN separates the control
and applications than was possible on closed networks de-
plane (which decides how to handle the traffic) from the
signed for a narrow range of telephony services. In other
data plane (which forwards traffic according to decisions
ways, SDN resembles past research on active networking,
that the control plane makes). Second, an SDN consoli-
which articulated a vision for programmable networks, al-
dates the control plane, so that a single software control
beit with an emphasis on programmable data planes. SDN
program controls multiple data-plane elements. The SDN
control plane exercises direct control over the state in thealso relates to previous work on separating the control and to enable greater to innovation; (2) control and data plane
data planes in computer networks. separation (from around 2001 to 2007), which developed
In this article, we present an intellectual history of pro- open interfaces between the control and data planes; and
grammable networks culminating in present-day SDN. (3) the OpenFlow API and network operating systems
We capture the evolution of key ideas, the application (from 2007 to around 2010), which represented the first
“pulls” and technology “pushes” of the day, and lessons instance of widespread adoption of an open interface and
that can help guide the next set of SDN innovations. developed ways to make control-data plane separation
Along the way, we debunk myths and misconceptions scalable and practical.
about each of the technologies and clarify the relation- Network virtualization played an important role
ship between SDN and related technologies, such as net- throughout the historical evolution of SDN, substantially
work virtualization. Our history begins twenty years ago, predating SDN yet taking root as one of the first signifi-
just as the Internet takes off, at a time when the Internet’s cant use cases for SDN. We discuss network virtualization
amazing success exacerbated the challenges of managing and its relationship to SDN in Section 3.
and evolving the network infrastructure. We focus on in-
novations in the networking community (whether by re-
searchers, standards bodies, or companies), although we 2.1 Active Networking
recognize that these innovations were in some cases cat- The early- to mid-1990s saw the Internet take off, with
alyzed by progress in other areas, including distributed applications and appeal that far outpaced the early applica-
systems, operating systems, and programming languages. tions of file transfer and email for scientists. More diverse
The efforts to create a programmable network infras- applications and greater use by the general public drew
tructure also clearly relate to the long thread of work researchers who were eager to test and deploy new ideas
on supporting programmable packet processing at high for improving network services. To do so, researchers de-
speeds [5, 21, 38, 45, 49, 72, 74]. signed and tested new network protocols in small lab set-
Before we begin our story, we caution the reader that tings and simulated behavior on larger networks. Then,
any history is incomplete and more nuanced than a single if motivation and funding persisted, they took ideas to
storyline might suggest. In particular, much of the work the Internet Engineering Task Force (IETF) to standard-
that we describe in this article predates the usage of the ize these protocols. The standardization process was slow
term “SDN”, coined in an article [36] about the OpenFlow and ultimately frustrated many researchers.
project at Stanford. The etymology of the term “SDN” is In response, some networking researchers pursued
itself complex, and, although the term was initially used to an alternative approach of opening up network control,
describe Stanford’s OpenFlow project, the definition has roughly based on the analogy of the relative ease of
since expanded to include a much wider array of technolo- re-programming a stand-alone PC. Specifically, conven-
gies. (The term has even been sometimes co-opted by in- tional networks are not “programmable” in any meaning-
dustry marketing departments to describe unrelated ideas ful sense of the word. Active networking represented
that predated Stanford’s SDN project.) Thus, instead of a radical approach to network control by envisioning a
attempting to attribute direct influence between projects, programming interface (or network API) that exposed re-
we instead highlight the evolution of and relationships be- sources (e.g., processing, storage, and packet queues) on
tween the ideas that represent the defining characteristics individual network nodes, and supported the construction
of SDN, regardless of whether or not they directly influ- of custom functionality to apply to a subset of packets
enced specific subsequent research. Some of these early passing through the node. This approach was anathema to
ideas may not have directly influenced later ones, but we many in the Internet community who advocated that sim-
believe that the connections between the concepts that we plicity in the network core was critical to Internet success.
outline are noteworthy, and that these projects of the past The active networks research program explored radical
may yet offer new lessons for SDN in the future. alternatives to the services provided by the traditional In-
ternet stack via IP or by Asynchronous Transfer Mode
(ATM), the other dominant networking approach of the
2. The Road to SDN early 1990s. In this sense, active networking was the first
Making computer networks more programmable en- in a series of clean-slate approaches to network architec-
ables innovation in network management and lowers the ture [14] subsequently pursued in programs such as GENI
barrier to deploying new services. In this section, we (Global Environment for Network Innovations) [33] and
review early work on programmable networks. We di- NSF FIND (Future Internet Design) [28] in the United
vide the history into three stages, as shown in Figure 1. States, and EU FIRE (Future Internet Research and Ex-
Each stage has its own contributions to the history: (1) ac- perimentation Initiative) [29] in the European Union.
tive networks (from the mid-1990s to the early 2000s), The active networking community pursued two pro-
which introduced programmable functions in the network gramming models:
21995 2000 2005 2010 2015
Tennenhouse/ Smart
Wetherall [76] Packets [67]
Active Networks
Section 2.1
ANTS [84], High Perf.
SwitchWare [3], Router [86],
Calvert [15] NetScript [20]
RCP [26],
Tempest [79] SoftRouter [47] IRSCP [78]
Control-Data Separation
Section 2.2
ForCES PCE [25], Ethane [16]
protocol [88] 4D [35]
OpenFlow [51] Onix [46] ONOS [56]
OpenFlow and Network OS
Section 2.3
Ethane [16] NOX [37]
Planet- Mininet [48],
MBone [50] Tempest [79] Lab [18] GENI [59] FlowVisor [68]
Network Virtualization:
Section 3
6Bone [43] RON [4] Impasse VINI [7] Open Nicira
[62] vSwitch [63] NVP [53]
Figure 1: Selected developments in programmable networking over the past 20 years, and their chronological relationship to advances in network
virtualization (one of the first successful SDN use cases).
• the capsule model, where the code to execute at the research work in active networks was funded by DARPA,
nodes was carried in-band in data packets [84]; and the funding program supported a collection of projects
and, perhaps more importantly, encouraged convergence
• the programmable router/switch model, where the code
on a terminology and set of active network components
to execute at the nodes was established by out-of-band
so that projects could contribute to a whole meant to be
mechanisms (e.g., [8, 69]).
greater than the sum of the parts [14]. The Active Net-
The capsule model came to be most closely associated works program placed an emphasis on demonstrations and
with active networking. In intellectual connection to sub- project inter-operability, with a concomitant level of de-
sequent efforts, though, both models have some lasting velopment effort. The bold and concerted push from a
legacy. Capsules envisioned installation of new data-plane funding agency in the absence of near-term use cases may
functionality across a network, carrying code in data pack- have also contributed to a degree of community skepticism
ets (as in earlier work on packet radio [90]) and using about active networking that was often healthy but could
caching to improve efficiency of code distribution. Pro- border on hostility and may have obscured some of the in-
grammable routers placed decisions about extensibility di- tellectual connections between that work and later efforts
rectly in the hands of the network operator. to provide network programmability.
The “use pulls” for active networking described in the
Technology push and use pull. The “technology pushes” literature of the time [15, 75] are remarkably similar to
that encouraged active networking included reduction in the examples used to motivate SDN today. The issues
the cost of computing, making it conceivable to put more of the day included network service provider frustration
processing in the network, advances in programming lan- with the timescales necessary to develop and deploy new
guages such as Java that offered platform portability and network services (so-called network ossification), third-
some code execution safety, and virtual machine tech- party interest in value-added, fine-grained control to dy-
nology that protected the host machine (in this case the namically meet the needs of particular applications or net-
active node) and other processes from misbehaving pro- work conditions, and researcher desire for a platform that
grams [71]. Some active networking research projects would support experimentation at scale. Additionally,
also capitalized on advances in rapid code compilation and many early papers on active networking cited the prolif-
formal methods. eration of middleboxes, including firewalls, proxies, and
An important catalyst in the active networking ecosys- transcoders, each of which had to be deployed separately
tem was funding agency interest, in particular the Ac- and entailed a distinct (often vendor-specific) program-
tive Networks program created and supported by the U.S. ming model. Active networking offered a vision of uni-
Defense Advanced Research Projects Agency (DARPA) fied control over these middleboxes that could ultimately
from the mid-1990s into the early 2000s. Although not all
3replace the ad hoc, one-off approaches to managing and • The vision of a unified architecture for middlebox or-
controlling these boxes [75]. Interestingly, the early liter- chestration. Although the vision was never fully real-
ature foreshadows the current trends in network functions ized in the active networking research program, early
virtualization (NFV) [19], which also aims to provide a design documents cited the need for unifying the wide
unifying control framework for networks that have com- range of middlebox functions with a common, safe pro-
plex middlebox functions deployed throughput. gramming framework. Although this vision may not
have directly influenced the more recent work on NFV,
Intellectual contributions. Active networks offered in- various lessons from active networking research may
tellectual contributions that relate to SDN. We note three prove useful as we move forward with the application of
in particular: SDN-based control and orchestration of middleboxes.
• Programmable functions in the network to lower the
barrier to innovation. Research in active networks pi- Myths and misconceptions. Active networking included
oneered the notion of programmable networks as a way the notion that a network API would be available to end-
to lower the barrier to network innovation. The notion users who originate and receive packets, though most in
that it is difficult to innovate in a production network and the research community fully recognized that end-user
pleas for increased programmability were commonly network programmers would be rare [15]. The miscon-
cited in the initial motivation for SDN. Much of the early ception that packets would necessarily carry Java code
vision for SDN focused on control-plane programmabil- written by end users made it possible to dismiss active
ity, whereas active networks focused more on data-plane network research as too far removed from real networks
programmability. That said, data-plane programmabil- and inherently unsafe. Active networking was also crit-
ity has continued to develop in parallel with control- icized at the time for not being able to offer practical
plane efforts [5, 21], and data-plane programmability performance and security. While performance was not
is again coming to the forefront in the emerging NFV a first-order consideration of the active networking re-
initiative. Recent work on SDN is exploring the evo- search community (which focused on architecture, pro-
lution of SDN protocols such as OpenFlow to support gramming models, and platforms), some efforts aimed
a wider range of data-plane functions [11]. Addition- to build high-performance active routers [86]. Similarly,
ally, the concepts of isolation of experimental traffic while security was under-addressed in many of the early
from normal traffic—which have their roots in active projects, the Secure Active Network Environment Archi-
networking—also appear front and center in design doc- tecture project [2] was a notable exception.
uments for OpenFlow [51] and other SDN technologies
In search of pragmatism. Although active networks ar-
(e.g., FlowVisor [31]).
ticulated a vision of programmable networks, the tech-
nologies did not see widespread deployment. Many fac-
• Network virtualization, and the ability to demultiplex tors drive the adoption of a technology (or lack thereof).
to software programs based on packet headers. The Perhaps one of the biggest stumbling blocks that active
need to support experimentation with multiple program- networks faced was the lack of an immediately compelling
ming models led to work on network virtualization. Ac- problem or a clear path to deployment. A significant
tive networking produced an architectural framework lesson from the active networks research effort was that
that describes the components of such a platform [13]. “killer” applications for the data plane are hard to con-
The key components of this platform are a shared Node ceive. The community proffered various applications that
Operating System (NodeOS) that manages shared re- could benefit from in-network processing, including in-
sources; a set of Execution Environments (EEs), each of formation fusion, caching and content distribution, net-
which defines a virtual machine for packet operations; work management, and application-specific quality of ser-
and a set of Active Applications (AAs) that work within vice [15, 75]. Unfortunately, although performance bene-
a given EE to provide an end-to-end service. Direct- fits could be quantified in the lab, none of these application
ing packets to a particular EE depends on fast pattern areas demonstrated a sufficiently compelling solution to a
matching on header fields and demultiplexing to the ap- pressing need.
propriate EE. Interestingly, this model was carried for- Subsequent efforts, which we describe in the next sub-
ward in the PlanetLab [61] architecture, whereby differ- section, were more modest in terms of the scope of prob-
ent experiments run in virtual execution environments, lems they addressed, focusing narrowly on routing and
and packets are demultiplexed into the appropriate ex- configuration management. In addition to a narrower
ecution environment on their packet headers. Demulti- scope, the next phase of research developed technologies
plexing packets into different virtual execution environ- that drew a clear distinction and separation between the
ments has also been applied to the design of virtualized functions of the control and data planes. This separa-
programmable hardware data planes [5]. tion ultimately made it possible to focus on innovations
4in the control plane, which not only needed a significant Some early proposals for separating the data and control
overhaul but also (because it is commonly implemented in planes also came from academic circles, in both ATM net-
software) presented a lower barrier to innovation than the works [10, 32, 80] and active networks [70].
data plane. Compared to earlier research on active networking,
these projects focused on pressing problems in network
2.2 Separating Control and Data Planes management, with an emphasis on: innovation by and
In the early 2000s, increasing traffic volumes and a for network administrators (rather than end users and re-
greater emphasis on on network reliability, predictabil- searchers); programmability in the control plane (rather
ity, and performance led network operators to seek bet- than the data plane); and network-wide visibility and con-
ter approaches to certain network-management functions trol (rather than device-level configuration).
such as the control over the paths used to deliver traf- Network-management applications included selecting
fic (a practice commonly known as traffic engineering). better network paths based on the current traffic load,
The means for performing traffic engineering using con- minimizing transient disruptions during planned routing
ventional routing protocols were primitive at best. Oper- changes, giving customer networks more control over the
ators’ frustration with these approaches were recognized flow of traffic, and redirecting or dropping suspected at-
by a small, well-situated community of researchers who tack traffic. Several control applications ran in opera-
either worked for or interacted regularly with backbone tional ISP networks using legacy routers, including the In-
network operators. These researchers explored pragmatic, telligent Route Service Control Point (IRSCP) deployed
near-term approaches that were either standards-driven or to offer value-added services for virtual-private network
imminently deployable using existing protocols. customers in AT&T’s tier-1 backbone network [78]. Al-
Specifically, conventional routers and switches embody though much of the work during this time focused on man-
a tight integration between the control and data planes. aging routing within a single ISP, some work [25, 26] also
This coupling made various network-management tasks, proposed ways to enable flexible route control across mul-
such as debugging configuration problems and predicting tiple administrative domains.
or controlling routing behavior, exceedingly challenging. Moving control functionality off of network equipment
To address these challenges, various efforts to separate the and into separate servers made sense because network
data and control planes began to emerge. management is, by definition, a network-wide activity.
Logically centralized routing controllers [12, 47, 78] were
Technology push and use pull. As the Internet flour-
enabled by the emergence of open-source routing soft-
ished in the 1990s, the link speeds in backbone net-
ware [9, 40, 65] that lowered the barrier to creating proto-
works grew rapidly, leading equipment vendors to imple-
type implementations. The advances in server technology
ment packet-forwarding logic directly in hardware, sepa-
meant that a single commodity server could store all of the
rate from the control-plane software. In addition, Inter-
routing state and compute all of the routing decisions for
net Service Providers (ISPs) were struggling to manage
a large ISP network [12, 81]. This, in turn, enabled sim-
the increasing size and scope of their networks, and the
ple primary-backup replication strategies, where backup
demands for greater reliability and new services (such as
servers store the same state and perform the same compu-
virtual private networks). In parallel with these two trends,
tation as the primary server, to ensure controller reliability.
the rapid advances in commodity computing platforms
meant that servers often had substantially more memory Intellectual contributions. The initial attempts to sep-
and processing resources than the control-plane processor arate the control and data planes were relatively prag-
of a router deployed just one or two years earlier. These matic, but they represented a significant conceptual de-
trends catalyzed two innovations: parture from the Internet’s conventionally tight coupling
of path computation and packet forwarding. The efforts to
• an open interface between the control and data planes,
separate the network’s control and data plane resulted in
such as the ForCES (Forwarding and Control Element
several concepts that have been carried forward in subse-
Separation) [88] interface standardized by the Internet
quent SDN designs:
Engineering Task Force (IETF) and the Netlink inter-
face to the kernel-level packet-forwarding functionality • Logically centralized control using an open interface
in Linux [66]; and to the data plane. The ForCES working group at the
IETF proposed a standard, open interface to the data
• logically centralized control of the network, as seen in
plane to enable innovation in control-plane software.
the Routing Control Platform (RCP) [12, 26] and Soft-
The SoftRouter [47] used the ForCES API to allow a
Router [47] architectures, as well as the Path Computa-
separate controller to install forwarding-table entries in
tion Element (PCE) [25] protocol at the IETF.
the data plane, enabling the complete removal of control
These innovations were driven by industry’s demands for functionality from the routers. Unfortunately, ForCES
technologies to manage routing within an ISP network. was not adopted by the major router vendors, which
5hampered incremental deployment. Rather than wait- into hardware meant that a router’s control-plane software
ing for new, open APIs to emerge, the RCP [12,26] used could fail independently from the data plane. Similarly,
an existing standard control-plane protocol (the Border distributed routing protocols adopted scaling techniques,
Gateway Protocol) to install forwarding-table entries in such as OSPF areas and BGP route reflectors, where
legacy routers, enabling immediate deployment. Open- routers in one region of a network had limited visibility
Flow also faced similar backwards compatibility chal- into the routing information in other regions. As we dis-
lenges and constraints: in particular, the initial Open- cuss in the next section, the separation of the control and
Flow specification relied on backwards compatibility data planes somewhat paradoxically enabled researchers
with hardware capabilities of commodity switches. to think more clearly about distributed state management:
the decoupling of the control and data planes catalyzed
• Distributed state management. Logically centralized the emergence of a state management layer that maintains
route controllers faced challenges involving distributed consistent view of network state.
state management. A logically centralized controller
must be replicated to cope with controller failure, but In search of generality. Dominant equipment vendors
replication introduces the potential for inconsistent state had little incentive to adopt standard data-plane APIs like
across replicas. Researchers explored the likely fail- ForCES, since open APIs could enable new entrants into
ure scenarios and consistency requirements. At least the marketplace. The resulting need to rely on existing
in the case of routing control, the controller replicas routing protocols to control the data plane imposed sig-
did not need a general state management protocol, since nificant limitations on the range of applications that pro-
each replica would eventually compute the same routes grammable controllers could support. Conventional IP
(after learning the same topology and routing informa- routing protocols compute routes for destination IP ad-
tion) and transient disruptions during routing-protocol dress blocks, rather than providing a wider range of func-
convergence were acceptable even with legacy proto- tionality (e.g., dropping, flooding, or modifying packets)
cols [12]. For better scalability, each controller instance based on a wider range of header fields (e.g., MAC and
could be responsible for a separate portion of the topol- IP addresses, TCP and UDP port numbers), as OpenFlow
ogy. These controller instances could then exchange does. In the end, although the industry prototypes and
routing information with each other to ensure consistent standardization efforts made some progress, widespread
decisions [81]. The challenges of building distributed adoption remained elusive.
controllers would arise again several years later in the To broaden the vision of control and data plane sepa-
context of distributed SDN controllers [46, 56]. Dis- ration, researchers started exploring clean-slate architec-
tributed SDN controllers face the far more general prob- tures for logically centralized control. The 4D project [35]
lem of supporting arbitrary controller applications, re- advocated four main layers—the data plane (for process-
quiring more sophisticated solutions for distributed state ing packets based on configurable rules), the discovery
management. plane (for collecting topology and traffic measurements),
the dissemination plane (for installing packet-processing
rules), and a decision plane (consisting of logically cen-
Myths and misconceptions. When these new architec-
tralized controllers that convert network-level objectives
tures were proposed, critics viewed them with healthy
into packet-handling state). Several groups proceeded to
skepticism, often vehemently arguing that logically cen-
design and build systems that applied this high-level ap-
tralized route control would violate “fate sharing”, since
proach to new application areas [16, 87], beyond route
the controller could fail independently from the devices
control. In particular, the Ethane project [16] (and its di-
responsible for forwarding traffic. Many network oper-
rect predecessor, SANE [17]) created a logically central-
ators and researchers viewed separating the control and
ized, flow-level solution for access control in enterprise
data planes as an inherently bad idea, as initially there was
networks. Ethane reduces the switches to flow tables that
no clear articulation of how these networks would con-
are populated by the controller based on high-level secu-
tinue to operate correctly if a controller failed. Skeptics
rity policies. The Ethane project, and its operational de-
also worried that logically centralized control moved away
ployment in the Stanford computer science department,
from the conceptually simple model of the routers achiev-
set the stage for the creation of OpenFlow. In particular,
ing distributed consensus, where they all (eventually) have
the simple switch design in Ethane became the basis of the
a common view of network state (e.g., through flooding).
original OpenFlow API.
In logically centralized control, each router has only a
purely local view of the outcome of the route-selection
process. 2.3 OpenFlow and Network OSes
In fact, by the time these projects took root, even In the mid-2000s, researchers and funding agencies
the traditional distributed routing solutions already vio- gained interest in the idea of network experimentation
lated these principles. Moving packet-forwarding logic at scale, encouraged by the success of experimental in-
6frastructures (e.g., PlanetLab [6] and Emulab [85]), and The initial OpenFlow protocol standardized a data-
the availability of separate government funding for large- plane model and a control-plane API by building on tech-
scale “instrumentation” previously reserved for other dis- nology that switches already supported. Specifically, be-
ciplines to build expensive, shared infrastructure such as cause network switches already supported fine-grained ac-
colliders and telescopes [54]. An outgrowth of this en- cess control and flow monitoring, enabling OpenFlow’s
thusiasm was the creation of the Global Environment for initial set of capabilities on switch was as easy as perform-
Networking Innovations (GENI) [33] with an NSF-funded ing a firmware upgrade—vendors did not need to upgrade
GENI Project Office and the EU FIRE program [29]. Crit- the hardware to make their switches OpenFlow-capable.
ics of these infrastructure-focused efforts pointed out that OpenFlow’s initial target deployment scenario was
this large investment in infrastructure was not matched campus networks, meeting the needs of a networking re-
by well-conceived ideas to use it. In the midst of this, search community actively looking for ways to conduct
a group of researchers at Stanford created the Clean Slate experimental work on “clean-slate” network architectures
Program and focused on experimentation at a more local within a research-friendly operational setting. In the late
and tractable scale: campus networks [51]. 2000s, the OpenFlow group at Stanford led an effort to
Before the emergence of OpenFlow, the ideas underly- deploy OpenFlow testbeds across many campuses and
ing SDN faced a tension between the vision of fully pro- demonstrate the capabilities of the protocol both on a sin-
grammable networks and pragmatism that would enable gle campus network and over a wide-area backbone net-
real-world deployment. OpenFlow struck a balance be- work spanning multiple campuses [34].
tween these two goals by enabling more functions than As real SDN use cases materialized on these campuses,
earlier route controllers and building on existing switch OpenFlow began to take hold in other realms, such as
hardware, through the increasing use of merchant-silicon data-center networks, where there was a distinct need to
chipsets in commodity switches. Although relying on manage network traffic at large scales. In data centers,
existing switch hardware did somewhat limit flexibility, the cost of hiring engineers to write sophisticated con-
OpenFlow was almost immediately deployable, allowing trol programs to run over large numbers of commodity
the SDN movement to be both pragmatic and bold. The switches proved to be more cost-effective than continu-
creation of the OpenFlow API [51] was followed quickly ing to purchase closed, proprietary switches that could
by the design of controller platforms like NOX [37] that not support new features without substantial engagement
enabled the creation of many new control applications. with the equipment vendors. As vendors began to com-
An OpenFlow switch has a table of packet-handling pete to sell both servers and switches for data centers,
rules, where each rule has a pattern (that matches on bits many smaller players in the network equipment market-
in the packet header), a list of actions (e.g., drop, flood, place embraced the opportunity to compete with the es-
forward out a particular interface, modify a header field, tablished router and switch vendors by supporting new ca-
or send the packet to the controller), a set of counters (to pabilities like OpenFlow.
track the number of bytes and packets), and a priority (to
disambiguate between rules with overlapping patterns). Intellectual contributions. Although OpenFlow embod-
Upon receiving a packet, an OpenFlow switch identifies ied many of the principles from earlier work on the sep-
the highest-priority matching rule, performs the associated aration of control and data planes, the rise of OpenFlow
actions, and increments the counters. offered several additional intellectual contributions:
• Generalizing network devices and functions. Previous
Technology push and use pull. Perhaps the defining fea-
work on route control focused primarily on matching
ture of OpenFlow is its adoption in industry, especially as
traffic by destination IP prefix. In contrast, OpenFlow
compared with its intellectual predecessors. This success
rules could define forwarding behavior on traffic flows
can be attributed to a perfect storm of conditions between
based on any set of 13 different packet headers. As such,
equipment vendors, chipset designers, network operators,
OpenFlow conceptually unified many different types of
and networking researchers. Before OpenFlow’s genesis,
network devices that differ only in terms of which header
switch chipset vendors like Broadcom had already begun
fields they match, and which actions they perform. A
to open their APIs to allow programmers to control certain
router matches on destination IP prefix and forwards out
forwarding behaviors. The decision to open the chipset
a link, whereas a switch matches on source MAC ad-
provided the necessary impetus to an industry that was al-
dress (to perform MAC learning) and destination MAC
ready clamoring for more control over network devices.
address (to forward), and either floods or forwards out a
The availability of these chipsets also enabled a much
single link. Network address translators and firewalls
wider range of companies to build switches, without in-
match on the five tuple (of source and destination IP
curring the substantial cost of designing and fabricating
addresses and port numbers, and the transport proto-
their own data-plane hardware.
col) and either rewrites address and port fields, or drops
unwanted traffic. OpenFlow also generalized the rule-
7installation techniques, allowing anything from proac- whether the controller handles any data traffic. Some SDN
tive installation of coarse-grained rules (i.e., with “wild- applications respond only to topology changes and coarse-
cards” for many header fields) to reactive installation of grained traffic statistics and update rules infrequently in
fine-grained rules, depending on the application. Still, response to link failures or network congestion. Other ap-
OpenFlow does not offer data-plane support for deep plications may send the first packet of some larger traffic
packet inspection or connection reassembly; as such, aggregate to the controller, but not a packet from every
OpenFlow alone cannot efficiently enable sophisticated TCP or UDP connection.
middlebox functionality. A second myth surrounding SDN is that the controller
• The vision of a network operating system. In contrast to must be physically centralized. In fact, Onix [46] and
earlier research on active networks that proposed a node ONOS [56] demonstrate that SDN controllers can—and
operating system, the work on OpenFlow led to the no- should—be distributed. Wide-area deployments of SDN,
tion of a network operating system [37]. A network op- as in Google’s private backbone [44], have many con-
erating system is software that abstracts the installation trollers spread throughout the network.
of state in network switches from the logic and appli- Finally, a commonly held misconception is that SDN
cations that control the behavior of the network. More and OpenFlow are equivalent; in fact, OpenFlow is merely
generally, the emergence of a network operating system one (widely popular) instantiation of SDN principles. Dif-
offered a conceptual decomposition of network opera- ferent APIs could be used to control network-wide for-
tion into three layers [46]: (1) a data plane with an open warding behavior; previous work that focused on routing
interface; (2) a state management layer that is responsi- (using BGP as an API) could be considered one instanti-
ble for maintaining a consistent view of network state; ation of SDN, for example, and architectures from vari-
(3) control logic that performs various operations de- ous vendors (e.g., Cisco ONE and JunOS SDK) represent
pending on its view of network state. other instantiations of SDN that differ from OpenFlow.
• Distributed state management techniques. Separating In search of control programs and use cases. Despite
the control and data planes introduces new challenges the initial excitement surrounding SDN, it is worth rec-
concerning state management. Running multiple con- ognizing that SDN is merely a tool that enables innova-
trollers is crucial for scalability, reliability, and perfor- tion in network control. SDN neither dictates how that
mance, yet these replicas should work together to act control should be designed nor solves any particular prob-
like a single, logically centralized controller. Previous lem. Rather, researchers and network operators now have
work on distributed route controllers [12, 81] only ad- a platform at their disposal to help address longstanding
dressed these problems in the narrow context of route problems in managing their networks and deploying new
computation. To support arbitrary controller applica- services. Ultimately, the success and adoption of SDN de-
tions, the work on the Onix [46] controller introduced pends on whether it can be used to solve pressing prob-
the idea of a network information base—a represen- lems in networking that were difficult or impossible to
tation of the network topology and other control state solve with earlier protocols. SDN has already proved use-
shared by all controller replicas. Onix also incorpo- ful for solving problems related to network virtualization,
rated past work in distributed systems to satisfy the state as we describe in the next section.
consistency and durability requirements. For example,
Onix has a transactional persistent database backed by 3. Network Virtualization
a replicated state machine for slowly-changing network
In this section, we discuss network virtualization, a
state, as well as an in-memory distributed hash table for
prominent early “use case” for SDN. Network virtualiza-
rapidly-changing state with weaker consistency require-
tion presents the abstraction of a network that is decoupled
ments. More recently, the ONOS [56] system offers an
from the underlying physical equipment. Network virtu-
open-source controller with similar functionality, using
alization allows multiple virtual networks to run over a
existing open-source software for maintaining consis-
shared infrastructure, and each virtual network can have a
tency across distributed state and providing a network
much simpler (more abstract) topology than the underly-
topology database to controller applications.
ing physical network. For example, a Virtual Local Area
Network (VLAN) provides the illusion of a single LAN
Myths and misconceptions. One myth concerning SDN spanning multiple physical subnets, and multiple VLANs
is that the first packet of every traffic flow must go to the can run over the same collection of switches and routers.
controller for handling. Indeed, some early systems like Although network virtualization is conceptually indepen-
Ethane [16] worked this way, since they were designed to dent of SDN, the relationship between these two technolo-
support fine-grained policies in small networks. In fact, gies has become much closer in recent years.
SDN in general, and OpenFlow in particular, do not im- We preface our discussion of network virtualization
pose any assumptions about the granularity of rules or with three caveats. First, a complete history of network
8virtualization would require a separate survey; we focus ing them much easier to deploy. To lower the barrier
on developments in network virtualization that relate di- for experimenting with overlay networks, researchers be-
rectly to innovations in programmable networking. Sec- gan building virtualized experimental infrastructures like
ond, although network virtualization has gained promi- PlanetLab [61] that allowed multiple researchers to run
nence as a use case for SDN, the concept predates modern- their own overlay networks over a shared and distributed
day SDN and has in fact evolved in parallel with pro- collection of hosts. Interestingly, PlanetLab itself was a
grammable networking. The two technologies are in fact form of “programmable router/switch” active networking,
tightly coupled: Programmable networks often presumed but using a collection of servers rather than the network
mechanisms for sharing the infrastructure (across multiple nodes, and offering programmers a conventional operat-
tenants in a data center, administrative groups in a campus, ing system (i.e., Linux). These design decisions spurred
or experiments in an experimental facility) and supporting adoption by the distributed-systems research community,
logical network topologies that differ from the physical leading to a significant increase in the role of experimen-
network, both of which are central tenets of network vir- tation with prototype systems in this community.
tualization. Finally, we caution that a precise definition of Based on the success of shared experimental plat-
“network virtualization” is elusive, and experts naturally forms in fostering experimental systems research, re-
disagree as to whether some of the mechanisms we discuss searchers started advocating the creation of shared exper-
(e.g., slicing) represent forms of network virtualization. In imental platforms that pushed support for virtual topolo-
this article, we define the scope of network virtualization gies that can run custom protocols inside the underlying
to include any technology that facilitates hosting a virtual network [7,62] to enable realistic experiments to run side-
network on an underlying physical network infrastructure. by-side with operational traffic. In this model, the network
equipment itself “hosts” the virtual topology, harkening
Network Virtualization before SDN. For many years, back to the early Tempest architecture [79] where multi-
network equipment has supported the creation of virtual ple virtual ATM networks could co-exist on the same set
networks, in the form of VLANs and virtual private net- of physical switches [79]; the Tempest architecture even
works. However, only network administrators could cre- allowed switch-forwarding behavior to be defined using
ate these virtual networks, and these virtual networks were software controllers, foreshadowing the work on control
limited to running the existing network protocols. As and data-plane separation.
such, incrementally deploying new technologies proved The GENI [33, 60] initiative took the idea of a vir-
difficult. Instead, researchers and practitioners resorted tualized and programmable network infrastructure to a
to running overlay networks, where a small set of up- much larger scale, building a national experimental infras-
graded nodes use tunnels to form their own topology on tructure for research in networking and distributed sys-
top of a legacy network. In an overlay network, the up- tems. Moving beyond experimental infrastructure, some
graded nodes run their own control-plane protocol, and researchers argued that network virtualization could form
direct data traffic (and control-plane messages) to each the basis of a future Internet that enables multiple network
other by encapsulating packets, sending them through the architectures to coexist at the same time (each optimized
legacy network, and decapsulating them at the other end. for different applications or requirements, or run by differ-
The Mbone (for multicast) [50], the 6bone (for IPv6) [43], ent business entities), and evolve over time to meet chang-
and the X-Bone [77] were prominent early examples. ing needs [27, 62, 73, 89].
These early overlay networks consisted of dedicated
nodes that ran the special protocols, in the hope of Relationship of Network Virtualization to SDN. Net-
spurring adoption of proposed enhancements to the net- work virtualization (an abstraction of the physical net-
work infrastructure. The notion of overlay networks soon work in terms of a logical network) clearly does not re-
expanded to include any end-host computer that installs quire SDN. Similarly, SDN (the separation of a logically
and runs a special application, spurred by the success centralized control plane from the underlying data plane)
of early peer-to-peer file-sharing applications (e.g., Nap- does not imply network virtualization. Interestingly, how-
ster and Gnutella). In addition to significant research on ever, a symbiosis between network virtualization and SDN
peer-to-peer protocols, the networking research commu- has emerged, which has begun to catalyze several new re-
nity reignited research on using overlay networks as a way search areas. SDN and network virtualization relate in
to improve the network infrastructure, such as the work three main ways:
on Resilient Overlay Networks [4], where a small collec-
• SDN as an enabling technology for network virtualiza-
tion of communicating hosts form an overlay that reacts
tion. Cloud computing brought network virtualization to
quickly to network failures and performance problems.
prominence, because cloud providers need a way to al-
In contrast to active networks, overlay networks did
low multiple customers (or “tenants”) to share the same
not require any special support from network equipment
network infrastructure. Nicira’s Network Virtualiza-
or cooperation from the Internet Service Providers, mak-
tion Platform (NVP) [53] offers this abstraction without
9requiring any support from the underlying networking physical resources or dynamically reconfiguring networks
hardware. The solution is use overlay networking to pro- in multi-tenant environments—that actually come from
vide each tenant with the abstraction of a single switch network virtualization. Although SDN facilitates network
connecting all of its virtual machines. Yet, in contrast to virtualization and may thus make some of these functions
previous work on overlay networks, each overlay node easier to realize, it is important to recognize that the ca-
is a actually an extension of the physical network—a pabilities that SDN offers (i.e., the separation of data and
software switch (like Open vSwitch [57,63]) that encap- control plane, abstractions for distributed network state)
sulates traffic destined to virtual machines running on do not directly provide these benefits.
other servers. A logically centralized controller installs
the rules in these virtual switches to control how packets Exploring a broader range of use cases. Although SDN
are encapsulated, and updates these rules when virtual has enjoyed some early practical successes and certainly
machines move to new locations. offers much-needed technologies in support of the specific
use case of network virtualization, more work is needed
• Network virtualization for evaluating and testing SDNs. both to improve the existing infrastructure and to explore
The ability to decouple an SDN control application from SDN’s potential to solve problems for a much broader set
the underlying data plane makes it possible to test and of use cases. Although early SDN deployments focused
evaluate SDN control applications in a virtual environ- on university campuses [34], data centers [53], and private
ment before the application is deployed on an opera- backbones [44], recent work explores applications and ex-
tional network. Mininet [41, 48] uses process-based vir- tensions of SDN to a broader range of network settings, in-
tualization to run multiple virtual OpenFlow switches, cluding home networks, enterprise networks, Internet ex-
end hosts, and SDN controllers—each as a single pro- change points, cellular core networks, cellular and WiFi
cess on the same physical (or virtual) machine. The use radio access networks, and joint management of end-host
of process-based virtualization allows Mininet to emu- applications and the network. Each of these settings in-
late a network with hundreds of hosts and switches on troduces many new opportunities and challenges that the
a single machine. In such an environment, a researcher community will explore in the years ahead.
or network operator can develop control logic and easily
test it on a full-scale emulation of the production data 4. Conclusion
plane; once the control plane has been evaluated, tested, This paper has offered an intellectual history of pro-
and debugged, it can then be deployed on the real pro- grammable networks. The idea of a programmable net-
duction network. work initially took shape as active networking, which es-
poused many of the same visions as SDN, but lacked both
• Virtualizing (“slicing”) an SDN. In conventional net-
a clear use case and an incremental deployment path. Af-
works, virtualizing a router or switch is complicated,
ter the era of active networking research projects, the pen-
because each virtual component needs to run own in-
dulum swung from vision to pragmatism, in the form of
stance of control-plane software. In contrast, virtualiz-
separating the data and control plane to make the network
ing a “dumb” SDN switch is much simpler. The FlowVi-
easier to manage. This work focused primarily on bet-
sor [68] system enables a campus to support a testbed for
ter ways to route network traffic—a much narrower vision
networking research on top of the same physical equip-
than previous work on active networking.
ment that carries the production traffic. The main idea
Ultimately, the work on OpenFlow and network oper-
is to divide traffic flow space into “slices” (a concept in-
ating systems struck the right balance between vision and
troduced in earlier work on PlanetLab [61]), where each
pragmatism. This work advocated network-wide control
slice has a share of network resources and is managed
for a wide range of applications, yet relied only on the
by a different SDN controller. FlowVisor runs as a hy-
existing capabilities of switch chipsets. Backwards com-
pervisor, speaking OpenFlow to each of the SDN con-
patibility with existing switch hardware appealed to many
trollers and to the underlying switches. Recent work
equipment vendors clamoring to compete in the growing
has proposed slicing control of home networks, to al-
market in data-center networks. The balance of a broad,
low different third-party service providers (e.g., smart
clear vision with a pragmatic strategy for widespread
grid operators) to deploy services on the network with-
adoption gained traction when SDN found a compelling
out having to install their own infrastructure [89]. More
use case in network virtualization.
recent work proposes ways to present each “slice” of
As SDN continues to develop, we believe that history
a software-defined network with its own logical topol-
has important lessons to tell. First, SDN technologies will
ogy [1, 22] and address space [1].
live or die based on “use pulls”. Although SDN is of-
ten heralded as the solution to all networking problems, it
Myths and misconceptions. People often refer to sup- is worth remembering that SDN is just a tool for solving
posed “benefits of SDN”—such as amortizing the cost of network-management problems more easily. SDN merely
10places the power in our hands to develop new applications Protocols on Programmable Hardware. In Proc. ACM SIGCOMM,
and solutions to longstanding problems. In this respect, New Delhi, India, Aug. 2010.
[6] A. Bavier, M. Bowman, D. Culler, B. Chun, S. Karlin, S. Muir,
our work is just beginning. If the past is any indication, L. Peterson, T. Roscoe, T. Spalink, and M. Wawrzoniak.
the development of these new technologies will require Operating System Support for Planetary-Scale Network Services.
innovation on multiple timescales, from long-term bold In Proc. First Symposium on Networked Systems Design and
Implementation (NSDI), San Francisco, CA, Mar. 2004.
visions (such as active networking) to near-term creative [7] A. Bavier, N. Feamster, M. Huang, L. Peterson, and J. Rexford. In
problem solving (such as the operationally focused work VINI Veritas: Realistic and Controlled Network Experimentation.
on separating the control and data planes). In Proc. ACM SIGCOMM, Pisa, Italy, Aug. 2006.
Second, we caution that the balance between vision and [8] S. Bhattacharjee, K. Calvert, and E. Zegura. An architecture for
active networks. In High Performance Networking, 1997.
pragmatism remains tenuous. The bold vision of SDN ad- [9] BIRD Internet routing daemon.
vocates a wide variety of control applications; yet, Open- http://bird.network.cz/.
Flow’s control over the data plane is confined to primi- [10] J. Biswas, A. A. Lazar, J.-F. Huard, K. Lim, S. Mahjoub, L.-F.
tive match-action operations on packet-header fields. We Pau, M. Suzuki, S. Torstensson, W. Wang, and S. Weinstein. The
ieee p1520 standards initiative for programmable network
should remember that the initial design of OpenFlow was interfaces. Communications Magazine, IEEE, 36(10):64–70, 1998.
driven by the desire for rapid adoption, not first princi- [11] P. Bosshart, G. Gibb, H. Kim, G. Varghese, N. McKeown,
ples. Supporting a wide range of network services would M. Izzard, F. Mujica, and M. Horowitz. Forwarding
metamorphosis: Fast programmable match-action processing in
require much more sophisticated ways to analyze and ma- hardware for SDN. In ACM SIGCOMM, Aug. 2013.
nipulate traffic (e.g., deep-packet inspection, and com- [12] M. Caesar, N. Feamster, J. Rexford, A. Shaikh, and J. van der
pression, encryption, and transcoding of packets), using Merwe. Design and implementation of a routing control platform.
In Proc. 2nd USENIX NSDI, Boston, MA, May 2005.
commodity servers (e.g., x86 machines) or programmable
[13] K. Calvert. An architectural framework for active networks (v1.0).
hardware (e.g., FPGAs, network processors, and GPUs), http://protocols.netlab.uky.edu/c̃alvert/arch-latest.ps.
or both. Interestingly, the renewed interest in more sophis- [14] K. Calvert. Reflections on network architecture: An active
ticated data-plane functionality, such as Network Func- networking perspective. ACM SIGCOMM Computer
Communications Review, 36(2):27–30, 2006.
tions Virtualization, harkens back to the earlier work on [15] K. Calvert, S. Bhattacharjee, E. Zegura, and J. Sterbenz.
active networking, bringing our story full circle. Directions in active networks. IEEE Communications Magazine,
Maintaining SDN’s bold vision requires us to continue pages 72–78, October 1998.
thinking “out of the box” about the best ways to program [16] M. Casado, M. J. Freedman, J. Pettit, J. Luo, N. McKeown, and
S. Shenker. Ethane: Taking control of the enterprise. In ACM
the network, without being constrained by the limitations SIGCOMM ’07, 2007.
of current technologies. Rather than simply designing [17] M. Casado, T. Garfinkel, M. Freedman, A. Akella, D. Boneh,
SDN applications with the current OpenFlow protocols in N. McKeown, and S. Shenker. SANE: A protection architecure for
enterprise networks. In Proc. 15th USENIX Security Symposium,
mind, we should think about what kind of control we want Vancouver, BC, Canada, Aug. 2006.
to have over the data plane, and balance that vision with a [18] B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson,
pragmatic strategy for deployment. M. Wawrzoniak, and M. Bowman. Planetlab: an overlay testbed
for broad-coverage services. ACM SIGCOMM Computer
Communication Review, 33(3):3–12, 2003.
Acknowledgments [19] M. Ciosi et al. Network functions virtualization. Technical report,
We thank Mostafa Ammar, Ken Calvert, Martin Casado, ETSI, Darmstadt, Germany, Oct. 2012. http:
//portal.etsi.org/NFV/NFV_White_Paper.pdf.
Russ Clark, Jon Crowcroft, Ian Leslie, Larry Peterson, [20] S. da Silva, Y. Yemini, and D. Florissi. The NetScript active
Nick McKeown, Vyas Sekar, Jonathan Smith, Kobus van network system. IEEE Journal on Selected Areas in
der Merwe, and David Wetherall for detailed comments, Communications, 19(3):538–551, 2001.
feedback, insights, and perspectives on this article. [21] M. Dobrescu, N. Egi, K. Argyraki, B.-G. Chun, K. Fall,
G. Iannaccone, A. Knies, M. Manesh, and S. Ratnasamy.
RouteBricks: Exploiting parallelism to scale software routers. In
REFERENCES Proc. 22nd ACM Symposium on Operating Systems Principles
[1] A. Al-Shabibi. Programmable virtual networks: From network (SOSP), Big Sky, MT, Oct. 2009.
slicing to network virtualization, July 2013. http://www. [22] D. Drutskoy, E. Keller, and J. Rexford. Scalable network
slideshare.net/nvirters/virt-july2013meetup. virtualization in software-defined networks. IEEE Internet
[2] D. Alexander, W. Arbaugh, A. Keromytis, and J. Smith. Secure Computing, March/April 2013.
active network environment archtiecture: Realization in [23] D. Erickson. The Beacon OpenFlow controller. In Proc. HotSDN,
SwitchWare. IEEE Network Magazine, pages 37–45, May 1998. Aug. 2013.
[3] D. S. Alexander, W. A. Arbaugh, M. W. Hicks, P. Kakkar, A. D. [24] D. Erickson et al. A demonstration of virtual machine mobility in
Keromytis, J. T. Moore, C. A. Gunter, S. M. Nettles, and J. M. an OpenFlow network, Aug. 2008. Demo at ACM SIGCOMM.
Smith. The SwitchWare active network architecture. IEEE [25] A. Farrel, J. Vasseur, and J. Ash. A Path Computation Element
Network, 12(3):29–36, 1998. (PCE)-Based Architecture. Internet Engineering Task Force, Aug.
[4] D. G. Andersen, H. Balakrishnan, M. F. Kaashoek, and R. Morris. 2006. RFC 4655.
Resilient Overlay Networks. In Proc. 18th ACM Symposium on [26] N. Feamster, H. Balakrishnan, J. Rexford, A. Shaikh, and
Operating Systems Principles (SOSP), pages 131–145, Banff, K. van der Merwe. The case for separating routing from routers.
Canada, Oct. 2001. In ACM SIGCOMM Workshop on Future Directions in Network
[5] B. Anwer, M. Motiwala, M. bin Tariq, and N. Feamster. Architecture, Portland, OR, Sept. 2004.
SwitchBlade: A Platform for Rapid Deployment of Network
11You can also read
