Cisco SD-WAN, and Why the Controller Never Touches Your Traffic

The four partsThe four parts, and the plane each one owns

Cisco's design guide splits the Cisco SD-WAN architecture into four planes, and naming the planes first makes the product names easy to place. The orchestration plane gets a new router into the overlay network. The management plane is central configuration and monitoring. The control plane builds the topology and decides where traffic flows. The data plane forwards the packets.

The SD-WAN Validator, formerly vBond, is the orchestration plane. It is the first thing a new router talks to. It authenticates each device that wants to join, then tells it where the Controllers are.

It is also the one component that sits in public address space, and Cisco gives the reason: that is what lets it reach Controllers and routers sitting behind NAT, for which it acts as the initial NAT traversal orchestrator.

The SD-WAN Manager, formerly vManage, is the management plane. The graphical dashboard where the whole overlay network is configured and watched. Templates, policies, upgrades and the monitoring screens all live here. It is what most people mean by SD-WAN management, and the component they actually log in to.

The SD-WAN Controller, formerly vSmart, is the control plane. Cisco calls it the centralized brain of the solution. Every edge router keeps a control connection to it, sends it the routes it knows, and receives back the routes and policy it should use.

The WAN edge routers are the data plane. Hardware or virtual devices at each branch, campus, data center or cloud site. They build the tunnels, carry the traffic, and apply the policy the Controller sent them.

The product name moved with the parts. The components came from Viptela, the SD-WAN company Cisco bought in 2017.

Cisco has since rebranded Cisco SD-WAN as Cisco Catalyst SD-WAN, and from IOS XE Catalyst SD-WAN release 17.12.1a and Catalyst SD-WAN release 20.12.1 the components carry the new names. Older designs and training material use vBond, vManage and vSmart, and they describe the same things.

vEdge and cEdge, the two kinds of WAN edge

Older material splits the edge devices in two. vEdge routers are the original Viptela hardware, running the Viptela operating system. Cisco's own platforms, the ISR, ASR and Catalyst 8000 families, run IOS XE SD-WAN, and engineers call them cEdge. Both join the same fabric and take policy from the same vSmart Controller.

Cisco's design guide lists the virtual options as well: the vEdge cloud router, ISRv, CSR1000v and Catalyst 8000v, for a cloud or a virtualized site.

Where the control components run

The Validator, Manager and Controller are software, not appliances. Cisco's design guide presents two deployments: Cisco cloud hosted, which it marks as recommended, and on-premises, where the customer runs the components as virtual machines in its own data center. The edge routers work the same way with either.

Control and dataControl goes up, data goes across

This is the idea the rest of the product follows from, and the figure draws it.

Each edge router keeps a control connection to the Controller, and Cisco runs every one of them as a DTLS tunnel. Over those connections travels OMP, carrying routes, next hops, encryption keys and policy.

What never travels over them is user traffic. Two branch routers that need to talk set up an IPsec security association between themselves, and the data goes straight from one to the other.

So the Controller can sit in a data center on the far side of the country without adding a hop to a branch to branch call.

Cisco's own wording is that the Controller does not participate in every flow going through the network. It has to be reachable for routers to learn and update what they know, and that is all.

Losing the Controller is therefore not the same as losing the WAN. OMP has a graceful restart timer, the mechanism for carrying on with what a router already learned while its control connection is down, and the default is 43,200 seconds, twelve hours. That buys time to restore a control component. It is not a reason to run with only one.

OMP and TLOCsHow OMP moves routes, and what a TLOC is

OMP, the Overlay Management Protocol, is on by default on every edge router, Manager and Controller, and Cisco states that the overlay does not function without it. It has one rule that surprises anybody who knows BGP: edge routers never exchange routes with each other.

Cisco's documentation says it plainly. Edge devices do not advertise routing information to each other by OMP or any other method. Each one advertises what it has learned to the Controller, and the Controller, applying policy, distributes overlay routing information to the rest.

The Controller is the route server for the whole WAN, and it is where you change what any branch can reach.

Two kinds of advertisement do most of the work.

OMP routes are prefixes: the subnets behind each router, learned on the LAN side and advertised into the overlay.

TLOC routes say where a router can be reached. A TLOC, a transport location, is the point where a router attaches to a WAN transport, identified by three things: the router's system IP address, a color, and an encapsulation type.

The color is the transport. Cisco defines colors as abstractions that identify individual WAN transports, so a router with a broadband circuit and an MPLS circuit has two TLOCs, one per color, and a prefix behind it is reachable through either.

That is how one branch subnet ends up with several paths, and by default a router installs up to four unique OMP paths for a destination.

TopologySD-WAN topology: full mesh until a policy says otherwise

By default, Cisco's design guide says, WAN edge routers attempt to connect to every TLOC over each WAN transport. Left alone, that is a full mesh: every site with a tunnel to every other site, on every transport that can reach the other end.

That is the right answer for a small network and gets expensive quickly, because the count grows with the square of the number of sites. Fifty sites make 1,225 site pairs.

With two transports at each end and every transport able to reach every other, that is up to four tunnels per pair and 4,900 tunnels across the WAN, each one carrying its own BFD probes.

The shape comes from the Controller. Since routers learn TLOCs only from the Controller, a router cannot build a tunnel to a TLOC it was never told about.

A policy that gives branches only the data center TLOCs produces a hub and spoke network; one that lets sites in a region see each other and send everything else through a hub produces a partial mesh. That is a consequence of how OMP distributes routes, and it is why topology in this product is a policy decision rather than a cabling one.

Path selectionApplication-aware routing, and how it picks a path

Several paths are only useful if something chooses among them, and this is the feature that does. Cisco defines application-aware routing as tracking the characteristics of the tunnels between routers and using what it collects to compute the best path for data traffic.

The measurement is BFD. Every tunnel carries BFD hello packets, by default one a second, and application-aware routing uses them to measure loss, latency and jitter. The default poll interval is ten minutes, so each figure is worked out from about 600 hellos.

What you configure is an SLA class: a maximum loss, latency or jitter, or a combination, that a tunnel must meet to carry a given kind of traffic. Voice gets a strict class, bulk transfer a loose one, and each application is steered to a tunnel that currently meets its class.

When nothing qualifies, traffic still flows. Cisco's documentation says that if no tunnel matches the SLA, data is sent through one of the available tunnels. That is the right default, and it is also why no amount of path selection rescues a site whose every circuit is bad, the same limit the MPLS comparison turns on.

Ten minutes is a long time for a voice call. The averages are computed over the poll interval, so with default settings a tunnel has to stay degraded long enough to move a ten minute average before traffic leaves it. Both intervals can be changed. Check them before promising anybody a failover time.

ArchitectureSegmentation, security and cloud in the Cisco SD-WAN architecture

Three more pieces of the architecture show up in every design, and in most of the pages written about Cisco SD-WAN.

Segmentation uses VPNs. Cisco's design guide says VPNs provide segmentation in this product, much like VRFs on a traditional router.

Two are reserved. VPN 0 is the transport VPN and holds the interfaces that connect to the WAN transports. VPN 512 is the management VPN and carries out-of-band management traffic. User data travels in service VPNs, which keep guest, corporate and payment network traffic apart across the whole fabric.

Security starts with identity. The guide states that all WAN edge devices and control components mutually authenticate each other using an authorized list model.

A router has to present a valid certificate, and its serial number has to be on the digitally signed authorized list, before the Validator lets it in. After that the control plane is secured with DTLS and the data plane with IPsec.

New devices provision themselves. Cisco calls this Zero-Touch Provisioning on vEdge routers and Plug-and-Play on IOS XE SD-WAN routers. A router shipped to a branch contacts a Cisco server, learns where its Validator is, authenticates, and pulls its configuration from the Manager.

Cloud OnRamp covers the cloud side. Cisco describes Cloud onRamp for SaaS as a way to configure access to SaaS applications, direct from the internet or through gateway locations, and Cloud onRamp for Multicloud as automating connectivity to workloads in the public cloud.

SD-AccessSD-Access vs SD-WAN

The names sound like two halves of one product. They are two products for two different parts of the network.

SD-WAN connects sites to each other across the WAN. SD-Access is Cisco's fabric for the campus, the wired and wireless network inside a site, and Cisco describes it as software that automates wired and wireless campus networks.

It is built from different parts: Catalyst Center for management, LISP for the control plane, VXLAN for the data plane, and Cisco TrustSec for policy.

A network can run both, and Cisco publishes validated designs that do: SD-Access inside the campus and SD-WAN between sites. When a proposal names only one of them, it is worth checking which part of your network it actually covers.

PitfallsWhere people go wrong

Treating the Controller as a hub. It is a control plane component, and it is not in the data path. Site to site traffic goes directly between routers unless a policy sends it through a hub.

Running a single Controller. Graceful restart buys hours, not a design. Two routers learning from one Controller share one point of failure for every change.

Putting the Validator behind something that hides it. Cisco places it in public address space so routers behind NAT can find it. If new routers cannot reach it they cannot join, although routers already in the overlay know their Controllers and carry on.

Letting the full mesh grow by default. Every router tries every TLOC on every transport. The policy that shapes the topology is easier to write at twenty sites than to retrofit at two hundred.

Expecting path changes in seconds on default settings. A ten minute poll interval measures trends, not blips.

Reading SD-Access and SD-WAN as one product. One is the campus, the other is the WAN. A quote can include either without the other.

ComparisonEvery component under its current name and the one it had before 20.12

The renaming applies from release 20.12.1, and Cisco also renamed vAnalytics to Catalyst SD-WAN Analytics at the same time. Material from before then is still accurate under the old names.

FAQFrequently asked questions

Keep readingRelated concepts

Read next · Remote access What Is an IPsec VPN? The tunnels every Cisco SD-WAN router builds to the others, explained on their own. Open this next11 min

• Infrastructure · 11 min MPLS Explained The WAN this replaced for most branches, and the one guarantee software cannot copy.
• Design · 9 min Hub and Spoke Topology, and the Traffic That Goes the Long Way The shape most SD-WAN policies produce once a full mesh gets too large.
• Design · 9 min Software Defined Networking, and Where a Business Actually Meets It The architecture behind cloud managed networking and controllers, and what a small office actually meets.
• Routing · 10 min Control Plane vs Data Plane, and Why the Split Matters to an Admin Control plane, data plane and management plane explained, with what fails when each one does.
• Infrastructure · 10 min Cloud Managed Networking, and What Happens When You Stop Paying The switches and access points that usually share the SD-WAN vendor’s cloud dashboard.