A top of rack switch is one that lives in the same rack as the servers it connects, so every server cable is a short patch lead that never leaves the cabinet and only the fiber uplinks cross the data center.
The alternative, end of row, puts fewer and larger switches at the end of a row and runs a long cable from every server to them. The name makes it sound like a decision about switches, and it is not.
It is a decision about cabling, about where a failure stops, and about whether you build a room at once or a rack at a time. What decides whether the design works is the ratio of server capacity to uplink capacity.
- The switch sits in the rack, so server cabling never leaves the cabinet
- End of row uses fewer switches and one long cable per server
- Copper is limited to a 100 m channel, which is why placement matters
- A ToR failure takes one rack; end of row takes part of a row
- The number to ask for is oversubscription, not the port count
On this page
The constraintWhy this is a cabling question
Start with the constraint, because the whole argument comes from it.
Twisted pair copper is specified to a 100 m channel, and that figure includes the patch leads at both ends, not just the horizontal run. A row of racks is easily long enough for that to matter, so the position of the switch decides whether a server connects with a two meter lead or a thirty meter one.
Top of rack takes the short answer. The ToR switch is in the cabinet, every server is a patch lead away, and the only cables leaving are the uplinks. End of row takes the other one: the aggregation switches are consolidated, and every server in every rack in that row is cabled back to them across the data center floor.
Either way the switch is doing the same job in the network. It is the access layer, the first switching hop a server reaches, and what changes is only where that hop physically sits.
That produces the trade in its simplest form. ToR means many switches and few long cables. EoR means few switches and many long cables. Everything else about the architecture follows from that sentence, including the parts that look like they are about something else.
Copper and fiberCopper inside the rack, fiber out of it
The 100 m limit is on the twisted pair. It is not on the uplinks, and that distinction is what makes the whole arrangement work.
A top of rack switch takes copper on its server-facing ports, where every run is a couple of meters, and fiber on its uplinks, where the run crosses the data center to an aggregation layer.
Fiber is not bound by the copper channel limit, so the rack can sit wherever the floor plan wants it. The design converts a long copper problem into a short copper problem plus a fiber problem, and fiber is the one that scales.
Inside the cabinet the server connections are often not twisted pair at all. Direct attach copper assemblies, a few meters of twinax with the optical module moulded onto each end, are the usual way to reach a server at 10 or 25 gigabits, and they are cheaper than a transceiver pair and draw less power.
The uplinks are where the optics live. A ToR switch normally carries a small number of high speed cages, and one of those ports can be split with a breakout cable into several slower connections, which is how a rack gets a sensible number of uplinks without buying a switch with more of them.
End of row does the opposite. It keeps the aggregation switches close enough that copper can reach the servers, which means the long runs are the copper runs, and the 100 m channel becomes a constraint on how the row is laid out rather than a detail.
| Server-facing cable | Cable that crosses the room | What limits the layout | |
|---|---|---|---|
| Top of rack | Copper, a few meters, inside the cabinet | Fiber uplinks | Almost nothing |
| End of row | Copper, tens of meters, along the row | Fiber, from the row upward | The 100 m copper channel |
The second cost that scales with switch count is management rather than money. Thirty racks means thirty or sixty switches to configure, keep on one firmware version, monitor and patch. That is a real networking burden and it is the honest argument for consolidating, which is what end of row is.
Automation makes it tolerable rather than free, and a data center running switch configuration by hand will feel every additional box. Each ToR switch also needs out of band access, a management port and a console connection that still works when the network it belongs to does not.
OversubscriptionThe number nobody prints
This is the part worth taking away, because it is the question that separates a design that works from one that merely connects.
Count the server capacity a switch faces and count the uplink capacity behind it. Forty eight servers at 10 gigabits each is 480 gigabits of possible demand. Two 40 gigabit uplinks is 80 gigabits of escape.
That is a 6:1 oversubscription ratio, and it is fine, or it is a disaster, depending entirely on whether those servers talk to each other inside the rack or to something outside it.
Traffic that stays within the cabinet never touches the ratio. Two servers in the same rack reach each other across one switch, which is both the lowest latency path in the data center network and the only path that costs no uplink capacity.
Nobody advertises this. A datasheet leads with port count and switching capacity, both of which describe the inside of the box. Oversubscription describes the relationship between the box and the rest of the data center network, which is where the performance problem actually lives.
The reason it matters more now than it used to is that traffic changed direction. Traffic between servers, east to west, grew faster than traffic in and out of the building, and virtualization and clustering put a great deal of conversation between machines that used to be conversation between a machine and a user.
A design tuned for traffic leaving the rack behaves badly when most of the traffic is trying to cross to the next rack.
That is the pressure that produced the leaf and spine architecture. Every top of rack switch becomes a leaf, every leaf connects to every spine, and any rack reaches any other rack in the same two hops.
It is a top of rack design with the uplink problem taken seriously, and the overlay networks that run across it exist because the underlying fabric is now routed rather than switched.
Failure domainWhere the failure domain sits
The other consequence of placement is what happens when something dies, and it is a genuine advantage rather than a marketing one.
A top of rack switch failing removes one rack. That is bad, and it is bounded, and if the workloads in that rack are spread across other racks it may be survivable without anybody being woken up.
An EoR aggregation switch failing removes the servers of several racks at once. The blast radius follows the cabling, which is the point: the cables all end in the same place, so the network failure does too.
The usual answer in both designs is a second switch and dual homing every server, which halves the elegance and doubles the switch count. In top of rack that means two switches per cabinet, which is where the port waste from the table above gets worse. Nothing here is free, and pretending otherwise is how a design gets sold.
PitfallsWhere people go wrong
Choosing on switch price. The switches are the visible cost and the cabling is the network infrastructure you live with. A data center cable plant is a ten year decision and a switch is a five year one.
Ignoring the empty ports. Part-full racks are the normal case, and every unused port in a top of rack switch is money spent early for capacity that may never be used.
Never calculating oversubscription. Port count and switching capacity describe the box. The ratio of server capacity to uplink capacity describes whether the design works.
Assuming east to west traffic is small. It usually is not any more, and an architecture that assumes traffic mostly leaves the data center will congest on the uplinks while every port looks healthy.
Treating leaf and spine as a different thing. It is top of rack with the uplinks designed properly. The leaves are the same switches in the same place.
Forgetting that the 100 m limit includes the patch leads. Designs get built to the horizontal run and then fail on the two three meter leads at either end.
ComparisonTop of rack and end of row, side by side
| Criterion | Top of rack | End of row |
|---|---|---|
| Switches to buy and manage | One or two per rack | A few aggregation switches per row |
| Long cable runs | Only uplinks | One per server |
| Cable plant decided when | As each rack is filled | Before the row is built, and it is permanent |
| A switch failure takes out | One rack | Part of a row |
| Ports wasted on part-full racks | Yes, this is the real cost | Very few |
| Suits growth by | Adding a rack at a time | Filling a room you already built |
The fifth row is the one that decides most real deployments and it rarely appears in the comparison. A 48 port switch in a rack holding twelve servers has 36 ports doing nothing, and you paid for them, powered them and have to manage them.
Multiply that across a data center and top of rack is expensive in a way that does not show up until the invoice. The row underneath explains why people accept that.
If racks fill up one at a time over three years, ToR lets the network infrastructure arrive with the servers. EoR needs the cable plant designed for a full row on the day the first server lands, and rebuilding it later means working over live equipment.
FAQFrequently asked questions
What is a top of rack switch?
A switch mounted in the same rack as the servers it serves, so all server cabling stays inside the cabinet and only the fiber uplinks leave it.
What is the difference between top of rack and end of row?
Where the switch sits, and therefore how the data center cabling runs. Top of rack means many switches and short cables. End of row means fewer aggregation switches and a long cable from every server.
Does the switch have to be at the top?
No. The name is conventional rather than physical, and middle of rack placement is common because it halves the worst case cable length inside the cabinet.
Why does cable length matter?
Twisted pair copper is specified to a 100 m channel including the patch leads at both ends. A long row can exceed that, which is the constraint the whole design question comes from.
Which is cheaper, top of rack or end of row?
It depends on how full your racks are. EoR wastes fewer ports and costs more in cabling. Top of rack wastes ports in part-full racks, costs far less to cable, and costs more to manage.
What is oversubscription in this context?
The ratio of the server capacity a switch faces to the uplink capacity behind it. Forty eight 10 gigabit servers behind two 40 gigabit uplinks is 6:1.
What oversubscription ratio is acceptable?
It depends on whether the traffic stays in the rack. Heavy east to west traffic needs a much lower ratio than a workload that mostly serves users outside the building.
What happens if a top of rack switch fails?
One rack loses its network. That is the main argument for the design, because an end of row failure takes several racks at once.
Is leaf and spine the same as top of rack?
The leaf and spine architecture uses top of rack switches as its leaves. It is the same placement with every leaf connected to every spine, so any rack in the data center reaches any other in two hops.
Do I need two switches per rack?
If the workloads cannot tolerate losing a rack, yes, and each server is cabled to both. It doubles the switch count and makes the unused port problem worse.
Does this apply to a small server room?
Less than people assume. With one or two racks the cable lengths are trivial, there is no row to run along, and the question is mostly about whether you want the switch in the cabinet or on a wall.
Why is east to west traffic mentioned so often?
Because virtualization and clustering moved a lot of conversation between servers rather than between a server and a user, and uplinks sized for traffic leaving the building congest when most of it is crossing to the next rack.
What is a ToR switch, and how does it affect data center switch placement?
A ToR switch is a top of rack switch: one or two switches in each rack serving only the servers in that rack. It is the most common data center switch placement because cabling stays short and inside the rack. The alternatives are end of row and middle of row, which use fewer switches and much longer cable runs.
Keep readingRelated concepts
Read next · Cabling and connectivity What UTP Cable Is, and Why the Twist Does the Shielding Where the 100 m channel limit comes from, and why it includes the patch leads rather than just the horizontal run. Open this next9 min- Switching · 11 min Spanning Tree Protocol What the older switched design needed to stay loop free, and what leaf and spine replaced it with.
- Switching · 12 min What Is VXLAN? The overlay that runs across a leaf and spine fabric, and the reason the underlying network became routed.
- Cabling and power · 9 min 802.3af, and Why Your Switch Still Says It The power budget question that decides how many devices one of those switches can actually run.
- Cabling · 13 min What Is a Patch Panel, and Why It Does Nothing At All The passive field a top of rack switch is cabled to.