← field notes

Designing Enterprise Networks

Sarooj Zawahir21 November 202652 min read

  • Networking
  • Data Centre
  • High Availability

A network design is a set of decisions about structure, each of which trades something away. This guide works through those decisions in the order they have to be made: the hierarchy, the Layer 2 and Layer 3 boundary, redundancy, routing, addressing, and the places where a design quietly fails.

What network design actually is

A network design is not a diagram. The diagram is the last artefact produced, and it is the least interesting one. The design is the set of decisions that produced it, and the reasons each decision went the way it did.

Most of those decisions are about where to put boundaries. Where does Layer 2 stop and Layer 3 start? Where does a failure stop propagating? Where does one address block end and the next begin? Where does policy get enforced? A good design puts these boundaries in the same place, so that one line on the diagram is simultaneously a broadcast boundary, a fault boundary, a summarisation boundary and a policy boundary. A bad design scatters them, and every subsequent change has to be reasoned about four times.

Requirements come before topology

The questions that actually determine a topology are rarely about networking:

Question What it determines
How many end devices, and where are they physically? Access port count, closet count, uplink count
What is the tolerable outage for each application? Redundancy model, convergence target, protocol timers
Which groups must be kept apart, and by what? Segmentation model, where firewalls sit, VRF or VLAN strategy
Does any application require Layer 2 adjacency? Whether the Layer 3 boundary can be pushed to the access layer
What is the growth expectation over five years? Address plan, module sizing, spare capacity
Who operates it, at what hour, with what skills? Protocol choice, feature count, how much cleverness is affordable

That last row is the one most often skipped and most often fatal. A design that only one person understands has a single point of failure that no amount of hardware redundancy addresses.

The three properties worth optimising

Everything below is in service of three things.

Determinism. You should be able to say, in advance and without logging in, which path a packet takes and which path it takes after any single failure. Designs that rely on spanning tree to pick a topology, or on routing protocols to find their own way through a mesh, fail this test. Equal-cost paths between known pairs of devices pass it.

Bounded failure domains. Any single fault should affect a predictable and small region. This is why Layer 2 domains are kept small, why routes are summarised, and why the core is kept free of features that can fail.

Modularity. The network is built from one block repeated, not from a hundred unique arrangements. When the building block is proven, the tenth one carries no new risk, and capacity planning becomes arithmetic rather than judgement.

On the age of the source material

Several of the figures in this guide come from Cisco's campus validated designs, some of which date from 2008. The platforms they name are long obsolete and some of the timer advice reflects hardware of that era. The structural guidance has not changed, because it follows from how the protocols work rather than from what the silicon could do, and Cisco's current campus guides reach the same conclusions. Where a number is tied to a specific platform or release, it is flagged as such.

The hierarchical model

The campus hierarchy is three layers: access, distribution, core. It has survived thirty years of changing hardware because it is not really about hardware. It is a rule for where to put the boundaries described in section 1.

Cisco's campus infrastructure design guide describes the layers as follows: the access layer is the network edge where end devices, IP phones and access points connect; the distribution layer is "the boundary between the Layer 2 wiring-closet network and the Layer 3 routed core", consolidating access networks and enforcing edge policy; and the core is the campus backbone.

three layer hierarchy · one distribution block

What each layer is for

The access layer is where the network meets everything it does not control. It terminates user ports, phones, access points, printers, cameras, badge readers and whatever appears next. Because it is the only layer that touches untrusted devices, it carries the port-level features: 802.1X, port security, DHCP snooping, dynamic ARP inspection, storm control, PoE and QoS classification. It is also the layer with the most devices, so anything configured here is configured hundreds of times, and anything that cannot be templated will be done inconsistently.

The distribution layer is the one that does the real design work. It is the aggregation point for a group of access switches, and in a traditional design it is also the Layer 3 boundary: the default gateway for every access VLAN lives here, as do the inter-VLAN routing, the summarisation towards the core, and most access control. It is deployed as a pair, because it is the point where a single failure would otherwise take out a whole building.

The core layer exists to connect distribution blocks to each other at speed, and to do nothing else. Cisco's campus high availability guide describes it as a Layer 3 switching environment using only hardware-accelerated services, and summarises the design philosophy as "less is more". Every feature added to the core is a feature that can fail in the one place where failure is least contained.

The two structural rules in the drawing

Two details in the diagram are not decoration.

Every access switch connects to both distribution switches. Cisco's phrase for this is to "build triangles, not squares", so that link and box failures are resolved by the hardware already carrying an equal-cost alternative, rather than by waiting for a protocol to converge. A single-homed access switch is a design that has chosen an outage.

The two distribution switches are connected to each other. The Cisco HA campus design guide is unambiguous: "An L3 link is required between the distribution nodes." Without it, when an access uplink fails, traffic returning from the core can arrive at the distribution switch that no longer has a path to the destination VLAN and be dropped. The guide adds a second requirement that is easy to miss: that link's address space must fall inside the block being summarised towards the core, or the summary advertises reachability the switch cannot deliver.

A core-to-core link is a different matter. With every distribution pair dual-homed to both core switches, the core nodes already have two paths to each other through the distribution layer, so the link is optional rather than required. Many designs include one anyway; it is a judgement call, not a rule.

Two tiers or three

A collapsed core merges the distribution and core functions into one pair of switches. It is the right answer for a large share of real networks, and the wrong answer past a threshold that is easy to calculate.

Cisco's campus infrastructure design guide gives the threshold directly: deploy "a three-tier LAN design when numbers of distribution blocks are greater than two". The reasoning is in the arithmetic of interconnections.

mesh versus core · four distribution blocks

The arithmetic

Without a core, N distribution blocks need every block connected to every other block, which is N(N-1)/2 adjacencies. With a core, each block connects only to the core, which is N adjacencies however large N becomes.

Blocks Adjacencies, full mesh Adjacencies, via core Uplinks per distribution switch, full mesh
2 1 2 2
3 3 3 4
4 6 4 6
6 15 6 10
8 28 8 14

The column that decides it is the last one. In a full mesh, each distribution switch needs a pair of uplinks towards every other block, so its uplink port count grows with the size of the campus. With a core, each distribution switch needs exactly two uplinks whether the campus has three blocks or thirty. That is what "modular" means in practice: the building block does not change when you add another one.

At two blocks the mesh wins outright, needing one adjacency against the core's two, and no extra pair of switches to buy, power and operate. At three it is a tie on adjacencies and the core is already ahead on ports. Past three the mesh loses on every measure. This is exactly why Cisco's threshold sits where it does.

The other reasons to add a core

The adjacency count is the clean argument, but it is rarely the one that settles the matter in a real building.

  • Physical layout. Blocks in separate buildings have to meet somewhere. That somewhere is a core, whether or not it is called one.

  • Change isolation. A core gives you a place to add, remove or rebuild a distribution block without touching any other block's configuration. In a mesh, every new block is a change on every existing block, which means a change window involving every part of the campus.

  • Port economics. Distribution switches are feature-rich and expensive per port. Core switches are simple and fast. Pushing transit ports onto the cheaper platform is usually the better buy once there are enough of them.

  • Failure domain. A mesh spreads the blast radius of a misconfiguration across every block it touches.

What a collapsed core looks like today

For the many campuses that sit under the threshold, the current pattern is a pair of Layer 3 switches running as a single logical device. Cisco's current cloud campus design guide builds exactly this: a collapsed core of two switches joined by StackWise Virtual, with all the SVIs on that pair and the access switches running pure Layer 2 with no SVIs of their own.

The logical-pair approach is worth understanding as a design choice rather than a product feature. Two switches presenting as one remove the blocked spanning tree ports from the access uplinks, because the access switch sees a single device at the far end of what is now one port channel. That changes the failure analysis: you gain a topology with no blocked links, and you accept a shared control plane across the two chassis, which is a different and more correlated failure mode than two independent switches running a routing protocol.

Where Layer 2 stops

This is the decision that shapes everything else in the campus. Every other choice in sections 5 to 12 depends on it.

In the traditional design, the access switch is a Layer 2 device. It carries VLANs up a trunk to the distribution pair, which owns the switched virtual interfaces, the default gateways and the inter-VLAN routing. In the routed access design, the access-to-distribution trunks are replaced by Layer 3 point-to-point links, and the gateway moves down to the access switch.

Layer 2 access versus routed access

What moves when the boundary moves

Moving the boundary down is not a single change. It relocates a cluster of functions that travel together.

Function Layer 2 access Routed access
Default gateway SVI on the distribution pair SVI on the access switch
First hop redundancy HSRP, VRRP or GLBP required Not needed; there is one gateway
Spanning tree root Distribution, co-located with the FHRP active node Access switch, and the topology below it is trivial
Access uplinks 802.1Q trunks, one blocking Routed point-to-point links, both forwarding
Inter-VLAN routing Distribution Access
ACLs and ip helper-address Distribution SVIs Access SVIs
Multicast PIM on distribution SVIs, with per-VLAN tuning PIM on the routed links
Recovery from an uplink failure Spanning tree plus FHRP Routing protocol, or no convergence at all on an equal-cost path

Cisco's routed access design guide notes that in the routed design the virtual gateway protocols are no longer needed and per-VLAN multicast tuning such as PIM query interval adjustment becomes unnecessary. Both of those are features you no longer have to get right.

The convergence argument

The numbers come from Cisco's measurements of both designs. An optimal Layer 2 access design converges in 800 to 900 milliseconds, and getting there requires a correct hierarchical design, tuned HSRP or GLBP timers and an optimal spanning tree design. A Layer 3 access design converges in under 200 milliseconds, and it needs only the correct hierarchical routing design. Cisco describes the difference as roughly fourfold.

The reason is structural rather than a matter of tuning. In the Layer 2 design, a failure has to be detected by one protocol, acted on by a second, and then have the gateway moved by a third, with the slowest of them setting the pace. In the routed design, both uplinks are already in the forwarding table. For most failures the hardware simply stops using one next hop and keeps using the other, and no protocol has to converge at all.

Cisco's guide is explicit about the one case that is not free: recovery from any single component failure happens without a routing convergence except for the loss of the distribution-to-access link, where the protocol does have to converge. That link is the worst case, because the access switch has no equal-cost alternative to the subnet behind it.

When Layer 2 access is still the right answer

Routed access costs you one thing, and Cisco states it plainly: Layer 2 access is preferred where VLANs must span multiple wiring closets, because the routed design gives that up entirely. A VLAN in a routed access design lives on exactly one access switch or stack.

So the question to ask is not which design is better. It is whether anything in the environment genuinely requires a single broadcast domain across closets:

  • Clustering or heartbeat products that require Layer 2 adjacency between members in different parts of the building.

  • Wireless designs that bridge client traffic locally at the access point rather than tunnelling it to a controller, and need roaming across closets within one subnet.

  • Building systems, lab equipment and instrumentation that assume a flat segment.

  • Appliances that are licensed, clustered or managed in a way that assumes their peers are on the same subnet.

If none of those exist, routed access is the stronger design. If some do, the usual answer is not to abandon the idea but to confine it: routed access everywhere, with a small number of deliberately-identified Layer 2 islands where something genuinely needs one.

One piece of Cisco's guidance applies regardless of which access design you choose: "Cisco recommends a routed network core in all cases." There is no campus-scale argument for a switched core.

Designing the access layer

The access layer is sized by counting, not by judgement. Four numbers decide it: how many ports, how much uplink, how much power, and how many VLANs.

Ports

Count the devices that terminate in each closet, then add the ones nobody mentioned: wireless access points, cameras, door controllers, display screens, printers, label printers, time clocks, building management controllers and the lab bench that has never been documented. Add growth, and round up to a whole switch rather than leaving two ports free.

The reason to be generous here is not elegance. Running out of ports in a closet forces an unplanned switch into a stack, which means a maintenance window, a new uplink pair, and often a second VLAN because the first subnet has also run out. One cheap decision at design time avoids all three.

Oversubscription is the ratio of the bandwidth that could arrive at the edge to the bandwidth leaving towards the distribution layer. Cisco's campus infrastructure design guide gives two figures: "The distribution layer should maintain a 20:1 oversubscription ratio in distribution-access networking blocks", and the distribution and core layers should maintain 4:1 oversubscription ratios.

The calculation is straightforward. A 48-port access switch with 1 Gbps edge ports offers 48 Gbps of edge capacity. Two 10 Gbps uplinks give 20 Gbps upstream, which is 2.4:1. That is far richer than the 20:1 guideline, which is the normal state of affairs with modern uplink speeds: the published ratio is a floor to stay above, not a target to design down to.

Access switch Edge capacity Uplinks Ratio, both active Ratio, one blocking
48 x 1 Gbps 48 Gbps 2 x 10 Gbps 2.4:1 4.8:1
48 x 1 Gbps 48 Gbps 2 x 25 Gbps 1:1 or better 1.9:1
48 x 2.5 Gbps multigigabit 120 Gbps 2 x 10 Gbps 6:1 12:1
48 x 2.5 Gbps multigigabit 120 Gbps 2 x 25 Gbps 2.4:1 4.8:1

The last column is the one that matters in a Layer 2 access design, and it is the column people forget. If spanning tree is blocking one of the two uplinks, the switch has half the upstream bandwidth the port count suggests. The multigigabit row with two 10 Gbps uplinks goes from a comfortable 6:1 to a 12:1 that will be visible to users during a backup window. In a routed access design, or behind a logical switch pair where both uplinks form one port channel, the ratio stays in the first column.

Power

PoE is now the constraint that bites first in a refresh, because device power demand has risen faster than switch power supplies. Cisco's comparison of the standards gives the per-interface maxima:

Cisco name IEEE standard Type Maximum power per interface Pairs used
PoE 802.3af Type 1 15.4 W 2
PoE+ 802.3at Type 2 30 W 2
UPOE Cisco, 802.3bt based Type 3 60 W 4
UPOE+ 802.3bt Type 4 90 W 4

The design mistake is to multiply the port count by the per-port maximum and specify a power supply for that. Nobody buys a 48-port switch with 4,320 W of PoE budget. The correct method is to build a device inventory per closet with the class of each, sum the actual demand, add headroom for growth, and then choose the supply. Then check what happens when a redundant supply fails, because a switch in power-budget shortfall denies power to ports by priority, and the ports it denies are the ones you did not set a priority on.

VLANs and subnets

In both the Layer 2 and routed access designs, Cisco configures each access switch with its own voice and data VLANs, and does not extend VLANs across access switches. That single rule prevents most of the trouble described in sections 8 and 12.

Size the subnet for the ports, not for the building. A 48-port switch does not need a /23. A /24 per VLAN per access switch is conventional, wasteful of address space in the IPv4 sense, and almost always the right call, because it keeps the addressing plan regular enough to summarise. Section 11 covers why that matters more than the waste.

Stacking

Where access switches are stacked, the stack is one logical device with one management plane, one configuration and one control plane. Two consequences follow for the design:

  • Take the uplinks from different stack members. A stack whose two uplinks both leave the same physical switch has a single point of failure that the diagram does not show.

  • The stack is a shared fate domain. A software upgrade, a crash or a bad configuration affects every port in the stack at once. This is the trade for the simplicity, and it is usually worth it, but it should be a decision rather than an accident.

The edge feature set

The access port is the only place where the network touches equipment it does not control, so it is where the controls belong. At minimum: PortFast with BPDU Guard on every user-facing port, DHCP snooping, dynamic ARP inspection built on the snooping database, an explicit allowed-VLAN list on every trunk, a native VLAN that is unused everywhere, and 802.1X or MAB where the organisation can operate it.

Cisco's campus infrastructure guide is specific about the trunk hygiene: allow only assigned VLANs on trunk ports, and use an unused, unique native VLAN. Both are one line of configuration and both close a class of attack that costs nothing to prevent and a great deal to investigate.

Designing the distribution layer

The distribution layer is where a campus design is won or lost. The access layer is repetitive and the core is deliberately dull; the distribution pair is where the boundaries actually sit.

It does five jobs:

  1. Aggregation. It terminates the uplinks from a group of access switches, so the core does not have to. Cisco's phrasing is that it "aggregates nodes from the access layer, protecting the core from high-density peering".

  2. The default gateway, in a Layer 2 access design, for every access VLAN in the block.

  3. Summarisation. It advertises one prefix for the whole block towards the core, instead of one per access VLAN.

  4. Policy. Access control between VLANs, QoS trust boundaries, and often the handoff to a firewall for traffic leaving the block.

  5. The fault boundary. Everything inside the block is a local problem. Everything outside sees only the summary.

Summarisation is the point

Of those five, summarisation is the one that is most often skipped and most consequential. Cisco's HA campus design guide states it flatly: "Summarize at the distribution."

The reason is about failure propagation, not route table size. When an access VLAN goes away, the distribution switch notices. If it is advertising a summary covering that VLAN, nothing outside the block changes: no update is sent, no query is generated, no SPF runs anywhere else. If it is advertising each VLAN individually, the withdrawal propagates across the campus, every router recomputes, and a single closet's problem has become a campus-wide event.

In EIGRP terms, summarising bounds the scope of queries. The Cisco guide's example configures ip summary-address eigrp 100 10.1.0.0 255.255.0.0 5 on the uplink towards the core. In OSPF terms, the distribution switch is an area border router and the summary is an area range. Either way the structural effect is identical, which is why the addressing plan in section 11 has to come first: you cannot summarise address space that was not allocated contiguously.

The distribution-to-distribution link deserves a second mention here. The Cisco guide warns that its address space must fall within the summarised range. If the interlink is addressed out of some unrelated block, the summary towards the core stops matching reality in exactly the failure case the interlink exists to cover.

How many access switches per block

There is no published number, because the real constraints are local: the physical reach of the cabling, the port count on the distribution pair, the aggregate uplink bandwidth, and the size of the failure domain you are prepared to accept.

The reasoning that matters is the last one. A distribution block is the unit of failure, the unit of change and the unit of address allocation. If one block covers an entire campus, a maintenance window on the distribution pair is a campus-wide outage, and the summary advertised to the core covers everything, which makes it useless as a boundary. If blocks are too small, you pay for distribution pairs that aggregate four access switches each.

A practical test: a block should be a region you would be willing to take down as a unit at 2 a.m. If the answer is no because it contains too much, the block is too big.

Load sharing across the pair

When both distribution switches forward, the traffic has to be divided between them. Two details are worth knowing.

In a Layer 2 access design with HSRP, one distribution switch is active for a given VLAN and the other is idle for it. The conventional answer is to alternate: make Dist 1 the HSRP active and spanning tree root for the odd VLANs and Dist 2 for the even ones, so both switches carry traffic. Cisco's rule is that whichever switch is the HSRP primary for a VLAN must also be the spanning tree primary for it, or traffic takes a two-hop path across the interlink to reach its own gateway.

In a routed design, load sharing is by equal-cost paths, and the hazard is CEF polarisation: if every layer in the hierarchy hashes on the same fields, the same flows take the same side of every pair, and half the links carry nothing. Cisco's HA campus guide addresses this by varying the input to the hash between tiers, using Layer 3 and Layer 4 inputs at the distribution layer with mls ip cef load-sharing full while the core keeps the default Layer 3 hash. The command is platform-specific and dated; the principle is not. Do not let two consecutive tiers make the same decision on the same inputs.

Designing the core

The core is the shortest section in any design document, and that is the point. Cisco's campus high availability guide describes it as a Layer 3 switching environment using only hardware-accelerated services, and sums up the philosophy as "less is more".

What the core does

It forwards packets between distribution blocks, at line rate, with as few decisions per packet as possible. It participates in one routing protocol. It has a small routing table, because every distribution block is summarised before it arrives. It has a configuration that fits on a page.

That is the entire specification.

What the core must not do

The list is longer, and every item on it is something that has been put in a core and later taken out of one.

  • No access ports. The moment a server, a firewall or a contractor's laptop terminates on a core switch, the core has acquired an edge, and with it a reason to carry an access VLAN, a reason to have an SVI and a reason to be reloaded.

  • No default gateways for user VLANs. The gateway belongs where the summarisation boundary is.

  • No policy. Access lists, NAT, inspection, deep classification and anything that reads past the forwarding decision belong at the distribution layer or in a dedicated services block. A core access list is a core that can be broken by a typo.

  • No Layer 2 transit. Cisco's recommendation that a routed core is correct "in all cases" is one of the few absolutes in campus design.

  • No asymmetry. All core links the same speed, all core devices the same platform and the same software version. An asymmetric core produces traffic patterns that are difficult to predict and harder to capacity-plan.

  • No convenience. The core is the wrong place for the thing that was quick to put somewhere, because it is the one failure domain that contains every other one.

Sizing

Core sizing is driven by three numbers: the aggregate bandwidth between distribution blocks, the port count needed to connect every block twice, and the forwarding capacity needed with one core switch out of service.

That last one is the test people skip. A two-node core must carry the full campus load on one node during a software upgrade. If it cannot, the redundancy is nominal: the network survives the failure in the sense that packets still flow, but the user experience during a planned maintenance window is indistinguishable from an outage.

A note on where the services go

Everything that does not belong in the core and does not belong in an access block goes into a services block: a distribution-style pair of switches dedicated to firewalls, load balancers, wireless controllers, VPN concentrators and the data centre handoff. It attaches to the core exactly like a distribution block, and gets summarised exactly like one.

This is not an extra layer. It is a distribution block whose access layer happens to be appliances rather than closets, and treating it that way keeps the core's rules intact.

Spanning tree, where Layer 2 remains

If any part of the design keeps Layer 2 between switches, spanning tree is running, and it is running whether or not anyone has thought about it. An unconfigured spanning tree is still a design; it is just one chosen by default bridge priorities and MAC address ordering.

Cisco's own troubleshooting guidance is blunt about the legacy protocol: "Classic STP is not recommended for production environments." It recommends a Layer 3 routed design where practical, and a modern mode otherwise.

Choose the mode deliberately

Mode Standard Convergence Where it fits
802.1D IEEE 802.1D Up to about 50 seconds Nowhere new
Rapid PVST+ Cisco, built on 802.1w, which was incorporated into 802.1D-2004 About 1 second Cisco-only campus, one instance per VLAN
MST IEEE 802.1s, incorporated into 802.1Q-2003 Rapid, 802.1w based Many VLANs, mixed vendors, large Layer 2 domains

The 50-second figure for 802.1D comes from its default timers: 20 seconds of max age, then 15 seconds of listening and 15 of learning. Cisco's HA campus guide reports roughly one second of convergence for Rapid PVST+ against up to 50 for 802.1D, which is the whole argument for not running the old one.

Between Rapid PVST+ and MST, the trade is instance count. Rapid PVST+ runs one spanning tree per VLAN, which gives per-VLAN load balancing at the cost of CPU and of a logical port count that grows with VLANs multiplied by trunks. MST maps many VLANs onto a few instances, which scales much better and is the interoperable choice; Cisco's current cloud campus guide recommends MST. Whichever you choose, run the same mode everywhere: a boundary between modes is a place where the topology is decided by fallback behaviour rather than by you.

root placement and the protection toolkit

Place the root yourself

Cisco's guidance is a single sentence: "Do not leave the STP to decide which bridge is root." Left alone, the root is elected by bridge priority and, on the usual tie, by the lowest MAC address, which means the root of your network is whichever switch happened to be manufactured earliest. That switch is frequently an access switch in a cupboard.

The root goes on a distribution switch, and the HA campus guide adds the constraint that makes the topology sane: "be sure to place the HSRP primary and the STP primary on the same distribution layer switch." If the root is on Dist 1 and the active gateway is on Dist 2, every frame from a host crosses to Dist 1, then back over the interlink to Dist 2, for no reason at all.

Set a secondary root on the other distribution switch, so that losing the primary produces a known topology rather than a new election.

The protection toolkit

Four features, each solving a different failure. They are not interchangeable and the places they belong do not overlap.

PortFast moves an edge port straight to forwarding instead of walking it through listening and learning. Cisco's caution is explicit: "Enable PortFast only on edge interfaces that do not connect to a Layer 2 bridging device." A PortFast port still runs spanning tree, so a switch plugged into one can still become root.

BPDU Guard closes that hole. It "disables a PortFast-configured port or interface if the port or interface receives a BPDU". The port is error-disabled and stays down until someone acts, which is the correct behaviour: a BPDU on a user port means something is wrong that a human should see.

Root Guard protects where the root may live. It keeps a designated port designated, so it can never become a root port. When a superior BPDU arrives, Cisco's documentation says the switch ignores it for root election and "moves this port to a root-inconsistent STP state", logging %SPANTREE-2-ROOTGUARDBLOCK. Recovery is automatic once the offending BPDUs stop. Enable it on every port where the root must not appear; together those ports draw a perimeter around the region where the root is allowed to be.

Loop Guard protects against a different failure entirely: a blocking port that stops receiving BPDUs because of a software fault rather than a link failure, and therefore ages out its information and transitions to forwarding, creating a loop. With Loop Guard, that port goes into a loop-inconsistent blocking state instead, and recovers automatically when BPDUs return. Enable it on root and alternate ports for every possible active topology, not only on the ports blocking today.

Root Guard and Loop Guard are mutually exclusive on a port, and Cisco notes that configuring Loop Guard overrides and disables Root Guard there. They apply to different port roles anyway: Root Guard to designated ports, Loop Guard to non-designated ones.

UDLD, and why it is not the same as Loop Guard

UDLD detects a link that passes traffic in one direction only, which on fibre is a common cause of bridging loops. Cisco's HA campus guide is unambiguous: "Enable UDLD aggressive mode in all environments where fiber optic interconnections are used."

The two features overlap but neither covers the other, and Cisco publishes the difference:

Failure Loop Guard UDLD
Unidirectional link, STP consequences Covered, if enabled on all root and alternate ports Covered, if enabled on all links in the redundant topology
Designated switch stops sending BPDUs through a software fault Covered Not covered
Miswiring Not covered Covered
Scope of action Per VLAN Per port

Use both. The combination is the only way to cover all three rows, and Cisco says as much: together they give the highest level of protection.

Keep the blocked ports few and the VLANs pruned

The risk in a Layer 2 design is concentrated in its blocked ports. Cisco puts it this way: "A single blocking port that mistakenly transitions to forwarding can melt down a large part of the network."

Two habits reduce the exposure. Prune unused VLANs from every trunk with an explicit allowed list, which Cisco warns is not the same as relying on VTP pruning. And bundle parallel links into a port channel, so that two physical links present one logical port to spanning tree and neither is blocked.

Finally, check the logical port count against the platform's limit. The number that matters is not VLANs and not ports but the product: trunks multiplied by active VLANs on each, plus non-trunking ports. Cisco's own formula and the show spanning-tree summary totals output are the way to measure it. A design that is comfortable on paper can exceed a supervisor's logical port budget purely through VLAN sprawl, and the symptom is instability rather than a clean error.

First hop redundancy

A host has one default gateway. It learns the gateway's MAC address once and caches it. If the router owning that address disappears, the host does not look for another one; it keeps sending frames to a MAC address nobody answers to, until the ARP cache entry ages out, which is far too long.

First hop redundancy protocols solve this by putting the gateway address on a virtual router that two or more physical routers take turns owning. The host never learns that anything changed, because the IP address and the MAC address both move.

active and standby gateway · one virtual address

The three protocols

HSRP VRRP GLBP
Specification RFC 2281, Informational, Cisco RFC 9568, which obsoletes RFC 5798 Cisco proprietary
Virtual MAC 00-00-0C-07-AC-XX, XX is the group 00-00-5E-00-01-XX for IPv4, 00-00-5E-00-02-XX for IPv6, XX is the VRID 0007.b400.xxyy pattern, one per forwarder
Transport UDP port 1985 to 224.0.0.2 IP protocol 112 to 224.0.0.18, or ff02::12 UDP port 3222 to 224.0.0.102
Default interval Hello 3 s, hold 10 s Advertisement 100 centiseconds Hello 3 s
Roles One Active, one Standby, others Listen One Active, the rest Backup One AVG, up to four AVFs
Election Highest priority, tie to highest IP Highest priority, tie to highest primary address Highest priority elects the AVG
Forwarding One router at a time One router at a time All forwarders at once

A few details in that table repay attention.

HSRP is Informational, not a standard. RFC 2281 says in terms that it "does not specify an Internet standard of any kind", and the IETF datatracker labels it Legacy. It is universal on Cisco equipment and absent or reimplemented elsewhere. If the design has to work across vendors, that decides it.

VRRP's terminology changed. RFC 9568, published in 2024, replaced RFC 5798 and renamed the Master role to Active. Documentation, command output and engineers will use both words for years.

VRRP's failover time is computed, not configured. The Backup declares the Active down after Active_Down_Interval, which RFC 9568 defines as (3 x Active_Adver_Interval) + Skew_Time, where Skew_Time is ((256 - Priority) x Active_Adver_Interval) / 256. The skew means a higher-priority backup declares failure slightly sooner, which breaks ties deterministically instead of by race. With the default one-second interval, a priority-100 backup waits about 3.6 seconds.

VRRP reserves priority 255 for a router that actually owns the virtual address as a real interface address. 100 is the default for a backup, 1 to 254 are the usable backup range, and a priority of 0 is the Active router announcing that it is standing down, which is what makes a graceful handover immediate rather than timed out.

GLBP forwards on every member. A GLBP group has one Active Virtual Gateway that answers ARP for the virtual IP, and it hands out a different virtual MAC to each requesting host. Cisco's documentation states the limit: "A GLBP group allows up to four virtual MAC addresses per group." Each gateway holding one is an Active Virtual Forwarder for that address. Load balancing is selected with glbp <group> load-balancing [host-dependent | round-robin | weighted].

Tuning, and what it costs

Default timers are slow. A three-second hello with a ten-second hold means a dead gateway can go unnoticed for ten seconds, which is a visible outage for anything stateful.

Cisco's HA campus guide recommends millisecond timers, giving standby 1 timers msec 200 msec 750 for HSRP and glbp 1 timers msec 250 msec 750 for GLBP. Those values are part of what produces the 800 to 900 millisecond campus convergence quoted in section 4.

Two cautions go with aggressive timers. They raise control-plane load on both switches, and they make the pair sensitive to anything that delays packet processing, so a switch under CPU pressure from a broadcast storm can lose its gateway role at the worst possible moment. And preemption needs a delay: a switch that has just rebooted will claim the active role as soon as the protocol starts, which can be well before its line cards, its routing adjacencies and its uplinks are ready. Cisco's advice is to measure the boot time and set the preempt delay 50 percent higher, with standby 1 preempt delay minimum 180 as the worked example.

The alignment rule, and the GLBP trap

In a Layer 2 access design the FHRP active node and the spanning tree root must be the same switch, for the reason given in section 8.2.

GLBP interacts badly with this, and Cisco's guide documents the trap. GLBP deliberately spreads hosts across both gateways, but spanning tree has blocked one access uplink, so the hosts assigned to the gateway on the far side of the blocked link send their traffic up the surviving uplink and then across the distribution interlink. Upstream traffic takes a two-hop Layer 2 path for half the hosts.

The fix in the guide is to arrange for the blocking port to be the distribution interlink rather than an access uplink, by raising its spanning tree cost on the secondary root, with spanning-tree cost 2000 on the interface facing the primary root. Both access uplinks then forward and GLBP does what it was installed to do.

This is worth noticing as a general pattern rather than a GLBP footnote. Two mechanisms that are individually correct, spanning tree choosing a loop-free topology and GLBP distributing gateways, combine into a design that is worse than either alone. Most campus design failures look like this.

The simplest answer

None of section 9 applies in a routed access design. There is one router on the subnet, it is the access switch, and there is no gateway to make redundant. This is the single strongest practical argument for routed access: an entire protocol, its timers, its preemption behaviour, its alignment rule with spanning tree and its interaction traps all stop being part of the design.

Routing design

Choosing a protocol

Protocol Status Where it fits
OSPF RFC 2328, Internet Standard Campus and enterprise, any vendor mix, where an area structure is wanted
EIGRP RFC 7868, Informational, Independent Submission, authored at Cisco Cisco-only estates, where fast convergence with less structure is wanted
IS-IS ISO standard, widely used by carriers Large service provider cores; rare in campus
BGP RFC 4271 and successors Between autonomous systems, at the internet edge, and as a fabric underlay

For a campus the realistic choice is OSPF or EIGRP, and the honest summary is that either works if the addressing plan is good, and neither saves you if it is not.

EIGRP is simpler to deploy because it does not impose an area structure, and the same property is its risk: without areas there is nothing forcing you to summarise, so an unsummarised EIGRP domain propagates queries across the whole campus. OSPF forces the structure on you, which is an irritation during the build and a protection afterwards.

RFC 7868 is worth knowing about for the same reason RFC 2281 is: it is an Independent Submission, Informational, not an IETF standard, and not endorsed by the IETF. EIGRP is documented, not standardised.

Mapping the hierarchy onto OSPF

Cisco's routed access design guide gives the mapping directly: each distribution block becomes its own OSPF area, with the distribution switches acting as area border routers, and the core in area 0.

OSPF areas on a campus hierarchy

This mapping works because the OSPF area boundary and the summarisation boundary and the distribution block boundary are all the same line. That is the principle from section 1 applied: one boundary doing four jobs.

The rule that constrains it is from RFC 2328 and restated plainly in Cisco's documentation: "All areas in an OSPF autonomous system must be physically connected to the backbone area (area 0)." An area that cannot reach area 0 directly needs a virtual link, and a virtual link in a campus is a sign that the physical design and the area design have drifted apart. Fix the topology rather than papering over it.

Area types, and which to use

Area type What is blocked
Normal Nothing
Stub No Type 4 or Type 5 AS-external LSAs
Totally stubby No Type 3, 4 or 5 LSAs except the default summary route
NSSA No Type 5, but Type 7 LSAs that convert to Type 5 at the NSSA ABR can traverse
NSSA totally stubby No Type 3, 4 or 5 except the default summary, with Type 7 still allowed

Cisco's routed access guide recommends totally stubby areas for distribution blocks, configured as area 120 stub no-summary, to keep the access switches' route tables minimal. Each ABR then injects a default route into the area, which is all an access switch needs: everything that is not local goes up.

The guide attaches a warning to that convenience, and it is a good one. The ABR generates the default route whenever it has an active area 0 interface. So an interface that ends up in area 0 by mistake, perhaps a link that should have been in the block's own area, makes that switch start advertising a default it cannot deliver on, and the result is a routing black hole rather than a visible failure.

Summarisation and the equal-cost trap

The summary is configured on the ABR with an area range, and Cisco's example is area 120 range 10.120.0.0 255.255.0.0 cost 10.

The cost 10 is not decoration. Without an explicit cost, each ABR derives the summary's cost from its own view of the component routes, and the two ABRs in a pair will usually arrive at slightly different numbers. The core then prefers one ABR over the other, and the design you drew with two equal paths is running on one. Cisco recommends a static cost specifically to guarantee equal-cost paths through both ABRs.

The EIGRP equivalent

EIGRP's problem is query scope. When a route is lost and there is no feasible successor, the router queries its neighbours, they query theirs, and the route stays active until every query is answered. The design job is to make sure a query stops quickly.

Cisco's routed access guide gives four mechanisms, and they work together:

  • Make access switches stubs. eigrp stub connected tells the distribution switches not to query them at all. The guide quantifies the effect: it limits distribution-generated queries to three or fewer per link failure.

  • Summarise at the distribution. A summary stops a query at the boundary, because the summarising router can answer it from its own table. The example is ip summary-address eigrp 100 10.120.0.0 255.255.0.0 on the uplinks to both core nodes. With that in place, core responses complete in under 100 milliseconds and access uplink convergence stays under 200.

  • Filter downwards. A distribute-list on the downlinks so access switches receive only a default route. Where no default exists, a single large summary serves instead.

  • Make neighbours passive by default. passive-interface default, then un-passive only the uplinks. This is a security measure as much as a scaling one: no adjacency is ever formed on a user-facing SVI.

Timers on top of that: the guide reduces EIGRP hello and hold to 1 and 3 seconds from the defaults of 5 and 15.

It also names the worst case honestly: a failure of the distribution-to-access link, because there is no equal-cost path that avoids a routing convergence. That is the same case section 4.2 identified for routed access generally.

OSPF timers, and the order to tune them in

OSPF's convergence is dominated by two throttles: how soon an LSA is generated after a change, and how soon SPF runs after an LSA arrives. Cisco's guide publishes measurements on the platform of the day, and the shape of the result is the useful part:

Configuration Measured convergence
Defaults 5.68 s
SPF start reduced to 1 s 1.7 s
SPF throttled from 10 ms 0.72 s
LSA and SPF throttling together 0.24 s

The recommended values are a start of 10 ms, a hold of 100 to 500 ms and a maximum wait of 5 seconds, applied to both throttles, with timers lsa arrival 80 set slightly below the generation hold so that an arriving LSA is not discarded as too frequent.

The important line in that guide is the caution, not the numbers: sub-second SPF timers should be used only with a well-structured area design, summarisation and link flap damping in place. Aggressive timers on an unsummarised, single-area network convert a small flap into a campus-wide CPU event. Structure first, then timers. Never the other way round.

The same guide recommends IP event dampening on every routed campus interface at its default values, and setting carrier-delay to 0 on critical interfaces so that a genuine link failure is reported immediately, with reduced protocol timers acting only as a backup to physical-layer remote fault detection.

The addressing plan

Everything in section 10 depends on this section, and this section has to be done first. You cannot summarise address space that was not allocated to be summarisable, and retrofitting an addressing plan onto a live campus means renumbering, which means touching DHCP scopes, access lists, firewall rules, monitoring, DNS, licence servers and the one application nobody documented.

Cisco's campus infrastructure guide puts the requirement in one line: use "structured IP subnet plans for easy expansion, better operation, management, and troubleshooting".

Allocate by block, not by need

The unit of allocation is the distribution block, and the size of the allocation is a power of two large enough to cover the block's eventual size, not its current one. Then every subnet inside the block comes out of that range and nothing else does.

A scheme that works, using 10.0.0.0/8:

Element Allocation Example
Distribution block A /16 per block Block 10 gets 10.10.0.0/16
Access VLAN A /24 inside the block's /16 VLAN 12 on closet 3 gets 10.10.12.0/24
Point-to-point links A /24 inside the block, split into /31s 10.10.254.0/24
Loopbacks A /24 inside the block, used as /32s 10.10.255.0/24
Distribution interlink Inside the block's /16, as section 6 requires 10.10.254.0/31

The block then advertises exactly one prefix, 10.10.0.0/16, and the core's routing table has one entry per block. Adding a closet, a VLAN or a link changes nothing outside the block.

Two details in that table are deliberate. Point-to-point links and loopbacks come from inside the block's own range, not from a separate global pool, because a separate pool cannot be summarised at the block boundary and so leaks one route per link into the core. And the distribution interlink is inside the range for the specific reason Cisco's guide gives, covered in section 6.1.

Summaries have to land on bit boundaries

A prefix can only be summarised at a bit boundary, which means allocations must be aligned as well as contiguous. 10.10.0.0/16 through 10.13.0.0/16 can be expressed as 10.10.0.0/14 only if the range starts at a multiple of four, which 10 is not: 10.8.0.0/14 covers 10.8, 10.9, 10.10 and 10.11.

This is why blocks should be allocated with gaps rather than packed. Four blocks allocated 10.0, 10.4, 10.8 and 10.12 can each later grow to a /14 without moving. Four blocks allocated 10.1, 10.2, 10.3 and 10.4 do not aggregate into any single prefix, and the first three cannot grow in place without colliding with each other.

IPv4 address space inside 10.0.0.0/8 is not scarce. Treat it as free and spend it on alignment.

Make the plan readable

A good plan can be read off an IP address without opening a spreadsheet. The conventions that achieve this are small:

  • Encode the distribution block in the second octet, so 10.20.x.x is always block 20.

  • Make the third octet match the VLAN ID where the numbering allows it, so VLAN 12 is 10.10.12.0/24 everywhere it appears.

  • Give the same role the same last octet in every subnet: .1 for the gateway, .2 and .3 for the physical interfaces behind an FHRP virtual address, a fixed range for infrastructure, a fixed range for DHCP.

  • Number loopbacks in one consistent sequence per device, because that is what appears in routing protocol output, syslog and traceroutes.

None of this changes how the network forwards. It changes how long it takes a tired engineer to understand an address at three in the morning, which is a real design property.

Routed links between switches need two addresses. A /30 provides two usable addresses out of four, wasting half. RFC 3021 defines 31-bit prefixes for point-to-point links, giving exactly the two addresses needed with no waste and no broadcast address, and it is supported on current platforms.

For a campus the saving is irrelevant, but the regularity is not: a /31 per link makes the link addressing uniform and keeps the point-to-point range small enough to sit comfortably inside the block's summary.

IPv6

The same structure applies with different arithmetic. Each subnet is a /64, without exception, because stateless address autoconfiguration and a good deal of other behaviour assumes it. The site gets a /48 or a /56 from the provider or from a unique local range, and the distribution block gets a fixed-size slice of that.

Summarise on nibble boundaries, meaning /48, /52, /56 and /60, because those fall on hexadecimal digit boundaries and are therefore readable. A /61 is as valid as any other prefix and nobody will ever read it correctly at speed.

The design error to avoid is treating IPv6 as a parallel universe with its own unrelated plan. The block that is 10.10.0.0/16 in IPv4 should be an obviously corresponding slice in IPv6, so that one topology diagram describes both.

Redundancy and convergence

Redundancy is not a property you add at the end. It is a claim about behaviour that has to be true for every component, and the design either supports the claim or it does not.

Detection is usually slower than reaction

A failure takes effect in three stages: something breaks, something notices, something reroutes. The third stage is fast on modern hardware. The second is where the time goes, and it is where the design choices are.

Mechanism What it detects Typical order of magnitude
Physical loss of light or carrier A cut fibre, a pulled cable, a dead optic Milliseconds
802.3z and 802.3ae remote fault A far-end failure on a fibre link Milliseconds
BFD Any forwarding-path failure between two peers Tens of milliseconds, configurable
Routing protocol hellos A peer that stopped responding Hundreds of milliseconds to seconds, by configuration
FHRP hellos A dead gateway 750 ms to 10 s, by configuration
Spanning tree A topology change About 1 s with Rapid PVST+, up to 50 s with 802.1D

The design preference follows from the table. Cisco's routed access guide makes it explicit: use point-to-point fibre with 802.3z or 802.3ae remote fault detection as the primary failure detection, set carrier-delay to 0 on critical interfaces so that detection is not artificially delayed, and reduce protocol timers only as a backup to the physical mechanism. The guide also recommends IP event dampening at default values on every routed campus interface, so that a flapping link does not translate every transition into a protocol event.

A media converter, a passive wavelength multiplexer, or a Layer 2 service in the middle of what the diagram shows as a point-to-point link breaks this chain. The far end goes away and the local port stays up, so detection falls back to protocol hellos and the convergence time multiplies. This is the most common reason a design that converges in 200 milliseconds in the lab takes nine seconds in the building.

Triangles, not squares

Cisco's phrase for the structural rule is "build triangles, not squares". A triangle means every node has an alternative path already installed in hardware, so a failure is handled by removing a next hop from an existing set rather than by computing a new path. A square means at least one node has exactly one way out, and its failure requires convergence.

squares against triangles · the same link failure

The difference in outcome is large. On an equal-cost triangle, recovery is the time to detect the failure plus the time to reprogram the forwarding hardware. On a square, it is the detection time plus a full protocol convergence, which section 10.6 shows is at best a few hundred milliseconds and at worst several seconds.

Cisco's HA campus guide is also direct about the economics: partial-mesh designs that save on links are "ultimately far more expensive".

Where single points of failure actually hide

The obvious ones get caught in review. These do not.

  • Both uplinks from one stack member. The stack is drawn as one box, so the diagram shows two uplinks and the reality is one switch.

  • Both uplinks in one conduit. Redundant fibre that shares a duct, a riser or a trench fails together. This is the classic finding after someone digs.

  • One supervisor, two line cards. Chassis redundancy means nothing if the control plane is single.

  • Two power supplies, one feed. Dual supplies plugged into the same PDU, or two PDUs on the same upstream breaker.

  • A shared control plane. StackWise Virtual, VSS and stacking all present two chassis as one device. That removes blocked ports and simplifies the topology, and it means a control-plane failure or a bad upgrade takes both halves. Dual-active detection on an independent path is not optional in these designs; it is the thing that stops a split brain from becoming a forwarding loop.

  • Identical software everywhere. A software defect is a common-mode failure, and a campus running one version on every switch will meet it on every switch at once. This is an argument for staged upgrades, not for version sprawl.

  • One person who understands it. Noted in section 1 and worth repeating here, because no amount of hardware addresses it.

The test that matters is the maintenance window

A redundant design is not proven by surviving a failure. It is proven by surviving a planned reload of one half while carrying production traffic, in the middle of the working day, without anyone noticing.

That test exercises everything at once: whether the remaining half has the capacity, whether the routing converges in the time claimed, whether FHRP preemption delays are long enough that the rebooted switch does not take traffic before it can forward, whether the applications tolerate the convergence, and whether the monitoring actually reports what happened.

If a design cannot pass that test, the redundancy is a line in a document rather than a property of the network. Most designs that fail it fail on capacity, because the sizing was done against the total and not against the surviving half.

The data centre is a different shape

Everything so far assumes the campus traffic pattern: many endpoints talking to a few places, mostly north and south through the hierarchy. A data centre has the opposite pattern. A single user request fans out into dozens of calls between servers that sit on different racks, and that traffic is east to west, between leaves of the tree rather than up and down it.

In a three-tier hierarchy, server-to-server traffic between two racks in the same block goes up to the distribution pair and back down. Between blocks it goes up to the core and back. The path length depends on where the two servers happen to be, so latency varies by placement, and the aggregation layers carry traffic that has no business being there.

spine and leaf fabric · four spines, four leaves

What the topology guarantees

A spine and leaf fabric is a two-stage Clos network. Cisco's massively scalable data centre paper describes the arrangement as one where "every leaf layer switch is connected to each of the spine layer switches in a full-mesh topology", with the spine layer "responsible for interconnecting all leaf switches".

Three properties follow directly from that shape, and they are the reason it is used:

Uniform path length. Any server reaches any other server in exactly two hops: up to a spine, down to a leaf. Latency becomes a property of the fabric rather than of where a virtual machine happened to be scheduled, which is what makes workload placement a capacity question instead of a network question.

Bandwidth scales by adding spines. Each additional spine adds one more parallel path between every pair of leaves. The paper notes that leaf and spine are connected with Layer 3 IP connectivity to provide equal-cost multi-path routing, so the fabric uses every spine simultaneously rather than keeping some in reserve.

Ports scale by adding leaves. A new rack is a new leaf with uplinks to the existing spines. No other leaf is reconfigured.

The constraint is the arithmetic: the number of leaves a fabric can hold is bounded by the port count of a spine, and the number of uplinks per leaf is bounded by the number of spines. Cisco's examples give a sense of the scale this reaches, with one two-tier design holding 288 leaves and 13,824 single-homed servers in a pod, and three-tier designs with a super-spine layer running to hundreds of thousands of servers.

Oversubscription is explicit here

In a campus, oversubscription is a ratio you check. In a fabric, it is a number you choose when you select the leaf switch, because it is fixed by the ratio of downlink to uplink ports on that switch.

Cisco's worked examples make this concrete: a leaf with 48 downlink ports at 25 Gbps and 8 uplinks at 100 Gbps is 1,200 Gbps down against 800 Gbps up, which is 1.5:1. Their designs run from 1:1 for non-blocking AI fabrics to 3:1 for general-purpose top-of-rack switches, with the pod-to-pod ratio in larger fabrics set by how many spines are populated.

That is a cleaner way to think about oversubscription than the campus version, and it is worth importing: state the ratio as a design decision, write it down, and check it against the actual traffic rather than inheriting whatever the chassis happened to provide.

Routing the fabric

The underlay is routed, and in large fabrics it is routed with BGP. Cisco's paper reports that "EBGP has been chosen by many MSDCs as the only routing protocol", following RFC 7938, with single-hop eBGP sessions over point-to-point Layer 3 links between tiers and private AS numbers from the 64512 to 65534 range.

The choice of eBGP over an IGP in this role surprises people who learned BGP as an internet protocol. The reasons are practical: the AS path gives loop prevention that scales to thousands of nodes without flooding a link-state database, policy is applied per neighbour, and the protocol's behaviour under churn is well understood at very large scale.

AS numbering has two common patterns, and Cisco documents the trap in the second. In the multi-AS design each leaf or leaf pair has its own AS and all spines share one, which works cleanly. In the dual-AS design all the spines in a pod share one AS and all the leaves share another, which, as the paper puts it, "breaks how eBGP works", because a route leaving a leaf and arriving at another leaf has an AS path that starts and ends in the same AS and is therefore discarded as a loop. The workarounds are as-override on the spine or allowas-in on the leaf, and the better answer is usually to choose the first pattern.

Do not import the fabric into the campus

Spine and leaf solves a traffic pattern the campus does not have. A floor of desks does not generate east-west traffic between closets; it generates north-south traffic to the data centre and the internet. Building a campus as a fabric adds cost and complexity for a property nobody needs.

The reverse mistake is more common and more expensive: building a data centre as a campus hierarchy, discovering that server-to-server latency depends on rack placement, and solving it by adding links until the aggregation layer is an accidental and unmanaged partial mesh.

The internet and WAN edge

The edge is where the design stops being entirely yours. Beyond it are providers with their own policies, address space you do not own, and failure modes you cannot instrument.

Structurally it is not special. The edge is a block that attaches to the core like any other, gets summarised like any other, and is sized like any other. What makes it different is that it is the only block where the policy boundary and the trust boundary coincide, which is why the firewalls live here and not in the core.

internet and WAN edge block

Keep the two edges separate

The internet edge and the WAN edge do different jobs and should be built as different blocks, even when they terminate in the same room.

The internet edge handles untrusted traffic, so it carries the firewall pair, the public address space, NAT, inbound published services and remote access. The WAN edge handles traffic to other sites that you also operate, so its policy is about routing and quality of service rather than about trust. Merging them produces a device that is simultaneously a routing policy point and a security boundary, and a change to either job becomes a change to both.

What each component is for

Component Job Design note
Edge routers Terminate the provider circuits and run BGP with them Two routers, ideally two providers, ideally diverse physical paths
Firewall pair Enforce the trust boundary Stateful, so both directions of a flow must traverse the same unit
DMZ Host published services away from the internal network Its own segment off the firewall, never a VLAN trunked into the campus
WAN routers Terminate site-to-site transport Where SD-WAN overlays, QoS policy and path selection live
Core attachment Join the block to the rest of the network Two links from each device to the two core switches, as everywhere else

Routing at the edge

Two directions, two different problems.

Inbound to the campus, what the edge sends to the core should be as little as possible. A default route is usually right, sometimes with a handful of specifics for WAN destinations. Do not redistribute the internet table into the campus IGP; there is no decision the campus can make with it that a default route does not already make.

Outbound to the providers, advertise only the address space you are entitled to advertise, with a prefix list that lists it explicitly rather than a filter that excludes what you can think of. Apply the same filter inbound: accept a default, or accept a filtered set, and never accept a full table you have no use for on a device sized for a default.

With two providers, the usual asymmetry is worth planning for rather than discovering. Outbound path selection is yours to make with local preference. Inbound path selection is the internet's to make, and you influence it only weakly with AS path prepending and multi-exit discriminators. A design that depends on symmetric inbound routing across two providers is a design that depends on other people's policy.

Stateful devices and asymmetric paths

The single most common edge failure is not a failure at all. It is a stateful firewall seeing one direction of a flow and not the other, dropping the half it does not recognise, and producing a fault that looks intermittent and application-specific.

This happens when there are two firewalls and more than one path between the same two points, which is exactly what redundancy produces. The ways out are to make the pair a single stateful unit with a synchronised session table, to force both directions through the same unit with routing metrics, or to remove the second path. All three are valid; having none of them is not.

The same reasoning applies to anything else that holds state in the path: load balancers, WAN optimisers, inspection devices and NAT. A stateless network tolerates asymmetry. A stateful device in that network does not, and the design has to decide which property it is going to have.

What not to do

  • Do not make the firewall the core. It is tempting when the firewall has enough ports. It makes every inter-VLAN flow in the campus depend on a device that is slower, harder to upgrade and far more likely to be reconfigured than a core switch.

  • Do not trunk the DMZ into the campus. A DMZ VLAN that appears on a campus access switch has moved the trust boundary to a cupboard.

  • Do not let the edge become the services block. Wireless controllers, load balancers and the rest belong in their own block, as section 7.4 describes, not bolted onto the device that faces the internet.

Validating the design

A design is a set of claims. Validation is checking that each one is true before it is built, and the useful form of the exercise is adversarial: assume the design is wrong and go looking for where.

Questions a design should survive

On structure

  1. Draw one distribution block. Is every other block identical to it? If not, why, and is the reason still valid?

  2. Where exactly does Layer 2 stop? Point at the line. Is it in the same place in every block?

  3. Does any VLAN appear on more than one access switch? For each one that does, what requires it?

  4. Is there a distribution-to-distribution link, and is its address space inside the block's summary?

On failure

  1. For each link in the drawing, what happens when it fails, how long does it take, and how do you know?

  2. For each device, the same question, including during a planned reload.

  3. Can the surviving half carry the full load? Not in principle: with the actual measured peak.

  4. Which two components fail together because they share a conduit, a power feed, a supervisor, a control plane or a software version?

On routing

  1. What does the core's routing table look like? One entry per distribution block, or one per VLAN?

  2. If a single access VLAN flaps, which devices run SPF or send a query? If the answer is more than the local block, the summarisation is not working.

  3. Are the two paths through the distribution pair genuinely equal cost, including after summarisation?

On addressing

  1. Can each block's address space be expressed as one prefix? Try writing it down.

  2. Can each block double in size without moving?

On operations

  1. Can someone who did not design it work out where a given IP address lives, from the address alone?

  2. How many people can perform the most dangerous routine change? If the answer is one, that is the network's weakest component.

  3. What does the monitoring see when each of the failures in questions 5 to 8 happens? A failure that produces no alert is a failure you will find out about from a user.

The mistakes that recur

These are the ones that show up again and again, across organisations and across decades.

The VLAN that spans everything. It usually starts with one legitimate requirement and grows, because once a VLAN is on the trunk it costs nothing to extend. Cisco's advice is to "only span VLANs across multiple access layer switches if you must", and the guide notes the consequences when you do: asymmetric routing and unicast flooding. The mitigation it offers, setting the ARP aging timer at or below the CAM aging timer, is a treatment rather than a cure.

Summarisation left for later. The addressing plan is made first, the summarisation is deferred to the implementation phase, and by then the DHCP scopes, firewall rules and static routes have been built against the unsummarised layout. This is the single most common reason a campus with a textbook topology still takes seconds to converge.

The core that acquired a job. One server, one firewall, one access list, one VLAN. Each addition is individually reasonable, and the aggregate is a core that cannot be upgraded without a change window involving half the organisation.

The redundancy that was never tested. Second supervisor never failed over, second power feed never proven, second path never carried traffic. Redundancy that has not been exercised is a hypothesis.

The spanning tree nobody configured. The root is wherever the election put it, no protection features are enabled, and the design is whatever the switches negotiated. This network works until someone plugs a small switch into a meeting room port.

Timers tuned before structure. Aggressive SPF and hello timers on an unsummarised, single-area network. Cisco's warning is that sub-second SPF timers belong only with a good area design, summarisation and flap damping in place. Without them, tuning converts small local events into campus-wide CPU load.

Two correct mechanisms combining badly. The GLBP and spanning tree interaction in section 9.3 is the clean example. Each feature is doing exactly what it was designed to do, and the result is wrong. This class of fault is only found by walking a packet through the design by hand, which is the main argument for doing exactly that before building it.

A design that only exists in one person's head. The drawing is out of date, the reasoning was never written down, and the one engineer who knows why the interlink cost is 2000 has left. Every item above is recoverable. This one is not.

Quick reference

Design numbers

Figure Value Source
Access to distribution oversubscription 20:1 Cisco enterprise campus infrastructure guide
Distribution and core oversubscription 4:1 Cisco enterprise campus infrastructure guide
Three tiers worth building at More than two distribution blocks Cisco enterprise campus infrastructure guide
Layer 2 access convergence, optimal 800 to 900 ms Cisco routed access design guide
Layer 3 access convergence Under 200 ms Cisco routed access design guide
Rapid PVST+ convergence About 1 s Cisco HA campus design guide
802.1D convergence Up to 50 s Cisco HA campus design guide
802.1D default timers Max age 20 s, forward delay 15 s per state Cisco STP tech note
802.1D network diameter 7 Cisco STP tech note
Leaf oversubscription, 48 x 25G down and 8 x 100G up 1.5:1 Cisco MSDC white paper

First hop redundancy at a glance

HSRP VRRP GLBP
Transport UDP 1985 to 224.0.0.2 IP protocol 112 to 224.0.0.18 or ff02::12 UDP 3222 to 224.0.0.102
Virtual MAC 0000.0c07.acXX 0000.5e00.01XX IPv4, 0000.5e00.02XX IPv6 0007.b400.xxyy
Default interval Hello 3 s, hold 10 s Advertisement 1 s Hello 3 s
Cisco tuned campus timers standby 1 timers msec 200 msec 750 glbp 1 timers msec 250 msec 750
Priority notes Higher wins, tie to higher IP 255 is the address owner, 100 default, 0 means standing down Highest priority becomes the AVG
Active forwarders 1 1 Up to 4 per group

Routing protocol tuning, as Cisco publishes it

Setting Value
EIGRP hello and hold 1 s and 3 s, from defaults of 5 and 15
EIGRP access switch role eigrp stub connected, limiting queries to three or fewer per link failure
EIGRP distribution summary ip summary-address eigrp 100 10.120.0.0 255.255.0.0 on both core uplinks
OSPF access area area 120 stub no-summary, a totally stubby area
OSPF summary area 120 range 10.120.0.0 255.255.0.0 cost 10, with the cost set explicitly
OSPF SPF throttle timers throttle spf 10 100 5000
OSPF LSA throttle timers throttle lsa all 10 100 5000
OSPF LSA arrival timers lsa arrival 80
Interface hygiene carrier-delay 0 on critical links, IP event dampening at defaults everywhere

Glossary

Term Meaning
ABR Area border router, an OSPF router with interfaces in more than one area
ASBR Autonomous system boundary router, which advertises external destinations into OSPF
AVG, AVF GLBP's active virtual gateway, which hands out virtual MACs, and the active virtual forwarders that use them
BPDU Bridge protocol data unit, the frame spanning tree runs on
CEF polarisation Consecutive tiers hashing on the same fields, so the same flows take the same side of every pair
Clos The non-blocking multistage switching topology that spine and leaf implements
Collapsed core Distribution and core functions in one pair of switches
Distribution block One distribution pair and the access switches it aggregates; the unit of failure, change and address allocation
ECMP Equal-cost multi-path, using several equal paths at once
FHRP First hop redundancy protocol, the family containing HSRP, VRRP and GLBP
Leaf, spine The access and interconnect tiers of a data centre fabric
MST Multiple Spanning Tree, IEEE 802.1s, mapping many VLANs onto few instances
Oversubscription Edge capacity divided by uplink capacity
Routed access A design where the Layer 3 boundary and default gateway sit on the access switch
Stub area An OSPF area into which external LSAs are not flooded
Totally stubby area A Cisco refinement that also blocks inter-area summaries, leaving only a default route
UDLD UniDirectional Link Detection, which finds links passing traffic one way only

References

Every figure, quotation and configuration example in this guide comes from one of the documents below. Where a Cisco design guide is cited, note the caveat in section 1.3: the structural guidance holds, the platform and release specifics do not.

Cisco design guides

Cisco technical notes

Standards

IEEE 802.1D, 802.1w, 802.1s and 802.1Q, and IEEE 802.3af, 802.3at and 802.3bt, are referenced by designation. They are published by the IEEE and are not freely redistributable; the behaviour described here is taken from the Cisco documents above.

share this note

LinkedIn WhatsApp X Email