Stratus everRun Explained: Fault Tolerance, Its Limits, and the Network Underneath

Fault tolerance protects a virtual machine from the failure of the hardware underneath it. It does not protect the VM from what happens inside it. This article explains how Stratus everRun works, what its HA and FT modes really do when a server dies, how quorum prevents split-brain, how everRun compares with VMware, Hyper-V and Proxmox, and how to design the network underneath an everRun pair. Every everRun statement here is taken from Stratus’s published documentation; where the documentation is silent, the article says so instead of guessing.

everRun in two minutes

everRun is software that joins two ordinary x86 servers into one highly available or fault-tolerant system. It was built by Stratus and is now sold by Penguin Solutions. It runs its own virtualization platform based on KVM, and you choose, per virtual machine, whether that VM gets high availability (HA) or fault tolerance (FT).

Stratus positions the two levels like this:

The availability percentages are Stratus’s own figures, not independent measurements. The everRun 8.0 User’s Guide also lists a third mode, unprotected operation, in which a VM runs on only one node and does not fail over if that node fails; Stratus recommends it only for special cases such as software licensing that prevents running on both nodes.

The terms you will meet throughout:

How everRun works under the hood

An everRun system is two complete servers that run the same software and hold the same data, memory and storage. Dedicated links between them keep those copies identical, and the everRun software on each node decides which node actively serves each VM.

The software stack on each node

• Host operating system. Both nodes run the same Linux host OS: CentOS in the 7.x documentation, Ubuntu in the everRun 8.0 User’s Guide.
• Virtualization. everRun uses an integrated virtualization platform based on KVM. KVM is the Linux kernel module that lets the kernel act as a hypervisor, using the CPU’s hardware virtualization extensions (Intel VT or AMD-V); it emulates very little hardware itself and relies on QEMU, a user-space program, for device emulation. The everRun 8.0 guide states that QEMU performs the hardware virtualization and executes guest code directly on the host CPU, and that users should make no changes to the QEMU engine or its configuration.
• Guest VMs. Windows and Linux guests run on top.
• Availability software and the AX. Stratus’s SplitSite documentation refers to an AX for each guest VM on each node, so every protected VM has an AX pair. Quorum decides which AX of the pair is active, and split-brain is defined as both AXs of a pair being active at once. An independent 2015 article expands the name as Stratus Availability Extensions. Stratus does not publish the AX’s internal design in its user guides, so this article does not describe it further.
• Fault handling. Stratus states that the software automatically detects, isolates and handles most hardware faults, and can notify the Stratus support centre of the fault type and its location.

The networks

everRun’s network segmentation logic detects a fault when the expected packet flow is interrupted even though both sides still show an active network link, so it can catch failures that a simple link-down check would miss.

Storage

Each node uses its own local storage, and everRun keeps the two copies in step over the A-Links rather than relying on a shared SAN. The RAID controller in each node builds logical disks from the physical disks. Logical disks are collected into storage groups, and they hold both the everRun system volumes and the VM volumes. Storage replication traffic between the nodes travels over the A-Links. The two nodes may have different disk capacities, but only the smaller capacity is usable in a storage group.

What Stratus does not publish

The user guides describe what the system does, not every internal component. They do not document the internal design of the AX, the exact replication algorithm used for FT, or the names and roles of internal services and log files. An independent 2015 article describes the FT state replication as “statepointing”. For anything deeper, rely on Stratus support rather than reverse-engineering.

Running mission-critical systems on everRun

Why choose fault tolerance

Many systems that must run around the clock depend on one or two central servers: a core application, its database, and the services that let users sign in. If one of those servers stops, everything that depends on it stops too.

Those applications were usually never written to be clustered. everRun protects them at the platform level: the VM runs on a pair of servers, and an FT VM survives the loss of either server. For an environment without a SAN or a large IT team, that is the main attraction.

An example layout

The protection level is chosen per VM, by asking how much a few minutes’ interruption would cost:

The A-Links run between the two nodes, the business and management networks go to a redundant switch pair, and the quorum service runs on a separate computer outside the everRun pair.

Operating an everRun pair day to day

On a healthy system there is little to do, which is the point. The routine work falls into a few areas:

• Loss-of-redundancy alerts. A node, a volume or an A-Link goes into a degraded state. The VMs keep running, but the pair is no longer redundant until the problem is fixed and the copies are back in sync. Treat these as urgent even though users notice nothing.
• Never take the healthy node down while the other is degraded. Before maintenance or a reboot, check the everRun Availability Console: both nodes healthy, volumes in sync, A-Links up.
• Planned maintenance one node at a time. Put one node into maintenance mode, work on it, return it, wait for full synchronisation, then do the other. In everRun 8.0, putting the preferred node into maintenance shuts down any unprotected VMs, so plan for them.
• Logs come from three places. An investigation needs the everRun logs from both nodes plus the guest OS and application logs. Keep time synchronised across all of them, or the timelines will not line up.
• Faults inside the guest are a separate matter: FT does not cover them, as a later section explains.

HA vs FT: what actually happens when a server dies

The difference is not “how fast it fails over”. An HA VM restarts; an FT VM never stops.

HA (high availability). The VM is active on one node, and its disk volumes are replicated to the other node over the A-Links. Stratus states that HA eliminates downtime for many, but not all, CPU, memory, I/O and other hardware failures; some faults need an automatic VM reboot to recover. If the node running the VM fails, the VM boots on the surviving node, and Stratus describes a brief period of downtime while the guest boots. A boot means the guest starts fresh: open sessions and anything held only in memory are lost, while data already written to disk is on the replicated volume. Stratus positions HA for applications that can tolerate occasional interruptions of a few minutes.

FT (fault tolerance). everRun runs an identical instance of the VM on the second node and keeps it in step over the A-Links. If a node fails, the instance on the other node simply carries on. Stratus states that FT eliminates downtime from any CPU, memory, I/O or other physical-machine failure and ensures no data loss.

Two details matter in practice:

• After any failure, the pair is no longer redundant. The survivor is now a single point of failure until the failed node is repaired and both copies are back in sync. The console shows this; treat it as urgent.
• The quorum server decides some edge cases. When the nodes lose contact with each other, the quorum server decides which node keeps each VM running. The next section explains how.

Quorum and split-brain

What split-brain is

Each node can only judge its partner through the links between them. If those links fail while both nodes are still running, each node sees silence and cannot tell whether its partner has died or has simply become unreachable. If both decide the partner is dead, both make themselves active for the same VM.

That is split-brain. Stratus defines it as both AXs of a guest VM’s AX pair being active at the same time: the same VM runs independently on both nodes. The two copies run independently, so their data can diverge. When the nodes are reconnected, they must resynchronise from one side, and the data written on the other side is lost.

Split-brain is a risk for any two-node system, so every two-node design adds an independent check. Switch pairs such as Dell VLT and Cisco vPC use a separate keepalive or heartbeat path between the two switches. everRun uses a quorum server: a third computer outside the pair.

How the quorum server prevents it

The quorum server is a Windows-based service that runs on a separate computer, outside the everRun pair, and both nodes reach it over the management network. When the nodes lose contact with each other, the quorum server arbitrates which node’s AX stays active for each VM. Stratus states that correct use of a quorum server is the only guaranteed way to prevent split-brain.

These three outcomes come from Stratus’s own SplitSite examples:

1. No quorum server. Both nodes lose the A-Links. Each one stops seeing its partner and makes itself active: split-brain. Data written on the node that later becomes standby is lost during resynchronisation.
2. Quorum server reachable from one node. node0 has lost every network path; node1 has lost the A-Links but can still reach the quorum server. node0 shuts down, node1 gets an answer from quorum and stays active. No split-brain, and the VM keeps running.
3. Quorum server unreachable from both. node1 cannot confirm with quorum either, so it terminates the VM. The VM is now down on both nodes. Split-brain is prevented, but the service is unavailable until a node or the quorum server is restored.

Other documented behaviour:

• Quorum lost, nothing else wrong. The quorum server can become unreachable without any other fault, for example while the quorum computer reboots for an OS patch. The AX then suspends synchronisation traffic until the connection returns. The VM keeps running on the node where it was active, but it does not move to the standby node, because further faults could occur. Synchronisation and normal fault handling resume once quorum is back. Patch quorum computers one at a time and keep the window short.
• Restart after a power loss. A restarted everRun system waits indefinitely for its partner before starting any guest VM. If the node that was previously active can reach the quorum server, it starts the VM immediately. Without a quorum server, a person must force the VM to boot, and the vendor warns that two people must never force-boot the same VM on both nodes, or they create split-brain themselves.
• Only two sites. With only two sites, if one node fails and the surviving node cannot reach the quorum server, the VMs at the surviving site are shut down automatically to avoid running in split-brain. This is why quorum belongs at a third site.
• HA and FT after a takeover. When the surviving node takes over, HA VMs boot (a brief outage); FT VMs continue to run.

Placing the quorum server

• Configure two: a preferred and an alternate quorum server, as Stratus recommends. A system can use zero, one or two.
• Reach them over the management network (ibiz0) using IPv4. Open the quorum UDP ports in every firewall in the path: the 7.4 guide documents UDP 4557, the 7.9 network requirements list UDP 4557 and 4558, and the ports can be changed in the everRun Availability Console.
• Keep round-trip latency to the quorum server under 500 ms; a 10 Mb link or T1 is adequate.
• Keep the quorum computers powered on at all times. If the preferred and alternate share a site, power them from separate AC sources or separate UPS units.
• Never run the quorum service as a VM on the everRun system it protects. Stratus also advises against running it as a guest VM on a different everRun pair, because a failure there would make the quorum VM fail over; if you do, that pair needs its own, second quorum computer. One quorum computer may serve many everRun systems.
• For SplitSite, place the preferred quorum server at a third facility and the alternate at a fourth (or carefully placed in the third). It must not depend on a switch or router at either node’s site.

What if the problem is inside the VM?

The short answer: if something goes wrong inside an FT VM, it goes wrong on both nodes. That is not a flaw in everRun. It is what fault tolerance is designed to do, and it is the same on every other FT platform.

Why both copies fail together

An FT VM is one virtual machine with two physical instances kept in step. Whatever the guest does on node0, it does on node1: the same instructions, the same memory, the same disk writes. That is exactly what lets the node1 instance take over a hardware failure without losing a single transaction.

The same property works against you when the fault is in the software. A hung application, a blue screen or a memory leak is part of the VM’s state, so it is copied too. To protect against a software fault, FT would have to tell “good” behaviour from “bad” and copy only the good. It cannot; it copies everything.

Meanwhile the platform sees nothing wrong. Both nodes are healthy, both instances are in sync, and the everRun console shows the VM as running. The VM is available. The service is not.

What FT covers, and what it does not

A note on wording: Stratus’s FT page says FT “transparently protects a VM from all faults”. Read in context, the faults that page lists are CPU, memory, I/O and other physical-machine failures. It does not claim to protect the guest operating system or application from their own faults, and by design it cannot.

Is it the same on other hypervisors?

For fault-tolerant VMs, yes. VMware vSphere FT works the same way, with a primary and a secondary VM kept in sync on two hosts, and it does not protect against guest OS or application failure: a crash in the primary generally means a crash in the secondary.

For platforms that only offer HA (Hyper-V Failover Clustering, Proxmox VE HA, and vSphere HA), the picture is different in an interesting way. They cannot survive a host failure without a reboot, but a reboot is exactly what cures many software hangs. vSphere HA VM Monitoring watches heartbeats from VMware Tools and disk and network activity; if both stop, it resets the VM automatically. Hyper-V Failover Clustering monitors the VM’s health and can restart it. So:

Neither is “better”. FT is unbeatable for hardware faults and does nothing for software faults; restart-based HA is weaker for hardware and can recover some software hangs on its own. Check what guest monitoring your everRun release offers; the everRun documentation cited in this article does not describe automatic restart of a hung guest, so confirm the behaviour with Stratus for your release.

How to protect what FT cannot

FT is the bottom layer of the answer, not the whole answer. The layers above it have to be built deliberately:

1. Application-level redundancy. Two instances of the application or database, so one can fail while the other serves (for example database replication or clustering, or two load-balanced application servers). This is the only thing that fully protects against an application fault, and it requires the application to support it.
2. Guest OS clustering. Windows Server Failover Clustering inside two VMs protects against guest OS and application failure, and allows rolling patching.
3. Automatic recovery inside the guest. Windows service recovery settings (restart the service on failure), watchdog scripts, and health checks that restart a stuck service before users notice.
4. Monitoring of the service, not just the VM. Alert on the application’s own health: response time, queue length, connected clients. A green VM in the console proves nothing about the service.
5. Capacity and scheduling. Size the VM for its worst moment, not its average. Test resource-heavy scheduled tasks under load and run them outside peak hours.
6. Change control. Test patches and configuration changes before production; FT will apply a bad change to both instances instantly.
7. Backups and point-in-time recovery. FT replicates corruption and deletion perfectly. Only a backup can take you back.

If someone asks “is the application redundant?”, the accurate answer is: the hardware under it is fully redundant; the application itself is still a single instance unless it is made redundant at the application or guest OS level. Saying that clearly, before go-live, builds more trust than promising five nines for everything.

everRun vs VMware vSphere vs Hyper-V vs Proxmox

everRun is built for one job: making a single pair of servers behave like one server that does not fail. The mainstream hypervisors are built to run many VMs across many hosts, with availability as one feature among many. That difference explains most of the table.

Where everRun stands out: FT on two commodity servers using their own local storage, with no SAN to run. For an environment with a handful of critical servers, that simplicity is the main reason to choose it.

Where the others stand out: scale, ecosystem and flexibility. They run many VMs across many hosts, move workloads freely, and integrate with far more backup, monitoring and automation tools. Only vSphere matches everRun’s zero-downtime mode, and it brings the same limit: FT protects the hardware, not the guest.

The fault-inside-the-VM behaviour from the previous section is therefore not an everRun weakness. Any replication-based FT, from any vendor, shares it. The real choice is between FT (best for hardware faults) and restart-based HA (can also recover some software hangs), plus whatever you build inside the guest.

Redundancy at every layer

Every availability technology protects one layer and assumes the layers above and below are handled by something else. A system is only as available as its weakest unprotected layer.

The same idea runs through a switch pair and a server pair. A Dell VLT or Cisco vPC pair lets either switch fail without dropping a dual-homed link; everRun lets either server fail without stopping an FT VM. Both work by keeping two independent boxes in sync over a dedicated link (the VLTi or peer-link, the A-Links), and both use a separate path (the keepalive or heartbeat, the quorum server) to tell a broken link from a dead partner.

They also share the same limit: neither looks inside what it carries. An MLAG pair will faithfully forward a broadcast storm on both switches; an FT pair will faithfully run a hung application on both servers. Protection at one layer never substitutes for protection at the layer above.

Two practical consequences:

• Match the failure domains. If both everRun nodes plug into the same switch, a single switch failure takes out the whole pair’s network, and FT cannot help. Spread the nodes across the switch pair, and spread the switches across power feeds.
• Know which layer failed. If an application hangs while both nodes are healthy, the server layer has done its job and the fault is in the guest. Naming the layer correctly in a root cause analysis keeps the fix in the right place and stops a platform being blamed for a problem it was never designed to solve.

Designing the network for an everRun pair

An everRun pair has two kinds of network with opposite needs. The A-Links must be private, fast and short. The business and management networks must reach the rest of the LAN and survive a switch failure. Design them separately.

A-Links: keep them private and short

The A-Links carry the replication traffic between the nodes: disk synchronisation, FT memory synchronisation and VM migration. Stratus warns that if the A-Link requirements are not met, guest VMs may run more slowly because synchronisation bandwidth is limited. If every path between the nodes fails, the quorum server decides which node keeps each VM (see the quorum section above).

The simplest design is also the most robust: two direct cables between the nodes, from two different NIC cards, with no switch in the path. Then no switch failure, misconfiguration or firmware upgrade can touch the sync traffic.

Business and management networks: ride the switch pair

The business networks carry the VMs’ own traffic. One of them, ibiz0, is the management network: it carries the everRun Availability Console and quorum traffic.

• Same Layer 2 on both nodes. Stratus requires both nodes to be on the same subnet on the management network (ibiz0). A guest VM keeps its own IP configuration whichever node is serving it, so each business network it uses must also be the same VLAN and subnet on both nodes, not routed between them. A VLT or vPC pair provides exactly this: the VLAN spans both switches through the VLTi or peer-link.
• Allow IPv6 multicast between the nodes on ibiz0, as Stratus requires. Check that multicast snooping or filtering on the switches does not block it.
• Spread the nodes across the switches. Connect node0 to Switch A and node1 to Switch B, so losing either switch removes one path, never both. everRun’s network segmentation logic is designed to detect network faults; confirm in commissioning tests exactly how your release behaves when one node loses a business link.
• Keep guest traffic off ibiz0 where you can. ibiz0 is the business network that carries management and quorum traffic. Putting guest VMs on other business networks keeps that traffic away from production load.

SplitSite: when the nodes are in different rooms

Stratus treats a system as a SplitSite configuration when its two nodes are connected through network infrastructure rather than direct cables, typically because they sit in different rooms or buildings. That protects against fire, power or cooling loss in one room, at the cost of stricter network rules:

• A-Links must be point-to-point and are never routed; in SplitSite they are implemented as isolated VLANs or dedicated fibre.
• The private network uses dedicated point-to-point fibre or a private VLAN, through non-routed, non-blocking switches with no filtering between the everRun ports.
• The latency limits above apply end to end. Stratus’s rule of thumb is about 1 ms per 100 miles of fibre, plus switch and converter delay.
• Stratus recommends a quorum server for every SplitSite, at a third facility that does not depend on either site’s switches or routers. Without it, the two-site shutdown behaviour described in the quorum section applies.

Key principles and checklist

Five principles for running everRun in production:

1. FT is a hardware promise, not a service promise. Make this clear to application owners and stakeholders before go-live.
2. Choose HA or FT per VM, by the cost of a few minutes’ outage. FT for the one or two servers everything depends on; HA for the rest.
3. A degraded pair is a single server. Treat any loss of redundancy (a node down, a volume out of sync, an A-Link down) as urgent, and never take the healthy node offline while the other is degraded.
4. The network is half the design. Private, direct A-Links; business networks spread across a VLT or vPC pair; quorum outside the blast radius.
5. Plan for faults inside the VM. FT does not cover them, so capacity, scheduled tasks, patching and application health need as much attention as the platform.

Design checklist

• Each VM’s protection level (HA or FT) chosen and agreed with the application owners
• Stakeholders informed in writing of what FT does not cover (guest OS and application faults)
• Two A-Links, on separate NIC cards, direct node to node (or each on its own VLAN)
• A-Link latency under 2 ms for FT VMs, 10 ms for HA VMs; bandwidth sized for all VMs
• Business networks on the same VLAN and subnet on both nodes; IPv6 multicast allowed
• node0 and node1 connected to different switches of a VLT or vPC pair
• Quorum server outside the pair, reachable over IPv4 with UDP 4557/4558 open
• Dual power supplies on separate feeds
• Application health monitoring and service auto-restart configured inside the guest
• Resource-heavy scheduled tasks tested under load
• Backups tested; FT does not protect against corruption or deletion

Commissioning tests (run them before go-live)

• Power off node0, then node1, in turn: FT VMs keep running, HA VMs boot on the partner
• Pull one A-Link, then the other: confirm behaviour and alerts
• Pull each node’s business link: confirm the VM stays reachable
• Fail each switch of the pair in turn
• Make the quorum server unreachable: confirm VMs keep running and alerts fire
• Return everything and confirm full resync before signing off

References

everRun (Stratus / Penguin Solutions)

• everRun 8.0 User’s Guide (PDF)
• everRun 8.0: Modes of Operation
• everRun 7.9: System Description
• everRun 7.9: Fault Tolerant Operation
• everRun 7.9: High Availability Operation
• everRun 7.9: Meeting Network Requirements
• everRun 7.9: Creating a SplitSite Configuration
• everRun 7.6: Storage Architecture
• everRun 7.5: Network Architecture
• everRun 7.5: Quorum Servers
• everRun 7.5: Understanding Quorum’s Effect on System Behavior
• everRun 7.4: Quorum Servers Considerations
• Stratus everRun product page (Penguin Solutions)
• everRun data sheet
• Fault Tolerance with everRun from Stratus, Availability Digest, December 2015

Other platforms

• VMware vSphere 6 Fault Tolerance: Architecture and Performance
• vSphere Availability: VM and Application Monitoring
• VMware HA, VMware FT and OS clustering (Scott Lowe)
• Understanding Hyper-V and Virtual Machines in the Context of a Cluster (Microsoft)
• Proxmox VE High Availability (ha-manager)
• Kernel-based Virtual Machine (Wikipedia)

everRun documentation is versioned, and some details differ between releases (for example the host operating system, and how priv0 and the A-Links are described). Check the user guide for the exact release you run.