Firm operations

How to test internet failover and know what passed

Listen to this piece read aloud in a recorded voice, narrated by Daniel.

A second internet connection gives us another path. The test is whether the firm can keep doing the work that matters when the first path becomes unavailable. That requires a defined scenario, observations from the applications people use, and a clear way to return to normal service. A useful failover record explains exactly what was exercised and what remains untested.

Source

Recovery products still need an application-level question

AWS Backup’s current restore-testing capability separates scheduled restoration from an optional validation step. It is a cloud example, but the distinction suggests a useful question for network resilience: what will demonstrate that work can resume after the path changes?

I would define success through a few real application behaviors, then use network events to explain the transition. This is an adaptation of the testing principle, not a claim that a cloud restore feature validates an office network.

Decide which failure you are exercising

“Internet outage” describes several different events. A local cable can disconnect. An upstream provider can stop carrying traffic while its handoff remains active. The device managing the two connections can fail. Each scenario tests a different part of the design.

Write down the scenario before the exercise. Identify the primary and alternate paths, the approved test window, the person authorized to stop the test, and the means of communicating if the ordinary network becomes unavailable. Confirm that recovery instructions are accessible outside the connection being tested.

A cable-disconnection exercise provides evidence about that condition. It does not establish that every upstream failure will be detected or that a shared power dependency has been addressed.

Proposal

Define success in application terms

Choose a small set of representative workflows. A document should open. An authorized user should be able to sign in. A new call should connect. An existing session may behave differently from a new one, so observe both where they matter.

Agree on tolerable interruption and acceptable degraded behavior with the people who own the work. The alternate connection may have different capacity, latency, or routing. A successful switch can still leave the firm unable to perform a critical activity.

Record the expected behavior before touching the network. That prevents the result from being redefined around whatever happened during the test.

Observe the transition and the return

Capture the time the failure condition begins, the time the network reports a path change, and the time each selected workflow becomes usable. These observations answer different questions. A device event is useful evidence, but users experience the application path.

One failover test on one axis: the failure condition, the network’s path change, each workflow usable again, and the return THE FAILURE EXERCISED Local cable disconnects Evidence about that condition only Upstream provider stops Carrying traffic while its handoff remains active The device itself fails The one managing the two connections ONE SCENARIO, WRITTEN DOWN IN ADVANCE THE CONDITION Failure begins Within the approved test window THE NETWORK Path change reported A device event: useful evidence THE APPLICATIONS Each workflow usable A document opens, a user signs in, a new call connects THE RETURN Primary path restored Routing state verified, application checks repeated THE INTERVAL THE TEST MEASURES Failure begins to each workflow usable Users experience the application path, not the device event Timestamps that bound the measure Recorded as evidence One test, one scenario; spacing is by event, not to scale, and no durations are implied.
Figure 1. The network reporting a path change is useful evidence, but it is not the result: I would time the test from the moment the failure begins to the moment each workflow people rely on is usable again, for one scenario written down in advance.
Describe this figure

A horizontal timeline of one failover test; spacing is by event, not to scale, and no durations are given. Above the timeline, a group names the failure being exercised, one of three scenarios written down before the exercise: a local cable disconnects, which gives evidence about that condition only; an upstream provider stops carrying traffic while its handoff remains active; or the device managing the two connections itself fails. The chosen scenario connects to the first event. Four events on the axis. The condition: the failure begins, within the approved test window. The network: a path change is reported, a device event that is useful evidence. The applications: each selected workflow becomes usable again, a document opens, a user signs in, a new call connects. The return: the primary path is restored, the routing state verified and the application checks repeated. Below, the interval the test measures, highlighted from the failure beginning to each workflow being usable, because users experience the application path, not the device event.

Use ordinary test accounts and non-sensitive material where practical. Keep a separate observation record so participants can note interrupted sessions, failed sign-ins, unexpectedly slow operations, and workarounds. Stop if the agreed limits are exceeded or the recovery path becomes unclear.

After restoring the primary path, verify the intended routing state and repeat the application checks. Look for services that appear recovered only because a user established a new session or switched to a different connection. Record those distinctions.

State the result narrowly

AWS’s recovery-testing guidance emphasizes exercising recovery paths against objectives. Applying that idea to an office network means writing a result that names the scenario, the observed interruption, and any unmet requirements.

A useful conclusion might say that new document and sign-in sessions worked over the alternate connection during the selected failure, while an existing call required reconnection. That is an illustrative reporting format, not a claim about a particular installation.

Assign an owner and a retest condition to every material exception. The test is complete when the intended state is restored, the evidence is recorded, and the remaining work has somewhere to go.

Proposal

What I would test next

I would begin with one controlled, reversible failure and explicit stop conditions. The result should change either the design, the recovery instructions, or our confidence in a narrowly defined scenario. If it changes none of those, reconsider what the test was intended to establish.

Companion resource

Internet failover test record

Open, print, or download the worksheet.