The Case of the Missing Packets

Every so often you get an internet problem that's just a genuinely fun puzzle. Not the normal kind where a modem needs a reboot, or the neighbourhood cable node needs a reset, or the fibre goes down and you know exactly what to do. This one was the interesting kind. It knocked out a very specific set of websites for a very specific set of customers, and figuring out why took our whole engineering team an entire weekend of detective work. This is that story.

What some customers saw

If you were one of the affected customers, your internet mostly worked. Speed tests looked normal. Most websites loaded. Your Wi-Fi was fine. But specific things kept failing: Crave and TSN wouldn't load. Facebook was intermittent. Yahoo was intermittent. A few mobile games and Activision titles couldn't connect. Everything else on the internet seemed fine.

The reason a random-looking subset of the internet wasn't working (and everything else was) is the first clue in the puzzle. On any internet provider, different destinations on the internet are reached via different network paths. If one path breaks and others don't, only the sites reached over the degraded path stop working. Which is what was happening here.

The technical story, in plain language

Purple Cow, like every internet provider, reaches the rest of the internet through several different upstream connections. Each connection is a different route out. When you visit Crave or TSN or Facebook, our routers pick which route to send you through based on which one gets to that destination most efficiently.

For a large chunk of the sites people watch and stream on, we send traffic through a shared exchange in Toronto called TorIX. TorIX is fast, cheap, and it's how most Canadian internet providers get you to Netflix, Crave, TSN, and a lot of other major services efficiently.

The physical circuit that carries our TorIX traffic between Halifax and Toronto is provided by a big American transit company called Zayo. On September 23, Zayo did something to that circuit (they still have not fully explained what) that we'd call a "silent reroute." Zayo didn't tell us. Our monitoring didn't catch it, because on the surface the circuit stayed up and small traffic still passed through.

But the rerouted circuit came back damaged. It could carry small internet packets fine. It could not carry packets larger than about 1478 bytes. Normal internet packets can be up to 1500 bytes. For two days, the circuit sat there quietly dropping any packet that was too large. Most traffic happened to be small enough to slip through, so nobody noticed.

Then on Friday evening, the wholesale cable network our cable customers operate on top of started rolling out new modem firmware. The new firmware handled packet sizes slightly differently than the old one. Modems that got the update started sending larger packets. Larger packets hit the degraded Zayo circuit. Larger packets died. Suddenly, if you happened to be on a modem that had received the new firmware, any service we reach through TorIX would fail. Crave. TSN. Some Facebook. Some Yahoo. A few games. Everything else on the internet worked.

That's why it looked like a random slice of the internet had disappeared. It wasn't random at all. It was a very specific slice, all reached through the same degraded path.

The investigation timeline

Here's how the weekend actually unfolded.

• Wed Sept 23, afternoonZayo silently reroutes the Halifax-to-Toronto circuit that carries our TorIX traffic. No notice. The circuit returns damaged, unable to carry packets over 1478 bytes. Nothing on our side detects it, because most traffic still passes.
• Fri Sept 25, eveningA new modem firmware rollout begins on the wholesale cable network. The first customer reports start trickling in: "Crave won't load." "TSN broken." "Everything else works fine though." That last part is the confusing one.
• Sat Sept 26, 10:28 a.m.Bell Media contacts us directly with hard data: over 1,000 unique customer IPs are failing to reach Crave. This is the moment the scale becomes clear.
• Sat Sept 26, 2:03 p.m.Status page goes up. Affected customers texted directly.
• Sat Sept 26, eveningTesting with the modem manufacturer isolates the pattern: only modems on the new firmware are affected. The new firmware is sending larger packets than the old one. Something upstream is dropping those larger packets. We don't yet know where.
• Sat Sept 26, 8:39 p.m.Test packets prove it: anything over 1450 bytes dies on the TorIX path. Every port on our side of the network is configured to allow the full 1514. The problem is not on our equipment. It's somewhere in a circuit we don't own.
• Sat Sept 26, 11:31 p.m.Plan agreed: withdraw the TorIX peering entirely and force our traffic to take a different route to those services. It's a workaround, not a fix, but it will restore service while we figure out the underlying problem.
• Sun Sept 27, 12:28 a.m.TorIX peering withdrawn. Traffic now flows via alternate routes. Large packets get through. Affected customers back online.
• Sun Sept 27, 11:46 a.m.Our engineering lead digs into the monitoring history and pinpoints the exact moment the Zayo circuit changed on September 23. Evidence goes to Zayo with a very direct message.
• Sun Sept 27, 1:31 p.m.Zayo repairs the circuit. Full-size packets now pass on the original path.
• Sun Sept 27, 2:05 p.m.Status page marked green.
• Mon Sept 28, late eveningTorIX peering re-enabled on the now-repaired circuit. Traffic returns to its normal route. All systems fully back to normal.

What actually caused it

The root cause was a silent, undocumented change Zayo made to our Halifax-to-Toronto transit circuit on September 23. After that change, the circuit could no longer carry full-size internet packets. It looked healthy on every dashboard, but it was quietly dropping anything over 1478 bytes.

The trigger that made the damage visible was a modem firmware update on the wholesale cable network that shifted the size of packets those modems send. The firmware itself wasn't at fault. It was doing what firmware should do. It just happened to expose a circuit failure that had been sitting there for two days.

None of the following were the cause: our own equipment configuration (which was correct), the firmware itself (which was normal), Akamai (which serves Crave and TSN and was fine), or the modem manufacturer.

A big thank-you to the team

This is the part that matters most. This puzzle got solved because of a very specific group of people who spent their weekend chasing it down.

An honourable mention also goes to Bell Media, who noticed the scale of the impact before we did and reached out on Saturday morning with the data we needed to size the problem. Great neighbours.

Where things stand now

Everything is fully back to normal. The Zayo circuit has been repaired. The TorIX peering has been re-enabled. Traffic is flowing over its normal, primary path. Your internet is doing what it should be doing.

Why we wrote this

Most internet providers, when something like this happens, put up a vague "we experienced technical difficulties" line on the status page and hope you forget about it by Monday. We'd rather do it differently. When something interesting happens on our network, we like to actually tell you what it was, because the story is often more interesting than the boring status-page version, and because you're paying for internet and you deserve to know how it works when it works, and how it breaks when it breaks.

Thanks for being in the herd. We had a wild weekend. We're back on our feet, we learned things we didn't know before, and we're going to keep working to be the internet provider Nova Scotia deserves.