The Time I Had To Troubleshoot 6% Packet Loss: Or, How We Accidentally Discovered the Network Had…

Every engineer has that one project.

The one where you begin with a perfectly reasonable objective_, “_We just need to load test the new service.”

Three months later, you’re learning things about enterprise networking that were never meant to be known by mortal beings.

This is one of those stories.

It Started with TLS

Our mission sounded delightfully straightforward: build a service capable of processing roughly 4,000 SOAP requests per second.

Yes, SOAP.

Because, this was 2014 and, apparently, REST hadn’t yet convinced everyone to abandon XML’s endless quest for matching angle brackets.

The application was ready.

The load generators were humming.

We turned the crank.

Then…

TLS failures.

Random.

Inconsistent.

Maddening.

Sometimes connections completed.

Sometimes they didn’t.

Sometimes handshakes failed.

Sometimes the application simply stared back at us with the networking equivalent of shrugging its shoulders.

The application logs were clean.

The CPU was loafing.

Memory was happy.

Disks were bored.

Everything looked healthy.

Except…

Random connection failures with TLS errors.

The First Clue

Eventually, someone asked the wonderfully dangerous question_,_ “Has anyone tried grabbing a network trace?”

It was loaded with packet retransmissions and reset packets.

There it was.

Approximately, 6% packet loss in the DMZ.

Six percent.

Not six packets.

Not six packets per million.

Six percent.

Every twentieth packet was apparently deciding it had better places to be.

This wasn’t an overloaded application.

It wasn’t TLS.

It wasn’t Java.

It wasn’t SOAP.

It was something in the network…and the network engineering team’s default position always started with, “Yeah, it’s always the network. Sure.”

Hope It’s in the DMZ

The lead network architect gathered everyone around an impressively complicated network diagram.

He pointed toward the DMZ.

Then, he pointed toward the sprawling mess of routers, switches, firewalls, load balancers, security appliances, and networking archaeology that made up the internal infrastructure.

Then, he delivered one of the most memorable lines I’ve ever heard in my caree_r,_ “Let’s hope the problem is in the DMZ.”

He paused.

Then, pointed toward the internal network_,_ “Because if it’s in this…”

His finger slowly traced the diagram.

“…we’re all in a lot of trouble.”

There are moments when you can hear everyone’s internal monologue simultaneously.

This was one of them.

The Network Team

Curiously…

The network team didn’t seem especially alarmed.

To them this wasn’t a crisis.

This was Tuesday.

Apparently, things had “always kind of behaved this way.”

Which is simultaneously comforting and deeply horrifying.

Can We Have a Network Diagram?

Being the hopeless optimist that I am, I asked for a network diagram.

Not because I enjoy network diagrams.

Nobody, enjoys network diagrams.

I, simply, wanted to know where packets were going.

The answer arrived almost immediately_,_ “We don’t share network diagrams with consultants.”

Fair enough.

So instead of receiving a nice architecture document…

…we reverse engineered the network one dropped packet at a time.

For the next several years after that, my network + architecture diagrams became the gold standard that every other team referenced.

Welcome to the Onion

As we peeled back the layers, the network became increasingly…

creative (the crack-enhanced kind).

Packets traversed:

• Load balancers
• Multiple firewall tiers
• DMZ infrastructure
• Core routing
• Internal routing
• Security appliances
• More firewalls
• Yet another firewall
• Several NAT boundaries

At one point I stopped counting after the fourth layer of Network Address Translation.

Not four translated addresses.

Four separate devices performing NAT.

When I asked about that being the best approach, I was told, “That’s nothing. In this other network path over there, a packet traverses a dozen layers of NATing.” Followed by some mumbled commentary about packets passing through layers of firewalls 60+ times — we’ll cover that part another time.

The packet left the load generator with one identity.

By the time it reached the application, it had participated in what can only be described as the witness protection program.

Ten Weeks

This became less of a troubleshooting exercise and more of an archaeological dig.

Week after week, we’d eliminate another possibility.

Firewall #1.

Not it.

Firewall #2.

Not it.

Router.

Not it.

Load balancer.

Nope.

TLS implementation.

Working perfectly.

Operating system.

Healthy.

Application.

Fine.

Another firewall.

Another NAT.

Another firewall.

Yet another firewall.

There were so many security appliances that packets were probably required to present identification just to continue existing.

Meanwhile…

While all this was happening…

…I’m attempting to build an application capable of processing 4,000 SOAP calls per second.

The middleware team had their own concerns.

Specifically…

A monitoring system.

One.

Single.

Monitoring system.

It made one request per second into this system.

And, there was serious discussion about changing it to one request every five minutes.

Because…

…it might put too much load on the system.

Meanwhile I’m trying to build something capable of surviving over 4,000 requests every second.

Apparently, we had very different definitions of “high traffic.”

I walked out of the third meeting on that topic. They continued to meet twice a week for a year to discuss it. Nothing ever changed.

Is a synthetic transaction once per second too often? On a system that gets hit once per minute, sure. On a system that is hit 1000s of times per second that is riddled with weird errors, you need data. I had to generate it somehow.

The Culprit

Ten weeks later…

We finally found it.

A security appliance performing deep packet inspection.

Packets disappeared.

Connections reset.

TLS handshakes failed.

Applications looked guilty.

The network was quietly dropping packets while insisting everything was perfectly fine.

That one firewall had some weird issues that I’m not sure were ever resolved. That particular security device was simply removed.

Suddenly…

Once the configuration was corrected…

Everything changed.

TLS errors vanished.

Packet loss disappeared.

Load tests scaled.

SOAP requests flew.

Monitoring could continue making its adorable one request every second without threatening civilization.

The application reached its performance targets.

The mystery was solved.

Lessons Learned

The application wasn’t broken.

TLS wasn’t broken.

SOAP wasn’t broken.

The operating system wasn’t broken.

Java wasn’t broken.

The new DataPower ESB wasn’t broken.

The network had been quietly dropping roughly 6% of all traffic for years.

Nobody had noticed because every application had learned to compensate.

Retransmissions.

Retries.

Timeouts.

Circuit breakers.

Everyone had simply adapted to living with a network that occasionally forgot its primary responsibility of…you know…delivering packets.

Summary

People often imagine performance engineering as tuning garbage collection, optimizing SQL queries, or shaving milliseconds off serialization code.

Sometimes it is.

Other times, it’s spending ten weeks discovering that packets are being NATted four times before encountering a crack-enhanced firewall.

The next time someone confidently declares_,_ “It’s definitely the application.”

Take a moment.

Capture a packet trace.

Ask awkward questions.

Request the network diagram.

And when someone tells you they don’t share those with consultants…

…bring extra coffee.

You’re probably going to be there a while.

If nothing else, this project permanently changed my definition of “network transparency.” Sometimes the most opaque part of an enterprise system isn’t the software — it’s the infrastructure (and the supporting network team) that’s convinced itself six percent packet loss is just part of the user experience.

Notes

• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• ChatGPT was used to generate the images.
• I used em dashes in my writing before the current GenAI wave was a thing. Not planning on changing now.
• Names have been changed to protect the guilty.
• None of the hostnames or users used in examples actually exist.
• Feel free to post any comments or suggestions below.