HollowByte: 11 bytes are enough to starve your server of memory (and OpenSSL fixed it without telling anyone)

Eleven bytes

Some attacks impress with their sophistication, others with their economy. HollowByte belongs to the second group: a payload of 11 bytes — shorter than the word "ateneasystems" — can force a server running unpatched OpenSSL to reserve up to 131 KB of memory per connection, before any authentication takes place. Repeat the trick a few thousand times and the server's memory balloons until the system starts killing processes.

Okta's security team disclosed it in mid-July, and on the 18th it hit the oss-security mailing list, which is when the sysadmin world found out en masse. But the most uncomfortable part of the story isn't the flaw itself: it's how it got fixed.

In one sentence

OpenSSL allocated its TLS receive buffer trusting the size the client declares, before a single real byte arrived. A remote, unauthenticated attacker can exploit that to bloat and fragment the server's memory permanently — the process doesn't release it even after the attacker disconnects — until it forces a denial of service. No data theft, no code execution: they "just" leave your server unusable.

How it works, in two steps

Step 1 — the blind reservation. During the TLS handshake, the client announces how much data it's about to send. Older OpenSSL versions allocated the full buffer based on that declared figure. Eleven well-crafted bytes are enough to claim the maximum reservation. The attacker never needs to send the promised data: the memory is already set aside.

Step 2 — the fragmentation that never leaves. Here's the refined nastiness. When the connection closes, OpenSSL frees the buffer… but glibc doesn't hand that memory back to the operating system: it keeps it around in case the process reuses it. By firing waves of connections with randomized declared sizes, the attacker ensures those freed chunks never fit the next reservations. The result: the process's resident memory (RSS) goes up and never comes back down, even after the attack ends. The only way to reclaim it is restarting the service.

Okta's tests against Nginx running unpatched OpenSSL make it plain: on a 1 GB RAM machine, the server ended up killed by the OOM killer with 547 MB of frozen, unusable memory. A perfectly healthy box, murdered by fragmentation.

The controversial part: the silent patch

The fix exists and it's elegant: OpenSSL no longer trusts the declared figure — it grows the buffer as real bytes actually arrive. A client that promises a lot and sends nothing now costs the server nothing.

The problem is how it shipped: no CVE and no security advisory. OpenSSL filed it as a robustness improvement ("hardening") and quietly folded it into version 4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6 and 3.0.21.

Why is that a problem? Because half the world prioritizes patches by CVE. Vulnerability scanners look for CVEs. Security teams triage by CVSS. If there's no CVE, for many organizations the patch simply doesn't exist — while the attack mechanism sits fully documented on the public internet. It's the least desirable combination: a publicly explained exploit, invisible urgency in the systems that manage patching.

Who's affected

Almost everyone, because OpenSSL sits underneath almost everything:

• Web servers: Apache, Nginx (when linked against OpenSSL for TLS).
• Runtimes: Node.js, Python, Ruby, PHP.
• Databases: MySQL, PostgreSQL, if TLS is enabled on exposed connections.

The condition is simple: a service accepting TLS connections from the internet with an OpenSSL version older than the patched ones. The obvious candidate is the web server, but a MySQL with exposed TLS qualifies too.

What you need to do today

1. Check your OpenSSL version

openssl version

Target depending on your branch: 3.0.21, 3.4.6, 3.5.7, 3.6.3 or 4.0.1 (or higher).

Mind the usual distro caveat, aggravated here by the missing CVE: Debian, Ubuntu or Alma may backport the fix while keeping the old version number — or take longer to pick it up precisely because there's no CVE flagging the urgency. Check the package changelog (apt changelog openssl or your distro's security tracker) instead of trusting the number alone.

2. Update the library

# Debian / Ubuntu
sudo apt update && sudo apt install --only-upgrade openssl libssl3*

# RHEL / AlmaLinux / Rocky
sudo dnf update openssl openssl-libs

3. Restart the services that use it (non-negotiable)

And here's the classic mistake: OpenSSL is a shared library. Updating the package achieves nothing while your processes keep the old version loaded in memory. A reload isn't enough either: the services need a restart.

# See which services still use the old library
sudo needrestart

# Or by hand: restart whatever faces the internet
sudo systemctl restart nginx    # or apache2, mysql, etc.

If you don't have needrestart, lsof | grep libssl | grep DEL shows the processes still holding the deleted version.

What we do

This flaw is a textbook example of why watching the CVE feed isn't enough: if your patching process depends on it alone, HollowByte slips right under your radar. On the servers we manage, the update pass includes the security mailing lists (oss-security, each distro's advisories) on top of CVEs — and after updating OpenSSL, exposed services get restarted, not just reloaded.

It's also a good sizing reminder: the attack killed a 1 GB machine with 547 MB frozen. Monitoring the resident memory of exposed processes (and alerting when it grows without ever coming back down) turns this kind of attack into an odd-looking graph on a Tuesday afternoon, instead of a website down on a Sunday night.

If you don't know which OpenSSL version runs underneath your website — or who would notice if your server started to bloat — let's talk.

References

• OpenSSL "HollowByte" DoS via attacker-controlled memory allocation — oss-security
• OpenSSL HollowByte: A DoS Hiding in 11 Bytes — Okta Security
• HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload — BleepingComputer
• OpenSSL Silently Fixes 'HollowByte' DoS Vulnerability — SecurityWeek
• HollowByte: Denial-of-service vulnerability in OpenSSL — heise online