A critical vulnerability in OpenSSL, named ‘HollowByte’, has been identified, allowing attackers to initiate a denial-of-service (DoS) attack with a payload of only 11 bytes. This flaw, uncovered by the Okta Red Team, exploits the way OpenSSL manages memory allocation during the TLS handshake, effectively enabling attackers to exhaust server resources before authentication.
Understanding the ‘HollowByte’ Vulnerability
The vulnerability lies in the initial stage of the TLS handshake, where a ClientHello message, containing a 4-byte header, prompts OpenSSL to allocate a receive buffer based on the declared size. Older OpenSSL versions allocate memory without validating the actual data, enabling a crafted 11-byte payload to trigger extensive memory allocation.
This process, involving functions like grow_init_buf() and OPENSSL_clear_realloc(), leads to the allocation of up to 131 KB of memory based on the attacker’s input. Consequently, server threads become indefinitely blocked, awaiting data that never arrives, aggravating the server’s vulnerability to connection-exhaustion attacks.
Impact on Internet Infrastructure
Traditional attacks such as Slowloris exploit open connections to deplete server resources. ‘HollowByte’ exacerbates this issue by inducing memory fragmentation, particularly impacting how glibc handles memory. When an attacker’s connection drops, OpenSSL frees the buffer, but glibc’s memory retention strategy prevents immediate resource recovery, resulting in continuous memory consumption.
Testing by the Okta Red Team demonstrated severe implications: in a 1 GB RAM environment, servers experienced out-of-memory termination after consuming 547 MB of fragmented memory. On a 16 GB system, attackers locked 25% of total memory, bypassing standard connection limits. This vulnerability’s reach extends to numerous internet frameworks, including NGINX, Apache, language runtimes, and databases.
Mitigation Measures and Future Outlook
OpenSSL has addressed this issue by implementing incremental buffer growth, ensuring that memory allocation only scales with actual data arrival. This fix has been integrated into OpenSSL version 4.0.1 and backported to earlier versions. However, the absence of a formal CVE advisory may leave many organizations unaware, necessitating proactive security measures.
Organizations are advised to upgrade their OpenSSL implementations to version 4.0.1 or relevant backports. Additionally, they should audit embedded OpenSSL versions within runtimes like Node.js and Python, as OS-level patches may not suffice. Monitoring for memory bloat on TLS-terminating servers can also help detect potential fragmentation attacks.
Ensuring robust defenses against ‘HollowByte’ is crucial, given OpenSSL’s widespread use. Security teams should prioritize these updates to mitigate potential threats effectively.
