Recent findings have highlighted a significant vulnerability in OpenSSL that can cause server memory to be exhausted with minimal data input. The issue, dubbed HollowByte, allows an unpatched OpenSSL server to allocate up to 131 KB of memory with just an 11-byte TLS request. This memory remains occupied until the server process is restarted.
Understanding the HollowByte Flaw
The HollowByte issue was disclosed by Okta’s Red Team, who discovered the denial-of-service (DoS) vulnerability and shared the details publicly. The flaw was quietly fixed by OpenSSL in a June update without an advisory, CVE, or changelog entry to indicate the correction. The affected versions include OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, with previous versions susceptible to the flaw.
The vulnerability arises from OpenSSL trusting the size declared in the header of a TLS handshake message. This trust leads the server to expand its receive buffer to the specified size before receiving the actual message body, resulting in memory allocation based on a potentially false header declaration.
Implications of the Vulnerability
The impact of HollowByte is particularly severe on systems using glibc, as this library retains small and medium memory chunks for reuse rather than returning them to the system. This retention can prevent memory from being reused effectively, leading to fragmentation and increased memory usage over time. In tests conducted by Okta, a 1 GB server was overwhelmed with 547 MB of fragmented memory, while a 16 GB server saw 25% of its memory locked without breaching the connection limit.
Despite these significant implications, OpenSSL has not classified HollowByte as a vulnerability. The security team opted to treat it solely as a bug or hardening issue, which does not require a CVE or formal advisories according to their policy. This decision means that many users might remain unaware of the fix unless they actively check for the specific updates.
Response and Future Outlook
The lack of an official vulnerability classification and associated advisories has led to questions about OpenSSL’s handling of the issue. While Okta has raised concerns about the persistent memory usage, OpenSSL maintains that the memory allocation per connection is typical and not inherently a vulnerability. However, the absence of public exploit codes or widespread proof-of-concept demonstrations keeps the potential risk contained for now.
As the security community awaits further clarification from OpenSSL and possible updates from Okta regarding other allocator behaviors, users are advised to update their OpenSSL installations to the fixed versions. This proactive approach can help mitigate potential risks associated with the HollowByte flaw. Moving forward, the emphasis will be on monitoring how OpenSSL addresses similar issues, especially in its extended-support branches, to ensure robust security measures are in place.
