An OpenSSL vulnerability, identified by Okta’s red team, poses a significant threat as it can lead to server memory exhaustion before any security handshakes occur. This flaw, dubbed ‘HollowByte’, can be exploited by attackers using a mere 11-byte malicious payload. By declaring a larger message body, attackers can initiate buffer pre-allocations that are not immediately released, causing a denial-of-service (DoS) scenario.
Understanding the HollowByte Vulnerability
The core issue with HollowByte lies in older versions of OpenSSL that pre-allocate buffer sizes based on the message body length declared in the handshake’s 4-byte header. This allocation occurs prior to data arrival, allowing attackers to send an 11-byte payload that results in an unvalidated buffer allocation reaching up to 131 KB. According to Okta, this leads to worker threads indefinitely waiting for non-arriving data.
Impact on Systems and Memory Management
The GNU C Library (glibc) complicates matters by retaining small-to-medium memory allocations for potential reuse, not returning them to the operating system when connections drop. Although OpenSSL may free the buffer, repeated payloads can exhaust server memory. Waves of connections with varied claimed sizes can prevent allocator reuse, bloating the server even after disconnection. The only remedy to recover memory is process termination. Testing revealed that systems with 1 GB RAM became unresponsive after 547 MB of memory was fragmented.
Patch Implementation and Affected Systems
OpenSSL version 4.0.1, along with backports to earlier versions, includes patches to address the HollowByte flaw by increasing buffer size based on actual data received, rather than relying on the handshake header. This update affects a wide range of applications and systems, including Apache, NGINX, Node.js, Python, Ruby, PHP, MySQL, and PostgreSQL, necessitating an upgrade to the patched OpenSSL version for continued secure operations.
In conclusion, the HollowByte vulnerability underscores the importance of timely software updates and patches. Organizations using OpenSSL should promptly upgrade to the latest patched versions to prevent potential DoS attacks and maintain secure system operations.
