Advisory

GHSA-73v3-wf97-vpgj

Also known as: CVE-2026-126366

Summary

msgpack-ruby JRuby decoder makes uncapped header-count allocations; a 5-byte message forces multi-GiB OOM

Details

Summary

The JRuby implementation of msgpack (msgpack gem, java platform — pure Java decoder under ext/java/) sizes containers directly from the element count written in the message header, with no cap at all: no equivalent of the C unpacker's SHRT_MAX clamp (since 1.7.3) and none of the buffered-bytes cap that commit c8d42d8 added for CRuby in 1.8.6. A single 5-byte message decides how much memory a JVM process tries to allocate:

  • array32 header declaring 0x7FFFFFFF elements → new IRubyObject[0x7FFFFFFF] (~8 GiB of reference slots) attempted immediately → java.lang.OutOfMemoryError: Requested array size exceeds VM limit.
  • Counts *below* the VM's max-array-size but above available heap (e.g. 5×10⁸ with a 512 MiB heap) produce a real allocation attempt — OutOfMemoryError: Java heap space — the JVM genuinely tries to take the memory before failing.
  • str32/bin32 headers declaring 0x7FFFFFFF bytes do the same via readBytes(size) → new byte[size].
  • Below the failure threshold there is a silent band: one streaming unpacker fed only an array32 header with count 2×10⁷ (5 wire bytes, zero elements) grows used heap +78 MiB immediately, no exception (each swallows the underflow), and the footprint stays elevated after Unpacker#reset + full GC. Linear in declared count (~4 B/element); nothing bounds it — not delivered bytes, not a per-container cap.

Only counts ≥ 2³¹ are refused (a clean MessagePack::UnderflowError from getUnsignedInt()), and only maps are safe (consumeHash grows per element instead of pre-sizing). Verified on the released gem msgpack 1.8.6 java, JRuby 9.4.12 / OpenJDK 25, 2026-10-30 — i.e. unfixed in the current release. Proposed fix mirrors c8d42d8: cap initial allocation by bytes actually buffered and grow as data arrives; read strings in bounded chunks instead of allocating the declared length up front.

Details

1. The four dispatch arms, and which pre-allocate (line numbers at tag v1.8.6 — current release; ext/java has no commits after it)

| header | dispatch (Decoder.java) | behaviour on the declared count | measured (released 1.8.6-java, Xmx512m) |
|---|---|---|---|
| array32 \xdd | :296 → `consumeArray(int size)` :139-145 | new IRubyObject[size] immediately, before reading any element | count 0x7FFFFFFF: OOME "Requested array size exceeds VM limit"; count 5×10⁸: OOME "Java heap space" (real allocation attempt); count 2×10⁷: +78 MiB, silent |
| str32 \xdb | :294 → `consumeString` :129 → `readBytes(int size)` :186-190 | new byte[size] immediately | len 0x7FFFFFFF: OOME "array size exceeds VM limit" |
| bin32 \xc6 | :269 → same consumeString/readBytes | same | len 0x7FFFFFFF: OOME "array size exceeds VM limit" |
| map32 \xdf | :298 → `consumeHash(int size)` :147+ | no pre-allocation — inserts per element, underflows on first missing byte | count 0x7FFFFFFF: no exception (swallowed), no large allocation |

The only guard is `getUnsignedInt()` :88-94: counts with the top bit set (≥ 2³¹) raise MessagePack::UnderflowError: Size too large (limited to 2**31 for the Java version). That bounds the *value* but leaves the entire positive band (available heap, 2³¹) unbounded — and that band is where real allocation attempts happen.

2. Contrast with the C implementation (same repo)

The CRuby unpacker has had a per-container cap since 1.7.3 (INITIAL_BUFFER_CAPACITY_MAX = SHRT_MAX) and, since c8d42d8 in 1.8.6, additionally caps initial size by bytes actually buffered ("an element needs at least one byte…"). Strings on C are bounded too (read_raw_body_cont allocates min(remaining, buffered)). The Java decoder has neither generation of cap — it is strictly behind the CRuby behaviour this repo already considers correct enough to ship.

3. Silent retention + mechanism control (measured 2026-10-30, released gem)

One MessagePack::Unpacker per process, fed only a header ("\xdd" + [count].pack('N')), zero elements; heap = Runtime.total_memory - free_memory after GC.start on both sides:

| declared count | used heap before → after | delta | exception? |
|---|---|---|---|
| 1,000,000 | 26.4 → 31.1 MiB | +4.7 | none (each swallows the underflow) |
| 5,000,000 | 53.3 → 74.0 MiB | +20.7 | none |
| 20,000,000 | 51.3 → 129.3 MiB | +78.0 | none |

Mechanism control: after the feed, Unpacker#reset + three GC.start cycles left used heap at +79.4 MiB above baseline (2×10⁷ arm) — the elevation is *not* unpacker-held state; it is JVM heap already spent on header-declared allocation and not returned. Either way the process footprint grows per attack and stays grown; the peak cost is paid synchronously during parsing regardless of GC behaviour afterwards. (Honest scope note: this is unbounded allocation, not a memory leak.)

4. Why JVM makes availability impact worse than the CRuby analogue

In CRuby, allocation failure raises NoMemoryError/fails inside one worker process. In a JVM, an OOME raised on the parser thread can land anywhere in the host application's shared runtime — long-lived JRuby servers (Rails, Logstash-class pipelines, JDBC apps) share heap across all threads and connections; a single crafted message can destabilize or kill the whole JVM, not just the request. The silent band means no exception at all is even required to spend memory.

5. Not claimed / scope

  • Fluentd is NOT affected (and this report deliberately does not involve it): Fluentd ships CRuby-only — CI pins engine: cruby, gemspec requires ruby >= 3.3 while JRuby 9.4 implements Ruby 3.1 semantics, and there are zero JRUBY guards in lib/. This is an upstream msgpack-ruby report only.
  • Maps (map32) do not pre-allocate — measured; the class applies to array32/str32/bin32.
  • Counts ≥ 2³¹ are cleanly refused — measured; no claim there.
  • Older java-gem releases were not individually probed; <= 1.8.6 means "every release up to the current one carries this code".

Severity

High Basis: The source's own band: HIGH

Dates

Published
2026-11-12 22:54 UTC
Modified
2026-11-12 23:00 UTC

Affected packages and fixes

msgpack

RubyGems · All systems

Affected

  • introduced 0, fixed 1.8.7 (ecosystem)

Fixed versions

1.8.7

Fix

The export has no generic command for this ecosystem.

Fix commands are the export's own generic commands. Ojuju never runs them. Check them against your setup first.

Check your own computer: guard packages check

View this record on OSV.dev

Source

Source: GitHub Advisory Database (reviewed advisories) · CC BY 4.0

Sources & licences