Benchmarks Are Free Now

My previous post walked through four bugs in ruby-enum, a gem I maintain, all stemming from the fact that class-level instance variables aren’t inherited by subclasses. The third fix, #59, made keys, key?, value?, key, value, to_h, parse and each walk up superclass and merge in a parent’s enums, so a subclass would see everything its ancestors defined. It was correct, fully tested, and shipped. It also made every one of those methods roughly 5x slower on any subclass.

def _enum_hash
  if superclass < Ruby::Enum
    superclass.send(:_enum_hash).merge(_own_enum_hash)
  else
    _own_enum_hash
  end
end

This recomputes the merged hash, walking the entire ancestor chain, on every single call. There’s no caching. A one-level subclass calling .value pays for building a brand new hash, on top of the superclass doing the same, every time. The test suite didn’t notice because tests check correctness, not speed, and correctness was fine.

I only found this because I asked Copilot CLI to add a benchmark script comparing lookups at different inheritance depths, mostly out of curiosity about how the “Benchmarks” section of the README would read next to the new feature. It took one prompt and about a minute to get a working script:

class Colors
  include Ruby::Enum
  define :RED, 'red'
  define :GREEN, 'green'
end

class SubColors < Colors
  define :BLUE, 'blue'
end

class SubSubColors < SubColors
  define :YELLOW, 'yellow'
end

benchmark('base class (no inheritance)', n) { Colors.value(:RED) }
benchmark('subclass (1 level)', n) { SubColors.value(:RED) }
benchmark('sub-subclass (2 levels)', n) { SubSubColors.value(:RED) }

The first run told the story immediately: a 1-level subclass’s .value calls took roughly 5x as long as the base class, and a 2-level subclass was worse still. Nobody had written this benchmark before merging #59 because writing a throwaway benchmark script used to feel like more ceremony than it was worth for a one-off performance check. Now it’s a single sentence to an agent, and the script stays in the repo afterward as a rake benchmark:inheritance task anyone can rerun.

The fix was small: memoize the merged hash and invalidate the cache only when define adds a new entry.

def _enum_hash
  @_enum_hash ||= if superclass < Ruby::Enum
                    superclass.send(:_enum_hash).merge(_own_enum_hash)
                  else
                    _own_enum_hash
                  end
end

Re-running the same benchmark after the fix (#60):

--- .value lookups, by depth of inheritance ---
base class (no inheritance): 0.0627
subclass (1 level): 0.0608
sub-subclass (2 levels): 0.0606

Subclass lookups are now indistinguishable from the base class, regardless of depth. I also had Copilot CLI write a second benchmark (#61) comparing basic Ruby::Enum operations against plain Ruby equivalents, since I was about to write a README claim about the gem’s overhead and didn’t want to write “negligible” without a number to back it up. Constant access turned out to be free (Colors::RED is exactly as fast as a plain Ruby constant), but hash-backed lookups like .value and .key? carry a real, if small, 3-5x cost versus calling Hash#[] directly - the price of the extra method dispatch and object wrapping Ruby::Enum does. Worth knowing, not worth avoiding the gem over. That number, along with the exhaustive Ruby::Enum::Case matcher’s much larger 50-100x overhead versus a native case statement, is now spelled out in a dedicated Performance section in the README, so nobody has to ask or guess.

None of this changes the underlying lesson that a performance regression is easy to introduce and easy to miss when your tests only check correctness. What’s changed is the cost of catching it. Writing a benchmark script used to be the kind of thing you’d skip on a Friday afternoon for a “just a getter method” change. Now, benchmarks are free.

Daniel Doubrovkine

Benchmarks Are Free Now ruby | open-source | ai

Benchmarks Are Free Now