Ruby Recursion Limit and Stack Overflow: The Interview Guide

Python gives you sys.setrecursionlimit(). A number to bump. A ceiling you can raise with a single line of code and a vague feeling of power. Ruby gives you nothing of the sort. No limit to configure, no soft guard, no polite warning before the cliff. MRI Ruby delegates recursion depth entirely to the OS thread stack, and when that fills up, you get a SystemStackError. In a coding interview, not knowing that distinction can quietly derail you.

The call stack in a Ruby process is a fixed-size memory region allocated before your code runs. You can't resize it at runtime. If recursion exceeds what fits, the OS terminates the stack walk and Ruby surfaces it as SystemStackError. You can rescue that error, but you can't meaningfully recover from it mid-stack. By the time you catch it, you're already out of stack space. The only real fix is not filling the stack in the first place.

The Error You'll See

def factorial(n)
  return 1 if n <= 1
  n * factorial(n - 1)
end

factorial(100_000)
# => SystemStackError: stack level too deep

SystemStackError is Ruby's signal that the call stack is full. It's not a soft limit you configure, and it's not a RuntimeError you can rescue and recover from in any meaningful way. By the time it fires, you're deep in the stack with no room left. A rescue block here is technically possible and practically useless:

begin
  factorial(100_000)
rescue SystemStackError => e
  puts "Stack overflow: #{e.message}"
  # You caught it. You're still out of stack. Congrats.
end

For what happens at the OS level, see what causes a stack overflow. In an interview, the right response is not a rescue block. It's converting to iteration.

How Deep Is "Too Deep"?

Ruby's call stack is backed by the C thread stack, a memory region the OS allocates when the thread starts. Default sizes by platform:

Platform	Default Thread Stack
Linux (x86_64)	8 MB
macOS (main thread)	8 MB
Windows	1 MB

Each MRI stack frame costs roughly 200 to 500 bytes depending on local variables, arguments, and any block closures the method captures. With an 8 MB stack and 400-byte frames, you get somewhere around 10,000 to 20,000 recursive calls before hitting SystemStackError.

That number shifts across Ruby versions, frame complexity, and OS configs. The interview takeaway: Ruby recursion is safe for trees of depth 100 to 200 (typical interview constraints), but anything approaching 10,000 frames is risky without iteration.

You can measure the practical limit on whatever machine you're running on:

depth = 0
begin
  recurse = ->(d) { recurse.(d + 1) }
  recurse.(0)
rescue SystemStackError => e
  depth = e.backtrace.length
  puts "Crashed at roughly #{depth} frames"
end

On most Linux systems with default settings, this lands between 8,000 and 18,000. Most people discover this number right after discovering they shouldn't have.

Asking Ruby how many recursive calls are too many.

Can You Push Ruby's Recursion Limit?

Two mechanisms exist. Neither belongs in an interview solution.

Environment variable (Ruby 2.0+):

RUBY_THREAD_STACK_SIZE=67108864 ruby your_script.rb  # 64 MB

This raises the stack size for Ruby-spawned threads. The main thread's stack is set by the OS before Ruby starts, so this alone doesn't help everywhere.

OS ulimit (Linux/macOS):

ulimit -s unlimited
ruby your_script.rb

Both are useful for production scripts that process enormous trees or graphs. In an interview, reaching for either signals you're working around a bad algorithm, not fixing it. Mention that you know these exist, explain the tradeoff briefly, and then convert to iteration.

Does Ruby Have Tail-Call Optimization?

Technically yes. Practically, no.

MRI ships with TCO that is off by default and must be enabled explicitly via the bytecode compiler. It looks like this:

RubyVM::InstructionSequence.compile_option = {
  tailcall_optimization: true,
  trace_instruction: false
}

RubyVM::InstructionSequence.new(<<~RUBY).eval
  def factorial(n, acc = 1)
    return acc if n <= 1
    factorial(n - 1, n * acc)
  end
RUBY

factorial(100_000)  # No SystemStackError

This is obscure, non-portable (JRuby and TruffleRuby handle it differently), and won't work in most interview coding environments. Tail-call optimization is worth understanding as a concept, but don't reach for it in a Ruby interview. Treat Ruby as a language without TCO. Any interviewer who objects to that simplification probably also wants to tell you about their Emacs config.

The Real Fix: Turn It Iterative

When a recursive solution risks a deep stack, rewrite it with an explicit stack (a plain Ruby array). The logic is identical. You're just managing the call stack yourself on the heap instead of letting MRI manage it on the thread stack. The heap can grow orders of magnitude larger than the thread stack, and it's not fixed at thread startup.

Recursive DFS:

def dfs(root)
  return if root.nil?
  process(root.val)
  dfs(root.left)
  dfs(root.right)
end

Iterative DFS, same preorder traversal:

def dfs(root)
  return if root.nil?
  stack = [root]
  until stack.empty?
    node = stack.pop
    process(node.val)
    stack.push(node.right) if node.right
    stack.push(node.left)  if node.left
  end
end

Note the order: push right before left so left gets processed first (LIFO). The pattern extends to graph traversal, backtracking, and most other recursive structures. For more on the space relationship between recursion and the call stack, see recursion space complexity.

Memoization Does Not Prevent Stack Overflow

This one trips up a lot of people. The thinking goes: "My recursive fib is hitting a stack overflow, I'll add memoization and it'll be fine." It won't.

def fib(n, memo = {})
  return n if n <= 1
  memo[n] ||= fib(n - 1, memo) + fib(n - 2, memo)
end

fib(50_000)  # Still SystemStackError

Memoization skips redundant subproblem computation, but the call chain still descends linearly. fib(50000) calls fib(49999), which calls fib(49998), and so on. Every frame is live on the stack simultaneously. Memoization helps on the second call to fib(49999), not on the first. For large inputs, the right answer is bottom-up DP:

def fib(n)
  return n if n <= 1
  a, b = 0, 1
  (n - 1).times { a, b = b, a + b }
  b
end

O(n) time, O(1) space, no stack involved. If an interviewer asks why you wrote it iteratively, you now have a complete explanation ready.

Where This Comes Up in Interviews

Binary trees. Interview trees are typically balanced, so depth is log n. A tree of 100,000 nodes sits at depth around 17. Recursive DFS is fine. If the problem says "the tree may be a degenerate chain" or the input looks like a linked list, ask about depth or switch to iterative by default.

Graph DFS. A path graph of n = 100,000 nodes has depth n. Recursive DFS will crash. Iterative DFS is the correct default when depth can be linear in n.

Nested parsing. Recursive parsers go as deep as the nesting level. Fine for typical contest constraints, not for "up to 10^5 levels of nesting."

Backtracking. Recursion depth equals the decision depth (path length), not the total state count. Backtracking over strings of length 20 is safe. Length 10,000 is not.

The heuristic that transfers: if recursion depth is bounded by log n or a small constant, recurse freely. If it's bounded by n, and n can reach 10,000 or more, use an explicit stack. An interviewer probing on this is testing whether you know the difference between problem complexity and stack usage, two things that often diverge. This pairs with the broader set of patterns in Ruby coding interview gotchas.

If you want to practice explaining tradeoffs like this under live pressure, SpaceComplexity runs voice-based mock interviews with rubric-based feedback, including gotcha questions on language-specific behavior.

The Numbers at a Glance

Item	Detail
Error type	SystemStackError
Default stack size	~8 MB (Linux/macOS)
Typical safe depth	Up to 10,000 to 20,000 frames
Practical interview safe depth	n up to several hundred
Tail-call optimization	Exists; off by default; non-standard
Increase stack	RUBY_THREAD_STACK_SIZE env var or ulimit -s
Memoization prevents stack overflow?	No
Correct fix for deep recursion	Explicit stack array on the heap

Further Reading

• Ruby documentation: SystemStackError
• MRI Ruby source on GitHub
• Ruby documentation: RubyVM::InstructionSequence
• Wikipedia: Call stack
• POSIX thread stack size (Linux man page)