Refactoring Large, Stubborn Codebases

November 19, 2024

Complaints about stubborn codebases

• Our code isn’t modular enough!

Best to centralize the refactor ☀️

Have one team drive the refactor:

• concentrates expertise
  most problems will be repeat problems

• incentivizes automation
  fewer engineer-hours overall

• more likely to finish
  no need to wait for each team to plan and prioritize

Centralized migration needs two things:

Agenda 🎯

• Improving developer satisfaction with Sorbet

• Making a Ruby monolith more modular

• Lessons learned from ratchets ratcheted

🥺 Stripe’s developers were unhappy

(sentiment from company-wide survey)

💡 Building Sorbet introduced leverage

(sentiment from company-wide survey)

Brief history of Sorbet 👨‍🏫

• Began fall 2017
• Stripe: ~800 employees (200 – 400 engineers)
• Initial project: 3 engineers, full time

Timeline

• 9 months to build Sorbet…
  … but has served as the foundation for hundreds of codemods
• 3 months to get to 75% adoption…
  …contained to just three engineers

Aside: you can do it too! 👷🏼‍♀️

Tools that you can use to bootstrap something:

• ESLint, RuboCop, your language’s linter
• Clang compiler plugins, C# static analyzers, go analysis package
• BYO: tree-sitter parsers, LSP server libraries, …
• GitHub code search for example projects using these tools

Ratcheting with # typed comments

# typed comment at the top of each file

typed: false → true

💡 local, incremental, and actionable

Alternatives to # typed: comment:

• by folder → too broad
  (not local enough, not incremental enough)

• by coverage percent → too granular
  (noisy, hard to action)

actionable = high signal, low noise

Developer satisfaction improved because

Agenda 🎯

• Improving developer satisfaction with Sorbet

• Making a Ruby monolith more modular

• Lessons learned from ratchets ratcheted

Why do we need modularity?

Simple example

# a toy logger
class Logger
  def log(message, **storytime)
    payload = storytime.map do |k, v|
      "#{k}=#{v.inspect}"
    end.join(" ")

    @output.puts("#{Time.now.to_i}: #{message} #{payload}")
  end
end

Simple example

# elsewhere
logger.log("Attempting operation", op: my_op, merchant: m)
# 1730756308: Attempting operation op=:update merchant=#<Merchant id=22 secret="hunter2">

Simple example

    # ...
    payload = storytime.map do |k, v|
      if v.is_a?(Merchant)  # if we're logging a merchant...
        "#{k}=Merchant(id=#{v.id}, ...)"  # redact most fields
      else
        "#{k}=#{v.inspect}"  # other objects can be logged as-is
      end
    end.join(" ")
    # ...

Simple example

    # ...
    payload = storytime.map do |k, v|
      if v.is_a?(Merchant)  # if we're logging a merchant...
        "#{k}=Merchant(id=#{v.id}, ...)"  # redact most fields
      else
        "#{k}=#{v.inspect}"  # other objects can be logged as-is
      end
    end.join(" ")
    # ...

Well-intentioned changes can produce tangled code

…and tangled code has non-local effects!

Why do we need modularity?

Point of leverage: packaging

# lib/logger/__package.rb
class Logger < PackageSpec
  import Merchant

  export Logger
end

# lib/merchant/__package.rb
class Merchant < PackageSpec
  import Logger

  export Merchant
end

Point of leverage: layering

The essential principle is that any element of a layer depends only on other elements in the same layer or on elements of the layer ’beneath’ it. Communication upward must pass through some indirect mechanism.

—Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software

Point of leverage: layering

Point of leverage: layering

Point of leverage: layering

class Logger < PackageSpec
  layer 'utility'
  import Merchant # <- ill-layered import!
  export Logger
end

class Merchant < PackageSpec
  layer 'business'
  import Logger
  export Merchant
end

Building a ratchet: strict_dependencies

Level zero: 'false'

Level one: 'layered'

Level one: 'layered'

Level two: 'layered_dag'

Level two: 'layered_dag'

Level three: 'dag'

Level three: 'dag'

Building a ratchet: strict_dependencies

• strict_dependencies 'false'
• strict_dependencies 'layered'
• strict_dependencies 'layered_dag'
• strict_dependencies 'dag'

Developer velocity and production latency are improving because

What makes a good ratchet?

• local
  • Sorbet: per-file
  • Dependencies: per-package
• incremental
  • Sorbet: false → true → strict
  • Dependencies: false → layered → layered_dag → dag
• actionable
  • Sorbet: “Where do I need types in my current files?”
  • Dependencies: “What bad edges can I remove from my current package?”

Agenda 🎯

• Improving developer satisfaction with Sorbet

• Making a Ruby monolith more modular

• Lessons learned from ratchets ratcheted

How can this approach fall down?

In theory, there is no difference between theory and practice. In practice, there is.

— Walter J. Savitch, relaying a quote overheard at a computer science conference

Beyond leverage and ratchets:

Important to have:

Tools aren’t always perfect at first!

Tools aren’t always perfect at first!

Corollary: don’t rush the launch!

Who ratchets the ratchets?

Our approach: two-level ratchets

class MyPackage < PackageSpec
  sorbet min_typed_level: 'strict'
  # ...
end

Our approach: two-level ratchets

class MyPackage < PackageSpec
  sorbet min_typed_level: 'strict', test_min_typed_level: 'true'
  # ...
end

Our approach: two-level ratchets

Our approach: two-level ratchets

Our approach: two-level ratchets

Sorbet supporting tooling

Packager supporting tooling

… and don’t forget to be patient—it’s hard work!

Questions?