Can we have the Standard Library for Macros?

Game of the Year Edition

Mateusz Kubuszok

About me

Scala developer for almost 11 years
co-author and maintainer of Chimney for about 9 of them
blogged at Kubuszok.com
wrote Things you need to know about JVM (that matter in Scala)
several presentations about metaprogramming in Scala

Why macros?

The background

The Chimney Story

Nine years ago, I had to solo maintain some backend service where we wanted to have a clear separation of concerns between the JSON models, database models, domain models, and converting back and forth was really tiring, which is why together with my friend Piotr Krzemiński, we kicked off Chimney.

The initial version was based on Shapeless, obviously as everything else, but very soon I ran into some very long compilation times and adding more features which were obviously needed not only by us but also by other users that appeared at the time. We learned that implementing it with Shapeless grew exponentially hard. So we rewrote the whole library to macros.

During our way we learned that macros are very rough around the edges, that you can very easily write code that is about as maintainable as a giant JavaScript blob with no types and with no hints and with no tooling. So even though on a regular basis we tried to refactor it, create some nice utilities, keep things nicely separated, after a while the development of the library slowed down to a halt.

And despite what many of you believe, it was even before Scala 3 appeared. The fact that Chimney needed something like two, maybe three years to get ported to Scala 3 was not interestingly caused by the difference between macro systems on Scala 2 and Scala 3. The issue was that even on Scala 2 we already got to the point that we the maintainers couldn’t fit the whole code in our heads, we couldn’t refactor it, we couldn’t follow what is actually happening there.

Even excluding Scala 3 migration, we learned that we have to somehow refactor this codebase into some more typed utilities. Ones which are telling us with what we are working at every single stage, telling us the type of every single intermediate expression, something which would handle the corner cases around pattern matching or creating types with type parameters, handling AnyVals, handling implicits, recursion, etcetera.

From that point of view Scala 3 was just the straw that broke the camel’s back. If we had these tools and used them consistently across the whole codebase, all we would have to do would be provide an implementation for Scala 3 and the porting of the whole library would be much easier.

We immediately discarded the possibility that we should just port it as a separate codebase because the complexity of this whole codebase was so big that we had absolutely zero confidence that if we just rewrote the code from scratch we would have any sort of behavior parity between both versions. It would be virtually a new library and quite a lot of our users were still on Scala 2.13 and 2.12 so it made no sense to have two versions of the library when changing the Scala version would also force them to rewrite how they are using Chimney because the behavior changed in basically unpredictable ways. The only way forward for us was to design in such a way that both versions have behavior parity, they could share as much tests as possible, they could share as much of the code as possible to make it aligned by design.

That was a huge effort which took about a year, year and a half, and implementing the Scala 3 version was basically a side product of this whole foundational rework of the Chimney implementation. Very early we figured out that we could have one common interface, shared by both versions of Scala that could be just used to implement the business logic so to say. While the Scala 2 and Scala 3 parts would be just implementation details. Scala 2 and Scala 3 macros would be just implementation details.

Additionally, we also researched how to make the automatic derivation with Chimney possible without using all these weird automatic versus semi-automatic split. We designed utilities to aggregate all of the errors in the macro so that the user would have full access to every single error at once so that it could be fixed in one go rather than running it again and again and again and learning of one issue at a time. We developed a way of logging what is happening inside a macro so that if you wanted to complain that you have no idea what is happening inside Chimney, you could just add a single implicit and get a whole log telling you what is happening, what kind of decisions were made inside a macro, why, which would be a huge improvement over whatever you would have in Shapeless or Mirrors based derivation. And also both of these things, much faster than Shapeless and Mirrors.

So then Chimney 0.8.0 was released and after a bit of polishing it got upgraded to Chimney 1.0.0. We believe it was a huge engineering success and for me it was an eye-opening experience that we can have much better derivation tools than basically what any other library is providing and that it’s not a matter of Chimney being special, it’s something that could be doable by every other library that uses type class derivation.

Then 3.7 came, and broke our automatic derivation, but we managed to convince the Scala team to add some utilities to the standard library.

What we discovered

we can share macros code between Scala 2 and Scala 3
macros are not slow - abusing type checker via implicit resolution is
macros can aggregate errors, and know their context - better errors
macros can log their internal logic and resulting expression
macros have more opportunities to micro-optimize the code
while usage of semi-auto still makes sense - it is not required to use everywhere at all time to keep the project maintainable - that’s issue of Shapeless/Circe approach, not the type class derivation itself
eye-opening: we can have much better derivation tools
not a matter of Chimney being special — doable by every library doing type class derivation

Glorious Shapeless Derivation Master Race vs Dirty Macro Peasant

"Words are cheap

Show me the code"

"The code" (& numbers)

implicit not found

 // vs

Failed to derive showing for value:
  example.ShowSanely.User:
No build-in support nor implicit
for type scala.Nothing

Benchmarks

Previous presentation

The downside

pain to implement for library authors
unless we make it easier, nobody would actually use this
Chimney’s tools, though a huge improvement, have rough edges
not suitable for generic type class derivation as-is

The horror

Why macros suck

Learning resources

virtually no sources
official Scala documentation is basically…

Some examples

Quoting and Splicing

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val expr1 =
  c.Expr[Int](q"21")
val expr2 =
  c.Expr[Int](q"37")

val expr3 = c.Expr[Int](
  q"""
  ${ expr1 } + ${ expr2 }
  """
)

val expr1 = Expr(21)
val expr2 = Expr(37)

val expr3 = '{
  ${ expr1 } + ${ expr2 }
}

Matching Types

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

def whenOptionOf[A:c.WeakTypeTag] =...

weakTypeOf[A]
 .dealias
 .widen
 .baseType(
  c.mirror.staticClass("scala.Option")
 ) match {
  case TypeRef(_, _, List(t)) =>
    whenOptionOf(
      c.WeakTypeTag(t.dealias.widen)
    )
  case _ => ...
}

def whenOptionOf[A: Type] = ...

Type.of[A] match {
  case '[Option[t]] =>
    whenOptionOf[t]
  case _ => ...
}

Instantiating an Arbitrary Type

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val args:List[List[c.Tree]]=
  ...

c.Expr[A](
  q"""
  new ${weakTypeOf[A]}(
    ...${args}
  )
  """
)

val ctor = TypeRepr.of[A]
  .typeSymbol
  .primaryConstructor

val args: List[List[Tree]] =
  ...

New(TypeTree.of[A])
  .select(ctor)
  .appliedToArgss(args)

Constructing a Pattern Match

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

def handleCase[
  A: c.WeakTypeTag
](name: c.Expr[A]) = ...

/* for each case: */
val name = c.internal
  .reificationSupport
  .freshTermName("a")
cq"""
$name: ${weakTypeOf[A]} =>
  ${handleCase(c.Expr[A](q"$name"))}
"""

/* then create the match: */
c.Expr[Result](
  q"""
  $expr match { ...${cases} }
  """
)

def handleCase[
  A: Type
](name: Expr[A]) = ...

/* for each case: */
val name = Symbol.newBind(
  Symbol.spliceOwner,
  Symbol.freshName("a"),
  Flags.Empty,
  TypeRepr.of[A]
)
CaseDef(
  Bind(
    name,
    Typed(Wildcard(),TypeTree.of[A])),
  None,
  handleCase(Ref(name).asExprOf[A]))

Match(expr.asTerm, cases)
  .asExprOf[Result]

Sealed Trait’s Children

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val symbol = c.weakTypeOf[A]
  .typeSymbol
if (symbol.isSealed) {
  // force Symbol initialization
  symbol.typeSignature
  val children = symbol.asClass
    .knownDirectSubclasses.map{sym =>
      val sEta = sym.asType
        .toType.etaExpand
      sEta.finalResultType
          .substituteTypes(
        sEta.baseType(symbol)
          .typeArgs.map(_.typeSymbol),
        c.weakTypeOf[A].typeArgs
      )
    }
  ...
} else {
  ...
}

val A = TypeRepr.of[A]
val sym = A.typeSymbol
if (sym.flags.is(Flags.Sealed)) {
  val c = sym.children.map: sub =>
    sub.primaryConstructor
        .paramSymss match:
      // manually reapply type params
      case syms :: _
      if syms.exists(_.isType) =>
        val param2tpe = sub.typeRef
          .baseType(sym).typeArgs
          .map(_.typeSymbol.name)
          .zip(A.typeArgs).toMap
        val types = syms.map(_.name)
          .map(param2tpe)
        sub.typeRef.appliedTo(types)
      // subtype is monomorphic
      case _ => sub.typeRef
  ...
} else { ... }

Jackson Pollock

(Not a macro, but looks very similar)

The dream

Let’s imagine such an API

Macro IO

val a = MIO {
  21
}
val b = MIO {
  37
}

a.map2(b)(_ + _) // applicative syntax

for {
  i <- MIO(1)
  j <- MIO(2)
} yield i + j // monadic syntax

List("1", "2", "3", "a", "b").parTraverse { a =>
  MIO(a.toInt)
} // .par* aggregates errors

Logging

Log.namedScope("All logs here will share the same span") {
  Log.info("Some operation starting") >> // standalone log

    MIO("some operation")
      .log.info("Some operation ended") >> // log after IO

    Log.namedScope("Spans can be nested") {
      Log.info("Nested log") // we can nest as much as we want
    }
}

All logs here will share the same span:
├ [Info]  Some operation starting
├ [Info]  Some operation ended
└ Spans can be nested:
  └ [Info]  Nested log

Let’s assume that it can be a thing

// Yet another utility, because .map/.flatMap
// cannot handle this:
MIO.scoped { runSafe =>

  Expr.quote { // <- instead of '{}/ q"..."
    new Show[A] {

     def show(a: A): String = Expr.splice {// <- instead of ${}
        runSafe {
          deriveShowBody(Expr.quote{ a })// : MIO[Expr[String]]
        } // : Expr[String]
      }
    }
  } // : Expr[Show[A]]
} // : MIO[Expr[Show[A]]]

And this as well:

val OptionType = Type.Ctor1.of[Option]
val EitherType = Type.Ctor2.of[Either]

Type[A] match {
  case OptionType(a) =>
    ... // a is A in Option[A]
  case EitherType(l, r) =>
    ... // l is L and r is R in Either[L, R]
  case _ =>
    ... // A is not an Option or Either
}

Imagine you created instances like this:

CaseClass.parse[A] match {
  case Some(caseClass) =>
   // A(summon[Arg1], summon[Arg2], ...)
   caseClass.construct { parameter =>
      import parameter.tpe.Underlying as Param // <- giving existential type a name!

      Expr.summonImplicit[Param] match {
        case Some(expr) => MIO.pure(expr)

        case None => MIO.fail(
          new Exception(s"No implicit for ${Type.prettyPrint[Param]}")
        )
      }
   }

  case None => MIO.fail(
    new Exception(s"Not a case class: ${Type.prettyPrint[A]}")
  )
} // : MIO[Expr[A]]

And pattern-matched like this:

Enum.parse[A] match {
  case Some(enumm) =>
    // expr match {
    //   case b: B => "B" + " : " + b.toString
    //   ...
    // }
    enumm.matchOn(expr) { matchedSubtype =>
      import matchedSubtype.{Underlying as B, value as b}
      // B <- named existential type
      // b: Expr[B]
      val bName = Expr(B.simpleName) // Expr[String]
      MIO {
        Expr.quote {
          Expr.splice { bName } + " : " + Expr.splice { b }.toString
        }
      }
    }

  case None => MIO.pure(Expr("not an enum"))
} // : MIO[Expr[String]]

And handled collections like this:

val expr: Expr[A] = ...

Type[A] match {
  case IsCollection(isCollection) =>
    import isCollection.{ Underlying as Item, value => isCollectionOfItem }
    // Now we can use `Item` as the type of the collection elements

    val iterable: Expr[Iterable[Item]] = isCollectionOfItem.asIterable(expr)
    Expr.quote {
      Expr.splice(iterable).map { (item: Item) => ... } // Type[Item] is handled
    }
  case _ => ...
}

And value classes like this:

val outer: Expr[A] = ...

Type[A] match {
  case IsValueType(isValueType) =>
    import isValueType.{ Underlying as Inner, value => outerIsValueType }
    // Now we can use Inner as the value class's underlying type

    val unwrapped: Expr[Inner] = outerIsValueType.unwrap(outer)
    outerIsValueType.wrap match {
      case CtorLikeOf.PlainValue(ctor, _) =>
        ctor(unwrapped) // <-- wrapped again, not really useful but possible
      case _ => // other constructors
    }
  case _ => ...
}

And integrated external libraries like this:

import cats.data.NonEmptyList
import eu.timepit.refined.Refined
import eu.timepit.refined.collection.NonEmpty
import eu.timepit.refined.numeric.Positive

case class WithNEL(values: NonEmptyList[Int])
case class RefinedPerson(
  name: String Refined NonEmpty,
  age: Int Refined Positive
)

import hearth.kindling.circederivation._
// No imports for derivation needed! Just add the integration as dependency.

// These just work — encoding, decoding, validation:
KindlingsEncoder.encode(WithNEL(NonEmptyList.of(1, 2, 3)))
// => {"values": [1, 2, 3]}

KindlingsDecoder.decode[WithNEL](Json.obj("values" -> Json.arr()))
// => Left(...) — rejects empty NonEmptyList!

KindlingsDecoder.decode[Int Refined Positive](Json.fromInt(-1))
// => Left(...) — validates the predicate!

And debugged it like this:

// Put outside of companion to prevent auto-summoning!
implicit val logDerivation: Show.LogDerivation = Show.LogDerivation

And traced it like this:

Test / scalacOptions ++= Seq(
  "-Xmacro-settings:hearth.mioBenchmarkScopes=true",
  s"-Xmacro-settings:hearth.mioBenchmarkFlameGraphDir=${crossTarget.value / "flame-graphs"}"
)

Actually, it’s already possible

with Hearth

Hearth

github.com/kubuszok/hearth

cross-compilable macros: Scala 2.13 & 3
including Cross-Quotes: Expr.quote / Expr.splice working on both Scala versions
high-level APIs like: CaseClass, Enum, IsCollection, IsValueType, …
MIO monad for laziness, structured logging and error aggregation
and flame graphs to give your macros observability and performance insights

Kindlings

The incubator for Hearth-based libraries

github.com/kubuszok/kindlings

my ShowPretty
Circe, Jsoniter Scala
scala-xml, scala-yaml
my UBJson
my Avro4s port
Tapir Schema
Cats' Kittens

And if you were worried who is actually using this thing — let me also introduce you to Kindlings, the incubator for Hearth-based libraries.

Sure, I have there a simple Show type class, which was the first type class I tested this library on.

But then I also ported Circe derivation into my sanely-automatic derivation approach: so with Kindlings you would be able to derive encoders and decoders for Circe. Both automatic and semi-automatic with virtually all of the features supported by circe-generic-extras on both Scala 2 and Scala 3 with the exactly same API. Probably faster than what Circe is doing right now. Probably it will also run faster, almost certainly with better error messages.

Also for my own amusement I ported jsoniter-scala to this approach. I added a few utilities that I liked like JSON AST that is not pulling the whole Cats ecosystem if you only want to use jsoniter, some extension methods allowing you to invariant-map over it, etcetera.

For my amusement I also generated type class derivation for scala-xml, scala-yaml, added my own UBJson library, ported avro4s.

Circe derivation

Encoder[A], Decoder[A], Codec.AsObject[A]
all circe-generic-extras features on Scala 2 and Scala 3
unified API across Scala versions
value class unwrapping (broken in upstream Scala 3)
recursive types without Lazy wrappers
@fieldName annotation on both Scala 2 and 3 (upstream: Scala 2 only)
JVM + JS + Native

Jsoniter-Scala derivation

JsonValueCodec[A], JsonCodec[A], JsonKeyCodec[A]
virtually all JsonCodecMaker configuration options
unified API across Scala versions
recursive types without special flags
JVM + JS + Native

Tapir Schema derivation

reuses your Circe or jsoniter-scala configuration
schema always matches your codecs
correct generic type parameter names

// Your Tapir schema automatically matches your JSON codec config
def schema[A: Schema]: Schema[Foo[A]] = Schema.derived
// schema[String] will be named Foo[String] instead of Foo[A]!

no more config drift between schema and codec

Cats' Kittens port

And if you think it’s still too limited to prove anything useful beyond encoders and decoders…

Monomorphic (kind *)

Polymorphic (kind * → *)

Show, Eq, Order
PartialOrder, Hash
Semigroup, Monoid
CommutativeSemigroup
CommutativeMonoid
alleycats.Empty

Functor, Contravariant, Invariant
Apply, Applicative
Foldable, Traverse
Reducible, NonEmptyTraverse
SemigroupK, MonoidK
alleycats.Pure, alleycats.EmptyK
NonEmptyAlternative (new!)
Alternative (new!)
alleycats.ConsK (fixed on Scala 3!)

All of this…

sharing the type class derivation logic
sharing the unit tests
exact same API on Scala 2.13 and Scala 3
use them today on Scala 2.13 — migration to Scala 3 is not a blocker

The secret weapon

AI-assisted development

virtually every single one of these libraries was completely vibe-coded
over one week
using Claude Code to implement all of it
a single person, one week, if really dedicated

libraries that took years to develop — ported in days
a lot of tokens burned, then bam — everyone benefits
the API is sane enough that AI can use it effectively
this is empowering for the community

Summary

macros can be better for users than Shapeless/Mirrors
Hearth makes macros approachable for library authors
Kindlings proves it works across many real-world libraries
AI makes the development fast
we can give ourselves better tools — today

Thank you

Hearth

Kindlings

Can we have the Standard Library for Macros?

About me

Why macros?

The Chimney Story

What we discovered

"Words are cheap

"The code" (& numbers)

The downside

The horror

Learning resources

Some examples

Quoting and Splicing

Matching Types

Instantiating an Arbitrary Type

Constructing a Pattern Match

Sealed Trait’s Children

Jackson Pollock

The dream

Macro IO

Logging

Let’s assume that it can be a thing

And this as well:

Imagine you created instances like this:

And pattern-matched like this:

And handled collections like this:

And value classes like this:

And integrated external libraries like this:

And debugged it like this:

And traced it like this:

Actually, it’s already possible

Hearth

Kindlings

Circe derivation

Jsoniter-Scala derivation

Tapir Schema derivation

Cats' Kittens port

All of this…​

The secret weapon

Summary

Thank you

All of this…