Can we have the Standard Library for Macros?

Mateusz Kubuszok

About me

Scala developer for 10 years
co-author of Chimney
blogged at Kubuszok.com
wrote Things you need to know about JVM (that matter in Scala)

What is a Macro?

a code generator
run during compilation by the compiler
working on the AST
pretending to be a method
Mirror s, inline if s, inline match , summonInline , summonFrom , … - metaprogramming but not macros

// Scala 2
import scala.language.experimental.macros

def macroMethod[A](a: A): SthOf[A] =
  macro macroMethodImpl[A]

import scala.reflect.macros.blackbox

def macroMethodImpl[A: c.WeakTypeTag](
  c: blackbox.Context
)(a: c.Expr[A]): c.Expr[SthOf[A]] = {
  import c.universe._
  ...
}

// Scala 3
import scala.quoted.*

inline def macroMethod[A](a: A): SthOf[A] =
  ${ macroMethodImpl[A]('{ a }) }

def macroMethodImpl[A: Type](
  using q: Quotes
)(a: Expr[A]): Expr[SthOf[A]] = {
  import q.reflect.*
  ...
}

Why do we need macros?

Alternatives

Shapeless (Scala 2)
inline , Mirrors & scala.compiletime (Scala 3)
Magnolia (Scala 2 & 3)
runtime reflection

But

implicit not found

 // vs

Failed to derive showing for value:
  example.ShowSanely.User:
No build-in support nor implicit
for type scala.Nothing

In other words

Shapeless/Mirrors/etc

You

Your users

easy start
rapid development
everything feels lean
everything feels principled and correct

pay with longer compilation times
pay with less performant runtime
pay with more time spent if the code doesn’t work

Macros

You

Your users

longer setup
no batteries included
development feel clunky
everything feels like a hack

may get better compilation times (than alternatives)
may get more performant runtime
may get better errors messages
or may get an undebugabble mess that cannot be understood even by the author

Macros API

Opportunities for improvement

Quoting and Splicing

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val expr1 =
  c.Expr[Int](q"21")
val expr2 =
  c.Expr[Int](q"37")

val expr3 = c.Expr[Int](
  q"""
  ${ expr1 } + ${ expr2 }
  """
)

val expr1 = Expr(21)
val expr2 = Expr(37)

val expr3 = '{
  ${ expr1 } + ${ expr2 }
}

Matching Types

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

def whenOptionOf[A:c.WeakTypeTag] =...

weakTypeOf[A]
 .dealias
 .widen
 .baseType(
  c.mirror.staticClass("scala.Option")
 ) match {
  case TypeRef(_, _, List(t)) =>
    whenOptionOf(
      c.WeakTypeTag(t.dealias.widen)
    )
  case _ => ...
}

def whenOptionOf[A: Type] = ...

Type.of[A] match {
  case '[Option[t]] =>
    whenOptionOf[t]
  case _ => ...
}

Instantiating an Arbitrary Type

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val args:List[List[c.Tree]]=
  ...

c.Expr[A](
  q"""
  new ${weakTypeOf[A]}(
    ...${args}
  )
  """
)

val ctor = TypeRepr.of[A]
  .typeSymbol
  .primaryConstructor

val args: List[List[Tree]] =
  ...

New(TypeTree.of[A])
  .select(ctor)
  .appliedToArgss(args)

Constructing a Pattern Match

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

def handleCase[
  A: c.WeakTypeTag
](name: c.Expr[A]) = ...

/* for each case: */
val name = c.internal
  .reificationSupport
  .freshTermName("a")
cq"""
$name: ${weakTypeOf[A]} =>
  ${handleCase(c.Expr[A](q"$name"))}
"""

/* then create the match: */
c.Expr[Result](
  q"""
  $expr match { ...${cases} }
  """
)

def handleCase[
  A: Type
](name: Expr[A]) = ...

/* for each case: */
val name = Symbol.newBind(
  Symbol.spliceOwner,
  Symbol.freshName("a"),
  Flags.Empty,
  TypeRepr.of[A]
)
CaseDef(
  Bind(
    name,
    Typed(Wildcard(),TypeTree.of[A])),
  None,
  handleCase(Ref(name).asExprOf[A]))

Match(expr.asTerm, cases)
  .asExprOf[Result]

Sealed Trait’s Children

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

val symbol = c.weakTypeOf[A]
  .typeSymbol
if (symbol.isSealed) {
  // force Symbol initialization
  symbol.typeSignature
  val children = symbol.asClass
    .knownDirectSubclasses.map{sym =>
      val sEta = sym.asType
        .toType.etaExpand
      sEta.finalResultType
          .substituteTypes(
        sEta.baseType(symbol)
          .typeArgs.map(_.typeSymbol),
        c.weakTypeOf[A].typeArgs
      )
    }
  ...
} else {
  ...
}

val A = TypeRepr.of[A]
val sym = A.typeSymbol
if (sym.flags.is(Flags.Sealed)) {
  val c = sym.children.map: sub =>
    sub.primaryConstructor
        .paramSymss match:
      // manually reapply type params
      case syms :: _
      if syms.exists(_.isType) =>
        val param2tpe = sub.typeRef
          .baseType(sym).typeArgs
          .map(_.typeSymbol.name)
          .zip(A.typeArgs).toMap
        val types = syms.map(_.name)
          .map(param2tpe)
        sub.typeRef.appliedTo(types)
      // subtype is monomorphic
      case _ => sub.typeRef
  ...
} else { ... }

Scala 2 (Quasiquotes)

Scala 3 (Quotes)

gluing strings instead of typed code
but at least these strings resemble the real code
and compiler checks them

gluing expressions as every other real code
but it’s not an expression, we are gluing together untyped trees

No `println` debugging

Macro reporting

println

c.echo("msg") // Scala 2
report.info("msg") // Scala 3

println("msg")

works in the terminal
works in the IDE
works in Scastie
logs only the first message from the macro

prints every time
works only in the terminal
unless we are using some compilation server

Avoiding runtime dependencies in macros

no Cats
no ZIO
nor any other library, that could be used in runtime

Let us imagine a better API

Macro IO

val a = MIO {
  21
}
val b = MIO {
  37
}

a.map2(b)(_ + _) // applicative syntax

for {
  i <- MIO(1)
  j <- MIO(2)
} yield i + j // monadic syntax

List("1", "2", "3", "a", "b").parTraverse { a =>
  MIO(a.toInt)
} // .par* aggregates errors

Logging

Log.namedScope("All logs here will share the same span") {
  Log.info("Some operation starting") >> // standalone log

    MIO("some operation")
      .log.info("Some operation ended") >> // log after IO

    Log.namedScope("Spans can be nested") {
      Log.info("Nested log") // we can nest as much as we want
    }
}

All logs here will share the same span:
├ [Info]  Some operation starting
├ [Info]  Some operation ended
└ Spans can be nested:
  └ [Info]  Nested log

Let’s assume that it can be a thing

// Yet another utility, because .map/.flatMap
// cannot handle this:
MIO.async { await =>

  Expr.quote { // <- instead of '{}/ q"..."
    new Show[A] {

     def show(a: A): String = Expr.splice {// <- instead of ${}
        await {
          deriveShowBody(Expr.quote{ a })// : MIO[Expr[String]]
        }
      }
    }
  }
}

And this as well:

val OptionType = Type.Ctor1.of[Option]
val EitherType = Type.Ctor2.of[Either]

Type[A] match {
  case OptionType(a) =>
    ... // a is A in Option[A]
  case EitherType(l, r) =>
    ... // l is L and r is R in Either[L, R]
  case _ =>
    ... // A is not an Option or Either
}

Imagine you created instances like this:

CaseClass.parse[A] match {
  case Some(caseClass) =>
   // A(summon[Arg1], summon[Arg2], ...)
   caseClass.construct { parameter =>
      import parameter.tpe.Underlying as Param

      Expr.summonImplicit[Param] match {
        case Some(expr) => MIO.pure(expr)

        case None => MIO.fail(
          new Exception(s"No implicit for ${Type.prettyPrint[Param]}")
        )
      }
   }

  case None => MIO.fail(
    new Exception(s"Not a case class: ${Type.prettyPrint[A]}")
  )
} // : MIO[Expr[A]]

And pattern-matched like this:

Enum.parse[A] match {
  case Some(enumm) =>

    // expr match {
    //   case b: B => "B" + " : " + b.toString
    //   ...
    // }
    enumm.matchOn(expr) { matchedSubtype =>
      import matchedSubtype.{Underlying as B, value as b}
      val bName = Expr(Type.simpleName[B])
      MIO {
        Expr.quote {
          Expr.splice { b } + " : " + Expr.splice { bName }
        }
      }
    }

  case None => Expr("")
} // : MIO[Expr[String]]

Actually, it’s already possible

with Hearth

Hearth

Hearth demo

Code at:

Show[A] demo at hearth-tests/src/
main/scala/demo/Show.scala
main/scala/demo/ShowMacrosImpl.scala
main/scala-2/demo/ShowCompanionCompat.scala
main/scala-3/demo/ShowCompanionCompat.scala
test/scala-3/demo/ShowSpec.scala

Regular code

package hearth.demo

trait Show[A] extends Show[A] {

  def show(value: A): String
}

object Show extends ShowCompanionCompat // Will provide .derived[A]

Cross-compilable macro

package hearth.demo

import hearth.*
import fp.effect.*, fp.instances.*, fp.syntax.*

private[demo] trait ShowMacrosImpl { this: MacroCommons =>

  def deriveTypeClass[A: Type]: Expr[Show[A]] = Expr.quote {
    new Show[A] {
      def show(value: A): String = Expr.splice {
        deriveOrFail[A](Expr.quote(value))
      }
    }
  }

  private def deriveOrFail[A: Type](
    value: Expr[A]
  ): Expr[String] = ...
  // ...
}

Adapters (necessary for now)

package hearth.demo

import scala.language.experimental.macros, scala.reflect.macros.blackbox

private[demo] trait ShowCompanionCompat { this: Show.type =>

  def derived[A]: Show[A] = macro ShowMacros.deriveTypeClassImpl[A]
}

private[demo] class ShowMacros(val c: blackbox.Context)
    extends hearth.MacroCommonsScala2
    with ShowMacrosImpl {
  def deriveTypeClassImpl[A: c.WeakTypeTag]: c.Expr[Show[A]] = deriveTypeClass[A]
}

package hearth.demo

import scala.quoted.*

private[demo] trait ShowCompanionCompat { this: Show.type =>

  inline derived[A]: Show[A] = ${ ShowMacros.deriveTypeClass[A] }
}

private[demo] class ShowMacros(q: Quotes)
    extends hearth.MacroCommonsScala3(using q), ShowMacrosImpl

private[demo] object ShowMacros {
  def deriveTypeClass[A: Type](using q: Quotes): Expr[Show[A]] =
    new ShowMacros(q).deriveTypeClass[A]
}

private[demo] sealed trait DerivationError
    extends scala.util.control.NoStackTrace
    with Product
    with Serializable

private[demo] object DerivationError {

  final case class UnsupportedType(typeName: String) extends DerivationError
  ... // other cases
}

private def deriveOrFail[A: Type](value: Expr[A]): Expr[String] =
  Log.namedScope(s"Derivation for Show[${Type.prettyPrint[A]}]") {
    attemptAllRules[A](value) // <- this is the core of the logic
  }
  .expandFinalResultOrFail(s"Show[${Type.prettyPrint[A]}]") {
      (errorLogs, errors) =>
    val errorsStr = errors.toVector.map {
      case DerivationError.UnsupportedType(typeName) =>
        s"Derivation of $typeName is not supported"
      ... // other cases
      case e =>
        s"Unexpected error: ${e.getMessage}:\n${e.getStackTrace.mkString("\n")}"
    }.mkString("\n")

    s"""Failed to derive Show[${Type.prettyPrint[A]}]:
       |$errorsStr
       |Error logs:
       |$errorLogs
       |""".stripMargin
  }

/** Idea:
  *   - successful Some -> rule applies, attempt succeeded
  *   - successful None -> rule doesn't apply, we should try the next one
  *   - failure -> rule applies but it failed, we should fail the whole derivation
  * If none of the rules matched, then we fail derivation as well.
  */
private type Attempt[A] = MIO[Option[Expr[A]]]

private def attemptUsingImplicit[A: Type](value: Expr[A]): Attempt[String] = ...

private def attemptAsBuiltIn[A: Type](value: Expr[A]): Attempt[String] = ...

private def attemptAsIterable[A: Type](value: Expr[A]): Attempt[String] = ...

private def attemptAsCaseClass[A: Type](value: Expr[A]): Attempt[String] = ...

private def attemptAsEnum[A: Type](value: Expr[A]): Attempt[String] = ...

private def attemptAllRules[A: Type](value: Expr[A]): MIO[Expr[String]] =
  MIO.async { await =>
    await {
      attemptUsingImplicit[A](value)
    } orElse await {
      attemptAsBuiltIn[A](value)
    } orElse await {
      attemptAsIterable[A](value)
    } orElse await {
      attemptAsCaseClass[A](value)
    } orElse await {
      attemptAsEnum[A](value)
    } getOrElse await {
      MIO.fail(DerivationError.UnsupportedType(Type.prettyPrint[A]))
    }
  }

But what about logging?

// inside Show companion
sealed trait LogDerivation
object LogDerivation extends LogDerivation

// Put outside of [[Show]] companion to prevent the implicit
// from being summoned automatically!
implicit val logDerivation: Show.LogDerivation = Show.LogDerivation

/** Enables logging if we either:
  *   - import [[demo.debug.logDerivation]] in the scope
  *   - have set scalac option `-Xmacro-settings:show.logDerivation=true`
  */
private def shouldWeLogDerivation: Boolean = {
  implicit val LogDerivation: Type[Show.LogDerivation] = Types.LogDerivation
  def logDerivationImported = Expr.summonImplicit[Show.LogDerivation].isDefined

  def logDerivationSetGlobally = (for {
    data <- Environment.typedSettings.toOption
    show <- data.get("show")
    shouldLog <- show.get("logDerivation").flatMap(_.asBoolean)
  } yield shouldLog).getOrElse(false)

  logDerivationImported || logDerivationSetGlobally
}

// ...
.expandFinalResultOrFail(
   s"Show[${Type.prettyPrint[A]}]"
   renderInfoLogs = shouldWeLogDerivation) { // <- enable conditional logging
// ...

Can we have the Standard Library for Macros?

About me

What is a Macro?

Why do we need macros?

Alternatives

But

In other words

Shapeless/Mirrors/etc

Macros

Macros API

Quoting and Splicing

Matching Types

Instantiating an Arbitrary Type

Constructing a Pattern Match

Sealed Trait’s Children

No `println` debugging

Avoiding runtime dependencies in macros

Let us imagine a better API

Macro IO

Logging

Let’s assume that it can be a thing

And this as well:

Imagine you created instances like this:

And pattern-matched like this:

Actually, it’s already possible

Hearth

Hearth demo

Regular code

Cross-compilable macro

Adapters (necessary for now)

But what about logging?

Summary

Thank you

Can we have the Standard Library for Macros?

About me

What is a Macro?

Why do we need macros?

Alternatives

But

In other words

Shapeless/Mirrors/etc

Macros

Macros API

Quoting and Splicing

Matching Types

Instantiating an Arbitrary Type

Constructing a Pattern Match

Sealed Trait’s Children

No println debugging

Avoiding runtime dependencies in macros

Let us imagine a better API

Macro IO

Logging

Let’s assume that it can be a thing

And this as well:

Imagine you created instances like this:

And pattern-matched like this:

Actually, it’s already possible

Hearth

Hearth demo

Regular code

Cross-compilable macro

Adapters (necessary for now)

But what about logging?

Summary

Thank you

No `println` debugging