override lazy val

Peregrinations of a developer in Scala land

Shapeless : not a tutorial - part 2

In the first instalment of this series, we discussed hlists and polys. But so far, we’ve not seen anything close to a usable piece of software.

The last piece we need to make this useful in real life is a way to convert our regular objects back and forth to their generic counterpart.

Generic : making it real

We’ve claimed earlier that hlists can represent any product type like tuples or case classes. Although we can easily devise a method to transform any product to and from hlists – the representation of tuple for example, would be the hlist of its elements – it is not possible to encode such generic conversion using only “normal” scala.

Fortunately, all the required information to express such algorithm is available through the reflexion API, so generic conversions can be implemented using macros.

That is precisely what shapeless.Generic does : it provides the hlist representation of a case class (or a tuple) and operations to convert an instance of the case class to its hlist representation, and back.

scala> case class User(id: Long, name: String)
defined class User

scala> val genUser = Generic[User]
genUser: shapeless.Generic[User]{type Repr = shapeless.::[Long,shapeless.::[String,shapeless.HNil]]} = fresh$macro$24$1@6bd96d74

scala> genUser.to(User(1L, "Homer"))
res3: genUser.Repr = 1 :: Homer :: HNil

scala> genUser.from(2L :: "Marge" :: HNil)
res4: User = User(2,Marge)

Please note though that product (ie hlists) is not a one-size-fits-all abstraction that can represent any type. Incidentally, Generic will not represent all possible types using HList, but that discussion is beyond the scope of the present article.

Singleton types

Before diving deeper into generic representations, let us make a little digression to discuss singleton types. A singleton type is a type for which there exists exactly one value (it is also said that such type has only one inhabitant).

You already have encountered singleton types, maybe without knowing it. Indeed, the type any object is a singleton type. So for example, when we write MakeBigger.type, we refer to the type that has for sole value object MakeBigger.

But there are also plenty other singleton types that scalac keeps hidden. In fact, scalac assigns a singleton type to every literal value.

For example, the type of 42L is internally represented as Long(42), the type of the only Long that represents the number 42.

Unfortunately, this internal representation of literals as singleton-typed values is not directly accessible to the developer. But it is accessible through the reflexion API, and shapeless internally uses macros to materialize singleton types.

Now why this digression ? We’ll see shortly that singleton types are useful to build enhanced generic representations of case classes. But a broader consequence of the ability to express singleton types is that we can now lift values into the type system.

Because referring to a type that can have only one value is equivalent to refer directly to that value.

Records and LabelledGeneric

import shapeless.LabelledGeneric
import shapeless.syntax.singleton._
import shapeless.record._

Shapeless uses the power of singleton types to build the record data structure. A record is a hlist which elements have a singleton-typed label attached to their types; these elements act as the fields of the record.

The label is a simple trait with no method and two types parameters :

// Labels a value of type V with a label of type K
trait KeyTag[K, +V] {}

The type for each field is literally “a value with a label” :

// A field containing a value of type V with a label of type K
type FieldType[K, +V] = V with KeyTag[K, V]

We could virtually use any type L as a label and create a field out of a value of type V by simply casting it to FieldType[L, V]. As KeyTag is an empty trait, the cast would always work.

But this would yield something completely useless since there would be no value of type L attached to it (one cannot, for example, get a usable value out of the type Option[String] itself). There’s nothing more we can do with a FieldType[L, V] compared to a simple V.

Now things are different if we use a singleton-type as the label. We can always extract the (unique possible) value from a singleton type, so using one as a label effectively attaches a value to our V.

That’s the purpose of the ->> operation defined in shapeless.syntax.SingletonOps : make a field out of an arbitrary value and a singleton-typed key :

scala> val field = 1 ->> Some("value")
field: Some[String] with shapeless.labelled.KeyTag[Int(1),Some[String]] = Some(value)

scala> val field2 = "foo" ->> List(1, 2, 3) 
field2: List[Int] with shapeless.labelled.KeyTag[String("foo"),List[Int]] = List(1, 2, 3)

scala> val field3 = false ->> 42L
field3: Long with shapeless.labelled.KeyTag[Boolean(false),Long] = 42

An instance of SingletonOps is implicitly available for every kinds of literals (roughly every subtype of AnyVal).

We will not extend any further our discussion on records, but you should know that shapeless provides us with many useful operations to manipulate them : selecting fields by their key, merging two records together, and so on.

More importantly, shapeless use records as the basis of a more precise generic representation of case classes : LabelledGeneric.

Our previous generic representation for product with simple hlists (the one that is accessible through Generic) didn’t retain the names of the case class’ fields. This is fixed using LabelledGeneric which represents case classes as records, using singleton-typed Symbols as keys (field names) :

scala> case class Rectangle(width: Int, height: Int)
defined class Rectangle

scala> val genRectangle = LabelledGeneric[Rectangle]
genRectangle: shapeless.LabelledGeneric[Rectangle]{type Repr = shapeless.::[Int with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("width")],Int],shapeless.::[Int with shapeless.labelled.KeyTag[Symbol with shapeless.tag.Tagged[String("height")],Int],shapeless.HNil]]} = shapeless.LabelledGeneric$$anon$1@70547aa9

scala> val repr = genRectangle.to(Rectangle(1,3))
repr: genRectangle.Repr = 1 :: 3 :: HNil

scala> repr.get('width)
res1: Int = 1

scala> val modified = repr.updated('height, 42)
modified: /* type omitted */ = 1 :: 42 :: HNil

scala> genRectangle.from(modified)
res4: Rectangle = Rectangle(1,42)

Notice how the names of the fields appear in the (huge) Repr type of genRectangle. More importantly, see how we were able to get and modify fields of the generic representation by their names.

Wrapping it up : a minimal JSON writer

Finally, lets tie it all up with a concrete example. We will write a very concise, and yet useful JSON writer that involves no runtime reflexion at all, and without having to write any macro.

Of course this minimalistic (less than 50 line of code) is not suitable for production use. It does not handle character escaping in strings and may render numbers oddly in certain edge cases.

But more importantly, it represents an occasion to put all we’ve learned so far in good use.

import shapeless.{HList, HNil, ::, Poly1, LabelledGeneric, Witness}
import shapeless.labelled.FieldType

So we want to define a poly ToJson that, given any object, will return a string containing the JSON representation of that object.

First, lets make our lives a bit easier by defining this little helper :

type ==>[In, Out] = Case.Aux[In, Out]

This will allow us to write In ==> Out instead of Case.Aux[In, Out] which I find a little clearer.

Now we can define ToJson for “primitive” types (booleans, numbers, strings), for lists (that render as arrays) and options (None is rendered as null).

object ToJson extends LowPriorityToJson {

  implicit def numericCase[N : Numeric]: N ==> String = at[N](_.toString)

  implicit def booleanCase: Boolean ==> String = at[Boolean](_.toString)

  implicit def stringCase: String ==> String = at[String]("\"" + _ + "\"")

  implicit def listCase[A]
  (implicit elemToJson: A ==> String)
  : List[A] ==> String =
    at[List[A]](l => "[" + (l map elemToJson.apply[A] mkString ", ") + "]")

  implicit def optionCase[A]
  (implicit elemToJson: A ==> String)
  : Option[A] ==> String =


We then want to render any other product types (tuples, case classes, etc…) as JSON objects. We need to isolate this case in a super type because lists and options are also products, and this would lead to ambiguity.

The mechanism to render a product type is quite simple :

  • transform the product to its generic representation with LabelledGeneric
  • for each FieldType[K, V] in the representation
    • recursively render V to JSON
    • use the value of K to output a field
  • wrap the fields in curly braces, separated by a coma
trait LowPriorityToJson extends LowerPriorityToJson {

  implicit def productCase[A <: Product, R <: HList]
  (implicit gen: LabelledGeneric.Aux[A, R],
            fieldsToJson: R ==> List[String])
  : A ==> String =
    at[A](a => "{" + fieldsToJson(gen.to(a)).mkString(", ") + "}")

  implicit def hnilCase : HNil ==> List[String] =
    at[HNil](_ => List.empty[String])

  implicit def hconsCase[K <: Symbol, H, T <: HList] // K is a singleton subtype of Symbol
  (implicit w: Witness.Aux[K], // this is how we get access to the single value of type  K
            headToJson: H ==> String,
            tailToJson: T ==> List[String])
  : (FieldType[K, H] :: T) ==> List[String] =
    at[FieldType[K, H] :: T](fields =>
      s""""${w.value.name}": ${headToJson(fields.head.asInstanceOf[H])}"""
      :: tailToJson(fields.tail)


Finally, we render every other types as JSON strings by using their toString method.

trait LowerPriorityToJson extends Poly1 {

  implicit def anyCase[T]: T ==> String = at[T]("\"" + _.toString + "\"")


Now we can use our ToJson poly to render arbitrary hierarchies of case classes. Here is a little example with characters from a famous TV show.

scala> :pa
// Entering paste mode (ctrl-D to finish)

import json._
import java.util.UUID
case class CharacterId(value: UUID = UUID.randomUUID())
case class Character(id: CharacterId, name: String, age: Int, hobbies: List[String], nemesis: Option[CharacterId] = None)
val seymourId = CharacterId()
val bart = Character(CharacterId(), "Bart", 12, List("skateboarding", "phone pranks"), Some(seymourId))
val lisa = Character(CharacterId(),"Lisa", 9, List("reading", "jazz music"))
val seymour = Character(seymourId, "Seymour", 42, List(), Some(bart.id))

// Exiting paste mode, now interpreting.

// ...

scala> ToJson(bart)
res0: String = {"id": {"value": "dc0ec251-f9c0-4e5c-8699-0f651cf3fd71"}, "name": "Bart", "age": 12, "hobbies": ["skateboarding", "phone pranks"], "nemesis": {"value": "3ef12363-50b9-423a-b93e-d77893f2aa69"}}

scala> ToJson(lisa)
res1: String = {"id": {"value": "f541f20c-7a34-4b16-89f0-489b78859200"}, "name": "Lisa", "age": 9, "hobbies": ["reading", "jazz music"], "nemesis": null}

scala> ToJson(seymour)
res2: String = {"id": {"value": "3ef12363-50b9-423a-b93e-d77893f2aa69"}, "name": "Seymour", "age": 42, "hobbies": [], "nemesis": {"value": "dc0ec251-f9c0-4e5c-8699-0f651cf3fd71"}}


We managed to write a generic program that handle arbitrary inputs while remaining completely type-safe. Hopefully, you should have begun to sense how useful generic programming can be.

By allowing us to reason about types’ structure, it gives us a way to get rid of boilerplate code in a wide variety of situations, without the need of runtime reflexion.

But the picture is still far from complete. In future posts, we’ll talk about coproducts and dependent functions.

comments powered by Disqus


Hi ! I'm Valentin Kasas and this is my programming blog.

I am a happy Scala user since 2012 and here is where I recount some of my wanderings in that vast landscape.

Follow @vil1