New Bamboo Web Development

Bamboo blog. Our thoughts on web technology.

Oat: explicit media-type serializers in Ruby

by Ismael Celis

Context

I've been writing HTTP APIs in Ruby for a while, and I've found that a common pain point involves how to design JSON/XML responses in a way that is sane, easy to test and doesn't lead to a lot of duplication.

There are plenty of tools and approaches out there. I will list a few and explain why I eventually decided to write another one.

Object#to_json

This approach was popularised by Rails years ago. Any object that implements #to_json can be serialized straight onto the world. Very useful for quick and dirty JSON APIs, since ActiveModel instances, including ActiveRecord and ActiveRecord associations, all implement this.

1 # some controller
2 def show
3   user = User.find(params[:id])
4   render json: user
5 end

This will serialize all attributes in the user model. You can customize what is and isn't exposed, but the baseline is exposing all attributes and then allowing you to blacklist/whitelist what you want. A big problem with this is that there's no easy way to include request data in the responses (such as session information) and for more complicated representations you end up building complicated hashes and conditionals. Moreover, it means that for any complex payload you end up defining representation logic inside your model layer, which goes against basic separation of concerns.

ActiveModelSerializers

ActiveModelSerializers introduce a separate layer where you can have serializer classes that declare how your domain objects should be serialized.

1 class UserSerializer < ActiveModel::Serializer
2   def full_name
3     [object.name, object.lastname].join(' ')
4   end
5 
6   attributes :id, :age, :full_name
7   has_many :friends
8 end

Certainly much better, but my problem with this is that it's still too tied to the model layer. Those attributes and has_many methods in the DSL are all about reflecting on the attributes and associations in your models. You certainly can customize serializers to output different attribute names (such as the custom full_name method in the example), but the emphasis is on exposing the data as modeled in the model layer. I find this problematic because in most Rails applications this means exposing your database tables to the world.

Another problem I see with this is that, by making the model layer the focus of your API design, it doesn't make it immediately clear what your API resources are going to look like. By looking at the serializer class above I have no idea how the rendered JSON is actually going to be structured. For example, it's somewhat unexpected that the serializer above will generate the following JSON.

1 {
2   "users": [
3     {"id": 1, "age": 35, "full_name": "John Doe"}
4   ]
5 }

So it inserts a root users node inferred from the serializer's class name (*). Again, you can configure all these things but the library's path of least resistance is all about magically turning your model layer into JSON rather than explicitly designing your API's data representations.

  • ActiveModelSerializer's latest incarnation will adopt the JSONAPi media-type, which dictates a root node named after the resource, in plural.

JBuilder

JBuilder installs by default on new Rails applications and is a very straight-forward DSL for writing JSON, in Ruby.

1 json.id @user.id
2 json.full_name @user.full_name
3 json.friends, @user.friends do |friend|
4   json.id friend.id
5   json.full_name friend.full_name
6 end

There's something refreshingly simple about this. You're basically writing JSON payloads, minus all the commas, quotes and string interpolation. What I do like about this is that it encourages you to think about what fields go in your JSON, and in what structure, much like what you do when building semantic HTML pages. In other words it helps enforce a layer of separation between your model layer and the outside world.

Having used jBuilder (and similar solutions like Rabl) in several projects, however, I've found that it can end up being too verbose and error-prone, precisely because you're writing raw JSON structures without a predefined schema or media-type.

For example, if you want to have some degree of structure in the way you encode links in your JSON responses, you end up with loads of duplication of things like this:

 1 json.links do
 2   json.self do
 3     json.href user_url(@user)
 4     json.type "application/json"
 5   end
 6   json.website do
 7     json.href @user.website_url
 8     json.type "text/html"
 9   end
10 end

This becomes obvious when trying to adopt an Hypermedia spec such as HAL or Siren, or really for any slightly complicated data structure that needs to encode links or embedded entities.

Roar

I haven't actually used Roar but it looks like a very complete solution that caters for both generating and parsing JSON payloads, aimed at (I think) building multiple apps in a SoA setup. Other than being way more than the simpler JSON generation part that my apps usually need, I do not agree with how it's designed to extend your models, effectively blurring the line between data models and resource representation.

Oat

So, after several of the approaches listed above, I decided my apps were missing a distinct abstraction layer that would

  • separate the model layer from API resource representations.
  • clarify how my API's resources look like, in terms of field names and structure.
  • make it easy to reuse resource definitions or parts of them.
  • allow me to try and switch different media types without too much impact on the resource definitions.
  • be framework-agnostic.
  • be easy to use outside of the request-response cycle, for example when generating resources for WebHooks or in background workers.

I've called this library Oat, and the Readme has plenty of examples and use cases.

The gist of it is that you create custom serializer classes that express the different components of a resource representation, including attributes, links and embedded resources.

 1 class UserSerializer < Oat::Serializer
 2   adapter Oat::Adapters::HAL
 3 
 4   schema do
 5     type ['user']
 6     link :self, href: user_url(item)
 7     link :website, href: item.website_url
 8 
 9     property :id, item.id
10     property :full_name, item.full_name
11 
12     entities :friends, item.friends, UserSerializer
13   end
14 end

This conveys enough information to tell you the general structure of your representations (it has 2 links, id and full_name properties, and a list of friends using the same schema), while abstracting away the details of the actual JSON being generated. Because you have to be explicit in what fields and sub-entities go in the representation, you're encouraged to think about your resources instead of reflecting from your model layer (of course you can still meta-program that in, this is Ruby after all).

The actual JSON generation is handled by adapters. An adapter class proxies the serializer's DSL and builds a hash that can then be generated by your JSON library of choice. For example the HAL adapter turns the serializer above into this JSON

 1 {
 2   "_links": {
 3     "self": {"href": "..."},
 4     "website": {"href": "..."}
 5   },
 6   "id": 1,
 7   "full_name": "John Doe",
 8   "_embedded": {
 9     "friends": [
10       {"id": 2, "full_name": "Joe Bloggs", "_links": {...}},
11       {"id": 3, "full_name": "Jane Doe", "_links": {...}}
12     ]
13   }
14 }

Switching to a Siren adapter will turn the same serializer definition into the Siren variety of JSON:

 1 {
 2   "class": ["user"],
 3   "links": [
 4     {"rel":["self"], "href":"..."},
 5     {"rel":["website"], "href":"..."}
 6   ],
 7   "properties": {
 8     "id": 1,
 9     "full_name": "John Doe"
10   },
11   "entities": [
12      {
13        "class": ["user"],
14        "links": [...]
15        "properties": {"id":2, "full_name": "Joe Bloggs"}
16      },
17      {
18        "class": ["user"],
19        "links": [...]
20        "properties": {"id":3, "full_name": "Jane Doe"}
21      }
22   ]
23 }

There is very little magic going on in here, and new adapters are fairly trivial to write. The gem ships with basic adapters for HAL and Siren, but you can write your own for your own cases.

Ultimately, Oat's serializer/adapter combo produces simple Ruby hashes (exposed as Serializer#to_hash), which is enough to use in your Rails controllers or your Ruby stack of choice (see the Readme for details and caveats). It's a rather small library I wrote to scratch my own itch, but I think it may introduce the right level of abstraction between your app's domain objects and your API resources.