Chillax and Protocols

Funny how a full-time job can pull you away from everything else…

I recently did a major rewrite of Chillax. It’s funny what they say about Common Lisp, how you never finish learning it. Ever since I started using it, I’ve gotten periodic “aha!” moments where something simply clicks. A memorable example was the moment when macros really made sense. The latest one, which came to me while working on Chillax, was the idea of designing your program around protocols.

Now, protocols are nothing new, and nothing unique to Lisp. CL doesn’t even have ‘explicit’ protocol support, like Clojure or Haskell do (although there’s a library). It does, though, have generic functions. “Write your code around defgeneric, not around defclass” is certainly something that gets regurgitated over and over. PCL even emphasizes the point by introducing generic functions before it even introduces CLOS classes.

Still, it’s not something that seems to stick right away for everyone. It certainly didn’t stick for me for a couple of years. Then it did — and it was like a tidal wave had just washed over me.

Consider this piece of code:

(defclass server ()
  ((host :accessor host :initarg :host :initform "")
   (port :accessor port :initarg :port :initform 5984)
   (username :accessor username :initarg :username :initform nil)
   (password :accessor password :initarg :password :initform nil)
   (securep :accessor securep :initarg :securep :initform nil)))

(defmethod couch-request ((server server) uri)

(defmethod stats ((server server))
  (couch-request server "_stats"))

... etc ...

Seems reasonable enough. in fact, a lot of “Object-oriented” code kinda flows like this. You first think of what kind of object you want to describe, and create a new class that has data fields that sort of describe that ‘object’, then you write methods for that class. The hope here is that you can just inherit from whatever class, and everything will be okay.

Unfortunately, there are a number of problems with this approach:

  1. It ‘covers its bases’ by just defining blanket accessors, initargs, and initforms for slots. This is both unnecessary and verbose.
  2. it creates potentially-unnecessary slots. — What if your subclass wants to store this data elsewhere, or construct it programmatically?
  3. The only way to reuse code is to subclass my-class. There’s perfectly reusable code there that is now stuck in a class-specific method.

We’re constantly being told about how object-orientation can make our code reusable and extensible — but this doesn’t seem very extensible at all!

Enter protocols: You design your program generically — that means, you don’t assume anything about the objects it handles except what it is -necessary- for the objects to be able to do. The rest of the code doesn’t even need to be polymorphic. After figuring out what the fuss with protocols was about, this is what the above Chillax code transformed into:

(defgeneric server-host (server))
(defgeneric server-port (server))
(defgeneric server-username (server))
(defgeneric server-password (server))
(defgeneric server-secure-p (server))
(defgeneric data->json (server data &key)
  (:documentation "Converts DATA to JSON suitable for sending to CouchDB."))
(defgeneric json->data (server json &key)
  (:documentation "Converts JSON to the desired data structure."))

(defun couch-request (server uri &key convert-data-p &allow-other-keys)
  ... code ...)

(defun server-uri (server)
  (format nil "~A://~A:~A/"
             (if (server-secure-p server)
             (server-host server)
             (server-port server)))

(defun couch-request (server uri)
  ... implementation using protocol ...)

(defun stats (server)
  (couch-request server "_stats"))

... etc ...

But that’s not the full implementation, is it? It’s not — but it’s the vast majority of the code needed in order to use the Chillax API. The magic happens when you realize how much implementation is needed to take advantage of this:

(defclass standard-server ()
    :reader server-host
    :initarg :host
    :initform "")
    :reader server-port
    :initarg :port
    :initform 5984)
    :reader server-username
    :initarg :username
    :initform nil)
    :reader server-password
    :initarg :password
    :initform nil)
    :reader server-secure-p
    :initarg :securep
    :initform nil))
   "Default implementation of the server protocol."))

(defmethod data->json ((server standard-server) data &key)
(defmethod json->data ((server standard-server) json &key)

(stats (make-instance 'standard-server)) => { server stats }

That’s really all of it. The beauty here is that you can reuse code without dealing with inheritance, or any sort of class-based hierarchy. Doesn’t seem like a big deal? Consider that CLOS allows you to define methods on built-in classes:

(defmethod server-host ((server hash-table)) (gethash 'host server))
(defmethod server-port ((server hash-table)) (gethash 'port server))
...and so on...

(let ((server (make-hash-table)))
  (setf (gethash 'host server) ""
         (gethash 'port server) 5984)
  (stats server)) => { server stats }

As long as you tell Chillax how to do what it needs to do on your own data types, it can take care of everything else.

I really like what this ended up doing to Chillax’s API. It’s made it very easy to reuse most of the Chillax code while still swapping out JSON libraries, and possibly (but not yet) http clients.

If you want to read more about protocols, I highly recommend you check out Art of the Metaobject Protocol. I have not read it yet myself (it’s been on my desk for a while), but Sonya Keene’s CLOS book is also recommended fairly often.

If you’re interested in reading about other languages’ takes on this concept, go read up on Haskell’s Type Classes, Java’s Interfaces, and Clojure’s Protocols. Common Lisp itself also has various examples of ‘protocol-oriented’ APIs, such as Gray Streams, and the more recent extensible sequences protocol in SBCL.

If you want to read about more interesting, advanced uses of this architectural technique, François-René Rideau’s postings about “Interface-passing Style” are good reads (and so is his fare-utils code).

Of course, I’ve left a small detail out: How do you actually design programs incrementally like this? The fact is that protocols are not always immediately obvious, and they’re not necessarily static as your application evolves. The process of writing extensible code using protocols is often incremental, and half of AMOP is dedicated to describing techniques for growing and developing these.

Do you write code like this? How has it worked out for you?

Creative Commons License
The Chillax and Protocols by sykosomatic, unless otherwise expressly stated, is licensed under a Creative Commons Attribution 3.0 Unported License.

This entry was posted in programming and tagged . Bookmark the permalink. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL.
  1. Brit Butler says:


    Thanks very much for writing about this. I’ve heard a lot of chatter on #lisp about these issues from you, drewc and others and am glad to read something more in depth which explains the basic idea. Glad you’re still enjoying lisp. Cheers.

  2. Alex Paes says:

    Great post Josh! It’s amazing how a simple change in the way one looks at something can make such a huge difference.

    I am a hobbyist lisper and I’ve always struggled with the correct way of working with object orientation in lisp. For some reason approaching objects and methods in lisp the same way I’ve done with other programming languages always felt kind of wrong, somewhat displaced.

    I’ve finally realized something, most other languages are probably dealing with object orientation in a completely wrong way but it’s the only possible approach in those languages and that kind of makes it the right way.

    Thank you for opening my mind a little more and making me a better lisper. Cheers.

  3. Duncan Mak says:

    I think this is a great way to structure code too, but I have to ask:

    How is this different from Smalltalk?

    • sykopomp says:

      Besides the multiple dispatch? I’m not sure it’s very different as long as you keep in mind that the protocol (the messages you expect will be there) is what’s important, not what the objects ‘are’.

    • Echoing sykopomp, I agree that the protocol focus over class structure up front is a big feature of Smalltalk style design. In Slate, my dialect with multi-methods, code definitely looks like this, without generics or prefixing typenames to every method name.

  4. Thomas M. Hermann says:

    Small critique, don’t mix :INITARG with :INITFORM. If you want the user to be able to be initialize the slot with an :INITARG but default to some value, use :DEFAULT-INITARGS. Otherwise, use :INITFORM if you want the slot to have an initial value. Most of the time, this is not an issue, but becomes a problem when you start getting into situations where you are directly using INITIALIZE-INSTANCE and REINITIALIZE-INSTANCE. See Chapter 9 of Keene for a detailed discussion of why these slot options conflict.

  5. Excellent post. This is pretty much how I tried to structure autobench, and it has made my life so much easier.

  6. I have nothing much to add to the post (other than I agree with you!), but I have to say I smile widely at the treatment that your blog engine gives to “:DEFAULT-INITARGS” :-)

  7. I actually just got around to reading Keene’s book last week. One thing she recommends that’s not immediately obvious (at least it wasn’t to me) is that “object” construction should be done with explicitly defined constructor functions instead of make-instance (so instead of (make-instance ‘standard-server) you’d have a make-server (generic) function). Then you have object creation as part of your protocol, and can swap out implementations (for example make the hypothetical “make-server” return those hash-tables instead of an actual CLOS “object”).

  8. Michael J. Forster says:

    As Alan Kay said, it’s really about the messages, not the objects.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>