Using Play! Iteratees and Enumerators with Redis Pub/Sub

A few weeks ago I started thinking about mixing together the Play! Framework's facilities for handling reactive data streams with Redis' pub/sub messaging system.  This is part of a larger project I'm working on which will allow operating on Scala types which persist to a Redis store.

This post covers my first pass at using Redis pub/sub as the source and destination of iteratees and enumerators.  The solution presented here is a bit hack-ish for me and I'm working on refining it but I wanted to get my initial pass at the problem out in the open.

Let's start with a quick review of Redis's pub/sub features.  Any client can publish to a channel and any client can subscribe.  Channels aren't anything special, just a key in the Redis store.  A client subscribed to that key will block until it unsubscribes.

Before I go on to describe iteratees and enumerators, let's take a look at the architecture we'll end up with at the end of this post.  Remember that we're building a distributed chat room.

Multiple instances of Play!, one Redis store.  First, the user in the top left sends a message ("Hi!") to the chat room (green).  Second, the Play! instance that user is on receives the message over a WebSockets connection in an enumerator.  Two iteratees have been applied on that enumerator, one to publish the message to the Redis store and another to publish it to the other users subscribed to this specific instance (blue).

Finally, the Redis store pushes the message out to all subscribed clients (red).  These clients will receive the message and push it into a channel (a type of push enumerator) which will then by sent onwards to whichever chat users might be subscribed.

The important thing here is that these operations are all asynchronous and non-blocking.  Moreover, the use of Redis pub/sub allows this architecture to scale horizontally—just add more Play! instances.  

Now, it's not perfect.  Given the current state of Redis clients for Scala, this only works if each instance is subscribed to a single channel, using it for some sort of multi-room chat would not scale as easily.  I'm hopeful that an asynchronous Redis client will appear in the future.

The source code for this little demo is available at https://github.com/ryantanner/websocketchat-redis.  Only a few modifications were made to Play!'s existing WebSocket chat demo, a testament to the flexibility of the enumerator/iteratee paradigm.

Batch inserts with Anorm (Play! Framework 2.0)

In trying to answer a question over at Stack Overflow I delved into Anorm's API for batch inserts.  It's usage is not immediately obvious from its type signature and I'd like to clarify it here.

Let's go straight to an example.  You need to store lists of numbers in a SQL table called UserActions with a single column named actions.  (I know this table is wholly unrealistic, just go with it).  You get these numbers in chunks as List[Long] and you want to insert them into this table.

Start with a basic insert query.

That's great if you've got a single variable to insert but how do you insert the entire list?  You could just foreach and create different insert statements for each element of your list but there's a better way: the BatchSql query.

Create the same insert query as above and then fold over your list of elements to insert.  The result is a BatchSql object.  Execute it and you're good to go!

Rewriting play20-auth with Slick

I've spent the last few months building a side project on top of the Play Framework 2.0 platform and while I've been impressed with its database library, anorm, I missed the power and utility of LINQ found on the .NET platform.

Enter Slick.

Slick is the new database query and access library from Typesafe and appears to be one of the few non-.NET DAO libraries to come to par with LINQ.  Anorm is good but this is so, so much better.  I started rewriting my project to use Slick instead of Anorm but I ran into a roadblock with my authentication layer provided by play20-auth.

play20-auth is written using anorm.  I easily could have left this in place given that both anorm and Slick can map to the same case class.  However, this bothered the purist in me so I decided to fork and port play20-auth to Slick.

play20-auth and anorm:

Overall it's not a huge change but I faced a few challenges in preserving some of the features offered by anorm.  Note that play20-auth uses a series of case objects extending the same sealed trait to indicate a user's permissions:

I wanted to maintain this feature. For those not familiar with sealed traits, the magic happens when pattern matching case objects (or case classes, but in this case case objects). Making a trait sealed causes the compiler to warn about any non-exhaustive match.  Also, the sealed trait can only be extended by objects or classes in the same file, allowing the compiler to make such reasoning.

The documentation on Slick is still lacking but buried in the test cases I found this gem...


...which pointed me in the right direction.

Looking at play20-auth's mapping from clob to string, I rewrote it using Slick as such:
This allowed me to use a Permission object within my Account case class, maintaining type safety.

I hope this helps anyone else trying to get started with Slick and the Play Framework.  Feel free to hit me up with any questions in the comments section.

You can view the complete port in my forked Github repo.

Squeryl and the REPL

Heads up to those playing with Squeryl. If you're getting strange runtime errors executing queries and inserts in the REPL like "java.lang.RuntimeException: next called with no rows available" (on an insert...) try sticking it in a unit test or at least a main(). This *could* also be a conflict due to running the REPL via SBT's console command. Running it via SBT instead of using the console runs fine on the same code. That said, I'm also willing to chalk this up to my own dumb mistake.

What I'm working with today: Squeryl

I've spent the last few days getting my head around Squeryl, an ORM for Scala.  Prior to this the only ORM I've had any exposure to is LINQ on the MS SQL platform.  It's good, quite good from my limited experience, and it does let you put together simple queries rather quickly (plus, aggregate() is a close-enough replacement for reduceLeft in a lot of simple CRUD-type situations).

Most of my work with Squeryl so far has been in defining my schema, understandably not something I'd want to rush.  My result set consists of Documents, comprised of Sentences, around which there are Properties which have Qualities and which are connected to other Properties via Connections throughout the entire set of Sentences and Documents.

It's a lot of data.

Squeryl's documentation isn't bad per se but it leaves a fair bit to be desired.  I expect that this is mostly due to my inexperience with databases and less so the documentation itself.  I've found that the tutorial posted by Srirangan is the best way to get started, preferably *after* reading through *all* of Squeryl's own documentation and stepping through all its sample code.

Of course, just getting the schema defined is only part of it.  Now it's time to learn about the actual *usage* of Squeryl.  I have high hopes.

Adventures in sbt and possibly a bug

I began working with sbt today.  Long overdue for someone working in Scala but I hadn’t strictly needed it until today when I began working with the Dispatch library.

sbt is brilliant but the real problem came about when I spent twenty minutes diagnosing what ended up being a whitespace issue:

  20     val lat = locResp \\ "location" \\ "lat" text 

  21     val lng = locResp \\ "location" \\ "lng" text 

  22     return new Location(lat,lng) 

is not the same as:

  20     val lat = locResp \\ "location" \\ "lat" text 

  21     val lng = locResp \\ "location" \\ "lng" text 

  22     

  23     return new Location(lat,lng) 

Only the latter compiles.

Scala’s inline XML handling has always been a bit picky with whitespace and typically I just wrap each selector in braces but this was a new one.  While I’m still a novice, I’d consider this a bug in Scala (somewhere).  I haven’t yet reproduced this outside the sbt console or even bothered looking into much but if anyone runs into newline issues with Scala and XML hopefully this will help.

Why Scala by Fire?

I first started learning Scala sometime in early 2011 after seeing the excitement of one of my professors over this new language.  He was so thrilled by it he began shifting the department’s introductory curriculum from C and Java over to Scala.  Obviously I needed to know what was going on and I was admittedly late to the party.

At the same time I was beginning preliminary work on my thesis, just playing with a few ideas and seeing what else had been done in the field.  As I delved further into each I realized how well they paired together.  A object-oriented functional language was exactly what I needed to make my thesis proposal possible.  Now, almost ten months in, I know I was right.

Learning a new language while also learning a field of CS you have no experience in is perhaps not the best approach but I can’t regret the decision.  This is where the name of this blog comes from, learning Scala while also learning a vastly complex field (natural language processing) and building the largest project I’ve yet to work on, an algorithm which uses the output of a lexical parser on Wikipedia to produce a temporal-spatial knowledge graph of as much of the corpus as possible.  I’ve had some pretty hot moments and wondered if I might get burned as a result, but so far it’s been a great experience.