Codes from the Underground

Objective-C Designing With Blocks

Sun, 15 Jan 2012 12:25:59 -0500

I recently started doing some Objective-C development at work. Coming from a Ruby/Python/JavaScript background, I was stoked to see that Objective-C had support for blocks.

Here’s a great tutorial on the subject.

Closures are a wonderful way to handle the complexities surrounding asynchrounous I/O. Ask any Node.js nerd, they’ll attest to this fact.

I’ve been leaning on blocks heavily while building libraries for handling I/O in Objective-C. The approach I use is as follows:

I create a delegate class that interacts with the foundation frameworks.
The methods of the API exposed by the delegate accept blocks, and invoke the blocks when a series of asynchrounous actions has reached a terminal point.
The class interacting with the library does not need to worry about the intermediate actions being taken by the delegate, it just trusts that its block will be invoked with the appropriate response.

As an example of this methodology in action, here’s an asynchrounous IMAP client I’ve been working on:

https://github.com/bcoe/IMAPClient

The IMAP class itself is an NSStreamDelegate. When a user invokes an IMAP command, a block is provided. The delegate methods handle the specifics surrounding the asynchrounous I/O. Once an operation eventually completes, the block is invoked. Here’s the code that handles running a command:

An individual consuming the library need not worry about these specifics. They simply provide the block and trust that it will eventually be invoked with the appropriate response. Ultimately, a series of IMAP commands takes on a form resembling this:

cool, eh?

In the same vein, here’s what an API call looks like to an OAuth HTTP framework I’ve been working on:

I find this is a great way to decompose asynchrounous problems. It’s been working great for us at Attachments.me, as we build out our iOS stack.

— Ben (@benjamincoe)

SMTP Server Cage Match

Sun, 04 Dec 2011 12:22:01 -0500

Last week I released smtproutes: A simple, Sinatra inspired, SMTP routing server.

Read What if SMTP and Sinatra Had a Baby?

Out of curiosity, I wanted to benchmark smtproutes vs. some other libraries that one might use for building SMTP-backed-web-services.

In this post I compare the performance of:

smtproutes (https://github.com/bcoe/smtproutes)
Lamson (http://lamsonproject.org/)
Pymilter (http://bmsi.com/python/milter.html)

“Three approaches to SMTP scripting enter, one approach to scripting SMTP leaves!”

The Test

I was interested in seeing the number of messages that each server could process per second, under various loads.

I created a mailing script which repeatedly mailed an 8kb message for the first benchmark and an 88kb message for the second, testing how each server dealt with small and relatively large email messages respectively.
I created a mail processing script for each of the servers. This received the messages and kept a running total of the number processed per second.
I tested each mail server with 1, 4, 8, and 16 concurrent mailing scripts, to simulate various loads on the server.
All the benchmarks were run in an Ubuntu Maverick VMware Image, on my Intel Core 2 Duo 2.26Ghz Macbook Pro.

The Results

Conclusions

Pymilter is painfully slow. Why? there’s a lot of OS level overhead that comes with receiving email via sendmail. I’m sure a Unix guru could configure sendmail in such a way that Pymilter performs much more admirably (I’m not that Guru).

Lamson is fast, it outperforms smtproutes when there are a small number of concurrent connections (a single SMTP connection sending a lot of messages to the server).

smtproutes, which uses an underlying process pool, excels when there are many concurrent connections to the server. With 16 concurrent mail scripts running, it more than doubles Lamson’s throughput.

So What?

The performance of Lamson and smtproutes helps validate the argument I’ve been putting forward in my last few blog posts (Writing a Secure SMTP Server in Python, What if SMTP and Sinatra Had a Baby?):

Running a lightweight SMTP server, independent of all the standard underpinnings of a captital ‘E’ email server, makes a ton of sense for a wide variety of applications that can be built using Email as a Service (EaaS, yep I’m going to keep using this horrible acronym).

— Ben (@benjamincoe)

What if SMTP and Sinatra Had a Baby?

Sat, 26 Nov 2011 20:33:00 -0500

Recently I’ve published several posts on the topic of composing software services on top of email. Why this recent obsession?

People understand the paradigm of email. For some, my grandfather being a prime example, email is their main motivation for having a computer.
Email is everywhere. It’s cross-platform, cross-device, cross-demographic. It’s estimated that there are nearly two billion email users.
We’ve only scratched the surface of the types of applications that can be built, using Email as a Service (EaaS, yep worst acronym ever).

It’s this last point that gets me the most excited — and I’m literally so excited I’m yelling as I type right now.

The Problem

Imagine if every time I wanted to rough out a website I had to spin up a whole J2EE stack, gross. In a similar vein, to compose a service on top of email, why should I have to run painful middleware? Quoting the ubuntu community website:

“Setting up an email server is a difficult process involving a number of different programs” yikes!

If only there were a better way!

The Solution (smtproutes)

smtproutes is what you’d get if Sinatra and SMTP had a baby. It’s not an email server with a capital S. smtproutes is a lightweight framework for rapidly prototyping web-services on top of SMTP.

Suppose I wanted to create an email service which lets you subscribe and unsubscribe to a mailing list. Here’s how we’d build this service using the smtproutes framework:

easy_install smtproutes

If the route regular expression is matched by the To: field of an inbound email message, the corresponding method will be executed.

In this example:

mailing subscribe-username@example.com will add the user, identified by username in the regular expression, to the subscribers collection.
mailing unsubscribe@example.com will remove the user.

It’s my hope that smtproutes can be a valuable tool for helping developers rapidly compose applications on top of SMTP. You can checkout some of its more advanced features, and contribute here:

https://github.com/bcoe/smtproutes

— Ben (@benjamincoe)

Writing a Secure SMTP Server in Python

Sun, 13 Nov 2011 15:57:00 -0500

Email as a Platform

Some startups are beginning to see existing email protocols as a rich platform to develop applications on top of. This is cool, email is an incredible tool for communication and I don’t think it has ever been fully utilized. Hell, even my mom knows her email address (and she thinks I fix printers for a living).

Here’s a list of some companies doing some awesome stuff with email protocols:

posterous.com
fiesta.cc
mailgun.net
attachments.me *cough*

Building an Application on SMTP

There are a million options for quickly getting an HTTP-based web application off the ground. The options available for building applications on top of SMTP/IMAP/POP aren’t quite as mature (or, as the case might be, simply don’t exist).

Python, being the swiss army knife of programming languages, has an SMTP server in its standard libraries:

http://docs.python.org/library/smtpd.html

I’ve been using this as a starting point for rapidly developing applications on top of SMTP. Having said that, it was missing some features I wanted:

There was no SSL support.
The library dealt somewhat poorly with multiple concurrent requests.
The library does not support AUTH.

Secure SMTPD

I have started a project that subclasses smtpd.py and adds support for SSL and AUTH. I have also changed the way in which new connections are handled (they are now forked), making the main process block less.

Other than that, Secure-SMTPD works pretty much exactly the same way as the standard library:

easy_install secure-smtpd

SSL SMTP Server

SSL SMTP Client

You can find and contribute to the project here:

https://github.com/bcoe/secure-smtpd

My goal is to create something that can act as a good foundation for rapidly prototyping rich SMTP-based applications that are: secure, fast, and simple.

— Ben (@benjamincoe)

Building Your First Node.js Library

Sat, 17 Sep 2011 15:44:00 -0400

I’ve been playing with Node.js since its early days. One of my favourite hacking experiences was the first Node Knockout Competition (what a fun language!)

What has really impressed me, is how quickly the community has matured. Taking into account the conventions that have grown up around Node.js development, I thought I’d write a post targeted at helping people get their first library out the door.

Structuring your Library

The root directory of your library should have a structure resembling this:

I’ll go over each of the important parts:

The lib/ directory

I like my code to be very explicit, for this reason I tend to have most of my code in logically named files under lib, e.g., my Person library might consist of a head.js file, a torso.js file, and an arm.js file.

index.js represents the main entry point of the library, and has responsibilities similar to __init__.py in Python. It pulls the namespace together. An example helps I think:

The exports object

Speaking of namespaces, there’s an exports object available in each of your library files. Anything placed on the exports array is exposed to the outside world. When you require(‘class.js’) you gain access to class.js’ exports object.

When we require() the above file, we can see that the Foo and Bar classes are exposed:

I try to keep what I punch onto the exports object to a minimum. Generally one class per library file.

Testing

Do a bit of a literature review, and find out which testing framework you’d like to use.

Recently I’ve been using a really light-weight approach:

I use Node.js’ built in assert module for performing assertions: http://nodejs.org/docs/v0.3.1/api/assert.html
I expose all the tests in an object with semantically named keys:

I expose a finished() callback which is used to move the tests forward, and can be called from deep within an asynchrounous stack of logic.
Finally, you simply run my tests by typing: node test/index.js

TDD is Your Friend

People complain about JavaScript being a pain in the ass to test, in principal I’ll agree.

It’s really annoying to take an existing JavaScript library and get it under test. It’s so easy, a few closures deep, to introduce hard to test dependencies and sideeffects.

Having said this, I find that if you use a TDD approach to building your libraries, from the ground up, you tease apart these problems from the start.

Creating Your API

A library isn’t a good library, without a clean API. There are a few conventions used in the Node.js that are worth sticking to:

asynchrounous calls tend to accept a callback of the form function(err, result). This allows errors occurring deep in an asynchronous set of operations, to bubble up to the application implementing your library.
optional function arguments make for a much cleaner API. You very often see this form: function(doc, options [optional], callback [optional)]. Where the callback might occur in place of any of the arguments. This is nice for someone using your library, but can be a pain to implement over and over again. I’ve built a library for dealing with this:

Contribute to Sexy-Args Here: https://github.com/bcoe/node-sexy-args

Finally, write a clean descriptive interface, as you would in any language. Naming is everything! name methods well, name classes well, name your library well.

Publishing Your Library

You’ve made a killer Node.js library, and you’re ready to share it with the outside world. Here’s what you’re going to want to do:

Create clean descriptive documentation, and examples as needed.
Throw everything up on Github. This is going to be your best bet for getting other people to contribute.
Update your package.json. This describes your library to a package manager. Most of the keys are pretty self explanatory. Most importantly make sure that lib points to your lib folder, and that main points to your lib/index file.

Now install NPM and publish your package:

Follow the instructions here, to install NPM: https://github.com/isaacs/npm
Now, from your packages root directory, do something resembling this:

Finally, hop on a computer that has NPM installed. install your newly published library npm install library. Hop in a node terminal, and make sure everything works as advertised (push point releases until this is the case).

Promote Your Library

Now that you’ve built something awesome for the Node.js community, don’t be shy to promote the hell out of it. People can’t benefit from what they don’t know about.

To help in this process, I built this recently:

NPM-Tweets: https://github.com/bcoe/npm-tweets (@nodenpm)

I hope some people found this useful,

— Benjamin Coe (@benjamincoe)

Attachments.Me: A Sexy DSL for JavaScript Arguments

Mon, 12 Sep 2011 18:19:18 -0400

Attachments.Me: A Sexy DSL for JavaScript Arguments:

attachmentsme:

I was working on one of my Node.js libraries and noticed I was doing something silly:

Karait (https://github.com/bcoe/karait)

There’s a lot of ritual around dealing with optional arguments and default parameters!

I did a little literature review, and found this problem was pretty…

Attachments.Me: Designing for Asynchronous Behaviour in ElasticSearch

Mon, 22 Aug 2011 22:30:11 -0400

Attachments.Me: Designing for Asynchronous Behaviour in ElasticSearch:

attachmentsme:

Let me preface this article by saying that I am relatively new to ElasticSearch. If anything I say is grossly inaccurate, please feel free to yell at me in the comments.

ElasticSearch is a great abstraction on top of Lucene:

It handles index replication and sharding.

Indexes documents very…

Ben Coe, Now in HD: Episode 1

Sun, 10 Jul 2011 21:19:07 -0400

Ben Coe, Now in HD: Episode 1

Why I Hate Ruby (Or, at least, some common practices of its developers*)

Sun, 05 Jun 2011 21:40:00 -0400

Let me preface this post by saying, I don’t hate Ruby the language as a whole. It has an elegant syntax and I enjoy working with it (some of my best friends are Ruby developers!). What I do hate are some of the bad habits adopted by its development community.

I will begin with a story. This past week, I have been setting up elasticsearch as a full text search solution for attachments.me. Here’s the play-by-play.

Java

I got the Java development environment for elasticsearch up and running. This way, if need be, I can submit patches to the elasticsearch community. It’s using a tool called Gradle for dependency management and, for once, getting all the Java dependencies installed didn’t have me tearing my hair out.

Python

I installed pyes, a Python client for elasticsearch. This also went very smoothly.

Ruby

Now it was time to install a Ruby client. The library Tire was the most popular solution, so I went with this. All the unit tests passed, and it seemed like a safe move to add Tire to our Gemfile. The next day the code was deployed and it proceeded to crash our production environment.

Why? Because Tire relied on yajl-ruby which punched the to_json() method on classes with a method incompatible with ActiveSupport (which also punched the same method onto classes). It gets better, in irb I happened to be including the JSON gem, this library expected a to_json() serialization method incompatible with both yajl/json and ActiveSupport. The situation is so bat-shit-crazy that there’s even a multi_json library, to provide cross-compatibility between these gems (note that this gem does not fix the serialization incompatibility).

Is the problem that there are so many JSON libraries? Not really, although this is a bit crazy. The problem is that each of the JSON libraries is mucking with the class methods.

I’m a big JavaScript groupie. Like Ruby, JavaScript is a very malleable language. It might arguably be a bit too malleable. Early in its history, it got a reputation for its ability to turn into an unmaintainable mess. If there’s one rule that saved JavaScript, it’s this “thou shalt not touch the global scope.” Well written JavaScript libraries take this commandment to heart and, as a rule, tend to only introduce one variable into global scope.

Punching methods onto classes is cool, but it pollutes scope and can cause major issues like those I had with to_json(). I think that modifying things high up in the inheritance hierarchy, as a side-effect of a require statement is, in many ways, much worse than leaking things into the global scope in JavaScript — it causes bizarre hard to track down bugs, when two libraries are contending for the same method on a class.

Do I actually hate Ruby? no. Ruby developers just need to learn from languages like JavaScript and adopt conventions which pollute scope less. That way, novice Ruby developers can utilize the language while setting off less land mines.

— Ben (@benjamincoe)

* It has come to my attention that, as I do not actually hate the Ruby language as a whole, this article’s original title “Why I hate Ruby” was confusing for various individuals (many of whom did not realize what was happening in time, and erroneously read a sentence or two of the article). I have amended the title, and have lashed myself in penance.

Attachments.Me: Pytoad: A Hoptoad Notifier for Python

Thu, 19 May 2011 19:07:35 -0400

Attachments.Me: Pytoad: A Hoptoad Notifier for Python :

attachmentsme:

We’ve recently started using Hoptoad for monitoring exceptions in our JavaScript and Rails code. Hoptoad is a web-service for centrally aggregating exceptions. We find it helps us much more proactively hunt down and fix bugs (getting annoying emails is a really good motivator).

Python is…

My Company is Hiring

Sun, 03 Apr 2011 19:13:42 -0400

My Company is Hiring

Attachments.me Searches Your Gmail Attachments by Filename and Content

Wed, 23 Feb 2011 11:19:00 -0500

Attachments.me Searches Your Gmail Attachments by Filename and Content

Attachments.me Puts A Visual, Social Spin On Email Attachment Search

Thu, 17 Feb 2011 22:52:43 -0500

Attachments.me Puts A Visual, Social Spin On Email Attachment Search

Securing Mongodb Behind Nginx

Sat, 08 Jan 2011 19:08:36 -0500

Securing Mongodb Behind Nginx

Endtable, an ORM for CouchDB on Node.JS

Tue, 23 Nov 2010 00:35:00 -0500

Two important facts about me: I hate relational-databases, and I absolutely love Node.JS (I would have its JavaScript babies).

Why do I hate relational-databases? it comes down to the way they’re usually used:

If I am modeling an automobile, I really don’t care that it can be stored in and retrieved from my garage. Furthermore, my garage really shouldn’t care about the attributes of the cars being stored within it.

The ORMs that I have worked with have generally felt too tightly coupled to the persistence mechanism underneath them.

Back when I was developing the MMO HackWars, I was a fairly green developer:

Having grown up developing games in C, I thought of objects as things that stuck around for a while, e.g., a player shooting at alien objects as they darted around the screen.
Coming from a strong OOP background, I had an object-centric view of software development, not a data-centric view.

I made the following design decisions for HackWars based on these facts:

MySQL was used purely as a key/value store. I just wanted a place to serialize my models to.
Saving was abstracted away from the domain model and overseen by a central thread. I think this decision related to my not thinking of an application in terms of a request/response paradigm.

It was a bit of a shift for me when I started doing professional web-development. The design approaches prevalent in the space (and encouraged by frameworks like Rails) did not really jive with my way of thinking about software design. I think this might be why I took to doing a lot of front-end JavaScript work. The type of design you do in browser-side JavaScript is actually shockingly close to what I was used to from game development.

Enter Node.JS. I recently saw a talk by Ryan Dahl, the creator of Node.JS. In a nutshell, he said it works “the way you feel things should”. In a similar vein, I’ve been working on an Object Mapper for CouchDB/Node.js that works the way I feel persisting data should. It’s modeled after the approach I used for my MMO:

Models are just dumped directly to CouchDB.
Relationships, validations, schemas, etc., are all implicit in the domain model, not imposed by the database.
Saving is performed asynchronously and automatically. You’re always interacting with a cached version of an object, with the understanding that it will save occasionally — this is freaking fast, and works well for domains like games.

Without further adieu, I present to you, Endtable:

You can install it either from github or NPM:

git clone https://github.com/bcoe/endtable.git

or,

npm install endtable

First you create an engine for connecting to CouchDB:

You then define the Plain-Old-JavaScript-Objects that you’ll be using: type is a reserved word in endtable that is used when permanent views are being auto-generated.

You can now create some objects:

As changes are made to the endtable objects, they will be automatically persisted.

You can retrieve objects from CouchDB using the following syntax, views will be created on the fly:

There you have it, Endtable is in its primordial stages, but I’m pretty happy with how it’s looking so far. Feedback is greatly appreciated.

Writing Clean Maintainable JavaScript

Mon, 15 Nov 2010 13:39:00 -0500

Writing Clean Maintainable JavaScript:

Some helpful tips for writing clean, maintainable, and easily testable, JavaScript.

Unit Testing in the Cloud with Node

Sun, 29 Aug 2010 03:37:00 -0400

I’ve been running on little sleep this weekend, why? It’s the Node.JS Knockout:

Node.js Knockout is a coding contest inspired by Rails Rumble. The rules are simple: teams of up to 4 have 48-hours to build any node.js-based web project they can imagine.

I have been hacking away on an entry for this competition with help from my friends Ben Vinegar and Jaco Joubert. Our project:

Running on a server-side mock-browser, a secured sandbox, WebHooks, and a lot of elbow-greese, CloudQ runs your client-side JavaScript on the server, so you don’t have to.

In the root of a public Github repo put a folder named ‘test’.
Place your JavaScript QUnit tests in this folder and a file named ‘index.html’ that points to them. For an example, see this repo.
Go to CloudQ and enter the path to the root of your Github project (for the repo above I’d enter ‘http://github.com/bcoe/schedgy’).

CloudQ executes the tests it finds in index.html directly off of your Github account. But wait, there’s more, you can setup a commit-hook on Github. CloudQ will tweet when one of your builds fails, just follow us @CloudQunit.

We’re hoping some JavaScript hackers will help us test our service before the competition ends today, please give me a shout if you’re interested.

Here’s some Important Information:

A repo setup to use CloudQ:

http://github.com/bcoe/schedgy

CloudQ is hosted here:

http://nko-team-discovery-channel.heroku.com/

I’ll write a thorough follow-up post after the competition, but here’s some quick thoughts while I’m in the thick of things:

Node is getting fairly mature, building CloudQ was a pleasure compared to work I was doing on Node a few months ago.
For certain classes of problems, e.g. WebHooks, asynchronous design is very elegant, Node lends itself beautifully to it.
Competing in this competition was a really fun way to hone my skills. If you’ve never tried out a hacking competition, give it a shot.

Porting Smarty to Twig with PEGs

Tue, 24 Aug 2010 17:46:51 -0400

Porting Smarty to Twig with PEGs

Niagara Glen 2010

Sun, 01 Aug 2010 19:37:05 -0400

This long weekend I went on a great two day trip to Niagara Glen to climb with my friend Christian. My hotel was in Grimsby, “The Gateway to Niagara”, so getting to climbing was a manageable drive away, but next time I’ll probably shoot for a hotel closer to the Glen.

I finally bought a crash-pad for this trip, which was awesome (more things to fall on).

My fancy new crash-pad — and Christian’s 1984 Saab.

Me awkwardly trying to look at a camera on an easy climb I did on my first day.

Me mid throw on a fairly hard roof problem that Christian and I were working on Sunday.

The move after that sick throw.

Christian climbed too.

All and all, a great way to spend a long weekend.

JavaScript, Comet, and RabbitMQ Fun

Wed, 21 Jul 2010 22:08:00 -0400

I’ve been kicking around the idea of making a sequel to HackWars, the game I created as a hobby in university. Many of the problems facing the original game were technological. Even though we had lots of coders willing to help out, it was difficult to bring people up to speed, iterating was brutal, and the whole beast was/is pretty tipsy.

My plan: Develop a game with similar play mechanics but with much more scalable and manageable architecture:

A key/value store, such as Redis, for saving player information.
RabbitMQ for message passing and parallelization.
JavaScript/CSS/HTML for the view (I sure as hell won’t miss Applets).

To test my plan, I recently started mucking with using Comet in association with RabbitMQ. The results were pretty cool, I thought I’d share them.

The Client Side

Comet is a buzz word that describes web-applications that operate based on messages pushed from the server, rather than the more common, “are we there yet?”, pull-based architecture.

Orbited provides a pure JavaScript/HTML socket in the browser.

I implemented a push-based architecture using orbited. Orbited creates a connection to a server that you proceed to keep open as long as possible, you allow orbited to re-connect when necessary. Here’s the nitty-gritty:

You create a configuration file that routes the orbited socket to the Comet server. My configuration file routes all connections made to the orbited server to a server I run on port 9000.
```
[access]
* -> localhost:9000
```
Using the orbited JavaScript libraries, you open a socket to port 9000.

I have started building a JavaScript library for handling messages to and from the Comet server. Outbound messages simply use jQuery and $.ajax() to post to a light-weight HTTP controller. Inbound messages arrive via the persistent connection opened by Orbited.

The Server Side

There are three parts to the server-side architecture:

A CherryPy server handles inbound messages and dispatches them to RabbitMQ — potentially this might be a user chatting, clicking somewhere in the GUI, whatever:

The Comet server listens for inbound connections from Orbited on port 9000, and proceeds to spawn the connection off into its own thread:

Finally, the thread listens on RabbitMQ for messages intended for a given user. These are dispatched to the client’s browser, via the open socket.

The code is pretty rough, but I’m pretty happy with how things are working.

I can’t wait to build out this architecture more, hopefully into a fairly addictive game — at which point I can move to the Caribbean, and live on my spoils.