Chicago Boss's Big Secret

By Evan Miller

DRAFT: May 20, 2010

"What's your secret, Boss?"
"You mean besides not telling secrets?"

I.1. Once Upon A Time In A Dry Goods Store

Before there were modern grocery stores and minivans, you had to go to a dry goods store like this and tell the man behind the counter what you wanted:

Have you ever gone to the deli part of the supermarket and had to wait for 10 minutes just to buy a few slices of cheese? Imagine just that, but the people in front of you aren't limited to buying cheese and specialty meats; when it's their turn, they also ask the man for two pounds of flour... eight cups of cereal... four sticks of butter... ten horseshoe nails... three tin cans of sardines... two bottles of laudanum... a replacement wagon wheel... two scrimshaw cane handles... and a pack of Wrigley's chewing gum. The store could handle maybe 1 customer every 15 minutes.

Now suppose you own the dry goods store and there's a big boom in the business cycle. Customers start lining up out the door, but damned if you can't serve them more quickly. What's a small business owner to do?

Your first idea is to hire another clerk behind the counter to serve two customers simultaneously. Well, the problem that immediately comes up is the clerks start reaching for the same shelves at the same time. They're getting in each other's way and on each other's nerves. When two customers ask for the last bottle of Dr. Milligan's Cure-All, a fight breaks out. The new system is an improvement, yes, but the line is still out the door, and adding any more clerks would just create chaos.

Then a brilliant idea dawns on you. You'll hire another clerk, but he'll be the only person allowed to fetch the dry goods! This way no one will get in each other's way. The clerks at the cash registers will simply exchange pleasantries with the customers, take orders, relay the orders to the Gopher Boy, tally up the totals, take the customers' money, and tell them all to have a nice day. The Gopher Boy will wait for the cashier to ask for something, fetch it, put it in the customer's bag, and wait for the next order. The system works because while one cashier is writing down an order or counting out change for a customer, Gopher Boy is usually busy fetching something for the other cashier. Sometimes both cashiers need something from Gopher Boy at the same time, and they have to wait a little longer to get it, but at least the cashiers aren't stepping on each other's toes.

This plan works so well that you are able to hire a third cashier, and a fourth. Now Gopher Boy is working his tail off, but here's the the strange thing: the cashiers never seem to be very busy; they spend a lot of time waiting for Gopher Boy to return with the customer's order, and in that time all they can do is stand there and smile at the customer.

You're not sure what to do about this problem, but shrug it off, and try to figure out how to serve more customers with the current system. Maybe you'll hire a smarter Gopher Boy who can fetch more than one item at time. Or maybe behind the counter you'll keep two mini-stores with the exact same items, but a separate Gopher Boy will be in charge of each one, and cashiers will be assigned to different Gopher Boys. Or maybe you'll partition the goods by name, and hire another Gopher Boy to be in charge of items starting with letters A–M while the current Gopher Boy mans items starting with letters N–Z. In any event, you'll be able to hire a couple more cashiers and serve that many more customers.

One evening, after the clerks have left for the day, you're counting your money and a young man walks in and asks for a job.

“Sorry,” you say. “No room here for more clerks.”

“So let 'em go,” he says.

“What?”

“Fire them. Hire me. I can do the work of four clerks. All I ask is double the going wage.”

“Take a hike, kid. Things work fine around here. Now if you'll excuse me, I have some money to count.”

“But I was thinking...”

“You were thinking. Beat it.”

He turns to leave and pauses at the door.

“Don't say...” he says.

“Don't say what?”

“Don't say I didn't offer.”

You go back to counting your money and don't think a whole lot of the incident. Meanwhile, business continues to boom. You plan ahead for planting season, the time of the year when sales are highest. You stock up on everything your customers are going to need — hoe handles, oxbows, you name it — and tell the clerks to plan to stay late because you're expecting a lot of business.

The first week of planting season arrives, but you don't quite get the number of customers you expect. In fact, sales are down. They're down the next week, too. And the next. One evening, you recognize a loyal customer outside carrying a brightly colored bag filled with all manner of dry goods. She's chatting enthusiastically with a friend. You step out and ask where the brightly colored bag came from.

“Well...” your customer says.

“You haven't heard??” her friend blurts out. “There's a new store across town and they've got butter and nails and flour and ham, and, and, the boy who works there... he's a genius!”

The next day you make a plan to visit the new store during lunch, and you arrive to a scene like this:

One clerk, one Gopher Boy, dozens of customers. You watch them closely, and the first thing you notice is that the clerk never stands there and smiles. He keeps track of ten customers' orders at once in his head, and while the Gopher Boy is getting something from the back, the clerk uses the time to count out change for one customer or take an order for another. He is busy, busy, busy, and miraculously, he seems to take care of more customers than all four of your clerks combined.

Boy, you think. If only my clerks could work like that.

The second thing you notice is that the clerk looks familiar somehow.

Didn't he ask me for a job?

I.2. The Boy-Genius Input/Output Strategy

If you saw through the thin veil of my dry-goods conceit above, you know that a dry goods store is a lot like a website. Cashiers are a lot like web application processes that talk to the customer and tell the Gopher Boy (database) what to retrieve.

The problem with almost every existing web framework is that the cashiers spend a lot of their time waiting to hear back from Gopher Boy. In the profession we call that synchronous network I/O. The process sends a network request and then sits and waits for a reply rather than work on other requests. This wastes server resources because:

Almost all existing web frameworks follow the synchronous network I/O model.

In constrast, asynchronous or event-driven applications act like Boy Geniuses. There is one process that keeps track of dozen, hundreds, or thousands of pending requests at once, and whenever the process is waiting to hear back from the database, it immediately starts doing something else to stay busy. Event-driven applications can almost always do more work in less time than their synchronous counterparts (how much more depends on what percent of the time the application spends waiting on Gopher Boy). If you're making dozens of database requests, the time can add up.

In practice, event-driven applications have been very successful at handling tons of traffic. Memcached is an event-driven cache server used by the web's most popular sites, including Facebook and YouTube. Nginx, an event-driven web server written in C by a lone Muscovite in 2004, today serves about 6% of all web domains.

But historically, event-driven MVC applications have been very difficult to write. The problem lies in how these programs are structured: an event-driven application has to tell the operating system, here's a request, tell me when you have a response, meanwhile I'm going to do some other things. In particular, it has to tell the operating system, execute this function as soon as Gopher Boy gets back. That's called a callback function.

Thus, in event-driven applications that have been written in C, Ruby, Perl, Python, PHP, Java, or almost any other language, every single network request requires its own callback function. Therefore, if you want to write an event-driven web application that makes a dozen database requests per page, you have to break the code up into at least a dozen different callback functions. Sounds like a nightmare, right?

Because of the difficulty in writing these applications, so far the technique has only been used in critical infrastructure like Nginx and Memcached. For everyday web programming, it has been a lot easier to use instead a synchronous I/O strategy which tells the operating system, here's a request, tell me when you have a response, meanwhile I'm just going to stand here and smile.

I.3. Enter Erlang

Erlang is different. Event-driven applications can be written in any language, but Erlang is the only language where event-driven applications do not need separate callback function for every network request. Erlang breaks up your program into those nice-sized chunks for you and keeps track of all the active network requests without you needing to lift a finger. Erlang is a Boy Genius, a pure event-driven language. In fact, it's impossible not to write an event-driven application in Erlang.

Chicago Boss is built with Erlang. That is Boss's big secret. You can build extremely lightweight, event-driven database applications without having to break up your code with callbacks for every database request. Ruby, Python, PHP, Perl, C, and Java simply can't compete, no matter what the framework.

Historically, writing web applications in Erlang has been cumbersome. Erlang was designed in the late 1980s for use by banks and telecoms, who weren't exactly attempting to put the fun into functional programming. But Chicago Boss has a trick up one sleeve, and two tricks up the other. We'll see in "An Evening With Chicago Boss" how building web applications with Boss can be a blast.