Convention for batching

Last month I talked about webhooks at Glue, a new conference on the “glue” of the web. One of the other speakers was Josh Elman of the Facebook Platform team. I ended up on the flight back with him, so we talked about webhooks. He seemed excited about them as a fan of the callback/hooking pattern in classical programming. In fact, he mentioned how they were experimenting with them already for notifications within the Facebook Platform, but he brought up the issue of batching. Something I hadn’t thought of before, but is important for large-scale implementations that are likely to be posting a lot of events to a target endpoint.

Later, when I read about Google Wave, and how they use webhooks in their API for creating bots, they mentioned that they may batch events. I’ve yet to crack open my Google Wave developer account and play with their implementation, but I’ve since realized a very simple convention for batching: JSON lists as the outer structure.

This works because an event object is ALWAYS going to be a key/pair object. So receiving code just has to check if the payload JSON is an array or an object. If it’s an array, handle each object inside as separate events. If it’s an object, handle as you would.

This doesn’t work for POST variables, since it’s very difficult to do arrays in general, but particularly as the outer data structure. It’s just not made to be able to do that. So this would require you to use at least JSON. It works in XML as well, but because there’s more variety in XML payloads, it’s probably harder to settle on such a simple convention as this.

Let me know what you think. Or if any of you already know how Google Wave batches events in their Robot API.

Pivotal Labs Video

Halp! Call for Contributors

The webhooks ecosystem is growing pretty fast and it’s really hard to keep up! There’s also a growing number of webhooks evangelists giving talks and writing blog posts about webhooks. It would be great if we could work together to help push this movement even further.

I really would like this blog (and eventually website) to be a place to learn about how to implement webhooks, best practices, and standards, as well as a place to see the movement grow. This means seeing services adopt webhooks, how they’re using them, how their users are using them … new services that provide infrastructure in this kind of event-driven programmable world… and software projects that help. Not just my projects like Hookah and Protocol Droid (yet to be announced), but all the plugins people are writing for various frameworks and open source projects.

So we need more writers on this blog. A lot of people send me links to services that adopted webhooks and I wish I could write about them all. I usually mention them on my Twitter, but that’s not enough to really say everything worth saying about some of them.

If you’d like to join us and write on this blog about webhooks, recurring or as a one-off piece, about anything from how you implemented webhooks, to how you did something clever with them, or why people should adopt them … get in touch with me. :)

PostBin now supports passthrough callback

Our beloved webhook debugging tool known as PostBin just got a little bit more useful. You can now include a URL in the query string when registering your PostBin callback that will get invoked when PostBin is posted to. This way you can use PostBin and still have your hook script run. Here’s an example of what it would look like:

picture-19

Notice you can just prepend your existing callback URL with a PostBin URL and a question mark. In this way, PostBin becomes a decorator to your hook scripts.

A lot of work went into this and a lot of plumbing and debugging. PostBin uses Hookah to invoke your passthrough callback, making PostBin the first real user of a Hookah instance. Hookah was rewritten in Twisted and got a lot of tweaks to make this work. Please consider using Hookah to handle asynchronous callback dispatching in your apps!

PostBin has a number of features planned, including private bins, request replaying (so you can capture events, then replay them to your passthrough endpoint for debugging), filtering, and more.

Slides from Pivotal Labs talk

I forgot to mention it on here, but yesterday I gave a talk at Pivotal Labs. It’s a whole new talk that tries to get a little bit more into technical implementation details. The slides are also arguably more stylish. Pivotal recorded the talk, so video will be up soon, too. Until then, here are the slides.

And feel free to use these slides or bits from the deck for your own webhooks talk! You can download the original Keynote presentation here.

Easy hook scripts with Scriptlets

Now that I’ve told you how cool AppJet is for hook scripts, allow me to obsolete it. This morning (as I try to avoid falling asleep as I type) I’m announcing Scriptlets, a tool for throwing online quick web scripts in any of several languages. It’s perfect for writing hook scripts. In fact, it’s what I’ve been dreaming of for web hooks for several years.

When I started thinking about web hooks I figured, as the early adopters, most advanced users would have access to some kind of cheap PHP web hosting for hook scripts. Only later did these cloud code runners appear like App Engine and AppJet. AppJet in particular is the closest to my ideal, as I mentioned before. But it was limited to a particular language and still felt slightly too heavyweight for simple hook scripts (by the name alone it was obviously intended for apps, not scripts).

The ideal I envisioned was a site like a paste bin (I’m keen on Pastie). You know, where you paste some code, pick a language, and you’d get a nicely syntax highlighted, numbered row display of the code with a simple, disposable URL you could use to show the kids in IRC your code that wasn’t working. I wanted that, but that also gave you a URL to run the code. This way I could use it for one-off web scripts that accepted request parameters and everything. No more random one-off PHP scripts on my server. And of course, it would be perfect for hook scripts.

Actually, when I started talking with friends about this, none of this cool cloud computing stuff was big yet. App Engine and AppJet weren’t even around. The closest thing was Ning, which used to host PHP code for you for free. One of my friends (actually, the one that coined “scriptlets”) was interning at Google and knew, but couldn’t tell me that App Engine was coming. Once it came out, he and I started a project like Scriptlets for just Python… but we ran into a wall when I misread the Python documentation and thought App Engine wasn’t letting us eval anything more than one line long. It turns out this was completely wrong, but we’re both pretty busy and we moved on to other things.

But when Google announced Java support on App Engine, even though I hate Java, I knew that the JVM meant they now supported a bunch of languages. I revisited our code, solved the stupid problem, and got several other languages working. And tonight I launched it… with support for 3 languages: JavaScript (uh oh, AppJet!), Python (uh oh, App Engine!), and PHP (uh oh.. uhh… commodity PHP hosting?). Ruby is on its way, and potentially others.

All Scriptlets languages provide an environment with access to the web request and response, and most of the features of that language. All the constraints of the App Engine environment are inherited, meaning no long running processes, no abitrary file system access and no socket connections. (I’m so glad I didn’t have to solve the sandboxing problem by myself!) But you do get to make web requests, and this is exposed with a custom function in JavaScript and PHP called fetch(). It’s a pretty limited environment and not even intended for state maintaining scripts (yet), but it should still prove useful.

Anyway, think of Scriptlets as another community service for the web hooks ecosystem. I have a bunch of these in mind that I’ll be building out. Like PostBin, Scriptlets is open source on GitHub and waiting for you to submit patches. Enjoy!

Extending hosted services as fast as lightning

I have a quick story to share that shows how cool it is to have web hooks. I was writing another post and ended up going on a tangent with this, so I figured I’d give it its own post.

At the last SuperHappyDevHouse, I presented Get Achievements, an XBox Live-style achievement platform I built one night. The only thing was, it had no pretty aesthetic to it, so I rushed what you see there now during DevHouse so I could present it at our lightning talks. I was hacking with a designer friend up until somebody came upstairs to tell me I was up soon, but just then I was struck with a great idea: what if I could add achievements to the DevHouse wiki? Get Achievements requires you to make backend changes to your code, and PBwiki is a hosted service. However, PBwiki recently added web hooks, so theoretically I could use them to trigger the achievements. Could I rig it up in the time I had left?

What I needed to do was have an event in PBwiki POST to a URL on Get Achievements that unlocks an achievement. I needed something simple and quick, so I figured I’d make a login achievement. With this, if I login to the wiki, I’d be rewarded with fanfare and a box saying “Login Achievement Unlocked” as you can do with Get Achievements. But how can I make PBwiki POST to Get Achievements when you login? Well, I can’t, it’s a hosted service. But like I said, they had web hooks, and they included logging in as a hookable event.

At this point I was downstairs less than 10 minutes from my 5 minute lightning talk. I let some people go ahead as I rushed through this sitting on the floor with my laptop. I needed a place to write a hook script for PBwiki that would POST to Get Achievements. Very simple. AppJet was the first place that came to mind, and because I didn’t have to set up an application config file or upload anything with an SDK, I was able to add achievements to the SuperHappyDevHouse wiki in less than 10 minutes… thanks to hooks and AppJet.

I got up there and showed them how cool Get Achievements was, and then I dazzled them with a demo of the DevHouse wiki with a login achievement. Sure, it was more a demo of web hooks than Get Achievements at that point, and because not everybody knows what the hell I’m talking about most of the time, I didn’t get the standing ovation I was looking for, but it went well. I had to go around after, asking my friends if they really got how awesome what I just did was.

From idea to a (mostly) working demo for something that would normally involve making backend code changes, I did as a user, in a panic, terribly fast. Just an example of what’s possible in a world with web hooks…

So simple you’ll think it’s stupid

Just because I think web hooks will change the world doesn’t mean I have to take it seriously. I sort of default to a serious tone in my writing. Hopefully this graphic will lighten things up a bit. It was thrown together by a friend after I came up with the tag line he used in it.

I’ve heard several reports of people getting excited about web hooks from my talks or posts here, and then when they find out what’s actually going on they somehow feel cheated. “As if something that simple could do all that!” I don’t know. Perhaps that’s what happened with this guy. Anyway, I’m considering using it as the unofficial web hooks tag line.

Webhooks: So simple youll think its stupid

Request for Feedback

I’m putting together a new version of my presentation on Web Hooks and I’d like to get feedback on my previous versions. I’m referring to the talks I have slides for on Slideshare. If you’ve seen any of them, at what point did it all click for you? What made most sense and what made the least sense? What was funniest? What did you think it could do without? I’m trying to see what I should elaborate on or just drop for my new deck.

Also, what do you think I skimped on? What do you want to hear more about? If you have any other kind of feedback, or even suggestions based on content on this blog, let me know in the comments. Thanks!

MailChimp adopts Web Hooks

MailChimp, an email marketing tool used by the likes of Firefox and Media Temple, recently announced the addition of web hooks to their API. Their primary use-case is to keep your email list in sync with theirs. They push to your callback URL with several types of events regarding list changes, including subscribes, unsubscribes, profile updates, cleaned addresses, and email changes.

Their API documention explains web hooks and how you use them with MailChimp. They also mention PostBin as an excellent utility to help see the event data as they’re triggered in their system.