Category Archives: Perspective

The Great XMPP Dispute and pubsubhubbub

Okay, it’s not really a dispute. That was sarcasm in the title. Just to be clear, since there’s always been a tiny bit of useful friction between the idea of XMPP and web hooks, it’s important to remember they are not mutually exclusive. Anybody hyping a battle between the two is trying to create or is imagining a controversy that isn’t real. Many proponents of web hooks are XMPP proponents as well. In fact, the video some are pushing around to promote this supposed throw down is of two XMPP supporters (including djabberd author Brad Fitzpatrick) demoing a pubsub system based on web hook callbacks. And it takes place at an XMPP meetup, so of course there was going to be some proselytizing.

Nevertheless, Jabber/XMPP is a messaging protocol. Web hooks provide a model for functional extensibility, so they are a platform for many different things. A push-based pubsub messaging system is just one use. Even though I originally wrote about web hooks as a notification mechanism, my mind was nowhere near pubsub. I was more focused on the idea of web service integration and orchestration. With the commoditization of CGI-enabled web hosting, I was thinking about how the popularity of web programming combined with the easy invocation of HTTP requests could be used to make a more useful and functionally extensible web. A more programmable web.

With that said, there is nothing wrong with discussing what you can and can’t do with web hooks and XMPP, but there is not some angry fight between camps. Done. Over. Moving on.

As it turns out, the project demoed in that video, which is called pubsubhubbub, is something I had previously stumbled upon while browsing projects a friend of mine was involved in. There wasn’t much of a description, and I didn’t dive too deep into the code, so I bookmarked it to come back to it. After that demo happened, it seemed I misheard (from several people) that it was some XMPP pubsub system, which totally confused me because… it’s not. It’s a neat, distributed pubsub implementation built on web hook callbacks created by Brad Fitzpatrick and Brett Slatkin (a Google App Engine developer). Hopefully they’ll put some more documentation up as it develops, but it’s just really neat to see some XMPP folks build an open pubsub system with web hooks. Cheers to them!

O’Reilly Broadcast on Web Hooks

My involvement at NASA inadvertently got web hooks written about on O’Reilly Broadcast yesterday. Kurt Cagle did a nice write-up on his take on web hooks, and it’s possible there will be more coming from Cagle on the topic. Although the post at first seemed pretty framed around syndication and push, the fact that he says things like “server-side mashups” and sees web hooks as a means to “create orchestration of web services” shows he gets the greater significance of this simple mechanism.

I just wanted to cover a few things that were brought up by Cagle and a few others that have written about web hooks recently.

Replacing Syndication

It seems a lot of people see web hooks as an alternative to poll-based feeds and syndication. Although I’ve claimed before that “feeds are not the answer,” it was in context of the vision of pipes for the web. Feeds were not invented for pipelining. They were invented for simple content syndication, and I think they do a pretty good job at that use-case: answering the question, “Hey, what’s new from you?” That said, Cagle seems to be spot on about web hooks and syndication working together.

One of my original arguments for web hooks was that polling sucks. “Hey, what’s new from you?” becomes “Are we there yet? Are we there yet? Are we there yet?” The thing is, web hooks alone don’t let you ask, “What’s new from you?” Nor do they provide a persistent reference to data. Web hook payloads should be ephemeral.

What seems like an obvious solution in this use-case is to provide a feed and a web hook for notifications of updates to the feed. This way you have the feed, which is nice for people that like polling, but also to have a persistent resource on the web for that content stream. And then you have the optional web hook for getting notified of updates, potentially with the updated data so if you don’t want to retrieve the feed, you don’t have to.

There’s a slightly verbose standard spec (as many are) called GetPingd that shows a way you can do this, but I imagine there are simpler approaches. One thing GetPingd points out is that this is all very similar to the ping services for blogs to notify search engines of new content. The missing element of that system is the ability for anybody to subscribe to the notifications. That is part of the essence of web hooks, as Timothy Fitz recently tweeted:

Remember, HTTP callbacks are nothing new. It’s exposing them to the user that makes it a web hook, and that’s where the emergence is.

Anyway, I can understand asking the question, “Will web hooks replace feeds/syndication?” as a thought experiment in trying to understand this new paradigm. But I have to say, after thinking about this for a long time, they won’t. They might replace certain use-cases for feeds, but if feeds were broken enough that web hooks would replace them, it would have happened already.

HTTP vs XMPP

Now this is where we get into some interesting waters. A lot of people bring up good points on both sides. My stance is simply that web hooks are simpler and just as effective for the majority of use-cases, and therefore the obvious winner. There are less pieces, simpler APIs, existing infrastructure, and it’s debatable whether XMPP is inherently any more performant.

You have to consider the use-cases, though. Part of the vision of web hooks is to have a standard HTTP event mechanism for the web. I just don’t see every web service throwing up an XMPP stack along with their HTTP stack. The two can and will work together when necessary, but as Cagle notes, “web hooks in general may be superior for orchestration.” Remember, web hooks are about more than message passing.

Standards

Cagle briefly touched on the standards issue. I’m sure that having a nice standards document would make for great adoption propaganda, and I know quite well the significance of agreed upon conventions in technology. However, I’m not in a hurry to over engineer anything, and I’m not going to assume we’ll know everything about the implications of this mechanism that we can encode them in a document that will either be ignored or adopted by everybody, making it harder to adapt to change. The longer we can put off standardizing, the better.

In the article, Cagle compares it to AJAX, in that the community isn’t very standards oriented. I’m not exactly sure what the AJAX community would benefit from standards. I’m quite happy that AJAX wasn’t limited by a standard to only use XML. There is nothing wrong with options. That’s kind of the whole point of technology: to provide new avenues, options, and choices for empowerment. Tools will always be used however the tool user finds useful, which is not always how the toolmaker intended.

I would much rather provide examples, and rigorously defined patterns of usage and implementation than try to define a standard. When a globally accepted convention is necessary, then we can work one out (with a useful, ideally proven, implementation), but it will probably be about some aspect of web hooks, not the model as whole. I think the biggest aspect ripe for standardization is for machine-friendly announcement of hooks and mechanism for registering callbacks (aka subscribing). But this is not preventing web hooks from being useful, otherwise nobody would be using them already.

Proposal submitted to OSCON

Hopefully web hooks are actually approaching critical mass. Just to be sure, a couple weeks ago I submitted a proposal to OSCON to speak about them. Here’s the abstract I sent.

Web hooks are going to make the web programmable. Three years ago, I stumbled upon this simple architectural pattern in web applications and was struck with a vision of the real programmable web. I’ve since realized today’s programmable web is hardly programmable. It is programmatic. Web API’s give you the power to programmatically use web applications, but they do not let you program them. That is, extend them, customize them, or fully integrate them with each other.

Mashups, the poster child of the programmatic web, are useful, but they also show what happens when you can’t directly integrate web services: you get new ones. Mashups represent the aggregation of services, not the integration of services. However, this is not about some grand proposal for a standardized way for all web services to integrate with each other. This is really about something simpler, and as a result has even richer implications.

Web hooks are about applying the old, simple concept of the callback to web applications. This simple mechanism is changing everything, perhaps even more than did the web API. By allowing users to hook into the logic and events of your systems, you go beyond user-generated content into user-generated functionality. Imagine users extending your web application with new features that can be shared with other users. Imagine discovering your web application suddenly fully integrated with complementary services. If there was a service equivalent of open source, this is pretty close.

After three years of letting just the mechanics of web hooks spread and develop, a number of solid case studies from tiny start-ups to giants like Google and Amazon have popped up, demonstrating the implementation. Sometimes it happened from my influence, sometimes it was just the practical way to solve a problem. Some people latch on to the notification use-case to eliminate polling. Only a few seem to see how the same mechanism can be used to create plugin architectures and platforms. But it also realizes a vision of pipes for the web, allowing you to conceptualize web applications as components that can be strung together to create something more than the sum of the parts.

The point of this talk is to spark public conversation about web hooks. To get us thinking about them. Possibly implementing them. The model has been cooking for a while, and there are lots of examples, demos and ideas to share after thinking and talking about it with other developers for the past three years. More recently, there’s been an influx of activity around the idea, so it’s getting hot. I want to show people just how far this simple, yet novel idea can take us…

User-contributed Functionaliy, Part II: Plugins for the Web

When the web was started, it was about these hyperlinked HTML documents… just pages of information. The web as a whole was collectively just about content. Then, thanks to a bunch of hacks that led to CGI, you had this optional tool as the webmaster to help augment static content with dynamic content. Most immediately, this was used for search, a feature that the web did not come with built-in.

Skip ahead about ten years and this concept of running arbitrary code on web requests with CGI was used and understood enough to finally fix this mostly centralized, one-way flow of information. Finally, anybody could easily publish content on the web through blogs, wikis and comments. This caused such a change in the use of the web that we decided to call it Web 2.0 (and haven’t been able to avoid going meta since–sorry). This two-way flow of information turned it into a collective conversation that the marketing folks today call “social media.”

As a byproduct of using CGI more and more, the web also became generally more dynamic. For example, the commercialization of the web happened once we figured out how to securely use this CGI business to do payment processing. The first killer “application” on the web was the shopping cart. The web slowly started to provide functionality along with content. Today in the industry we talk less about web pages (providing information) and more about web applications (providing functionality).

Our web today is not just a social media platform, but an application platform. And applications do things. Applications represent the augmentation and automation power of computing. I know we’re social beings and communication is our primary means of interaction, but forgive me if I think the empowerment of computational utility is cooler than social media. I’d rather use computers to solve problems holding humanity back from self-actualization than to merely add more channels to the echo chamber. Ahem.

I think social media is important, but it’s stealing the spotlight from the functional potential of the web. Beyond sharing-information-with-people-we-know. We’re still caught up in content, yet it was functionality (e-commerce or “the shopping cart”) that allowed the web to be commercialized. It seems like we under appreciate this aspect of the web.

So what’s the point? This whole time I’ve been trying to set up for an assertion about the future of the web. Here it is:

The web was originally about content (web pages). Then it got functionality (CGI, early web apps). It used this functionality to fully democratize its content (blogs, wikis, etc). Next it will democratize functionality. We’ll have user-contributed functionality just as we had user-contributed content.

What does it mean to have user-contributed functionality? Kind of what it sounds like. Just like you can “contribute” a photo to Flickr, you’d be able to “contribute” a feature (new functionality) to Gmail.

It’s kind of like open source, although a bit more consumer friendly. Like open source, if you want a program to do something different or work with another program, you can make it do that yourself. You can even share a patch so others can get that same functionality. The difference with web applications is that most will never give you their source. And if they do, it would be a nightmare to try and integrate everybody’s patches with the latest deployment. Open source just doesn’t quite translate to the world of web applications.

So if you can’t have access to the source, how can you contribute functionality? Gee, that ad-blocker you have in your browser is pretty slick. Did they need the browser source to make it? No? What’s that? Yes, a plugin system! How do plugin systems work? Right, they provide hooks for external code to run.

I think you see what I’m getting at. Web hooks open web applications up to functional extension and personalization. The plugin metaphor also holds about the ease of use. Not everybody can write a plugin, but anybody can install a plugin. User-contributed functionality will be just as easy to install (if at all), and even easier to write than most plugins. Plus, not only can it be shared between users, but potentially across web applications because the web is a common protocol.

So is user-contributed functionality just plugins for web applications? Yes! I’ve been saying web hooks will enable push, pipes and plugins for a while… but who knows what I mean by that. It’s taken several years just to get people to understand what web hooks are, hopefully it won’t take as long to convey what role they can play. User-contributed functionality seemed like a pretty good way to convey their power to customize and extend.

Anyway, there you have it. It’s already starting anyway. What are Facebook Applications but plugins over HTTP, submitted by users? How long before we see Gmail Labs go from just features by internal teams to features by users?

So, I invite you to imagine a world where what you can do with applications on the web is not limited by those that made them.

User-contributed Functionality, Part I: The CGI-powered Web

I’ve got a story for you. And it ends with a fairly unique take on the future of the web. I’m not about to call it Web 3.0 because I detest this long running meme of “let’s define an era before it happens,” but it is the future. And it’s two parts because it’s late and I’m tired.

So one of the key characteristics of Web 2.0 was this idea of “two-way media” or “conversational media” where users could easily publish their own content and comment on the content of others. This gave rise to the phrase “user-contributed content,” where we let the end-user create the content. It took over ten years for this idea to catch on and become a reality. It’s so great, and we’d never go back.

Some of us are privy to the fact that this idea of a two-way, read-write web was the original vision for the web. Tim Berners-Lee’s first web browser included an editor, but publishing didn’t work because the write verbs in HTTP weren’t implemented in early web servers…

Wait, come to think of it, even to this day web servers don’t implement write methods.

Does Apache itself handle PUT or POST requests as intended? No, Apache delegates the proper handling of these to CGI scripts or a module that essentially runs CGI scripts more efficiently. Most web servers still do not implement the write methods of HTTP. They don’t because before we got around to it, something happened.

In fact, I’m not sure many of us realize this fundamental change (other than the original developers of the web) because we’ve all just accepted the way things are as the way things are. Something big happened in 1993 that changed the entire conceptual model of the web.

webhooks08_nodes1138

Web of linked HTML documents

In the beginning, the web was conceptually about serving up these HTML files that would link to each other. If you recall, the path of the URL was just a subset of the server’s filesystem that was mounted to be served by CERN HTTPd or whichever web server. Apparently the killer feature of the web was rendering HTML (so you had inline hyperlinks, among other things), which you had to get somehow, and so GET was all that mattered from HTTP. If you wanted to put something online, you didn’t need a browser, screw HTTP, you just had to FTP a file to the server. Easy enough, right? It was, back then.

Nobody cared to do anything interesting with PUT or POST. However, they were interested in using this fancy new web protocol to access other protocols. So they started hacking the web servers to run scripts that would query WAIS or some other obscure protocol, usually for search because that was a big problem then.

Eventually Rob McCool drafted a spec for CGP or Common Gateway Protocol that would make a standard way for these little scripts to be run by web servers. It was eventually renamed to CGI (since it’s more of an interface than a protocol), implemented in Apache, and that was that. Now you could run scripts on web requests! Most people used CGI for search, which makes a lot of sense, but the rest of the content was still directly served up HTML files because, well, that’s how the web works, yeah?

Slowly, more and more people started doing clever things with CGI. A few people decided to respect the HTTP spec and utilize the proper verbs for write actions, but this CGI thing was kind of a hack, I don’t think ever seriously intended by Tim Berners-Lee, and so people were abusing the semantics, doing destructive things with GETs, ignoring DELETE and PUT. Well, even today people still do this, just less so… anyway…

Eventually our websites got complicated enough and ambitious enough with CGI that almost all requests would go through CGI instead of serving up static HTML documents. And instead of silly filename paths of files that aren’t really on the server, we can make up useful, descriptive paths with the date and title in them.

Today our web is not about serving up files on the server, but generating files to serve. Our web of pages is now generated by “higher-order” web applications that are no longer simple scripts, but complex software.

higherorderweb

Apps are "higher-order" nodes

If you ask me, those little hacks got a little out of hand, but I guess it’s for the best. Otherwise, we wouldn’t have Gmail or Amazon or Wikipedia… we wouldn’t have web applications that did useful things, we’d just have a bunch of static HTML documents linked to each other managed by webmasters and central authorities. None of this democratized media business.

Thanks to CGI we got the read-write web, but we also made the web way more useful than it was intended. Suddenly browsing to a URL would run some code. And code… well, code can do anything.

Next we’ll build on this idea and see how web hooks can change the game again!

Amazon gets clever with Web Hooks

I once had to build a shopping cart framework for a friend’s company in about three days. It was a challenge considering all the complexities involved in pretty much any instance of a shopping cart. Most shopping cart solutions would hardcode predefined rules or configurable options for determining tax, shipping, and promotional discounts. This was kind of ridiculous since those are usually the points where it varies quite a bit from cart to cart. My solution was to provide hooks because I knew these guys were programmers and were smart enough to implement their own callback functions to define whatever rules they wanted. It turned out to be very powerful, one of their favorite features of the framework, and it was trivial to implement.

So how does an architecture like that translate to a hosted e-commerce checkout solution? Obviously, web hooks! Amazon recently figured this out as they just announced a Callback API for their Checkout solution under their Amazon Payments arm. It lets people use their hosted “checkout solution” while letting them customize the rules for calculating taxes, shipping, and promotional discounts in the most powerful way: code.

This is a fairly significant example of web hooks because it breaks away from the notification use-case that people seem to be caught up with. By actually processing the results of a hook callback request, you’re opening up your application logic to the end user. This is user contributed logic, an under appreciated and mostly unrecognized trend for the emerging service platform aspect of the web.

I don’t want to tangent too much into a rant on the web I want (and I think we all want, but don’t realize it). Instead, I’ll summarize some details on this new Amazon announcement of web hooks. The full documentation for which can be downloaded in PDF format.

You define the callback endpoints in a configuration XML document used for setting up Checkout. They use HMAC authentication for hook requests with added timestamp nonsense, very similar to the way they do Amazon Web Services. The payload is somewhat surprisingly not XML, but instead a URL encoded key/value pair format. The strange thing is that they try and do nested data structures this way, which makes me think JSON would be the better choice. And just to confuse you slightly, in the documentation they use XML to describe the data structures they pass to you and expect back, even though XML is not actually involved. Nevertheless, the documentation is quite good.

Anyway, I’m happy to see a use-case emerge in a way that starts to hint at the aspect of this model I’m more interested in: customization via code. Notifications, or “push,” are really just something along the way. Although this is a trivial example, you should be able to start imagining how it could be used to build a plugin architecture for your web app. The ability to allow your users to extend and build new functionality they can share with others does not seem like an empty value proposition!

Beyond Mashups: Service Integration and More

Today I went to Mashup Camp and gave a session on Web Hooks. It was mostly my standard talk with minor updates, but framed in the context of mashups. It seemed to go well, and I think I was even invited to talk about them at Google. John Musser from Programmable Web was there, but missed the talk. However, I did get to talk to him and he seems genuinely interested in it. He mentioned perhaps even including event hooks as part of the PW directory.

Anyway, here are the slides I unshifted upon the deck I usually show. The notes are sort of poor quality since I added them afterward in a rush to get these online.

Pioneering Web Hooks since Web 1.0

I always try to point out that web hooks are nothing new. In fact, most “new” technology is not new. The mouse, for example: in 1984, when Apple introduced the Macintosh 128K, the mouse and GUI exploded in popularity and quickly became the standard model for interfacing with computers. Little did people know, the mouse had already been around for 21 years! It was invented at SRI by our pal Douglas Engelbart way back in 1963.

It’s interesting to note that the mouse is sort of useless without the GUI. The two are pretty strongly coupled. Perhaps the latent success of the mouse was in part due to the GUI taking about 20 years to be perfected and made ready for mainstream acceptance. In any case, most new technology is not new. Remember that, kids!

The name “web hooks” might be new, but the idea isn’t. The technology certainly isn’t; it’s just HTTP. In fact, you can say the same thing about REST, which wasn’t coined until 7 years after HTTP.

So what was the first major application of web hooks?


The classic example of a web hook is PayPal’s Instant Payment Notification feature, also known as IPN. This feature allows online retailers to handle real-time purchase confirmations from PayPal. After giving PayPal a URL for IPN, they perform an HTTP POST to it whenever the seller receives a payment. This allows the retailer to better integrate the rest of their system with PayPal.

It was a pragmatic solution to achieve simple server-to-server communication. What blows my mind is why it hasn’t been used for much more since then, hence this blog and my slight obsession with the idea. There have been others that have made use of this more recently, and I’m going to cover those guys on this blog, but still… IPN has been around for over 5 years.

I’m not sure how long PayPal has had the feature, but I know it dates back to at least 2003. And PayPal wasn’t the only one. There used to be (and still are) a lot of alternatives to PayPal which provide the same feature. Even Google Checkout has an IPN equivalent.

But the popularity and success of PayPal has given them loads of experience with IPN. They’re basically web hook veterans. I’d love to talk to some of their engineers about their experience providing the IPN feature. How it evolved, maybe how it scales, how the community likes using it, what kinds of tools have been made around it…

Actually, one thing I didn’t realize until I researched it for this post, is that there are 3rd party services that consume the IPN web hook to provide extra features for the retailer. For example, Scrobbld extends the functionality of PayPal with a nice web UI to manage post-sale affairs without even entering your PayPal username. It’s based entirely on consuming the PayPal IPN.

Now an interesting thing to note about PayPal’s implementation of IPN is their confirmation step. Instead of just sending the IPN in a fire-and-forget fashion, they want you to perform an HTTP POST back to them with all the data from the IPN. They use this to prevent spoofing and let you know the IPN did, in fact, come from them.

The HTTP status code you respond to the IPN with is used to determine if they should resend the notification. If you return anything other than 200 OK, they try to resend the IPN in full for up to four days.

And although some people like the idea of a structured payload document in their web hook posts, PayPal IPN data is straight up simple key/value request parameters. This eliminates the need for the consuming script to parse anything beyond what their CGI library probably already takes care of. In the case of PHP, all the data is right there in $_POST.

So in general, I like PayPal’s approach to IPN. They’ve had plenty of time to get it right and do a good job at keeping it simple. Although they might not be the very first example, they’re definitely the most used example of web hooks in action.

Now, I was just thinking about the mouse and GUI analogy I made at the beginning here. Web hooks are a complement to web API’s in much the same way the mouse is a complement to the GUI. In fact, like the mouse, without their API counterpart, I think web hooks are much less useful. Without a GUI, how useful is a mouse?

Perhaps the reason web hooks are taking longer to be adopted is similar to the mouse waiting for the GUI to develop. Even though IPN was around in 2003, it wasn’t until 2005 that web API’s started to become popular. Now that API’s are pretty commonplace, I guess it makes sense web hooks are starting to grow in popularity…

Web Hooks and the Programmable World of Tomorrow

It’s always nice to start a blog with some really juicy content. Luckily I recently put together a 40 minute talk on web hooks and put all the slides on SlideShare, which I can share here. The talk itself is mostly in the notes, but the slides play a big part. At some point I should have a video recording to put up here. Until then, enjoy my slides since I put a lot of work into them! I also wrote an introduction to the talk on my blog.