Comments on: Convention for batching http://blog.webhooks.org/2009/06/14/convention-for-batching/ User-defined HTTP callbacks for push, pipes and plugins Tue, 04 May 2010 19:35:11 +0000 hourly 1 http://wordpress.com/ By: DJ Adams http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-176 DJ Adams Fri, 10 Jul 2009 15:58:13 +0000 http://blog.webhooks.org/?p=189#comment-176 Hi Oliver In danger of pointing out the obvious, but this sounds like the classic 'does it scale' question is being applied here as it's applied to everything. The point with webhooks is that the model is inverted: Whereas polling the 'million-to-single endpoint' problem is hard to scale, with webhooks/push, scaling of the 'single-to-million endpoints' is at least addressable (e.g. queues). As Jeff says, webhooks are not a silver bullet, but in flipping the comms model they give us a weapon to resist the polling onslaught (groan). dj Hi Oliver

In danger of pointing out the obvious, but this sounds like the classic ‘does it scale’ question is being applied here as it’s applied to everything. The point with webhooks is that the model is inverted: Whereas polling the ‘million-to-single endpoint’ problem is hard to scale, with webhooks/push, scaling of the ‘single-to-million endpoints’ is at least addressable (e.g. queues).

As Jeff says, webhooks are not a silver bullet, but in flipping the comms model they give us a weapon to resist the polling onslaught (groan).

dj

]]> By: Jeff Lindsay http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-161 Jeff Lindsay Mon, 15 Jun 2009 09:04:31 +0000 http://blog.webhooks.org/?p=189#comment-161 Yes, multipart messages seem like a great way to batch requests. However, they're a little cumbersome to deal with on the receiving end and aren't terribly well supported by many libraries. Yes, multipart messages seem like a great way to batch requests. However, they’re a little cumbersome to deal with on the receiving end and aren’t terribly well supported by many libraries.

]]> By: tamberg http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-160 tamberg Mon, 15 Jun 2009 08:46:33 +0000 http://blog.webhooks.org/?p=189#comment-160 There is an RFC for batched HTTP requests (http://www.snellspace.com/wp/?p=991) Regards, tamberg There is an RFC for batched HTTP requests (http://www.snellspace.com/wp/?p=991)

Regards,
tamberg

]]> By: Oliver http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-158 Oliver Sun, 14 Jun 2009 22:18:03 +0000 http://blog.webhooks.org/?p=189#comment-158 No reason really. I've just been following it's growth and that has been the one major problem I see with it. CNN and Twitter were actually the kinds of services I was thinking would have a scaling problem with web hooks. But I like how you explained that it isn't necessarily the proper answer for those types of services. There are so many services that it is a superior answer to however, that it should definitely have its place in design discussions. It could still be a problem for smaller sites only having thousands of requests to make though. Most small services don't want to use an "enterprise" solution because of costs, so they will have to solve the scaling problems themselves. For example, lots of people have great services running on shared hosting plans. If they start having to send thousands of requests they could start having problems pretty quickly, maybe forcing them into higher hosting costs a little sooner than they would normally need to. (I don't think shared hosting providers would look to kindly on lots of requests going out all day) ps. I just noticed the google groups link at the top. That's probably a better place for my curiosities. ;) No reason really. I’ve just been following it’s growth and that has been the one major problem I see with it.
CNN and Twitter were actually the kinds of services I was thinking would have a scaling problem with web hooks.
But I like how you explained that it isn’t necessarily the proper answer for those types of services. There are so many services that it is a superior answer to however, that it should definitely have its place in design discussions.

It could still be a problem for smaller sites only having thousands of requests to make though. Most small services don’t want to use an “enterprise” solution because of costs, so they will have to solve the scaling problems themselves.
For example, lots of people have great services running on shared hosting plans. If they start having to send thousands of requests they could start having problems pretty quickly, maybe forcing them into higher hosting costs a little sooner than they would normally need to. (I don’t think shared hosting providers would look to kindly on lots of requests going out all day)

ps. I just noticed the google groups link at the top. That’s probably a better place for my curiosities. ;)

]]> By: Jeff Lindsay http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-157 Jeff Lindsay Sun, 14 Jun 2009 21:34:40 +0000 http://blog.webhooks.org/?p=189#comment-157 Sure. One solution is to delegate it to a service that specializes in queuing up those requests. Like blog ping services. But perhaps something like Hookah will grow into a highly scalable "enterprise" event dispatcher. Scaling is going to be different for different topologies though. If you have a large number of events, but not necessarily millions of consumers, perhaps something like the Twitter Stream API is better suited. But I think it's not terribly likely there will be a service that will have a million webhook consumers. That would be nice though. It really depends on what you're trying to accomplish. I think in the example of CNN, a simple RSS feed is fine. It will be timely enough and scales much easier. Webhooks are not a silver bullet by any means. I guess, I'm curious why you're interested in scaling webhooks at that level? This question keeps making me think pre-mature optimization. Sure. One solution is to delegate it to a service that specializes in queuing up those requests. Like blog ping services. But perhaps something like Hookah will grow into a highly scalable “enterprise” event dispatcher.

Scaling is going to be different for different topologies though. If you have a large number of events, but not necessarily millions of consumers, perhaps something like the Twitter Stream API is better suited.

But I think it’s not terribly likely there will be a service that will have a million webhook consumers. That would be nice though.

It really depends on what you’re trying to accomplish. I think in the example of CNN, a simple RSS feed is fine. It will be timely enough and scales much easier.

Webhooks are not a silver bullet by any means.

I guess, I’m curious why you’re interested in scaling webhooks at that level? This question keeps making me think pre-mature optimization.

]]> By: Oliver http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-156 Oliver Sun, 14 Jun 2009 21:19:36 +0000 http://blog.webhooks.org/?p=189#comment-156 A question I have about webhooks is how will large sites handle having to send out potentially hundreds of thousands or millions of http requests to everyone who requested a hook? Like if CNN had a hook people could sign up for to get the latest stories, they would probably get a lot of subscribers. Having to make a million http requests for every new story seems like a large load for a content deliverer to have to handle. Especially assuming that each request waits for an ack. What if a lot of the endpoints are slow or down? It just doesn't seem to scale very well. What are some of the potential solutions to sending out a million http requests? A question I have about webhooks is how will large sites handle having to send out potentially hundreds of thousands or millions of http requests to everyone who requested a hook?
Like if CNN had a hook people could sign up for to get the latest stories, they would probably get a lot of subscribers. Having to make a million http requests for every new story seems like a large load for a content deliverer to have to handle.
Especially assuming that each request waits for an ack. What if a lot of the endpoints are slow or down? It just doesn’t seem to scale very well.
What are some of the potential solutions to sending out a million http requests?

]]> By: Jeff Lindsay http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-155 Jeff Lindsay Sun, 14 Jun 2009 19:38:30 +0000 http://blog.webhooks.org/?p=189#comment-155 Right, they put everything under a single value, which can then be an array (which btw, seems to be slightly unstandardized in many CGI implementations). Right, they put everything under a single value, which can then be an array (which btw, seems to be slightly unstandardized in many CGI implementations).

]]> By: rick http://blog.webhooks.org/2009/06/14/convention-for-batching/#comment-154 rick Sun, 14 Jun 2009 19:27:48 +0000 http://blog.webhooks.org/?p=189#comment-154 Ruby frameworks fake arrays in POST vars with field names like event[0][name]. Kind of hackish, so I tend to prefer JSON as you suggested. Ruby frameworks fake arrays in POST vars with field names like event[0][name]. Kind of hackish, so I tend to prefer JSON as you suggested.

]]>