The internet has been dead and boring for a while now. It has reached a point of stability where flashes of technological creativity are rare, but every now and then some new technology can put a spark back in the ole gal (no sexism intended).
If you haven’t heard of WebHooks or PubSubHubBub its about time you did. Both are designed to simplify and optimize the web.
Webhooks let applications talk to each other using very simple HTTP. Webhook enabled applications run (so far) on app hosting sites in the cloud. What makes them different is that they constantly scan for POSTS to a designated URL. To use the application, you register your application with the other webhook enabled application and provide a callback URL. You POST data from your app to the url of the receiving app, and monitor the callback URL for its response. Your app then takes the POST it received and processes it.
I will give you a simple example. Your accountant sets up an app for all their customers that has all the sales tax rules for every community in every state. Every time your company makes a sale, your oline store application sends the transaction ID, amount of taxable goods and the long zip code where the sale was made. The webhooks enabled app receives it, calculates the correct salestax and immediately sends back the result to your online store which incorporates the local sales tax information it received in the invoice as the customer checks out.
This is a very simple example of using Webhooks, you will be able to come up with much better.
Pubsubhubbub (PSHB) is a realtime, multicasting webhooks enabled publish and subscribe system. Historically on the net, most information is received after it is pulled. For example, we set up receive intervals for our email. Our browsers update our RSS feeds at pre determined intervals. We repeat the same searches over and over, just looking to see if there is anything new. Even when we get alerts for new email or information, the alerts are generated by actively polling the source. PSHB changes that.
The PSHB hubs are cloud based distribution centers. Publishers choose to distribute their data through any number of publicly availabe hubs. Subscribers choose to receive their “subscriptions” or data through the Hub. The beauty of the hub and why this makes sense is because the HUB multicasts the data to each publishers’ subscribers, is easily scalable and it distributes to subscribers in realtime. Every time a publisher has something new it can post the data to the PSHB, which knows who that publishers’ subscribers are and immediately multicasts the new data to all the subscribers. In real time.
The implications and opportunities to change business on the web, and actually to any device that can subscribe to the HUBs are huge.
This could be an open door for the content business. For instance, currently aggregators have to get their news the old fashioned way, through RSS feeds and news alerts that they retrieve throughout the day. That is not realtime news. Using The Associated Press as an example, AP could post their stories to a HUB. In realtime, the HUB can update member websites so that they will always have information first, before any aggregator. It may not take long for aggregators to recognize the new data on the member sites, but they won’t have it first.
The New York Times could do the same thing. Subscribers could get everything first, in realtime. Then after some delay which might be 1 minute, it might be 30 minutes depending on what the paper thinks is the value related to timeliness, it could post on the website and on twitter and facebook as updates. Would NY Times online readers pay $1 a month to be guaranteed that they get their news first, before anyone else ? I dont know.
In the sports world, text based play by play websites could be updated in realtime rather than pulling every 30 seconds or requiring the user to hit refresh every few seconds.
Huge databases can talk to huge databases and exchange data more efficiently, hopefully increasing the value of the information. Medical databases, crime databases, any database hosted by different organizations could use webhooks and HUBS to agree to keep each other up to date.
This is all JUST beginning. Its all brand spanking new. Already extensions like Superfeedr are starting to appear, designed to scale.
Dive in and take advantage of the opportunity, ignore it at your own peril
m
This is some good stuff. I wonder what took people so long to invent something “else” that is going to change the internet?
Comment by econ365 -
Pingback: Transliacijos technologijos rytdienos internete
I’m excited to see what the media does with this technology. Rather than throwing out information and hoping to draw in readers and viewers, relevant and instant news is now delivered directly to the end user. Great post as always
Comment by Lin and Jirsa Photography -
Pingback: What I don’t like about RSS « Altmode
Pingback: Webhooks Are The New Black
Specifically regarding comments from Saleh and those who agree with him, I think you have not thought this matter through very clearly and you are totally wrong.
You are asserting that youtube, after building a massive “customer” base by giving away its product for free, will then start charging customers who are “addicted” to the product.
I don’t see any way that this business model can work.
First, it rests on the assumption that demand for what youtube provides is extremely in-elastic. I think clearly it is opposite – a small increase in price (e.g. from zero to anything else) will drastically reduce demand. How many people are going to check out youtube videos when they are procrastinating at work or school if they have to pay for it?
Second, the barriers to entry are so low that anyone can compete once there is a non-zero cost to the customer. If there is any profitability in the customer-fee based model for video hosting, it will quickly get driven very close to zero by competition. What they provide is like a commodity, so there is no reason for the price they can charge to be anything except very slightly higher than the marginal cost. Given that there are well-capitalized potential competitors, this model does not seem like a good one (I wouldn’t want to compete with NBC, via General Electric Cap Corp, on the ability to raise capital at low borrowing costs.)
Finally, your argument that customers are addicted to the product is logically false (question-begging argument, technically). If you give away something for free, there will always be a lot of demand for that thing. To assume that people need, or are addicted, to this thing because they consume a lot of it is not true. There is no evidence that significant demand exists at non-zero prices, and the fact that youtube has not begun charging supports the hypothesis that the demand is not there.
At best, I think youtube becomes similar to a utility.
Comment by jlai24 -
This is exactly the kind of hype and exagerations we don’t need- as if we didn’t learn from the previous Internet hype period. Fact is- PubSubHubbub is an “evolution”, not a revolution vs. the current RSS push mechanism. Secondly, it uses RSS, so it would be misleading to imply that RSS is the “old fashioned way”.
The issue with content dissemination is not that it was lacking real-time, but rather in its organization and relevancy for a particular user. So, passing content in real-time will just accelerate the mess if we don’t work also improving knowledge management. Most users aren’t glued to their Twitter account anyways, so they will not be able to discern the difference between a 5 min or 25 min delay in news arrival.
Comment by William Mougayar -
Pingback: This week in the paid content debate | Sips from the Firehose
“Skynet versus the Human Hive Mind” is getting closer. 🙂
We have the intelligent Skynet computer as part of our modern cultural lore as a result of the Terminator series and the “Collective Hive mind” as part of it due to the Star Trek:TNG “Borg”. But if technology that grows out of PubSubBubHub enables not only a true machine based AI but also enables a human hive mind the question is begged; Would an evil machine intelligence emerge and truly win out as it did for the Borg in the TNG series t?
OR, would a hyper-connected and biologically based human hive mind emerges out of advanced and futuristic versions of twitter and evolve nearly immediately to be optimally moral in the image of human ideals. Would our Morality/Ethics/Philanthropy elements dominate and win out over our darker side?
Interesting premise for an Sci Fi book I think.
I blog at http://over40innovator.blogspot.com
Roge
Comment by rtoennis -
Pingback: Mark Cuban says “The Internet is about to change” due to buzzword-of-week « The Packet Rat
Something would quality as an “Internet changing” technology if it majorly impacts Internet users. Pub-Sub does not.
This is because the user right now is getting closely same effect with Pub-Sub or without it. Whehter Pub-Sub or RSS, right now, the user is getting what he or she subscribed to. No big change there.
Think of Pub-Sub as just a technological optimization, that is behind-the-scenes from the consumer. It doesn’t quality as a “new” internet changing thing that brings the internet back from the dead.
Comment by Saleh Najar -
Pingback: Mike Villar: Rising Internet Star » Blog Archive » What about the Real Time Web?
Pingback: What about the Real Time Web? - Aerofoil Media | Affiliate Marketing, Display Advertising and Social Media Services That Fly
Pingback: Flow » Blog Archive » Daily Digest for August 27th - The zeitgeist daily
Pingback: Sailing on Dry Land | Taylor Davidson
Pingback: The Internet is about to change | Digital Asset Management
Pingback: Information is Faster than Nicholas Carr « Bobby Karalla's Blog
Pingback: links for 2009-08-26 « riverrun meaghn beta
Pingback: D.C.’s Cupcake Battle and Is Twitter Only For Old Folks? : Floriday Properties
Mark,
I heartily agree with your premise even if I care not one wit about whether it’s WebHooks/PubSubHubbub, protocol soup or chicken soup – the market/developers will decide that.
While there is a tendency (of some) to confuse attributes with outcomes (i.e., didn’t XYZ do this?), I think that the key take-away here is that:
1. Message oriented approaches are integral to evolution towards a real-time web;
2. The model has a metaphorical patron saint in Twitter (and its ecosystem) so there’s an example that people understand;
3. This same model works really well in compositing information/services and happens to fits the mobility constructs of the mobile broadband generation, something that I blogged about in:
The Mobile Broadband Era: Messages, Mobility and The Cloud (Guest post @ O’Reilly Radar)
http://bit.ly/PqarM
Check it out if interested.
Mark
Comment by marksigal -
Pingback: Macro De-Duplication of data « Recalibrate
Pingback: New Internet Revolution: Web-Hooks - Global Affairs Forum, Politics, Law, Science, Health
I agree with dereksthered and Jonathan Gardner. This is nothing new.
Comment by newsbliss -
How is this any different from Marimba’s Castanet?
http://news.cnet.com/Marimba-adds-controls-to-Castanet/2100-1001_3-203952.html
It was supposed to be the “next big thing” back about 12 years ago, with its “Push” technology (sounds like what you describe here) set to “revolutionize” the internet, but it fizzled out.
Comment by gambit3 -
“Now, what are the benefits of having news pushed to my browser instead of polling the RSS feeds every minute?”
There isn’t any as long as you know all of the RSS feeds that you’re interested in, and you continue to update as new ones are created. A hub allows all of the information to be aggregated at a single source. For example, if you’re interested in Dallas concerts you can subscribe to RSS feeds for Ticketmaster, Live Nation, and all of the local venues, but if a new venue opens you won’t know about shows there until you find their feed. With a hub the new venue would add their events and the information would be pushed to you with no action on your part.
Comment by dbaderf -
Mark – I love ya, man. But, I respectfully have to disagree with you.
Let’s pretend that cyber-security isn’t an issue – ever. And cloud computing results are consistently reliable. (pause here for laughter) And that the majority of internet users possess a modicum of intelligence when it comes to the application of new technology. Then, ideas like WebHooks and PubSubHubBub will be viable and realistic.
Comment by buzzsaw55 -
Pingback: Good Morning Forbes | Forbes Avenue
I invented (but did not implement) a similar technology over 9 years ago, as explained at http://www.1729.com/blog/HowIInventedAMicronewsSystem.html, which includes a link to http://web.archive.org/web/20010223204516/miski.sourceforge.net/miski-white-paper.html . If implemented, it would have been like a scaleable multi-server cross between Twitter and Delicious, with real-time posting and reposting of links to new content (or old content), all spam-free.
Comment by pjdorrell -
Pingback: The Internet is About to Change [Voices] | UpOff.com
Pingback: What are webhooks really about? « blogrium
Pingback: The Internet is About to Change | Mark Cuban | Voices | AllThingsD
As many people have rightly noted, push notifications and “webhooks” in particular are nothing new. These things are not going to change the internet indeed.
The second notion here is “OK, the tech is nothing new, but the people just started getting it, and that will revolutionize the internet”. I believe this is not true either.
The tech for push existed for a long time and whenever there was a benefit or a customer pain to fix, the tech was used to solve it, see IMAP as example.
Now, what are the benefits of having news pushed to my browser instead of polling the RSS feeds every minute? What customer pain will it cure if I get the New York Times news in real time as opposed to 30 second delay? I get so many feeds, that even if something arrives in real-time, I won’t be able to read it immediately because of all the other stuff that arrived before.
In other words, there is no business model for immediate news delivery. People won’t pay for it. If you need it because you are a trader, you get a Bloomberg terminal and pay a fortune for it. If you are not a trader, you won’t pay a penny to deliver it 30 seconds earlier.
Then, what is the benefit for me, if my bandwidth and CPU is saved by using “push” instead of regular polling? Nil. Bandwidth and CPU are utility. I have huge bandwidth required for video streaming. So it can handle RSS polling just fine. I have really good CPU, because it’s needed for video decoding, accounting software, for running Javascript from all those fancy Web 2.0 sites. Thus “push” instead of “poll” doesn’t solve anything for me. No customer pain here = no business opportunity.
By “me” here I mean average customer. Well, not everyone got a lot of bandwidth/CPU, but with costs falling every year we are all getting there quickly.
Comment by Oleg Kokorin -
Pingback: The Internet is about to change « blog maverick « sull is vocally active
Mark, in response to your replies, I’m glad to hear that you get it more than most people! My comments here were in response to other comments as opposed to your post. 🙂
Comment by Jeff Lindsay -
The real change is going to come from emergent properties of the ubiquity of an event-driven programmable web. These come from subtle qualities of the nature of and evolving ecosystem of HTTP.
Realtime is just a side effect of event-driven systems.
Anyway, I agree the Internet is about to change because of webhooks. I just don’t think realtime alone is a game changer.
From MC> agree 100pct. But realtime is a concept that people understand as a starting point. Its the low hanging fruit that get people thinking about what event driven apps can do
Comment by Jeff Lindsay -
Mark,
Thanks for your well written explanation of these technologies. Yet, you don’t explain what is different today that will cause the internet to change that was not present last month, last year, or even a few years ago? What specifically leads you to advise that NOW is finally the time these nascent technologies are going to lead to changes when they have largely not revolutionized the daily experience for non-tech internet users yet? (By saying the Internet is “about” to change, you are conceding that the change has not yet occurred, just curious what makes you feel it will change shortly?)
Comment by smeade -
Pingback: Peter Renshaw (bootload) 's status on Wednesday, 26-Aug-09 00:20:42 UTC - Identi.ca
Pingback: The Internet is about to change « blog maverick « Netcrema – creme de la social news via digg + delicious + stumpleupon + reddit
In programming circles, this is not new technology. Protocols such as IMAP have been doing this for a very long time. TCP/IP was designed from the ground up to support this.
It’s exciting that the rest of the world is start to get it, however. When you have a network with true push, you can do some really exciting things. Compare IMAP versus POP and you get a hint at what is possible.
As we see a shift from poll to push, there will be a reduction in web traffic and decrease in message latency. Imagine instead of polling an RSS URL every ten minutes and pulling down megabytes of data, you register that you want to be informed when a new article is available. Then, you get sent the new article and only the new article within microseconds of the article being published.
There are also ideas that are simply impossible with polling that are easily solved with push. Let’s say you want to know when Amazon changes prices on any of its products. If Amazon allowed people to subscribe to the price change service, they can keep the entire world informed on the current price of every product in their store without breaking their bank on bandwidth and CPU costs.
Comment by Jonathan Gardner -
WebHooks and PubSubHubbub don’t intend to reach the end-user consumer. They can work in concert with Comet and Web Sockets, or even XMPP to achieve this.
From MC>thx for the comment Jeff. I know you are one of the gurus ! I wasnt suggesting everyone would write their own hooks, but as best I could, I had to put it in terms the genpop would follow. I know you have been pushing this hard for awhile. I just thought i would try to help it along
Comment by Jeff Lindsay -
Its funny that solving the polling problem of HTTP (which is built on TCP which doesn’t have this limitation) is groundbreaking.
We already have Comet which is widely used to establish server side push. Eventually you must reach the consumer who probably only has a browser and cannot allow inbound connections.
As for large commercial databases, I doubt that they would gain a profit from pushing to competitor databases, and if they did, they are probably using their own lightweight TCP connection.
Having said this, I do believe that PubSubHubBub is only a step rather than a big leap in moving away from inefficient polling, but only because of the limitations of HTTP. I think pushing for “Web Sockets” http://dev.w3.org/html5/websockets/ will bring larger changes to the internet, though it may take too long for browsers to start supporting it.
Comment by codexon -
Does this potentially mean if we pre-determine the sites we would visit there will be no load time because we will be fed instantaneously when new content is available?
Comment by starwinar -
Brian, as long as your app that’s making this API call is defining the API, and it was the user who defines where it makes the call, it will be a webhook. I believe this example came from Amazon’s callback API that I showcased in my talk that Mark linked to at the beginning.
Just to be clear, as the progenitor of webhooks “the movement” (as opposed to the technology, which, sure, it’s been around. In fact, it’s just HTTP), here is how I define webhooks as an architectural pattern: allowing users to specify URL callbacks in your application that you POST to on events. These can be for fire-and-forget notifications (PayPal IPN), or for building an extension API (HookPress), or many other uses.
Real-time, or push, is just the beginning. That was a nice side effect of this whole thing, but there are grander implications that have motivated me to push this idea for the past 3 years.
Comment by Jeff Lindsay -
Ahh I see the utility of web hooks now. Thanks Brian.
Comment by dereksthered -
Pingback: Dallas Blog, Daily News, Dallas Politics, Opinion, and Commentary FrontBurner Blog D Magazine » Blog Archive » Mark Cuban: “Internet Interesting Again”
Just want to point out I think your example of the sales tax is incorrect….what you’re describing is a standard API, not a web hook. Webhooks sounds like it is specifically push notification.
If I were to morph your example…a webhook would be if the accountant’s app automatically notified you whenever a new tax rate goes into effect. Then you wouldn’t have to go check the accountant’s server after every sale. You could store your tax rates locally and use that, confident in the fact that the accountant app will contact YOU if the numbers need to be updated. This is push notification.
API’s are what is really revolutionary here and we’ve been seeing that trend for years. This is really just another extension of APIs…for example I was using Paypal’s IPN for a while now and never thought of it as a “webhook”. It’s a good idea and I’d like to see more of it, but I don’t think it’s accurate to think of it as something new on the scene.
Brian Armstrong
http://www.StartBreakingFree.com
From MC> the point i was trying to make was that any authorized app can talk to any other that uses webhooks. the ecommerce app could be from anyone. Because its using webhooks, it simplifies it dramatically. But then again, Ive still got a lot to learn about it.
Comment by krypton1 -
FYI, the correct spelling is PubSubHubbub.
Comment by dlanstein -
Mark, thanks a lot for mentionning our Superfeedr. We’re slightly more than a PSHB extension, though, and I’d really love to prove it!
I can’t agree more on the value of “real-time” vs. “delayed”, even by a few minutes information. It is hard to perceive for the “average joe” right now, but I definetely agree that the news business is going to be washed away (again!) but that real-time wave.
I would even go further and say that the web’s infrastructure (poll-based as you explains) will soon be dual : geographically (as it is now) and timely organized. The only issue is to put all that data in movement : take it from it’s “permanent” location and put it into the real-time stream that is flowing from a service to another. And that is exactly what Superfeedr is all about. Let’s meet and chat about it! julien@superfeedr.com
But, again, thank you for waking up people!
Comment by julien51 -
Love this technology. Combined with semantic web tech, rdf / microformating, and you have the internet we will see come to life in the near future. Devices, content and people integrated and communicating at a level we have never seen before. Awesome. Thanks for the great post. You have the same knack for explaining new tech and trends as Steve Yelvington. Hope you don’t mind the comparison.
Best,
Brandon
Comment by bfmount -
Mark, your articles on the future of the media are far and away the best thing out there. Would love to know your thoughts on Huffington’s social news.
I don’t think the NY Times can compete on real-time news and should leave that to the AP or Reuters, but the NY Times can compete on content and should utilize social aspects, logins, and demographic data to better serve their readers and advertisers.
I wrote an article on what the future newspaper may look like. You can find it here: http://bit.ly/future_news.
I would love to hear your thoughts on what you think a future news platform would look like. You have come out in favor of a subscription based approach that effectively blocks aggregators. Do you think this would work with limited buy in from the newspaper industry? Can media generate a substantial portion of their revenue from readers or are they going to have to rely on advertisers? Etc.
Comment by anvilmike -
I’m confused. What does web hooks give you that web services do not? And I don’t think everyone truly understands how HTTP works. If every web developer knew how HTTP worked, there would be a lot more parameter validation going on. Instead HTTP’s stateless model opens up a lot of potential for abuse, but not everyone knows that. So does registering with web hooks allow for identity assurance? If not there could be an opening for hackers to exploit it.
Comment by dereksthered -
I completely agree web hooks combined with PubSubHubBub and the like are a huge development, Marc. The RSS examples are slightly off in the sense that ping servers have been using a PubSub architecture for years, but the interesting part about web hooks to me is the simplification of the PubSub model. Just like REST has come to dominate web services, web hooks will come to dominate PubSub, and for exactly the same reason: everyone understands HTTP and tools for using it are ubiquitous.
In my p2p world, it will be interesting to see if web hooks can come into play for making HTTP calls directly to HTTP servers running locally on users’ machines with ports opened through UPnP or PMP. You could automatically notify all subscribers computers directly whenever there’s a new bid on eBay, for example, a price drops, a user comes online, etc etc. The HTTP server on the client side potentially becomes even more interesting.
Glad you’re covering it.
Comment by adamfisk -