Analysis: The broken cloud

The cloud makes some big promises about the future of enterprise IT. Cheaper, better, easier and above all, simpler.

When it comes to applications, the software-as-a-service (SaaS) model is seen as an opportunity to ditch the baggage of the past - lengthy, complex integrations and vendor lock-in - and make a fresh start.

But has the cloud lived up to that promise yet? The answer, sadly, is no.

I found that out the hard way, when iTnews editor Brett Winterford asked me to write a report on integrating SaaS solutions.

“Just the top 20 providers," he said. "Shouldn’t be too hard, right?”

“Yeah, that’s fine,” I answered.

That was six months ago. In the intervening time I’ve produced graphs, summaries, scorecards, written nearly ten thousand words and made spreadsheets big enough to cover a moderately sized meeting room wall - all of which we're releasing as a lengthy report next week.

Brett, it wasn’t fine. Not at all.

So what was so difficult?

The rules are different now

My first mistake was to assume that my experience as a software developer would carry over well. But I soon learned the work I’d done in offline integration was all but useless in the SaaS world.

Ask me how to make the MoinMoin wiki engine work with LDAP for authentication, or integrate it with Trac to do issue tracking. Not so difficult once you get the hang of it: find the parts that need to integrate, figure out how to access that data and write some code to make it work.

Unfortunately, none of that makes sense in the cloud.

Firstly, where do you put your code? There are plenty of bad integration solutions -- even from companies that should know better -- that involve running a plugin on your computer that downloads data from one cloud service and uploads it to another.

Great. So now I shut down my computer for the night and everything stops.

Or you run it on your own servers, but then, what’s the point of the cloud? Aren’t you meant to be paying someone else to run all that for you?

Writing code to integrate various cloud services is antithetical to the very idea of the cloud. We’re meant to be in the age of “it works out of the box, just sign up and we’ll take care of the hard stuff”.

If it works out of the box except when you want to actually use it in the real world, it may as well not work at all.

Accessing data is also different now. In the old days, when we stored things in “files” and “databases” like the cavemen that we were, you could just open that file or connect to that database. The data’s right there, naked as the day it was born!

For security and technical reasons, that’s never going to happen in the cloud. Ask Sony about what happens when people get raw access to your database.

No, there’s only one way of accessing data in a cloud service: the web API.

Before I go too much further, I should say that I actually like web APIs. A well-designed one can be a pleasure to use and they actually paint a tempting future for online integration.

Offline, the best integrations don’t access data through files or a database. They connect with code directly, which can yield some really fantastic deep integration possibilities like live synchronisation, notifications, and custom widgets displayed inside the application.

The problem with offline APIs is that they’re usually a pain to work with. Some only work for services using the same programming language, or use some great new custom communication protocol that you have to bash your head against for weeks to figure out.

The redeeming feature of SaaS is that we get to do it over. Do it better. We can learn from these mistakes and build something nicer and easier.

Only we aren’t. We’re making the same mistakes again.

Standards. Anyone?

You know what made the nineties awful, other than Pauly Shore? Too many file formats.

Everyone who wrote software would immediately go and design their own format, and before too long we had hundreds.

The problem was that there were almost no standards, and the few that existed were basically ignored. We’re still suffering today. Look at music files. MP3? Windows Media? iTunes AAC? Linux’s OGG? What a mess.

Read on to page two for more about standards and 'marketplaces' or register now to attend iTnews' launch of 'Which Clouds Play Nice' at vForum, Sydney Exhibition and Convention Centre on October 19, 1pm.

Things are, fortunately, getting better for file formats.

Microsoft has published a specification for the Office file format. All the big calendaring programs now support the iCalendar format. We’re even down to about three popular image formats.

I suspect that the internet is largely to thank for this; once everyone actually had to deal with each other’s files on a regular basis, standards started to seem like a really good idea.

Sadly, we’re quite close to repeating the exact same multi-decade cycle of pain again; this time in the cloud.

In the report we cover twenty cloud providers across five industry groups: Sales/CRM, Office Communication, Collaboration/Project Management, Accounting, and HR. How many formats would you expect to have to research to figure out if these solutions can integrate with each other?

If you answered somewhere between five and twenty, as I did, you’re very wrong.

In fact, there are almost no standards at all. If you want to find out which providers integrate with which, you have to check each pair individually.

There are about two hundred complex interactions to analyse to even get close to knowing which solutions could potentially work best together.

I'm only grateful we weren't assessing the top 50 SaaS providers -- that would be more than 1200 things to study.

If this complexity only affected the job I was asked to do, it wouldn’t be such a big deal. But this is research done on a daily basis by rapidly greying IT managers around the country.

They need this information to work out which services to choose, and if they get it wrong the consequences can be expensive.

Marketplaces aren't a solution

A number of cloud providers, big ones in particular, have jury-rigged a solution on top of this integration mess. That solution is called a cloud marketplace: a place where third party vendors can sell pre-built custom integrations to their beleaguered customers who just want the thing to work.

Nice idea, but in fact it just makes the whole thing worse.

Now instead of just asking “does AwesomeCRM integrate with Cool Dude Accounting?”, you have to ask “does AwesomeCRM integrate with Cool Dude Accounting on its own? Does it integrate via a third party? Which of their marketplaces is it listed on? Do I trust the third party too? How do I choose if there’s more than one?”

That’s not to say marketplaces aren't useful. They are a great way of collecting third-party additions to a cloud service and making it easier for customers to add functionality.

But providers should never use them to mask holes in core functionality. Any cloud provider who doesn’t treat “working as part an entire cloud IT stack” as core functionality is ignoring the real needs of their customers.

How to fix it

The fix is actually pretty simple. We just need standards. Not standards for file formats, but standards for cloud data. Standards that anyone can use to integrate with anyone else.

The SaaS industry needs to agree on a CRM standard, an HR standard, an Accounting standard. There needs to be new standards bodies, or standards around SaaS need to be submitted to the old ones.

Either way, the only solution is to work together. Microsoft and Netscape/Mozilla didn’t with web browsers, and progress on web standards languished for nearly a decade. We can do better.

Some existing standards are already becoming popular in the cloud space. I mentioned iCalendar before. It’s already used by all the big cloud calendar providers.

Same with the IMAP standard for email. That means any provider who wants to integrate with cloud email only has to write one integration.

But there are only a few of these, mostly in services, like calendaring, that have been online for a while. Time is running out to standardise the rest before the corporate and government worlds sign up en masse.

If you’re in the market for cloud services, make sure to ask your prospective providers, “do you support a standard cloud data format for integration?” and into their confused silence interject, “Do you have a timeline for supporting one? It’s important to us that our Cloud services are standards-based.”

If enough people ask, providers will listen.

The cloud has the potential to be so much better than what came before. To make IT easier and cheaper for companies and let them focus on what they really want to be doing: running their businesses well.

This is an opportunity to start with a fresh slate. Let’s not waste it.

iTnews launches 'Which Clouds Play Nice' at vForum, Sydney Exhibition and Convention Centre, on October 19, 1pm. iTnews readers can register to attend free.