The hottest tech trend of the year is Big Data, the application of powerful new analytical tools to the mountains of data now being created by the Cloud and by millions of new sensors embedded into the natural world.

Almost everywhere you look these days, from commerce to government to scientific research, people are talking about Big Data. It’s a hot target for new start-ups and venture capitalists. And even that celebrated creator of photography books about tech topics, Rick Smolan (“24 Hours in Cyberspace”, “One Digital Day”) is preparing to publish a big new book on. . . Big Data. It is slated to be introduced at the annual “Silicon Valley comes to Oxford” in November.

Data used to be boring. Now it’s exciting. It’s trendy. It’s even hip. But the real question is: Is Big Data real? The answer is yes, but. . .

The greatest strength, but also the greatest weakness of high tech is that whenever a cool new technology comes along we all race to implement it into whatever we are doing. What’s good about this is that it leads to the very rapid dissemination of anything useful and valuable across the industry. What’s bad is that it often leads to a lot of money and time being wasted trying to shoehorn the hot new technology into a business that really can’t use it.

So the real question is not whether Big Data is valuable – it is, in the right applications – but rather what is the competence of its users to put it to work in a productive way? Guys like me, who work in that world, have already seen cases where Big Data can be a real ‘silver bullet.’ But we’ve also already seen too many examples of companies and other institutions implementing Big Data solutions just because it seems cool. And I’ve yet to encounter a case of installing infrastructure purely for infrastructure’s sake that has actually worked.

The fact that you are reading this article already tells me that you are pursuing the implementation of Big Data to your organization because a) your boss keeps asking why you haven’t done it yet; b) because it is the theme of every conference and trade show you’ve been to in the last year, so it must be the Next Big Thing; and/or c) because it is supposed to do miraculous things, and your organization or your career could use a miracle right now.

And, in fact, Big Data just may prove to be that miracle you are praying for. The greatest strength of this technology is that it can quickly sift through unimaginably large caches of data and find the needle in the haystack, that one incredibly valuable – and until now unattainable – piece of information that could change your entire business. In the world of scientific research, Big Data can often mean replacing the traditional but imprecise science of sampling with looking at every single subject – say every tree in the Amazon rainforest. In the commercial world, Big Data has proven very useful in digging through all of the data from, say, the movement of every customer through a store in a month’s time, to test the effectiveness of display location, pricing, discounts, all the variables of retail.

But Big Data can also be an enormous waste of time and money if that mountain of raw data consists of only dross, or the only patterns that emerge from all of your analysis are those you could have discovered with a couple hours of simple observation.

So, let me suggest a few questions you might want to ask yourself before you jump aboard the Big Data train:

1. Is your business challenge unique? You may think so, but look around – there are probably scores of companies facing problems just like yours. And if that’s the case, there’s probably already an off-the-shelf Big Data solution that suits your needs. Big Data may seem brand new, but in fact, giant companies like Wal-Mart have been pursuing it for years, and have invested hundreds of millions of dollars developing powerful new tools applications around retail point-of-sale. Use their creations and save some money.

2. Do you know what you’re looking for? The story of Big Data is often couched as one of unexpected discovery – and indeed, you may just have such a revelation about your business that utterly changes the way you operate. But the reality is that to find that elusive bit of illumination you kind of have to know where to look and what you are looking for. And to do that, you first have to properly frame your question. If you haven’t already done that, you will likely spend your time running in circles.

3. Do you have the right scope? Like a lot of hot new technologies of the past, there is currently a tendency to see Big Data as a panacea to all of your business problems. It isn’t, and it will never be. But if you have a general idea of what you are looking for and you keep your ambitions small, Big Data can be a powerful tool. In other words, instead of setting out to revamp your entire retail operation, and instead try to determine, say, the most effective location for the cash registers in your stores, or what day of the year mothers of elementary school students are most likely to buy rain gear, or the most likely key to malfunction on your new smartphone, or profiling the customers who click on your web page then you may be thrilled with the power of Big Data.
4. Is there anything there? Obviously, this is something for which there is no easy answer. But the fact of the matter is that a lot of those mountains of raw data are just that – and no amount of analytical processing is going to glean from them any valuable wisdom. The biggest mistake many companies will make in the next few years will be to assume that, because they produce so much of it, that data is now free. It isn’t. It’s just a hundred or a thousand times cheaper than it used to be. But if you spend a ton of money on tools that filter through a million times as much raw data as you used to do with comparatively rich data, you are going to end up spending a lot more, and getting a lot less for your trouble.

Do all of my warnings mean that you should stay away from Big Data? Hardly. Only that you should go in with your eyes open. Remember, it is not about the process, but the results. And it is not about finding hidden gems, but empowering smart business decisions. If that is your strategy, welcome to the Big Data revolution.