Building world-class data systems

 

“What does it take to be world-class?” It’s a question I’ve had put to me – almost always rhetorically – by fellow executives, by self-styled thought-leaders, and by other, less convincingly “world-class” authorities. 

At a TDWI conference in Las Vegas recently, a clerk cheerfully described the hotel in which I was staying as “world-class.” Right. Between the house music boom-booming down in the courtyard all night – audible as hell thanks to the single-paned glass in my 12th-floor room! – and the airborne toxic event that was the hotel’s ventilation system, I judged her claim … less than convincing. 

It was also recently asked of me by an organization looking to embark on a complete refresh of their decision-making capabilities; this is what really made me engage with the question. 

I’m going to approach this question from a different angle, however. Because those of you who know me know that’s how I roll. So without further ado, I put it to you: What does it take not to be world-class? 

Here’s one formulation: You can’t ignore the time dimension. You can’t decide that you’re going to build the best system in the world and take five years to do it and still call what you’ve done world-class. The two goals aren’t just completely incompatible, they’re mutually contradictory. They’re yin and yang, matter and anti-matter: Jimmie Dale Gilmore and Eminem. 

The time dimension matters. Time isn’t just some abstract thing; it’s an expression of what it means to live and work and decide – you know, to do business – on an ongoing basis. The time dimension has to do with the decisions you make or don’t make today, as well as with the opportunities that exist now that you identify, seize, or just plain miss. 

It’s easy to build a Perfect Data Ecosystem – if you can ignore time. It’s also easy to build a Perfect Data Ecosystem if you can ignore resource constraints, financial constraints, or – most important – human-talent constraints. 

It’s easy to do this, in other words, if you don’t live and do business in the real world. In the real world, however, stuff changes continuously. And what are you doing in the meantime, while your IT thaumaturgists are off building your Perfect Data Ecosystem? How are you making decisions? How are your employees making decisions? Business decision-making doesn’t stop because business doesn’t stop. Challenges and opportunities don’t suddenly cease to appear. 

And to think that you can bet big on a years-long effort to produce a Perfect Data Ecosystem is to blithely ignore all of this. To think that anyone can produce a “Perfect” platform of any kind is to be dangerously naive. As you work and make decisions and do business, your requirements for a Perfect Data Ecosystem system will likewise change. 

So let’s revisit the question. What does it take not to be world-class? Here’s another formulation: You focus too much on speed and too little on quality. You say, “We’ll never get it perfect, let alone right, so let’s get it up and running as quickly as possible.” Or – worse still – let’s shoot for “Good enough for now.” 

These words have been like kryptonite to human excellence throughout history. It’s easy to build something quickly and to pronounce it “good enough” if you don’t have to worry about how long it has to work for, about what different circumstances it has to cater to, or about how to support it if there is a problem. 

It’s especially easy to do this if you don’t have to worry about how to make it flexible for the Unknown Future. You know, if you build it thinking that you won’t ever have to change it at some point. 

Two extremes: On the one hand, you’ve got formal perfection: the Platonic Ideal of a data integration infrastructure. On the other, you’ve got corner cutting: a functional solution, designed for current systems, configurations, and conditions, that works in a limited set of circumstances and points in time. These are two opposite extremes. 

Being world-class is delivering quickly and sustainably. It’s building a system that addresses current needs, but which – as a function of its inherent adaptability and resiliency – can be rapidly (and mostly painlessly) reconfigured to address changing conditions and requirements, too. 

This isn’t a weaselly or milquetoast formulation. It’s an issue of maximizing what you can do with respect both to time and to excellence – and making use of a methodology and automation tools to help you do this. 

The key, as we see it, is intelligent automation. Start by automating the repetitive, time-consuming, thankless stuff. Automate the stuff that requires laborious – and unnecessary – human interventions. When you automate, you gain efficiency, alacrity, and resiliency. When you automate, you abstract your conceptual architecture – your data warehouse, your big data platform, and your analytic front-end tools – from the physical instantiation of that architecture. 

You can use automation tools like WhereScape RED and WhereScape 3D to rapidly build a data warehouse or big data system, or to pretty painlessly move from one data warehouse platform from another, to automatically generate a presentation layer for new and existing analytic tools, and to move data to and from different versions of Hadoop. Automation is the key to rapidly building resilient systems. It is the key to building right, now. The key to world-class.