Opening Pandora’s Box

Is it too late for Data Governance?

I responded recently to a question in a post on LinkedIn, on whether Data Governance should drive how Data Management evolves.

In summary, the consensus on the thread was “Yes, it should.” (Life isn’t quite that simple of course, especially when the Data Governance function probably doesn't even exist in the first place!) The other consideration being, that any activity in the data management space really should link back to, and be driven by, a clear business need. (See my earlier post “Information as a Service” for more on that subject.)

“What’s that got to do with Pandora’s Box”, I hear you ask? Bear with me…

In responding to the thread, I started thinking more generally about the need for Data Governance as an organisational capability. In particular, I began comparing the situation as it used to be when I entered the workforce with the scenarios we find ourselves dealing with today. (Hark at Old Man Duncan…)

And after a bit of pondering, I’ve come to the conclusion that in many respects, we may well have actually gone backwards in terms of data quality and business responsibility for data!

Some perspective. I started working in 1992, when server-based database applications running on DEC/VAX and early Unix minicomputers were order of the day. Users accessed the systems using green-screen terminals with forms-based text-only interfaces, exclusively using a keyboard. PCs had evolved to the point where DOS-based personal productivity applications were reasonably prevalent (WordPerfect was the word processor of choice, and SuperCalc was the market leading spreadsheet). Programming was all done with text editors, and there was not a “Window” ™ or mouse in sight.

In hindsight, guess I was fortunate, in that I joined the “new wave” of business systems developers working with 3GLs and RDBMS – Oracle V6, Forms3.0, Pro*C ReportWriter to be precise.

And SQL. Lots, and lots and LOTS of SQL.

Basically, I learned an awful lot about how to decompose, model, structure and interact with data – the foundations for a career in what we now call Information Management. I guess I can be satisfied that (to date at least) I’ve not been found out. Indeed, some people actually seem to like what I’ve done and what I’m still doing. Yay me...!

Anyway, what all this meant was that a lot of effort went into developing these business systems, and more importantly, into ensuring that the data definitions were well understood. Not just for referential integrity purposes, but at the member record level too. Many businesses still had dedicated data processing teams (clerks) who were responsible for – and took pride in – the accuracy and completeness of the data.

And because this was more or less their only role, they were damned good at it; diligent, conscientious and fast. Result? High-quality data that could then be queried, reported, and acted upon by the business. Everything was pretty focussed on execution on whatever process mattered, and the computer systems were simply there to speed up and ensure rigour of the recording process.

Speed forward twenty years and the world looks like a pretty different place.

We’re living in a mobile, connected, graphical, multi-tasking, object-oriented, cloud-serviced world, and the rate at which we’re collecting data is showing no sign of abatement. If you’re in Information Management, then that’s got to be a good thing, right?!

Not so fast young Grasshopper...

While the tools, technologies and methods available to us are so much more advanced and powerful than those green-screen, one-size-fits all centralised systems of the mid-eighties and early nineties, I think our progress has come at a significant (unacknowledged or even unrecognised) cost. Distributed computing, increased personal autonomy, self-norming organisations and opportunity for self-service were meant to lead to better agility, responsiveness and empowerment. The trade-offs are in the forms of dilution of knowledge, hidden inefficiencies, reduced commitment to discipline and rigour, and unintended consequences. 

And people these days have the attention span of the proverbial goldfish. (Ooh look, a bee…)

Which leads me to my conclusion. That the advent of "Data Governance" as an emerging discipline (and indeed other forms of governance - process, architectural, security etc,) could be considered as a reactionary an attempt to introduce a degree of structure, moderation and resilience into this ever-evolving state of business entropy.

Can we succeed? Or are we trying to close the data version of Pandora's Box…?

We can but hope.