Data Governance - lets make beautiful music together.

I've been in my new post as Director of Data Governance at the University of New South Wales for about a month now.  

Most of my time has been spent trying to get to meet people and hear their perspectives on how things work around campus (or don't), what the key issues are, and just getting to grips with the scale of the challenge in an organisation of approximately 5000 staff (Or 8000. Or 13,500. It kind of depends on which definition of "staff" you're using - but that's another story...)

Anyway, I had the opportunity last week to present to the University's IT investment committee, a senior executive group comprising faculty managers, academic leaders and financial governance. I was allocated 10 minutes to introduce myself and provide an update on the Data Governance agenda. Now, it's already clear that the University has an appetite to do something significant with respect to the usage and accountability for data (the fact that they've hired me is a statement of intent!). But let's be honest - there are going to be relatively few members of the university community who need to (or want to) get to grips with the wider scope, complexity and subtlety of and Enterprise-wide Data Governance and Information Management operating model! 

So I was struggling to think of a way to frame my agenda in a manner that was going to convey the scale of the task ahead without scaring everyone (or boring them rigid), and I came up with an analogy that seemed to resonate with the audience.

If we think of the University (or any other organisation for that matter) as an orchestra, then the various Faculties, Schools and Business Units are the musical sections, and the staff are the individual musicians. The organisation's data is the music to be played and the role of the Data Governance function is analogous to the conductor. 

Now, the outcome for any orchestra is to be able to give a concert to a paying audience (and get them to come back again…). The enabling capabilities which are required to get the concert on stage are a complex mix of contributions from numerous different people, applying multiple processes and skills, with the agreement of all participants to work towards the same goals:

1. All musicians need to have the skills to play their allocated instrument. (not necessarily a given!)
2. Someone has to write the music, orchestrate it, publish it, make sure the sheet music is brought to rehearsals.
3. We have an agreed running order for the concert.
4. Everyone needs to be playing the same tunes, at the same speed, and starting at the same time.
5. Rehearsals and practice (individually, in Sections, and together as an Orchestra).
6. Marketing the concert, booking the tickets, ushering the audience.
7. The performance.

THE CONCERT IS A FAILURE IF ANY PART OF THE PROCESS DOESN’T HAPPEN, AND IT WOULD STILL ALL BE LIKELY TO FAIL WITHOUT THE CONDUCTOR TO BRING IT ALL TOGETHER.

So Data Governance is about consciously and competently orchestrating and conducting all the various efforts throughout the organisation, so that whatever data we have (or need to have) is available, in context, and usable for a range of well-defined purposes.

Underpinning our data orchestra are an established set of formal practices, processes and protocols that define the institutional capability required, many of which may only exist in part, some not at all. Initially, we need to establish which of these services are needed, where are the gaps, and how they will be fulfilled. e.g.

Organisational Capabilities:

• Data Governance Steering Committee, focussing on Data/Information issues.
• Formal identification of Data Owners & Stewards for each data asset.
• Fora that bring together the community of Collectors and Consumers of each given information domain. 

People/Skills:

• Information Management/Business Intelligence Competency Centre
• Value-added analytic services and support the whole-of-institution approach.

Process:

• Business Glossary and metadata management techniques for collating and communicating common & consistent data definitions;
• Data Quality Profiling & remediation service;

Systems:

• Robust, scalable Data Warehouse & BI that is insulated from, and supports, change. 

Now, does everyone need to know or care about all the detailed practices & protocols, or how all the complexity happens? NO.

But I’d content that everyone does need to know about “their bit”, where the information touch-points are with other parts of the organisation, and that whatever their doing fits into the overall context.

To go back to the orchestra analogy, each individual musician doesn’t need to know how the composer and music publisher does their job, but they do need to know that the sheet music is going to be available, and that they’ve got to turn up at the Opera House on Tuesday week for a recital…

Which I guess makes yours truly the baton-wielding eccentric with the funny tailcoat suit and mad hairdo... 

CxO’s are blithering idiots! But what if they’re Idiots Savant…?

A client of mine (let’s call him “Mr. X”) who is the CIO at a government agency (let’s call then “Agency Y”) recently spent a fairly significant proportion of his organisation’s ICT budget for this year on a fairly large data warehouse appliance {let’s call it “Product Z”). Agency Y also got a 20% discount off list price for buying immediately.

“Nice work!” I hear you cry, “They’ve bought a market leader in the field which is constantly improving its technology, they’ve purchased at a discount, and if Agency Y is investing in Business Intelligence platforms, then they must be thinking progressively and wanting to drive more value from the information that’s available in the organisation. Splendid!”

Now, put that purchasing decision in the context of some additional information about the current situation at Agency Y:

• There are no current business projects identified that require BI capability;
• There are no statements of requirement for information-enabled use cases;
• There is no inventory of data holdings;
• There is no Business Intelligence architecture defined;
• There are no plans at present to buy any ETL or BI end-user tools;
• The purchase of Product Z was made without reference to any formal solution selection process (e.g. due diligence evaluation of options in the market, price-testing to see whether other vendors would be more cost-effective, or proof-of-concept to validate that the product actually works).

Does Mr. X’s decision to buy Product Z seem so sensible to you now?! Effectively, at this point in time Agency Y has a 900kg, multi-million dollar paperweight sitting in its data centre.

I was discussing this scenario with my distinguished colleague Ben Bor today and I had started from the perspective that Mr. X is clearly a blithering idiot (like most of the Senior Execs I come across…). However, Ben posed a very interesting question, viz:

What if Mr. X is actually a genius?

Wow! That made me think!

Could it possibly be that the executive decision-making process is such that it is beyond our reckoning? What if this seemingly nonsensical waste of money is actually part of grand plan that we just don’t have the tools to comprehend? Given that Mr. X is at CxO level and we’re not (and probably never likely to be), maybe CxO’s have decision-making insights that transcend the normal logical approach that we mere mortals apply in these scenarios? Is Mr. X (and other executive decision-makers) actually a Savant?

Or is making a multi-million dollar purchasing decision with no evidence, no substantiation, no process and no currently identifiable requirements just really, really, REALLY stupid?

You, dear reader, can decide. I clearly don't have what it takes to understand such things.

BREAKFAST BRIEFING 05/09: Policy, Process & Culture in the lifecycle of an Information Asset

I will be running an interactive breakfast briefing on the topic of "Policy, Process and Culture in the Lifecycle of an Information Asset", to be held on Tuesday 05/09 at 8am, at the Boat House in Canberra.

For further details and to register, please click here: http://www.smsmt.com/About/Events/ACT-Information-Management

Revolutionising the Data Warehouse & Business Analytics: Top-10 Takeaways

I was delighted to chair Ark Group’s inaugural “Revolutionising the Data Warehouse and Business Analytics” conference this week.

The event featured participants from a diverse range of organisations such as Telstra, Australia Post, Asciano, Deloitte, PBT Group, Macquarie University, Uniting Care Heath, University of Technology Sydney and the Australian Sport Commission. Their though-provoking content provided stimulus for a highly interactive forum with some great debate.

A wide range of topics and themes were touched upon during the event, however some concepts were cropping up consistently throughout the various sessions. I’ve captured my own personal “Top 10 Takeaways” from the conference (not in any particular order):

1. Think about “Data Warehouse” as a set of services, not a single monolithic IT system. There needs to be a mind-shift away from the technology-centric view of a highly controlled Enterprise Data Warehouse.

Instead, think of the “warehouse” more as a set of organisational capabilities or services that facilitate information delivery and context-based analytics at point-of-need. The important aspect is not the technology, it’s the usage (and the action that is taken as a result).

A further enabler of this way of thinking is to adopt Agile methods within the data warehouse and analytic environment. Deliver to the requirements that achieve visible business value and accept that ongoing iteration and rework are simply a business-as-usual cost of doing effective analytic business.

2. Apply the principle of “Data First”. The organisation needs to understand the inventory of what data is being captured (or could be captured) in order to start exploring what that information could then be used for. Where possible and within reason, source all of the data that you can, even if you don’t have a specific business requirement in mind.

This also will help to facilitate the “second-sight” required to be ready to meet future emerging business requirements. For example, Google have adopted a philosophy of “throw nothing away” (right down to each mouse-click event and change of font colour on a web page), on the basis that they may be able to derive additional insight in the future that has not even been envisioned yet.

As a starting point, conduct a comprehensive audit of the organisation’s data holdings, so that you understand the information content that is available and can start to align it with the business context(s) that it can be applied to.

3. Ground everything you do in business requirements. Don’t start with the technology and then go looking for a problem to apply it to. Find the business requirement and then apply technology as a tool for enabling the solutions.

This also helps as a principle to guide your relationships with the technology vendors, who’ll happy try to sell you products as the solution to your requirements, even when you haven’t actually defined the problem yet!
(Note that this learning point is complementary to the “Data First” principle, not in conflict with it.)

4. Big Data: there are five “Vs”, not three. In addition to the now ubiquitous Big Data dimensions of “Volume”, “Variety” and “Velocity”, we should also be thinking about “Variability” and “Value”, and the most important considerations are these last two.

Complexity of the data landscape means that we’re not dealing with “structured” or “unstructured” information, we’re dealing with a diverse range of multi-structured data sets.

Meantime, if we don’t understand the context of the information that we’re dealing with and have mechanisms to navigate the impact on successful outcomes, then we need to question whether the activity is worthwhile.

Look for tangible value and ensure that you can explicitly identify and communicate sufficient value to justify the investment (accepting that there will be areas of benefit that may be less quantifiable.)

5. Information Governance is crucial. There needs to be an explicit information value chain that identifies what the decision rights and business rules are, and who is accountable.

There needs to be formal and visible measurement, with control points that then hold people to account. Successful organisations are investing in data quality as a human issue with process and cultural implications, not just technical ones.

6. The success of any data warehouse initiative is directly related to the level of Change Management capability. Transformational impact is achieved when data warehouse and analytics are rolled out pervasively, paying full attention to the human aspects of the change that arises.

This has to be done as a conscious discipline by members of the team who are practiced in Change Management techniques. Organisations will derive more benefit from a relatively small-scale solution that is well socialised and accepted as part of day-to-day business operations, than from a technically excellent analytic warehouse solution with no engagement.

7. A measure of information’s potential value is its likelihood of having unintended consequences. Think about the impact that knowing a particular piece of information could have. What unintended consequences could it lead to if that information is made available to the wrong person or used in the wrong way? If there are far-reaching consequences, then the information probably has significant intrinsic value.

e.g. Knowing someone’s favourite colour probably has little impact on them regardless of who that information was share with, however knowing their political allegiances could be potentially damaging if made known out of context.

8. There is still a lot of “biological ETL” going on. Human beings are still spending a lot of their time mashing up data in spreadsheets, re-keying data from one system to another, collating multiple reports to then derive a subsequent calculation.

Every function that can be rationalised or automated frees up more time for people to think and act.

9. Engage. Visible, active and regular communication is crucial. Share ideas, collaborate, ask awkward questions and expect them to be answered.

Use intellectual curiosity and sceptical scrutiny as tools to drive better thinking.

10. The future is the Cloud and Open-source, and it’s already here. ICT infrastructure is moving to the Cloud at an accelerating rate and organisations should expect (indeed demand) their data warehouse and analytics solutions to operate on Cloud-based platforms.

Open-source solutions are changing the game to the point where it becomes questionable whether costly, proprietary technologies are required any more. NOSQL will become an increasingly pervasive element of the analytic environment.

We all need to adapt our thinking to maximise the benefits of transitioning to Cloud and manage the legislative and regulatory implications (e.g. with respect to Australian Privacy legislation), but Cloud not going to be negotiable.

Challenges for the Information-Enabled Organisation (pre-conference article)

In advance of this month's "Revolutionising the Data Warehouse and Business Analytics" event, Ark Group are running a series of articles that address some of the topics that will arise at the conference.

My contribution, "Challenges for the Information-Enabled Organisation", is published here: http://www.arkgroupaustralia.com.au/News-DWnewperspect.htm

Other associated articles worth reading:

"Adding Social Customer Analytics to the Data Warehouse"  http://www.arkgroupaustralia.com.au/News-socialcustanalytics.htm
Top-10 tips for Data Warehouse Developers" http://www.enterprise-advocate.com/2012/04/top-10-tips-for-data-warehouse-developers/ 
"Big Data Analytics: 2012 predictions" http://tdwi.org/blogs/philip-russom/2012/01/big-data-analytics-2012-new-years-predictions.aspx

Further details for the conference can be found here:
http://www.arkgroupaustralia.com.au/Events-E024datarev.htm
Event entry on LinkedIn

Event Notification 23/24 August: "Revolutionising the Data Warehouse & Business Analytics", hosted by Ark Group

I am delighted to be chairing the forthcoming Ark Group conference "Revolutionising the Data Warehouse & Business Analytics", to take place in Melbourne on 23rd & 24th August.

The conference agenda will explore the following challenges:

• Demonstrating the tangible benefits and ROI of data warehousing
• Effectively managing Big Data
• Overcoming the barriers of to move to agile data ware house
• Maturing organisation’s data quality capability
• Bringing IT costs into control by effectively using Cloud
• New technologies and trends in business analytics

For more details, conference brochure and registration, please click here: http://www.arkgroupaustralia.com.au/Events-E024datarev.htm

Information as a Service, Part 2: what do I do about it?

In an earlier post, I defined the concept of “Information as a Service”, and covered off why successful Information Management requires a change of mindset that emphasises the wider context of why information is required, and for whom. I’d like to now like to turn attention to the impact that Information as a Service can have and some hints-and-tips on how to achieve better outcomes.

IT departments have traditionally struggled to engage with an information-aligned business agenda. The rigours and constraints of IT delivery are focussed on managing the technology infrastructure and applications that store and distribute data (containers and connectors), rather than having linkage to the relevance of the contents used in context. This focus typically requires an approach that is oriented towards policing of controls, process compliance and gatekeeping of expenditure.


In contrast to a technology-centric mindset, an information-oriented approach requires a different and complementary set of skills. Most importantly, it requires a fundamental change of mindset for the information team (whether for data warehousing, business intelligence, analytics or Documents & Records Management). To be able to encapsulate the information content within its business context, the team needs to articulate the relevance and impact that the organisation’s information has on business performance and process effectiveness. This in turn creates a clear line-of-sight from business value, through analytic information services to the underlying data warehousing of information assets focuses the people, processes, and technology towards the optimal management of information.

It becomes clear that without a critical mass of these competencies, organisations will struggle to balance investment in information with the information requirements of the organisation. Having a data warehouse and analytic environment that is fully aligned to the business view of the organisation ensures that management of the Information competency is fully congruent with the required business capability.

Data Warehousing and Analytics programs can deliver value to organisations at any strategic pain-point.  However, each initiative must be matched to the performance management strategies appropriate to your organisation’s business model. This process will drive the right investment in the data warehouse and associated analytic outputs.

A “top 10” of actions that help to develop better engagement and alignment with the “Information As a Service” mindset might include:

1. Managing business performance through visible measurement: this requires explicit investments in executive decision-making via executive dashboards which report directly from operational and management information systems. A strong link is created between reporting processes and actual business performance, setting the foundation for analytics-driven performance optimisation. 
2. Using the information that is currently available to drive new information capture: focusing data warehousing initiatives on making decisions with the best existing information focuses attention on understanding what additional information is required information but is currently not available. This supports proactive management of decision risks through highlighting the gaps to drive prioritisation of new data sources for data warehousing initiatives.
3. Driving commonality of analytic solutions and services: optimising the information delivery process by consolidating and standardising the inventory of common management and operational reports. A coherent approach that co-ordinates across all system enhancement, performance improvement, and transformation initiatives means building an integrated suite of reports that delivers repeatable elements common to overall business performance. Resources can then be re-focussed towards specific value-adding tasks, rather than maintaining multiple, duplicated copies of basic reporting services. 
4. Completeness of data integration: managing a continuous, prioritised pipeline of data integration for all organisational data to ensure the data warehouse suite supports both current and future demands. Data integration requirements should be driven from strategic drivers, proactively acquiring data for future analytic initiatives to enhance responsiveness.
5. Explicit management of ad hoc reporting and analysis: Implementing a Centre of Excellence / Competency centre that is formally accountable for managing a tiered community of users from information consumers, information integrators, statistical analysts, to decision makers
6. Metadata Driven Governance and Delivery: driving scope management, governance and delivery through a co-ordinated approach to metadata management. This ensures the requirements quickly find their way into working solutions and are validated by the business early in the project lifecycle. Using a Business Glossary and metadata-centric Agile delivery methods shifts the focus to the organisation information needs instead of the technology used to deliver the information.
7. “Steel thread” to manage data warehouse delivery risk: When developing business analytic solutions, the focus of planning and management effort should be on business requirements and objectives.  However, there is also a hidden risk in all projects - the unbounded effort in the development work stream caused by detailed technical unknowns. The concept of a “steel thread” maps the end-to-end linkages and dependencies from deploying the underlying warehouse technical platform, through data and analytics application development and on to the final business outcome, based on a tightly bounded sub-set of the overall business requirements. Potential issues in the technical environment, in the use of new technologies, or in the understanding of business logic are identified, tested and mitigated early in the delivery cycle, so that the overall solution implementation is de-risked.
8. Proactive Data Quality Management: assessing overall Data Quality for data sources and implementing appropriate controls and remedial actions to ensure Data Quality is managed in a proactive manner, both during programme execution and beyond into business-as-usual operations. Measurement and profiling, root-cause diagnosis, remedial action and continuous improvement are all necessary elements of a proactive approach to Data Quality. (The linkages between data and its usage in context are explicitly identified, as data quality is only ever quantified as a function of its usage).
9. Formal Data Governance: The people and functions who produce and use information are the people who know its value, understand what they need to save and should know how long a given set of data is going to be useful.  And while business people may know these things, it is often difficult – or even impossible – to get them to articulate their own information needs. Additionally, many departments only give consideration to their own information needs and opportunities for re-use, combination and added value are often missed. A robust and sustainable Data Governance regime provides a common foundation of “Rules of Engagement” and identifies explicit decision rights and accountability for the analytic business mandate. 
10. Delivering analytic solutions that are fun to use: people are more likely to make use of tools and services that are not only easy to use but are also attractive and engaging. Application of user experience (UX) disciplines and data visualisation techniques ensures that information is presented pleasingly and effectively for the intended decision making process.

Aligning data warehouse and analytic services with business outcomes requires a conscious focus and effort, either as an initiative in its own right or as part a holistic approach to implementing Information Governance within an information-enabled business transformation.  Once the overall scope and context of the implementation has been established (e.g. as part of defining the organisation’s Information Management Strategy and Roadmap), the aim is to stand up the new analytic competency as quickly as possible, so that your organisation can begin to realise the benefits of the transformed data warehouse capability.