"What's your point, caller?!"

Think about Information Use Cases, not technology.

I really enjoyed Leena Rao's recent article on TechCrunch, "Why We Need to Kill "Big Data"", not least because it reflected pretty much how I've been feeling for a long time about the lazy, misleading use of jargon and never-ending hype cycle as the tech vendors jump upon the latest bandwagon. (Pick one, any one, because they're all at it. Yes, I'm looking at you, IBM, Oracle, Microsoft, Teradata, SAP...) 

Leena's perspective also provides a healthly counterpoint to the views currently being positioned by IDC (as reported in the Data Informed article "Now is the time to buy into Big Data"), which seems to offer a very tech-centric point of view and plays right into the hype cycle without actually saying anything.

We've been here before  - Decision Support, OLAP, Management Information Systems, Performance Measurement, Business Intelligence, Master Data Management - all are catch-all term that actually had little if any real meaning in and of themselves. "Big Data" is just the latest entry in the Buzzword Bingo Lexicon.

In the twenty-odd years I've been involved in business solutions delivery and management consulting, it seems that *insert tool of choice* is promoted as the "next big thing to solve all your problems", without giving any thought whatsoever to what the actual problem, scenario or challenge actually might be. The technology is almost always the totally wrong entry point for the conversation, because technology only becomes of relevance when it is applied to solving a particular problem - and depending on the problem at hand, some technologies are more equal than others. 

In the Information Management space, almost all problems relate to the capture, exchange, dissemination, sharing, interpretation and acting upon one or more data sets. (Data Governance then sets out to ensure that these tasks can happen in a repeatable, consistent, efficient and effective manner). Which got me thinking. Are there generic categories of "Information Use Case" that we can use to describe various business problem scenarios, so that we can then start to make more informed choices about what sorts of technologies might be appropriate?

Or to put it another way: "what's your point, caller?!"

Anyway, here are some information-related problem situations that I could identify (you might think of others, or take issue with some of mine. I'd be delighted to hear from you if you do, because it means you've been thinking about the business problem, and not about the technology!):

Note that there is likely to be interaction (or even significant overlap) between these classes of use case. Each individual class of Use Case should be considered a necessary, but not sufficient, element of the organisation's Data Governance and Information Management capability.

Structured Data Reporting

Data is of little value if all it does is sit in the data warehouse. As a result, the presentation layer is of very high importance.

Most On-Line Analytic Processing (OLAP) vendors have a front-end presentation layer that allows users to call up pre-defined reports or create ad hoc reports. The aim is to synthesise large quantities of raw data into meaningful views that can be acted upon in context.

As such, reporting against structured data can be viewed as a specific type of authoring process; any reporting output is likely to be produced and submitted to the more general publishing process.

A number of key considerations need to be taken into account as part of the reporting capability:

• Number of reports: The higher the number of reports, the more likely it is that purchasing a pre-built vendor solution is the right approach. Reporting tools typically make creating new reports easier (by offering re-usable components) and also provide report management systems to make maintenance and support functions easier.

• Desired Report Distribution Mode(s): reports will only be distributed in a single mode (for example, email only, or over the browser only), or will users access the reports through a variety of different channels?

• Ad Hoc Report Creation: in most environments, it is expected that end- users will be able to create their own ad hoc reports. Ad hoc report creation necessarily relies on a strong metadata layer and shared understanding of what the information presented in the report is communicating.

• Data source connection capabilities: in most modern environments, users will need to access data sources using both relational database and OLAP multidimensional data technologies.

• Scheduling and distribution capabilities: in a realistic data usage scenario, senior executives will only have time to come in on Monday morning and look at the most important information from the previous week. To meet this need, the reporting tool must have scheduling and distribution capabilities. Weekly reports are scheduled to run on Monday morning, and the resulting reports are distributed to the senior executives either by email or web publishing.

• Security Features: reporting tools are geared towards a number of users in different Business Units and teams, with different priorities and responsibilities. Therefore, ensuring that people see only what they are supposed to see is important. Most reporting tools have capabilities to manage security at different levels, including at the report level, folder level, column level, row level, or even individual cell level. Furthermore, they have a security layer that can interact with the common corporate login protocols and "single sign-on" policies.

• Export capabilities: data export is commonly required for Excel, flat file, and PDF formats. It may also be desirable and time-saving to export the reporting format as well as the data itself.

• Integration with the Microsoft Office environment:
It is likely that reporting information will need to be incorporated into documents created with Microsoft Office products, especially Excel, both for manipulating data and for publishing. Some reporting tools now offer a Microsoft Office-like editing environment for users, so all formatting can be done within the reporting tool itself, with no need to export the report into Excel.

Strategic Intelligence and Data Mining

Data mining is the process of discovering new patterns and inferences in large data sets, involving a range of methods and techniques such as artificial intelligence, machine learning, statistics and database systems

The goal of data mining is to extract knowledge from a data set in a human-understandable structure and may involve a complex process of database and data management, data pre-processing, model and inference considerations, interestingness metrics, complexity considerations, post-processing of found structure, visualisation and online updating.

It is likely that a risk-based approach will need to incorporate information processing and data analytic features including:

• Anomaly detection:  Identification of outliers, changes and deviations in the data records that might be interesting or data errors and require further investigation.

• Association rule learning: searching for relationships and dependencies between variables.

• Clustering: discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data.

• Classification: identifying and applying a known, generalised structure or categorisation to new data. (For example, an email program might attempt to classify an email as legitimate or spam.)

• Regression: discovery of an approximation function that models the data with the least error.

• Summarisation:  to provide a more compact representation of a data set, including visualisation and report generation.

Publish Content to a Community

At a minimum, content has to be written and it has to be posted. Between those two steps, it is usually checked for its writing quality and correctness. Legal and compliance may need to review it. Ideally, any publishable material will be reviewed by a high-level editor or editorial board to make sure that it is consistent in style and fact with other information already in the published domain.

As with the searching process, the authoring process will need to be context-aware to ensure that information is defined and used appropriately, within both the context within which it was created, and the context of any intended (or unintended) usage.

Executive Briefings

The capacity to provide timely, compelling and concise advice to inform senior decision makers and executives is a vital capability for any organisation.

The Executive Briefing process therefore requires departments and business units to be able to locate, collate, and interpret the available information, such that the context and rationale for any decision can be supported and substantiated.

Education, Training and Learning

In simple terms, education provides a knowledge base that underpins any other activities the individual may engage in at a later stage. Training is not as general and tends to concentrate on skills development for the purposes of a specific skill or task. Learning tends to be associated with the self-developed of the individual.

Capability for education, training and learning of staff is a key aspect of service improvement in the University. In support of this, organisations need to provide an information sharing capability that enables all staff to access the process, policy and knowledge resources pertinent to their role.

Search for Content Previously Created

Many information users will be interested in finding material that has been authored by someone else in the organisation.

Assuming this content has been made available for others to access, the capability for finding, retrieving and accessing the required material may be many and varied, depending on a number of factors including: the nature of the content medium; the physical locations of the originator and consumer; the mechanisms available; other content that the consumer may wish to combine.

The nature of information content will also be dependent upon both the context within which it was created, and the context of the intended usage. Any search and retrieval process will need to be context-aware to ensure that information is used appropriately.

A technology-enabled approach to content search and retrieval will become increasingly important. However, it is also important to give due consideration to the governance authorities and control processes that define what content is to be made available.

Is should also be noted that any content search capability does not stand alone and needs to be fully integrated with content authoring and publication processes and systems. As such, the search process is likely to be implemented as part of integrating Records Management, Document Management and Knowledge Management solutions.

Records Management, Compliance and Audit

Records Management is the practice of maintaining the records of an organisation from the time they are created up to their eventual disposal. This may include classifying, storing, securing, and destruction (or in some cases, archival preservation) of records. A record can be either a tangible object or digital information, such as office documents, databases, application data, and e-mail.

The ISO 15489-1: 2001 Standard defines Records Management as "[the] field of management responsible for the efficient and systematic control of the creation, receipt, maintenance, use and disposition of records, including the processes for capturing and maintaining evidence of and information about business activities and transactions in the form of records". The standard defines “records” as "information created, received, and maintained as evidence and information by an organisation or person, in pursuance of legal obligations or in the transaction of business"

Records Management is primarily concerned with the evidence of an organisation's activities, and is usually applied according to the value of the records rather than their physical format. While there are many purposes of and benefits to records management, as both these definitions highlight, a key feature of records is their ability to serve as evidence of an event. Proper records management can help preserve this feature of records.

Freedom of Information & Privacy Responses

Many jurisdictions now make legislative provision based on the principle that government information is a public resource to be managed in the public interest. Such instruments give citizens the right to make requests to access Government documents. Similarly, where personal information is retained by an Agency, the individual has the right to request access to those records.

Data Quality 2013: Asia Pacific Congress

I'm delighted to be participating in the Ark Group's forthcoming Data Quality Asia Pacific Congress, to be held from 5-7 March at Citigate Central in Sydney.

The DQ Congress has been running annually since 2005 and features international keynote presenters and practitioner case studies, bringing attendees up to date information on such topics as: managing unstructured and complex data, master data management, effective data governance, new innovations and technologies; communicating about data quality.

As well as contributing to an "Ask the Experts" trouble-shooting panel, I will also be hosting a half-day interactive workshop, "Maximising value of your information assets: Creating and maintaining the Information Asset Register".

This workshop will equip participants with the understanding of key factors associated with a successful organisational approach to Information Asset Management, and focuses on practical methods and processes of building and maintaining the Information Asset Register as a building-block for maximising business value. Key topics include:

• What do we mean by “Information as an Asset”?
• What do I need to include in the Information Asset Register?
• Who needs to be accountable for information?
• How does the Information Asset Management process work?
• How do I measure the value of an Information Asset?
• What other Information Management best-practices do I need to be aware of? 

I hope to see you there!

Who is the Highlander?

Whenever we discuss the need for Data Governance, one area that rightly receives significant attention is to start explicitly identifying those stakeholders who need to be more formally accountable for the organisation’s data. The roles and responsibilities of “Data Owner” and “Data Steward” are debated, and Steering Committees, Information Boards and other collective groups are eventually formed which bring the participants together on an ongoing basis.

With a fair following wind, people will start talking to each other for a change and data will be the topic of conversation. If you’re really lucky, everyone might start playing nicely in the sandpit and you might actually get some positive action.

So far, so egalitarian. However, this collective, co-operative approach is not without its challenges. I’ve worked with several organisations that have started out on a path towards more formal Data Governance but have failed to get beyond the “talking shop” stage. They’ve established working parties to focus on data quality issues, they’ve agreed formal Data Principles that define the expectations, and some groups have even put in place monitoring, measurement and cleansing services to correct “bad” data. Yet somehow the overall result is that nothing much actually changes and that “Data Governance” is ultimately seen as delivering little real value. (If things go really badly, then behaviours of distrust, protectionism, and entrenchment across territorial boundaries can even become reinforced).

Not so when I was working on a major data and analytics project with my colleagues in Germany a while back.

The project had solid foundations. The overall benefits case was strong, the investment had been approved, there was clear focus on the data rather than the technology, we had trialed the solution within our UK business operations and there was already active and willing participation from the leaders of each business line. To this point, the initiative had progressed to investment approval stage through collaboration and consensus and I was expecting this to continue as the decision-making approach for the project proper. However, when I brought everyone together to confirm their roles within the collective Governance model, the project sponsor (Projektsführer) asked the room “Who is the Highlander?” (Imagine this spoken with a thick German accent for best effect…)

At first, I thought this was a Teutonic joke at my expense – as the token “Ausländer” on the project and a proud Scotsman, we’d already exchanged plenty of jokes about kilts, bagpipes, haggis and whisky. But my amusement (and slight bemusement) wasn’t shared by the rest of the group, who were clearly treating this as a serious question. What the heck was going on?!

The next part of the conversation went something like this:

Projektsführer: “Ach so. You have seen ze film, ‘Ze Highlander’ ja? Mit Christopher Lambert und Sean Connery?”

Me: “Ummm, yes….”

PF: “Zere can be only vun.”

Me: “Errrr. Pardon?”

PF: “There can be only one. In ze film, ze immortals all die until zere is only one left, und ze Highlander is ze one who gets all their power. So for our data project, who is the Highlander? Who is ze person that we all trust to make it work?”

And then the penny dropped. Somewhat bizarrely, my German colleagues had established a common metaphor for their Governance model, based on a poor-quality but highly entertaining science fiction from the 1980’s in which a French actor plays the Scottish hero and a world-famous Scottish actor plays a Egyptian swordsman working for the King of Spain. (http://en.wikipedia.org/wiki/Highlander_(film) and http://www.imdb.com/title/tt0091203/ )

By asking “Who is the Highlander”, everyone in the room (except me) knew implicitly what was expected – that whoever was assigned the role of leading the whole initiative would be granted the authority to act on everyone’s behalf, but would be ultimately answerable to the whole group. They would be need drive the whole process, facilitate and arbitrate, and have to ensure that everything came together and be held totally accountable for achieving a successful outcome. The German approach to Governance requires strong leadership from a committed individual, as well as active and willing participation from the wider stakeholder community.

And then they all turned to me and started laughing. Because for this project, the Highlander was, quite literally, going to be a Highlander…

Data Governance - lets make beautiful music together.

I've been in my new post as Director of Data Governance at the University of New South Wales for about a month now.  

Most of my time has been spent trying to get to meet people and hear their perspectives on how things work around campus (or don't), what the key issues are, and just getting to grips with the scale of the challenge in an organisation of approximately 5000 staff (Or 8000. Or 13,500. It kind of depends on which definition of "staff" you're using - but that's another story...)

Anyway, I had the opportunity last week to present to the University's IT investment committee, a senior executive group comprising faculty managers, academic leaders and financial governance. I was allocated 10 minutes to introduce myself and provide an update on the Data Governance agenda. Now, it's already clear that the University has an appetite to do something significant with respect to the usage and accountability for data (the fact that they've hired me is a statement of intent!). But let's be honest - there are going to be relatively few members of the university community who need to (or want to) get to grips with the wider scope, complexity and subtlety of and Enterprise-wide Data Governance and Information Management operating model! 

So I was struggling to think of a way to frame my agenda in a manner that was going to convey the scale of the task ahead without scaring everyone (or boring them rigid), and I came up with an analogy that seemed to resonate with the audience.

If we think of the University (or any other organisation for that matter) as an orchestra, then the various Faculties, Schools and Business Units are the musical sections, and the staff are the individual musicians. The organisation's data is the music to be played and the role of the Data Governance function is analogous to the conductor. 

Now, the outcome for any orchestra is to be able to give a concert to a paying audience (and get them to come back again…). The enabling capabilities which are required to get the concert on stage are a complex mix of contributions from numerous different people, applying multiple processes and skills, with the agreement of all participants to work towards the same goals:

1. All musicians need to have the skills to play their allocated instrument. (not necessarily a given!)
2. Someone has to write the music, orchestrate it, publish it, make sure the sheet music is brought to rehearsals.
3. We have an agreed running order for the concert.
4. Everyone needs to be playing the same tunes, at the same speed, and starting at the same time.
5. Rehearsals and practice (individually, in Sections, and together as an Orchestra).
6. Marketing the concert, booking the tickets, ushering the audience.
7. The performance.

THE CONCERT IS A FAILURE IF ANY PART OF THE PROCESS DOESN’T HAPPEN, AND IT WOULD STILL ALL BE LIKELY TO FAIL WITHOUT THE CONDUCTOR TO BRING IT ALL TOGETHER.

So Data Governance is about consciously and competently orchestrating and conducting all the various efforts throughout the organisation, so that whatever data we have (or need to have) is available, in context, and usable for a range of well-defined purposes.

Underpinning our data orchestra are an established set of formal practices, processes and protocols that define the institutional capability required, many of which may only exist in part, some not at all. Initially, we need to establish which of these services are needed, where are the gaps, and how they will be fulfilled. e.g.

Organisational Capabilities:

• Data Governance Steering Committee, focussing on Data/Information issues.
• Formal identification of Data Owners & Stewards for each data asset.
• Fora that bring together the community of Collectors and Consumers of each given information domain. 

People/Skills:

• Information Management/Business Intelligence Competency Centre
• Value-added analytic services and support the whole-of-institution approach.

Process:

• Business Glossary and metadata management techniques for collating and communicating common & consistent data definitions;
• Data Quality Profiling & remediation service;

Systems:

• Robust, scalable Data Warehouse & BI that is insulated from, and supports, change. 

Now, does everyone need to know or care about all the detailed practices & protocols, or how all the complexity happens? NO.

But I’d content that everyone does need to know about “their bit”, where the information touch-points are with other parts of the organisation, and that whatever their doing fits into the overall context.

To go back to the orchestra analogy, each individual musician doesn’t need to know how the composer and music publisher does their job, but they do need to know that the sheet music is going to be available, and that they’ve got to turn up at the Opera House on Tuesday week for a recital…

Which I guess makes yours truly the baton-wielding eccentric with the funny tailcoat suit and mad hairdo... 

CxO’s are blithering idiots! But what if they’re Idiots Savant…?

A client of mine (let’s call him “Mr. X”) who is the CIO at a government agency (let’s call then “Agency Y”) recently spent a fairly significant proportion of his organisation’s ICT budget for this year on a fairly large data warehouse appliance {let’s call it “Product Z”). Agency Y also got a 20% discount off list price for buying immediately.

“Nice work!” I hear you cry, “They’ve bought a market leader in the field which is constantly improving its technology, they’ve purchased at a discount, and if Agency Y is investing in Business Intelligence platforms, then they must be thinking progressively and wanting to drive more value from the information that’s available in the organisation. Splendid!”

Now, put that purchasing decision in the context of some additional information about the current situation at Agency Y:

• There are no current business projects identified that require BI capability;
• There are no statements of requirement for information-enabled use cases;
• There is no inventory of data holdings;
• There is no Business Intelligence architecture defined;
• There are no plans at present to buy any ETL or BI end-user tools;
• The purchase of Product Z was made without reference to any formal solution selection process (e.g. due diligence evaluation of options in the market, price-testing to see whether other vendors would be more cost-effective, or proof-of-concept to validate that the product actually works).

Does Mr. X’s decision to buy Product Z seem so sensible to you now?! Effectively, at this point in time Agency Y has a 900kg, multi-million dollar paperweight sitting in its data centre.

I was discussing this scenario with my distinguished colleague Ben Bor today and I had started from the perspective that Mr. X is clearly a blithering idiot (like most of the Senior Execs I come across…). However, Ben posed a very interesting question, viz:

What if Mr. X is actually a genius?

Wow! That made me think!

Could it possibly be that the executive decision-making process is such that it is beyond our reckoning? What if this seemingly nonsensical waste of money is actually part of grand plan that we just don’t have the tools to comprehend? Given that Mr. X is at CxO level and we’re not (and probably never likely to be), maybe CxO’s have decision-making insights that transcend the normal logical approach that we mere mortals apply in these scenarios? Is Mr. X (and other executive decision-makers) actually a Savant?

Or is making a multi-million dollar purchasing decision with no evidence, no substantiation, no process and no currently identifiable requirements just really, really, REALLY stupid?

You, dear reader, can decide. I clearly don't have what it takes to understand such things.

BREAKFAST BRIEFING 05/09: Policy, Process & Culture in the lifecycle of an Information Asset

I will be running an interactive breakfast briefing on the topic of "Policy, Process and Culture in the Lifecycle of an Information Asset", to be held on Tuesday 05/09 at 8am, at the Boat House in Canberra.

For further details and to register, please click here: http://www.smsmt.com/About/Events/ACT-Information-Management

Revolutionising the Data Warehouse & Business Analytics: Top-10 Takeaways

I was delighted to chair Ark Group’s inaugural “Revolutionising the Data Warehouse and Business Analytics” conference this week.

The event featured participants from a diverse range of organisations such as Telstra, Australia Post, Asciano, Deloitte, PBT Group, Macquarie University, Uniting Care Heath, University of Technology Sydney and the Australian Sport Commission. Their though-provoking content provided stimulus for a highly interactive forum with some great debate.

A wide range of topics and themes were touched upon during the event, however some concepts were cropping up consistently throughout the various sessions. I’ve captured my own personal “Top 10 Takeaways” from the conference (not in any particular order):

1. Think about “Data Warehouse” as a set of services, not a single monolithic IT system. There needs to be a mind-shift away from the technology-centric view of a highly controlled Enterprise Data Warehouse.

Instead, think of the “warehouse” more as a set of organisational capabilities or services that facilitate information delivery and context-based analytics at point-of-need. The important aspect is not the technology, it’s the usage (and the action that is taken as a result).

A further enabler of this way of thinking is to adopt Agile methods within the data warehouse and analytic environment. Deliver to the requirements that achieve visible business value and accept that ongoing iteration and rework are simply a business-as-usual cost of doing effective analytic business.

2. Apply the principle of “Data First”. The organisation needs to understand the inventory of what data is being captured (or could be captured) in order to start exploring what that information could then be used for. Where possible and within reason, source all of the data that you can, even if you don’t have a specific business requirement in mind.

This also will help to facilitate the “second-sight” required to be ready to meet future emerging business requirements. For example, Google have adopted a philosophy of “throw nothing away” (right down to each mouse-click event and change of font colour on a web page), on the basis that they may be able to derive additional insight in the future that has not even been envisioned yet.

As a starting point, conduct a comprehensive audit of the organisation’s data holdings, so that you understand the information content that is available and can start to align it with the business context(s) that it can be applied to.

3. Ground everything you do in business requirements. Don’t start with the technology and then go looking for a problem to apply it to. Find the business requirement and then apply technology as a tool for enabling the solutions.

This also helps as a principle to guide your relationships with the technology vendors, who’ll happy try to sell you products as the solution to your requirements, even when you haven’t actually defined the problem yet!
(Note that this learning point is complementary to the “Data First” principle, not in conflict with it.)

4. Big Data: there are five “Vs”, not three. In addition to the now ubiquitous Big Data dimensions of “Volume”, “Variety” and “Velocity”, we should also be thinking about “Variability” and “Value”, and the most important considerations are these last two.

Complexity of the data landscape means that we’re not dealing with “structured” or “unstructured” information, we’re dealing with a diverse range of multi-structured data sets.

Meantime, if we don’t understand the context of the information that we’re dealing with and have mechanisms to navigate the impact on successful outcomes, then we need to question whether the activity is worthwhile.

Look for tangible value and ensure that you can explicitly identify and communicate sufficient value to justify the investment (accepting that there will be areas of benefit that may be less quantifiable.)

5. Information Governance is crucial. There needs to be an explicit information value chain that identifies what the decision rights and business rules are, and who is accountable.

There needs to be formal and visible measurement, with control points that then hold people to account. Successful organisations are investing in data quality as a human issue with process and cultural implications, not just technical ones.

6. The success of any data warehouse initiative is directly related to the level of Change Management capability. Transformational impact is achieved when data warehouse and analytics are rolled out pervasively, paying full attention to the human aspects of the change that arises.

This has to be done as a conscious discipline by members of the team who are practiced in Change Management techniques. Organisations will derive more benefit from a relatively small-scale solution that is well socialised and accepted as part of day-to-day business operations, than from a technically excellent analytic warehouse solution with no engagement.

7. A measure of information’s potential value is its likelihood of having unintended consequences. Think about the impact that knowing a particular piece of information could have. What unintended consequences could it lead to if that information is made available to the wrong person or used in the wrong way? If there are far-reaching consequences, then the information probably has significant intrinsic value.

e.g. Knowing someone’s favourite colour probably has little impact on them regardless of who that information was share with, however knowing their political allegiances could be potentially damaging if made known out of context.

8. There is still a lot of “biological ETL” going on. Human beings are still spending a lot of their time mashing up data in spreadsheets, re-keying data from one system to another, collating multiple reports to then derive a subsequent calculation.

Every function that can be rationalised or automated frees up more time for people to think and act.

9. Engage. Visible, active and regular communication is crucial. Share ideas, collaborate, ask awkward questions and expect them to be answered.

Use intellectual curiosity and sceptical scrutiny as tools to drive better thinking.

10. The future is the Cloud and Open-source, and it’s already here. ICT infrastructure is moving to the Cloud at an accelerating rate and organisations should expect (indeed demand) their data warehouse and analytics solutions to operate on Cloud-based platforms.

Open-source solutions are changing the game to the point where it becomes questionable whether costly, proprietary technologies are required any more. NOSQL will become an increasingly pervasive element of the analytic environment.

We all need to adapt our thinking to maximise the benefits of transitioning to Cloud and manage the legislative and regulatory implications (e.g. with respect to Australian Privacy legislation), but Cloud not going to be negotiable.