Data-driven or jelly based decision culture?
In this post we explore these questions… Do you make decisions armed with quality data? Or do you rely on your “jellies” to inform the actions you decide to take? Are you at risk of trying to become a data-driven business and failing with all the cost and risk consequences that entails?
Simply put, if you want to be a data-driven business then you need to invest in embedding a sustainable data quality management process into your organisation. If you don’t, you will waste a lot of time and money on initiatives like ML/AI, RPA, Data warehouses, Lakes, Lakehouses, digital transformation and more that will ultimately fail, as you will have created a jelly monster masquearading as a data-driven culture. The worst of both worlds!
Every business is looking for answers to an ever increasing number of questions. How many customers do we have in this area? What is our best selling product? What is the most common customer complaint? What aspects of our service do our customers talk about the most? In a data-driven organisation, to get answers to questions like this you need consistently reliable, good quality sources of data and typically lots of it. In “Jelly decision” cultures, someone will make a decision based on experience, instinct and almost always with a much smaller amount of data. Essentially they will make a decision that is a greater risk to the successful outcome for the organisation based on the limited quality data available to them.
Some “jelly decisions” occasionally produce favourable outcomes however, many do not! Indeed, supporters of the “jelly decision” regularly point to their past successes as to why you should run the business on their “jellies” (instincts). Rarely if ever do they mention, the costly, risk crystalising howlers that they and others have inflicted on the business, brushed conveniently under the carpet in support of the populist “jelly” culture in days of yore.
Organisations on the journey to data-driven decision making are increasingly looking at business intelligence (BI), data analysis and data science (machine learning and AI) functions as the means to get answers to their burning questions. Sadly, unless they address the issue of sustainable organisational data quality management they are running the risk of catastrophic failure…
Why? Well, the reason is that incoming data is the raw fuel of the insight laden, data-driven organisation. To have any value, data needs to be first “curated”, i.e. It needs to be made available along with other data in a form that can be analysed and interrogated by the business, empowering them to get their questions answered, ideally by themselves. In short you can’t answer questions without data of good quality and provenance to use as the basis of your decision making. The essence of a data-driven culture!
The data curation (sometimes referred to as the munge, collect & prepare, ETL, ELT, transformation) process is perhaps one of the most time consuming and inefficient processes that are undertaken within the data-driven enterprise. You need to curate data to get answers to new questions, it is happening more and more in data-driven organisations as more curation is needed to answer more questions. In the world of data-driven businesses, it’s an essential process. It is not uncommon for data scientists, engineers or analysts to spend 80% of their valuable time doing just this. I’ll let you into a secret, ask your data curators if you don’t believe me, the primary reason for this inefficiency is that raw data is not being managed from a data quality perspective by the business, data quality has instead become the responsibility of the data curators to massage it into an approximation of the truth in an attempt to gain answers to questions. Poor quality or limited data generally leads to bad decisions and outcomes, the sort of outcomes “jelly decision” cultures attain!
Once you have curated your data, the concept of the data pipeline is typically introduced into the data-driven organisation’s armoury. A data pipeline is typically an automated process to get the data to the point where it can be analysed for answers to questions as business as usual. A data pipeline typically involves a number of steps to enable the business to get answers to common questions. You might for example have in your data pipeline an ELT feed into a data warehouse with data marts provisioned dynamically and dashboards refreshed with answers to common business questions. However, data pipelines can only be trusted if the quality of the data going into the pipe at source is managed. i.e. you must have a data quality management process in place, owned by the business to sustainably manage the quality of the data that is being used for data dependent initiatives and decision making. Otherwise you’re just pumping poor quality data to the recipient to make an ill informed decision.
If attempting to transform to a data-driven culture, you don’t put in place the means to sustainably manage and improve the quality of your data at source, then you will fail. You will be less efficient, less competitive, introduce more risk and not be able to get the answers to questions that your organisation needs to be successful. You will make data curation harder and fill your data pipelines with poor quality data. You will require more and more of the scarce data curation skills to be successful and you will have created a far worse situation for your business. Alas, it will now be running under the misaprehension of being data-driven when it has just spent a lot more time, resources and money to remain a jelly culture.
To discover more about how infoboss can help support your data quality and data protection initiatives in a data-driven business, then please get in touch.