April 1, 2023, by Paola Reyes
In the last decade, we have seen an increase in the collection of data from businesses, governments, websites, social media, and others. This increment has awakened my curiosity to explore and create projects to understand, predict, and gain insight into the collected data. However, have most of the big data projects succeeded? And why do big data projects fail?
The answer to the first question is no. And second, there is a list of factors that impact the success of a project, which are lack of knowledge, well-defined objectives, infrastructure, skills of the analysts, communication skills as well as security and privacy. I will detail the explanation of these factors in the following paragraphs.
Before exploring the causes of big data project failures, let us check some of the statistics about them. Do you have trouble believing that 87% of data science projects never reach production? Moreover, 85% of big data projects fail (DataSciencePM, 2020). I could not believe it when I read about this, especially because those higher percentages are most likely made up of a big amount of money and people’s time lost during the process of the projects. So, what are the factors that contributed to these high percentages?
Some of the causes of big data project failures are the lack of knowledge of the business problem and, subsequently, the absence of well-defined objectives for the project.
If there is not enough information about the problem that we are trying to solve, the creation and accomplishment of the objectives will be hard to achieve. Another cause is not having good data quality. Imagine a group of analysts that are trying to identify what products will be the most popular for the winter season, but they are analyzing and creating predicted models based on data sales from summer. They will be finding products such as swimwear and beach towels instead of scarf or slippers.
The infrastructure and the skills of the analyst take an important role in the success of a project. As we know, big data is synonymous with large amounts of information that cannot be handled by conventional software and storage, the reason why it is important to select the right infrastructure for storage and developing the analysis. As well as the expertise of the team is crucial in the project. If they lack knowledge and the required experience, it will be difficult to develop and finalize the project.
Finally, communication and security and privacy concerns. If the communication is poor between the client/boss and the analyst team, there will be misunderstandings and delays that might lead to the project’s failure. And, if privacy and security are not sufficiently handled, the firm or analyst team may be exposed to legal liability, which would lead to the failure of the project and cause extra reputational harm.
An example of a failed big data project is the Target data breach in 2013, where the store analyzed the customer’s behavior to enhance marketing and sales. However, Millions of client credit card details were stolen because of the project's inability to detect a security vulnerability in its system (Computerworld, 2014).
This project was part of a group of big data projects that fail because of weak security and privacy protocols, due to the poor skills of the analyst team to determine how to manage and protect the customer’s data. Another factor is an inability to recognize and address suspicious activity in the data. As a consequence, Target was exposed to financial losses as well as legal and reputational harm.
As I covered, the success of a big data project is based on a group of factors working together to finalize the project. The lack of some of them can drastically affect the future of the analysis. Therefore, it is important to have a clear understanding of the business problem to create the right objectives, have concerns and protocols to protect the data, and ensure that the infrastructure, and skills of expertise, are met. As well as having good channels of communication before starting the project and while developing it, to reduce the chances of failure.
References
Computerworld. (2014, February 12). Target breach happened because of a basic network segmentation error. Retrieved from https://www.computerworld.com/article/2487425/target-breach-happened-because-of-a-basic-network-segmentation-error.html
DataSciencePM. (2020, June 15). Why do big data projects fail? Retrieved from https://www.datascience-pm.com/project-failures/
Comments