A 2024 research study on artificial intelligence (AI) from the RAND Corporation estimated that “more than 80 percent of AI projects fail — twice the rate of failure for information technology projects that do not involve AI.”[1] The root causes of failure include miscommunicating or misunderstanding the problem to be addressed, lack of data to train AI models, focus on technology instead of the problem, inadequate infrastructure to support AI operations, and problems that are too difficult for AI to address. Two of the five causes of AI project failures were related to the problems being addressed and are preventable. Given the transformational potential that AI brings, this finding is alarming and should serve as a warning to business leaders and Chief AI Officers. If you don’t get the problem “right” from the start, you’re setting yourself up for failure before the first algorithm is even written.
In this article, we’ll discuss best practices for considering and selecting the “right” problems for successful AI projects, and provide actionable guidance for business leaders. This blog is part of a continuing series of articles aimed at providing senior leaders and managers with a practical working knowledge of artificial intelligence.
Best Practices
While innovative technologies like the Internet of Things (IoT) and artificial intelligence (AI) bring the potential to address old challenges in new ways, they cannot solve all problems. These technologies are merely tools, and it is important to use the right tools to address the right problems. From our years of experience using innovative and emerging technologies to address a variety of challenges, we’ve developed four best practices for sourcing, assessing and selecting problem sets.
Best Practice #1: Make sure you understand the problem, and specify it completely and clearly.
A successful AI project starts with a strong understanding of the problem to be solved. This may sound obvious, but in practice, it’s where many well-intentioned initiatives go wrong right away. AI can’t address a problem if it doesn’t know what the exact problem it is solving is. Well understood problems start with a well-specified problem statement. It must be specific and clearly describe the challenge to be addressed. This requires a deep understanding of the nature of the problem, who is impacted and in what ways. The problem statement must discuss the scope and boundaries of the problem, the symptoms, the type of data needed, what a successful outcome looks like, and key metrics for success. Furthermore, all “problems” must have an owner, someone who is impacted by the problem and is willing to commit resources and time to resolve it. A problem without an owner is not a problem that leaders and managers should waste resources and time on.
Best Practice #2: Know what types of problems AI can and cannot solve.
AI is not a magic wand that can fix everything. The truth is that AI is a good fit for only a certain subset of problems. Misunderstanding this reality is a common pitfall for otherwise promising AI projects. Business leaders must know what AI can and can’t do, and only apply AI when it is most appropriate to do so.
AI is good at (currently) |
Pattern recognition in large datasets |
Image and speech recognition, fraud detection in financial transactions, weather forecasting |
Data processing and analysis |
Market trend analysis, scientific research data processing, personalized recommendations |
|
Automating repetitive tasks |
Quality control in manufacturing, customer service chatbots, data entry and processing |
|
Finding optimal answers in complex situations |
Supply chain management, traffic flow optimization, resource allocations |
|
AI is not good at (currently) |
Creative thinking and innovation |
Writing a completely original novel, or inventing a new scientific theory. [2] |
Emotional intelligence and empathy |
Mental health counseling, and complex customer support situations requiring empathy |
|
Contextual understanding and common-sense reasoning |
Language translation of idiomatic expressions, understanding sarcasm or humor and making judgments in ambiguous situations [3] |
|
Ethical decision-making |
Autonomous vehicles deciding between two harmful outcomes and AI-assisted judicial systems |
|
Situations outside of the scope data models were trained on |
Responding to a never-before-seen type of cyber-attack and addressing sudden, drastic changes in societal norms |
Even when AI is used in situations is it well-suited for, there are situations where the risks and harms to human safety, and society and community well-being are high. Human judgement is needed to complement AI to achieve the intended results. Examples include medical diagnoses and treatment, financial investments and risk management, criminal justice and law enforcement and content moderation.
Best Practice #3: Match the right AI method to the problem being addressed.
AI is not one technology, but many different types of technologies. Each type of AI has its own capabilities. Just as you wouldn’t use a screwdriver to hammer a nail, you shouldn’t use the wrong type of AI to solve a problem. Different AI technologies excel at different tasks. Machine learning is well suited for analyzing statistics from vast datasets to identify patterns and relationships. In contrast, expert systems are used in structured environments with clear rules, such as compliance checks, medical diagnosis, or troubleshooting technical issues. A discussion of the different AI types and the types of tasks they are well-suited for is discussed in a previous article here.
Success hinges on aligning the problem with the most appropriate AI approach. If you attempt to use generative AI to predict customer churn—a task better suited for traditional ML—you’re likely to fail. In other cases, using the wrong AI type may mean you are using a more complex and expensive approach for something that a simpler technology could have accomplished. By aligning the problem to the right AI type, you significantly improve the chances of creating successful AI outcomes.
Best Practice #4: Don’t apply AI to problems where you cannot address poor data quality, or where the quantity of available data and access to the data is limited.
Data is the lifeblood of AI, and many initiatives stumble because they underestimate the challenges of sourcing, cleaning, and maintaining it. AI can’t solve a problem if there is no data or if the data quality is poor. For example, in cases involving rare medical conditions, there may not be enough medical data to train AI models to reliably and accurately detect these diseases. Similarly, niche industrial processes may have limited data available to train models to improve operational efficiency.
Another critical issue is the quality and relevance of the data that is available. The data for AI projects must correctly represent the real-world phenomena it’s supposed to describe, with minimal errors or inconsistencies. The data should also be comprehensive, covering a wide range of relevant scenarios and edge cases. Diversity in the data is essential to ensure the AI model can generalize well and avoid biases. Additionally, good AI data is up-to-date and relevant to the current context in which the AI will operate. Low-quality data can lead to models that produce inaccurate or harmful outputs. Additionally, the process of labeling data for supervised learning is often labor-intensive and costly, creating bottlenecks in data preparation that can delay AI development. [4]
In these situations, the problem, no matter how well defined it is and well-suited for AI, cannot be addressed adequately. Even with the right problem and method, your AI project will falter without quality data.
Actionable Steps for Business Leaders
With an understanding of these best practices, we share some basic steps to enable business leaders to enhance their ability to bring well-defined problems for AI to solve. These steps not only improves the likelihood of successful AI projects but also fosters a culture of continuous improvement within the organization. Business leaders and Chief AI Officers should consider these actions:
Establish a process and a pipeline for finding qualified problems to be considered
Successful AI projects start with well-qualified “good” problems. However, finding good problems to draw from consistently doesn’t just happen. Business leaders have to establish a process that identifies, sources, develops and qualifies problems to be considered. This builds a pool of qualified “problems” that leaders draw from to consider for AI project candidates. This entails:
- Building a systematic problem collection and curation capability across the organization. Conduct surveys or interviews periodically to understand internal and external stakeholder pain points and what problems they genuinely care about solving. Implement intake systems that allow employees, partners and customers to provide input and suggestions. Work with teams to identify high-impact pain points, particularly those with strong data trails.
- Clearly Define Problems. Use structured methods like the “5 Whys” or workshops with diverse stakeholders to articulate and refine problems. Document problems using a standardized problem statement template so that it follows a consistent format to facilitate review and consideration.
- Rank Problems by Importance and Feasibility. Regularly evaluate and score problems based on their relevance to business goals, potential ROI, and feasibility. Focus on addressing high-priority issues and those that have business or organization owners who will commit time and resources, and are aligned with organizational goals.
Establish a process to vet and select “good problems”
Bringing “good problems” to AI projects is facilitated by drawing from a pool of well-qualified problem sets. But not all well-qualified problems are suitable for AI projects. Business leaders need to create a process and criteria to evaluate which of the collected problems are “AI suitable”. Some considerations may include:
- Is this problem within the capabilities of AI to address, and can the “right” type of AI be identified?
- Is there sufficient, high-quality data for AI to address the problem?
- How strategic is it to address this problem (relative to other candidate problems being considered)?
- What is the potential return on investment (relative to other candidate problems being considered)?
- What are possible risks and complicating factors that may hinder the resolution of the problem?
Set realistic expectations for AI across the organization
Business organizations in the age of AI must be AI-savvy. Leaders and managers need to understand what AI can and cannot realistically do. This sets expectations and minimizes pairing “poor” problems with AI, which ultimately wastes resources, creates suboptimal outcomes and lead to poor perceptions of AI. Leaders should consider the following activities:
- Provide training on AI types, capabilities, and limitations. This builds realistic expectations and aids in selecting suitable problems.
- Update training periodically to keep pace with rapid advancements in fields like generative AI.
- Develop a Problem-AI Mapping framework to align challenges with the right AI techniques.
Establish a process and pipeline of AI-ready data for projects
Manufacturing companies need raw materials to build products. Similarly, businesses need data for AI to solve problems and create outcomes. Data availability is a critical enabler for AI project success. However, having the “right” data is not enough. The data must be also be in a state and format that is usable by the AI for model development and training. Managers and AI leaders should:
- Conduct data audits to evaluate availability, quality, relevance, ownership, privacy considerations, and usage restrictions for potential AI projects.
- Identify and address gaps in data collection or quality before starting projects. Develop approaches to address the gaps, such as the use synthetic data in cases where data or access is limited, or secure access agreements to external data sources.
- Develop processes and capabilities to prepare data for use by AI. This includes data ingestion, transformation, cleaning and preparation, and storage.
- Review, develop and implement governance policies to maintain and ensure consistent access to high-quality and current data for training and operational use.
Conclusion
The allure of AI is undeniable, but its success depends on solving the right problems. By clearly defining the problem, understanding what AI can and cannot do, matching the right method to the task, and ensuring quality data, you can drastically improve the odds of success.
As a business leader, your role is to steer your team toward the right challenges and ensure they have the resources to succeed. Remember, the question isn’t whether AI can solve a problem but whether it should. By following these best practices and actionable steps, you can turn AI into a strategic capability rather than a costly experiment. Successful AI projects don’t happen by chance—they happen by design.
References
[1] J. Ryseff, B. De Bruhl, and S. Newberry. “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed,” Research Report, Rand Corporation. August 13, 2024. Link.
[2] Generative AI may create hallucinations. AI hallucinations are instances where generative AI models produce false, misleading, or nonsensical information while presenting it as factual. This phenomenon occurs when large language models (LLMs) perceive patterns or objects that don’t exist, resulting in inaccurate or illogical outputs.
[3] While advancements in agentic AI technologies may enable some systems to make judgements in certain situations, this capability is still in early stages of maturity.
[4] “Data Scarcity: When Will AI Hit a Wall?”, M. Lindeman, Pieces for Developers, June 17, 2024. Link.
This article was co-authored by Benson Chan of Strategy of Things and Peter Williams of Peter Williams Consulting. This is a second in a series of blogs aimed at providing senior leaders and managers in mid-market organizations with a practical working knowledge of artificial intelligence (AI). If there are specific topics you wish for us to address in the future, please comment below.
Thanks for reading this post. If you found this post useful, please share it with your network. Please subscribe to our newsletter and be notified of new blog articles we will be posting. You can also follow us on Twitter (@strategythings), LinkedIn or Facebook.