Addressing the 85% AI Failure Rate: Why We Have Built an Open-Source Data Maturity Toolkit and Consortium
From theoretical frameworks to validated implementation pathways
In our previous exploration of Data Centres of Excellence (DCoE), we found ourselves increasingly drawn to a fundamental question: what transforms data from operational necessity into strategic intelligence? As we dug deeper into Decision Support Systems, Business Intelligence and Analytics Maturity within the DCoE framework, a persistent pattern emerged that we couldn’t ignore.
The more we explored how organisations might build systems that enable intelligent decision-making, the clearer it became that we’re consistently addressing symptoms rather than causes. We design sophisticated analytical frameworks, implement cutting-edge AI systems, and create elegant theoretical models, yet Gartner’s research shows that 85% of AI projects deliver erroneous outcomes, with 60% of new AI projects lacking AI-ready data expected to be abandoned by 2026.
This realisation led us to step back from theoretical exploration and ask a more fundamental question: what if the problem isn’t that we lack sophisticated tools, but that we’re building those tools on foundations designed for entirely different purposes?
The Evolution from DCoE Theory to Data Reality
Our work on Data Centres of Excellence had us examining how Decision Support Systems (DSS), Business Intelligence (BI) and Analytics Maturity might strengthen organisational frameworks. What became increasingly apparent, however, was that even the most sophisticated DSS, BI or Analytics systems cannot overcome fundamental issues in how data is collected, managed, and prepared for intelligent analysis.
We began to see that whilst we focus considerable attention on what happens to data once it reaches our analytical systems, we pay remarkably little attention to whether that data was ever designed for intelligent consumption in the first place. Most organisations treat data as an incidental by-product of their technology systems rather than engineering it as a strategic product designed for intelligent analysis, AI enablement and predictive analytics capabilities.
This insight proved transformative. If the foundation determines whether sophisticated analytical capabilities can deliver their promised value and robust AI enablement, then perhaps our focus on algorithmic sophistication and technical infrastructure was missing the most critical element: systematic data collection designed for intelligent analysis.
Building Beyond Theory
Rather than developing another theoretical framework to add to the existing collection, we found ourselves compelled to build something practical. Over the past twelve months, working with The Talent Intelligence Collective, we’ve developed what we believe represents a significant departure from traditional approaches: a comprehensive academically informed toolkit of standards and validation frameworks specifically designed to address the systematic challenges that undermine both AI enablement and Predictive People Analytics effectiveness.
As we progressed our academic paper, we made critical decisions about how to translate foundational research into practical methodology. This work builds on foundational research by Richard Wang and colleagues on Information Products, Yang Lee and Leo Pipino’s Total Data Quality Management (TDQM) approach, and Herbert Simon’s decision-theoretic frameworks. What we’ve created, however, isn’t simply an application of existing theory, it’s a systematic methodology that addresses the practical challenges we encountered in our DCoE explorations.
The Ten Root Conditions: From Pattern Recognition to Systematic Solutions
Drawing extensively from Lee and Pipino’s research framework, we have drawn on these ten conceptual conditions that frequently appear in organisational contexts and consistently affect both AI enablement and Predictive People Analytics effectiveness. As we developed our academic paper, these conditions became central to our understanding of why traditional approaches consistently fall short:
Multiple Data Sources as Conflicting Information
Subjective Judgement in Data Production
Resource Limitations Affecting Data Access
Security and Accessibility Balance Considerations
Diverse Coding Systems Across Functions
Complex Data Representation Challenges
Data Volume and Processing Relationships
Data Input Standards and User Behaviour Patterns
Evolving Information Requirements
System Integration and Information Architecture
What makes our approach different is that we’ve moved beyond Lee and Pipino’s original diagnostic framework to developing specific methodologies for addressing these conditions systematically. Our toolkit provides structured approaches that organisations can implement rather than simply understand. These decisions emerged naturally as we worked through the academic paper, recognising that identification without solution creates limited practical value.
Data Readiness Levels: A Practical Framework
Our exploration of DCoE frameworks led us to appreciate NASA’s Technology Readiness Levels for assessing technical maturity. This inspired our development of Data Readiness Levels (DRL), a systematic approach to evaluating and progressing data collection maturity for AI enablement and Predictive People Analytics. During our academic paper development, we realised that existing maturity models lacked the specificity needed for data collection assessment.
Building on Wang et al.’s Information Product principles, our research reveals that most organisations plateau at DRL 5-6, implementing sophisticated data collection techniques while continuing to treat data as a by-product of technology systems. The critical breakthrough occurs at DRL 7, where organisations implement Wang’s Information Product principles, deploy Data Product Manager (DPM) roles, and establish systematic quality management based on Lee and Pipino’s TDQM methodology that transforms data into strategic assets designed for intelligent consumption.
Focus on the Most Pressing Challenge: People Data
As we progressed through our research, it became clear that whilst data quality issues affect all domains, the challenge is particularly acute in people data. The complexity of human behaviour, subjective assessments, and fragmented systems creates the most pressing data quality issues across organisational contexts when it comes to AI enablement and Predictive People Analytics.
Most organisations combine and repurpose people data from disparate systems (time reporting, salary systems, sick leave tracking, ATS systems, payroll) alongside scraped data, or subjective self or manager reported assessments like skills profiles, personality assessments, feedback, and performance ratings. This approach treats people data as incidental by-products of HR technology systems rather than strategic assets designed for intelligent analysis of human capital.
The difference between DRL 6 and DRL 7 isn’t technological sophistication, it’s a fundamental shift in how data is conceptualised, collected, and managed, directly reflecting Wang et al.’s core insight about treating information as a strategic product rather than operational by-product. This is particularly critical for people data because the stakes are higher: poor quality people data doesn’t just impact analytical accuracy, it affects decisions about organisational capability and strategic workforce planning through AI enablement and Predictive People Analytics initiatives.
From Individual Insight to Collective Action
What started as academic exploration of DCoE possibilities has evolved into something much larger. As we progressed through our research paper, we realised that the frameworks we were developing could benefit from collaborative validation rather than remaining purely theoretical. Rather than keeping our developed toolkit proprietary, we’ve established the Data Maturity Matters Consortium as an open-source initiative. This decision reflects our growing conviction that the challenges we’re addressing, particularly in people data for AI enablement and Predictive People Analytics, are too significant and too widespread for any single organisation to solve independently.
The consortium brings together solution providers, academic researchers, policy makers, and organisations ready to validate systematic improvements. Our validation methodology, which crystallised during our academic paper development, centres on six-week implementation protocols where participants work together to test our established solutions, measuring results against specific DRL progression criteria informed by Herbert Simon’s decision-theoretic frameworks.
Validation Through Collaborative Implementation
What excites us most about this initiative is its departure from traditional consulting or academic approaches. Instead of developing frameworks that look impressive in presentations but prove difficult to implement, especially in the complex reality of people data management for AI enablement and Predictive People Analytics, we’re validating practical solutions through collaborative testing with organisations facing real challenges. This methodology emerged as we recognised during our academic paper development that theoretical frameworks require validation through practical implementation to demonstrate their effectiveness.
This represents a fundamental shift from individual expertise to collective validation where we’re testing whether our systematic approaches, grounded in the work of Wang, Lee, Pipino, and Simon, actually work when implemented in diverse organisational contexts, particularly in the challenging domain of people data for AI enablement and Predictive People Analytics.
Building Transparent Foundations for Competitive Advantage
The results belong to the entire consortium, creating an open-source collective that provides transparent foundations upon which organisations can build genuine competitive advantage. Rather than relying on black box vendor solutions where the methodology remains opaque, our approach enables organisations to bring critical knowledge in-house, building internal capabilities in collaboration with the specific expertise they need.
This represents a new way of disrupting the traditional market of vendor-led proprietary technology stacks. Instead of organisations remaining dependent on external systems they cannot fully understand or control, the consortium model enables them to develop sophisticated internal capabilities while leveraging collective expertise. The transparency of open-source methodologies means organisations understand exactly how their data maturity solutions work, allowing them to adapt and evolve these approaches as their needs change.
This collaborative approach transforms the traditional vendor-client relationship into genuine partnership, where organisations contribute to and benefit from shared knowledge development while maintaining the flexibility to build solutions that precisely match their unique contexts and requirements for AI enablement and Predictive People Analytics success.
Looking Forward
Our journey from exploring DCoE frameworks to building practical data maturity solutions has reinforced our conviction that sustainable improvement requires systematic approaches rather than ad-hoc interventions. The 85% AI failure rate isn’t a technology problem, it’s a foundation problem that requires fundamental changes in how we approach data collection and management, precisely as Wang et al. predicted in their foundational Information Product research. This is especially critical in people data, where organisations face the most complex data quality challenges for AI enablement and Predictive People Analytics.
Through the Data Maturity Matters Consortium, we’re moving beyond theoretical possibilities to validated practical solutions. What began as academic exploration of how organisations might build intelligent decision-making capabilities has become collaborative development of the data foundations, particularly for people data, that make robust AI enablement and Predictive People Analytics capabilities and their associated ROI possible.
For those interested in validating our toolkit through consortium participation: www.datamaturitymatters.tech