Multiplicity of compute options, failed POCs, budget woes and governance nightmares make for a tough drive!
I remember discussing Amazon’s secret 2018 trial of an AI-based recruitment tool with a group of tech journalist friends. It ultimately served as a warning sign about the risks of relying entirely on technology. Amazon developed the tool to streamline hiring but systematically discriminated against female candidates. This bias arose because the AI was trained on historical hiring data from Amazon’s existing tech workforce, which is predominantly male.
Other companies faced similar problems with their integrated AI models back then. IBM Watson for Oncology gave incorrect treatment advice. Google Photos had issues with racial bias in its image recognition. Tesla’s self-driving technology raised safety concerns. Of course, those were the early days of AI. At the time, these AI projects excited us, but their rapid failures made us question whether AI could deliver results in the short term and what it would take to ensure its success. Many organizations focus on automating their processes for exceptional user experience and productivity. Still, these experiences served as a reminder that AI requires continuous bias checks, diverse training data, and strong human oversight to be genuinely effective.
While AI has made considerable progress since then, with generative AI and agentic AI opening up significant opportunities for productivity and innovation, not all those challenges have been addressed.
For instance, McDonald’s recently ended its much-publicized AI voice ordering system at more than 100 drive-thru locations in the U.S. However, this service, introduced to improve customer experience and reduce wait times, could not meet the objectives. Instead, it frustrated McDonald’s customers and pushed
them to mock the initiative as AI could not understand their orders.
Today, many organizations strategically deploy AI to transform key business functions, from personalized product recommendations and enhanced customer service to optimizing complex supply chain operations. However, for many enterprises, the biggest challenge remains: scaling AI implementations while identifying unique use cases and achieving a clear return on investment. Our discussions with technology leaders reveal that over 60% of CIOs’ current AI investments are driven more by competitive anxiety—the fear of being left behind—than a well-defined strategic vision for technological transformation.
CIOs who have successfully deployed AI say that being excited about AI isn’t enough. AI success requires careful planning, a strong data foundation, and aligning AI initiatives with business goals. But IT leaders face tough questions:
◼ How can AI address specific organizational needs?
◼ How will AI models be governed?
◼ Can implementations scale from pilot to enterprise-wide?
◼ What measurable benefits can be achieved?
◼ What is the actual return on investment (ROI)?
Unclear expectations
Enterprises are dealing with a big headache, as over 70% of AI projects fail to deliver meaningful results. Research from RAND highlights the magnitude of this issue, demonstrating that AI initiatives fail at twice the rate of traditional IT projects, with more than 80% falling short of their intended objectives. According to the research firm Gartner, by the end of 2025, at least 30% of generative AI projects will be abandoned after proof of concept. This is due to poor data quality, inadequate risk controls, escalating costs, and unclear business value.
The root of these failures lies in a fundamental misalignment between technological potential and organizational expectations. CIOs face a critical challenge in translating AI’s promising capabilities into tangible business outcomes, struggling to justify substantial investments in productivity enhancement technologies. The complexity results from businesses’ inability to convert AI initiatives into financial benefits, coupled with organizations’ lack of clear objectives, insufficient understanding of AI limitations, and inadequate implementation strategies.
The key to success remains developing precise roadmaps, investing in data quality, fostering a culture of realistic innovation, and establishing robust measurement frameworks beyond traditional return on investment metrics.
Cost management and optimization
As enterprises scale their AI initiatives, managing infrastructure costs has become a critical focus for CIOs. According to Enterprise AI Pulse, a recent survey conducted by CIO&Leader based on responses from over 300 IT leaders, cost management is the primary concern for 32% of CIOs, preventing them from scaling AI initiatives.
A growing debate between CIOs and CFOs revolves around balancing AI’s potential with its financial implications. Making AI work effectively requires substantial investment, from modernizing legacy systems and implementing essential guardrails to ensuring data readiness.
Enterprises must allocate significant budgets for infrastructure upgrades, AI model training, and ongoing operations. AI can be expensive, and many IT leaders struggle to make it financially sustainable in the long run.
Gartner reinforces this concern, estimating that over half of organizations abandon their AI and Gen AI initiatives due to cost-related issues. The research firm further notes that fewer than 15% of organizations successfully identify, quantify, and measure these efforts’ costs, risks, and value.
According to the Enterprise AI Pulse survey responses on the biggest challenges in managing AI infrastructure costs, CIOs cite GPU and accelerator expenses (30%) as their top concern, followed by data storage and transfer costs (27%), software licensing and tooling (25%), and personnel and training expenses (18%). These findings highlight the growing demand for high-performance AI computing, the escalating costs of managing large datasets, and the critical need for skilled talent to operate and optimize AI systems.
The dominance of hardware costs reflects the growing dependence on GPUs, TPUs, and AI accelerators for training and inference workloads. High-end GPUs come at a premium price. Enterprises are exploring cloud-based AI processing and workload optimization techniques such as model quantization and federated learning to reduce infrastructure costs. Similarly, data storage and transfer costs remain a big concern, especially as AI models consume vast datasets.
Cloud egress fees and real-time data processing needs add to the financial burden. Hybrid cloud strategies, edge AI, and tiered storage policies can help enterprises mitigate these costs while ensuring seamless AI operations.
According to industry experts, CIOs will continue to look for ways to manage costs in the year ahead by negotiating software licensing agreements, adopting open-source tools, and implementing AI model lifecycle management to optimize resource utilization.
AI chip shortages
The next big challenge is procuring specialized hardware, such as GPUs and TPUs. Around 27% of ITDMs reported that it is a challenge. The demand for these chips increased with large AI models. Supply chain complications and exorbitant prices, though, are complicating the situation. Companies are now turning
to cloud alternatives and other categories of AI chips to cope with these issues.
The rapid development of AI has hugely fueled the need for high-performance chips, especially for natural language processing (NLP) workloads, computer vision, and reinforcement learning. Enterprises focusing on AI-driven automation, tailored customer experiences, and AIenabled big data analytics need greater computational power, increasing the demand for specialized hardware like GPUs and TPUs.
Acquiring these chips has been a significant roadblock with supply chain constraints, sky-rocketing prices, and procurement issues. The semiconductor sector has been struggling with disruptions due to geopolitical tensions, trade barriers, and pandemic-led production slowdowns.
With chip production clustered in the hands of a limited number of players—TSMC, NVIDIA, Intel, and AMD—supply is exposed to market risks and external disruption. Consequently, AI chip costs have skyrocketed, making it challenging for companies, particularly mid-tier companies and startups, to procure specialized AI hardware.
In the meantime, major cloud vendors and hyperscalers tend to obtain priority access to these chips, leaving other players in an unfavorable position. This imbalance pushes many organizations to consider cloud-based AI infrastructure, different AI accelerators, and hybrid AI approaches to bridge the prevailing hardware shortage.
While cloud adoption for AI workloads is growing, most enterprises (53%) still have less than a quarter of their AI workloads in the cloud, indicating a firm reliance on on-premise infrastructure. Only a tiny fraction (6%) have moved more than 75% of their AI workloads to the cloud, suggesting that full-scale cloud adoption remains rare. Data security, compliance, cost, and performance concerns may slow cloud migration.
However, a gradual shift is underway, with 22% of enterprises having 25-50% of AI workloads in the cloud and 19% reaching 51-75%. This suggests that organizations are adopting a hybrid approach, leveraging both on-premise and cloud environments based on business needs and technical feasibility.
Data storage and management
The third biggest roadblock in scaling AI infrastructure, cited by 21% of CIOs surveyed in Enterprise AI Pulse, is data storage and management. AI models rely on vast datasets distributed across on-premises data centers, cloud platforms, and edge devices. This fragmentation creates challenges with data accessibility, latency, and integration, making it difficult to streamline AI workflows.
Transferring large datasets between storage solutions or cloud providers further compounds the issue. High data movement costs, bandwidth bottlenecks, and performance constraints can significantly slow AI adoption. As organizations embrace hybrid and multi-cloud AI strategies, efficient data pipeline management becomes critical to ensure seamless model training and inference.
Beyond infrastructure hurdles, regulatory compliance adds another layer of complexity. Data protection laws such as GDPR and India’s DPDP Act mandate stringent controls over data storage, processing, and movement, restricting how organizations can leverage data for AI applications. Maintaining data consistency and quality across distributed environments is also essential for AI accuracy and reliability.
To overcome these challenges, enterprises increasingly adopt hybrid storage architectures, AI-driven data management solutions, and edge computing to minimize data movement and optimize storage efficiency. Implementing a well-structured DataOps strategy can enhance AI scalability by automating data workflows, improving governance, and enabling organizations to maximize AI’s potential while staying compliant.
Skill shortage and staffing
A major challenge in scaling AI initiatives is the shortage of skilled AI professionals, such as data scientists and machine learning engineers, with 21% of CIOs citing this as a key concern. The lack of in-house expertise makes it difficult for enterprises to build and maintain robust AI teams, slowing innovation and
increasing reliance on third-party vendors. At the same time, organizations grapple with data storage and movement challenges, whether managing on-premises infrastructure or navigating complex multi-cloud environments. Efficient data handling remains critical for AI performance, but many companies struggle with accessibility, integration, and cost constraints.
In 2025 and beyond, enterprises are expected to focus on AI automation (AIOps) to streamline IT operations and reduce manual workload. Upskilling internal IT teams will also be a priority, helping businesses develop AI capabilities in-house rather than depending solely on external talent. Additionally, AI-driven automation will minimize the need for highly specialized roles, allowing organizations to deploy and manage AI models more efficiently while addressing skill gaps.
AI Governance
Many enterprises struggle with the absence of clear regulatory guidelines and industry-wide best practices, making it difficult to establish consistent governance policies for AI models. This gap leads to accountability, risk management, and compliance issues, especially as AI regulations evolve globally. The most significant challenge, cited by 30% of respondents, is the lack of standardized frameworks.
Other significant challenges include technical complexity (28%) and resource constraints (28%). AI governance requires specialized data science, ethics, and compliance expertise, which many organizations lack. Additionally, managing AI systems at scale demands significant financial investment, making budget
constraints a critical barrier. Lastly, organizational resistance (15%) remains a challenge, as businesses often face internal pushback when mplementing AI governance measures, either due to cultural inertia or concerns about operational disruptions. Addressing these challenges will require a combination of regulatory clarity, investment in AI talent, and cultural change within enterprises.
The road ahead
The mandate is evident from the survey and insights from CIOs who have successfully implemented AI. Achieving AI success isn’t about jumping on the bandwagon; it requires a structured, well-governed strategy aligned with business objectives. Based on their experiences, enterprises must take decisive action to move AI initiatives beyond pilots and into scalable, high-impact deployments.
◼ Get clarity on AI goals – AI should not be a solution in search of a problem. Organizations
must define clear, outcome-driven AI goals that align with their business strategy. Whether enhancing operational efficiency, driving customer engagement, or unlocking new revenue streams, AI investments must be tied to measurable business impact. A well-defined AI roadmap, with input from cross-functional teams, ensures that initiatives remain focused, feasible, and value-driven.
◼ Fix the data first – AI is only as good as the data it runs on. Organizations must invest in data governance, classification, and cleansing strategies to ensure high-quality, bias-free, structured data. Implementing strong DataOps frameworks can help streamline data accessibility, integration, and quality
control.
◼ Optimize infrastructure with cost in mind – Managing AI-related infrastructure expenses—whether for GPUs, cloud egress fees, or storage—is a top priority. Enterprises should explore hybrid cloud models, federated learning, and model optimization techniques to balance performance with cost efficiency.
◼ Prioritize use cases with measurable impact – CIOs recommend identifying AI use cases that directly enhance business outcomes instead of broad, vague AI ambitions. Whether it’s supply chain optimization, customer experience personalization, or financial automation, AI must drive measurable value.
◼ Bridge the talent gap – With a shortage of skilled AI professionals, companies need to upskill internal teams and leverage AI automation tools like AIOps to reduce reliance on specialized talent. Building an intense AI center of excellence can accelerate internal expertise development.
◼ Build strong AI governance frameworks – AI governance isn’t just a compliance requirement—it’s a competitive advantage. Organizations must establish clear accountability, ethical AI guidelines, risk assessment models, and explainability frameworks to ensure AI is trustworthy and compliant with evolving regulations.
◼ Scale AI with a long-term vision – CIOs emphasize that AI is a marathon, not a sprint. Companies must move beyond Proof-ofConcept (PoC) trials and pilot projects, investing in scalable AI architectures, automation frameworks, and iterative deployment strategies that ensure long-term success.