How enterprises can modernise data infrastructure to scale AI adoption, control costs and deliver ROI.
As enterprises move from AI experimentation to deployment at scale, the focus is shifting from model capabilities to the infrastructure, data architecture and governance required to support AI in production. For CIOs, the challenge is to balance performance, scalability, security and cost while modernising legacy environments and demonstrating measurable business value. Hybrid infrastructure, the convergence of transactional and analytical systems, and the growing adoption of AI agents are further complicating these decisions.
In a recent interaction with CIO&Leader, Sanjay Agrawal, Head of Presales and Chief Technology Officer (CTO) at Hitachi Vantara India and SAARC, discusses how enterprise data infrastructure is evolving to support AI workloads, the trade-offs between cloud and on-premises environments, and the architectural bottlenecks organisations must address as data volumes grow. He also examines the role of storage in AI infrastructure, the case for shared versus separate data platforms, and the priorities CIOs should consider when building an AI-ready enterprise. Excerpts from the interaction.
CIO&Leader: How is the rise of AI reshaping enterprise data infrastructure, and what are the biggest technology shifts CIOs need to prepare for as they move towards AI-ready architectures?
Sanjay Agrawal: Technology and market expectations are changing rapidly. Traditionally, enterprises primarily used IT for databases and structured data, with volumes often limited to a few hundred terabytes. Insights were largely derived from transactional data. For example, a bank could understand its customers based on the accounts they held, the transactions they conducted, and the services they consumed.
With the emergence of customer 360 initiatives, enterprises began incorporating unstructured data to gain deeper insights into customer sentiments, interactions, behaviours, locations, preferences, and loyalty. This led to a significant increase in the volume of unstructured data, in some cases by 400%-500%.
As AI adoption accelerated, organisations began extracting greater value from these data repositories. The AI journey progressed from machine learning and deep learning to generative AI, AI agents, and, more recently, agentic AI.
The AI journey has evolved from machine learning and deep learning to generative AI, AI agents and agentic AI.
Throughout this evolution, data quality, availability and infrastructure have remained critical. Without sufficient, high-quality data and infrastructure capable of delivering it to GPUs at the required speed, AI systems cannot perform effectively, regardless of how advanced the underlying models or large language models (LLMs) are.
At Hitachi Vantara, we focus on the entire technology stack, with a particular emphasis on data infrastructure. Our solutions are designed to maintain the right balance between compute and data infrastructure while adapting to evolving AI workloads.
As enterprises adopt agentic AI, requirements for contextual data, access frequency and concurrency rise, potentially increasing infrastructure demands by five to ten times. Our architecture addresses these needs by decoupling the data layer from the compute layer, enabling enterprises to scale storage and data infrastructure independently for AI training, inference and agentic decision-making.
Our architecture decouples the data layer from the compute layer, allowing enterprises to scale infrastructure independently.
Through Hitachi IQ, we aim to deliver a unified solution that combines compute capabilities with a flexible, independently scalable data layer, enabling enterprises to support evolving workloads as they progress from traditional AI and machine learning to generative AI and agentic AI.
CIO&Leader: How are you balancing the growing demand for cloud and hybrid infrastructure with the need for performance, security, governance and cost control?
Sanjay Agrawal: CIOs are increasingly reassessing cloud economics as data volumes and processing requirements grow. While the cloud offers flexibility and access to on-demand computing resources, costs can be difficult to manage when workloads involve large datasets, sustained processing or frequent data transfers. Moving everything back on-premises is not necessarily the answer either, as some workloads benefit from the cloud’s elasticity.
The challenge is to determine where each workload should run based on its performance, security, governance and cost requirements. For many enterprises, this is leading to a hybrid model in which core data repositories remain on-premises, while selected workloads use cloud resources as needed.
For example, an organisation could retain its primary data lake or lakehouse on-premises and move only the relevant subset of data to the cloud for a specific AI workload. The results can then be returned to the enterprise environment. This can help limit data movement and egress costs, although the economics depend on the workload and frequency of data transfers.
Object storage is increasingly being used to support large on-premises data repositories. For CIOs, the priority should be to establish clear workload-placement policies, understand the total cost of ownership across environments, and ensure that data governance and security controls remain consistent across on-premises and cloud infrastructure.
CIO&Leader: As data volumes and AI workloads continue to grow rapidly, what do you see as the biggest infrastructure bottlenecks enterprises are likely to face in the years ahead?
Sanjay Agrawal: One of the biggest challenges is ensuring that the data architecture can scale with AI workloads without creating performance bottlenecks or high costs. Enterprises often focus on compute capacity when planning for AI, but data access, throughput, and the ability to handle concurrent requests can be equally important.
The choice of storage architecture is therefore critical. Traditional storage systems may continue to serve transactional workloads effectively. Still, large-scale AI applications involving unstructured data may require object storage, high-performance parallel file systems, or a combination of technologies. The appropriate design depends on the workload’s latency, throughput and scalability requirements.
Another challenge is integrating data warehouses and data lakes. Data lakehouse architectures offer a way to bring these environments closer together, supporting analytical workloads across structured and unstructured data. Object storage can provide the underlying capacity, while compatible table formats and processing engines enable data to be accessed and analysed.
For CIOs, the key is to avoid designing infrastructure around a single AI use case. They need to assess how data volumes, concurrent access and performance requirements are likely to evolve, and build an architecture that can accommodate growth without requiring major redesigns. This also means identifying where additional performance is genuinely necessary rather than investing in high-performance infrastructure across the entire data estate.
CIO&Leader: As AI reshapes enterprise infrastructure, how is Hitachi Vantara’s role evolving beyond traditional storage towards becoming a core part of the data infrastructure powering enterprise AI?
Sanjay Agrawal: The distinction is between infrastructure that supports business transactions and infrastructure that supports data analysis and AI. Both are essential, but their requirements differ.
Transactional systems, such as core banking platforms, need low latency, predictable performance, high availability, and reliability. A delay or disruption in processing a customer transaction can directly affect business operations. These workloads require infrastructure designed to maintain consistent performance, including during peak demand. Hitachi Vantara’s Virtual Storage Platform (VSP) portfolio addresses this category of enterprise storage requirements.
Analytical and AI workloads have distinct demands. Enterprises need to store, discover and process large volumes of structured and unstructured data, including documents, customer records and other information used for analytics and AI applications. They must also manage access, retention, privacy and deletion requirements in line with applicable regulations.
This is where object storage and platforms such as Hitachi Content Platform become relevant. As enterprises move towards generative AI and AI agents, they also need to consider how data is made accessible to AI applications, how access is governed, and how the underlying infrastructure scales with demand. Hitachi IQ is part of our approach to this broader AI infrastructure requirement.
For CIOs, the decision is not simply about choosing a storage product. It is about building a data architecture that meets the performance and availability requirements of core business systems while supporting the scale, accessibility, and governance needed for analytics and AI.
CIO&Leader: CIOs are facing increasing pressure to demonstrate measurable business value from technology investments. As AI moves from observing the business to executing it, does the infrastructure need to differ between systems of record and systems of intelligence?
Sanjay Agrawal: Yes. Systems of record and systems of intelligence have different performance and scalability requirements, and these differences become more important as AI moves from generating insights to taking actions.
Systems of record support business transactions and therefore prioritise low latency, predictable performance, reliability and availability. Systems of intelligence typically process larger datasets for analytics, predictions and AI-driven recommendations, making scalability and efficient data access particularly important.
For example, a transactional environment may manage hundreds of terabytes of data, while an AI or analytics environment may need to process multiple petabytes. Data lakes and lakehouses can grow to tens of petabytes, requiring a different approach to capacity, data access, and workload management.
The distinction becomes especially important when AI agents begin interacting with operational systems. If an AI application needs to retrieve information and trigger a business transaction, the architecture must account for both analytical processing and the transactional system’s performance requirements.
Enterprises should continue using differentiated infrastructure for these workloads over the next few years. However, the degree of separation will depend on their applications, performance requirements, and existing architecture.
For CIOs, the priority is to ensure that AI initiatives do not compromise the reliability of mission-critical systems, while avoiding unnecessary duplication of infrastructure wherever workloads can safely share resources.
CIO&Leader: Is there an opportunity to create a common data foundation underneath both systems, or does it make more sense to maintain separate infrastructure?
Sanjay Agrawal: Both approaches are technically viable, and the decision should be based on workload requirements rather than a preference for complete consolidation or separation.
A common infrastructure can support different data types and services, including structured and unstructured data, as well as block, file, and object storage. This can simplify capacity management, improve utilisation, and reduce the cost and complexity of maintaining separate environments.
However, consolidation has limits. Latency-sensitive transactional workloads may be affected if they share infrastructure with resource-intensive analytics or AI processing. CIOs therefore need to consider workload isolation, performance guarantees, availability, and the consequences of competing resource demands.
A shared foundation may work well where workloads have compatible performance requirements, and the infrastructure can manage resource contention effectively. Separate environments may be more appropriate when transaction processing requires tightly controlled latency or when regulatory, security or operational considerations demand greater isolation.
The decision should follow an assessment of workload characteristics, service-level requirements and total cost of ownership. A common data foundation can be valuable, but it does not necessarily mean that every workload should run on the same underlying infrastructure.
CIO&Leader: As enterprise AI architectures become more complex, no single vendor can deliver the entire stack. How are you leveraging partnerships across hyperscalers, AI platform providers and system integrators to help enterprises move from AI pilots to production at scale?
Sanjay Agrawal: Enterprise AI involves several layers, including infrastructure, data management, models, applications, security and integration with existing business systems. No single provider necessarily covers all these requirements, so partnerships are important in delivering solutions that work in an enterprise environment.
Hyperscalers can provide cloud infrastructure and services for workloads that benefit from elastic capacity. System integrators help organisations connect AI applications with existing systems, data sources and business processes. AI technology providers contribute models, development tools and application capabilities.
Our role includes providing storage and data infrastructure for these environments, including through software-defined storage offerings available in public cloud environments. For on-premises deployments, we work with system integrators and AI partners to combine infrastructure with the software and applications required for specific use cases.
As part of the Hitachi Group, GlobalLogic brings digital engineering and AI expertise that can support customer projects.
From a CIO’s perspective, the important consideration is how well these components work together. Organisations need clarity on integration responsibilities, data access, security, performance, support and accountability when problems arise. They should also assess the risk of vendor lock-in and the portability of their data and applications.
The objective is to start small, demonstrate value and progressively expand the AI journey.
The objective should be to build an ecosystem that addresses business requirements while allowing the organisation to evolve its AI architecture as technologies and use cases change.
CIO&Leader: With budgets under pressure and legacy systems limiting AI adoption, what should CIOs prioritise over the next 12–18 months to deliver measurable ROI?
Sanjay Agrawal: CIOs should focus on two priorities: identifying high-value AI use cases and building a data foundation to support them.
They should start with clearly defined business problems and assess potential returns, implementation costs, risks, and time to value. Starting small and scaling successful initiatives can demonstrate ROI before committing to larger investments.
Equally important is addressing fragmented data, quality, accessibility and governance. Rather than replacing their entire infrastructure, enterprises can build on existing investments while ensuring scalability for future AI workloads.
Security, resilience and governance must also be built in from the outset, particularly as AI agents begin taking actions autonomously. Over the next 12–18 months, CIOs should prioritise measurable use cases, data readiness and scalable infrastructure, expanding investments as business value becomes evident.
CIO&Leader: With responsible AI becoming increasingly important, challenges around data governance, security, privacy and lineage are affecting organisations’ ability to move AI workloads into production. What are some of the best practices enterprises should follow?
Sanjay Agrawal: Moving AI from experimentation into production requires organisations to address data quality, security, governance and accountability alongside model performance. These are operational requirements, not simply compliance considerations.
The first step is to establish where the data comes from, whether the organisation has the right to use it, how it is protected, and who can access it. Data lineage is important because organisations need to understand which sources informed an AI-generated output and whether the underlying information is accurate, up to date, and appropriate for its intended use.
Several practices are particularly important:
- Data governance and security: Establish clear data ownership, access controls, retention policies and protection against cyber threats. Sensitive information should be handled in accordance with applicable privacy and regulatory requirements.
- Fairness: Assess training data and AI outputs for bias or discriminatory patterns, particularly in applications that influence customer access, employment, credit or other consequential decisions.
- Transparency and explainability: Maintain appropriate documentation of the models, data sources and assumptions involved. Where AI informs a decision, users should be able to understand the key factors behind the outcome to the extent appropriate for the application.
- Accountability: Define who is responsible for AI-generated outputs and actions across business, IT, data and technology teams. These responsibilities should be established before deployment, particularly when AI agents can initiate actions across business systems.
- Human oversight: Set clear boundaries for autonomous actions and identify decisions that require human approval. Organisations should also have mechanisms to monitor performance, investigate errors, and intervene when systems behave unexpectedly.
CIOs should integrate these controls into the AI lifecycle rather than treat governance as a final approval step. The level of oversight should reflect the sensitivity of the data, the potential impact of an incorrect outcome and the degree of autonomy given to the system.
The objective is to make AI systems sufficiently secure, traceable and controllable for their intended business use, while enabling organisations to move beyond pilots into production.