Two important technologies associated with Exadata networking are InfiniBand and RoCE (RDMA over Converged Ethernet). Earlier generations of Exadata relied heavily on InfiniBand for the high-speed private network, while newer Exadata platforms use Ethernet-based networking with RoCE capabilities.

The fundamental idea remains similar: provide a low-latency, high-bandwidth path between database servers and storage servers while supporting RDMA (Remote Direct Memory Access) and intelligent database I/O.

Understanding the Oracle Exadata RoCE and InfiniBand Switch Architecture helps Oracle DBAs, Exadata administrators, database architects, and infrastructure engineers understand how database traffic travels through the system and why the network fabric is essential to Exadata performance. This article explains the architecture, major components, RDMA, RoCE, InfiniBand, network topology, I/O flow, redundancy, performance considerations, and troubleshooting concepts.

The Exadata network fabric is the high-speed communication infrastructure connecting the major components of an Exadata environment.

The network fabric allows database servers to communicate with storage servers and, depending on the Exadata configuration and generation, supports different categories of traffic.

The key requirements for this network include:

Exadata is designed around a scale-out architecture. As database and storage resources are added, the network fabric must provide sufficient connectivity and bandwidth to support parallel workloads.

An Exadata environment generally contains several logical network functions rather than one network used for everything.

The major networking areas include:

• Client access network
• Management network
• Private cluster/interconnect network
• Database-to-storage network

The database-to-storage network is especially important for Exadata performance because it carries the communication between database servers and storage servers.

The exact physical topology, switch configuration, and supported network technologies depend on the Exadata generation and platform.

InfiniBand is a high-performance networking technology designed for low-latency communication and high-throughput data transfer.

Earlier generations of Oracle Exadata used InfiniBand as the high-speed fabric connecting database servers and storage servers. InfiniBand was particularly suitable for Exadata because database workloads can generate substantial I/O traffic between compute and storage nodes.

Its architectural characteristics include:

Multiple switches and redundant links can be used to provide resilient connectivity.

RoCE stands for RDMA over Converged Ethernet. It combines the RDMA communication model with Ethernet networking. The objective is to provide efficient, low-latency memory-to-memory communication over an Ethernet fabric without requiring traditional CPU-intensive data-copy operations for every transfer.

Modern Exadata platforms use Ethernet-based networking and RoCE capabilities to provide high-performance communication between system components.

The exact implementation varies by Exadata generation, so architecture documentation for the specific platform should always be consulted when designing or troubleshooting an environment.

RDMA (Remote Direct Memory Access) is one of the key concepts behind both InfiniBand and RoCE. Traditional network communication often involves multiple interactions between the operating system, CPU, network stack, and memory. RDMA allows a system to transfer data directly between memory regions of communicating systems with significantly less CPU and software-stack involvement.

This can reduce CPU overhead and latency while improving data-transfer efficiency. For Exadata, RDMA is particularly important because database and storage servers need to exchange I/O information efficiently at high speed.

Although RoCE and InfiniBand are different technologies, both can support RDMA-based communication.

The comparison should not be interpreted as meaning that every Exadata platform supports both technologies simultaneously. Exadata generations have different hardware and networking architectures.

A RoCE-based Exadata environment uses high-speed Ethernet switching to connect database servers and storage servers.

Redundant network paths are normally used so that the failure of a single component does not isolate a database server or storage server. RoCE networks also require appropriate Ethernet configuration to support reliable, predictable RDMA traffic.

RoCE operates over Ethernet, but high-performance RDMA traffic has specific network requirements.

A poorly configured Ethernet fabric can negatively affect the performance expected from a RoCE-based architecture. Therefore, Exadata networking should be treated as an integrated engineered-system component rather than as a generic Ethernet network.

In earlier Exadata architectures, InfiniBand switches formed the high-speed fabric between database servers and storage servers.

Redundant switches and links provide multiple communication paths. The architecture allows database servers to access multiple storage cells concurrently, creating a highly parallel storage fabric. This parallelism is important for large database workloads because a query can potentially generate I/O across multiple storage servers rather than depending on one centralized storage device.

The database-to-storage communication path is central to Exadata's performance model.

The storage server can perform eligible processing before returning data to the database server. This is one of the fundamental differences between Exadata and conventional storage architectures.

Exadata uses specialized communication mechanisms for database-to-storage interaction. iDB, or Intelligent Database, is associated with communication between Oracle Database and Exadata Storage Servers for intelligent I/O operations. The database server can send storage requests that contain information allowing the storage server to perform appropriate processing.

The storage server can then:

This architecture reduces unnecessary data movement.

RoCE and RDMA provide the high-speed communication foundation, while Exadata software provides database-aware processing. These technologies work at different architectural layers.

This layered architecture is important when troubleshooting performance because a problem can originate at the database, software, network, switch, or storage layer.

Smart Scan allows eligible SQL processing to be offloaded to Exadata Storage Servers. For example:

In a traditional architecture, the database server may need to retrieve a large amount of data and perform filtering locally. With eligible Exadata Smart Scan processing:

This can significantly reduce the amount of data transferred across the network. Oracle's Exadata architecture is designed to offload eligible database operations to storage servers and return relevant results rather than unnecessarily transferring all underlying data.

The network fabric connects database servers to the intelligent storage layer. At the storage-server side, CELLSRV is the core software component responsible for most storage-server services.

This means the network cannot be analyzed independently from Exadata System Software. A database administrator troubleshooting slow I/O should consider both the network path and what CELLSRV is doing with the request.

High availability is a fundamental design requirement for engineered systems.

Redundancy helps protect against failures involving:

The actual redundancy design depends on the Exadata generation and hardware architecture.

Network performance directly affects database I/O when database servers communicate with storage servers.

The objective is not simply to maximize bandwidth. The complete system must provide balanced performance across CPU, memory, network, and storage resources.

A RoCE-based architecture requires careful engineering.

InfiniBand environments have their own management and operational requirements. DBAs and infrastructure teams should monitor:

A healthy InfiniBand fabric is essential for maintaining predictable database-to-storage communication.

When diagnosing an Exadata performance issue, avoid assuming that the database or storage is automatically responsible. A structured troubleshooting approach is better.

The most important architectural distinction is the underlying network fabric.

RoCE provides RDMA capabilities over Ethernet, while InfiniBand is a dedicated high-performance networking technology with native RDMA capabilities. For modern Exadata deployments, the exact network technology should be determined from the supported hardware architecture rather than assumed from older Exadata documentation.

The combination of high-speed networking, RDMA, intelligent storage, and database-aware software provides several benefits.

The Oracle Exadata RoCE and InfiniBand Switch Architecture is a fundamental part of Exadata's high-performance database design. The network fabric connects database servers with intelligent storage servers and provides the communication path required for high-speed database I/O.

InfiniBand played an important role in earlier Exadata generations, while newer Exadata platforms use Ethernet-based architectures incorporating RoCE and RDMA. Although the underlying networking technology has evolved, the architectural objective remains consistent: provide high bandwidth, low latency, scalable connectivity, and efficient communication between database and storage resources.

The complete Exadata performance model depends on several layers working together:

Understanding this complete path helps Oracle DBAs and infrastructure professionals diagnose performance problems more accurately. A slow database operation may involve SQL execution, storage processing, network congestion, switch behavior, or hardware configuration rather than a single isolated component.

For professionals working with Oracle Exadata, knowledge of RoCE, InfiniBand, RDMA, network fabrics, Smart I/O, CELLSRV, and database-to-storage I/O flow provides a strong foundation for Exadata administration, performance tuning, architecture design, and troubleshooting.