Technology ๐Ÿ‘ค KP Expert ๐Ÿ“… Nov 16, 2026 ๐Ÿ‘๏ธ 289 views

Oracle Exadata RoCE Switch Snapshot Report: What It Is and What Information It Collects

A deep look at the report DBAs use to prove the RoCE fabric came back after maintenance: where it comes from, how it is structured, and exactly what each layer records.

Updated 16 November 2026 by KP Expert ยท 9 min read

Key takeaways

  • It is the Cisco NX-OS snapshot feature on Exadata's RoCE switches, not a separate Oracle tool.
  • It collects in two layers: a built-in feature summary and Exadata detail sections you add.
  • LLDP neighbors turn port numbers into named database servers and storage cells.
  • Its real value is comparison: before vs after a change, matched row by row.
  • It does not collect logs, configuration, hardware health or the server side of the link.

01Introduction

A RoCE switch snapshot report is a saved, point-in-time record of what an Exadata RoCE switch looks like while it is running: which ports are up, at what speed, which server or storage cell sits behind each port, and whether the links are clean and lossless. You take one before a change and one after, then compare them.

Since Exadata X8M, the internal network that carries Smart Scan, Exadata RDMA Memory (XRMEM) and RAC interconnect traffic runs on RDMA over Converged Ethernet (RoCE) instead of InfiniBand. RoCE runs the RDMA protocol on top of Ethernet, so existing Exadata performance features continue to work unchanged. The switches that carry this traffic are Cisco Nexus switches, and the snapshot report is built on their NX-OS snapshot feature.

This article explains the concept in depth: what the report really is, where it comes from, and exactly what it captures and what it does not. For hands-on commands, see our companion step-by-step guide to generating a RoCE switch snapshot report.

Who this is forExadata DBAs and DMAs on X8M, X9M, X10M and X11M systems who patch, cable or troubleshoot the RoCE fabric, and anyone preparing for Exadata administration interviews.

02Background: the Exadata RoCE Network Fabric

Every Exadata rack from X8M onward has at least two RoCE leaf switches, and every database server and storage server has one RoCE port cabled to each. The fabric connects the database and storage servers through a bonded interface to these switches. Multi-rack systems add spine switches to join the leaves of different racks.

ComponentTypical nameRole in the fabric
Lower leaf switch (Cisco Nexus 9336C-FX2)<rack>sw-rocea0First RoCE path for every server and cell
Upper leaf switch (Cisco Nexus 9336C-FX2)<rack>sw-roceb0Second RoCE path for every server and cell
Spine switch (multi-rack only)<rack>sw-roces0Links leaf switches across racks
Database server / storage cell<rack>dbadm01, <rack>celadm01Two RoCE ports (re0, re1), one per leaf

Three properties of this fabric explain why a snapshot report is so useful:

  • Redundancy hides failures. If one link drops, traffic moves to the other leaf and the database keeps running. A port that stays down after maintenance can go unnoticed until the second path fails.
  • The configuration is fixed by Oracle. Switches run Oracle's golden configuration; Exadata 20.1.0 and later include a simplified procedure to apply it. You verify state rather than change settings.
  • RDMA needs a lossless network. The golden configuration enables Priority Flow Control (PFC) and congestion signalling. A port can be up yet still hurt performance if its error or pause counters climb.

03What the RoCE switch snapshot report is

The RoCE switch snapshot report is not a separate Oracle utility. It is the Cisco NX-OS snapshot feature, run on the Exadata RoCE switches, combined with a few Exadata-specific sections and a before-and-after comparison. Oracle has no tool by that exact name, which is why the term confuses many DBAs.

An NX-OS snapshot captures the running states of selected features, can be extended with new show commands, and can be compared with or deleted from other snapshots. In practice, the "report" is three outputs:

  1. The snapshot itself: a named, stored copy of show-command output, kept on the switch.
  2. The dump: a readable listing of everything one snapshot captured.
  3. The comparison: a field-by-field diff of two snapshots, as a summary of counts or in full detail.

How a snapshot is structured

A snapshot is a set of sections. Each section is one show command plus the information NX-OS needs to match rows between two snapshots. Cisco's own example section, v4route, captures show ip route detail vrf all, with ROW_prefix as the row id and ipprefix as the key.

Part of a sectionWhat it doesExadata example
Section nameLabel shown in dumps and comparisonslldpnbr
Show commandCommand whose structured output is storedshow lldp neighbors detail
Row idTag that marks one row in the XML/JSON outputROW_nbor_detail
Key 1 (and optional key 2)Field that uniquely identifies each rowLocal port id

Because rows are matched by key, the comparison does not just say "something changed". It names the exact interface, neighbor or route that changed, and shows the old and new values side by side.

Where it fits among Exadata diagnostics

Most Exadata diagnostics describe one moment so Oracle Support can investigate a fault. The snapshot report is different: its value comes from comparison. It answers one question quickly: did the RoCE fabric come back exactly as it was?

04How the report is generated (overview)

You generate the report by logging in to each RoCE switch as admin over SSH, adding the Exadata sections once, then creating, dumping and comparing snapshots. Oracle requires switch commands to be run while connected to the switch management interface as the switch administrator.

on the switch, exec mode
exa01sw-rocea0# snapshot section add lldpnbr "show lldp neighbors detail" ROW_nbor_detail l_port_id
exa01sw-rocea0# snapshot create pre_patch_16nov2026 Before RoCE firmware upgrade
exa01sw-rocea0# show snapshots dump pre_patch_16nov2026
exa01sw-rocea0# snapshot create post_patch_16nov2026 After RoCE firmware upgrade
exa01sw-rocea0# show snapshots compare pre_patch_16nov2026 post_patch_16nov2026 summary

Repeat on both leaves and every spine. Row ids and keys can differ between NX-OS releases, so confirm them with show <command> | json before adding a section. Full prerequisites, sample outputs and an automation script are in the step-by-step guide.

05What information the report collects

The report collects operational state in two layers: a built-in summary that every NX-OS snapshot includes, and the Exadata detail sections you add. Together they answer one question: is every server and cell still connected, at full speed, on a clean and lossless path?

Layer 1: built-in feature summary

Every snapshot records the state of the switch's interfaces and of whichever protocols are enabled. The exact list depends on the NX-OS release and the features in the golden configuration, so check it with show snapshots dump <name> after your first snapshot.

Data pointWhat it tells youHealthy result on a RoCE leaf
Total interfacesHow many interfaces the switch knows aboutSame before and after
Ethernet ports up / downHow many physical ports have a live linkUp count equals cabled ports
VLANsNumber of VLANs definedUnchanged (Secure Fabric isolation adds more)
IPv4 / IPv6 routesSize of the routing tablesUnchanged
Protocol neighbors (if enabled)Sessions between switches in multi-rack fabricsAll sessions established
Spanning tree (if enabled)Layer 2 loop protection stateUnchanged

Layer 2: Exadata detail sections

The built-in summary only counts. The Exadata sections identify each port and each host, which is what you need to act.

Section and commandInformation collected per portWhat it reveals on Exadata
Interface status: show interface briefAdmin state, operational state, speed, mode, down reasonEvery cabled port should be up at 100 Gbps; reasons such as Link not connected or XCVR not inserted explain failures
LLDP neighbors: show lldp neighbors detailNeighbor system name, neighbor port (re0/re1), management addressMaps each switch port to a named database server or storage cell
Transceivers: show interface transceiverCable or optic type, vendor, part number, serial numberA changed serial confirms a part swap; a missing entry means the module is not detected
Error counters: show interface counters errorsFCS, alignment, receive and transmit errorsA jump on one port points to a bad cable, dirty connector or failing optic
Lossless QoS: show interface priority-flow-controlPFC mode, pause frames sent and receivedPFC off or unusual pause counts signal a QoS or congestion problem
Routes and inter-switch links (multi-rack)Route tables, leaf-to-spine neighbor sessionsA lost session or prefix means part of the fabric is unreachable

Snapshot metadata

Each snapshot also stores its name, creation time and the free-text description you give it. Putting the change ID or Oracle SR number in the description makes the report self-documenting.

What the snapshot does not collect

  • Logs and crash history: use show logging log and show tech-support details.
  • The running configuration: back it up separately; a snapshot is state, not configuration.
  • Hardware health such as fans, power supplies and temperature: use show environment.
  • The server side of the link: use Exadata tools such as rocelinkinfo, ip link and Exachk.
  • Traffic performance over time: snapshots are single points, not trends.

06How to read and use the report

Read the summary comparison first, then open the full comparison only for features whose counts changed. In the detailed output, NX-OS marks values that differ between the two snapshots; no marked lines means the switch returned to exactly the same state.

  1. Check the summary counts: interfaces up and down, VLANs, routes and neighbor sessions.
  2. If a count differs, open the full comparison and find the marked lines.
  3. For each change, note the key (port, neighbor or prefix) and its before and after values.
  4. Map the port to a host using the LLDP section of the before snapshot.
  5. Decide whether the change was planned or needs action.
Difference seenLikely meaningNext step
Port up before, down afterHost still booting, cable unseated or transceiver faultWait for the host, reseat the cable, check the port detail
LLDP neighbor missing on an up portLLDP not exchanged yet, or wrong host connectedRecapture after 1 to 2 minutes; check the cabling map
LLDP neighbor name changedCables swapped between portsCompare with Oracle's cabling tables and correct
Speed dropped (100G to 40G)Wrong or degraded cable or opticReplace the cable; verify the part number
Transceiver serial changedA component was replacedExpected only if you replaced it
Error counters jumped on one portPhysical-layer problemClean or replace optics and cable
PFC off or pause frames spikingQoS misconfiguration or congestionCheck golden configuration; open an Oracle SR
Fewer routes or neighbor sessionsInter-switch link or spine problemCheck uplinks and the spine snapshots
Ignore traffic countersOrdinary packet and byte counters always change between snapshots. Only error and pause counters matter in a comparison.

A worked example

After a rolling cell patch, the summary on rocea0 shows 30 ports up before and 29 after, while roceb0 is unchanged. The full comparison marks Ethernet1/12 as down. The before snapshot's LLDP section shows that port belonged to celadm03, port re0.

That cell is now running on its roceb0 link alone, a hidden single point of failure. Reseating the cable restores the link, and a fresh after snapshot shows no differences.

07How it compares with other Exadata diagnostic tools

The snapshot report is the only tool in the Exadata kit that compares switch state before and after a change; the others describe one moment or one side of the link. Use it alongside them, not instead of them.

ToolRuns onWhat it collectsBest used for
RoCE switch snapshot reportRoCE switch (NX-OS)Port state, LLDP neighbors, optics, errors, PFC; compared over timeProving the fabric returned to its prior state
show tech-support detailsRoCE switchVery large dump of switch state, logs and configurationFirst thing Oracle Support asks for on a switch fault
rocelinkinfo (Exadata 24.1+)Database serverReal-time status of each fabric link, in iblinkinfo formatViewing the fabric from the Exadata side
configure_roce_hostinfo.sh (24.1+)Database serverLabels each switch port with the connected host, IP and portMaking port-level output easier to read
verify_roce_cables.pyDatabase serverChecks hosts and switches are correctly cabledValidating cabling after installation or changes
Exachk (AHF)Database serverBest-practice and health checks, including RoCE switchesWhole-rack health before and after maintenance
SundiagDatabase and storage serversServer hardware, disk and OS diagnosticsServer or storage hardware faults
ILOM snapshotServer ILOMService processor, firmware and hardware dataHardware faults below the OS
Oracle ASR (Exadata 21.2+)MonitoringAutomatic service requests for Cisco switch faultsHands-off fault reporting

If patchmgr fails on a RoCE or management switch, Oracle publishes a My Oracle Support note listing the support data to collect for the 9336 and 9348 switches (MOS Doc ID 2871074.1). Attach your snapshot comparison to that data. For server-side diagnostics, see our Exadata Sundiag guide.

08Best practices and security considerations

Snapshot every RoCE switch before you touch any of them, and save every output off the switch. Snapshots live on the switch, so they are lost if the switch is replaced.

  • Cover the whole fabric: both leaves in every rack and all spines, never one switch.
  • Name consistently: pre_<activity>_<date> and post_<activity>_<date>, with the change ID or SR number in the description.
  • Wait before the after snapshot: let every database server and cell finish booting, or you will chase false "port down" differences.
  • Keep a healthy baseline: refresh one per switch monthly so a sudden complaint can be compared against a known-good state.
  • Add sections once: define the Exadata sections once per switch and reuse them.
  • Clean up: snapshots are never deleted automatically; remove old ones after saving them.

Security and change control

  • Read-only by design. Creating, dumping, comparing and deleting snapshots does not change interfaces, traffic or the golden configuration. Only snapshot section add changes anything, and it edits the snapshot definition, not the network configuration.
  • Do not use Cisco maintenance mode (GIR) on Exadata unless an Oracle procedure tells you to; take snapshots manually instead.
  • Protect the output. Dumps contain hostnames, management IPs, serial numbers and cabling. Store them like other infrastructure records, and share them only through your Oracle SR.
  • Protect switch access. Use the admin account only from trusted database servers, and do not change switch SSH settings outside Oracle's documented procedures.

09Frequently asked questions

Q.Is the RoCE switch snapshot report an Oracle tool?

No. It is the Cisco NX-OS snapshot feature used on Exadata's Cisco Nexus RoCE switches. Oracle's own fabric tools, such as rocelinkinfo and Exachk, run from the database servers.

Q.Does taking a snapshot affect database traffic?

No. It only runs show commands and stores the output, so it is safe on a live production rack.

Q.Is a snapshot the same as a switch configuration backup?

No. A snapshot records operational state (what is up and connected). A backup records configuration (how the switch is set up). You need both before maintenance.

Q.Which Exadata models does it apply to?

RoCE-based systems: X8M, X9M, X10M and X11M. Exadata X8 and earlier use InfiniBand switches, which have different tools.

Q.What is the single most useful section to add?

LLDP neighbors. It turns anonymous port numbers into named database servers and storage cells, so every other difference becomes actionable.

Q.Will Oracle Support accept a snapshot comparison?

It is useful supporting evidence, but Support normally also asks for show tech-support details, patchmgr logs and Exadata-side diagnostics.

10Conclusion

The RoCE switch snapshot report turns "the fabric looks fine" into proof. It records port state, neighbors, optics, error counters and lossless QoS on every switch, and its comparison names exactly what changed. Identical counts, the same LLDP neighbors on the same ports, stable error counters and active PFC mean the Exadata RoCE fabric is fully restored.

Master the Exadata RoCE fabric hands-on

KP Expert's live Oracle Exadata course covers the RoCE network fabric, switch patching with patchmgr, diagnostics like Sundiag and Exachk, and real troubleshooting scenarios.

Explore the Oracle Exadata courseBook a free demo class
Oracle Exadata series ยท DiagnosticsPublished by KP Expert
Recently Enrolled

Student enrolled in this course.

View course
Explore Courses

Latest from @kp__expert

Follow on Instagram
Loading Instagram posts...

AI Course Assistant

Share your details and goals to get the best course recommendations.

Recommended Courses

Select a course name to view full details.

Course Details
Enrollment & Contact
  • Review selected course and confirm your enrollment request.
  • Click checkout to move into the full payment process.
  • After payment submission, your enrollment is processed by our team.
Admissions Contact
Email: info@kpexpert.com
Phone: +91 92708 37105
Your submitted details
Name, email and phone will appear here.
Your request has been submitted successfully. Our team will contact you shortly.