ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency AdapterProduct Guide

Author
Updated
25 Sep 2019
Form Number
LP0901
PDF size
14 pages, 260 KB

Abstract

The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter delivers low sub-600µs latency, extremely high message rates, RoCE v2, NVMe over Fabric offloads and embedded PCIe switch. Mellanox provides the highest performance and lowest latency for the most demanding applications: High frequency trading, Machine Learning, Data Analytics, and more. This adapter ships ready to accept 10GbE, 25GbE or 40GbE network connections.

This product guide provides essential presales information to understand the ConnectX-5 Ex adapter and its key features, specifications, and compatibility. This guide is intended for technical specialists, sales specialists, sales engineers, IT architects, and other IT professionals who want to learn more about the adapter and consider its use in IT solutions.

Change History

Changes in the September 25 update:

Introduction

The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter delivers low sub-600µs latency, extremely high message rates, RoCE v2, NVMe over Fabric offloads and embedded PCIe switch. Mellanox provides the highest performance and lowest latency for the most demanding applications: High frequency trading, Machine Learning, Data Analytics, and more. This adapter ships ready to accept 10GbE, 25GbE or 40GbE network connections.

The following figure shows the ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter.

ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter
Figure 1. ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter (heatsink removed)

Did you know?

NVMe storage solutions are gaining popularity, offering very fast storage access. The evolving NVMe over Fabric (NVMe-oF) protocol leverages the Mellanox RDMA connectivity for remote access. ConnectX-5 offers further enhancements by providing NVMe-oF target offloads, enabling very efficient NVMe storage access with no CPU intervention, and thus improved performance and lower latency.

Part number information

The ordering information is listed in the following table.

Table 1. Ordering information
Part
number
Feature
code
Mellanox
equivalent
Description
4XC7A08229 B31C MCX516A-BDAT ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter

The adapter, when shipped as a stand-alone option, includes the following items:

  • One Mellanox ConnectX-5 Ex EN adapter
  • Two QSFP28 to SFP28 transceiver adapters (QSA28)
  • Full-height (3U) bracket attached with low-profile (2U) bracket included in the box
  • Documentation flyer

Supported transceivers and cables

The adapter has two empty QSFP28 cages. The adapter also includes two QSA28 adapters that convert the QSFP28 cages to SFP28 for 25Gb connectivity.

The adapter either supports a connection to a 1Gb, 10Gb, 25Gb or 40Gb switch or can share a connection to a 100Gb switch using a 4:1 breakout cable.

The following table lists the supported transceivers.

Table 2. Transceivers
Part number Feature code Description
1Gb transceivers
00FE333 A5DL SFP 1000Base-T (RJ-45) Transceiver
10Gb transceivers
46C3447 5053 SFP+ SR Transceiver
90Y9412 A1PM SFP+ LR Transceiver
25Gb transceivers
7G17A03537 AV1B Lenovo 25GBase-SR SFP28 Transceiver
40Gb transceivers
49Y7884 A1DR QSFP+ 40GBASE-SR4 Transceiver
00D9865 ASTM Lenovo QSFP+ 40GBASE-iSR4 Transceiver
Converters
00D9676 ARZH Mellanox QSFP to SFP+ Adapter
4G17A10853* B306 Mellanox QSA 100G to 25G Cable Adapter

* Two QSA adapters are included with the ConnectX-5 Ex adapter.

The following table lists the supported fiber optic cables and Active Optical Cables.

Table 3. Optical cables
Part number Feature code Description
LC-LC 0M3 Fiber Optic Cables (these cables require a transceiver)
00MN499 ASR5 Lenovo 0.5m LC-LC OM3 MMF Cable
00MN502 ASR6 Lenovo 1m LC-LC OM3 MMF Cable
00MN505 ASR7 Lenovo 3m LC-LC OM3 MMF Cable
00MN508 ASR8 Lenovo 5m LC-LC OM3 MMF Cable
00MN511 ASR9 Lenovo 10m LC-LC OM3 MMF Cable
00MN514 ASRA Lenovo 15m LC-LC OM3 MMF Cable
00MN517 ASRB Lenovo 25m LC-LC OM3 MMF Cable
00MN520 ASRC Lenovo 30m LC-LC OM3 MMF Cable
SFP+ 10Gb Active Optical Cables
00YL634 ATYX Lenovo 1M SFP+ to SFP+ Active Optical Cable
00YL637 ATYY Lenovo 3M SFP+ to SFP+ Active Optical Cable
00YL640 ATYZ Lenovo 5M SFP+ to SFP+ Active Optical Cable
00YL643 ATZ0 Lenovo 7M SFP+ to SFP+ Active Optical Cable
00YL646 ATZ1 Lenovo 15M SFP+ to SFP+ Active Optical Cable
00YL649 ATZ2 Lenovo 20M SFP+ to SFP+ Active Optical Cable
QSFP Active Optical Cables
00YL652 ATZ3 Lenovo 3m 40G QSFP+ to QSFP+ Active Optical Cable
00YL655 ATZ4 Lenovo 5m 40G QSFP+ to QSFP+ Active Optical Cable
00YL658 ATZ5 Lenovo 7m 40G QSFP+ to QSFP+ Active Optical Cable
00YL661 ATZ6 Lenovo 15m 40G QSFP+ to QSFP+ Active Optical Cable
00YL664 ATZ7 Lenovo 20m 40G QSFP+ to QSFP+ Active Optical Cable
7Z57A04256 AX42 Lenovo 1m 40G QSFP+ Active Optical Cable
QSFP 40Gb Optical Cables
00VX003 AT2U Lenovo 10m QSFP+ MPO-MPO OM3 MMF Cable
00VX005 AT2V Lenovo 30m QSFP+ MPO-MPO OM3 MMF Cable
SFP28 25Gb Active Optical Cables
7Z57A03541 AV1F Lenovo 3m 25G SFP28 Active Optical Cable
7Z57A03542 AV1G Lenovo 5m 25G SFP28 Active Optical Cable
7Z57A03543 AV1H Lenovo 10m 25G SFP28 Active Optical Cable
7Z57A03544 AV1J Lenovo 15m 25G SFP28 Active Optical Cable
7Z57A03545 AV1K Lenovo 20m 25G SFP28 Active Optical Cable
QSFP28 100Gb Breakout Active Optical Cables
7Z57A03551 AV1R Lenovo 3m 100G to 4x25G Breakout Active Optical Cable
7Z57A03552 AV1S Lenovo 5m 100G to 4x25G Breakout Active Optical Cable
7Z57A03553 AV1T Lenovo 10m 100G to 4x25G Breakout Active Optical Cable
7Z57A03554 AV1U Lenovo 15m 100G to 4x25G Breakout Active Optical Cable
7Z57A03555 AV1V Lenovo 20m 100G to 4x25G Breakout Active Optical Cable

The following table lists the supported direct-attach copper (DAC) cables.

Table 4. Copper cables
Part number Feature code Description
QSFP-to-QSFP Cables
49Y7890 A1DP 1m QSFP+-to-QSFP+ cable
49Y7891 A1DQ 3m QSFP+-to-QSFP+ cable
00D5810 A2X8 5m QSFP+ to QSFP+ Cable
00D5813 A2X9 7m QSFP+ to QSFP+ Cable
SFP+ 10Gb Passive DAC Cables
00D6288 A3RG 0.5m Passive DAC SFP+ Cable
90Y9427 A1PH 1m Passive DAC SFP+ Cable
00AY764 A51N 1.5m Passive DAC SFP+ Cable
00AY765 A51P 2m Passive DAC SFP+ Cable
90Y9430 A1PJ 3m Passive DAC SFP+ Cable
90Y9433 A1PK 5m Passive DAC SFP+ Cable
00D6151 A3RH 7m Passive DAC SFP+ Cable
SFP+ 10Gb Active DAC Cables
00VX111 AT2R Lenovo 1m Active DAC SFP+ Cables
00VX114 AT2S Lenovo 3m Active DAC SFP+ Cables
00VX117 AT2T Lenovo 5m Active DAC SFP+ Cables
SFP28 25Gb Passive DAC Cables
7Z57A03557 AV1W Lenovo 1m Passive 25G SFP28 DAC Cable
7Z57A03558 AV1X Lenovo 3m Passive 25G SFP28 DAC Cable
7Z57A03559 AV1Y Lenovo 5m Passive 25G SFP28 DAC Cable
QSFP28 100G-to-4x25G Breakout Cables
7Z57A03564 AV22 Lenovo 1m 100G QSFP28 to 4x25G SFP28 Breakout DAC Cable
7Z57A03565 AV23 Lenovo 3m 100G QSFP28 to 4x25G SFP28 Breakout DAC Cable
7Z57A03566 AV24 Lenovo 5m 100G QSFP28 to 4x25G SFP28 Breakout DAC Cable

Features

The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter has the following key features:

Ultra-low latency

Mellanox ConnectX-5 adapter cards feature exceedingly low latency and hardware-based RDMA technology that is perfect for meeting the needs of the financial services industry. Based on advanced technology providing sub-microsecond latency and a capacity of 200 million messages per second, ConnectX-5 offers the next generation of connectivity today.

Mellanox’s Messaging Accelerator (VMA) transparently boosts performance for socket-based applications without having to modify the applications. By offloading network processing from the OS, traffic passes directly to the network adapter from the application user space, bypassing the kernel IP stack and minimizing context switches, buffer copies, and interrupts. The result is a reduction in latency by 500% and an increase in application throughput, as compared to applications running on a standard network kernel stack.

Cloud, SDN, and NFV environment support

Mellanox Accelerated Switching and Packet Processing (ASAP2) Direct technology allows to offload vSwitch/vRouter by handling the data plane in the NIC hardware while maintaining the control plane unmodified. As a result there is significantly higher vSwitch/vRouter performance without the associated CPU load

The vSwitch/vRouter offload functions that are supported by ConnectX-5 include Overlay Networks (for example, VXLAN, NVGRE, MPLS, GENEVE, and NSH) headers’ encapsulation and de-capsulation, as well as stateless offloads of inner packets, packet headers’ re-write enabling NAT functionality, and more.

Moreover, the intelligent ConnectX-5 flexible pipeline capabilities, which include flexible parser and flexible match-action tables, can be programmed, which enable hardware offloads for future protocols.

ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VMs) within the server. With ConnectX-5 Network Function Virtualization (NFV), a VM can be used as a virtual appliance. With full data-path operations offloads as well as hairpin hardware capability and service chaining, data can be handled by the Virtual Appliance with minimum CPU utilization.

With these capabilities data center administrators benefit from better server utilization while reducing cost, power, and cable complexity, allowing more Virtual Appliances, Virtual Machines and more tenants on the same hardware.

Specifications

The adapter has the following technical specifications:

  • PCIe 4.0 x16 host interface
  • Support for MSI/MSI-X mechanisms
  • Two QSFP28 external connectors supporting a transceiver, converter, direct-attach copper (DAC) cable or active optical cable (AOC).
  • Data rates supported: 1, 10, 25 and 40 Gb/s Ethernet
  • TLP (Transaction Layer Packet) Processing Hints (TPH)
  • Embedded PCIe Switch: Up to 8 bifurcations
  • PCIe switch Downstream Port Containment (DPC) enablement for PCIe hot-plug
  • Access Control Service (ACS) for peer-to-peer secure communication
  • Advance Error Reporting (AER)
  • Process Address Space ID (PASID) Address Translation Services (ATS)
  • IBM CAPI v2 support (Coherent Accelerator Processor Interface)
  • Support for MSI/MSI-X mechanisms

Ethernet:

  • 40GbE / 25GbE / 10GbE / 1GbE
  • IEEE 802.3by, Ethernet Consortium 25, supporting all FEC modes
  • IEEE 802.3ba 40 Gigabit Ethernet
  • IEEE 802.3ae 10 Gigabit Ethernet
  • IEEE 802.3az Energy Efficient Ethernet
  • IEEE 802.3ap based auto-negotiation and KR startup
  • IEEE 802.3ad, 802.1AX Link Aggregation
  • IEEE 802.1Q, 802.1P VLAN tags and priority
  • IEEE 802.1Qau (QCN) Congestion Notification
  • IEEE 802.1Qaz (ETS)
  • IEEE 802.1Qbb (PFC)
  • IEEE 802.1Qbg
  • IEEE 1588v2
  • Jumbo frame support (9.6KB)

Enhanced Features:

  • Hardware-based reliable transport
  • Collective operations offloads
  • Vector collective operations offloads
  • PeerDirect RDMA (GPUDirect) communication acceleration
  • 64/66 encoding
  • Extended Reliable Connected transport (XRC)
  • Dynamically Connected Transport (DCT)
  • Enhanced Atomic operations
  • Advanced memory mapping support, allowing user mode registration and remapping of memory (UMR)
  • On demand paging (ODP)
  • MPI Tag Matching
  • Rendezvous protocol offload
  • Out-of-order RDMA supporting Adaptive Routing
  • Burst buffer offload
  • In-Network Memory registration-free RDMA memory access

CPU Offloads:

  • RDMA over Converged Ethernet (RoCE)
  • TCP/UDP/IP stateless offload
  • LSO, LRO, checksum offload
  • RSS (also on encapsulated packet), TSS, HDS, VLAN and MPLS tag insertion/stripping, Receive flow steering
  • Data Plane Development Kit (DPDK) for kernel bypass applications
  • Open VSwitch (OVS) offload using ASAP2:
    • Flexible match-action flow tables
    • Tunneling encapsulation/ de-capsulation
  • Intelligent interrupt coalescence
  • Header rewrite supporting hardware offload of NAT router

Storage Offloads:

  • NVMe over Fabric offloads for target machine
  • Erasure Coding offload – offloading Reed Solomon calculations
  • T10 DIF – Signature handover operation at wire speed, for ingress and egress traffic
  • Storage protocols: SRP, iSER, NFS RDMA, SMB Direct, NVMe-oF

Overlay Networks:

  • RoCE over Overlay Networks
  • Stateless offloads for overlay network tunneling protocols
  • Hardware offload of encapsulation and decapsulation of VXLAN, NVGRE, and GENEVE overlay networks

Hardware-Based I/O Virtualization

  • Single Root IOV
  • Address translation and protection
  • VMware NetQueue support
  • SR-IOV: Up to 1K Virtual Functions
  • SR-IOV: Up to 16 Physical Functions per host
  • Virtualization hierarchies (e.g., NPAR and Multi-Host, when enabled):
    • Virtualizing Physical Functions on a physical port
    • SR-IOV on every Physical Function
  • Configurable and user-programmable QoS
  • Guaranteed QoS for VMs

HPC Software Libraries:

  • Open MPI, IBM PE, OSU MPI (MVAPICH/2), Intel MPI
  • Platform MPI, UPC, Open SHMEM

Management and Control:

  • PLDM for Monitor and Control DSP0248
  • PLDM for Firmware Update DSP0267
  • SDN management interface for managing the eSwitch
  • I2C interface for device control and configuration
  • General Purpose I/O pins
  • SPI interface to Flash
  • JTAG IEEE 1149.1 and IEEE 1149.6

Remote Boot:

  • Remote boot over Ethernet
  • Remote boot over iSCSI
  • Unified Extensible Firmware Interface (UEFI)
  • Pre-execution Environment (PXE)

Server support

The following table lists the ThinkSystem servers that are compatible.

Table 5. ThinkSystem server support
Description and part number E 1S Intel 2S Intel AMD 4S Intel Dense/ Blade
SE350 (7Z46 / 7D1X)
ST50 (7Y48/7Y50)
ST250 (7Y45/7Y46)
SR150 (7Y54)
SR250 (7Y51/7Y52)
ST550 (7X09/7X10)
SR530 (7X07/7X08)
SR550 (7X03/7X04)
SR570 (7Y02/7Y03)
SR590 (7X98/7X99)
SR630 (7X01/7X02)
SR650 (7X05/7X06)
SR670 (7Y36/7Y37/7Y38)
SR635 (7Y98 / 7Y99)
SR655 (7Y00 / 7Z01)
SR850 (7X18/7X19)
SR860 (7X69/7X70)
SR950 (7X11/12/13)
SD530 (7X21)
SD650 (7X58)
SN550 (7X16)
SN850 (7X15)
ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter, 4XC7A08229 N N N N N N N N N N Y Y N N N Y Y Y Y N N N

Operating system support

The adapter supports the following operating systems:

Tip: This table is automatically generated based on data from Lenovo ServerProven.

Table 6. Operating system support for  ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter , 4XC7A08229
Operating systems
SD530 (Gen 2)
SR630 (Gen 2)
SR650 (Gen 2)
SR850 (Gen 2)
SR860 (Gen 2)
SR950 (Gen 2)
SD530 (Gen 1)
SR630 (Gen 1)
SR650 (Gen 1)
SR850 (Gen 1)
SR860 (Gen 1)
SR950 (Gen 1)
Microsoft Windows Server 2012 R2 N N N N N N Y Y Y Y Y Y
Microsoft Windows Server 2016 Y Y Y Y Y Y Y Y Y Y Y Y
Microsoft Windows Server 2019 Y Y Y Y Y Y Y Y Y Y Y Y
Microsoft Windows Server version 1709 N N N N N N Y Y Y Y Y Y
Red Hat Enterprise Linux 6.10 N N N N N N Y Y Y Y Y Y
Red Hat Enterprise Linux 6.9 N N N N N N Y Y Y Y Y Y
Red Hat Enterprise Linux 7.3 N N N N N N Y Y Y Y N Y
Red Hat Enterprise Linux 7.4 N N N N N N Y Y Y Y Y Y
Red Hat Enterprise Linux 7.5 N N N N N N Y Y Y Y Y Y
Red Hat Enterprise Linux 7.6 Y Y Y Y Y Y Y Y Y Y Y Y
Red Hat Enterprise Linux 7.7 Y Y Y Y Y Y Y Y Y Y Y Y
Red Hat Enterprise Linux 8.0 Y Y Y Y Y Y Y Y Y Y Y Y
Red Hat Enterprise Linux 8.1 Y Y Y Y Y Y Y Y Y Y Y Y
SUSE Linux Enterprise Server 11 SP4 N N N N N N Y Y Y Y Y Y
SUSE Linux Enterprise Server 12 SP2 N N N N N N Y Y Y Y N Y
SUSE Linux Enterprise Server 12 SP3 N N N N N N Y Y Y Y Y Y
SUSE Linux Enterprise Server 12 SP4 Y Y Y Y Y Y Y Y Y Y Y Y
SUSE Linux Enterprise Server 15 Y Y Y Y Y Y Y Y Y Y Y Y
SUSE Linux Enterprise Server 15 SP1 Y Y Y Y Y Y Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.0 U3 N N N N N N Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.5 N N N N N N Y Y Y Y N Y
VMware vSphere Hypervisor (ESXi) 6.5 U1 N N N N N N Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.5 U2 Y Y Y Y Y Y Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.5 U3 Y Y Y Y Y Y Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.7 N N N N N N Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.7 U1 Y Y Y Y Y Y Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.7 U2 Y Y Y Y Y Y Y Y Y Y Y Y
VMware vSphere Hypervisor (ESXi) 6.7 U3 Y Y Y Y Y Y Y Y Y Y Y Y

Physical specifications

The adapter is a low-profile adapter with the following dimensions:

  • Length: 14.2 cm (5.6 in)
  • Height: 6.9 cm (2.71 in)

Operating environment

The adapter is supported in the following environment:

  • Temperature (operating): 0 to 55 °C (32 to 131 °F)
  • Temperature (storage): -40 to 70 °C (-40 to 158 °F)
  • Humidity (operating): 10 to 90% non-condensing
  • Humidity (storage): 5 to 90% non-condensing

Warranty

One-year limited warranty. When installed in a supported server, the adapter assumes the system’s base warranty and any warranty upgrade.

Agency approvals

The adapter conforms to the following standards:

  • Safety: CB / cTUVus / CE
  • EMC: CE / FCC / VCCI / ICES / RCM
  • RoHS: RoHS-R6

Top-of-rack Ethernet switches

The following table lists the Ethernet LAN switches that are offered by Lenovo.

Table 7. Ethernet LAN switches
Part number Description
1 Gb Ethernet Rack switches
7Y810011WW Lenovo ThinkSystem NE0152T RackSwitch (Rear to Front)
7Z320O11WW Lenovo ThinkSystem NE0152TO RackSwitch (Rear to Front, ONIE)
7159BAX Lenovo RackSwitch G7028 (Rear to Front)
7159CAX Lenovo RackSwitch G7052 (Rear to Front)
7159G52 Lenovo RackSwitch G8052 (Rear to Front)
7165H1X Juniper EX2300-C PoE Switch
7165H2X Juniper EX2300-24p PoE Switch
1 Gb Ethernet Campus switches
7Z340011WW Lenovo CE0128TB Switch (3-Year Warranty)
7Z360011WW Lenovo CE0128TB Switch (Limited Lifetime Warranty)
7Z340012WW Lenovo CE0128PB Switch (3-Year Warranty)
7Z360012WW Lenovo CE0128PB Switch (Limited Lifetime Warranty)
7Z350021WW Lenovo CE0152TB Switch (3-Year Warranty)
7Z370021WW Lenovo CE0152TB Switch (Limited Lifetime Warranty)
7Z350022WW Lenovo CE0152PB Switch (3-Year Warranty)
7Z370022WW Lenovo CE0152PB Switch (Limited Lifetime Warranty)
10 Gb Ethernet switches
7159A1X Lenovo ThinkSystem NE1032 RackSwitch (Rear to Front)
7159B1X Lenovo ThinkSystem NE1032T RackSwitch (Rear to Front)
7159C1X Lenovo ThinkSystem NE1072T RackSwitch (Rear to Front)
7159CRW Lenovo RackSwitch G8272 (Rear to Front)
7159GR6 Lenovo RackSwitch G8296 (Rear to Front)
25 Gb Ethernet switches
7159E1X Lenovo ThinkSystem NE2572 RackSwitch (Rear to Front)
7Z210O21WW Lenovo ThinkSystem NE2572O RackSwitch (Rear to Front, ONIE)
100 Gb Ethernet switches
7159D1X Lenovo ThinkSystem NE10032 RackSwitch (Rear to Front)
7Z210O11WW Lenovo ThinkSystem NE10032O RackSwitch (Rear to Front, ONIE)

For more information, see the list of Product Guides in the following switch categories:

Related product families

Product families related to this document are the following:

Trademarks

Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.

The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
RackSwitch
ServerProven®
ThinkSystem

The following terms are trademarks of other companies:

Intel® is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.

Linux® is a trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft®, Windows Server®, and Windows® are trademarks of Microsoft Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.