Mellanox ConnectX-3 and ConnectX-3 Pro Adapters

High-performance computing (HPC) solutions require high bandwidth, low latency components with CPU offloads to get the highest server efficiency and application productivity. The Mellanox ConnectX-3 and ConnectX-3 Pro network adapters for System x® servers deliver the I/O performance that meets these requirements.

Mellanox's ConnectX-3 and ConnectX-3 Pro ASIC delivers low latency, high bandwidth, and computing efficiency for performance-driven server applications. Efficient computing is achieved by offloading from the CPU routine activities, which makes more processor power available for the application. Network protocol processing and data movement impacts, such as InfiniBand RDMA and Send/Receive semantics, are completed in the adapter without CPU intervention. RDMA support extends to virtual servers when SR-IOV is enabled. Mellanox's ConnectX-3 advanced acceleration technology enables higher cluster efficiency and scalability of up to tens of thousands of nodes.

Figure 1. Mellanox Connect X-3 10GbE Adapter for System x (3U bracket shown)

Mellanox Ethernet and InfiniBand network server adapters provide a high-performing interconnect solution for enterprise data centers, Web 2.0, cloud computing, and HPC environments, where low latency and interconnect efficiency is paramount. In additiona, Virtual Protocol Interconnect (VPI) offers flexibility in InfiniBand and Ethernet port designations.

With the new ConnectX-3 Pro adapter, you can implement VXLAN and NVGRE offload engines to accelerate virtual LAN ID processing, ideal for public and private cloud configurations.

Table 1 shows the part numbers and feature codes for the adapters.

The part numbers include the following:

• One adapter with a full-height (3U) bracket attached
• Additional low-profile (2U) bracket included in the box
• Documentation

Figure 2 shows the Mellanox ConnectX-3 40GbE / FDR IB VPI Adapter (the shipping adapter includes a heatsink over the ASIC but the figure does not show this heatsink)

Figure 2. Mellanox ConnectX-3 40GbE / FDR IB VPI Adapter for System x with 3U bracket (attached heatsink removed)

The 10GbE Adapter (00D9690) supports the direct-attach copper (DAC) twin-ax cables, transceivers, and optical cables that are listed in the following table.

The Mellanox ConnectX-3 40GbE / FDR IB VPI Adapters have the following features:

• Two QSFP ports supporting FDR-14 InfiniBand or 40 Gb Ethernet
• Low-profile form factor adapter with 2U bracket (3U bracket available for CTO orders)
• PCI Express 3.0 x8 host-interface (PCIe 2.0 and 1.1 compatible)
• Support for InfiniBand FDR speeds of up to 56 Gbps (auto-negotiation FDR-10, DDR and SDR)
• Support for Virtual Protocol Interconnect (VPI), which enables one adapter for both InfiniBand and 10/40 Gb Ethernet. Supports three configurations:
  • 2 ports InfiniBand
  • 2 ports Ethernet
  • 1 port InfiniBand and 1 port Ethernet
• SR-IOV support; 16 virtual functions supported by KVM and Hyper-V (OS dependant) up to a maximum of 127 virtual functions supported by the adapter
• Enables Low Latency RDMA over 40Gb Ethernet (supported with both non-virtualized and SR-IOV enabled virtualed servers) -- latency as low as 1 µs
• Microsoft VMQ / VMware NetQueue support
• Sub 1 µs InfiniBand MPI ping latency
• Support for QSFP to SFP+ for 10 GbE support
• Traffic steering across multiple cores
• Legacy and UEFI PXE network boot support (Ethernet mode only)

The Mellanox ConnectX-3 Pro ML2 2x40GbE/FDR VPI Adapter and ThinkSystem Mellanox ConnectX-3 Pro ML2 FDR 2-Port QSFP VPI Adapter have the same features as the ConnectX-3 40GbE / FDR IB VPI Adapter with these additions:

• Mezzanine LOM Generation 2 (ML2) form factor
• Offers NVGRE hardware offloads
• Offers VXLAN hardware offloads

Performance

Based on Mellanox's ConnectX-3 technology, these adapters provide a high level of throughput performance for all network environments by removing I/O bottlenecks in mainstream servers that are limiting application performance. With the FDR VPI IB/E Adapter, servers can achieve up to 56 Gb transmit and receive bandwidth. Hardware-based InfiniBand transport and IP over InfiniBand (IPoIB) stateless off-load engines handle the segmentation, reassembly, and checksum calculations that otherwise burden the host processor.

RDMA over InfiniBand and RDMA over Ethernet further accelerate application run time while reducing CPU utilization. RDMA allows very high-volume transaction-intensive applications typical of HPC and financial market firms, as well as other industries where speed of data delivery is paramount. With the ConnectX-3-based adapter, highly compute-intensive tasks running on hundreds or thousands of multiprocessor nodes, such as climate research, molecular modeling, and physical simulations, can share data and synchronize faster, resulting in shorter run times.

In data mining or web crawl applications, RDMA provides the needed boost in performance to enable faster search by solving the network latency bottleneck that is associated with I/O cards and the corresponding transport technology in the cloud. Various other applications that benefit from RDMA with ConnectX-3 include Web 2.0 (Content Delivery Network), business intelligence, database transactions, and various cloud computing applications. Mellanox ConnectX-3's low power consumption provides clients with high bandwidth and low latency at the lowest cost of ownership.

TCP/UDP/IP acceleration

Applications utilizing TCP/UDP/IP transport can achieve industry leading data throughput. The hardware-based stateless offload engines in ConnectX-3 reduce the CPU impact of IP packet transport, allowing more processor cycles to work on the application.

NVGRE and VXLAN hardware offloads

The Mellanox ConnectX-3 Pro adapters offers NVGRE and VXLAN hardware offload engines which provide additional performance benefits, especially for public or private cloud implementations and virtualized environments. These offloads ensure that Overlay Networks are enabled to handle the advanced mobility, scalability, serviceability that is required in today's and tomorrow's data center. These offloads dramatically lower CPU consumption, thereby reducing cloud application cost, facilitating the highest available throughput, and lowering power consumption.

Software support

All Mellanox adapter cards are supported by a full suite of drivers for Microsoft Windows, Linux distributions, and VMware. ConnectX-3 adapters support OpenFabrics-based RDMA protocols and software. Stateless offload is fully interoperable with standard TCP/ UDP/IP stacks. ConnectX-3 adapters are compatible with configuration and management tools from OEMs and operating system vendors.

InfiniBand specifications (all three FDR adapters):

• Supports InfiniBand FDR-14, FDR-10, QDR, DDR, and SDR
• IBTA Specification 1.2.1 compliant
• RDMA, Send/Receive semantics
• Hardware-based congestion control
• 16 million I/O channels
• 256 to 4 KB MTU, 1 GB messages
• Nine virtual lanes: Eight data and one management
• NV-GRE hardware offloads (ConnectX-3 Pro only)
• VXLAN hardware offloads (ConnectX-3 Pro only)

Enhanced InfiniBand specifications:

• Hardware-based reliable transport
• Hardware-based reliable multicast
• Extended Reliable Connected transport
• Enhanced Atomic operations
• Fine-grained end-to-end quality of server (QoS)

Ethernet specifications:

• IEEE 802.3ae 10 GbE
• IEEE 802.3ba 40 GbE (all three FDR adapters)
• IEEE 802.3ad Link Aggregation
• IEEE 802.3az Energy Efficient Ethernet
• IEEE 802.1Q, .1P VLAN tags and priority
• IEEE 802.1Qbg
• IEEE P802.1Qaz D0.2 Enhanced Transmission Selection (ETS)
• IEEE P802.1Qbb D1.0 Priority-based Flow Control
• IEEE 1588v2 Precision Clock Synchronization
• Multicast
• Jumbo frame support (9600B)
• 128 MAC/VLAN addresses per port

Hardware-based I/O virtualization:

• Address translation and protection
• Multiple queues per virtual machine
• VMware NetQueue support
• 16 virtual function SR-IOV supported with Linux KVM
• VXLAN and NVGRE (ConnectX-3 Pro adapters)

SR-IOV features:

• Address translation and protection
• Dedicated adapter resources
• Multiple queues per virtual machine
• Enhanced QoS for vNICs
• VMware NetQueue support

Additional CPU offloads:

• TCP/UDP/IP stateless offload
• Intelligent interrupt coalescence
• Compliant with Microsoft RSS and NetDMA

Management and tools:

InfiniBand:

• Interoperable with OpenSM and other third-party subnet managers
• Firmware and debug tools (MFT and IBDIAG)

Ethernet:

• MIB, MIB-II, MIB-II Extensions, RMON, and RMON 2
• Configuration and diagnostic tools
• NC-SI (ML2 adapters only)

Protocol support:

• Open MPI, OSU MVAPICH, Intel MPI, MS MPI, and Platform MPI
• TCP/UDP, EoIB and IPoIB
• uDAPL

The adapters have the following dimensions:

• Height: 168 mm (6.60 in)
• Width: 69 mm (2.71 in)
• Depth: 17 mm (0.69 in)
• Weight: 208 g (0.46 lb)

Approximate shipping dimensions:

• Height: 189 mm (7.51 in)
• Width: 90 mm (3.54 in)
• Depth: 38 mm (1.50 in)
• Weight: 450 g (0.99 lb)

The following tables list the ThinkSystem servers that are compatible.

The adapters are supported in the System x servers that are listed in the following tables.

Memory requirements: Ensure that your server has sufficient memory available to the adapters. The first adapter requires 8 GB of RAM in addition to the memory allocated to operating system, applications, and virtual machines. Any additional adapters installed each require 4 GB of RAM.

The adapters support the following operating systems:

• Mellanox ConnectX-3 10 GbE Adapter, 00D9690
• Mellanox ConnectX-3 40GbE / FDR IB VPI Adapter, 00D9550
• Mellanox ConnectX-3 Pro ML2 2x40GbE/FDR VPI Adapter, 00FP650
• ThinkSystem Mellanox ConnectX-3 Pro ML2 FDR 2-Port QSFP VPI Adapter, 7ZT7A00501

* InfiniBand mode not supported with VMware: With VMware, these adapters are supported only in Ethernet mode. InfiniBand is not supported.

The adapters have the following regulatory approvals:

• EN55022
• EN55024
• EN60950-1
• EN 61000-3-2
• EN 61000-3-3
• IEC 60950-1
• FCC Part 15 Class A
• UL 60950-1
• CSA C22.2 60950-1-07
• VCCI
• NZ AS3548 / C-tick
• RRL for MIC (KCC)
• BSMI (EMC)
• IECS-003:2004 Issue 4

The adapters are supported in the following environment:

• Operating temperature:
  • 0 - 55° C (-32 to 131° F) at 0 - 914 m (0 - 3,000 ft)
  • 10 - 32° C (50 - 90° F) at 914 to 2133 m (3,000 - 7,000 ft)
• Relative humidity: 20% - 80% (noncondensing)
• Maximum altitude: 2,133 m (7,000 ft)
• Air flow: 200 LFM at 55° C
• Power consumption:
  • Power consumption (typical): 8.8 W typical (both ports active)
  • Power consumption (maximum): 9.4 W maximum for passive cables only,
  • 13.4 W maximum for active optical modules

The following 10 Gb Ethernet top-of-rack switches are supported.

For more information, see the Lenovo Press Product Guides in the 10Gb top-of-rack switch category:
https://lenovopress.com/networking/tor/10gb

One year limited warranty. When installed in a supported server, these cards assume the server’s base warranty and any warranty upgrades.

For more information, refer to these documents:

• Lenovo ThinkSystem networking options product page
  https://lenovopress.com/lp0765-networking-options-for-thinksystem-servers

• ServerProven compatibility
  http://static.lenovo.com/us/en/serverproven/xseries/lan/matrix.shtml
• Mellanox User Manual - Mellanox ConnectX-3 40GbE / FDR IB VPI Adapter
  http://bit.ly/YJ3R0x
• Mellanox User Manual - Mellanox Connect X-3 10GbE Adapter
  http://bit.ly/YJ4oQa