NVMe-Rich Configurations of the ThinkSystem SR650

The Lenovo ThinkSystem SR650 is a mainstream 2U 2-socket server with industry-leading reliability, management, and security features, and is designed to handle a wide range of workloads.

New to the SR650 is support for up to 24 NVMe solid-state drives. With this support, the SR650 is an excellent choice for workloads that need large amounts of low-latency high-bandwidth storage, including virtualized clustered SAN solutions, software-defined storage, and applications leveraging NVMe over Fabrics (NVMeOF).

Figure 1. ThinkSystem SR650 with 24 NVMe drives

Three new configurations are now available:

• 16 NVMe drives + 8 SAS/SATA drives
• 20 NVMe drives
• 24 NVMe drives

NVMe (Non-Volatile Memory Express) is a technology that overcomes SAS/SATA SSD performance limitations by optimizing hardware and software to take full advantage of flash technology. Intel Xeon processors efficiently transfer data in fewer clock cycles with the NVMe optimized software stack compared to the legacy AHCI stack, thereby reducing latency and overhead. NVMe SSDs connect directly to the processor via the PCIe bus, further reducing latency. NVMe drives are characterized by very high bandwidth and very low latency.

These configurations are available configure-to-order (CTO) in the Lenovo Data Center Solution Configurator (DCSC), https://dcsc.lenovo.com. The following table lists the feature codes related to the NVMe drive subsystem. The configurator will derive any additional components that are needed.

Note the following requirements for any of the three NVMe-rich configurations:

• Two processors
• No high-thermal processors:
  • 200 W or 205 W TDP are not supported
  • Gold 6126T, Gold 6144, Gold 6146, or Platinum 8160T processors are not supported
  • Gold 6230N, Gold 6240Y, and Gold 6244 processors are not supported
• No GPU adapters installed
• No PCIe flash adapters installed
• No PCIe adapters with more than 25 W TDP installed
• 1100 W or 1600 W power supplies installed.
• Ambient temperature of up to 30 °C (86 °F)
• If a fan fails and the ambient temperature is above 27 °C, system performance may be reduced.

Although not required, it is expected that these configurations will be fully populated with NVMe drives. Maximum performance is achieved when all NVMe drive bays are filled with drives.

To verify support and ensure that the right power supply is chosen for optimal performance, validate your server configuration using the latest version of the Lenovo Capacity Planner:
http://datacentersupport.lenovo.com/us/en/solutions/lnvo-lcp

See the ThinkSystem SR650 product guide for the complete list of NVMe drives that are supported in the server: https://lenovopress.com/lp0644#drives-for-internal-storage

The NVMe drives listed in the following table are not supported in the three NVMe-rich configurations.

The 16x NVMe drive configuration has the following features:

• 16 NVMe 2.5-inch drive bays plus eight SAS/SATA 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap).
• The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
• The eight SAS/SATA drive bays are connected to a supported 8-port RAID adapter or SAS HBA.
• One PCIe x16 slot is available for high-speed networking such as a 100 GbE adapter, InfiniBand or OPA adapter. If you elect not to configure the eight SAS/SATA drive bays, then you can free up an additional x8 slot for a second networking adapter.
• The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
  • ThinkSystem 1Gb 2-port RJ45 LOM
  • ThinkSystem 1Gb 4-port RJ45 LOM
  • ThinkSystem 10Gb 2-port Base-T LOM
  • ThinkSystem 10Gb 2-port SFP+ LOM
  • ThinkSystem 10Gb 4-port Base-T LOM
  • ThinkSystem 10Gb 4-port SFP+ LOM
• Additional support for one or two M.2 drives, if needed

The 16x NVMe drive configuration has the following performance characteristics:

• Balanced NVMe configuration. In this 16-NVMe drive configuration, each processor is connected to 8 drives. Such a balanced configuration ensures maximum performance by ensuring the processors are equally occupied handling I/O requests to and from the NVMe drives.
• No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.

The 20x NVMe drive configuration has the following features:

• 20 NVMe 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap). The other 4 bays are unavailable and are covered by a 4-bay blank.
• The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
• One PCIe x8 slot is available for networking or other needs. The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
  • ThinkSystem 1Gb 2-port RJ45 LOM
  • ThinkSystem 1Gb 4-port RJ45 LOM
  • ThinkSystem 10Gb 2-port Base-T LOM
  • ThinkSystem 10Gb 2-port SFP+ LOM
  • ThinkSystem 10Gb 4-port Base-T LOM
  • ThinkSystem 10Gb 4-port SFP+ LOM
• Additional support for one or two M.2 drives, if needed

The 20x NVMe drive configuration has the following performance characteristics:

• No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.
• Near-balanced NVMe configuration. Unlike the 16-drive and 24-drive configurations, that 20-drive configuration has eight NVMe drives connected to processor 1, and 12 NVMe drives connected to processor 2. As a result, we recommend you to only choose this configuration if you need the additional capacity that four drives provide above the 16-drive configuration, and your workload can fully operate without an equal number of drives connected to each processor.

The 24x NVMe drive configuration has the following features:

• 24 NVMe 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap).
• The NVMe drives are connected to the processors via NVMe Switch Adapters. The onboard NVMe connectors are routed to a riser card installed in Riser slot 1.
• Two x16 slots (one connected to each processor) are available for high-speed networking such as a 100 GbE adapter, InfiniBand or OPA adapter.
• The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
  • ThinkSystem 1Gb 2-port RJ45 LOM
  • ThinkSystem 1Gb 4-port RJ45 LOM
  • ThinkSystem 10Gb 2-port Base-T LOM
  • ThinkSystem 10Gb 2-port SFP+ LOM
  • ThinkSystem 10Gb 4-port Base-T LOM
  • ThinkSystem 10Gb 4-port SFP+ LOM
• Additional support for one or two M.2 drives, if needed

The 24x NVMe drive configuration has the following performance characteristics:

• Balanced NVMe configuration. In this 24-NVMe drive configuration, each processor is connected to 12 drives. Such a balanced configuration provides maximum performance by ensuring the processors are equally occupied handling I/O requests to and from the NVMe drives.
• 2:1 oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration each drive is allocated 2 lanes from the processor, resulting in a 2:1 oversubscription of the PCIe lanes. With 24 drives, there are simply not enough PCIe lanes in a two-socket server to support no oversubscription. As a result, the design objective is to minimize the oversubscription while still maintaining balance across all lanes.
• Balanced open slots. This configuration has two open PCIe x16 slots, one connected to each processor. These slots could be used for a pair of high-speed network cards and the result would be balanced configuration.

The following two field upgrade option kits are available to upgrade existing SAS/SATA or AnyBay drive configurations based on the 24x 2.5" chassis (feature code AUVV) to either the 20-drive or 24-drive NVMe configurations.

These kits include drive backplanes and required NVMe cables, power cables, drive bay fillers, and NVMe switch adapters.

The ThinkSystem SR650 U.2 20-Bays Upgrade Kit includes the following components:

• Two 810-4P NVMe Switch Adapters
• Three 1610-4P NVMe Switch Adapters
• One x16/x8 PCIe Riser for Riser 1
• One x16/x16 PCIe Riser for Riser 2
• Three 8-bay NVMe drive backplanes
• One 4-bay drive bay filler
• NVMe and power cables
• Brackets and screws
• Drive bay labels for the front bezel

The ThinkSystem SR650 U.2 24-Bays Upgrade Kit includes the following components:

• Four 810-4P NVMe Switch Adapters
• One 1610-8P NVMe Switch Adapter
• One x16/x8/x16 PCIe Riser for Riser 1
• One x16/x16 PCIe Riser for Riser 2
• Three 8-bay NVMe drive backplanes
• NVMe and power cables
• Brackets and screws
• Drive bay labels for the front bezel

For more information, see these resources:

• ThinkSystem SR650 product guide
  https://lenovopress.com/lp0644-lenovo-thinksystem-sr650-server
• Product Guides for ThinkSystem NVMe drives:
  https://lenovopress.com/servers/options/drives#term=nvme&rt=product-guide
• Paper, Implementing NVMe Drives on Lenovo Servers
  https://lenovopress.com/lp0508-implementing-nvme-drives-on-lenovo-servers
• Paper, Comparing the Effect of PCIe Host Connections on NVMe Drive Performance
  https://lenovopress.com/lp0865-comparing-the-effect-of-pcie-host-connections-on-nvme-drive-performance
• Data Center Solution Configurator (DCSC) configurator
  https://dcsc.lenovo.com/