Lenovo ThinkSystem SD650 Server (Xeon SP Gen 2)

The ThinkSystem SD650 direct water cooled server is an open, flexible, and simple data center solution for users of technical computing, grid deployments, analytics workloads, and large-scale cloud and virtualization infrastructures. The direct water cooled solution is designed to operate by using warm water, up to 50°C (122°F). Chillers are not needed for most customers, meaning even greater savings and a lower total cost of ownership.

Now with second-generation Intel Xeon Scalable processors, the ThinkSystem SD650 server and the n1200 DWC enclosure are designed to optimize density and performance within typical data center infrastructure limits. The n1200 DWC enclosure is a 6U rack mount unit that fits in a standard 19-inch rack and houses up to 12 water-cooled servers in 6 trays.

The two ThinkSystem SD650 servers are shown installed in a tray in the following figure.

Figure 1. Two Lenovo ThinkSystem SD650 servers on the Compute Tray that provides water cooling

The ThinkSystem SD650 server and n1200 DWC enclosure with direct water cooling provide the ultimate in data center cooling efficiencies and performance. You can obtain upwards of 40% energy savings in data center expenses with the reduced need for air conditioning and air chillers. Direct water cooling means that the processors installed in the servers can enable the Intel Turbo Boost performance feature more often to further increase processor performance.

The ThinkSystem SD650 direct water cooled offering is a dense platform from Lenovo that achieves maximum performance with innovative warm-water cooling functionality. The DWC offering includes a dense chassis and two half-wide compute nodes on a compute tray, all fitting in a standard rack.

Direct water cooling is part of Lenovo’s Neptune family of liquid cooling technologies which drive both greater energy efficiency and higher performance.  Lenovo Neptune’s approach uses liquid cooling to dissipate heat from systems with high thermal output, which enables data centers to run up to 50% more efficiently while maintaining uncompromised performance and preserving a dense data center footprint.

With direct water cooling, Lenovo drives increased compute density, performance, and cooling efficiency for High Performance Computing and other workloads that require dense compute performance, such as Cloud, Grid, and Analytics. Direct water cooling is achieved by circulating the cooling water directly through cold plates that contact the CPU thermal case, DIMMs, and other high-heat-producing components in the server.

One of the main advantages of direct water cooling is the water can be relatively warm and still be very effective, as water conducts heat much more effectively than air. Depending on the server configuration, 85 - 90% of the heat is removed by water cooling; the rest can easily be managed by a standard computer room air conditioner. With allowable inlet temperatures for the water being as high as 50°C (122°F), in many cases the water can be cooled by using ambient air and chilled water and a heat exchanger is not required.

The rear view of the two ThinkSystem SD650 servers in the water-cooled tray is shown in the following figure.

Figure 2. Rear view of two SD650 servers showing the water inlet and outlet

Designed to industry-standards, the SD650 is a general-purpose platform that is designed to give customers a flexible IT infrastructure. Customized solutions can be configured to provide an application-appropriate platform with a choice of servers, networking switches, adapters, and racks.

ThinkSystem SD650 servers provide a dense, flexible solution with a low TCO. The half-wide, dual-socket server is designed for data centers that require high performance but are constrained by floor space. By taking up less physical space in the data center, the SD650 enhances density and supports the Intel Xeon Scalable Family of processors up to 28 cores and up to 240W TDP. The SD650 is designed to support future processors of up to 245W, thus providing for even more performance per server. The SD650 contains only essential components in the base architecture to provide a cost-optimized platform.

The NeXtScale n1200 DWC Enclosure is an efficient, 6U, 12-node chassis with no built-in networking or switching capabilities; therefore, it requires no chassis-level management. Sensibly designed to provide shared, high-efficiency power and cooling for housed servers, the n1200 DWC enclosure is designed to scale with your business needs.

Scalability and performance

The ThinkSystem SD650 server NeXtScale n1200 DWC chassis offer the following features to boost performance, improve scalability, and reduce costs:

• Up to 12 compute nodes, each with two second-generation Intel Xeon Scalable Family processors, 12 TruDDR4 DIMMs, and one PCIe slot, up to two drive bays, in 6U of rack space. It is a highly dense, scalable, and price-optimized offering.
• The second-generation Intel Xeon Scalable Family processors improves productivity by offering superior system performance. Supported processors have up to 28 cores, core speeds of up to 3.6 GHz, TDP of up to 240W, DDR4 memory operating at 2933 MHz, and UPI interconnect links running at 10.4 GTps.
• Two processors, up to 56 cores, and up to 112 threads maximize the concurrent execution of multi-threaded applications.
• Intelligent and adaptive system performance with Intel Turbo Boost Technology 2.0 allows CPU cores to run at maximum speeds during peak workloads by temporarily going beyond processor TDP.
• By providing better cooling than possible with air cooling, direct water cooled servers optimize Turbo Boost 2.0 enablement. With water cooling, it may be possible to operate the processors in continuous Turbo mode.
• Intel Hyper-Threading Technology boosts performance for multithreaded applications by enabling simultaneous multithreading within each processor core, up to two threads per core.
• Intel Virtualization Technology integrates hardware-level virtualization hooks that allow operating system vendors to better use the hardware for virtualization workloads.
• Intel Speed Select Technology provides improvements in server utilization and guaranteed per-core performance service levels with more granular control over processor performance.
• Intel Deep Learning Boost (Vector Neural Network Instruction set or VNNI) is designed to deliver significant, more efficient Deep Learning (Inference) acceleration for high-performance Artificial Intelligence (AI) workloads.
• Intel Advanced Vector Extensions 512 (AVX-512) enable acceleration of enterprise-class workloads, including databases, and enterprise resource planning.
• Each processor has six memory channels with memory speeds of up to 2933 MHz to maximize system performance.
• A total of 12 DIMMs of TruDDR4 ECC memory provide speed, high availability, and a memory capacity of up to 1.5 TB. Higher memory capacity by using Persistent Memory.
• Supports the new Intel Optane DC Persistent Memory; up to four Data Center Persistent Memory Modules (DCPMMs) can be installed in conjunction with regular memory DIMMs, two per processor. DCPMMs are up to 512 GB each, for a total of up to 2 TB of Persistent Memory.
• One full-height half-length PCIe 3.0 x16 slot. Supports an Intel Omni-Path Architecture (OPA) adapter or Mellanox ConnectX-5 VPI adapter delivering ultra-high performance connectivity.
• One mezzanine LOM Generation 2 (ML2) slot, with a x16 PCIe 3.0 host interface. The ML2 slot supports NC-SI for out-of-band systems management.
• Supports SATA or NVMe SSDs drives to maximize I/O performance in terms of throughput, bandwidth, and latency.
• Supports a new Lenovo patented-design M.2 adapter for convenient operating system boot functions. Available M.2 adapters support either one M.2 drive or two M.2 drives in a RAID 1 configuration for boot drive performance and reliability.
• The use of solid-state drives (SSDs) instead of, or along with, traditional hard disk drives (HDDs) can improve I/O performance. An SSD can support up to 100 times more I/O read operations per second (IOPS) than a typical HDD.
• PCI Express 3.0 I/O expansion capabilities improve the theoretical maximum bandwidth by 60% compared with the previous generation of PCI Express 2.0.

Energy efficiency

The direct water cooled solution offers the following energy efficiency features to save energy, reduce operational costs, increase energy availability, and contribute to a green environment:

• Water cooling eliminates power that is drawn by cooling fans in the chassis and dramatically reduces the required air movement in the server room, which also saves power. Savings as much as 40% are possible in the data center due to the reduced need for air conditioning.
• Water chillers may not be required with a direct water cooled solution. Chillers are a major expense for most geographies and can be reduced or even eliminated because the water temperature can now be 50°C instead of 18°C in an air-cooled environment.
• 85-90% heat recovery is possible with the DWC design. Heat energy absorbed may be reused for heating buildings in the winter for further operating expense savings.
• The processors and other microelectronics are run at lower temperatures because they are water cooled, which uses less power.
• 80 Plus Platinum and Titanium power supplies ensure energy efficiency.
• Intel Intelligent Power Capability can power on and off individual processor elements as needed to reduce power draw.
• Low-voltage 1.2 V DDR4 memory DIMMs use up to 20% less energy than 1.35 V DDR3 DIMMs.
• There are power monitoring and power capping capabilities through the FPC in the chassis
• Optional Lenovo XClarity Energy Manager provide advanced data center power notification, analysis, and policy-based management to help achieve lower heat output and reduced cooling needs.

Manageability and security

The following powerful systems management features simplify local and remote management of the SD650 server:

• The server includes an XClarity Controller (XCC) to monitor server availability. Optional upgrade to XCC Advanced to provide remote control (keyboard video mouse) functions. Optional upgrade to XCC Enterprise enables the additional support for the mounting of remote media files (ISO and IMG image files), boot capture, and power capping.
• Lenovo XClarity Administrator offers comprehensive hardware management tools that help to increase uptime, reduce costs and improve productivity through advanced server management capabilities.
• New UEFI-based Lenovo XClarity Provisioning Manager, accessible from F1 during boot, provides system inventory information, graphical UEFI Setup, platform update function, RAID Setup wizard, operating system installation function, and diagnostic functions.
• Support for Lenovo XClarity Energy Manager which captures real-time power and temperature data from the server and provides automated controls to lower energy costs.
• Integrated Trusted Platform Module (TPM) 2.0 support enables advanced cryptographic functionality, such as digital signatures and remote attestation.
• Supports Secure Boot to ensure only a digitally signed operating system can be used. Supported on standard drives as well as M.2 drives in the M.2 Adapter.
• Industry-standard Advanced Encryption Standard (AES) NI support for faster, stronger encryption.
• Intel Execute Disable Bit functionality can prevent certain classes of malicious buffer overflow attacks when combined with a supported operating system.
• Intel Trusted Execution Technology provides enhanced security through hardware-based resistance to malicious software attacks, allowing an application to run in its own isolated space, protected from all other software running on a system.
• The n1200 DWC chassis includes drip sensors that monitor the inlet and outlet manifold quick connect couplers; leaks are reported via the Fan and Power Controller (FPC).

Availability and serviceability

The SD650 server and NeXtScale n1200 DWC chassis provide the following features to simplify serviceability and increase system uptime:

• The NeXtScale n1200 DWC chassis supports N+N and N+1 power policies for its six power supplies, which means greater system uptime.
• The power supplies are hot-swappable.
• Toolless cover removal provides easy access to upgrades and serviceable parts, such as adapters and memory.
• The server offers Single Device Data Correction (SDDC, also known as Chipkill), Adaptive Double-Device Data Correction (ADDDC, also known as Redundant Bit Steering or RBS), memory mirroring, and memory rank sparing for redundancy in the event of a non-correctable memory failure.
• The Dual M.2 Boot Adapter supports RAID-1 which enables two installed M.2 drives to be configured as a redundant pair.
• Proactive Platform Alerts (including PFA and SMART alerts): Processors, voltage regulators, memory, internal storage (HDDs and SSDs, NVMe SSDs, M.2 storage), fans, power supplies, and server ambient and subcomponent temperatures. Alerts can be surfaced through the XClarity Controller to managers such as Lenovo XClarity Administrator and other standards-based management applications. These proactive alerts let you take appropriate actions in advance of possible failure, thereby increasing server uptime and application availability.
• The XCC offers optional remote management capability and can enable remote keyboard, video, and mouse (KVM) control and remote media for the server.
• The power source-independent light path diagnostics functionality provides individual light path LEDs that lead the technician to failed (or failing) components, which simplifies servicing, speeds up problem resolution, and helps increase system availability.
• Built-in diagnostics in UEFI, using Lenovo XClarity Provisioning Manager, speed up troubleshooting tasks to reduce service time.
• Lenovo XClarity Provisioning Manager supports diagnostics and can save service data to a USB key drive or remote CIFS share folder for troubleshooting and reduce service time.
• Auto restart in the event of a momentary loss of AC power (based on power policy setting in the XClarity Controller service processor)
• There is a three-year customer replaceable unit and onsite limited warranty, with next business day 9x5 coverage. Optional warranty upgrades and extensions are available.
• With water cooling, system fans are not required; only small power supply fans operate. This results in significantly reduced noise levels on the data center floor, a significant benefit to personnel having to work on site.

The front of the SD650 tray (two servers) is shown in the following figure.

Figure 3. Front view of ThinkSystem SD650 tray (two independent servers)

The following figure shows the locations of key components inside the server.

Figure 4. Inside view of two SD650 servers in the water cooled tray

The compute nodes are installed in the NeXtScale n1200 DWC enclosure, as shown in the following figure.

Figure 5. Front view of the n1200 DWC enclosure

The rear of the enclosure contains the power supplies, cooling water manifolds, and the Fan and Power Controller, as shown in the following figure.

Figure 6. Rear view of the n1200 DWC enclosure

The following figure shows the architectural block diagram of the SD650, showing the major components and their connections.

Figure 7. SD650 system architectural block diagram

The following table lists the standard specifications of the SD650 server.

The NeXtScale n1200 DWC enclosure provides shared high-efficiency power supplies. The SD650 trays connect to the midplane of the n1200 enclosure. This midplane connection is for power and control only; the midplane does not provide any I/O connectivity.

The following table lists the standard specifications of the enclosure.

There are no standard SD650 models; all servers must be configured by using the configure-to-order (CTO) process. The ThinkSystem SD650 machine type is 7X58.

The following table lists the base CTO models.

The following table lists the base chassis choices for CTO configurations with second-generation Intel Xeon Scalable processors.

There are no standard models of the water-cooled chassis; all NeXtScale n1200 DWC chassis must be configured by using the CTO process. The machine type is 5468.

The water manifold a separate machine type 5469.

The SD650 supports the second-gneration Intel Xeon Scalable processors that are listed in the following table. Two processors must be selected - the server is not supported with only one processor installed.

Some processors include a suffix letter in the processor model number:

• L: Large memory tier (supports total memory up to 4.5TB per processor)
• M: Medium memory tier (supports total memory up to 2TB per processor)
• N: NFV optimized
• S: Search optimized
• T: High Tcase
• U: Single socket
• V: VM Density optimized
• Y: Speed Select

Processors with a suffix other than L or M, as well as those without a suffix, support up to 1TB per processor.

Memory capacity of processors

Second-generation Xeon Scalable processors are limited to the amount of memory they can address, as follows:

• Processors with an L suffix (eg 8280L): Up to 4.5 TB per processor
• Processors with an M suffix (eg 8280M): Up to 2 TB per processor
• All other processors: Up to 1 TB per processor

The calculation of the total memory per processor includes both the system memory DIMMs and the Persistent Memory DCPMMs installed in the server.

For example:

• A configuration using 6x 128GB DIMMs per processor is 768GB per processor, which means that neither an M nor an L processor is required
• A configuration using 6x 32GB DIMMs + 2x 256GB DCPMMs is a total of 1.1875 TB which means an M processor is required
• A configuration using 6x 128GB DIMMs + 2x 512GB DCPMMs is a total of 1.75 TB which means an M processor is required

* L3 cache is 1.375 MB per core or larger. Processors with a larger L3 cache per core are marked with an *
** The Intel Xeon Gold 5218 and 5218B processors have similar specifications; however, they use different silicon designs and cannot be mixed in the same system.

Each SD650 server has 12x TruDDR4 DIMM slots, plus 4x additional slots reserved for Persistent Memory modules (DCPMMs), as shown in the following figure.

Figure 8. Location of DIMMs in each server (two servers per tray)

The SD650 uses Lenovo TruDDR4 memory operating at up to 2933 MHz. The server supports 12 DIMMs with two processors installed, 6 DIMMs per processor. Each processor has six memory channels with 1 DIMM per channel. With 12x 128 GB LRDIMMs, each server supports up to 1.5 TB of memory.

The SD650 with second-generation Intel Xeon Scalable processors also supports Intel Optane DC Persistent Memory, as described in the Persistent Memory section.

The DIMMs supported with the SD650 server are listed in the following table.

Lenovo TruDDR4 memory uses the highest quality components that are sourced from Tier 1 DRAM suppliers and only memory that meets the strict requirements of Lenovo is selected. It is compatibility tested and tuned to maximize performance and reliability. From a service and support standpoint, Lenovo TruDDR4 memory automatically assumes the system warranty, and Lenovo provides service and support worldwide.

The following rules apply when selecting the memory configuration:

• The server supports RDIMMs, LRDIMMs and 3DS RDIMMs. UDIMMs are not supported.
• Mixing RDIMMs and LRDIMMs is not supported.
• Mixing 3DS RDIMMs with either RDIMMs or LRDIMMs is not supported.
• Mixing x4 and x8 DIMMs is supported.
• Mixing of 2666 MHz and 2933 MHz is supported, however, all installed memory will operate at the lowest speed.

The following memory protection technologies are supported:

• ECC
• SDDC (also known as Chipkill, for x4-based memory DIMMs; look for "x4" in the DIMM description)
• ADDDC (also known as Redundant Bit Steering or RBS, for x4-based memory DIMMs)
• Memory mirroring
• Memory rank sparing

If memory channel mirroring is used, then DIMMs must be installed in pairs or sets of three (minimum of one pair or set of three per processor), and all DIMMs in the pair or set of three must be identical in type and size. 50% of the installed capacity is available to the operating system.

If memory rank sparing is used, then a minimum of two single-rank or dual-rank DIMMs must be installed per populated channel (the DIMMs do not need to be identical). In rank sparing mode, one rank of a DIMM in each populated channel is reserved as spare memory. The largest rank in the channel will be automatically selected as the spare rank. The amount of memory available to the operating system depends on the number, capacity and rank counts of the DIMMs installed.

The SD650 server supports Intel Optane DC Persistent Memory, a new class of memory and storage technology explicitly architected for data center usage. Persistent Memory offers significantly lower latency than fetching data from SSDs, even NVMe SSDs, and offers higher capacities than system memory.

Using Lenovo ThinkSystem servers running applications that are tuned for Intel Optane DC Persistent Memory will result in lower data latency compared to solid-state drive technology. When data is stored closer to the processor on nonvolatile media, applications can see significant overall improvement in performance. 

The following table lists the ordering information for the DC Persistent Memory modules (DCPMMs).

The following are the requirements when installing DCPMMs:

• Only second-generation Intel Xeon Scalable Family processors support DCPMMs. First generation Xeon Scalable processors are not supported.
• Only the Platinum level, Gold level, and the Silver 4215 processor models support DCPMMs. All other Silver and all Bronze level processors do not support DCPMMs. See the Processor features section for details.
• All installed DCPMMs must be the same size. Mixing DCPMMs of different capacities is not supported
• DCPMMs are installed in the black DIMM slots (see the Internal view of the server)
• Minimum 2 TruDDR4 DIMMs per processor (1 per memory controller)
• For Memory Mode, minimum 2 DCPMMs per processor (install 1 per memory controller)
• For App Direct Mode, mimimum 1 DCPMM installed in the server (any processor)
• When either Memory Mode or Mixed Mode is used, the ratio of memory to DCPMMs must be between 1:16 and 1:4, but the recommended ratio is 1:4 for the best performance. For example, 6x 16GB DIMMs + 2x 256GB DCPMMs is a ratio of 1:5.33. This ratio requirement does not apply to App Direct mode.
• In configurations with DCPMMs installed, memory mirroring is supported, with two restrictions:
  • Mirroring is only enabled on the DRAM DIMMs installed in the server; The DCPMMs themselves do not support mirroring.
  • Only App Direct mode is supported. Memory mirroring cannot be enabled when DCPMMs are in Memory Mode or Mixed Mode.
• Memory sparing is not supported with DCPMMs installed

DCPMMs offer the following memory protection technologies:

• ECC
• SDDC
• DDDC
• Patrol scrubbing
• Demand scrubbing

In the App Direct mode, the DCPMMs also support memory mirroring that is performed by the processor's integrated memory controllers. Memory mirroring is not supported in Memory Mode or Mixed Mode. Memory rank sparing is not supported by DCPMMs in any mode.

For more information, see the Intel Optane DC Persistent Memory (DCPMM) product guide, https://lenovopress.com/LP1066

The SD650 server supports one or two 2.5-inch drives internally in each server. These are Simple-Swap drives that are not front accessible so will require both servers in the tray to be powered off and the tray removed from the chassis before replacing the drives.

Each server supports two 7mm high drives or one 15mm drive:

• Two 7mm SATA SSDs
• One 15mm SATA SSD and NVMe SSD

The following figure shows two 7mm drives installed in a drive cage. See the Components and connectors section for the location of the drive bays.

Figure 9. Two 7mm SATA SSDs installed in a drive cage

Supported drives are listed in the following table. The table also lists the drive cage that is required to mount the drive in the server.

In addition, the SD650 supports M.2 adapters and drives, similar to other servers in the ThinkSystem portfolio. The M.2 adapter is installed in a dedicated slot on the system board. The M.2 drives attach to the adapter: the single M.2 adapter supports 1 drive and the dual M.2 adapter supports two drives and supports RAID-0, RAID-1, or JBOD.

Note: When the dual M.2 adapter is selected, only 32GB M.2 drives can be used. The 128GB M.2 drives are not supported with the dual M.2 adapter when installed in the SD650.

The following table lists the supported M.2 adapters and drives.

* The SD650 only supports one 128GB  or one 240GB M.2 drive; only supported in the Single M.2 Adapter. See the environmental requirements in the Operating environment section
** The use of 2 requires the Dual M.2 Adapter

For more information about the M.2 adapters and M.2 drives, see the M.2 product guide:
https://lenovopress.com/lp0769-thinksystem-m2-drives-adapters

The SD650 does not support internal tape drives.

The server supports the external USB optical drive listed in the following table.

The drive is based on the Lenovo Slim DVD Burner DB65 drive and supports the following formats: DVD-RAM, DVD-RW, DVD+RW, DVD+R, DVD-R, DVD-ROM, DVD-R DL, CD-RW, CD-R, CD-ROM.

The SD650 has the following I/O expansion options:

• One PCIe 3.0 x24 front-accessible FHHL slot with riser card
• One ML2 x16 front-accessible slot with riser card. Supports NC-SI.

The server also has the following slot ready for future support of adapters:

• One internal PCIe x16 slot

The front accessible slots are shown in the following figure.

Figure 10. Front accessible PCIe slots

To use the front-accessible slots, the riser cards listed in the following table is required.

Table 12. Riser card options

The server offers one Gigabit Ethernet port with an RJ45 connector, based on the Intel Ethernet Connection X722, shared between the operating system and the XClarity Controller (XCC) management processor. The second standard Gigabit Ethernet port is reserved for systems management.

The server supports high-speed network adapters in the front-accessible full-height half-length PCIe slot and the ML2 slot. Supported adapters are listed in the following table. The use of an adapter in these slots also requires the installation of a riser card.

The Mellanox ConnectX-6 HDR adapter (feature B4RG) implements SharedIO, also known as Mellanox Multi-Host technology. With SharedIO, a Mellanox VPI adapter is installed in a slot in one SD650 server in a tray and an auxiliary adapter (feature B4RB) is installed in a slot in adjacent server in the same tray. A cable connects the two adapters together. The result is that the two servers share the network connection of the VPI adapter with significant savings both in the cost of the adapters but also the cost of switch ports.

The SD650 server offers two Omni Path Architecture solutions. Feature AU0B is a standard PCIe x16 OPA adapter with an air-cooled heatsink, whereas feature B21S is the same adapter but with a cold plate and copper tubing that connects to the water infrastructure of the server. Select feature B21S in your configuration when you want to maximize heat recovery through water cooling, rather than relying on room air conditioning.

The SD650 server currently does not support storage host bus adapters.

The SD650 server currently does not support Flash storage adapters.

The SD650 server currently does not support GPUs.

One of the most notable features of the ThinkSystem SD650 offering is direct water cooling. Direct water cooling (DWC) is achieved by circulating the cooling water directly through cold plates that contact the CPU thermal case, DIMMs, and other high-heat-producing components in the server.

One of the main advantages of direct water cooling is the water can be relatively warm and still be effective because water conducts heat much more effectively than air. Depending on the server configuration, 85-90% of the heat is removed by water cooling; the rest can be easily managed by a standard computer room air conditioner.

Allowable inlet temperatures for the water can be as high as 50°C (122°F). In most climates, water-side economizers can supply water at temperatures below 50°C for most of the year. This ability allows the data center chilled water system to be bypassed thus saving energy because the chiller is the most significant energy consumer in the data center. Typical economizer systems, such as dry-coolers, use only a fraction of the energy that is required by chillers, which produce 6-10 °C (43-50 °F) water. The facility energy savings are the largest component of the total energy savings that are realized when the SD650 is deployed.

The advantages of the use of water cooling over air cooling result from water’s higher specific heat capacity, density, and thermal conductivity. These features allow water to transmit heat over greater distances with much less volumetric flow and reduced temperature difference as compared to air.

For cooling IT equipment, this heat transfer capability is its primary advantage. Water has a tremendously increased ability to transport heat away from its source to a secondary cooling surface, which allows for large, more optimally designed radiators or heat exchangers rather than small, inefficient fins that are mounted on or near a heat source, such as a CPU.

The ThinkSystem SD650 offering uses the benefits of water by distributing it directly to the highest heat generating server subsystem components. By doing so, the offering realizes 7% - 10% direct energy savings when compared to an air-cooled equivalent. That energy savings results from the removal of the system fans and the lower operating temp of the direct water-cooled system components.

The direct energy savings at the chassis level, combined with the potential for significant facility energy savings, makes the SD650 an excellent choice for customers that are burdened by high energy costs or with a Green mandate.

Water is delivered to each of the nodes from a chiller distribution unit (CDU) via the water manifold. As shown in the following figure, each manifold section attaches to a chassis and connects directly to the water inlet and outlet connectors for each compute node to safely and reliably deliver water to and from each Compute Tray.

The DWC Manifold is modular and is available in multiple configurations that are based on the number of chassis drops that are required in a rack. The Manifold scales to support up to six Enclosures in a single rack, as shown in the following figure.

Figure 11. n1200 DWC Enclosure and Manifold assembly showing scaled up Water Manifold

The water flows through the SD650 as shown in the following animation. The inlet water is split into two parallel paths, one for each server in the tray. Each path is then split further, one for the processors and one for the memory, drives and PCIe adapter.

Figure 12. Animation showing water flow through the SD650 tray
(for PDF users, see the animation at https://lenovopress.com/LP1042#animation)

The NeXtScale n1200 DWC enclosure supports up to six redundant hot-plug power supplies, which provides N+N or N+1 redundancy. Power policies with no redundancy also are supported. These power supplies are either 80 PLUS Platinum or Titanium certified to allow for the best efficiency values of your data center. The following table lists the supported power supplies.

The power supply options have the following features:

• Supports N+N or N+1 Power Redundancy, or Non-redundant power configurations to support higher density
• Power management controller and configured through the Fan and Power Controller
• Integrated 2500 RPM fan
• 80 PLUS Platinum or Titanium certified
• Built-in overload and surge protection
• Supports high-range voltage only: 200 - 240 V
• 200 - 240 (nominal) V AC; 50 or 60 Hz; 6.9 A (maximum)

The SD650 provides local console access through the KVM connector at the front of the server. A console breakout cable is used with this connector, which provides a VGA port, two USB ports, and a DB9 serial port. The cable is shown in the following figure.

Figure 13. Console breakout cable

One console breakout cable is shipped with the NeXtScale n1200 DWC enclosure. Additional cables can be ordered per the following table.

Table 15. Console breakout cable

To aid with problem determination, the server includes light path diagnostics, which is a set of LEDs on the front of the server and inside the server that show you which component is failing. The LEDs are shown in the following figure.

Figure 14. Power button and system LEDs

When an error occurs, the system error LED lights up. Review the logs through the web interface of the XClarity Controller (see the Remote server management section). If needed, power off the server and remove it from the enclosure. Then, press and hold the light path diagnostics button to activate the system board LEDs (see Figure 4 for the location of the button). The LED next to the failed component then lights up.

Note: The server does not support local management using XClarity Mobile running on a tethered smartphone or table.

The SD650 contains an integrated service processor, XClarity Controller (XCC), which provides advanced service-processor control, monitoring, and alerting functions. The XCC is based on the Pilot4 XE401 baseboard management controller (BMC) using a dual-core ARM Cortex A9 service processor.

The server offers two Ethernet ports that can be use for remote management:

• An Ethernet port that is dedicated for remote management, connected directly the XCC. The port supports 10/100/1000 Mbps speeds.
• An Ethernet port that is shared between the operating system and remote management. The shared-NIC functionality can be enabled or disabled in the server's UEFI panels.

Figure 15. Ethernet ports on each SD650 server

Remote server management is provided through industry-standard interfaces:

• Intelligent Platform Management Interface (IPMI) Version 2.0
• Simple Network Management Protocol (SNMP) Version 3 (no SET commands; no SNMP v1)
• Common Information Model (CIM-XML)
• Representational State Transfer (REST) support
• Redfish support (DMTF compliant)
• Web browser - HTML 5-based browser interface (Java and ActiveX not required) using a responsive design (content optimized for device being used - laptop, tablet, phone) with NLS support

There are two XClarity Controller upgrades available for the server, Advanced and Enterprise.

Lenovo XClarity Controller Advanced adds the following remote control functions:

• Remotely viewing video with graphics resolutions up to 1600x1200 at 75 Hz with up to 23 bits per pixel, regardless of the system state
• Remotely accessing the server using the keyboard and mouse from a remote client
• Capturing blue-screen errors
• International keyboard mapping support
• LDAP-based authentication

Lenovo XClarity Controller Enterprise enables the following additional features:

• Boot Capture
• Remote mounting of CD-ROM (ISO) and diskette (IMG) files as virtual drives
• Virtual console collaboration - Ability for up to 6 remote users to be log into the remote session simultaneously
• Power capping
• License for XClarity Energy Manager

When configuring the server for ordering via configure-to-order (CTO), you can elect to have XCC Advanced or XCC Enterprise by selecting the appropriate feature codes as listed in the following table:

• XCC Standard - select neither feature listed in the table
• XCC Advanced - select feature AVUT
• XCC Enterprise - select feature AUPW

The NeXtScale n1200 DWC Enclosure offers a remote management capability through the Fan and Power Controller (FPC), which is at the back of the enclosure, as shown in the following figure.

Figure 16. Fan and Power Controller

The Fan and Power Controller has one external RJ45 port, which provides a 10/100 Mbps Ethernet connection for browser-based remote control.

The FPC also includes a USB key that is housed inside the unit. The USB key stores the event log for the FPC plus power, cooling, and network configuration data.

The server offers the following security features:

• Administrator and power-on password
• Trusted Platform Module (TPM) supporting both TPM 1.2 and TPM 2.0
• Optional Trusted Cryptographic Module (TCM), available only in China

The server is NIST SP 800-147B compliant.

The TCM module, available only for China customers, is installed in a dedicated socket on the system board, as shown in Figure 4. Ordering information is shown in the following table.

The server supports the following operating systems:

• Red Hat Enterprise Linux 7.6
• Red Hat Enterprise Linux 7.7
• Red Hat Enterprise Linux 7.8
• Red Hat Enterprise Linux 7.9
• Red Hat Enterprise Linux 8.0
• Red Hat Enterprise Linux 8.1
• Red Hat Enterprise Linux 8.2
• Red Hat Enterprise Linux 8.3
• Red Hat Enterprise Linux 8.4
• Red Hat Enterprise Linux 8.5
• Red Hat Enterprise Linux 8.6
• Red Hat Enterprise Linux 8.7
• Red Hat Enterprise Linux 8.8
• Red Hat Enterprise Linux 8.9
• Red Hat Enterprise Linux 9.0
• Red Hat Enterprise Linux 9.1
• Red Hat Enterprise Linux 9.2
• Red Hat Enterprise Linux 9.3
• SUSE Linux Enterprise Server 12 SP4
• SUSE Linux Enterprise Server 12 SP5
• SUSE Linux Enterprise Server 12 Xen SP4
• SUSE Linux Enterprise Server 12 Xen SP5
• SUSE Linux Enterprise Server 15
• SUSE Linux Enterprise Server 15 SP1
• SUSE Linux Enterprise Server 15 SP2
• SUSE Linux Enterprise Server 15 SP3
• SUSE Linux Enterprise Server 15 SP4
• SUSE Linux Enterprise Server 15 SP5
• SUSE Linux Enterprise Server 15 Xen
• SUSE Linux Enterprise Server 15 Xen SP1
• SUSE Linux Enterprise Server 15 Xen SP2
• SUSE Linux Enterprise Server 15 Xen SP3
• SUSE Linux Enterprise Server 15 Xen SP4
• SUSE Linux Enterprise Server 15 Xen SP5

Also review the latest LeSI Best Recipe to see the operating systems that are supported via Lenovo Scalable Infrastructure (LeSI):
https://support.lenovo.com/us/en/solutions/HT505184#5

For a complete list of supported, certified and tested operating systems, plus additional details and links to relevant web sites, see the Operating System Interoperability Guide: https://lenovopress.com/osig

The compute tray with two SD650 servers has the following dimensions:

• Width: 432 mm (17.0 in.)
• Height: 41.0 mm (1.6 in.)
• Depth: 659 mm (25.9 in.)
• Weight (two servers on each compute tray):17.2 kg (38 lb)

The NeXtScale n1200 DWC enclosure features the following dimensions:

• Width: 447 mm (17.6 in.)
• Height: 262.2 mm (10.3 in.)
• Depth : 915 mm (36.0 in.)
• Weight:
  • Fully configured (stand-alone): approximately 135.5 kg (298 lbs)
  • Empty enclosure (with midplane, FPC, and cables): approximately 25 kg (55 lbs)

The system operates at the following electrical specification:

• 200 - 240 (nominal) V AC, 50 Hz or 60 Hz
• Maximum input current:
  • 1300W power supply: 6.9A
  • 1500W power supply: 8.2A
  • 2000W power supply: 10A

The SD650 tray and n1200 DWC enclosure are supported in the following environment:

Water requirements:

• Water temperature: ASHRAE class W4: 2°C - 50°C (35.6°F - 122°F)

  Note: The minimum inlet water temperature is data center dependent, that is, the dew point at which condensation occurs. A minimum of 2°C is supported provided ambient air conditions are not such that condensation will occur. A minimum of 18°C is recommended.

• Maximum pressure: 4.4 bars
• Minimum water flow rate: 6.0 liters per minute per enclosure, assuming 1.0 lpm per compute tray with 6 trays per enclosure (1 tray consists of 2 compute nodes)
  • For processors below 205 W: 6.0 liters per minute per enclosure, assuming 1.0 lpm per compute tray with 6 trays per enclosure
  • For processors 205W and above: 7.5 liters per minute per enclosure, assuming 1.25 lpm per compute tray with 6 trays per enclosure

Note: The water required to initially fill the system side cooling loop must be reasonably clean, bacteria- free water (<100 CFU/ml) such as de-mineralized water, reverse osmosis water, de-ionized water, or distilled water. The water must be filtered with an in-line 50 micron filter (approximately 288 mesh). The water must be treated with anti-biological and anti-corrosion measures.

Air temperature requirements:

• Operating: ASHRAE class A2: 10°C - 35°C (50°F - 95°F); when the altitude exceeds 900 m (2953 ft), the maximum ambient temperature value decreases by 1°C (1.8°F) with every 300 m (984 ft) of altitude increase.
• Powered off: 5°C - 45°C (41°F - 113°F)
• Shipping/storage: -40°C - 60°C (-40°F - 140°F)
• Maximum altitude: 3048 m (10 000 ft)

To use the 128GB M.2 drive or 240GB M.2 drive, the following conditions must be met:

• If inlet water temperature is 45°C, then the ambient air temperature cannot exceed 27°C.
• If inlet water temperature is 30°C, then the ambient air temperature cannot exceed 35°C.

Relative humidity (non-condensing):

• Operating: ASHRAE Class A2: 8% - 80%, maximum dew point : 21°C (70°F)
• Shipment/storage: 8% - 90%

Heat output:

• Minimum configuration (with one minimal configuration tray): 433 BTU per hour (127 watts)
• Maximum configuration (with six maximal configuration trays): 40946 BTU per hour (12000 watts)

The server conforms to the following international standards:

• FCC - Verified to comply with Part 15 of the FCC Rules, Class A
• Canada ICES-003, issue 5, Class A
• UL/IEC 60950-1
• CSA C22.2 No. 60950-1
• Japan VCCI, Class A
• IEC 60950-1 (CB Certificate and CB Test Report)
• China CCC GB4943.1, GB9254, Class A, and GB17625.1
• Russia EAC/GOST ME01, IEC-60950-1, GOST R 51318.22, and GOST R 51318.24, GOST R 51317.3.2, GOST R 51317.3.3
• CE Mark (EN55032 Class A, EN60950-1, EN55024, and EN61000-3-2, EN61000-3-3)
• CISPR 22, Class A

The warranty terms are as follows:

• ThinkSystem SD650: Three years
• NeXtScale n1200 DWC Enclosure: Three years

The standard warranty terms are customer-replaceable unit (CRU) and onsite (for field-replaceable units FRUs only) with standard call center support during normal business hours and 9x5 Next Business Day Parts Delivered.

Lenovo’s additional support services provide a sophisticated, unified support structure for your data center, with an experience consistently ranked number one in customer satisfaction worldwide. Available offerings include:

• Premier Support

  Premier Support provides a Lenovo-owned customer experience and delivers direct access to technicians skilled in hardware, software, and advanced troubleshooting, in addition to the following:

  • Direct technician-to-technician access through a dedicated phone line
  • 24x7x365 remote support
  • Single point of contact service
  • End to end case management
  • Third-party collaborative software support
  • Online case tools and live chat support
  • On-demand remote system analysis
• Warranty Upgrade (Preconfigured Support)

  Services are available to meet the on-site response time targets that match the criticality of your systems.

  • 3, 4, or 5 years of service coverage
  • 1-year or 2-year post-warranty extensions
  • Foundation Service: 9x5 service coverage with next business day onsite response. YourDrive YourData is an optional extra (see below).
  • Essential Service: 24x7 service coverage with 4-hour onsite response or 24-hour committed repair (available only in select markets). Bundled with YourDrive YourData.
  • Advanced Service: 24x7 service coverage with 2-hour onsite response or 6-hour committed repair (available only in select markets). Bundled with YourDrive YourData.
• Managed Services

  Lenovo Managed Services provides continuous 24x7 remote monitoring (plus 24x7 call center availability) and proactive management of your data center using state-of-the-art tools, systems, and practices by a team of highly skilled and experienced Lenovo services professionals.

  Quarterly reviews check error logs, verify firmware & OS device driver levels, and software as needed. We’ll also maintain records of latest patches, critical updates, and firmware levels, to ensure you systems are providing business value through optimized performance.

• Technical Account Management (TAM)

  A Lenovo Technical Account Manager helps you optimize the operation of your data center based on a deep understanding of your business. You gain direct access to your Lenovo TAM, who serves as your single point of contact to expedite service requests, provide status updates, and furnish reports to track incidents over time. In addition, your TAM will help proactively make service recommendations and manage your service relationship with Lenovo to make certain your needs are met.

• Enterprise Server Software Support

  Enterprise Software Support is an additional support service providing customers with software support on Microsoft, Red Hat, SUSE, and VMware applications and systems. Around the clock availability for critical problems plus unlimited calls and incidents helps customers address challenges fast, without incremental costs. Support staff can answer troubleshooting and diagnostic questions, address product comparability and interoperability issues, isolate causes of problems, report defects to software vendors, and more.

• YourDrive YourData

  Lenovo’s YourDrive YourData is a multi-drive retention offering that ensures your data is always under your control, regardless of the number of drives that are installed in your Lenovo server. In the unlikely event of a drive failure, you retain possession of your drive while Lenovo replaces the failed drive part. Your data stays safely on your premises, in your hands. The YourDrive YourData service can be purchased in convenient bundles and is optional with Foundation Service. It is bundled with Essential Service and Advanced Service.

• Health Check

  Having a trusted partner who can perform regular and detailed health checks is central to maintaining efficiency and ensuring that your systems and business are always running at their best. Health Check supports Lenovo-branded server, storage, and networking devices, as well as select Lenovo-supported products from other vendors that are sold by Lenovo or a Lenovo-Authorized Reseller.

Examples of region-specific warranty terms are second or longer business day parts delivery or parts-only base warranty.

If warranty terms and conditions include onsite labor for repair or replacement of parts, Lenovo will dispatch a service technician to the customer site to perform the replacement. Onsite labor under base warranty is limited to labor for replacement of parts that have been determined to be field-replaceable units (FRUs). Parts that are determined to be customer-replaceable units (CRUs) do not include onsite labor under base warranty.

If warranty terms include parts-only base warranty, Lenovo is responsible for delivering only replacement parts that are under base warranty (including FRUs) that will be sent to a requested location for self-service. Parts-only service does not include a service technician being dispatched onsite. Parts must be changed at customer’s own cost and labor and defective parts must be returned following the instructions supplied with the spare parts.

Lenovo Services is a dedicated partner to your success. Our goal is to reduce your capital outlays, mitigate your IT risks, and accelerate your time to productivity.

Here’s a more in-depth look at what we can do for you:

• Asset Recovery Services

  Asset Recovery Services (ARS) helps customers recover the maximum value from their end-of-life equipment in a cost-effective and secure way. On top of simplifying the transition from old to new equipment, ARS mitigates environmental and data security risks associated with data center equipment disposal. Lenovo ARS is a cash-back solution for equipment based on its remaining market value, yielding maximum value from aging assets and lowering total cost of ownership for your customers. For more information, see the ARS page, https://lenovopress.com/lp1266-reduce-e-waste-and-grow-your-bottom-line-with-lenovo-ars.

• Assessment Services

  An Assessment helps solve your IT challenges through an onsite, multi-day session with a Lenovo technology expert. We perform a tools-based assessment which provides a comprehensive and thorough review of a company's environment and technology systems. In addition to the technology based functional requirements, the consultant also discusses and records the non-functional business requirements, challenges, and constraints. Assessments help organizations like yours, no matter how large or small, get a better return on your IT investment and overcome challenges in the ever-changing technology landscape.

• Design Services

  Professional Services consultants perform infrastructure design and implementation planning to support your strategy. The high-level architectures provided by the assessment service are turned into low level designs and wiring diagrams, which are reviewed and approved prior to implementation. The implementation plan will demonstrate an outcome-based proposal to provide business capabilities through infrastructure with a risk-mitigated project plan.

• Basic Hardware Installation

  Lenovo experts can seamlessly manage the physical installation of your server, storage, or networking hardware. Working at a time convenient for you (business hours or off shift), the technician will unpack and inspect the systems on your site, install options, mount in a rack cabinet, connect to power and network, check and update firmware to the latest levels, verify operation, and dispose of the packaging, allowing your team to focus on other priorities.

• Deployment Services

  When investing in new IT infrastructures, you need to ensure your business will see quick time to value with little to no disruption. Lenovo deployments are designed by development and engineering teams who know our Products & Solutions better than anyone else, and our technicians own the process from delivery to completion. Lenovo will conduct remote preparation and planning, configure & integrate systems, validate systems, verify and update appliance firmware, train on administrative tasks, and provide post-deployment documentation. Customer’s IT teams leverage our skills to enable IT staff to transform with higher level roles and tasks.

• Integration, Migration, and Expansion Services

  Move existing physical & virtual workloads easily, or determine technical requirements to support increased workloads while maximizing performance. Includes tuning, validation, and documenting ongoing run processes. Leverage migration assessment planning documents to perform necessary migrations.

The NeXtScale n1200 DWC enclosure is supported in the following racks:

• 42U 1100mm Enterprise V2 Dynamic Rack and Rack Expansion, type 9363
• 42U 1100mm Deep Primary Rack and Expansion Rack, type 1410

For more information, see the list of Lenovo Press Product Guides in the Rack cabinets and options category:
http://lenovopress.com/systemx/rack

The following table lists the uninterruptible power supply (UPS) units that are offered by Lenovo.

† Only available in China and the Asia Pacific market.

For more information, see the list of Product Guides in the UPS category:
https://lenovopress.com/servers/options/ups

The following table lists the power distribution units (PDUs) that are offered by Lenovo.

For more information, see the Lenovo Press documents in the PDU category:
https://lenovopress.com/servers/options/pdu

Lenovo Financial Services reinforces Lenovo’s commitment to deliver pioneering products and services that are recognized for their quality, excellence, and trustworthiness. Lenovo Financial Services offers financing solutions and services that complement your technology solution anywhere in the world.

We are dedicated to delivering a positive finance experience for customers like you who want to maximize your purchase power by obtaining the technology you need today, protect against technology obsolescence, and preserve your capital for other uses.

We work with businesses, non-profit organizations, governments and educational institutions to finance their entire technology solution. We focus on making it easy to do business with us. Our highly experienced team of finance professionals operates in a work culture that emphasizes the importance of providing outstanding customer service. Our systems, processes and flexible policies support our goal of providing customers with a positive experience.

We finance your entire solution. Unlike others, we allow you to bundle everything you need from hardware and software to service contracts, installation costs, training fees, and sales tax. If you decide weeks or months later to add to your solution, we can consolidate everything into a single invoice.

Our Premier Client services provide large accounts with special handling services to ensure these complex transactions are serviced properly. As a premier client, you have a dedicated finance specialist who manages your account through its life, from first invoice through asset return or purchase. This specialist develops an in-depth understanding of your invoice and payment requirements. For you, this dedication provides a high-quality, easy, and positive financing experience.

For more information, see these resources:

• ThinkSystem SD650 home page
  https://www.lenovo.com/us/en/data-center/servers/high-density/ThinkSystem-SD650/p/77XX7DSSD65
• ThinkSystem SD650 datasheet
  https://lenovopress.com/DS0024
• Interactive 3D tour of the SD650
  https://lenovopress.com/LP0840
• ThinkSystem SD650 drivers and support
  http://datacentersupport.lenovo.com/products/servers/thinksystem/sd650-dwc-dual-node-tray/7x58/downloads
• Blog post: Lenovo’s Customer-Driven Cooling Innovations Push New Performance and Energy Efficiency Limits In High Performance Computing, by Vinod Kamath, Lenovo Thermal Architect
  https://lenovoxperience.com/shareContent?newsId=283yi044hzgcdv7snkrmmx9ovczuovmrbj7g8uxrx4d1j94v
• ThinkSystem SD650 and NeXtScale n1200 DWC enclosure product publications:
  http://thinksystem.lenovofiles.com/help/topic/SD650/introduction.html
  • Quick Start
  • Setup Guide
  • Maintenance Manual
  • Messages and Codes Reference
  • Fan and Power Control User's Guide
• Lenovo Scalable Infrastructure home page:
  https://www.lenovo.com/us/en/data-center/servers/high-density/Lenovo-Scalable-Infrastructure/p/WMD00000276
• Lenovo Capacity Planner
  https://datacentersupport.lenovo.com/us/en/products/solutions-and-software/software/lenovo-capacity-planner/solutions/ht504651
• Power planning documents
  https://support.lenovo.com/documents/LNVO-POWINF