close

Вход

Забыли?

вход по аккаунту

?

Sun Blade Servers Architecture

код для вставкиСкачать
 SUN BLADE
™
8000 AND 8000 P
MODULAR SYSTEMS
Modular Architecture for Business and Mission
Critical Applications and High Performance Computing
White Paper
November 2006
Sun Microsystems, Inc.
Table of Contents
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Modular Architecture for Enterprise Applications and HPC . . . . . . . . . . . . . . . . . . 2
The promise of modular architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
Sun Blade 8000 and 8000 P modular systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Sun Blade 8000 and 8000 P Modular System Overview . . . . . . . . . . . . . . . . . . . . 10
Chassis front perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Chassis rear perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
Passive midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Industry Standard Server, Networking, and Expansion Modules . . . . . . . . . . . . . 15
Sun Blade X8400 server module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
PCI Express ExpressModules (EM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Network Express Module (NEM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
Open and Standard Chassis and System Management . . . . . . . . . . . . . . . . . . . . 25
Chassis Monitoring Module (CMM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
Server module ILOM service processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Sun N1 system management software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Executive Summary
Sun Microsystems, Inc.
Executive Summary
The emerging Participation Age is marked by radically increasing individual connectedness and participation in a wide range of networked global communities. With ever-increasing demands from customers and partners, organizations need to deploy services more rapidly while retaining sufficient agility to take advantage of business opportunities when they occur. At the same time, high data center real estate costs and other pressures are driving many to increase density and raise the levels of abstraction in order to simply infrastructure and keep management costs reasonable. Even as organizations consolidate applications from multiple legacy platforms, however, they must assure predictable performance that meets service-level requirements while delivering high levels of utilization.
Blade server architecture offers considerable promise toward addressing these goals, with its presumption of increased compute density, improved serviceability, and lower levels of exposed complexity. Unfortunately, most legacy blade platforms don't provide the computational horsepower, memory capacity, and I/O bandwidth needed by today's demanding application tier and other high-performance applications. Complicating matters, many legacy blade server platforms often lock customers into an extensive proprietary infrastructure that requires redesign of existing network, management, and storage environments. These legacy chassis designs also often artificially constrain expansion capabilities and provide only limited and expensive expansion options, ultimately dictating network topologies. Many organizations have found these limitations simply unacceptable for their essential business- and mission-
critical application environments. In contrast to earlier blade server designs, the Sun Blade
™
8000 and 8000 P modular systems provide a fundamentally new approach that combines the best aspects of modular architecture with a design focused on the needs of the data center. Delivering serious computational performance, flexible networking, and industry-standard high-
bandwidth I/O capabilities, the innovative Sun Blade 8000 modular system provides up to three times the throughput of today’s blades, while using 20 percent less power and 30 to 50 percent less space than modern rack mount servers. With support for up to 40 dual-core AMD Opteron
™
processors in a single chassis and 120 in a single rack (up to 240 processor cores) the Sun Blade 8000 P provides effective high-density infrastructure for high performance computing (HPC) environments such as compute grids.
With chassis designed for the future, Sun Blade 8000 and 8000 P modular systems can not only host significant computation, I/O, and expansion, but offer the headroom to power future server modules based on next-generation processors as well. This document describes the Sun Blade 8000 and 8000 P modular systems, along with their key applications, architecture, and components.
2
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
Chapter 1
Modular Architecture for Enterprise Applications and HPC
Steadily improving technology is yielding new opportunities for increased efficiency and flexibility in the data center. Dual- and multi-core processor technologies are doubling compute density every other year. Virtualization technologies and more powerful servers are making it possible to consolidate widely distributed data centers onto smaller numbers of more powerful servers. High bandwidth networking technologies are becoming more affordable. Modern provisioning technology makes it possible to dynamically readjust workloads on the fly.
Regrettably, most current server form factors have failed to take full advantage of these trends. For instance, most traditional rack-mount servers require a box swap in order to allow an organization to deploy new CPU and I/O technology. Modular architecture offers the opportunity to rapidly harvest the returns of new technology while serving the constantly changing needs of the enterprise.
The promise of modular architecture
At its best, modular or blade server architecture blends the enterprise availability and management features of vertically-scalable platforms with the scalability and economic advantages of horizontally-scalable systems. In general, modular architectures offer considerable promise, contributing to:
• Higher compute density — providing more processing power per rack unit than with rack-mount systems
• Increased serviceability and availability — featuring shared common system components such as power, cooling, and I/O interconnects
• Reduced complexity — through fewer required components, cable and component aggregation, and consolidated management
• Faster service expansion and bulk deployment — letting organizations expand or scale existing services and flexibly pre-provision chassis and I/O components
• Lowered costs — since modular servers can be less expensive to acquire, easier to service, easier to manage
Unfortunately, legacy blade architectures have failed to deliver on this promise for more demanding applications. Most legacy blade systems are based on proprietary architectures that lock customers into an extensive infrastructure that constrains deployment. In addition, though vendors typically try to price server modules economically, they often charge a premium on the required proprietary I/O and switching infrastructure. 3
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
Together, these constraints have caused trade-offs in both features and performance that had to be weighed when considering blade technology for individual applications: • Power and cooling limitations often mean that processors are limited to less powerful mobile versions.
• Limited processing power, memory capacity, and I/O bandwidth severely constrain the applications that can be deployed on blade server platforms.
• Proprietary tie-ins and other constraints in chassis design dictate networking topology for customers and limit I/O expansion possibilities to a small number of proprietary modules.
These shortcomings in chassis design were largely the result of a primary focus on density, with relatively small blade server chassis, and small-format server modules. Ultimately these designs limited the application of blade technology. While many organizations would have liked to deploy modular technology for applications, databases, and HPC environments, these limitations caused legacy blade servers to be more commonly deployed for less demanding applications such as web servers and simple IT infrastructure. Sun Blade
™
8000 and 8000 P modular systems
A fundamental rethinking of blade architecture was required to address these issues and to make modular servers a viable consideration for applications with serious computational and I/O demands. At Sun, this process began with a design point that focused on the needs of the data center rather than preconceptions of chassis design. The result is the Sun Blade 8000 and 8000 P modular systems, offering high performance, scalability, and easy integration into legacy data centers. These platforms offer considerable advantages:
• Serious performance for enterprise and HPC applications
Powered by the fastest AMD Opteron processors available, large memory capacity, and massive I/O throughput on each server module, high performance I/O intensive enterprise or technical computing applications can be deployed with ease. The Sun Blade 8000 modular system provides breakthrough I/O performance offering two to 20 times the I/O performance of competing four-socket blade servers and allowing entire multi-tier applications or to be deployed in a single chassis or rack. With support for up to three Sun Blade 8000 P modular systems in a single rack, flexible and dense HPC deployments can be built in a very small physical footprint.
4
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
• Simpler to manage
With the Sun Blade 8000 modular system, all management, monitoring, and control functions execute directly on each server module through an Integrated Lights Outs Manager (ILOM) service processor, removing the need to manage at a chassis level. Coupling ILOM capabilities with Sun N1
™
System Manager and the Sun N1 service Provisioning System provides a unique and powerful management architecture that offers the highest levels of automation.
• More energy efficient
Consolidated power and cooling modules in the Sun Blade 8000 chassis improve energy efficiency with 20 percent lower power consumption and 30 percent better cooling efficiency over equivalent rack-mount servers. The result is greatly improved system reliability, allowing organizations to maximize throughput per watt, and per square foot.
• Lower cost
The Sun Blade 8000 modular system is based on an industry-standard PCI Express midplane, allowing Sun and third-party peripheral vendors to offer a broad range of compatible I/O modules at lower costs. In addition, Sun’s implementation of industry-standard PCI Express networking helps eliminate expensive and proprietary network interfaces while providing an open platform for innovation of different I/O modules.
The result of this comprehensive no-compromise approach is a design that is literally better than both rack-mount server and legacy blades in terms of energy consumption, density, and I/O capacity. Table 1 demonstrates that the Sun Blade 8000 modular system provides greater density at lower power with far greater I/O capacity when compared with rack-mount servers providing equivalent numbers of processor cores.
Table 1. The Sun Blade 8000 modular system provides superior characteristics in terms of power consumption, density, and I/O capacity over rack-mount servers
Table 2 illustrates similar characteristics by comparing the I/O throughput along with the power needed to support 80 processor cores. Again, the Sun Blade 8000 modular system provides a significant advantage in energy consumption and I/O capacity over competitive four-socket modular server offerings.
Platform Dell 6950 HP DL585 G2 IBM x3755
Sun Fire V40z server
Sun Blade 8000 modular system
CPU cores 80 80 80 80 80
Watts (maximum configuration) 15,700W 15,660W
a
15,000W 9,800W 9,000W
I/O Capacity (per 10 servers) 1,440 Gb/s 1,664 Gb/s 1,205 Gb/s 510 Gb/s 1,920 Gb/s
Space (rack units) 40 RU 40 RU 30 RU 30 RU 19 RU
a. HP DL585 G2 consumes 11,580W at 115 V, and 15,660 at 220 V; Sun Blade 8000 modular systems operate on 220 V power
5
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
Table 2. The Sun Blade 8000 and 8000 P modular systems provide superior characteristics in terms of power consumption, density, and I/O capacity when compared to other blade servers
Sun makes every effort to quote performance and throughput numbers that are verifiable and conservative, in line with real-world situations and customer expectations. When evaluating I/O throughput in particular, it is important to compare systems using similar metrics. For example, HP claims 5 terabits per second of I/O bandwidth for its C-Class blade server chassis by counting SerDes connections on the backplane and calculating throughput based on a theoretical maximum data transfer speed over copper, without accounting for protocol or other overhead. By this measure, the Sun Blade 8000 modular system provides I/O throughput of 9.6 terabits per second.
Third-generation modular architecture
Starting with a clean-slate design, the Sun Blade 8000 modular system provides a new approach to system architecture. This approach starts by disaggregating the system into modular components, and then recreating a balanced system for enterprise computing that offers both superior investment protection and support for a complete range of applications.
• Powerful server modules
Each Sun Blade 8000 modular system supports up to 10 Sun Blade X8400 server modules in a single chassis. For high-performance applications, each server module in turn provides four sockets for the fastest available dual-core AMD Opteron processors. High memory density for large-memory applications is provided on each server module with sixteen DDR1-400 DIMM sockets (for a total of up to 64GB of RAM). Each server module provides high I/O bandwidth as well, with a total of 40 lanes of PCI Express bandwidth delivered from each server module to the multiple available I/O expansion modules (a total of 160 Gb/s per supported per server module at initial product introduction, with 192 Gb/s available at a later date). To enhance availability, the Sun Blade X8400 server module has no power supply or fans and features two hot-swap disks with RAID 0 or 1 built in.
• Complete separation between CPU and I/O modules
The Sun Blade 8000 modular system design avoids compromises because it provides a complete separation between CPU and I/O modules. Two types of I/O modules are supported, including up to two industry-standard PCI Express Platform HP BL45p IBM LS41
Sun Blade 8000 modular system
Sun Blade 8000 P modular system
CPU cores 80 80 80 80
Watts 9,000W 11,600W 9,000W 9,000W
I/O Capacity 120 Gb/s 440 Gb/s 1,920 Gb/s 640 Gb/s
Memory 32 GB 64 GB 64 GB 64 GB
6
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
ExpressModules (EMs) dedicated to each server module as well as multiple Sun Blade 8000 Network Express Modules (NEMs) that provide bulk IO for all of the server modules installed in the system. Through this flexible approach, server modules can be configured with different I/O options depending on the applications they host. I/O modules are hot-plug, and customers can choose from Sun-branded or third-party adapters for networking, storage, clustering, and other I/O functions. Sun Blade 8000 P modular systems offer only NEM support.
• Industry-standard management infrastructure
Each Sun Blade X8400 server module contains its own directly addressable management service processor that is compatible with other Sun x64 systems management. Referred to as Integrated Lights Out Management (ILOM), this capability supports IPMI, SNMP, CLI, and HTTP management methods, and provides transparent management of individual server modules.
• Integrated chassis management infrastructure
Within each Sun Blade 8000 and 8000 P modular system, optional dual redundant Chassis Monitoring Modules (CMM) work in conjunction with the ILOM service processor on each server module to form a complete chassis management solution. Sun N1 management software provides discovery, aggregated management, and bulk deployment for multiple systems. • Innovative and highly-reliable chassis design
The Sun Blade 8000 and 8000 P modular systems are designed for a long life cycle and their designs assume ongoing improvements in technology. The chassis integrates AC power supplies and cooling fans for all of the server and I/O modules. This approach keeps these components off of the server modules, making them efficient and more reliable. Power supplies and fans in the chassis are designed for ease-of-service, hot-swappability, and redundancy. The chassis provides power and cooling infrastructure to support current and future CPU and memory configurations, helping to ensure that the chassis life cycle will span multiple generations of processor upgrades. All modular components such as CMMs, EMs, and NEMs are hot-plug, and can be configured in a redundant fashion.
• Targeted chassis for specific needs
Sun understands that different applications require different approaches to modular infrastructure. The Sun Blade 8000 modular server is ideal for business and missions critical applications that need density along with maximized I/O capacity and N+N power redundancy for high availability. The Sun Blade 8000 P modular server is ideal for HPC applications such as compute grids that primarily need performance and density with lower requirements for I/O and tolerance for N+1 power redundancy. Both chassis provide the same computational capacity.
7
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
The Sun Blade X8400 server module — enterprise class data center compute engine
Application performance and the ability to host demanding compute, memory, and I/O-
intensive applications is ultimately dependent on the characteristics of the actual server module itself. The innovative chassis design of the Sun Blade 8000 modular system allowed designers considerable flexibility when it came to delivering powerful server modules for serious computational tasks.
Compute and memory capacity
Each Sun Blade X8400 server module provides substantial computational and memory capacity to support demanding enterprise and HPC applications. With support for up to 10 server modules per chassis, each Sun Blade X8400 server module provides:
• Four sockets for dual-core processors, yielding eight cores per board, 80 cores per chassis, 160 cores per rack (Sun Blade 8000 modular system), or 240 cores per rack (Sun Blade 8000 P modular system)
• The fastest available dual-core AMD Opteron processors available, including Special Edition (SE) Opteron processors
• 16 DIMM slots (4 per socket) on each Sun Blade X8400 server module with 1GB, 2GB, and 4GB DDR400 DIMMs for up to 64 GB of memory per server module
Leading I/O throughput
The Sun Blade X8400 server module provides from two to 20 times more I/O throughput than any other server module and up to three times the I/O options, allowing modular servers to be used for applications that require significant I/O throughput.
• 160 Gbps of I/O throughput are provided on each Sun Blade X8400 server module, delivered through 40 lanes of PCI Express I/O (up to 1.6 Tbps of I/O throughput per Sun Blade 8000 chassis). The complete throughput of six PCI Express interfaces (four x8, and two x4) are delivered from each server module to the passive midplane and the I/O devices connected to it in the Sun Blade 8000 chassis.
• Two SAS or SATA 2.5-inch disk drives are supported per server module (PCI-based)
• In the Sun Blade 8000 modular system, two hot-plug PCI Express ExpressModules (EM) slots are dedicated to each server module (20 per chassis) for granular blade I/O configuration.
• Network Express Modules (NEMs) provide bulk I/O across multiple server modules and aggregate I/O functions. The Sun Blade 8000 modular system supplies up to four NEMs while the Sun Blade 8000 P modular system supplies up to two NEMs.
8
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
A choice of operating systems
In order to provide maximum flexibility and investment protection, the Sun Blade X8400 server module supports a choice of operating systems, including:
• Solaris
™
10 Operating System (OS)
• The Linux operating system (64-bit RedHat or SuSE Linux)
• Microsoft Windows
• VMware ESX Server 3.0.1
This list is accurate as of this writing, please see www.sun.com/servers/blades/8000
for the latest supported operating systems and environments.
Enterprise-class features
Unlike most traditional blade servers, Sun Blade X8400 server modules provide a host of enterprise features that help ensure greater reliability and availability:
• Each server module supports hot plug with the chassis
• Each server module supports two hot-plug disks, and built-in support for RAID 0 or 1 (diskless operation is also supported)
• The Sun Blade 8000 chassis supports hot plug I/O in the form of both EM and NEM I/O modules while the Sun Blade 8000 P chassis supports only NEM I/O modules
• Redundant hot-swap chassis-located fans mean greater reliability through decreased part count and no fans located on the server modules
• Redundant hot-swap chassis-located power supply units (PSUs) means that no power supplies are located on individual server modules
Enterprise-class management
Together, the Sun Blade X8400 server module and the Sun Blade 8000 and 8000 P modular system provide a robust and comprehensive list of management features, including:
• A dedicated ILOM service processor on each server module for blade-level management granularity
• Optionally redundant Chassis Monitoring Modules (CMMs) for integrated server module and chassis management
• Sun N1 System Manager software for server module discovery and OS provisioning
• Sun N1 Service Provisioning System for bulk application-level provisioning
Modular and “future-proof” chassis design
Sun Blade 8000 and 8000 P modular systems provide significant improvements over legacy server module platforms. While others treat the chassis itself as the design center, Sun chose to center its design on the needs of the data center. This approach ultimately resulted in chassis designs that don’t force compromises in the performance and capabilities ultimately delivered by the server modules. For example, in addition to offering four-socket Sun Blade X8400 server modules that support the latest dual-core 9
Modular Architecture for Enterprise Applications and HPC
Sun Microsystems, Inc.
AMD Opteron processors, the Sun Blade 8000 modular system delivers 100 percent of system I/O to the I/O modules through a passive midplane.
Two modular chassis are provided to help maximize both compute and I/O density, depending on the needs of the organization (Figure 1). The Sun Blade 8000 modular system is provided in a 19 rack unit (19U) chassis with up to two chassis supported in a single 42U rack. The Sun Blade 8000 P modular system features a 14U chassis with up to three chassis supported in a single rack.
Figure 1. The Sun Blade 8000 and 8000 P modular systems (left and right respectively)
Each chassis supports:
• Up to 10 Sun Blade X8400 server modules per chassis (20 per rack for the Sun Blade 8000 modular system, 30 per rack for the Sun Blade 8000 P modular system) • Up to 20 blade-dedicated PCI Express ExpressModules (EM), supporting industry-
standard PCI Express interfaces (Sun Blade 8000 modular system only)
• Network Expansion Modules (NEMs), providing access and an aggregated interface to all of the server modules in the chassis
• Optionally redundant Chassis Monitoring Modules (CMMs) for integrated chassis-
level management
• Hot-swap chassis-based power-supplies and fans
• Redundant power grid connection capability
10
Sun Blade 8000 and 8000 P Modular System Overview
Sun Microsystems, Inc.
Chapter 2
Sun Blade 8000 and 8000 P Modular System Overview
The Sun Blade 8000 modular system is ideal for highly-available applications that need significant amounts of computational performance, large memory support, and high-
bandwidth I/O throughput:
• Business- and mission-critical applications (mid-tier) need availability features along with scalability in terms of scalable processing performance, memory, and I/O.
• High-performance databases need processing power, large memory support, and high-bandwidth I/O
• Application consolidation efforts require predictable and scalable performance with integral partitioning and virtualization technologies.
The Sun Blade 8000 P modular server is ideal for applications where computational density is the overriding concern, coupled with lower needs for I/O and strategies such as grid computing provide for availability requirements:
• High-performance Computing (HPC) in areas such as research, finance, energy exploration, the life sciences, EDA, and MCAE requires floating point performance, large memory support, and computational density to optimize real estate and power consumption.
Together, these modular systems address all of these application areas with a unified design that focuses on innovative integration in addition to density. The result is a platform that supports the most challenging computational problems, scalable applications, large databases, and even entire multi-tier applications in a single chassis.
Chassis front perspective
The Sun Blade 8000 and 8000 P chassis contain the server modules and I/O modules, connecting the two through the passive midplane. Redundant and hot-swappable power supplies and fans are also hosted in the chassis. All slots are accessible from the front or the rear of the chassis for easy serviceability. Server modules, I/O modules, power supplies, and fans can all be added and removed while the chassis and other elements in the enclosure are powered on. This capability yields great expansion 11
Sun Blade 8000 and 8000 P Modular System Overview
Sun Microsystems, Inc.
opportunity and provides considerable flexibility. The front view of the Sun Blade 8000 and 8000 P chassis is shown in Figure 2.
Figure 2. Server modules, power supplies, and fan modules load from the front of the Sun Blade 8000 and 8000 P chassis
As shown in the illustration, three front fan modules cool the PCI Express ExpressModules in the top of the Sun Blade 8000 chassis. These fans are hot-swap capable and have LEDs that indicate their status. The fans are located behind a grill that can be removed by unscrewing two captive screws on either end. Below the fans are the six power supply units (PSUs) that are also hot-swap and redundant (N+N). Each of the PSUs has an integral fan that cools the power supply as well as the Network Express Modules located behind it. The fans in the PSUs get their power from the chassis power grid, so that the fans keep running even if the power supply were to fail. Each PSU is rated at 3 kW for a total N+N PSU rating of 9 kW, providing the chassis with the head room to power a full complement of server modules as well as server modules based on future processor technology. The Sun Blade 8000 P chassis supports four power supplies for an N+1 power rating of 9 kW. Power plugs are separate and independent of the power supplies, and are located on the back of the chassis.
Beneath the PSUs, up to 10 Sun Blade X8400 server modules are inserted vertically. The dual hard drives on each server module are available for easy hot-swap from the front of the chassis. Indicator LEDs and I/O ports are also provided on the front of the server modules for easy access. The I/O ports available on each server module include a VGA HD-15 monitor port, two USB 2.0 ports, and a DB-9 serial port that connects to the ILOM service processor and console. Table 3 contrasts the capacities of the Sun Blade 8000 and Sun Blade 8000P chassis.
Three front fan modules Up to ten Sun Blade X8400 (cooling for PCI Express EMs)
server modules per chassis
Hot-swappable power supplies with integral fans
Sun Blade 8000 modular system
Sun Blade 8000 P modular system
12
Sun Blade 8000 and 8000 P Modular System Overview
Sun Microsystems, Inc.
Table 3. The Sun Blade 8000 and 8000 P modular systems provide a range of capabilities to suit diverse needs.
Chassis rear perspective
The rear of the Sun Blade 8000 and 8000 P chassis (Figure 3) provide access to the back side of the passive midplane for I/O modules. Twenty hot-plug capable PCI Express ExpressModule slots are accessible at the top of the Sun Blade 8000 chassis. Two PCI Express ExpressModule slots are dedicated and directly connected to each server module through the passive midplane. Slots 0 and 1 from right to left are connected to server module 0, slots 2 and 3 are connected to server module 1 and so on. Figure 3. The rear of the chassis provides access for I/O modules (both EM and NEM), Chassis Monitoring Modules, redundant hot-swap fans, and AC power plugs
Space is provided for up to four NEMs in the Sun Blade 8000 chassis, and up to two NEMs in the Sun Blade 8000 P chassis. NEMs provide the same I/O capabilities across all of the server modules installed in the chassis. All the server modules are directly connected to each of the configured NEMs through PCI Express connections. More details on NEMs are provided in chapter 3
. Below the NEM slots, up to two redundant Chassis Monitoring Modules (CMMs) provide remote monitoring and a central access Modules and capacities
Sun Blade 8000 modular server
Sun Blade 8000 P modular server
Sun Blade X8400 server modules 10 10
AMD Opteron processors 40 40
Processor cores per chassis/rack 80/160 80/240
PCI Express ExpressModules (EMs) 20 —
Network Express Modules (NEMs) up to 4 up to 2
Chassis Management Modules (CMMs) 2 2
Power supplies 6 (N+N) 4 (N+1)
Chassis per 42U rack 2 3
Up to 20 PCI Express ExpressModules Network Express Modules
Nine redundant Plugs/cords
and hot-swappable fans
Chassis Monitoring Modules
(two per server module)
Sun Blade 8000 P modular system
Sun Blade 8000 modular system
13
Sun Blade 8000 and 8000 P Modular System Overview
Sun Microsystems, Inc.
point to the chassis. The CMM includes an integrated switch that gives LAN access to the CMM's Ethernet port and to the individual server module ILOM Ethernet ports. Individual server module management is completely transparent and independent from the CMM. A single CMM is provided by default but a redundant unit can be configured for greater availability if desired.
Power supply inlets (plugs) are available from the rear of the chassis, and correspond to the number of power supply units — six in the case of the Sun Blade 8000 modular system and four in the case of the Sun Blade 8000 P modular system. These plugs connect the power cables to the front-loaded power supplies. Each of the cables require a 220V, 20A circuit, and a minimum of three circuits are required to power each chassis. For full redundancy, six circuits are required by the Sun Blade 8000 modular system (N+N) while the Sun Blade 8000 P modular system requires four circuits (N+1) for maximum redundancy. Nine rear fan modules provide cooling for the server modules. Each of the rear fan modules is hot-swap capable and includes two internal fans for complete redundancy. Passive midplane
All modules, front and rear, with the exception of the AC input and the system fans connect directly to the passive midplane. The power supplies connect to the midplane through a bus bar. AC distribution is provided via a cable harness from the AC inlets into floating connectors for each power supply. The redundant fan modules plug individually to a set of three fan boards, where fan speed control and other chassis-level functions are implemented. The front fan modules that cool the ExpressModules each connect to the chassis via blind-mate connections. The main functions of the midplane include:
• Providing a mechanical connection point for all of the server modules
• Providing 48V and 12V standby power from the power supplies to each customer-
replaceable module.
• Providing a PCI Express interconnect between the PCI Express root complexes on each server module to the EMs and NEMs installed in the chassis.
• Connecting the server modules, CMMs, and NEMs to the chassis management network
14
Sun Blade 8000 and 8000 P Modular System Overview
Sun Microsystems, Inc.
Figure 4. Distribution of PCI Express links from the Sun Blade X8400 server module
The passive midplane in the Sun Blade 8000 chassis supports an aggregate PCI Express throughput of up to 1.92 Tbps (192 Gbps supported for each server module, equivalent to six x8 PCI Express interfaces). The Sun Blade X8400 server module currently supports up to 160 Gbps, so the initial throughput available from Sun Blade 8000 chassis is 1.6 Tbps. As shown in Figure 4, each Sun Blade X8400 server modules delivers six PCI Express links through the passive midplane:
• Two x8 PCI Express links connect from each server module to each of the dedicated EM's
• Two x8 PCI Express links connect from each server module to the top two NEM's
• Two x4 PCI Express links connect from each server module to the bottom two NEM's
x8
x8
x8
x8
x4
x4
15
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
Chapter 3
Industry Standard Server, Networking, and Expansion Modules
The Sun Blade 8000 modular system provides high performance, capacity, and massive levels of I/O through full featured interfaces that use the latest technology and make the most of innovative chassis design. The Sun Blade X8400 server module, PCI Express ExpressModule (EM), and Network Express Module (NEM) are described in this section with the Chassis Monitoring Module (CMM) described in chapter 4
.
Sun Blade X8400 server module
The Sun Blade X8400 server modules provide significant computational, memory, and I/O density in a compact, efficient, and flexible package. Architectural overview
Each Sun Blade X8400 server module provides four sockets for 800 Series AMD Opteron dual core processors, yielding eight high-performance processor cores per server module. Each processor accesses four DDR400 DIMM slots. 1GB, 2GB, and 4GB DIMMS are supported for a maximum of 16GB of RAM per processor and a maximum total of 64GB per server module. Each server module also features two hot-swappable SAS or SATA disk drives and integrated per-module ILOM capabilities. As discussed, six PCI Express interfaces are provided directly to the passive midplane in the chassis. Figure 5 16
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
illustrates the physical layout of the Sun Blade X8400 server module, with each server measuring approximately 19.5 in. x 18.5 in. x 1.75 in. (height x depth x width).
Figure 5. Physical layout of the Sun Blade X8400 server module with support for four dual-core 800 Series AMD Opteron processors and up to 64 GB of memory
Figure 6 illustrates a logical block diagram of the Sun Blade X8400 server module. Each AMD Opteron processor features three 1GHz HyperTransport connections. These links connect either to the other processors, or directly to NVIDIA nForce Professional I/O bridges. Each processor also has a 6.4GB/s connection to its four DDR SDRAM DIMMs.
The I/O bridges are implemented with the nVidia NForce4 Professional 2200 (CK8-04) and the nVidia nForce4 Professional 2050 (IO-04) media and communications processors (MCPs). Each MCP provides 20 lanes of PCI Express I/O divided into 3 links (two x8 links and one x4 link) for a total of 40 lanes of PCI Express I/O bandwidth. These links are all directly wired to the midplane connector for connectivity to the two EMs dedicated to each server module and to the slots for up to four shared NEMs. In addition to providing flexible and upgradeable I/O connectivity, each server module provides 160 Gb/s of I/O throughput
1
. The CK8-04 bridge also implements the southbridge functionality including multiple USB2.0 ports, a serial port, and a 33MHz/32bit PCI bus. The PCI bus connects the internal RAID controller and the ATI Rage XL video card. For boot or local storage, the hardware RAID LSI 1064 SAS controller provides connectivity to the internal hot-swap Serial Attached SCSI (SAS) or low-cost Serial ATA (SATA) hard drives. RAID 0 or 1 are supported by this controller. The front panel of the Sun Blade X8400 server module exposes a VGA port for monitor connectivity and two USB ports for ILOM service processor
Four x8 and two x4 PCI Express slots
Four high-performance dual-core
AMD Opteron processors with
four DIMM slots per processor
Hot-plug SAS or SATA
2.5-inch drives with
integrated RAID 0 or 1
1. Next-generation server modules will take full advantage of the full midplane capacity of 192 Gbps
17
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
keyboard, mouse, or storage devices.
Figure 6. Block-level diagram of the Sun Blade X8400 server module in the Sun Blade 8000 chassis
Each server module contains its own power distribution fed by the 48V provided by the power supplies. The server module also provides 12V and standby power to its two associated EM modules. The Sun Blade X8400 server module also has its own ILOM Service Processor, described in more detail in chapter 4
. The front panel features the standard system indicators (Power, Attention, Locate, OK-to-Remove) and Power and Locate push-buttons.
Dual-core AMD Opteron
™
processors
• The AMD Opteron processor extends the ubiquitous x86 architecture to accommodate x64 64-bit processing. Formerly known as x86-64, AMD’s enhancements to the x86 architecture allow seamless migration to the superior performance of x64 64-bit technology. The AMD Opteron processor (Figure 7) was designed from the start for dual-core functionality, with a crossbar switch and system request interface. This approach defines a new class of computing by combining full PCIe Bridge
IO4
PCIe x4 - 16Gbps
PCIe x8 - 32Gbps
PCIe x8 - 32Gbps
PCIe x4 - 16Gbps
PCIe x8 - 32Gbps
PCIe x8 - 32Gbps
CK8-04
nForce4
6.4
GB/sec
6.4
GB/sec
6.4
GB/sec
6.4
GB/sec
PassiveMidplane
EM#0
NEM#2
NEM#0
EM#1
NEM#3
NEM#1
3 USB 2.0 Ports - Remote KMS
2 x 10/100MBps
Management Ethernet
DVI Video
Output
Serial
VGA Video Output
PCI32-bit/33MHz
LPC33MHz
Blade Module Front Panel
USB 2.0
VGA HD-15
DB-9 Serial
Video
Over LAN
Redirect
LSI
SAS1064
LSI
LOGIC
ATI
RageXL
Motorola
MPC8275
SP
BCM
Super I/O
Controller
18
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
x86 compatibility, a high-performance 64-bit architecture, and the economics of an industry-standard processor.
Figure 7. High-level architectural perspective of a dual-core AMD Opteron processor
Enhancements of the AMD Opteron processor over the legacy x86 architecture include:
• Sixteen 64-bit general-purpose integer registers that quadruple the general-purpose register space available to applications and device drivers as compared to x86 systems
• Sixteen 128-bit XMM registers for enhanced multimedia performance to double the register space of any current SSE/SSE2 implementation
• A full 64-bit virtual address space with 40 bits of physical memory addressing and 48 bits of virtual addressing that can support systems with up to 256 terabytes of physical memory
• Support for 64-bit operating systems to provide full transparent, and simultaneous 32-bit and 64-bit platform application multitasking
• A 128-bit wide, on-chip DDR memory controller that supports ECC and ChipKill technologies and provides low-latency memory bandwidth that scales as processors are added Each processor core has a dedicated 1MB Level-2 cache, and both cores use the System Request Interface and Crossbar Switch to share the Memory Controller and access the three HyperTransport links. This sharing represents an effective approach since performance characterizations of single-core based systems have revealed that the memory and HyperTransport bandwidths are typically under-utilized, even while running high-end server workloads.
Memory
Controller
HyperTransport 0
HyperTransport 1
HyperTransport 2
CPU 1
CPU 0
1MB L2 cache
1MB L2 cache
System request interface
Crossbar switch
Dual-core AMD Opteron
19
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
The AMD Opteron processor with integrated HyperTransport technology links provides a scalable bandwidth interconnect among processors, I/O subsystems, and other chip-
sets. HyperTransport technology interconnects help increase overall system performance by removing I/O bottlenecks and efficiently integrating with legacy busses, increasing bandwidth and speed, and reducing processor latency. At 16 x 16 bits and 1 GHz operation, HyperTransport technology provides support for up to 8GB/s bandwidth per link.
A choice of operating systems
Sun Blade X8400 server modules are certified to run a wide range of operating systems, providing considerable choice. The operating systems supported as of this writing include:
• The Solaris 10 Operating System with support for Solaris Containers (Solaris 10 OS for x64 Update 2 or later)
• Red Hat Linux (Red Hat Enterprise Linux AS4 update 3, 64-bit)
• SuSE Linux (SuSE Linux Enterprise Server 9 SPS, 64-bit SP3 or later)
• Microsoft Windows (Windows Server 2003, 32-bit Enterprise Edition and Standard Edition, Windows Server 2003, Enterprise x64 Edition and Standard x64 Edition)
• VMware ESX Server 3.0.1
Please visit http://www.sun.com/servers/blades/8000
for the current list of supported operating systems. Solaris
™
Operating System support for flexibility and enhanced performance
Among the available operating systems, the Solaris OS is ideal for large-scale enterprise deployments, and has features that can enhance the flexibility and performance of Sun Blade X8400 server module.
• Solaris Containers for consolidation, secure partitioning, and virtualization
Solaris Containers comprise a group of technologies that work together to efficiently manage system resources, virtualize the system, and provide a complete, isolated, and secure runtime environment for applications. Solaris Zones and Solaris Resource Management work together with the Solaris fair-share scheduler. Solaris Containers can be used to partition and allocate the considerable computational resources of the Sun Blade X8400 server module. Solaris Zones enable an administrator to create multiple separate environments for applications on a single system, while the resource management framework allows for the allocation, management, and accounting of system resources such as CPU and memory.
20
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
– Solaris Zones — Solaris Zones can be used to create an isolated and secure envi-
ronment for running applications. A zone is a virtualized operating system envi-
ronment created within a single instance of the Solaris OS. Zones can be used to isolate applications and processes from the rest of the system. This isolation helps enhance security and reliability since processes in one zone are prevented from interfering with processes running in another zone.
– Resource Management — Resource management tools provided with the Solaris OS help enable system resources such as CPU cycles to be dedicated to specific applications. CPUs in a multi-core multi-processor system such as the Sun Blade X8400 server module can be logically partitioned into processor sets and bound to a resource pool, which in turn can be assigned to a Solaris zone. Resource pools provide the capability to separate workloads so that consumption of CPU resources do not overlap, and also provide a persistent configuration mechanism for processor sets and scheduling class assignment. In addition, the dynamic fea-
tures of resource pools let administrators adjust system resources in response to changing workload demands.
• Solaris Dynamic Tracing (DTtrace) to instrument and tune live software environments
When production systems exhibit nonfatal errors or sub-par performance, the sheer complexity of modern distributed software environments can make accurate root-cause diagnosis extremely difficult. At the same time, the inability to meet anticipated user load or the failure to meet service-level requirements can be very costly to the business. Unfortunately, most traditional approaches to solving this problem have proved time-consuming and inadequate, leaving many applications languishing far from their potential performance levels.
The Solaris DTrace facility provides dynamic instrumentation and tracing for both application and kernel activities — even allowing tracing of application components running in a Java
™
Virtual Machine (JVM
™
)
1
. DTrace enables developers and administrators to explore the entire system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior. Tracing is accomplished by dynamically modifying the operating system kernel to record additional data at locations of interest. Best of all, although DTrace is always available and ready to use, it has no impact on system performance when not in use, making it particularly effective for monitoring and analyzing production systems.
• NUMA optimization in the Solaris OS
With memory managed by each processor, the Sun Blade X8400 server module represents a non-uniform memory access (NUMA) architecture. The speed for a processor to access its own memory is slightly different than to access memory managed by another processor. The Solaris OS provides technology that can help applications improve performance on NUMA architectures.
1. The terms “Java Virtual Machine” and “JVM” mean a Virtual Machine for the Java platform
21
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
– Memory Placement Optimization (MPO) — The Solaris 10 OS uses MPO to improve the placement of memory across the physical memory of a server, resulting in increased performance. Through MPO, Solaris 10 works to ensure that memory is as close as possible to the processors that access it while still maintaining enough balance within the system. As a result, many database and HPC applications are able to run considerably faster.
– Hierarchical lgroup support (HLS) — HLS improves the MPO feature in the Solaris OS. HLS enables the Solaris OS to optimize performance for systems with more complex memory latency hierarchies. HLS enables the Solaris OS to distin-
guish between the degrees of memory remoteness, allocating resources with the lowest possible latency for applications. If local resources are not available by default for a given application, HLS helps the Solaris OS allocate the nearest remote resources.
PCI Express ExpressModules (EM)
Industry-standard I/O, long a staple of rack-mount and vertically-scalable servers has been elusive in legacy blade platforms. Unfortunately the lack of industry-standard I/O has meant that customers often paid more for fewer options, and were ultimately limited by a single vendor’s innovation. Unlike legacy blade platforms, the Sun Fire 8000 modular system accommodates up to 20 PCI Express ExpressModules (EM) compliant with new PCI SIG form factor. This approach allows for a wealth of expansion module options from multiple ore I/O options from multiple expansion module vendors, and avoids a single-vendor lock on innovation.
The passive midplane implements connectivity between the EMs and the server modules, and physically assigns pairs of EMs to individual server modules. As shown in Figure 8, EMs 0 and 1 (from right to left) are connected to server module 0, EMs 2 and 3 are connected to server module 1, EMs 4 and 5 are connected to server module 3, and so on. Each EM is supplied with an x8 PCI Express link back to its associated server module providing up to 32 Gb/s of I/O throughput. EMs are hot-plug capable according to the standard defined by the PCI SIG, and fully customer replaceable without opening either the chassis or removing the server module. 22
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
Figure 8. A pair of 8-lane (x8) PCI Express slots allow up to two PCI Express ExpressModules per server module in the Sun Blade 8000 chassis
With the industry-standard ExpressModule form factor, a wide range of EMs are anticipated. Initially available EMs include:
• Dual-port GbE EM (Intel NICs)
• Dual-port FiberChannel EM (Qlogic)
• Dual-port 4X InfiniBand EM (Mellanox)
For the latest information please refer to www.sun.com/servers/blades/8000
Network Express Module (NEM)
Many legacy blade platforms include integrated network switching as a way to gain aggregate network access to the individual server modules. Unfortunately, these switches are often restrictive in their options and dictate topology and management choices. As a result, data centers often find legacy blade server platforms difficult to integrate into their networks, or are resistant to admitting new switch hardware into their chosen network fabrics.
The Sun Blade 8000 and 8000 P modular systems address this problem through a specific Network Express Module (NEM) form factor that provides configurable network I/O for up to 10 server modules at a time. Connecting to all of the installed server modules through the passive midplane, NEMs represent a space efficient mechanism for providing high-density, configurable I/O. A NEM provides bulk or aggregated I/O options for the entire chassis, ranging from simple network connectivity to implementations of high-speed interconnects such as FiberChannel or Infiniband.
ServerModule9
ServerModule5
ServerModule4
ServerModule3
ServerModule2
ServerModule1
ServerModule0
ServerModule8
ServerModule7
ServerModule6
23
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
NEMs appear to the server modules as a standard PCI Express adapter and both Sun-
supplied and third-party NEMs are anticipated. Having a model based on PCI Express is highly flexible because the NEM can be built using any standard PCI Express silicon available in the market, and integrated into a 10-blade modular I/O device. Individual NEMs may provide simple pass-through networking or may implement functions such as switching. Management may be exposed or hidden depending on the function of the NEM.
The initial NEM released with the Sun Blade 8000 and 8000 P modular systems is a 20-port Gigabit Ethernet pass-through module, providing a dual Gigabit Ethernet NIC to each server module and exporting all of the resulting 20 Gigabit Ethernet links out the rear panel via RJ-45 connections. Other possible NEMs from Sun or third parties include Fiber Channel HBAs for SAN connectivity or InfiniBand HBA's for I/O virtualization applications in the same pass-through format. A second class of possible NEMs involves the addition of an aggregation function to the I/O interface. In this model, each server module still owns a dedicated I/O chip, such as an Ethernet NIC, but instead of bringing each blade's I/O interface out to the rear panel, the NEM provides an aggregation function such as an embedded Ethernet switch. By providing high bandwidth links at the rear panel, such as 10 Gigabit Ethernet, the NEM implements an aggregation function, thereby reducing external switching and cabling needs and simplifying configuration and management.
Gigabit Ethernet Network Express Module
The Gigabit Ethernet NEM initially available with the Sun Blade 8000 and 8000 P modular system contains a dual Gigabit Ethernet NIC device for each of the 10 server modules supported in a chassis. Each of the NIC's Ethernet ports is connected to two RJ-45 connectors providing two 10/100/1000BaseT ports. Figure 9 illustrates the mapping of Gigabit Ethernet NICs specific to each server module to RJ-45 connectors on the back panel.
24
Industry Standard Server, Networking, and Expansion Modules
Sun Microsystems, Inc.
Figure 9. The Gigabit Ethernet NEM provides two 10/10/1000 BaseT ports for each installed Sun Blade X8400 server module
ServerModule0
ServerModule1
ServerModule2
ServerModule3
ServerModule4
ServerModule5
ServerModule6
ServerModule7
ServerModule9
ServerModule8
25
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
Chapter 4
Open and Standard Chassis and System Management
Management in legacy blade platforms has typically either been lacking, or administrators have been forced into adopting unique and platform-specific management infrastructure. To address this issue, the Sun Blade 8000 and 8000 P modular systems provide a wide range of flexible management options at both the server module and chassis level.
Individual Sun Fire X8400 server modules provide IPMI, HTTPs, CLI (SSH), SNMP, and file transfer interfaces that are directly accessible from the Ethernet management port on the Chassis Monitoring Module (CMM). Each server module is assigned an IP address (either manually, or via DHCP) that is used for the management network. Each Sun Blade X8400 server module contains its own ILOM service processor, giving it similar remote management capabilities to Sun Fire
™
X4000 servers. The Chassis Monitoring Module (CMM) provides the basic chassis monitoring and management functions, and also implements the management network. Similar to the service processor on individual sever modules, the CMM includes an implementation of ILOM that has the purpose of monitoring and management of the chassis along with monitoring the server modules.
Chassis Monitoring Module (CMM)
The CMM is the primary point of management of all shared chassis components and functions, providing a set of management interfaces in addition to the overall monitoring of the chassis and chassis functions.
CMM network functionality
One or two CMMs can be configured into the Sun Blade 8000 and 8000 P chassis with the optional second CMM providing redundancy and failover. Each CMM has an individual IP address assigned either statically or dynamically via DHCP. If dual redundant CMMs are configured, a third Master IP Address may be configured that “floats” between the two CMMs depending on which one is currently acting as the master. In this fashion, a single IP address automatically identifies the active master CMM.
The CMM provides complete chassis monitoring and management functionality while providing access to server module management functions. In addition, the CMM supports HTTP and CLI “pass-thru” interfaces that provide transparent access to each server module. For the purposes of IPMI environmental management, the CMM appears to each blade's ILOM service processor as a satellite Baseboard Management 26
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
Controller (BMC), providing the status of the chassis elements to the server module service processor. In addition, the CMM provides access to each blade via a single serial port through which any of the individual server module’s ILOM consoles can be accessed. The CMM's management functions include:
• Implementation of an IPMI satellite controller, making the chassis environmental sensors visible to the blades' BMC functions
• Direct environmental and inventory management via HTTP/CLI/SNMP/IPMI interfaces
• CMM, ILOM, and NEM firmware management
• Pass-through management of blades using HTTP links and command line interface (CLI) SSH contexts
The management network internal to the CMM joins the local management processor on each server module to the NEM — providing connectivity between the ILOM service processors, NEM management processors, and the CMM processors. The management network is formed via a set of Ethernet switch chips on the CMM modules.
CMM architecture
Each of the CMMs are composed of an Ethernet switch and an enclosure service processor. The switch is dedicated exclusively to remote management network traffic, enabling administrators to access the remote management functions of both the CMM and the server modules. The switch in each CMM provides a single network interface to each of the server modules and to each of the NEMs. With dual redundant CMMs installed, the server modules are provided with redundant network interfaces to the management network. The interfaces in the server modules’ ILOM are configured using bonding technology that provides failover functionality in case of a CMM failure, or in case a network path is broken. Figure 10 provides a block-level diagram of the CMM.
Figure 10. CMM block diagram
Management
Switch
ExpansionBus
Switch
2
Switch
1
To
Management
Switch
PCI
UART
Boot
Flash
64MB
SC RAM
Motorola
MPC8275
SP
UART
To Power
Supplies
ToOtherCMM’s
ManagementSwitch
ToI2CSubsystem
To Misc.Status & Control Logic
(Server Modules,PEM,SC,Fan Board)
PLD
Ethernet
Ethernet
Buddy CMM (MPC)
Server Module 0
Server Module 1
Server Module 5
Server Module 2
Server Module 3
Server Module 4
Buddy CMM (GBE)
Server Module 6
Server Module 7
Server Module 9
Server Module 8
NEM 3
NEM 2
NEM 1
NEM 0
Buddy CMM (GBE)
27
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
The CMM’s ILOM functionality enables various management functions, including power control of the chassis as well as hot-plug operations of infrastructure components such as power supply modules, fan modules, server modules, and NEMs. The CMM’s ILOM acts as a conduit to server module ILOM settings, allowing settings such as network addresses and administrative users to be configured or viewed. Each CMM’s processor originates two Ethernet links. One link connects to the local Ethernet switch and the other is connected to the ethernet switch on the other (optionally redundant) CMM module through the passive midplane. The internal switch is built out of two smaller interconnected switches. The two links which connect to each server module and each NEM are connected directly to the two NIC interfaces provided by the management processor on each of these devices. As a result, every management processor in the system has two paths to every other management processor as well as two paths out of the chassis via the external Gigabit links that leave the rear of each CMM module.
When two CMMs are configured for redundancy, their Gigabit Ethernet connections and management switches are connected through the passive midplane (Figure 11).
Figure 11. Optional dual redundant Chassis Monitoring Modules (CMMs) connect through the passive midplane Server module ILOM service processor
By utilizing an ILOM service processor, the Sun Blade X8400 server module brings flexible and sophisticated management capabilities to a modular environment. Critical to effective system management, the ILOM service processor:
• Implements an IPMI 2.0 compliant BMC, providing IPMI management functions to the server module's BIOS, OS and applications, and to IPMI-based management tools accessing the BMC either thru the OS interfaces, or via the ILOM Ethernet Buddy CMM
(GB Ethernet)
xxx
Passive
Midplane
Buddy CMM
(GB Ethernet)
xxx
CMM’s
Connections to the
Buddy Management
Switch
28
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
management interface, providing visibility to the environmental sensors (both on the server module, and elsewhere in the chassis using a satellite controller model)
• Manages inventory and environmental controls for the server module, including CPUs, DIMMs, and EMs, and provides HTTP/CLI/SNMP access to this data
• Supplies remote textual and graphical console interfaces, as well as a remote storage (USB) interface (collectively these functions are referred to as Remote Keyboard Video Mouse and Storage (RKVMS)
• Provides a means to download BIOS images
On-board ILOM firmware and connections
The ILOM service processor connects to all major components via on-board interfaces such as I
2
C with a separate management network provided for remote access. Equipped with field-upgradeable firmware, each ILOM supplies management functions for fan speed control and diagnostic LEDs, and provides a wealth of connections to individual server module components, including:
• An LPC bus for KCS (including three interfaces for BIOS, OS driver, and SMI) and BIOS ROM programming (with the host off or in reset mode)
• 100 Mbit/sec Ethernet management network interface
• Two serial ports (one to the host SIO and one to the DB-9 port) allowing forwarding of the host serial port to the DB-9 connector or to the network via a command line interface over secure shell (ssh).
• Three USB 2.0 ports (floppy, DVD, and composite keyboard/mouse)
• One digital video port
• GPIOs to reset the system, control the system power, and signal graceful shutdown via ACPI
• I
2
C connections to various sensors (voltage, temperature, and fans) as well as field-
replaceable ROMs
Server module ILOM communication channels, user management, and security
Access to ILOM functionality on the Sun Blade X8400 server modules can be made through a variety of both out-of-band and in-band communication channels. Out -of-
band communication helps ensure that effective management can take place even in the event of hardware or networking failures, and includes:
• A management serial port that provides direct console access via a command line interface (CLI) • A dedicated Ethernet port that provides a web-based GUI (over HTTPS), a CLI via SSH, IPMI 2.0, and SNMP v1, v2c, and v3
In-band communication to the ILOM service processor is provided via the host OS running on the Sun Blade X8400 server module. A variety of management tools can be used to access management information on individual server modules:
29
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
• Intelligent Platform Management Interface (IPMI) with IPMItool
IPMItool is a simple command-line interface to systems that support the Intelligent Platform Management Interface (IPMI) v2.0 specification. IPMItool provides the ability to read the sensor data repository and print sensor values, display the contents of the system event log, print field-replaceable unit information, read and set LAN configuration parameters, and perform remote chassis power control. IPMItool was originally written to take advantage of IPMI-
over-LAN interfaces but it is also capable of using the system interface as provided by a Linux kernel device driver such as OpenIPMI or a Solaris OS driver called BMC that is provided with the Solaris 10 OS. IPMItool is available under a BSDcompatible license.
IPMItool is not designed to replace the OpenIPMI library but instead provides a completely command-line oriented tool that can be used by administrators in conjunction with other tools. Where possible, IPMItool supports comma-
separated values for output to facilitate parsing by other scripts or programs. IPMItool is designed to run quick command-response functions that can be as simple as turning the system on or off, or as complex as reading in the sensor data records while extracting and printing detailed sensor information for each record.
• Simple Network Management Protocol (SNMP) management
SNMP management provides remote access by SNMP-compliant entities to monitor and control network devices and manage configurations including statistics collection, performance, and security on a network. SNMP is a network management protocol used almost exclusively in TCP/IP networks. The Sun Blade X8400 server modules provide SNMP MIBs to manage and monitor the servers using any SNMP-capable network management system, such as HP OpenView Network Node Manager (NNM), Tivoli, CA Unicenter, or IBM Director. The MIB data describes the information being managed, reflects current and recent server status and provides server statistics.
SNMP v1, v2c, and v3 are supported with v3 enabled by default (v1 and v2c are disabled by default). SNMP “sets” can be enabled or disabled (default). An IPMI-
specific trap called a Platform Event Trap, or PET, may also be generated. The following SNMP MIBs are supported:
– SNMP-FRAMEWORK-MIB
– SNMP-USER-BASED-SM-MIB
– SNMP-MPD-MIB
– ENTITY-MIB
– SUN-PLATFORM-MIB
• Role-based administration
Different management users can be defined with corresponding roles and responsibilities. Up to 10 user IDs can be created locally on the service process 30
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
with each user ID consisting of a user name and the roles that are allowed. By default, Administrator and Operator roles are defined. Authentication is carried out against a local service processor database. Alternately, an LDAP client is implemented in the ILOM service processor as well to allow authentication against an LDAP server (LDAP groups must be mapped to service processor roles). Up to 10 concurrent active sessions are supported on the service processor, including serial, secure shell and web clients.
Remote keyboard, video, mouse, and storage (RKVMS)
To facilitate effective and full-featured remote management, the ILOM service processor provides remote keyboard, video, mouse, and storage (RKVMS) support that is tightly integrated with the Sun Blade X8400 server module. Together these capabilities allow the server module to be administered remotely, while accessing keyboard, mouse, video and storage devices local to the administrator (Figure 12). ILOM Remote Console support is provided on the ILOM service processor and can be downloaded and executed on the management console. Input/output of virtual devices is handled between ILOM on the Sun Blade X8400 server module and ILOM Remote Console on the web-based client management console.
Figure 12. Remote keyboard, video, mouse, and storage (RKVMS) support in the ILOM service processor allows full-featured remote management for Sun Blade X8400 server modules
• Remote keyboard and mouse support
Through the ILOM service processor, the Sun Blade X8400 server module detects a USB keyboard and mouse. ILOM Remote Console captures mouse and keyboard input on the management console and sends it to the ILOM service processor. The Management
Console
CDROM,DVDROM
or.iso Image
Keyboard,Mouse,CDROM,
and Floppy are Seen as
USB Devices by BIOS and O
S
ILOM Remote Console
Displays Remote Video in
Application Window
Video
(Up to 1024x768@60Hz)
ILOM Remote Console
Connected to ILOM Over
Management Ethernet
Local Mouse and
Keyboard
Sun Blade X8400
Serve
r Module
Graphics Redirect Over Ethernet
Floppy Disk or
Floppy Image
Remote Keyboard,Mouse and Storage
Emulated as USB Devices by ILOM
31
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
service processor then transmits mouse and keyboard inputs on the respective USB buses on the server. The server receives keyboard entries and mouse movements as if they were generated by local USB devices.
• Remote video support
Each Sun Blade X8400 server module incorporates an ATI Rage XL graphics controller capable of up to 1024x768@60Hz display. Graphics from the ATI graphics controller is sent to the ILOM service processor. The service processor then redirects the video signal to ILOM Remote Console running on the management system over a network connection, where the video is displayed on the management console. ILOM Remote Console supports both 8- and 16-bit video modes to accommodate needs for both low bandwidth connections (8-bit) and higher quality (16-bit).
• Remote virtual storage
The Sun Blade X8400 server module detects two USB storage devices through the ILOM service processor that can be set up in the BIOS as floppy, CD/DVDROM, or disk image (floppy and CD/DVDROM by default). When the server tries to access either the virtual floppy disk or the virtual CDROM, the ILOM service processor redirects the request over the Ethernet connection to ILOM Remote Console on the management console. ILOM Remote Console accesses the content from the physical floppy disk drive, CD/DVDROM, or disk image, returning it across the network to the server module. The ILOM service processor presents the data to the server as if it were coming from a local USB storage device. This capability allows the Sun Blade X8400 server module to boot remotely from a virtual USB storage device. The content of the storage device can be an actual CD, floppy disk, or disk image (ISO or IMG).
Sun N1
™
system management software
Beyond local and remote management capabilities, data center infrastructure needs need to be agile and flexible, allowing not only fast deployment but streamlined re-
deployment of resources as needs dictate. Sun N1 management technology provides software and services for life-cycle management of business services and infrastructure, including the Sun Blade 8000 modular system. To improve life-cycle management and change management, N1 software supports management of business services and the servers on which they run.
•Sun N1 System Manager
Sun N1 System Manager simplifies infrastructure life cycle management by enabling administrators to perform standardized actions across logical groups of systems. Sun N1 System Manager can automatically discover and group bare-
metal systems, performing actions on the entire group as easily as operating on a single system. Sun N1 System Manager remotely installs and updates operating systems, firmware, Solaris patches and packages, and RedHat Linux RMP packages 32
Open and Standard Chassis and System Management
Sun Microsystems, Inc.
onto selected systems. In addition, the software provides considerable lights-out monitoring of both hardware and software, including fans, temperature, disk and voltage levels as well as swap space, CPU utilization, memory capacity, and file systems. Role-based access control lets IT staff grant specific management permissions to specific users. A convenient hybrid user interface integrates both a command-line interface (CLI) and an easy-to-use graphical user interface (GUI), providing remote access to manage systems from virtually anywhere.
Sun N1 System Manager provides advanced management and monitoring features to the Sun Blade 8000 modular system. The remote management interface discovers and presents the Sun Blade X8400 server modules in the enclosure as if they were individual servers. In this fashion, the server modules appear in exactly the same way as individual rack-mount servers, making the same operations, detailed inventory, and status pages available to administrators. The server modules are discovered and organized into logical groups for easy identification of individual modules, and the system chassis and racks that contain them. Organizing servers into groups also allows features such as OS deloyment across multiple server modules. At the same time, individual server modules can also be managed independently from the rest of the chassis. This flexibility allows for management of server modules that may have different requirements than the other modules deployed in the same chassis.
Some of the functions available through Sun N1 System Manager software include operating system provisioning, firmware updates (for both the BIOS and ILOM service processor firmware), and health monitoring. In addition, Sun N1 System Manager includes a shell tool that allows easy access to inventory information, and also simplifies the task of running jobs on multiple servers thanks to the server grouping functionality.
• Sun N1 Service Provisioning System
Tightly integrated with Sun N1 System Manager, the Sun N1 Service Provisioning System increases productivity by automating application provisioning by eliminating manual tasks and time spent maintaining custom scripts. Instead, administrators can deploy, configure, and update network services from a Web browser. At the touch of a button, the Sun N1 Service Provisioning System automatically checks application dependencies, deploys needed software from a central repository, configures applications based on a predefined component model, and logs all actions. By eliminating manual, repetitive tasks, the Sun N1 Service Provisioning System greatly simplifies complex software installation and configuration, helping to speed deployment of new business services, increase application availability, and reduce operating expenses. At the same time data center personnel are freed to focus on more strategic initiatives and operating expenses are reduced through one-touch deployment to many systems.
33
Conclusion
Sun Microsystems, Inc.
Chapter 5
Conclusion
To support the endless demand for scalability, reliability, and manageability in the data center, application infrastructure needs to provide ever-increasing performance and capacity while becoming simpler to deploy, adjust, and manage. Unlike traditional legacy blade architecture, the Sun Blade 8000 and 8000 P modular systems deliver the required power and performance for compute-, memory-, and I/O-intensive applications in a dense and highly manageable package with a choice of operating systems. With Sun Blade X8400 server modules supporting the fastest dual-core AMD Opteron processors, large memory capacity, and high-bandwidth industry-standard I/O capabilities, high-performance applications can finally leverage the advantages of modular architecture.
Ultimately, effective and successful modular platforms require more than just high compute density and processor performance. By placing the data center as a design focus, the Sun Blade 8000 and 8000 P modular systems succeed through innovative architecture in combination with robust operating system and management software. This approach helps ensure that individual server modules are as reliable as possible by moving key functionality to the chassis itself. Redundant power and cooling help ensure uptime for the entire chassis while the passive midplane delivers maximum I/O to individual server modules through industry-standard interfaces. Integral chassis and server module management along with Sun N1 management tools help ensure uptime and rapid deployment of essential services. With the Sun Blade 8000 and 8000 P modular systems, organizations can finally begin to consolidate mission- and business-critical applications onto a high-performance modular platform. Performance-hungry enterprise applications, high-performance databases, and even compute-intensive HPC and grid applications can be run and consolidated on these systems with confidence that they are designed to accommodate the future.
Sun Blade 8000 and 8000 P Modular Systems On the Web sun.com
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
© 2006 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, Java, JVM, N1, Solairs, Sun Blade, and Sun Fire are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. AMD, Opteron, and the Opteron logo are trademarks of Advanced Micro Devices, Inc. Information subject to change without notice. Printed in USA 11/06 
Автор
golick
Документ
Категория
Информатика
Просмотров
63
Размер файла
1 885 Кб
Теги
sun, blade servers
1/--страниц
Пожаловаться на содержимое документа