29
Abr
09

Net App Virtualization….. Presentacion de un caso

Como todo mundo esta apostando a la virtualizacion he aqui un caso que nos presenta la gente de Net App.

tech-on-taptech-on-tap1

Consolidating Up to 1000 Physical Servers onto
20 VMware ESX Servers

At the NetApp engineering lab in Bangalore, India, our operational goal is to meet the compute and storage needs of roughly 700 development engineers working on a wide variety of critical NetApp development efforts, including WAFL®, NAS and SAN, storage management, VTL, open systems SnapVault® (OSSV) and SnapDrive® for UNIX®.

As of August 2008, we had deployed about 1,000 physical x86 servers to meet the diverse needs of these and other engineering projects, and the demand was continuing to grow at a rate of about 40 servers per month—roughly two racks of compute servers every month. This growing demand created a concern about available rack space, power, and cooling capacity. The situation was further complicated by the slow and inflexible provisioning processes of physical servers, making it difficult for us to meet engineers’ demands.

Project COLD (Consolidation and Optimization of Lab/Datacenter) was initiated with the goal of consolidating and virtualizing these critical engineering resources. We are currently in the middle of this transition. At completion, we expect to replace 50% or more of our original 1,000 physical servers with just 20 VMware® ESX servers. A limited number of physical servers (final number to be determined) will be retained for application testing that requires physical hardware.

Our VMware servers will also accommodate future growth while greatly improving our ability to adapt to changing requirements from engineers.

In this article we’ll describe:

  • Consolidation and virtualization objectives
  • Key challenges
  • Configuration and provisioning processes
  • P2V migration
  • Current state of the transition
  • Future plans

Consolidation and Virtualization

Server and storage consolidation reduces costs by improving operating efficiency and effectiveness, while virtualization seeks to maximize the value (in terms of both capital expenses and operating expenses) of server and storage technology through increased automation, scalability, and the ability to offer multiple functions or services from a single platform. Our new data center is headed toward a virtualized model that will allow us to utilize more versatile technologies to respond to fluctuating engineering demands and meet business requirements as they rise and fall. Virtualization technology adds value to our data center by increasing flexibility, scalability, ease of management, and responsiveness.

Virtualization allows IT operations to be performed with far better economies of scale, maximizing the utilization of existing resources and allowing infrastructures to be managed efficiently even as they undergo high rates of growth. We evaluated various virtualization solutions before deciding on VMware in conjunction with NetApp® storage. Ultimately we chose VMware because of the wide range of guest operating systems it supported, which was a key requirement for this project.

Initial Challenges

At the outset, we faced a variety of organizational and technical challenges that we had to overcome to move the project forward.

Budget constraints. First, since our budget was limited, this virtualization project had to start with minimal equipment. The project was initiated with two loaned servers (which were already installed with VMware ESX server) plus a standalone NetApp FAS3050 storage system. This gave us enough equipment to do a proof of concept (POC). These loaned servers were upgraded with additional memory so they could handle more virtual machines.

Concerns from engineers. At the outset, engineers were skeptical. They didn’t believe that a virtual machine would be able to handle the same load as the physical servers they were used to. This was further complicated by the fact that the needs of each engineering project were somewhat unique. However, using just the two initial virtual servers we were able to convince enough engineers to keep moving forward.

Network integration. With that hurdle out of the way, we then had to determine how we could integrate a virtual server farm into our existing environment. The engineering lab network was designed to minimize Layer 2 broadcasting. Multiple VLANs are configured in access layer switches with an uplink to core backbone switches using Layer 3. (This network architecture corresponds to a campus LAN where each function/department uses a separate VLAN.)

We either had to provide an individual ESX server on each VLAN or modify the network topology. With the first option, resources would not be fully utilized, we would need more ESX licenses, and management would be more difficult. However, the second option was complex and could require significant downtime to accomplish.

After a lot of discussion with our networking partners and engineering customers, we settled on a solution consisting of a consolidated virtual server farm to host all our ESX servers, storage systems, and networking gear with links to each project VLAN.

<!––><!––>

Virtual server farm and network configuration

Figure 1) Virtual server farm and network configuration.

Virtual Server Farm Configuration and Provisioning

We’ve followed the guidelines in TR-3428: NetApp and VMware Virtual Infrastructure 3: Storage Best Practices in designing this setup. Our virtual infrastructure has now grown to a farm of 8 servers and a NetApp FAS3050 cluster, hosting 436 virtual machines to support 17 engineering teams. We’ve so far accomplished 150 physical-to-virtual (P2V) conversions and 100 GSX to ESX migrations. We had been doing a limited amount of virtualization in a distributed way for some projects using VMware GSX. We typically had four or five virtual machines per physical server.

Resource pools are configured on VMware Virtual Center to group and manage sets of virtual machines on a per team basis. Network connectivity to each team is provided by two 1GB network ports, teamed for load balancing and redundancy.

Our clustered FAS3050 is equipped with four disk shelves using 300GB FC drives and multipathing for storage failover. The network ports of the storage system are trunked and configured as NetApp VIFs for redundancy and load balancing.

All ESX datastores are hosted from this storage system using NFS. We opted for NFS because it was cost effective and easy to configure and manage. Fibre Channel datastores would have required additional hardware such as FC switches, HBAs, and cabling, which could not be procured given the budget constraints we had. Moreover, performance using NFS datastores was comparable with that of FC.

New virtual machines are provisioned with NetApp’s rapid cloning utility version 1. This process leverages NetApp FlexClone® so that similar virtual machines can share the same storage without wasting a lot of space storing redundant copies of the same operating system files. You can find out more about this process in a recent Tech OnTap article, which describes the same process for VMware virtual desktops.

Migrating from Physical to Virtual

Because engineering teams want existing server configurations to remain unchanged, we are faced with the challenge of retaining hostnames, IP addresses, and operating system configurations as we transition servers from physical to virtual.

To accomplish these migrations, we first extend the data network for each team to the Virtual server farm. P2V conversions are achieved using VMware Virtual Center. Most of the migrations happen during the weekends or on holidays to reduce the impact of downtime. New virtual machines are monitored regularly for performance issues. We also work closely with teams in case of any performance issues and allocate more resources to virtual machines as necessary.

Current Status

Virtual machine growth and physical server decommissioning count are monitored regularly and plotted in graphical format. We plan to bring down the physical server count to 500 within the next six to nine months. Virtual machine count is expected to grow to 1,500 during the same period. We expect to accommodate all 1,500 virtual machines with just 20 VMware ESX servers. Our current eight servers are supporting 450 virtual machines with an average CPU and memory utilization hovering around 30%, leaving us plenty of headroom to support more virtual machines.

Virtual server farm and network configuration

Figure 2) Transition progress.

Future Plans

While we’re very happy with our progress so far, we can also see some additional opportunities made possible by our virtual environment:

  • Multiplatform support. Our current environment only includes servers based on Intel®. We’d also like to support virtualzation of the PowerPC platform and AIX operating environment using IBM logical partitions (LPAR) and SPARC using Solaris™ containers.
  • Single dashboard. Our current management environment relies on Virtual Center to monitor and manage VMware servers/virtual machines and NetApp Operations Manager to monitor and manage storage. We’re currently exploring the use of NetApp SANscreen® to allow us to see both servers and storage on a single dashboard.
  • Business continuity. Recovery in a physical server environment is difficult. Our new virtual environment will give us the ability to provide a much higher level of business continuity. We’d like to use SnapMirror to mirror all virtual machine data to a single NetApp NearStore® system. This would give us the ability to recover quickly from any server or storage hardware failure and provide an off-site copy of this data for site recovery.
  • Servers and storage on demand. Our ultimate dream is to create a self-service environment in which engineers can request server and storage resources online and those resources can be immediately provisioned without administrator involvement.

Conclusion

Although we are only partway through our transition, we are already seeing substantial benefits from the virtual environment:

  • Faster provisioning. Because of our rapid growth, it was previously difficult to stay ahead of demand or accommodate unexpected requests. If an engineering project needed a couple additional servers to run some tests, it could take up to four weeks to obtain and configure the necessary hardware. Now we can provision new virtual servers in a matter of minutes.
  • Load balancing. Similar to provisioning, if a physical server became overloaded, a painful and lengthy reprovisioning process was often needed. Today, we regularly monitor VMs for performance issues and quickly add resources using VMware tools as necessary. Should a particular VMware server become overloaded, we can use VMotion® to move virtual machines and rebalance the load with minimal disruption.
  • Improved resiliency. We can now recover more quickly from server/operating system failures. If a physical server suffers a hardware failure, that’s obviously a long process. If a failure occurs on a VM, we can quickly restart it. If a virtual server were to fail, we can quickly restart its load on the remaining virtual servers using VM migration.
  • Decreased downtime. Maintenance features of both VMware and NetApp storage makes it possible for us to perform maintenance without incurring downtime, lessening the impact to engineers.

The net result of all these enhancements is a more flexible, resilient development and test environment that ultimately improves engineer productivity and decreases time to market. Recognizing the advantage of this approach, other NetApp engineering labs are following a similar approach.

The authors would like to extend a special thanks to the entire NetApp Bangalore Engineering Support team, all of whom worked tirelessly to make the project a success! Jim Harrigan, Engineering Support, and Sunita Rao, NFS Product Manager, provided particularly valuable guidance.

Got opinions about VMware and NetApp?

Ask questions, exchange ideas, and share your thoughts online in NetApp communities.

Rick Scherer

John Cherian
Site Manager, Engineering Support
NetApp Bangalore

When John (Middle) joined the Bangalore Engineering Support team as an individual contributor almost five years ago, the team supported just five racks of equipment. As site manager, he has built a 24-member team currently responsible for some 360 racks of equipment. John also played a lead role in evolving Engineering Support Global Technical Operations during last year. He was trained as a medical doctor, but a passion for technology resulted in a switch to IT 12 years ago.

Suresh Kumar
Senior UNIX Administrator
NetApp Bangalore

Suresh (right) has worked as a UNIX administrator for eight years; he worked at HP before joining NetApp a little over two years ago. At NetApp he is focused on the Bangalore engineering data center and has been a key contributor to the COLD project.

George Stephen
Windows Administrator
NetApp Bangalore

George (left) has been with NetApp for the last three years as a Windows administrator. He has focused specifically on virtualization technologies for two years.

Anuncios

0 Responses to “Net App Virtualization….. Presentacion de un caso”



  1. Dejar un comentario

Responder

Por favor, inicia sesión con uno de estos métodos para publicar tu comentario:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s


A %d blogueros les gusta esto: