This is part 1 of the following blog series:
- My first FlexPod! (Part 1 – Introduction)
- My first FlexPod! (Part 2 – Hardware Overview)
- My first FlexPod! (Part 3 – Hardware Configuration)
- My first FlexPod! (Part 4 – Quality of Services)
- My first FlexPod! (Part 5 – Hyper-V Cluster)
- My first FlexPod! (Part 6 – System Center)
Introduction:
Two years ago I had the opportunity to design, develop and implement a new public and private cloud infrastructure. Because we were going to host multiple customers (tenants) including our selves on a shared infrastructure this required a Secure Multi-Tenancy environment. As you may probably know, a Secure Multi-Tenancy environment brings a lot of challenges that you have to deal with. We were going to use Microsoft Hyper-V on Windows Server 2012 R2 and SCVMM (System Center Virtual Machine Manager) 2012 R2 as the virtualization platform. Because it was going to be a private and public cloud service it required the right hardware that is highly scalable. At that time we looked at DELL, HP and Cisco for compute, network and storage. At the end we pulled the plug and chose for a Cisco FlexPod solution. And this is where it all started…
Well, I can tell you now; Cisco FlexPod is awesome! Now that we have our infrastructure running for about two years I am so enthusiastic about this system that is literally made a kind of career shift for me. I am not going totally for Cisco instead of Microsoft, but more a combination of both. As we would say, best of both worlds! And this already brought me a lot of new opportunity’s. Because I am so enthusiastic about a Cisco FlexPod (or better known as Cisco UCS with NetApp) I get a lot of questions from people around me. That’s why I decided to write this blog, to share my experience and hopefully make other people as enthusiastic as me. Did I just say the word ‘enthusiastic’ four times? 😉
If you are interested in this topic than I recommend you to read this blog. Please understand that there are so many things to write about and that I can’t go into too much detail on each topic. I have chosen only a few topics that might be interesting. You can expect more blogs from me that go into more technical detail. So, stay tuned…
Traditional hardware requirements (Hyper-V Cluster):
This blog would have looked nice if I started thrown in a nice picture of our Cisco FlexPod solution. Be patient, we’ll get there soon. To better understand the benefits of Cisco UCS lets first look at how a Hyper-V Cluster would look like with traditional hardware and what challenges arise.
Let’s imagine you need to build a Hyper-V Cluster with three Hyper-V Servers. For network connectivity and redundancy you would need two Ethernet switches. Each servers would need one or more Ethernet Adapters (e.g. Management, Cluster Heartbeat and Live Migration) to connect to those switches. For VM (Virtual Machine) network connectivity you would probably need an additional Ethernet Adapter. And let’s say for storage you would need two Fiber Channel switches. To connect each server to those switches you would need Host Bus Adapters. If you want to give your VM’s direct storage connectivity you would probably need an additional Host Bus Adapter. And because everything needs to be connected redundantly you would need twice the amount of Ethernet and Fiber Channel ports. Soon your physical hardware would like this:
As you can see this results in a bunch of network adapters and cables. In some scenarios this can be quit complex and what most people don’t realize is cabling like this consumes a lot of power. In fact not every bandwidth is used efficiently, which makes the price per port costly. As you know, with Windows Server 2012 and later it is possible to team several network interfaces and configure converged networking on top of the team or even on top of a vSwitch. But to get enough bandwidth and redudancy you still need multiple interfaces. And depending on the NIC teaming mode in most cases it does not increase bandwidth for a single session as it only passes through one network interface. I will write a blog about teaming at a later time. Of course you can use 10Gb interfaces instead of 1Gb interfaces, but one way or another you still need to separate interfaces for each function.
Each physical server assembly is built upon a so called hardware profile. For standardization it is best-practice you keep that hardware profile the same on every physical server. So the choices you make can stick with you for a long time.
Traditional hardware limitations (scalability):
With hardware at datacenter infrastructures (as described above) there are two kind of scalability:
Scale-up:
“Scale-up” means you (can) add things like more memory, additional processors or even network adapters. Most often that is quite easy to do. But there are some limitations. Suppose you want to replace a motherboard with a newer processor architecture. Or depending on your hardware model, you might not have enough room (e.g. slots). In most cases that would result in total hardware replacement and re-installation of the Operating System, which is a pain.
Scale-out:
“Scale-out” means you (can) add more physical servers. But more servers means more cabling and before you know it you run out of switch ports. Of course you can add switches, but those switches need to interconnect with your existing switches. A network can grow quit complex and at some stage have bandwidth issues. And as a result you get more network devices to manage.
Now… this is almost history. There is an evolution going on at the datacenters. Cisco and other vendors are working hard to change the way we deploy compute and connect network resources. That’s why you read al these new terms like “Converged Infrastructure”, “Converged Networking”, “Unified Fabric”, “Unified Ports”, “Stateless Computing” and so on. And particularly Cisco has come a long way in the industry to make our job much easier.
Cisco FlexPod to the rescue:
So what is a Cisco FlexPod solution you might ask. A FlexPod is a datacenter reference architecture that contains Cisco and NetApp hardware. To be more specific:
- Cisco UCS (Unified Computing System) for compute.
- Cisco Nexus (and UCS) for network.
- NetApp FAS for storage.
Please look at www.cisco.com/go/ucs or www.cisco.com/go/flexpod for more information. Cisco has build this fully tested and validated datacenter architecture which offers high performance, high availability and is highly scalable. It contains all the ingredients you need, especially for those who want to build there own cloud infrastructure. Although a Cisco FlexPod can be combined with all kind of hardware it is based on a certain validated design. To give you an example, this is what a typical Cisco FlexPod solution may look like:
As you can see the example above it contains a Blade Chassis with Blade Servers. Cisco also has Cisco UCS C-Series Rack Servers. They even have E-Series Blade Servers that can be mounted into routers, firewalls and some other devices (which is a totally different story). At the time of writing Cisco offers three FlexPod solutions each with there own combination of hardware:
- FlexPod Datacenter (medium and large enterprises, for a highly scalable infrastructure)
- FlexPod Express (small and medium enterprises, optimal value for their infrastructure)
- FlexPod Select (focuses on high-capacity and performance for specialized workloads)
Cisco also has a solution with EMC storage called Cisco Vblock. Of course you can buy Cisco UCS without NetApp or EMC storage devices. You can even use or combine your own storage devices. Nevertheless, a Cisco FlexPod or Cisco Vblock is a good solution to start with if you are looking for a robust datacenter architecture with dozens of validated designs, white papers and deployment guides. And from my own experience; Cisco- and NetApp Support is very good! Now let’s have a look at some of the benefits of Cisco UCS and wat it does in terms of scalability.
Converged Infrastructure:
A Cisco FlexPod offers a Converged Infrastructure which serves as a platform for hosting a public-/private cloud service, including IaaS (Infrastructure as a Service), PaaS (Platform as a Service) and SaaS (Software as a Service). A number of features are ideally suited for deploying a Cloud service. This includes pooling of shared resources, automate and allocating resources. With a Converged Infrastructure everything is brought together in a single fabric.
Unified Fabric:
As mentioned; For Ethernet you need Ethernet adapters and Ethernet switches, for Fiber Channel you need Host Bust Adapters and Fiber Channel switches. As with most cloud infrastructures, a lot of I/O protocols are involved which normally requires a lot of components and cabling. Which results in complex network infrastructures and high energy consumption.
Cisco UCS is a platform where various I/O protocol are brought together in one ‘single-wire and management’, known as a Unified Fabric. No need for separate switches. UCS is a system that supports almost every protocol and transport carrier, like Ethernet, FC (Fiber Channel) and even FCoE (Fiber Channel over Ethernet). A Unified Fabric simplifies the network significantly by means of network virtualization and efficiently utilizes shared resources. It is perfect suitable for dynamic workloads and reduces energy consumption.
You might ask, what is so special about a Unified Fabric? Well, this is what makes it so much easier and in the same time highly scalable. For instance, you can connect almost any network device to Cisco UCS. Let’s say you have an uplink switch that connects to your corporate network, a storage device that uses iSCSI and another storage device that uses Fiber Channel. You can connect all of them easily. Then you can provision (and even virtualize) them as resources to your Cisco UCS servers. For example, you also configure the VLAN’s from your uplink switch in Cisco UCS. Those VLAN’s are then available as network resources. When a server administrator configures a server (known as a Service Profile) it can select those network resources without needing a network engineer. In the same time those network resources are optimally configured on the physical server and switches. Once you have used it, you love it!
Virtualization Support:
A Converged Infrastructure is in fact a new evolution of data center network infrastructures. It allows for both a technical and business efficiency. We already experience the benefits of server virtualization. Server virtualization cannot operate without other resources, such as network and storage. Cisco UCS focuses on all those aspects, what makes for the best flexibility in the field of hosting. In addition Cisco UCS has a tight integration with virtualization platforms like VMware and Hyper-V. Cisco UCS offers full support for a wide range of innovative features, such as VM-FEX (Virtual Machine Fabric Extender), SR-IOV (Single-Root I/O Virtualization), VMQ (Virtual Machine Queuing), vRSS (virtual Receive Side Scaling), Nexus 1000V and more.
Stateless Computing:
A key benefit of Cisco UCS is the concept of Stateless Computing. In fact, a physical UCS Blade Server has no configuration at all. It has no embedded UUID, no MAC Addresses, not even BIOS settings. A physical UCS server does contain a CNA (Converged Network Adapter), like a so called Cisco VIC (Virtual Interface Card). At first the server is kind of brainless, just a hardware resource that is available to you. Instead, Cisco UCS allows you to build your own server configuration. As if you are building a recipe with all the required ingredients. May your wish come true. A server configuration is put together in a so called Service Profile. A Service Profile contains all your ingredients, like settings, resources and polices. It also contains converged network interfaces called vNICs (virtual Network Interface Cards) and vHBAs (virtual Host Bust Adapters), in a desired PCI order. Each vNIC will have it’s own MAC Address (from a resource pool you created manually), optimal settings and etc. With a Cisco VIC within a physical UCS server, it is even possible to create a Service Profile with 256 vNICs/vHBAs. A Service Profile can then be applied to a Blade- or Rack Server. The server will boot with exactly that configuration. The vNICs and vHBAs are presented to the Operating System as Gen* PCI-E Network Adapters.
Cisco UCS allows you to create a Service Profile Template (for a typical server role) and clone new Service Profiles. This ensures consistency across all servers. For example, you can build a Service Profile Template with all the ingredients for a VMware Server or Hyper-V Server. That Service Profile Template contains all the vNICs (and vHBAs) you need, each with the right VLANs (and VSANs) assigned, all the required BIOS settings, including several policies.
When you need to replace or upgrade a UCS server, you can simply ‘disassociate’ a Service Profile from the original server and ‘associate’ it to another server. The exact configuration with all the ingredients move with it. Almost similar like you would move a VM from one host to another host. You literally virtualize your server configuration like you would with a VM. Of course you need to boot from SAN to make it work. Cisco UCS provides full and robust support for boot from SAN, for both iSCSI and Fiber Channel. Each cloned server has exactly the same configuration, but each with a unique UUID, MAC Addresses and etc. You can upgrade servers easily, you can replace servers easily, and you can add servers easily. All can be done within minutes! Stateless Computing adds flexibility in terms of Scale-Up and Scale-Out.
Personally I am quit impressed about all this. It makes deployment, scale-out and such so much easier. No need to switch fabrics and keep track of complex cabling. Adding (I must actually say creating) a new servers, giving it vNICs (and vHBAs) interfaces and connecting them is so easy!
Fabric Failover:
A FlexPod is fully redundant. With traditional physical servers it is often common to use NIC teaming. NIC teaming requires the necessary configuration within the Operating System. With Cisco UCS there is no need for NIC teaming. Cisco UCS offers Fabric Failover.
Fabric Failover ensures that if a physical link or whole network component (from Fabric A or B) fails, it will do an automatic failover to another network component (from Fabric B or A). This is an active/passive configuration. The Operating System running on a Cisco UCS server only detects a single converged network interface. Fabric Failover offers hardware based redundancy, which simplifies your configuration and deployment. Fabric Failover is optional. If Fabric Failover is used you need to choose whether the vNIC is actively connected to Fabric A or B. Of course you still have the option to add two vNICs to a Service Profile and configure NIC teaming, although it is not recommended. And if you need to have iSCSI connectivity with MPIO (Multi-Path I/O). You simply create two vNICs, the first vNIC connected to Fabric A and the second vNIC connected to Fabric B. Same thing for vHBAs with MPIO.
Scale-up and Scale-out:
Cisco UCS offers enormous flexibility in terms of Scale-Up (upgrade) and Scale-Out (expand). Among other things, thanks to a Unified Fabric and Stateless Computing it is very easy to add new Blade Chassis and Rack Servers, upgrade or replace them. You can connect multiple network devices to Cisco UCS, like uplink switches, directly attached storage devices. Especially for a Public Cloud service, where one has a lot to do with a dynamic workload, it offers many benefits. You can connect as many network devices to Cisco UCS as long as you don’t run out of ports. You can connect a maximum of 20 Blade Chassis or 160 servers to Cisco UCS. You can connect both Blade Chassis and Rack Servers.
Firmware Management
Cisco UCS has a simplified and automatic firmware management system. If configured; when you connect a new Blade Chassis, all components (like Fabric Extenders and Blade Servers) will be automatically updated for you. Same thing when you insert a new Blade Server in an existing Blade Chassis, all server components will be automatically updated. And with updated I mean every single component! Including motherboards, CNAs, RAID controllers, Video Adapters, KVM consoles, disks, Fabric Extenders, I/O Modules and etc. When a new firmware-/software package is released; you can import it and Cisco UCS allows you to automatically update all your servers automatically! This can save you a lot of time and ensures consistency across all your servers.
UCS Manager:
The entire UCS system is managed by a system called UCS Manager. Everything in terms of compute and network is configured and managed through UCS Manager. UCS Manager provides a single-point of management for network- and server administrators.
UCS Manager is accessible through a web console. It is also possible to use PowerShell scripts against UCS Manager. UCS Manager includes all the KVM consoles. If you want, you can monitor your UCS environment with SCOM (System Center Configuration Manager). SCOM only has to connect with UCS Manager. SCOM will then collect the information from all your UCS devices.
Ok, for now I think this is enough information about Cisco UCS. It’s time to show you my first Cisco FlexPod configuration. Continue reading by clicking on the link below.
