Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Monthly Archives: October 2009

iSCSI LUN is very slow/no longer visible from vSphere host

 

I encountered this situation in my home lab recently – to be honest I’m not exactly sure of the cause yet, but I think it was because of some excessive I/O from the large number of virtualized vSphere hosts and FT instances I have been using mixed with some scheduled storage vMotion – over the weekend all of my virtual machines seem to have died and crashed or become unresponsive.

Firstly, to be clear this is a lab setup; using a cheap/home PC type SATA disk and equipment not your typical production cluster so it’s already working pretty hard (and doing quite well, most of the time too)

The hosts could ping the Openfiler via he vmkernel interface using vmkping so I knew there wasn’t an IP/VLAN problem but access to the LUNs was very slow, or intermittent – directory listings would be very slow, time out and eventually became non-responsive.

I couldn’t power off or restart VMs via the VI client, and starting them was very slow/unresponsive and eventually failed, I tried rebooting the vSphere 4 hosts, as well as the OpenFiler PC that runs the storage but that didn’t resolve the problem either.

At some point during this troubleshooting the 1TB iSCSI LUN I store my VMs on disappeared totally from the vSphere hosts and no amount of rescanning HBA’s would bring it back.

The Path/LUN was visible down the iSCSI HBA but from the storage tab of the VI client

Visible down the iSCSI path..

image

But the VMFS volume it contains is missing from the list of data stores

image

This is a command line representation of the same thing from the /vmfs/devices/disks directory.

image

OpenFiler and it’s LVM tools didn’t seem to report any disk/iSCSI problems and my thoughts turned to some kind of logical VMFS corruption, which reminded me of that long standing but never completed task to install some kind of VMFS backup utility!

At this point I powered down all of the ESX hosts, except one to eliminate any complications and set about researching VMFS repair/recovery tools.

I checked the VMKernel log file (/var/log/vmkernel) and found the following

[root@ml110-2 /]# tail /var/log/vmkernel

Oct 26 17:31:56 ml110-2 vmkernel: 0:00:06:48.323 cpu0:4096)VMNIX: VmkDev: 2249: Added SCSI device vml0:3:0 (t10.F405E46494C454009653D4361323D294E41744D217146765)

Oct 26 17:31:57 ml110-2 vmkernel: 0:00:06:49.244 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x410004168500) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

Oct 26 17:31:57 ml110-2 vmkernel: 0:00:06:49.244 cpu1:4097)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

Oct 26 17:32:00 ml110-2 vmkernel: 0:00:06:51.750 cpu0:4103)ScsiCore: 1179: Sync CR at 64

Oct 26 17:32:01 ml110-2 vmkernel: 0:00:06:52.702 cpu0:4103)ScsiCore: 1179: Sync CR at 48

Oct 26 17:32:02 ml110-2 vmkernel: 0:00:06:53.702 cpu0:4103)ScsiCore: 1179: Sync CR at 32

Oct 26 17:32:03 ml110-2 vmkernel: 0:00:06:54.690 cpu0:4103)ScsiCore: 1179: Sync CR at 16

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.700 cpu0:4103)WARNING: ScsiDeviceIO: 1374: I/O failed due to too many reservation conflicts. t10.F405E46494C454009653D4361323D294E41744D217146765 (920 0 3)

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.700 cpu0:4103)ScsiDeviceIO: 2348: Could not execute READ CAPACITY for Device "t10.F405E46494C454009653D4361323D294E41744D217146765" from Plugin "NMP" due to SCSI reservation. Using default values.

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.881 cpu1:4103)FSS: 3647: No FS driver claimed device ‘4a531c32-1d468864-4515-0019bbcbc9ac’: Not supported

Due to too many SCSI reservation conflicts, so hopefully it wasn’t looking like corruption but a locked-out disk – a quick Google turned up this KB article – which reminded me that SATA disks can only do so much 🙂

Multiple reboots of hosts and the OpenFiler hadn’t cleared this situation – so I had to use vmkfstools to reset the locks and get my LUN back, these are the steps I took..

You need to find the disk ID to pass to the vmkfstools –L targetreset command, to do this from the command line look under /vmfs/devices/disks (top screenshot below)

You should be able to identify which one you want by matching up the disk identifier.

image

Then pass this identifier to the vmkfstools command as follows (your own disk identifier will be different) – hint: use cut & paste or tab-completion to put the disk identifier in.

vmkfstools-L targetreset  /vmfs/devices/disks/t10.F405E46494C4540096(…)

You will then need to rescan the relevant HBA using the esxcfg-rescan command (in this instance the LUN is presented down the iSCSI HBA – which is vmhba34 in vSphere)

esxcfg-rescan vmhba34

(you can also do this part via the vSphere client)

if you now look under /vmfs/volumes the VMFS volume should be back online, or do a refresh in the vSphere client storage pane.

All was now resolved and virtual machines started to change from (inaccessible) in the VM inventory back to the correct VM names.

One other complication was that my DC, DNS, SQL and vCenter server are all VMs on this platform and residing on that same LUN. So you can imagine the havoc that causes when none of them can run because the storage has disappeared; in this case it’s worth remembering that you can point the vSphere client directly at an ESX node, not just vCenter and start/stop VMs from there – to do this just put the hostname or IP address when you logon rather than the vCenter address (and remember the root password for your boxes!) – if you had DRS enabled it does mean you’ll have to go hunting for where the VM was running when it died.

image

In conclusion I guess there was gradual degradation of access as all the hosts fought with a single SATA disk and increased I/O traffic until the point all my troubleshooting/restarting of VMs overwhelmed what it could do. I might need to reconsider how many VMs I run from a single SATA disk as I’m probably pushing it too far – remember kids this is a lab/home setup; not production, so I can get away with it 🙂

In my case it was an inconvenience that it took the volume offline and prevented further access, I can only assume this mechanism is in-place to prevent disk activity being dropped/lost which would result in corruption of the VMFS or individual VMs.

With the mention of I/O DRS in upcoming versions of vSphere that could be an interesting way of pre-emotively avoiding this situation if it does automated storage vMotion to less busy LUNs rather than just vMotion between hosts on the basis of IOPs.

Installing VMware Workstation on Windows 7

You may recall I previously posted on problems installing VMware Workstation 6.5 on Windows 7, this problem seems to have been resolved with the upcoming VMware Workstation 7 which adds support for Windows 7 as a guest and as a host.

You can download the Workstation 7 RC build here Release Build here and see the full features list, I can confirm it installed perfectly on my Windows 7 Ultimate x64 machine.

image (Screenshot from RC build, see above link for RTM build)

Some new features include official support for Windows 7 *with Aero support!* (shown below)

image

And best of all – it now provides "official support for ESX as a guest VM under Workstation (my previous posts on workarounds for Workstation 6.5  here)

image

As an aside I’m running Windows 7 on a machine with a 64-bit SSD hard drive, I’m hoping to make use of the linked clone functionality to save disk space as I often run VM’s which are built from a common base OS template (see this post here for more info on how I’ve managed linked images in the past)– performance so far has been great both for host and guest as I/O doesn’t get as bogged down as it does with traditional spindle based disks.

**UPDATE: ah, the perils of the scheduled post – as this article went live the final RTM build of Workstation 7 has been released, I’ve updated the links in this post**

Getting access to VMworld content if you couldn’t make it in person

 

Now the noise around VMworld has calmed down I thought I would let you know that the vast amount of excellent technical content that was presented at the event itself is available to stream online or as an MP3 (audio only) or slide download (audio only).

As you’d expect, the catch is it’s not free to you unless you attended VMworld in-person. However, you can purchase a VMworld subscription which costs $699 USD per annum and gives you full access to stream and download content from the event, and all previous events back to 2004 – so if travel and time out of the office is not an option for you – how about you (or your employer) pay for a subscription to the content itself – which is obviously cheaper than attending in person.

image 

I have pasted a full list of all sessions from VMworld 2009 US below – please don’t ask me to post the sessions online, this is explicitly forbidden as you’d expect – if you want the content I’m afraid you’ll have to pay – click the graphic below (but it’s excellent value IMHO

image

Note: you’ll need a vmworld.com account to view the session details linked below (it’s free and can be done here)

Super Sessions

SS4880

NetApp: Clear up the Cloud – Key Infrastructure Requirements and Real-World Implementations

SS5000

Dell: How to Get Ahead in the Cloud With Your Feet Planted Firmly on the Ground

SS5001

VMware: Extending Your IT Beyond the Datacenter: The vCloud Initiative

SS5081

Wyse: Desktop Virtualization / Cloud Computing: We Did It – Here’s How and What we Learned

SS5082

Cisco and VMware: Delivering Innovation for Virtualization

SS5120

IBM: What You Need to Know to Virtualize Today’s Data Center

SS5121

Intel: Technology transformations central to the evolution of flexible computing

SS5140

EMC: Infrastructure Architectures Purpose Built for the Virtual Datacenter

SS5160

HP: Stop Virtualizing Servers, Start Virtualizing Infrastructure

SS5220

Symantec: Complete the Promise of Virtualization

SS5240

VMware, Cisco and EMC: Engineering Developments Enabling the Virtual Datacenter

SS5241

VMware: Question and Answer Session with Paul

SS5440

VMware: Enabling Better Business Outcomes with Policy-Driven Service Level Management

Business Continuity & Disaster Recovery

BC1500

vCenter Site Recovery Manager "Up and Running" – Best Practices & Avoiding the Pitfalls

BC2082

VMware vCenter Site Recovery Manager Performance and Best Practice (online only)*

BC2142

Data Recovery – Install / Configure and troubleshooting

BC2253

Pain-free VMware Agentless Backup AND Recovery – VCB & Beyond (online only)*

BC2260

Automated Disaster Recovery for Branch Offices using SRM and vSphere 4

BC2412

Managing SAP Virtualization, DR and Upgrades with EMC Ionix Technology (online only)*

BC2425

Using CDP for Cost-Effective Application Recovery in vSphere Environments

BC2520

911 Call Center Leverages VMware To Ensure Survivability

BC2537

VMware Site Recovery Manager with Enterprise Replication – A Customer Testimonial

BC2541

Re-architecting Backup and Recovery for Virtual Environments: Best Practices

BC2565

An Optimized Approach to Workload Availability in Virtualized Environments

BC2704

Site Recovery Manager, a real user experience

BC2760

Protecting the corporate infrastructure: Leveraging Site Recovery Manager for multi-site DR protection.

BC2961

VMware Fault Tolerance Architecture and Performance

BC3042

VMware SRM Enhanced Testing (online only)*

BC3083

Mission Critical: Virtualization and Robust DR Architectures for Vital Systems (online only)*

BC3086

A practical guide for recovering key applications using VMware Site Recovery Manager

BC3189

How Storage Enhances Business Continuance: From Backup to Fault Tolerance

BC3197

High Availability – Internals and Best Practices

BC3209

Creating the Fastest Possible Backups Using VMware Consolidated Backup -A Design Blueprint

BC3210

Preview of Best Practices for Recovery with Site Recovery Manager and NFS

BC3223

Planning for Optimized and Cost Effective Storage Utilizing Deduplication and Virtualization Technologies

BC3301

DR Architecture Design Workshop with SRM

BC3369

VMware Fault Tolerance Real-World Use Cases

BC3370

VMware Fault Tolerance – Overview and Best Practices

BC3384

Using IP based replication as an Enabler for Server Virtualization and Storage Repurposing

BC3386

DR Solutions and Services with Sungard and VMware

BC3387

Advanced Data Protection for Microsoft SQL and Exchange Server in a VMware Environment

BC3396

How VMware uses Site Recovery Manager for its own disaster recovery

BC3421

SRM Architecture & Features: The Road Ahead

BC3425

VMware Availability Solutions and Futures

BC3602

VMware Fault Tolerance – vSphere Workflows and API Considerations (online only)*

BC3780

How Customers are Using VMware Site Recovery Manager for Automated Disaster Recovery

BC4440

Increase consolidation ratios in vSphere environments while simplifying and improving backup and recovery

BC4580

Why you should be using Virtual Backup Appliances for Business Continuity & Disaster Recovery?

BC4840

Exploring vStorage API’s and how Veeam and Dell partner for Success

BC5380

How to bring V-Cloud & Cloud Computing to Backup & Recovery

Business Workshops

BW4740

vSphere – Evangelizing the Value Proposition

BW4741

VMware View – Evangelizing the Value Proposition

BW4742

Realizing the Return on Investment In VMware

BW4743

Planning Your Virtualization Journey

Desktop Virtualization

DV1392

Lead Practices for Integrating VMware View with Active Directory

DV1406

Integrating VMware View with your VPN (online only)*

DV1460

Leveraging Vmware View and Cisco WAAS in a ROBO deployment

DV1666

VMware View Reference Architecture

DV1667

Norton Healthcare Desktop

DV1788

The 4 C’s of Desktop Virtualization for Healthcare: Costs, Clients, Continuity, and Compliance

DV1790

Understanding TCO & ROI for VMware View (next gen VDI)

DV1961

VMware Workstation and Player Technology Preview

DV2000

VMware View Client Virtualization Platform and its Use Cases

DV2181

Leveraging SRM with VMware View – Lessons Learned

DV2223

Printing Considerations in a View Architecture

DV2243

The View Test – Running a POC

DV2363

Next Generation Client Virtualization – Technology Deep Dive

DV2380

Getting the Most from View Composer – Tips, Tricks, and Best Practices

DV2387

Scripting within VMware ThinApp

DV2439

Breaking Down Desktop Virtualization Alternatives

DV2461

Virtualization on mobile phones? Why do I need that?

DV2478

Application Troubleshooting in VMware ThinApp

DV2484

Server and Storage Sizing for VMware View

DV2485

Choosing the Right Virtual Desktop Infrastructure Platform

DV2507

VMware View – Upcoming Attractions

DV2546

Case Study: Making the Grade with VDI and K12 Education

DV2626

VMware View Security Architecture

DV2672

Cerner Millenium deployed in a VMware View environment

DV2697

VMware View – Remote Display Experiance

DV2705

ThinApp Packaging & Deployment

DV2714

Optimizing VDI Storage with VMware View: Strategies for Success

DV2734

VMware ThinApp â?? Integration with VMware View (online only)*

DV2782

Application and Desktop Virtualization

DV2785

ThinApp Reference Architecture

DV2787

VMware ThinApp – A Review of Best Practices and Design Considerations (online only)*

DV2801

Integrating VMware View into your environment

DV2861

The Journey to a Dependable VDI Solution

DV2941

Operationalizing Desktop Virtualization

DV2943

VDI to the edge, deploying virtual desktops to remote and branch offices

DV2981

What’s new with ThinApp

DV3081

From Strategy to Reality – VDI lessons learned at Embarq

DV3200

Creating a VMware View Design for the Enterprise

DV3243

ViewPlanner: An Automated Measurement Infrastructure for Large-Scale VDI Deployments

DV3258

From NT4.0 to Virtualized Desktops and Applications

DV3260

Comprehensive Performance Analysis of Remote Display Protocols in VDI Environments

DV3266

Efficient VDI Design – The Math behind VDI

DV3292

The Effective use of VMware ThinApp in Higher Education

DV3300

Switching to the Mac with VMware Fusion: Lessons Learned & Deployment Tips

DV3458

Better Software Development and Testing using VMware Workstation

DV3498

VMware View Performance and Scalability on Cisco UCS

DV3532

Virtual Desktop Security

DV3567

Don’t throw that PC! How to convert old PCs to Thin Clients using a thin Linux OS and VMware View

DV3660

VMware Fusion – Present and Future

DV3746

How to Optimize VMware View ROI with Thin Computing

DV4243

How to make BYOPC (Bring Your Own PC) a Reality?

DV4381

An Insider’s View of Mobile Phone Virtualization

DV4521

From Servers to Desktops Virtualization in a "Next Gen Data Center" for Telecommunications Companies

DV5080

Desktop Virtualization / Cloud Computing: Using the Vendor Ecosystem to Drive Maximum Success

Enterprise Applications

EA1396

SQL, Exchange and SharePoint Deployments on EMC Celerra Unified Storage (online only)*

EA1480

Best Practices for Java and J2EE Applications on VMware ESX

EA1640

SAP Scaling with VMware vSphere and IBM x3850 M2

EA1820

Virtualizing Critical Healthcare Applications

EA1941

Accelerate the Enterprise Java Application Lifecycle with SpringSource

EA2342

Oracle Database Virtualization Strategies

EA2436

Efficient and High Performance Virtualization of Oracle Database Environments using vSphere

EA2442

Software Licensing in the Virtual Enterprise: Current Problems and Future Trends

EA2526

SharePoint vSphere Best Practices

EA2583

HPC/Grid Computing and Virtualization

EA2631

Virtualizing Exchange 2007 on vSphere 4 – Technical Considerations and Customer Success Story

EA2649

Virtualizing IBM Lotus Domino and Sametime: Planning to Successful Deployment at Whirlpool

EA2652

IBM DB2 on VMware: The Most Cost-Efficient and Dynamic Database Platform

EA3104

Enterprise Application Performance and Scalability on vSphere

EA3196

Virtualizing Blackberry Enterprise Servers (BES) on VMware vSphere 4

EA3216

Best Practice for Virtualizing Active Directory using vSphere (online only)*

EA3234

Virtualizing SQL Server in a VMware vSphere environment

EA3241

Beyond Infrastructure as a Service: Developer and Runtime Services with VMware and our Partners

EA3347

Analyzing Application Stack Performance using AppSpeed

EA3376

Virtualizing Citrix XenApp using vSphere

EA3389

US Navy Marine Corps virtualizes 700,000+ Microsoft Exchange users to Achieve Mission Critical Reliability

EA3408

MySQL on VMware: Performance and Deployment Best Practices

EA3448

SAP on VMware: Advanced Networking for High Performance

EA3481

Virtualization of Analytic Databases

EA3582

Developing and Deploying Virtualized Real Time Communications within the Enterprise

EA3579

Oracle Enterprise Workloads on VMware How-To (online only)*

EA3605

Virtualizing Tier 1 Applications: The Value of the vSphere Internal Cloud as a Better Platform for Apps

EA3606

Database Consolidation on vSphere (SQL and Oracle)

EA3820

The Spheres – Using WebSphere with vSphere

EA3940

Cerner Millennium Scalability when deployed on VMware vSphere and Nehalem

EA4403

VMware IT: Exchange 2007 at VMware

EA4404

Oracle E-Business Suite: Virtually Powering VMware

EA4760

Building Cloud Ready Applications

Technology and Architecture

TA1394

vSphere 4 Advanced Storage Log Analysis

TA1440

Networking in ESX: VM DirectPath Dynamic – the road to Direct VM to Hardware (online only)*

TA1523

The Path to vSphere Unleashed

TA1541

Cool little things marketing did not tell you about vSphere 4.0

TA1670

Overview of VMware vSphere

TA1962

How and Why we Upgraded Herning Kommune’s Production Environment to vSphere 4.0 at GA

TA2021

Linux Virtual Machines: Templates, P2V and Other Tips and Tricks

TA2103

Virtual Networking with vSphere 4.0 – What’s New

TA2105

Virtual Networking Concepts and Best Practices

TA2222

ROBO – ESX implementation at Kroger Store Systems

TA2231

Securing Virtual Environments – Before and After VMsafe

TA2254

vStorage – Storage Integration for Cloud OS

TA2259

Ask the Experts – Virtualization Design

TA2262

vSphere Enterprise Stability – It’s all in the Design

TA2266

Designing the Next Generation Data Center with Unified Computing & VN-Link

TA2381

Warehouse In A Box

TA2384

Deploying Cisco Nexus 1000V in a VMware vSphere Environment

TA2400

Hypervisor Competitive Differences: What the Vendors Aren’t Telling You

TA2405

Nielsen Company Leverages VMsafe Virtual Firewalls from Altor Networks for Unprecedented Security

TA2467

Best Practices to Increase Availability and Throughput for the Future of VMware

TA2509

Storage Best Practices for Scaling Virtualization Deployments

TA2525

VMware vSphere 4 Networking Deep Dive

TA2543

VMware’s Secure Software Development Lifecycle

TA2544

A Comprehensive Look at the Security and Compliance of vSphere 4

TA2623

Enhanced Storage VMotion in vSphere 4

TA2627

Understanding "Host" and "Guest" Memory Usage and Other Memory Management Concepts

TA2646

Creative Solutions: How Florida Hospital virtualized AIX and Mastered SAN Replication for DR

TA2650

Take PowerCLI to the Next Level

TA2682

Achieving 10+ Gbps File Transfer Throughput Using Virtualization – End-User Case Study

TA2689

Controlling the Storage Impact of Virtual Server Sprawl

TA2699

Measuring Virtualization Platforms and Trends Using VMmark (online only)*

TA2713

Safe At Any Speed with VMware DRS & DPM

TA2731

Tips for Planning and Upgrading to vSphere 4

TA2732

Enterprise Approach to Deploying vSphere

TA2942

Performance Best Practices

TA2955

Sizing Tools for Storage in vSphere and VMware View Environments

TA2963

ESXtop for Advanced Users

TA3045

Implementing VMware vSphere 4.0 with HP Virtual Connect Flex-10 technology

TA3105

Long Distance VMotion

TA3195

Stateless ESXi: Scalable Rollout and Management of Virtualized Hosts

TA3212

Virtualizing Resource Intensive Applications, a Case Study with SAP

TA3220

VMware vStorage VMFS-3: Architectural Advances since ESX 3.0

TA3261

Using VMware to Conduct Digital Forensics and Criminal Investigations (online only)*

TA3264

iSCSI Scalability and Storage Performance Enhancements in vSphere 4.0

TA3286

Applications in the Cloud: Getting off the ground

TA3302

Security Considerations When Building Virtual Infrastructures Across Security Zones

TA3324

Performance Troubleshooting in Virtual Infrastructure

TA3326

Building an Internal Cloud-the Journey and the Details

TA3353

Building the TVE a Collaboration Story

TA3402

How Dell Uses Virtualization: Paving the Path to Large-scale Distributed Computing

TA3406

What is New for Storage in vSphere 4.0

TA3438

Top 10 Performance Features of VMware vSphere 4

TA3461

Tech Preview: IO DRS – Providing Performance Isolation to VMs in Shared Storage Environments

TA3488

All Hypervisors Are Not Created Equal – The Unique Advantages of VMware ESX (online only)*

TA3521

vNetwork Monitoring, Troubleshooting, Security, and Management (with Live Demo)

TA3555

Getting to over 90% Virtualization

TA3557

Using NFS Storage Infrastructure for vSphere

TA3576

Early vSphere Deployment Stories

TA3603

Getting The Most Out Of VMotion: EVC, Performance Tuning, and Troubleshooting

TA3880

Head-To-Head Comparison: VMware vSphere and ESX vs. Hyper-V and XenServer

TA3882

The Cloud- What is it and why should I care

TA3901

Security and the Cloud

TA4020

Beyond the Data Center – Virtualizing your remote and distributed IT workloads

TA4060

The Path to COS-less ESX: Migrating Server Operations from ESX to ESXi

TA4100

Internal Clouds: Customer perspective and implementations

TA4101

Buying the Cloud: Customer perspective and considerations on what to send to an external cloud

TA4102

Unveiling New Cloud Technologies

TA4103

Engineering the Cloud-The Future of Cloud

TA4160

What is new in the vSphere SDKs and APIs

TA4242

Securing the Virtual Data Center in Enterprises and Clouds

TA4341

Virtual Network Performance

TA4720

Virtualization and Cloud Computing with AMD Opteron™ Processor-based Platforms

TA4820

What Keeps Clouds Up?

TA4860

Deploying the Foundation for a Virtualized Dynamic Data Center

TA4881

Designing Dynamic Data Centers with NetApp and VMware

TA4901

IBM System x Delivers Innovation for Virtualization Solutions for Today and Tomorrow

TA4902

IBM’s Cloud Computing Solutions

TA4940

Navigating the Cloud: IT Management Challenges and Opportunities

TA4941

I/O Clouds in the Virtualized Data Center

TA4960

Virtualizing Email Security â?? The St Lawrence College Story (online only)*

TA5161

How Expedia uses Storage VMotion to Allow Datacenter to Take Flight

TA5360

Enabling the Virtual Data Center with Symmetrix V-Max

TA5400

The Soft Underbelly of Bare Metal – Real World Security Lessons from the Datacenter to the Cloud

TA5460

Optimize Your Storage: A Blueprint for Storage Resource Management in VMware Environments

TA5461

Securing the Cloud

TA5540

The Evolution of Open Source Virtualization

TA5600

Virtualization: How Cisco IT Leverages Operational Excellence to Drive Innovation

Virtualization 101

V11721

Best Practices for Successful VI Design

V12226

Building a High Availability and Disaster Recovery Solution with VMware

V12644

Designing a Virtualization Infrastructure for the Small Environment

V12789

VMware vCenter Converter 101 (online only)*

V13100

The VMware Competitive Advantage – A Comparison of Server Virtualization Offerings

V13226

Getting Started with Virtualization Using VMware ESXi

V13227

Introduction to Virtualization

V13229

Introduction to VMware vSphere

V13395

Getting to Yes! Keys to Launching a Successful Data Center Virtualization Program

V13478

Executing Enterprise Virtualization – Continuing Case Study with USMC

V13496

VMware vSphere and VI Best Practices – Tips and Tricks (online only)*

V13760

Overcoming the Hidden Challenges of Virtualization

Virtualization Management

VM1401

Using vCenter Lab Manager and View for the Education Cloud

VM1461

Avoiding the hurdles in scaling Lab Manager

VM1682

vCenter Database Architecture: Technical Deep Dive

VM1700

vApps and advanced VM templates in vSphere 4

VM1724

Technical Preview: CapacityIQ

VM1882

VMware Update Manager 4 Performance and Best Practices

VM1900

Accelerating Large Scale Migration – Best Practices, Tools, and Processes to Address Challenges

VM1960

Datacenter Consolidation and Migration with VMware Site Recovery Manager

VM2100

vSphere Orchestration for SAP’s Demo Business

VM2120

Assessment methodology for LCM customization with Orchestrator

VM2164

Virtualization and Compliance: The Auditor’s Perspective

VM2166

VMware’s New Licensing Model & Compliance Best Practices

VM2201

Turfwars! : The Future Staffing of Virtualized Environments

VM2241

Managing vSphere with VMware PowerCLI

VM2265

Achieving Measurable Advantages Through Better Virtual Systems Management

VM2280

vCenter Mobile Access – Managing Virtual Infrastructure From Your Phone

VM2408

Tech Preview: VMware vCenter ConfigControl

VM2409

Virtual Center: Troubleshooting Unleashed

VM2440

Deploying vSphere Mgmt Automation: Dependency Mapping, Root Cause, Configuration & Compliance

VM2472

Introduction to VMware vCenter Chargeback

VM2506

Automating the Virtual Data Center

VM2511

Automate Routine Actions and Enforce process in Virtual Environments (online only)*

VM2630

Using Unnoc to Perform Monitoring of vCenter, ESX Servers, and Virtual Guests

VM2643

Infrastructure Performance Panel

VM2648

Managing Compliance in Virtual Environments

VM2657

Best Practices for deploying VMware vCenter Lifecycle Manager.

VM2674

VMware vCenter Server Heartbeat Best Practices

VM2684

What’s New in vCenter Server 4

VM2700

Lab Manager Best Practices

VM2706

Improve Cloud Interoperability Using Virtualization Management Standards

VM2711

VI Performance Optimization: From the Datacenter to the Desktop (online only)*

VM2717

What’s New In Lab Manager 4?

VM2724

How to Prevent Headaches in your Virtual Environment

VM2764

From Virtualization, Automation to From Virtualization, Automation to Cloud Managemen

VM2766

Integrated and Application Centric Deployment Approach to Achieve Wider Virtualization Adoption

VM2847

Upgrading to vSphere – Things to consider

VM3021

VMware’s Tooling of VI Operations Best Practices

VM3041

Integrating Virtualization with Capacity Management

VM3085

Top 10 Reasons vCenter is the Best Platform for Virtualization Management

VM3103

How VMware Reduces Cost-per-Application and OpEx Costs

VM3120

How does VMware fit in with your management processes? VMware Integration with Big 4 + Microsoft

VM3142

Large Scale Virtualization

VM3217

vCenter Databases: Setup, Management and Best Practice (online only)*

VM3235

Introducing VMware vCenter Product Family: Managing Service Levels Across Dynamic IT Infrastructure

VM3237

VC Linked Mode in vSphere 4.0 (online only)*

VM3263

Sustainable Change Management Processes in a Virtual World

VM3325

vSphere and ESXi Log Files 101 & 102 (online only)*

VM3352

Operational Savings Achieved through Virtualization

VM3360

Assuring Service Levels for P2V migrations with AppSpeed

VM3404

Alarms for vCenter 4.0 (online only)*

VM3405

A Spotlight on vSphere Storage Management: Today and Tomorrow

VM3414

Study reveals production best practices for virtualization (online only)*

VM3427

VMware Administration for The Average Administrator

VM3433

Technical Deep Dive: VMware Host Profiles

VM3463

Monitoring Hardware Health with vCenter 4

VM3470

Seven Steps To Understanding And Mitigating Virtualization Security Risks

VM3483

Managing Application Performance with vCenter AppSpeed

VM3528

Does Virtualization Change The Way We Secure IT Environments?

VM3566

Best Practices for Managing and Monitoring Storage in vSphere

VM3580

Capacity Planning and RoI – Before, During & After the Project

VM3607

Before Chargeback, I Need To Know What It Costs

VM3609

Assessment Best Practices for VMware Partners

VM3620

Incident Handling in a Virtualized Data Center

VM3881

Business Objects SAP Virtual Infrastructure Lab Mgr Deployment and Migrating Between Network Ranges

VM3903

Practical Virtual Systems Management – Climbing The Hierarchy of Needs

VM4000

Operationalizing Software Update Management in a Growing Virtual Environment

VM4061

How to Generate Reports out of the vCenter Database

VM4120

Introduction to VMware vCenter Lifecycle Manager

VM4261

Ensuring Database Performance using VMware AppSpeed

VM4320

Vision for virtualization-aware IT management

VM4380

Tech Preview: vCenter Server

VM4520

Analyze to Automate – The Evolution of Vizioncore and How We Truly Extend VMware’s vSphere

VM4800

The "Next Generation Data Center" for Telecommunication Companies

VM4900

Building a Service-Driven Data Center: Making an Internal Cloud A Reality

VM5002

How to Gain Sponsorship and Successfully Enable Virtualization Transformation with VMware

VM5040

Amplify the Economic Benefits of Virtualization

VM5361

An Introduction to VMware vCenter Orchestrator

VM5401

Conquering Costs and Complexity in a Virtualized Environment: Research and Case Studies

VM5420

Using Lab Manager in a Regulated Healthcare Environment

VM5521

Unified Computing System – Infrastructure Management and Provisioning

VM5560

Data Center Optimization through Virtualization: Compute More, Consume Less

Instructor-Led Labs (PDFs only)

LAB01

VMware vSphere 4 – New Features, Best of, Advanced Topics

LAB02

VMware vCenter Site Recovery Manager – Advanced Operations

LAB03

VMware View Advanced Config & Troubleshooting

LAB04

VMware vSphere 4 – Performance Optimization & Troubleshooting

LAB05

VMware vSphere 4 – Security Hardening & Best Practices (vShield Zones)

LAB06

VMware vCenter Server Heartbeat

LAB07

VMware vCenter Lab Manager 4

LAB08

VMware vCenter CapacityIQ

LAB09

VMware vCenter AppSpeed

LAB10

VMware vCenter Lifecycle Manager

LAB11

VMware vCenter Chargeback

LAB12

Scripting VMware Infrastructure (PowerShell/Perl Toolkits)

Self-Paced Labs (PDFs only)

SPL13

VMware® Infrastructure 3 to VMware vSphere™ 4 Upgrade

SPL14

VMware vSphere™ 4 Base Install

SPL15

VMware View Base Install & Config

SPL16

VMware vCenter Site Recovery Manager (SRM) Basic Install & Config

SPL17

VMware vCenter Converter

SPL18

VMware vCenter Data Recovery

SPL19

VMware® ThinApp™

SPL20

VMware® Fusion™

SPL21

VMware View Experience

SPL22

VMware vCenter Orchestrator

SPL23

VMware vSphere™ 4 Virtual Networking Fundamentals

I wasn’t able to to go VMworld US in person this year because my wife and I were expecting a baby at that time, but luckily I received a VMworld subscription from VMware as a benefit of the vExpert programme; had I not I would have probably shelled out of my own pocket for one.

Justifying the spend

Whilst we seem to be slowly emerging from the economic apocalypse of the last 18 months it’s still very hard to get sign-off to attend such events in person and too many org’s treat VMworld/Tech-Ed as marketing type events – unless you are a vendor with a stand this couldn’t be further from the truth – these conferences are primarily technical training boot-camp camps, with some networking and general trade show features thrown in. However, they are what you make of them – the onus is on you to hunt down the sessions/track or people you are interested in – nobody drives your schedule but you – non self-starters need not apply.

I use the following analogy – which applies equally to Microsoft Tech-Ed and VMworld (..and I’m sure Oracle World, Apple World, etc.)

A typical 5-day technical training course on an individual product (Exchange, ESX, Windows 2008) in the UK will cost in the region of £1,500-3,000 GBP and those 5 days will be slow-paced (9.30 –> 4.30pm affairs). The course content and material has to cater to the lowest common denominator delegate, for a geek/experienced tech this can make for frustratingly slow progress and means you only cover a very narrow technical focus, or broad high-level overview – you can’t easily dive in and out of the bits that are relevant to you with a traditionally delivered course and even the best instructor in the world can’t dedicate that much time to you in a classroom environment.

So compare that training course is £2-3000 + travel + accommodation + time out of the office to VMworld (for example..), even at the most expensive register on-site on the day prices

  • VMWorld Full Conference Pass* 1,260 EUR (£1,176 GBP at current exchange rate) (Tech-Ed 5 days c.£2000 full price* ticket)
  • Travel (airfare from most of continental Europe, economy/flexible flight) c.£400**
  • Hotel (normal business hotel, 4 nights) £900**

+Access to on-demand streamed and downloaded content following the conference (access allowed until the next VMworld) included

+Lunch/breakfast usually included

+Networking opportunities, access to product teams and managers included

+trade show with relevant vendors/suppliers included

+bag and pen included (ok, I’m struggling with that one! :))

note:

*Early registration attracts a large discount on the full conference pass – look for “early bird” tickets which can knock a significant percentage off the full price

**If you are prepared to “slum” it with budget airlines and hotels this is significantly cheaper.

Prices for reference:

VMworld Europe 2009 prices

Tech-Ed Europe 2009 Prices

With Microsoft Tech-Ed they usually give a complimentary Technet Direct subscription – which is worth hundreds of pounds on it’s own and gives you multiple copies of almost every Microsoft product for your own use.

So if you look at it pragmatically – VMworld/Tech-Ed give you the flexibility to tailor your content to what is important to you; as well as the ability to take all the information away with you to review online post-conference (even for the sessions  you didn’t make in person)

With a training course you walk away with a nice certificate, some spiral bound manuals and if you are lucky – a pen 🙂

And they both come out to roughly the same price.

I’m not saying this is for everyone – you need to be a self-starter to make the most of these conferences, and if you do a limited scope day-job and that is all you are interested in doing, traditional training courses are probably your best bet but for those of us that work as consultants or want to broaden our horizons – go for it!

My write-ups of previous VMWorld and Tech-Ed events can be found at the following links:

Tech-Ed EMEA 2008

https://vinf.net/2008/10/31/off-to-microsoft-teched-emea-2008/

https://vinf.net/2008/11/04/teched-emea-2008-it-pro-day-1/

https://vinf.net/2008/11/04/teched-emea-2008-it-pro-day-2/

https://vinf.net/2008/11/06/teched-emea-2008-it-pro-day-3/

https://vinf.net/2008/11/06/teched-emea-2008-it-pro-day-4/

https://vinf.net/2008/11/07/teched-emea-2008-it-pro-day-5/

VMworld Europe 2009

https://vinf.net/2009/02/23/vmworld-partner-day-keynote/

https://vinf.net/2009/02/24/vmworld-partner-day-wrap-up/

https://vinf.net/2009/02/24/vmworld-europe-day-1-keynote/

https://vinf.net/2009/02/24/dc02-best-practices-for-lab-manager-vmworld-europe-2009/

https://vinf.net/2009/02/24/dc14-overview-of-2009-vmware-datacenter-products-vmworld-europe-2009/

https://vinf.net/2009/02/24/vmworld-europe-day-1-wrap-up/

https://vinf.net/2009/02/24/vmware-client-hypervisor-cvp-grid-application-thoughts/

https://vinf.net/2009/02/25/vmworld-europe-day-2-keynote/

https://vinf.net/2009/02/25/how-vmware-it-use-vmware-internally/

https://vinf.net/2009/02/25/vmworld-europe-day-2-wrap-up-a-good-day-despite-the-curious-lack-of-forks/

https://vinf.net/2009/02/26/hands-on-lab-01-vsphere-features-overview/

https://vinf.net/2009/02/26/hands-on-lab-12-cisco-nexus-1000v-switch/

Windows 7 and the Intel 855 video driver problem

 

Judging by the several hundreds of hits that this post on my blog gets every day since January I would say there is a *serious* demand from the general internet community to use laptops with the Intel 855 family video chipset with Windows 7, and it’s not even been released to the general public until now!

Whilst this may be an “older” chipset; in reality those laptops aren’t beyond serviceable life – and if anything the performance increases in Win7 will make them more usable and extend their life a bit further.

Nobody realistically expects fantastic 3D/Aero graphics from this combo, but a driver that supports the panel’s native resolution would be more than adequate for browsing/word processing etc. no doubt these ex-corporate laptops are being cycled around family members/students for some time to come rather than forcing people to squint at standard VGA.

There are ways to get the Vista driver bodged into Win7 (see this post and it’s comments), but it’s far from ideal or stable

Please Intel/Microsoft – surely you must be able to produce a basic/compatible driver by today! otherwise I’m sure some of the Windows launch parties could be something of a disappointment!

if it goes by way of evidence – this is the number of hits my original blog post about this problem has had in the last 10 months, it will be interesting to see what it does from launch day onwards!

screenshot_01

BlackBerry app for WordPress

I must be a bit behind the times, but I have just stumbled across the beta version of a blackberry app that let’s you publish and manage your blog directly from your phone, there is also an iPhone version

Very cool, you can now all look forward to accidental posts from my pocket and I can look forward to RSI from my blackberry Pearl 🙂

If you are interested you can download it OTA from httphere

Incidentally it also supports the builtin camera, and this post was typed on on a blackberry 8120 pearl.

Can’t see a way of viewing stats yet, which would be a nice addition.

VMware AppSpeed Probes and more 2% Maintenance Mode Problems

 

Following on from my last post on problems entering maintenance mode with FT-enabled VMs, I seem to have found another one – if you have the rather excellent AppSpeed product deployed on an ESX cluster and you want to put a host into maintenance mode it gets stuck at 2% as it can’t move the AppSpeed probe VM onto an alternative host

image

If you try to manually vMotion the problematic probe off to another host in the cluster you get the following error

image

If you shutdown or suspend the AppSpeed probe VM then the switch to maintenance mode continues as expected.

image

This would make sense as it plugs directly into a dedicated vSwitch on that host to monitor network traffic so vMotioning it off wouldn’t be of any use – assuming the other nodes in the cluster are also running AppSpeed probes.

However it would be great if there was a more automated way to handle this? guess it’s tricky as on one hand its great that AppSpeed doesn’t rely on any ESX-host agents and is essentially self-contained with probes running as VM appliances but on the other hand the probe doesn’t know the guest is being put into maintenance mode so should be shut down/suspended rather than vMotioned to an alternative host.

There is integration with the vCenter server via a plug-in so maybe in future versions that could trap a maintenance mode event and initiate (or suggest) shutting down the AppSpeed probes.

VMware FT, 2 Nodes and stuck on 2% entering maintenance mode

 

I have a 2 node vSphere cluster running on a pair of ML115g5 servers (cheap ESX nodes, FT compatible) and I was trying to put one into maintenance mode so I could update its host profile, however it got stuck at 2% entering maintenance mode, it appeared to vMotion off the VMs it was running as expected but never passed the 2% mark.

image

After some investigation I noticed there were a pair of virtual machines still running on this host with FT enabled – the secondary was running on the other server ML115-1 (i.e not the one I wanted to switch to maintenance mode)

image

image

I was unable to use vMotion so that the primary and secondary VMs were temporarily running on the same ESX host (and that doesn’t make much sense anyway)

image

That makes sense, the client doesn’t let you deliberately do something to that host that would break the FT protection as there would be no node to run the secondary copy. incidentally this is good UI design – you have to opt-in to break something – so you just have to temporarily disable FT and should be able to proceed.

If I had a 3rd node in this cluster there wouldn’t be a problem as it would vMotion the secondary (or primary) to an alternative node automatically (shown below is how to do this manually)

image 

However in my case all of the options to disable/turn-off FT were greyed out and you would appear to be stuck and unable to progress.

the fix is pretty simple and you just need to cancel the maintenance mode job by right-clicking in the recent tasks pane and choosing cancel, which then re-enables the menu options and allows you to proceed. Then turn-off (not disable – that doesn’t work) fault tolerance for the problematic virtual machines

image

image

image

The virtual machine now doesn’t have FT turned on, if you just disable FT it doesn’t resolve this problem as it leaves the secondary VM in-situ, you need to turn it off.

So, moral of the story is – if you’re stuck at 2% look for virtual machines that can’t be vMotioned off the host – if you want to use FT – a 3rd node would be a good idea to keep the VM FT’d during individual host maintenance; this is a lab environment rather than an enterprise grade production system but you could envision some 2-node clusters for some SMB users – worth bearing in mind if you work in that space.

Designing Active Directory – Talk by Brian Desmond in London

 

If you’re in the UK and are interested in Active Directory – Brian Desmond (an MVP for AD) will be giving a talk on Active Directory design on 29th October in London.

Details here on the ADUG website here and – registration is free.

I’ve been working with Active Directory for a long time but it’s always refreshing to have some Q&A with industry peers, so if you can make it it promises to be an interesting evening.

Not sure if it will be recorded/streamed for people that can’t make it – if not can I make that suggestion to the ADUG team- I can bring some recording equipment if you need it.

Performance Update on Cheap vSphere Server

 

My home lab has a pair of HP ML110 servers with 8Gb of RAM running vSphere 4 (more info here) it’s configured in a cluster with iSCSI storage running from an old HP D530 PC with a 1Tb hard disk running OpenFiler. it performs pretty well and meets most of my needs, I thought I’d do a quick couple of screenshots of the average performance I have seen on it over the last 3 months.

image

it’s running a constant load of about 17 mostly Windows virtual machines and a varying load of test environments which are suspended to disk – think the most I have ever had running on the 2-node cluster at one time was about 45 VMs and performance was ok – trying to use VUM to patch all those VMs at the same time killed things though, as all the VMs are running from a single 1Tb SATA disk over OpenFiler.

This is a list of all the VMs, you can create your own html list as follows, or you can also save it as a CSV to import into Excel to manipulate.

image image

image

the following screenshots show the last 3 months of performance stats from vCenter as the number of VMs has increased and decreased as I’ve provisioned and removed VMs for testing.

Overall CPU usage for the cluster

image 

vMotion and VM Reconfiguration activities

image

Cluster Memory consumption

image

The new overview page feature can show you a quick summary of virtual machine performance

image

Drilling down into the performance tab on each host gives more information on specific performance like disk and network

image

image

You can also produce a stacked graph showing guest CPU usage of each VM on a host

image

or identify which VMs have the busiest virtual disks

image

image

You can also view a stacked (per VM) graph showing on a per-host basis how much physical RAM the guests are consuming, relative to each other over time.

image

My Technical Book Reading List

 

I work as a technical architect in a consultancy role in the Windows Infrastructure & Virtualization space, and as such I’m paid for my experience and opinion, but it’s also an important part of my job to proactively keep on to top of industry trends and investigate new technologies that may solve customer problems, even problems they don’t know they have {yet}.

Twitter and blogs are a useful resource for current and less formal or structured information if used carefully, but it’s all too easy to be distracted from the job at hand by the latest shiny tech or flamewar so you need to measure the amount of time you spend there and they are no substitute IMHO for traditional study and hands-on “playing” time.

Like many I’m currently studying for my VCP4 exam, and have recently upgraded my MCSE to the current MCITP:EA 2008 certification so I thought I would publish a reading list for anyone else looking to do the same as well as general industry books I have been reading recently.

Maintaining a balance between this study/pro-active time and staying a well utilised/billed out resource is hard and I find it often spills into my own personal time so you need to have a good level of personal interest/dedication otherwise you will struggle – I’m a geek at heart and I look upon this as investing in my career but it’s more important that your kids remember what you look like!

This is one area where travelling for work via public transport rather than driving works well – plenty of study time to be had with a good pair of headphones

(i.e not the rubbish ones that ship with the iPod), a good list of tunes to block everyone else out and a book/laptop with VMs.

Maintaining a good home lab environment is also critical for me, and I run all the services for my demanding demo environments as well as my home users (read: wife & kids) there and that really does teach you something about availability :)  I wrote some details of my lab setup here but I have an updated post in the pipeline as there have been some significant changes to support my vSphere study.

Anyways, enough rambling and on with the list..

VCP4 Upgrade

Mastering VMware VSphere 4: Scott Lowe: Books

ISBN: 0470481382
ISBN-13: 9780470481387

You can’t go far wrong with the Lowe, this is the definitive vSphere book at the moment.

VCP4 practice exams online at the SLOG here

VCP4 resource list here

if you wanted a similar book for VI3 I would recommend this one;

Microsoft Certified IT Professional (MCITP)

Windows 2008 MCITP Core Exams study guide

This is a comprehensive set of books for the 4 core exams, not as dry as previous versions and lots of practical lab excercises, as usual good set of practice tests on the included CD

For the client component of the MCITP:Enterprise Administrator cert

MOSS 2007 Exam

Not the most exciting product for me, but it does cover the exam requirements – seems to spend a lot of time explaining what each menu item is within the MOSS UI, which was a bit frustrating.

Very good book, puts a lot of real world around MOSS deployments

General Good Books to Read

Cisco UCS

Good book covering what Cisco’s new UCS blade system and consolidated I/O model is; not much information available elsewhere at the moment, although 1/2 the book is spent discussing the various CPU/memory bus architectures which is a good update to your knowledge but would have liked to have seen more time dedicated to how UCS works and some example configurations

General computing/tech

An excellent book, with a focus on open source technologies and a lot of practical insight from the building of Flickr – good for briding the infrastructure/application divide.

 

 

If you don’t get cloud computing or understand where things could go, you need to read this book, brilliant and not a long, drawn-out read and not that technical.

Aimed more at developers rather than infrastructure people but if you want to try things out for yourself it has some easy to understand examples.

I hope you found this list useful, you may notice that these are affiliate links – should you wish to purchase any of these books Amazon will pay me a nominal commission which I can use towards the normally ad-free funding of this site and my time, although you are entirely able to go and purchase any of these books directly from the Amazon site.

Baby and non-IT related books

We recently had our 2nd child and it reminded me of these great books for all fathers to-be 🙂

The Bloke’s Guide to Pregnancy: Jon Smith: Books

ISBN: 140190288X
ISBN-13: 9781401902889