Tuesday, December 6, 2011

iSCSI Multipathing (MPIO)

With the enhancements of iSCSI devices over these years, it’s not unusual to find some environments choosing iSCSI implementation over Fiber Channel implementations.

I believe it’s clear to everyone that FC provides the fastest and more reliable solution these days and of course it’s the most expensive solution, right ?!?

But iSCSI solutions have it’s merits . So, how do you get the best performance of it ?

Well, the more obviously approach would be use faster connections.
If you are already using 10gb connections you probably wont see much difference having more than one connection.
Now if you have 1gb connection and cannot migrate to 10gb, add multiple NICs and path to your configuration.

Configuring iSCSI Multipathing.
I’ll not try to cover here all the aspects of how to accomplish that, it’s because different iSCSI storage vendors present storage to servers in different ways. Some vendors present multiple LUNs on a single target, while others present multiple targets with one LUN each.

My best advice is to check with your storage vendor how to configure it for your specific environment; they all have documentation about it.

So, what’s this post about ?

My first though was to alert about the misconception about it, some people tent to believe that if you just add more NICs to your virtual switch where the VMKernel port is configured it will automatically provide load balance and bigger throughput, just like the virtual machines connections do.

That’s not true !!!

Since vSphere iSCSI stack acts like a “port binding” you will end up with just one active connection per iSCSI initiator X iSCSI target, regardless of how many NIC you have attached to your vSwitch.

To accomplish multipathing you will need to configure additional vmkernel portgroups and bind each NIC to each portgroup.

Let’s see how it works.

1 – Configure additional vmkernel portgroups

Configure as much portgroups as NICs will have for iSCSI traffic

2 – Map each iSCSI port to just one active NIC.
By default all NICs are active, you will need to overwrite vSwitch failover order policy so that each port maps to only one corresponding active NIC

3 - Binding ports
Now the final piece: you will bind vmknics to iSCSI initiators.
First identify the name of iSCSI ports. (get them from the VI client, Networking option)

Second, you need to identify the vmhba names. (get them from the VI client, Storage Adapters option)

Finally you just run the command which will bind them.

esxcli swiscsi nic add -n port_name -d vmhba

on our example it will be this:

esxcli swiscsi nic add -n vmk1 –d vmhba32
esxcli swiscsi nic add -n vmk2 –d vmhba32

If you display the Paths view for the vmhba32 adapter through the vSphere Client, you see that the adapter uses two paths to access the same target. The runtime names of the paths are vmhba32:C1:T1:L0 and vmhba32:C2:T1:L0. C1 and C2 in this example indicate the two network adapters that are used for multipathing.

You can now configure your discovery initiators and rescan your datastore.

AGAIN: it’s more a heads up than a procedure to follow, remember: there are several factors that could contribute to how do you set it up, like software assistance or hardware dependent of your card connections, so check with your vendor.

If you want to read more:
VMWARE has a good guide: iSCSI SAN Configuration Guide
Virtual Geek blog has also very good information about it

Now it’s up to you ; )

Wednesday, November 9, 2011

NetApp, Cisco and VMWARE Cloud solution

A couple of months ago I wrote an article about some Cloud Solution that are out there.

A friend of mine how works at NetApp shared with me their solution for Cloud and Virtualization.

It's called FlexPod

The good thing about those solutions is that you can rely on them about the certification components, performance capacity , proof of concept, etc. It's just because the major players already made that for you.

so, here's another solution for your consideration when choosing your Cloud solution

Tuesday, November 1, 2011

VMWARE is not just about virtualization.

If you think VMWARE is all just about virtualization, you are wrong.
It’s also playing well on SaaS (Software as a Service) field.

VMWARE has several companies/projects in parallel and I’m sure you will be amazed with those products. As every new app design for the Cloud, they provide access to several types of devices, like PCs, smatphones, tablets, etc..

SocialCast, think about it as your internal Facebook. You can share ideas among the peers or your project or the entirely company. With this collaborative environment you can increase productivity and communication.

SlideRocket, it’s a communication platform intended to create presentations and share in a more rousted way. You can for example update your presentation and everybody how has a copy of it would automatically see the new version.

Zimbra, is the email, calendar and collaboration system. It has great flexibility and integration with others tools, it really looks like the email of the future.

Project Octopus, it’s your central location to store your files, it looks like the file share you have at work right, but because it’s on the internet, or should I say cloud, it’s accessible anywhere you are not just inside your company. Also you can share those files with anyone with all the secure you want.

Horizon App Manager, is a centrally dashboard for all application will need access to, it also provides single-sing-on to all of them. But the great thing about it is that you can enforce your policies even with the 3rd party applications and all your password stay behind your firewalls.

Shavlik, VMWARE has also your own patch management system, it works pretty much as Microsoft Baseline Security Analyzer, but the engine has been improved to include much more systems and applications to check the vulnerabilities.

you can learn more about which one of them, just going into theirs home pages.

see you.

Thursday, October 20, 2011

Slots size and offender guests.

Hey there, it has been a long time ....
The past few weeks I went very busy and just today I could afford some minutes to post something new.

At this point you should be familiar with HA and slots sizes, right…if you are not I recommend you to read this post at Yellow Bricks.

Now you might realize that the slot size will direct impact your failover capacity.

So be careful when setting up reservations on your guests.

vSphere 4.x gives us good information what the slots sizes are:
Selecting the cluster, click on the Summary Tab,
You will find the Vmware HA information box

If you click “Advanced Run time Info” you will see it’s slot size

Ok, now you know what your slot size is. But how do you identify which individual VM is contributing to the slot size ?

Well, I found this great script at Virtu-Al blog.

Identifying the offender VMs is helpful to understand if you made a mistake configuring them.

Check the video bellow on that in action.

SlotInfo from Alan Renouf on Vimeo.

Wednesday, September 21, 2011

VMWARE Memory Management and More

Today I was watching the presentation which Jonathan Klick did for VMWorld 2011, “Correctly Sizing Memory in Virtual Environments”

While it brings some good information, metrics and insights about what you should be watching when right sizing your environment, it just mention a lot of terms that you should me familiar with, like : EPT, Large Pages, TPS, etc..

So I decided it was time to shake off the dust and give you some more information about them.

First things first, when monitoring the memory metrics of your guests, understand what each one means: this will help you a lot.

Second, you need to understand what Transparent Page Sharing (TPS) is.
Let’s forget all the bits and bytes behind how a virtual memory page translates into a physical host memory, the important piece to know is that ESX was used to use small pages , 4k in size, so it was easy to ESX to identify common pages and instead of load 2 or more pages with the same data into memory, they just load on physical memory once and share them with the others guests. In basic terms it’s TPS. Easy ?

Now with the advent of new processors like Nehalem some new virtualization functions have been introduced to the processor, one of them is EPT (Extended Page Tables). When ESX identifies that capability on it’s processor they start using Large pages, 2mb in size, instead of using small pages

What Large pages has to do with TPS ? Well, because it’s unlikely that there will be 2 or more 2mb pages identicals, there’s no gain on wasting resources to identify them.
With consequence you will see a lot more host memory consumed and less memory being shared among guests.
So, is there any performance improvement using Large Pages ?
Definitely. Check the VMWare performance evaluation.

But what about TPS, has it been disabled ?
No. It still there, but it will be used more when there’s memory constraints.
When your host is suffering memory pressure the ESX will break these Large pages into small pages, that’s when TPS is back to game finding more easily equal pages and sharing them between guests, increasing the available memory on the host.

Making short a long history.
Keep Large Pages and EPT on. Let ESX take care of the memory management.

As you might know, there’s no size that fits all.
What I want to say with that is, I’m giving you some information about what’s out there, now it’s up to you to understand your environment, your applications, monitor them and tweak them to get the best of your software can offer.

good luck.

More reading:
Large Pages from Yellow Bricks

Monitoring TPS from Yellow Bricks

Large Pages a problem of perception


EPT affects TPS from VMWARE

Wednesday, September 14, 2011

Custom ESX firewal rules

As you know ESX has a built-in firewall. It means that no communication is allowed unless you specify it. Glad a few basic ports are opened by default.

Check here a list of the ports required for the majority of VMWARE functions.

You can open/close ports through vSphere Client or with esxcfg-firewall command.
But let’s agree that doing that through the vSphere Client is a lot easier: you don’t have to know the syntax of the command, logon through SSH on your host or even run remote commands.
You just go on Security Profile and check the desired rule.

There’s a problem with that approach, Security Profile just shows a couple of rules and ports to be checked.

What if you have an application that needs communication through a port which is not there ?
Would not be nice if you could create your own firewall rule ?

It’s possible!!!
VMWARE KB1021779 gives you the directions to accomplish that, changing the /etc/vmware/firewall/services.xml file.

I would be very carefully changing this file, it’s a system file which controls a lot of services, also it’s probably that some future patch will get in place and replace it, whipping out your customization.

So, my advice is to create an xml file in /etc/vmware/firewall/ and then create your rule details inside it.
There are a bunch of files at /etc/vmware/firewall/ just take a look at them for syntax understanding , it should be very easy to build your own.

Here’s an example of a rule I created called ExtraPort which opens TCP 3434 for inbound and outbound:

Once you create your file restart the mgmt-vmware service.

Next time you went through Security profile you will see your firewall rule in there.

You are now ready to give your junior system admin the task to open and close ports without worrying too much ; )

Wednesday, August 31, 2011

DRS/vMotion sudently stops working

Hi there,

It’s easy to forget about a few things when you have a lot of things we need to take care on our daily basis activities.

DRS is one of those things you set it up, watch it working a couple of times, get comfortable on how it works, forget about it and never revisit this topic. (until it goes bad).

Despite the fact my hosts were well balanced, I decide to take a look at the vMotion activities on my cluster and figured out that vMotion metric was really low.
My next step was to check the migration tasks history. It turned out that only 2 hosts were vMotionning VMs between them, there were no activity from the others hosts.

So, I tried to make a manual migration, to my surprise I got an error message during the validation test:

vMotion interface “VMotion” on destination host “hostA” does not have any operational physical network connections.

And another message said that my vMotion network label does not match destination/source.

That’s odd ! We all know the vMotion portgroup label MUST be the same among all hosts within a cluster.

I checked the label on them and they were the same on all hosts.
That’s when I remembered they were first configured with a mistyped label and to fix that I just rename it to match the others.

Well, this time I wanted to make it right, so I deleted the vMotion portgroup and created a new one with the right label.
After that the migrations are back in business.

Remember: does not rename the vMotion portgroup label, delete and create it instead

See you ; )

Tuesday, August 23, 2011

New vSphere Licensing 5 Model

Well, despite all the amazing new features VMWARE is launching with vSphere 5 the hottest topic seems to be the new licensing model.

At first I thought they were going into the wrong direction but as I started learning the “why” and motivator to this change I realized it has value.

If you consider in the future you will have hosts with less physical CPUs (sockets) and more cores, it definitely means less licenses.

But what about the vRAM pool concept ?

I believe everybody likes to have some room for growing or even temporary high utilization. So, it’s note unusual that you have 20% of your total amount memory reserved for future utilization.
Also as memory might be the less expensive component these days, you tend to buy as much as possible.

So instead of licensing all the physical RAM you have available, you will have to license just what has been utilizes for your VMs. (vRAM pool takes into consideration the amount of allocated vRAM on each powered on VMs)

Yeah, just powered on VMs, that means you will pay what you use.

It’s better to pay attention now on how much of memory you allocate to any given VM and monitor it’s utilization to right size it accordingly, or you will be just wasting your money on licenses.

I’m not here to debate if it’s right or wrong, it’s just different, but to be honest it seems a lot fairer to me.

I encourage all of you to read the VMware vSphere License White Paper.

You now might be wondering, how do I know how many licenses do I need when upgrading to vSphere 5 ?

Good news !! VMWARE released a tool to help you identify your vRAM pool needs and amount of licenses you will need.
It’s called vSphere Licensing Advisor and can be downloaded here.

Here’s a video from http://www.virtu-al.net/ that shows you how it works

VMware License Advisor from Alan Renouf on Vimeo.

Friday, August 12, 2011

VPXD.EXE at 100%

After a relaxing 20 days vacation I’m back.

And right on my first day back a weird issue had pop up.

Virtual Center could not be opened, the O.S. was very unresponsible, when checking the processor utilization it was at 100% all the time and the most offender process was vpxd.exe

Getting a little of background of what have happened during my absence I realized the VC has been moved from one cluster to a new one. (it was supposed to happen since we are migrating our environment).

I started wondering if the issues could be because of that change and then I decided to revert it.

After shutdown the VC and register it back to it’s previous host/cluster the O.S just started normally, Virtual Center was working fine again.

But several v-motions tasks (started by DRS) started.
I wait all v-motion tasks to finish and the work load to establish on the new cluster.

At this point I can only guess
When VC started on the new cluster and tried to load balance it’s load through DRS it got kind of crazy with the situation of a new guest has been added to the cluster while it’s off, not just any guest, in fact the problem was because it’s new guest is the virtual center itself, so it went in a kind of loop.

That’s what I did to successfully migrate the VC to the target cluster.

I disabled DRS for the new cluster.
Shutdown VC again
Register it on the new host/cluster
Start the VC again

This time the VC has started properly and I just needed to enable DRS back.

Lesson learned:
When moving your VC to another cluster, first disabled DRS on the target cluster and re-enable it back after the migration.

See you.

Sunday, July 24, 2011

New vSphere 5

Last week VMware has announced the new release of its new Cloud platform: vSphere 5.

There are a lot of new enhancements and features; as you would expect all of them are intent to improve performance, availability, scalability, automation, etc..

Some other productions will got a new release too: vShield 5, Site Recovery Manager 5, vCloud Director 5.

Specifically about the vSphere 5 suite, you now can have VMs with 1TB of RAM and up to 32 vCPUs. There are new features as Storage DRS, to help you load balance the utilization of your storage, Virtual Center itself could be deployed as a Virtual Appliance running Linux, a new license model, etc…

Well, there is hundreds of news, so why don’t you take a look at this document for a comprehensive list ?!!?

vSphere 5 will just be available at 3Q of 2011 and until there, there’ll be a lot to study.

Start learning it here.

Thursday, July 7, 2011

How to align disk partitions

I own you guys this topic, It has been more than 7 months since I made this promises.
So it’s time to make it right!!!

You remember about unaligned partitions, and how to identify if your partition is mis-aligned, right ? If you dont get back and read these posts before we proceed.

First thing we need to remember is Windows 2008, Vista, Windows 7, they all align their partitions correctly (64KB) by default, so you don’t need to worry about them.

But what about your Windows 2000/2003/XP ? They align it with 63KB making then unaligned with your datastore and in some cases causing performance issues. But you already know that.

Without further ado, let’s see how to align those partitions.

Manual alignment of your partition is a destructive task, it re-creates your partition and no data can be kept while performing that.
So, it’s perfect for your new data drivers.
You just attach a new VMDK file to your guest, create partition, align it, format it and your are ready to go.
But, what if your data driver already has data?
Take a backup of the data, destroy the volume, re-create the partition , align it, format it and restore the data.

What about the system disk ?
It’s a little bit harder
If it’s a new guest and the O.S. has not being installed yet, attach the system VMDK file to another guest system (the disk will show as a secondary disk), create partition, align it, format it.
Then disconnect the disk from the guest and attach it back to your original guest, now it’s just start it up and install the O.S as you would normally do.

If your O.S. is already installed, you will need to take a backup of your system state , perform the same step above as if not O.S. has been installed and then restore the system state.

Manual alignment procedure
- Open a command prompt, run diskpart;
- list disk – then select disk #;
- create partition primary align=64
- Format the disk as you would do normally

Thanks God there are a few tools that can help us with this job without all this manual effort.

If you have some extra bucks to spare you can use vOptimizer Pro from Quest. I have never used it but I’m sure it gets the job done.

if you are a Netapp customer you can use the free tools MBRSCAN / MBRALING tools. Since it’s my case, I’ll show you how it works.

First your need to download it from the NOW Support WebPage (you will have to have an account to access it).
Mbrscan and mbralign are part of the Netapp Host utilities.

Before run the tools make sure your guest is powered off and has no snapshots.

mbrscan checks your VMDK files to see if they are aligned or not. You will need to run it against each disk of your guest.

- Copy the mbrscan file to the /tmp directory of the ESX server (I use /tmp, you can use the one that fits better for you)
- In the /tmp directory run the command ‘sudo chmod 555 mbrscan’ to set the execution
- Run /tmp/mbrscan -flat.vmdk
(NOTE: you are checking the –flat.vmdk file)

mbralign will aligned your VMDK file to 64KB without destroying the data. You will need to run it against each disk of your guest.

- Copy the mbralign file to the /tmp directory of the ESX server (I use /tmp, you can use the one that fits better for you)
- In the /tmp directory run the command ‘sudo chmod 555 mbralign’ to set the execution
- run /tmp/mbralign .vmdk
(NOTE: you are aligning the .vmdk file).
*Also, make sure you have enough space on the SAN lun to have backup files that will be aligned

Run mbrscan again to make sure it’s aligned.

Mbralign will make a copy of your VMDK files with an extenstion called -mbralign-backup before it aligns them.
So it would be easier to restore your old VMDKs in case of corruption (just delete the new vmdk and renames the files back to the original name)
In case everything went fine, delete the backup file to do not waste space on your datastore

But how to avoid new unaligned disks ?
Make sure all your templates are aligned, that way all new guest will be aligned as well ; )

Good Luck.

Thursday, June 30, 2011

HA and DRS Best Practices

It has been quite a while since my last post, that’s because I was wrapping up my studies to take 70-649 Microsoft exam, it’s intent for the ones upgrading their MCSE to the new Microsoft Titles as MCTS and MCITP

I know I’m a “little” behind the schedule, but so much to learn on so many areas…anyway, first step has been taken. I passed !!

But that’s not I want to talk about on this post.

Do you remember when I wrote about the Vmware Vsphere HA and DRS Technical deepdive book ?

It has happen that Alan Renouf, owner of the Virtu-al.net web site has created a script that queries your cluster and compare it to the best practices presented by the book and provide recommendations.

Be aware, the recommendations are based on book's best practices and despite the fact it has been written for some skilled guys there's not size that fits all.
So, the idea is to learn the details, understand your environment and if it’s the case, fix them.

At least it’s a wonderful tool to document your environment.

How to use. (dont need to mention to download the script, right)
- Start vSphere PowerCli
- Run the script from the powercli console
- Fill the fields with your Virtual Center Name, your ID, password and the location to safe your report

- Then click Connect

In a few seconds the report will open, as you can see bellow there are so many cool details.

I recommend a try

Thursday, June 16, 2011

Update Manager not working after June 01st.

Probably you guys are aware of VMWARE KB1030001, which states:
“To continue applying patches on ESX 3.5 hosts, the secure key needs to be updated before June 1, 2011. This patch updates the secure key.”

As state on the KB you need to install the patch ESX350-201012410-BG.

That’s OK so far, but, it was not clear, at least for me, how your environment will behavior in this situation: will it provide a clear message about the requirement, will just the patches released after June 01st be affected ?

I’ll try to answers these questions and how to solve it.

First, you will not be able to even scan your host, so it will affect not only the new patches but the entirely Update Manager functionality.
Second, it will not give you a clear message about this issue. Instead you get an general error, like:
“VMware vCenter Update Manager had an unknown error. Check the Tasks and Events tab and log files for details.”

It does not help much, right ?

To get the confirmation about this error is being caused by the secure keys you need to check the update manager log. (/var/log/vmware/esxupdate.log)
There you will find detailed description as bellow:
- Integrity Error!
- keyExpired

Now that you are sure about the cause, let’s see how to fix it

At this point you are unable to use Update Manager on this host, so will need to manually update the new secure keys.

Download ESX350-201012410-BG.

As you can see there’s a dependency patch called ESX350-201012404-BG.
If your host does not have it installed already, download it as well.

Unzip the packets and copy them to your destination host.
Log on the host through the console or a SSH session.
Navigate to the folder where you copied the patches to be installed and run the following command:
esxupdate--noreboot--nosig update

Again, if you don’t have ESX350-201012404-BG installed you will need to install it before you proceed with ESX350-201012410-BG.

Both patches don’t require a reboot of your host.

After the installation you should be able to use Update Manager to scan and patch your host.

Piece of Cake, right ; )

Thursday, June 9, 2011

VMFS Version – VMWARE Datastore

Do you know the LUNs on your environment might have different versions ?

Yeah, that’s true!!
To check tha LUN version, on the configuration tab, select the Storage option and click on the desired LUN.

On Datastore Details, you will see it’s VMFS version

So, why did it happen ?

VMFS version depends on the version of your ESX host by the time when you last formatted the LUN.

• ESX 3.0.0 is provided with 3.21 (initial release)
• ESX 3.5.0 is provided with 3.31
• vSphere (ESX 4.0) is provided with 3.33
• vSphere (ESX 4.1) is provided with 3.46

Now you might be wondering, Does the version of my VMFS impact any new feature or performance ?
The answer is NO. Features and enhacements are implemented at the system drivers level.

But if your environment is standing long enough, since ESX 2 days, you might have one or two LUNs formatted with VMFS2, but you would probably notice that a long time ago.

If you have a mixed environment with ESX with different versions, my advice is to always format new LUNs using the host with your latest version, that way the LUN will be created on the latest version. Would not hurt, right ; )

Now if you are a kind o control freak and want all your LUNs at the same and latest level the only way to upgrade it is deleting the LUN and creating it again.
Make sure to move your data to another LUN before delete it.

If you want more details VMware has a KB about it: KB1005325

Thursday, June 2, 2011

ESX Active Directory Integration 2/2

On my last post we talked about the AD integration with ESX 3.5.
Now it’s time to cover it under the new ESX version, vSphere 4.1

Instead of having to create local users on your hosts and configure LPAD direction you can make your ESX 4.x member of the domain. That’s right. It’s a new functionality for ESX 4.x, you can actually make your ESX host a member of the domain and setup privileges based on the AD users and groups.

Just for the record, ESX make is possible through Likewise.

Because I don’t like to reinvent the wheel, I will not describe here all the steps to configure it, I’m pretty sure if you do a google search you will find a lot of references on how to accomplish that, So I will show will this video that covers all the aspects on how to configure it.

If you want to know more check it out.
Have fun

Tuesday, May 24, 2011

ESX Active Directory Integration 1/2

Today let’s talk about the AD Integration with ESX 3.5
I believe most of you have to, at least once, to log in the ESX console for some troubleshooting, some third part app installation or maintenance, whatsoever.

You can do this with root account.
But you don’t want to share it’s password with dozens of administrators of your company. The auditors will be pulling their hairs off if you do that.

The best practice is to disable the remote access (SSH) for the root account.
Edit the /etc/ssh/sshd_config file
And change the PermitRootLogin from yes to no
Restart the sshd service

Now you should be safe ; )

But you still need the capability of login remotely, right ?!?

Here’s the thing:

- create a local account, with blank password, where the ID’s match the ID’s on your AD
run: useradd bob
there are a few options for group, comments, shell, etc.. (Check useradd man page for all options)

now that you have your accounts create locally you can set up LDAP redirection to your domain.

esxcfg-auth --enablead --addomain=example.com --addc=dc1.example.com

now you should be able to login through a SSH session with the ID and password from your AD domain.

Now you realized you cannot run the command which require admin privileges.
It’s time to tweak the /etc/sudoers file. Use VISUDO to do that.
if you have doubts, here’s the SUDO website about it.

Next post I will talk about the Active Directory integration with ESX 4.x

Friday, May 13, 2011

Cannot add existing disks to my guest

Since I migrated my VMWARE environment to vSphere 4.1 Virtual Center, I started getting a few errors when adding existing disks to virtual machines.

The error says:
Adding existing IDE disk is not supported at the moment. IDE disks cannot be hot added or there are no free available IDE controller slots.

I realized that disk are old disks that are on my environment since older versions of virtual center and VMDK descriptor file contains a legacyesx value instead of the lsilogic value of current ESX versions.

Seems this parameter is not compatible with vSphere 4.1 anymore.
So, you need to edit the VMDK description file and replace

ddb.adapterType = legacyESX
ddb.adapterType = lsilogic

After that the disk could be attached to the virtual machine without any problem.
Check VMWARE KB 1025883 for more details

Friday, May 6, 2011

Cloud in a Box

Today let’s talk about what I call as Cloud in a box.

Probably you realized that to build your virtual infrastructure you have to deal with a lot of stuff, servers, storage, network , softwares, etc…. everything needs to be certified to interact with each other.

VMware provides a Hardware Compatibility List (HCL) which you can use to search every component to make sure it’s certified.
But, let’s be honest it’s not an easy/fast take.
Also you have to be an expert on network, storage and hardware to make the right call.

May be if you are a small or medium company you don’t have the right guy on your staff.

Well, but you do want to enter into the virtualization/cloud world, everybody does!!

That’s where the big players getting in… IBM, DELL and VCE have their own solution. In general it’s a rack with everything you need, servers, storage, network, software, installed, connected, tested, certified, ready for use.

It’s the new plug-and-play for Cloud.

Of course IBM will use it’s own brand, VCE will use EMC storage and Cisco switches. Despite the price you can pick up the one of your preference.

The only common thing among them is that VMWARE vSphere is running behind the scenes for the provisioning. How cool is that ?

IBM has CloudBurst
VCE (witch is a join venture between VMWare, Ciscos and EMC) has vBlock
Dell has vStart
NetApp has Flexpod





But, that has been always my choice. Enjoy

Friday, April 29, 2011

Vmware PowerCLI script for Licensing

Do you think using the Licensing tab for checking what features are licensed on each host kind of frustrating?

Welcome aboard!!!

I created a powerCLI script which will dump ESX 3.5 Licensing usage into a csv file, so you can import into Excel and play with it.

Here it’s:

$path = ".\ResultESX.txt"

$ESXHosts = Get-VMHost |Get-View

$ServiceInstance = Get-View ServiceInstance
$LicenseMan = Get-View $ServiceInstance.Content.LicenseManager
$Query = $LicenseMan.QueryLicenseUsage
foreach ($ESXHost in $ESXHosts)
$LicUse = $LicenseMan.QueryLicenseUsage($ESXHost.MoRef)
Write-Host -ForegroundColor Yellow $ESXHost.Name
Write-Host "License Server: $($LicUse.Source.LicenseServer)"
$resultWrite = $ESXHost.Name + ";" + "License Server: $($LicUse.Source.LicenseServer);"
foreach ($Reservation in ($LicUse.ReservationInfo |Sort Key))
If ($Reservation.State-eq "licensed")
switch ($Reservation.key)
"esxFull" {$FriendlyName = "ESX Server Standard"; break}
"esxExpress" {$FriendlyName = "ESX Server Foundation"; break}
"backup" {$FriendlyName = "VMware Consolidated Backup Usage"; break}
"vmotion" {$FriendlyName = "VMotion"; break}
"drs" {$FriendlyName = "VMware DRS"; break}
"das" {$FriendlyName = "VMware HA"; break}
"esxHost" {$FriendlyName = "VirtualCenter Agent for ESX Server"; break}
"nas" {$FriendlyName = "NAS Usage"; break}
"iscsi" {$FriendlyName = "ISCSI Usage"; break}
"san" {$FriendlyName = "SAN Usage"; break}
"vsmp" {$FriendlyName = "Up to 4-way SMP"; break}
default {$Friendlyname = "Feature not yet in script";break}
$resultWrite = $resultWrite + ";$FriendlyName licensed for $($Reservation.Required) Processors"
Add-Content $path $resultWrite

If you have no idea how to start, take a look at PowerCLI Basics

Thursday, April 14, 2011

Pegasus service failing

Recently I figured out that one of my ESX hosts were taking a long time to reboot.
Watching the POST I realized it was taking a long time trying to start Pegasus service, after a few minutes it fails and the remaining services start as usual.

It made me wondering why it was failing.

Pegasus is an open implementation of CIM, which monitors the hardware healthy and presents this information on the “Hardware Healthy” tab inside Virtual Center.
Learn more about Pegasus...

VMware has first identified this issue on KB1004257, but it specifies as an issue for ESX 3.5 update 1 and fixed with update 2. But my server is at update 5 !!!!

I found an article on Yellow Bricks blog about how to solve it. Take a look. (I did not use it to solve my issue)

What did I do to solve my issue ?

When looking at /var/pegasus/vmware/install_queue I found a few folders and files in there.
Those are the list of new MOF files waiting to be compiled; I can just guess why they are sitting in there. May be the first MOF file fails and the remaining ones does not get processed.
So, at every new patch/update I applied, the new MOF files are not getting compiled and queue get’s increasing.

To get all the files compiled I ran.

/etc/init.d/pegasus restart (sometimes I needed to run it twice.)

After each file got compiled, the pegasus service was back starting fine and fast.

I think you should take a look at the content of your folders (/var/pegasus/vmware/install_queue), right ; )

Thursday, April 7, 2011

Vmware View for iPad

Last post
we talked about the new Vmware vSphere client for iPad, but Vmware still developing tools to iPad, this time Vmware View is the tool.
Is not that exciting ?!?

I can see all new technology converting into a single solution, virtualization, VDI solutions where your desktop is virtual and why not on the cloud, now we don’t need to have a “regular’ pc to access it. You can do it with your iPad.

You can just go to iTunes and download it FOR FREE. Oh man , I love free stuff ; )

Here you can find all the details about it.

Thursday, March 31, 2011

Vmware vSphere Client for iPad

Last year I wrote about a vSphere client App for iPad which vmware was developing.

Well, it’s finally available and you can download from free form Apple app store, iTunes.
That’s it, for free !!! is not that cool ?

With the iPad fever and the new iPad 2 it’s a tool that you should not miss.

This App will cover the majority of common tasks you might want to perform on your environment, but it does not cover all single feature of the full client and that’s by design.

If you have iPad, lucky you, go test it and let me know, until I don’t have one my own.

Here’s VMWARE’s web site about it, where you can find more details about the configuration….. yeah you need to configure your environment with vCMA first.

Wednesday, March 23, 2011

ESX Scripted Install – part 4/4

Finally we reached the last part of this series.

Let’s review what we learned so far.

1 – How to get started
2 – Working with %pre section
3 – Working with %post section

Now it’s time to wrap up your installation.

I believe by now you have finished you installation script, if you do not rename the ks.cfg file so far that’s your opportunity and have it called something more intuitive.

Then you need to incorporate that file within ESX installation along with the custom files and RPM you want to be available (if you have one, check %post section).

But, there’s a trick in here, be aware when changing an ISO file, the hash check sum will be changed and the installation will failure. Use a program that can change a ISO file and keep the hash check sum. I use MagicISO to change ISO files.

Now you can burn your CD/DVD.

Insert the CD into the server and power on.
The Installation Choice screen will appear:

Choose the option “ESX Scripted Install using USB ks.cfg” and press F2

The boot option line will show, as bellow

replace just the word usb to cdrom:/inst.cfg

The line must be just like above. Attention, my script is called inst.cfg, replace it by yours.

Press Enter to start the installation.

Easy right ?!?!

Fell free to let a comment if it works for you.

See you next

Thursday, March 17, 2011

ESX Scripted Install – part 3/4

Last post we worked on how to pass parameters to the installer.

Now it’s time for the customization you can make after the Operational System is installed.

The %post section will carry on all the commands you want to execute after your ESX has been installed, that means the ESX is loaded and all the ESX’s commands are available too you.

The installation sequence is something like:
1 – installer loads
2 - %pre section loads and you input your parameters
3 – installation occurs based on the %pre section information.
4 – installer executes the commands on %post section
5 – Server reboots

Did you realized the power of this section ?!!?!

You can run all the commands you normally do when configuring your host after installing it, like
esxcfg-firewall command to open/close ports
esxcfg-auth command for the authentication settings

Anything you normally do on a host session. The command syntax is just the same.

So, how to use the %post section ?

Start with the following line after all the %pre section commands
%post --interpreter=bash

Then just place the commands, one per line.

But the trick I can show you, is that you can also copy files to the host, may be the sshd_config file, with your already configured SSH options or even coping rpm packages for agents installations.

How does it work ?

- first you will need to create a folder with all the required files and packages you want to use.
- Then burn a CD with the ESX installation and your new files (we will cover this process on the next post)

Now that your CD is ready with the files you need, let’s configure the %post section

First you will need to mount the cd-rom on the system
mount /dev/cdrom /mnt/cdrom

them you can just copy the files from the cd to the system, like
cp /mnt/cdrom/Custom/sshd_config /etc/ssh/sshd_config

or installing the RPMs on your system, like
rpm –ivh /mnt/cdrom/Custom/agent.rpm

Here’s just a sample of a script to help you start your own


%post --interpreter=bash

#mount CDROM
mount /dev/cdrom /mnt/cdrom

#enabling ssh on firewall
esxcfg-firewall -e sshClient

#configuring ssh
mv /etc/ssh/sshd_config /etc/ssh/sshd_config.old
cp /mnt/cdrom/Custom/sshd_config /etc/ssh/sshd_config
chmod 600 /etc/ssh/sshd_config

#unmont cdrom
umount /mnt/cdrom


I bet your are anxious to get it working ; )

Friday, March 11, 2011

ESX Scripted Install – part 2/4

Now, that we learned how to get started with ESX Scripted Installation.
It’s time to starting customizing your script.

I would not go into details about every setting on the script because you can check them on ESX Installation Guide.

But, one thing I wanted to pointed it out is that I needed to make one adjustment on the disk section to be able to install it on next hosts.

I commented the line with specifies what disk to use and uncomment the line to use the first disk detected.
See example bellow:

#part '/boot' --fstype=ext3 --size=1100 --ondisk=mpx.vmhba0:C0:T0:L0
# Uncomment to use first detected disk:
part '/boot' --fstype=ext3 --size=1100 --onfirstdisk
#part 'none' --fstype=vmkcore --size=110 --ondisk=mpx.vmhba0:C0:T0:L0
# Uncomment to use first detected disk:
part 'none' --fstype=vmkcore --size=110 --onfirstdisk

But what now ?!?!

You probably don’t want to install all your hosts with the same name and same IP configuration, right ?

You can find several blogs to teach you how to use input files or websites with required information, but I personal believe, just and input prompt during install would works better.

There’s a special entrance called %pre, where you can specify everything you want before start your installation. So, how to get it working ?

After the last line on the script you add the %pre section. See my example bellow with comments:

Now…how to get the installation read my /tmp/networking file with the information I just typed in ?!?!

Look for the network section above, should be near the timezone section, comment it (add # at the start of the line)

Then add the line bellow:
%include /tmp/networkconfig
It’s done!!
The installation will starts, jump to the %pre section where you can enter the information to be used by the installer.

Now that you learned how to use the %pre section you can use the same concept to gather all kind of information you want , like time zone, license, keyboard, etc..

See you next.

Thursday, March 3, 2011

ESX Scripted Install – part 1/4

Eventually you reach a point when you want/need your ESX installations to be consistent across your environment.
That’s when scripted installations get on board.

It would help you saving some man hour costs solving software conflicts and installation issues just creating a more in-depth automation process for repeatability and predictability installation results

But how to get started ?

If you search the web, you will find several posts and lot of fancy scripts, but you are just a rookie on this new world and probably these result will just confuse you even more.

You probably can start with ESX Installation Guide, there’s a section just for scripted installation. But you will realize there are some pitfalls you would need to overcome.

So in this series of 4 posts I will show you how I conquered this task and how can you too.

The easiest way to start is doing a normal ESX installation through the ESX Graphical mode.
That’s your best opportunity to set up your disk layout , root password , time zone, etc…

Once your installation is done, there’ll be a file called ks.cfg at the root folder of your system,
This file will be the bases of your scripted installation. Copy it to a place where you can store and make changes. (do not use Windows text editor to make changes on it)

When editing ks.cfg then you will see all the choices you did during the installation:
As root password (remember it will be encrypted on the file).

Also you can see all the sections and compare with the ESX Installation Guide to understand what they do and how to use them.

During the next post we will cover more about customizing it.

On this journey together you will see that we can start simple and make just the changes we want for our environment

ESX Scripted Install – part 2/4 coming soon on a laptop near you ; )

Friday, February 25, 2011

Building custom PowerCli scripts to manager vmware environment

Now that you learned the basics of PowerCli you might be realized the power of this tool and the thousands of changes you can make on your environment with it.

But, if you want to accomplish some specific task and have no idea how to start it. Would not be nice to have Virtual Center telling you the scripts/command required to perform that change?

Well, it’s possible with Onyx.

Onyx is a standalone application that serves as a proxy between the vSphere Client and the vCenter Server. It monitors the network communication between them and translates it into an executable PowerShell code. Later, this code could be modified and saved into a reusable function or script.

Thursday, February 17, 2011

Finally it arrived.

Yesterday the post office delivery my copy of Vmware Vsphere HA and DRS Technical deepdive book.

Want to know more about this book…check it here
Why don’t get your own ? I’m sure it worth every penny …buy in here

The history behind it is that a few weeks ago I joined Veeam on a webcast called 5 New ways to use Sure Backup (just click on the link to watch the replay).

At the end of the webcast there was a contest where the firsts correct answers would win this book. Guess what ?!?! I was one of them.

I’d like to thanks Rick Vanover not just by the book or even the opportunity to learn more about their tool, but also for showing care and doing constantly follow up’s to be sure the book has been delivery properly. It’s definitely something we don’t see a lot over there.
If you have a Twitter account (not my case) why not follow him on @RickVanover

Friday, February 11, 2011

Update Manager not downloading ESX 3.5 patches

Part of building a new system is to make sure the new one works, at least, as the old one.

When working on my new Update Manager I configured the download scheduler settings, notification, baselines, etc…

But one thing was not working fine…. There was any patches for ESX 3.5.
Despite the fact the download task just ends very quickly and with a successfully message, there was no patch for ESX 3.5 on my repository, but I do have patches for ESX 4.x.

Well, I started with the basics, checking firewall, internet access, NTFS privileges on the folders, database availability….everything was good.

So after hours of troubleshooting and searching the web I finally found vmware KB 1008308.

It means that if you don’t have an ESX 3.5 attached to your Virtual Center it would not download the patches for it. I assume it’s always true for ESX 4.x patches.
It should be an issue for the majority of fresh environments where no host will be attached until the system is made fully operational.

Then I just added one ESX 3.5 to my Virtual Center and run again the download task…this time the patches for ESX 3.5 were downloaded to my repository.

What just confuses me is that you have the option to select what patch we want to download.

But the true is regardless the option you choose on “Patch Download Settings” it will just download the patches for the systems you already have attached to your Virtual Center.

Weird, right…

Thursday, February 3, 2011

Thin Provisioning Performance

I never know how to start a post : (
Let’s make it simple.

Hi !!!

I believe everybody on the VMware world is aware of Thin Provisioning, right…. It has been out there for a long time as one of the new features of vSphere.

What ?!? You did not have time to check it out. I know the felling.

Thin Provisioning in VMware’s word is: “VMware thin provisioning enables virtual machines to utilize storage space on an as-needed basis, further increasing utilization of storage for virtual environments. vCenter Server 4.0 enables alerts and provides alarms and reports that specifically track allocation and current usage of storage capacity to allow administrators to optimize the allocation of storage for virtual environments. Thin provisioning allows users to safely optimize available storage space by using over-allocation and to reduce the storage costs for virtual environments”

It has been introduced (officially) with vSphere 4.0. You can check all new vSphere 4.0 features in here.
But, if you are a smart guy, you are planning to jump directly to vSphere 4.1, which has a couple of new features…see here.

Ok, now that you understand Thin Provisioning you may be wondering, is there any performance impact or benefit ?!?

I don’t want to reinvent the wheel, so check it out VMware’s performance study of Thin Provisioning. There’s a bunch of good information in there, concepts, performance rates, etc.

Wanna a resume ? Performance results between thin and thick disks are similar.
But remember it’s a controlled environment, make your own tests and measure them.

There’s no single solution that fit’s all.

Friday, January 28, 2011

Virtual Center Custom Alarms

Monitoring the health of your environment is a crucial step to maintain a good level or service and availability.
I’m sure all of you have realized the default alarms Vcenter generates for your hosts and clusters, but I’m not convinced all of you understand the power and capacity of the Virtual Center’s alarms.

VMWARE provides several triggers that you can customize to monitor several components of you environment, like, Hosts, Clusters, Networks, Datastores..

Also it can sends notification traps, e-mails and run commands. BTW running commands based on specific alerts is something amazing, you just need a little imagination.

For example, let’s suppose one of you hosts have lost half of it’s network capability, depending on the workload the guests running on this host could suffer from a network delay.
But if you have a custom alert based on this issue, it could run a PowerCli script to set your host to maintenance mode (which would v-motion the guests to another hosts) and send you and e-mail so you could take care of the issue.

Nice right ?!?!

Bellow is a cool video about a custom Datastore alarming, I’m sure it will open your mind for the VMWARE’s alarms

Wednesday, January 19, 2011

VMWARE Log Control 2/2

Last post I talked about ESX’s log retention and how to configure it using logrotate.conf
Today I want to mention that you can also control the log of your guests.

I’m not talking about the guest’s log location, which is by default on the same directory of your guest’s configuration (vmx file). You can change that when you add a host to Virtual Center, lets say you want to consolidate all log’s on the same location (LUN), but this post is not about it.
It’s about having the control of you guest’s log, just like on the host you can control the maximum size of your log and the rotation of them.

Let’s see how it works.

- First you need to power off your guest
- Right click your guest and choose “Edit Settings”
- Go to the “options” tab and under Advanced / General settings
- Click on the button “Configuration Parameters”

There will be a list of configuration parameters, if the ones you need are not listed just click “Add Row” and add them.

- On the new row, click under the Name column and specify the configuration option
- On the new row, click under the Value column and specify the configuration value

log.rotateSize is the maximum size allowed for the log file before a new one gets created. (it’s kilobytes, so 1000000 means 100kb)
log.keepOld is the amount of logs you will have before you start rewriting the olds one.

So on my example I will have a maximum of 10 logs of 100kb each.

When you are done, just power on the guest again.

Also, you can do change the vmx file directly.
Just add the following lines to the end of file
log.rotateSize = "100000"
log.keepOld = "10"

On more tip before we wrap up.
By default guest’s log name is vmware.log, if you want to change it’s name just add on more Row called log.fileName and then the name you wish for the log file.

Easy , right ?!?!?
See you

Wednesday, January 12, 2011

VMWARE Log Control 1/2

Today let’s talk about the log control of your VMWARE system.

I believe most of you must have some kind of policy or regulatory law to be compliant with.
In most cases it involves to keep a history of log files.
Also it’s a good practice to have control of your log files to avoid them to fill up your system volume with data.

Well, on ESX it’s controlled by logrotation.conf found at /etc/

Bellow you can see an example of my logrotation.conf

Log rotate is a powerfull resource and there’s a lot of options you can set up.
I just gave you the basics. If you want to read more about it just type: man logrotate on you host’s console

Before change you log configuration make sure you stop syslog to avoid any issue:

Let’s see how implement that

Log on your host’s console
Run: /etc/init.d/syslog stop
Run: vi /etc/logrotate.conf
Make the changes you want, save and close the file
Run: /etc/init.d/syslog start

Next post I will show how to control the log of your guests. Don’t miss that

Who am I

My photo
I’m an IT specialist with over 15 years of experience, working from IT infrastructure to management products, troubleshooting and project management skills from medium to large environments. Nowadays I'm working for VMware as a Consulting Architect, helping customers to embrace the Cloud Era and make them successfully on their journey. Despite the fact I'm a VMware employee these postings reflect my own opinion and do not represents VMware's position, strategies or opinions. Reach me at @dumeirell

Most Viewed Posts

Blog Archive