Tuesday, July 24, 2018

ESXi host upgrade failed - 0x8b in position 513


Most of my time as a Consulting Architect at VMware Professional Services I spend with clients, helping them to create innovative solutions, overcoming challenges, etc.
Since every environment is unique, sometimes I stumble to some weird situations, this past week was one of them.

The client was upgrading their ESXi hosts from version 6.0 to 6.5, while the majority of the hosts went smoothly, a couple of them presented some undesired behavior.

Update Manager was used to remediate the hosts, everything was going fine, the patches have been staged and the first reboot occurred as expected, but during the installation, it crashed with a blue screen and an error message:

*******************
An expected error occurred
See logs for details
UnicoDecodeError: ‘utf-8’ codec can’t decode byte 0x8b in position 513: invalid start byte
*******************

(I'm sorry about the image quality, I was in a hurry trying to figure it out)

And then the installation rollback automatically to ESXi 6.0
Surprisingly all hosts were the same model, installed at the same period, the same way with the same ISO, so there’s nothing special about those hosts we could think off.

After some basic troubleshooting nothing pops up and an internet search for this error did not return anything relevant.
Time to search internally, VOILA ….that’s when I found a couple of past cases with the same behavior.

Long story short, the altbookbank for some reason was corrupted, we never found out why.
The solution was to recreate the altbookbank from the bootbank partition.
First, we got rid of the content in /altbootbank and then we copied the content from /bootbank to it.

Wait a minute, what /altbootbank and /bootbank is all about ?

ESXi keeps two independents copies of its boot partition, bootbank and altbootbank. One of them will have the active image, bootbank, which is used to boot up the system and the other one will have an alternate image, altbootbank, you can imagine that as the last good known state, so in case your boot partition becomes corrupted you can reboot your host from the last good know state (altbootbank).

It really took me a while to figured out how to solve it. I’m publishing it hoping it can save some of your time too, just let me know if you faced this issue too.


Tuesday, July 17, 2018

vSphere Integrated Containers – Affinity rules


Managing a vSphere environment is not about making use of the technology by itself, in fact, it’s leveraging this technology to fulfill business needs what it really matters. Often vSphere Administrators utilize DRS affinity rules to control virtual machines placement specifying a group of hosts which might fulfill those needs, reasons vary from license constraints, specific hardware needs, increase availability etc.

With the advent of vSphere Integrated Containers, VIC, developers can instantiate their own containers, container-vms to be more precise, without the interference of a vSphere Admin, while it increases the agility of the business it also places a new challenge; as containers-vms come and go as need how admins can keep their affinity rules updated in order to fulfill the business need ??? For sure a manual intervention is not up for debate.

Luckily VIC 1.4 brought a new functionality, host affinity. When enabled, there will be a DRS VM Group for each Virtual Container Host, VCH, and as containers are created or deleted this group will be updated accordingly, helping administrators and developers life to adhere to those business need automatically.

During the creation of a VCH, you enable host affinity just specifying ”--affinity-vm-group” option on the vic-machine command line (not yet available on VCH wizard).

A new DRS VM Group will be created with the same name of the VCH, you will also notice VCH VM is part of this group, it’s made that way because it’s impossible to create an empty VM Group, while an empty group can exist as a result of removing all VMs from it.

But what about new existing VCHs ???
Starting with VIC 1.4.1 you can reconfigure them enabling host affinity as well.
 
After creating the VCH, vSphere Administrators just need to create a VM-Host Affinity rule that matches this newly created VM Group and a Host Group, before handling the VCH for the developers.

So every time a developer creates or deletes containers on the VCH, the VM group membership will be updated accordingly and DRS will take care of the scheduling container-vm based on the rule create before automatically.
Enabling higher agility and improving operational efficiency while keeping the business need into account.

If you are still not sure why would you use this feature, I’d like to expose a few use cases;
On a hypothetical scenario of a single vSphere cluster made of 10 hosts distributed on 2 physical racks you may have;

*** Licensing needs ***
Let’s imagine you have an application that is licensed per physical host or processor, to decrease your license cost you might create a host group containing just the hosts you have this application license for and match this group with an affinity rule for the VCH VM Group, this way you don’t need to license your entire cluster;



*** Specific Hardware needs ***
Now if your container benefits from a graphical intensive processor, GPU, you can create a host group containing those hosts and match this group with the VCH VM Group, now those intensive containers will always be scheduled on the right host;



*** Fault Domains ***
Increasing your fault domain is always a plus when it comes to availability.
You can use a Host Group to create a kind of virtual cluster inside your vSphere cluster where the members of this host are spread among the racks. While you cannot guarantee your application will always be spread evenly between racks, HA will restart your container-vms on the remaining hosts in case of rack failure.



But, if you wanna make sure your application will always be spread evenly between racks you can create two VCHs, where each one will have an affinity rule to a Host Group based on hosts of a single rack.
Like VCH01 will use hosts from rack-A and VCH02 will use hosts from rack-B, now you can control the placement of your containers assuring that your application will always be available in case of a rack failure.


As you can see there's so many use cases for this feature but even more important is to support the agility you need while still aligned with your business need.

Do you have a different use case for this feature ? let us know...

Tuesday, June 5, 2018

vSphere Integrated Containers 1.4 - Upgrade 3/3


Finally, we get to the last step on this journey of upgrading my environment to the latest and greatest vSphere Integration Containers so far.

Today I'll upgrade my Virtual Container Host, VCH, to version 1.4

VCH management and lifecycle actions, like upgrades, are performed through the use of VIC Engine Bundle.
If you haven't that available yet or is using an older version, grab it now.
Engine Bundle can be found on the VIC Getting Start page (https://VIC:9443)

Unpack the binaries from Engine bundle tar file;
Run: tar -zxf vic_1.4.0.tar.gz

Check the VCH version on your environment;
Run: vic-machine ls
As you can see it’s running version 1.3 and has one container running.

The amazing thing about VIC is that interruptions, like upgrades, to VCH, does not cause any outage to the containers, basically because they are running independently as container-vms;
if you are using NAT based port forwarding then communication will be briefly interrupted but if you are using the exclusive VIC feature, Container Network, you are in good shape them.

Now that we know the ID of our VCH we can upgrade it.
Execute the vic-machine upgrade command and specify the VCH's ID we just got from the previous step.
Run: vic-machine upgrade --id "VCH_ID"

In less than 3 minutes my VCH has been upgraded to version 1.4 and as you can see my container kept running for the entire process.


WHAT ABOUT MY CONTAINERS ???

There’s no process to upgrade your running containers !!!
Containers are ephemeral by nature, so if you want a newer version, delete it and create a new one. Welcome to the container world !!!!

VIC containers are based on Photon OS, and with VIC 1.4 comes a new bootstrap.iso version; don't get too excited, it was not this time it got upgraded to version 2.0, but it got some nice minor packages updates.
So any previous container will still be running the old Photon OS version but the new ones will get the new bootstrap.iso version.

For comparison, I just run a new container based on the same image and compared the OS versions.


That’s all folks there are no more excuses to be running an older VIC version if you still unsure about why to upgrade, take a look at the complete list of what's new.

See you
 

Wednesday, May 30, 2018

vSphere Integrated Containers 1.4 - Upgrade 2/3


Hello there, following my series of upgrading vSphere Integration Containers to version 1.4, I'll cover today how to upgrade the vSphere Web Client Plug-in.

Let’s start with some housekeeping:
I’m considering you are running vCenter Server Appliance right, who is running the Windows version anyway ?!?!?
  • You already upgraded your VIC appliance to version 1.4;
  • VIC plugin 1.2.x or 1.3.x is already installed on vCenter;
  • The bash shell is enabled on VCSA; (just check it on vCenter's VAMI page)
 
Copy the VIC Engine Bundle file from the new appliance to VCSA;
Engine Bundle can be found on the VIC Getting Start page (https://VIC:9443)

Unpack the binaries from Engine bundle;
Run: tar -zxf vic_1.4.0.tar.gz

Execute the upgrade script;
Run: ../vic/ui/VCSA/upgrade.sh

Once on the VCSA bash shell, set up some environments variables;
Run: export VIC_ADDRESS="VIC_v1.4_IP"
Run: export VIC_BUNDLE="vic_engine_bundle_version"




Provide the vCenter name, the user with privileges to register plugins, If the plugin version you want to upgrade is correct, just hit yes to proceed


The log is provided on the screen during the process;

If everything ran as expected, just restart the web client services for the new version takes place.

Run: service-control --stop vsphere-ui
Run: service-control --stop vsphere-client
Run: service-control --start vsphere-ui
Run: service-control --start vsphere-client

When logging back to vCenter we see the plug-in has been upgraded


Unfortunately, this new plug-in has no new features, but VCH Creation Wizard has gone through some design improvements, collapsing and sorting some information in order to make the deployment more intuitive and easier.


Now we are missing the last piece of it, upgrading Virtual Container Hosts....keep watching.


Who am I

My photo
I’m an IT specialist with over 15 years of experience, working from IT infrastructure to management products, troubleshooting and project management skills from medium to large environments. Nowadays I'm working for VMware as a Consulting Architect, helping customers to embrace the Cloud Era and make them successfully on their journey. Despite the fact I'm a VMware employee these postings reflect my own opinion and do not represents VMware's position, strategies or opinions. Reach me at @dumeirell

Most Viewed Posts

Blog Archive