Friday, November 17, 2017

vSphere Integrated Containers – additional Docker commands

vSphere Integrated Containers brings an enterprise container runtime into vSphere environments, where developers, who are familiar to Docker, can run their containerized applications transparently without even noticing it’s running on a nontraditional Docker host.

In order to provide this seamless experience, VIC must provide a completely parity of supported commands with Docker. VMware has been putting a lot of efforts on each release to add as much commands as possible, making developers life easier and easier.
Along with a bunch of new features and fixes, VIC 1.2 also made a step closer to the 100% parity with Docker adding support for the following most used commands;

- docker cp
description: Copy files to and from containers
syntax: docker cp "src_file" container_ID:"dst_path"
- docker exec 
description: run commands on a running container
syntax: docker exec container_ID [command]

- docker commit
description: Commit changes to a container into a new image
syntax: docker commit container_ID new_image:tag

- docker diff
description: Inspect for differences on the containers’ file system since it’s creation
syntax: docker diff container_ID
A = file added
C = file changed
D = file deleted

- docker stats (not really new, but on 1.2 we added support for network statistics)
description: Shows container’s statistics
syntax: docker stats container_ID

For a full list of supported Docker command on VIC check the documentation.

Tuesday, November 7, 2017

vSphere Integrated Containers - Custom Certificates

vSphere Integrated Containers (VIC) leverages TLS (transport layer security) to ensure communication security between client and Virtual Container Hosts (VCH) providing not only traffic encryption but also guaranteeing the identity of the certificate and it’s creator.

In a previous post I already talked about protecting your VCHs with TLS, but it lacks a process on how to generate your own Certificate Signing Request (CSR), which can be used to request a valid certificate from an internal or public Certificate Authority (CA), keep in mind it’s not intended to be a definitive guide, certificate is a wider subject with a lot of options that can be leveraged, like encryption methods, key size, certificate requirements, etc.. with that said, use at your own risk and make sure to test it before going into production.

I will use Openssl tool to generate an RSA Private Key and CSR, if you don’t have it yet download and install it first, it has a version available for each of the popular O.S.

For the purpose of this example, my VCH will be named vch01, defining a name for your VCH is  important for the certificate specifications like common name, make sure you adjust it accordingly  when creating your owns.

The certificate requirements for VCH are: 
  • an X.509 certificate
  • KeyEncipherment 
  • DigitalSignature 
  • KeyAgreement 
  • ServerAuth
Let’s start creating a directory where we will store our files
Run: mkdir /tmp/vch01

To make things easier I created a configuration file where I added all the certificate settings, it could also be used as a template for future requests
Run: vi /tmp/vch01/vch01.cnf

 Add the fields as the screen bellow, updating the fields related to your environment
Now let’s create our RSA Private Key, this key will be used to signed the CSR
Run: openssl genrsa -des3 -out vch01.key 2048 pkcs8
The options on this command are:
  • des3 is the cryptographic method
  • 2048 is the size of private key 
  • pcks8 is the standard syntax for storing information
VIC does not support PCKS#1, so if you have a key like that convert it before proceed.

To make sure the key is OK
Run: openssl rsa -in /tmp/vch01/vch01.key -check

By default the key is encrypted and some applications does not support it, such as VIC, so to proceed we need to remove the passphrase from it to avoid issues when creating your VCHs
Run: openssl rsa -in vch01.key -out newvch01.key

Now that we have our private key we can generate the CSR.
Run: openssl req -new -key newvch01.key -out vch01.csr -config vch01.cnf

Since we are using a config file to provide the certificate information, nothing will be prompted but the key passphrase
The syntax on this command contains:

  •  -key; which specified what key to use to sign the CSR;
  • -out; csr output file;
  • -config; the configuration file with certificate details;

Before sending the CSR to your CA check if it contains the information you need
Run: openssl req -in vch01.csr -noout -text

If everything is fine you can send it to your CA in order to generated a signed certificate.
Once you get your signed certificate back is just a matter of creating your VCH with the certificate as I stated on Protecting your VCH post.

At this point the process of generating your CSR is done,  but if either you don’t plan on having your certificate signed by a CA or wish to test your new implementation while the CA is signing your certificate what you need then is to self sign your own certificate.

Self-signed certificate can also be done with openssl, you only need a RSA key and a CSR, which we just created so we are good to go.

To generate a self-signed certificate which is good for 365 days, issue the following command
Run: openssl x509 -req -days 365 -in vch01.csr -signkey vch01.key -out vch01.pem

That's it, you now have your self signed certificate to create your VCHs.

  • The certificate process wen fine but during the VCH creation you got the following error:
ERROR Failed to parse certificate: tls: failed to parse private key
ERROR Unable to load certificates: tls: failed to parse private key
ERROR --------------------
ERROR vic-machine-linux create failed: tls: failed to parse private key

It’s just because your key is encrypted, you need to remove the passphrase from the key, check the steps above on how to remove passphrases from your key.

  • Once your VCH is created with a self-signed, the CA cert will be the same as your server certificate, just create a copy of it and name it ca.pem, otherwise when you try to communicate with your host you will got the certificate signed by unkown authority error message
x509: certificate signed by unknown authority 

That's all folks.....

Monday, October 30, 2017

vSphere Integrated Containers – Application Repackaging

When talking about containers it’s inevitable to think about modern application built natively for the cloud, but the reality for most companies is that the majority of, if not all, their applications still traditional applications and probably they will still that way for many years to come.

Does that mean you cannot join the containers party ? ABSOLUTELY NOT !!!

One of vSphere Integrated Containers use cases is target specific for you, Application Repackaging;
What it means is that you can take your traditional application, monolithic or even a 3-tier application and repackage them as containers, without having to change a single line of code.
You don’t even need to change all layers of your application, start simple. Containerize only the web/presentation layer and give it a try.

You will immediately see the benefits of running applications within containers, like:

  • Faster Recovery: in case of disasters, you don’t need to recreate your VMs, patch them, install app, configure etc.. just instantiate a new image in a matter of seconds and you are back in business;
  • Easy scale-out: adding a new node to an application is just as simple as running a new container, just select the same image as a base source and it’s done;
  • Faster life-cycles: upgrading a container with a new application is much simpler and faster, you can upgrade your application in minutes;
  • Portability: container images are becoming the de-facto application delivery method, just because it runs anywhere, desktops, vSphere Infrastructure and to the cloud without any modification what so ever.

And it’s not all, with vSphere Integrated Containers every container is a VM, so you also get all vSphere benefits:

  • Increased availability: leveraging High Availability (HA), in case of hardware failures, your containers will be restarted automatically on another healthy host, reducing your downtime and increasing your application availability
  • Predicted performance: Distributed Resource Scheduler (DRS), will make sure your container gets the performance it needs, moving the VMs/containers around to load balance the workload among the hosts of your cluster, ensuring the ESXi host has the free resource it needs to fulfill your containers needs.

If you are as excited as I am with all these possibilities, watch the demo video to see it in action.


Thursday, October 19, 2017

vSphere Integrated Containers – working with variables

If you play long enough with vSphere Integrated Containers (VIC), sooner you will realize how long and tedious are the vic-machine commands.

Do you know you can make use of VIC variables to simplify those day to day activities?!?

Taking as an example the simple task of listing all your created VCHs, you would need a command like that:

vic-machine ls --target vcenter.vsphere.local --user administrator@vsphere.local --password Secure123 --thumbprint 2F:C3:3C:5D:99:B6:31:87:77:58:4D:8F:2F:75:D9:0C:01:F8:FE:6B

As I said….loooong !!!!
What if I tell you can just run: vic-machine ls instead
Much simpler, right ?!?!

Well, that’s all possible setting up the VIC variables, that will store your values and use them on future commands; these are the ones:

  • VIC_MACHINE_TARGET: that’s the variable that tells which is your target vCenter to use;  
  • VIC_MACHINE_USER: the username with privileges to run commands against your target vCenter;   
  • VIC_MACHINE_PASSWORD: the password for the username you just specified with VIC_MACHINE_USER;
  • THUMBPRINT: the variable that contains the thumbprint of your target vCenter;

You just need to set your environment variables to make it works.
Depending on your operations system the command to set environment variables are different;
For Linux:       export “variable”=”value”
For Windows: set “variable”=”value”

Now, let’s see how our example will looks like:

Run: export VIC_MACHINE_TARGET=vcenter.vsphere.local
Run: export VIC_MACHINE_PASSWORD=Secure123
Run: Export VIC_MACHINE_THUMBPRINT=2F:C3:3C:5D:99:B6:31:87:77:58:4D:8F:2F:75:D9:0C:01:F8:FE:6B
Run: vic-machine ls

Now you can just run vic-machine commands like ls, create, delete, configure and others without having to provide the same information over and over again.

Since we are talking about making your life easier, do you know Docker has it’s own variables too?

To check the information of your docker host you would run a command like that:
docker –H --tls --tlscacert /home/vch01/ca.pem --tlscert /home/vch01/cert.pem --tlskey /home/vch01/key.pem info

Would it not be better just run: docker info

Yes you can, it works the same way, setting up DOCKER variables, these are the most common ones;

  • DOCKER_HOST: set your target docker host or VCH in our case; 
  • DOCKER_TLS_VERIFY: set your docker client to check for TLS, set to 1 to enable verification; 
  • DOCKER_CERT_PATH: the path where docker host certificate and keys could be found, keep all certificates for the same docker host on the same folder, that makes things easier down the road;   
  • DOCKER_CONTENT_TRUST: when set to 1, it enables content trust;  
  • DOCKER_CONTENT_TRUST_SERVER: set your notary server for content trust verification and signature;
 Give it a shot, I bet you will feel more productive and less tedious.

See you next.

Monday, October 9, 2017

vSphere Integrated Containers – Name Convention

During the past several years, infrastructure Administrators came up with different methods to organize its vCenters, some with folders structures, some with fancy VMs nomenclature and prefixes.

When we think about vSphere Integrated Containers (VIC) consumption model, containers as Virtual Machines (containerVMs), we realize it kinds of disrupted this organizational model, mainly because VM's creation is no longer under administrators control,  but also because developers are now creating and deleting their own containerVMs inside vCenter without even noticing the existence of it, breaking the standards and controls applied so far by this new dynamic nature.

When a container is created by a developer within VIC, it’s respective containerVM, by default, is created in vCenter using the nomenclature of CONTAINER NAME + CONTAINER ID

For an infrastructure administrator, these VMs names are not very helpful, especially if you have controls and policies in place which depend on those VMs name.
Tools like vRealize Operations, VMware NSX or chargeback tools, might be ineffective when trying to manage and control those containerVMs.

To address these challenges, VIC 1.2.1 introduces the --container-name-convention option, which allows you to specify a prefix that will be applied to every containerVM during its creation, giving back to the administrator the control they require.

This convention option is determined during Virtual Container Host (VCH) creation. When creating your VCH there are many options available depending on your environment and your needs, I’m just focusing on --container-name-convention today, but if you are interested, check this link for the full options.

First, let’s check the prefix option based on the container name.
--container-name-convention “prefix”-{name}

This option enforces that each containerVM name will be made of the prefix you specify (in this example DEV) + the CONTAINER NAME

The second option is the prefix based on the container ID. 
--container-name-convention “prefix”-{id}

This option enforces that each containerVM name will be made of the prefix you specify (in this example PROD) + the CONTAINER ID

As you can see, it’s a win-win situation; from a developer’s point of view, nothing changes, they can still see it’s containers name and IDs just before, from a vSphere Infrastructure point of view it can create a standard that will be honored during containerVMs creation.

Monday, October 2, 2017

vSphere Integrated Containers – Developer Workflow

There’s no doubt vSphere Integrated Containers (VIC), is getting more and more powerful at each version, just check the several enhancements and capabilities which were released with version 1.2 and you will see what I'm talking about.

Today, I wanna walk you through a developer point of view workflow where we can create an image, storage it on a registry and then run it in production, with just 6 easy steps, all within VIC 1.2.

note: I already deployed a Virtual Container Host (VCH),  I'm leveraging Harbor, which is also part of VIC as my registry. If you think you need some help with those steps, let me know and I'll write another post explaining how I did it.

Let's do it;

Step 1 – run my Docker Container Host
Since VIC engine does not support (yet), docker build and docker push I will make use of a Docker Container Host (DCH), a native Docker host leveraged directly from VCH.

With the use of port mapping, DCH's services will be available through my VCH on port 12375.

docker run -d -p 12375:2375 vmware/dch-photon:1.13

Wait until it pulls the image from Docker Hub and starts it.
Step 2 – Set my docker client to the newly DCH
From this point on, I want all my commands to run against the recently created DCH, pointing my docker client to it. not really a required step but it makes things easier.


Step 3 – Build my image
It’s time to build my new image.
I’m using a simple Dockerfile, which uses nginx:latest image and update it. 
But you can be as much creative as Docker permits, remember; DCH is 100% Docker API compatible.

For easier identification, I’m building it and tagging it with a meaningful name. 
You can tag it something else later as well.

docker build -t registry.corp.local/justait/nginx:patched . 

it will take some time to pull the image and apply the updates. don't you worry.

Step 4 – Push to registry
My image is ready!! 
Now I can push it to my registry, everyone will be able to consume it.

docker push registry.corp.local/justait/nginx:patched 

I also pushed the unpatched version of nginx, just in case someone wants to use it too.
As you can see, both images are now stored in my registry under justait project

Step 5 – Set my docker client to my VCH
I want all my production containers to run on VIC, where they can benefit from vSphere features like DRS and HA.
So, let's set my docker client back to VCH,  


Step 6 – Run your container
It’s just a matter of running the container based on the new image.
But I'll not just simply run the container, I want to use this unique VIC feature of exposing the container's service directly on the network (Container Network).

docker run -d --net routable registry.corp.local/justait/nginx:patched

Here it is, from building an image to running a container in production and all we needed was a VCH. 

Writing a blog on your own free time, sometimes leave you without imagination, this topic was a suggestion from one of my readers, if you want to see something here, please, leave a comment or contact me on twitter, I really appreciate those ideas ;)

Monday, September 18, 2017

vSphere Integrated Containers – container network firewall

One of the unique and amazing features of vSphere Integrated Containers, VIC,  is its ability to expose containers services directly on a network, which means the traffic would not need to pass through the container host (port mapping), full network throughput per container and outages at the container host DO NOT cause any outages to the container service itself.
This capability is possible through the use of Container Network option.

On traditional Docker implementation, you could just pass the option -P and all container’s exposed ports will be published, while it’s great, it also raises security concerns about publishing ports and services that you are unaware of and might, potentially, increase your attack surface.

With that in mind, VMware, enhanced the security and control of container services with a new security feature, container network firewall, available starting from VIC 1.2.

This new feature comes with 5 levels of security trust, as follow;

  • Closed: no traffic come in or out of the container interface;
  • Open: all traffic is permitted; it allows the use of option -P during container creation;
  • Outbound: only outbound connections are permitted, good for containers consuming services but not providing any
  •  Published: only connections to published ports are permitted; you need to explicitly tell which port will be permitted during container creation; Ex: docker run -d –p 80 nginx
  • Peers: only containers on the same “peer” interface are permitted to communicate with each other. To establish peers you need to provide a range of IPs to the container network during VCH creation, (--container-network-ip-range)

By default the behavior of container network firewall is Publish, that’s why -P option might suddenly stop working after you upgrade to VIC 1.2.

To control the container firewall behavior you need to specify the trust level during VCH creation:
--container-network “PortGroup”:Internet --container-network-firewall "PortGroup":open

Now you have all the control you need on your container’s services.

Tuesday, September 5, 2017

vSphere Integrated Containers – Protecting VCH 2/2

This is post two of protecting your Virtual Container Host (VCH), if you did not check post one, I really encourage you to check it out before proceeding.

As promised, now I will show how we can secure our VCH leveraging two-way authentication with TLS certificates.

vSphere Integrated Containers (VIC), provides self-signed certificate capability, where, during VCH creation, it creates it's own CA in order to create and sign server and client certificates.
Bear in mind that self-signed certificates provide all the security and encryption required, but they don’t provide aspects such expiration, intermediate certificate authorities and so on.

*** Certificate Base Authentication and Traffic Encryption ***
Unlike the previous methods, now the users MUST provide client certificate in order to authenticate to the VCH endpoint any time they want to issue Docker commands, if you are using a self-signed or untrusted certificate, you also need to provide the CA certificate which signed them. 
Besides authentication, the traffic between client station and VCH are encrypted as well (Docker API service is listening on port 2376).
Being this method the one recommended for Production environments.

You just need to provide --tls-cname “name” option during VCH creation.
This name is the common name that will be added to the certificate and how your users will connect to the endpoint.

VIC will create a folder with the VCH name in the current directory and all certificates will be stored within it. 
Those are the self-signed certificates generated;
ca.pem    -> send to users
cert.pem -> send to users
key.pem  -> send to users

if you want to specify a different location to store the certificates once created, use the option --tls-cert-path “path”

Now if I try to connect to my recently created VCH just pointing its endpoint... think again...

You, as a Cloud Admin, must deliver to the users who will connect to your endpoint the required certificates, cert.pem and key.pem, that were generated during VCH creation, remember to send the ca.pem as well (just in case it’s a self-signed certificate).

The users will then copy the certificates to the client station, personally, I like setting up some variables to tell Docker client that I need to enable TLS check and where my certificates could be found.

export DOCKER_CERT_PATH=”path_to_certificates”

As you can see now, I can securely connect to my VCH endpoint.

You might be asking, OK, but what about the use of custom/trusted certificates ?

YES !!! VIC allows the use of them as well.

Make sure your certificate is:
  • an X.509 certificate
  •  KeyEncipherment 
  • DigitalSignature 
  • KeyAgreement 
  • ServerAuth

*** Leveraging Custom certificates ***
First, make sure you have your valid certificate signed by a trusted CA to a folder where you have access to.

Besides the --tls-cname “name” option, now you need to provide some few other options  during VCH creation:
--tls-ca “file” the location for the CA certificate.
--tls-server-cert “file” the location for the custom server certificate.
--tls-server-key “file” the location for the private key which generated the server certificate.
--tls-cert-path “path” for the location to save your client certificate

As we could see, VCH has loaded server certificate in order to generate the client certificate, which the users will be required to connect to it.
Again, delivery the client certificates to your users and don’t forget to adjust the environment to point to the new certificates and you are ready to go.

As a last tip, do not delete the folders and certificates of your VCH, it's might be useful if you need to redeploy a VCH, reusing the certificates means you dont need to send new certificates to your users.

I hope by now you are empowered with all the knowledge to protect your environment.

See you next

Who am I

My photo
I’m an IT specialist with over 15 years of experience, working from IT infrastructure to management products, troubleshooting and project management skills from medium to large environments. Nowadays I'm working for VMware as a Consulting Architect, helping customers to embrace the Cloud Era and make them successfully on their journey. Despite the fact I'm a VMware employee these postings reflect my own opinion and do not represents VMware's position, strategies or opinions. Reach me at @dumeirell

Most Viewed Posts

Blog Archive