Error : Unknown CPI error "Unknown" with message 'getaddrinfo': Name or service not known (esxiXX.XX.XX:443) in 'create stemcell' CPI method (CPI request ID:''cpi-XXXXXX) in BOSH DIRECTOR on vSphere

I got stuck on this error in my homelab while learning BOSH Director from scratch. And it is a well-known issue which we could get stuck sometimes and its "DNS".

I am deploying bosh director where I have DNS working perfectly fine from my machine, within and outside. Even my mobile was able to resolve DNS for all my servers, just to ensure its working fine. But the error kept on saying its a DNS issue.

So BOSH Director was successfully deployed, but while uploading stemcell failed because of the DNS issue. 

What I have figured out, BOSH Director VM in my case is 172.20.10.6 which must have DNS resolution to ESXi as it is going to upload the stemcell VM to a shared Datastore of an ESXi host and onwards you could create your deployment. I was trying to dig into BOSH Director VM for which I was not able to find the password with default installation as mentioned on bosh.io 

 

Now after a while, I figured that you can SSH into bosh director via a jumpbox user which is to be specified at the time of create-env for which you have to add -o bosh-deployment/jumpbox-user.yml.

After creating jumpbox user I logged into BOSH Director, I found the default DNS to be 8.8.8.8 and there was no I was able to make any changes with the default installation.

Now, this global DNS setting is supposed to be pushed by some configuration files. I even logged into stemcell having USERNAME:- vcap & PASSWORD to be c1oudc0w as mentioned on bosh.io to verify if this DNS setting could be already there but no luck!

After a lot of internet search was not able to come across an article that could have helped me to resolve this.

This 8.8.8.8 global DNS setting is pushed by /bosh-deployment/bosh.yml

So there are two ways to fix this:- 

1. You edit this file with your own DNS Server IP then initiate deployment.

2. You can edit this file and declare an ARRAY type variable at the time of environment creation.

And here I declared my DNS to be array type and it will be there in my bosh0 VM which is created after this create-env.

And finally, uploading stemcell completed successfully.

Kubernetes the Hard Way on VMware Workstation

Time to play with some Github. I am new to this ecosystem of DevOps where one can easily fork the content published and modify according to their requirement.

I am learning Kubernetes for the past few months and is preparing for CKA(Certified Kubernetes Administrator).  One of the most important parts as per some findings and experience shared by the opensource community is KUBERNETES-THE-HARD-WAY.

It took some time for me to understand this so-called "KUBERNETES". I must say I learned a lot from an online Udemy course Certified Kubernetes Administrator (CKA) with Practice Tests by Mumshad Mannambeth. He is a kind of Gem for this technology.  

In this course, labs are available on your web browser with no additional charge and you can spin it N number of times in a day. I have completed all the labs once.

Afterward, I had to get my hands dirty on the Ubuntu terminal to configure the Kubernetes Cluster manually.

Instead of posting it on the blog, I have posted it on Github.

• Prerequisites
• Installing the Client Tools
• Provisioning a CA & Generating TLS Certificates
• Generating Kubernetes Configuration files for Authentication
• Generating the data encryption config and key
• Bootstrapping the etcd cluster
• Bootstrapping the Kubernetes Control Plane
• Bootstrapping the Kubernetes Worker001 Node
• TLS Bootstrapping Worker002 Node
• Deploy Pod Networking
• RBAC for kubelet Authorization
• Deploying DNS for pods

A majot difference in bootstrapping cluster using hard-way and using Kubeadm is:-

In case of Manual Bootstrap:- Kube-Scheduler, Kube-Proxy, Kube-Controller-Manager, etcd, Kube-APIServer are running as a Service

In case of Cluster Bootstrapped using Kubeadm:- All of the components are running as pods under kube-system namespace.

Invoke-NSXWebRequest in PowerCLI

From a few days, I was trying to use Invoke-NSXWebRequest to make a RESTApi Call in PowerCLI.

I was not sure how to do this at once, Learning something is always good. I know I am not that good at scripting, but at least I know how to get my work done.
Not every cmdlet is available in POWERNSX  as of now, but nowadays EVERYTHING IS API & API IS EVERYTHING.

So here I have enabled high availability for NSX Edge using POWERNSX & RESTAPI

$uri = '/api/4.0/edges/edge-2/highavailability/config'
from NSXRestAPI Documentation  following syntax

PUT /api/4.0/edges/{edgeId}/highavailability/config
 where method type is PUT


$Body ='<highavailability> <ipaddresses> <ipaddress>192.168.222.1/30</ipAddress> <ipaddress>192.168.222.2/30</ipAddress> </ipAddresses> <declaredeadtime>6</declareDeadTime> <enabled>true</enabled> </highAvailability>'
Defined a Variable named $Body and passed onto the given content.

$MyConnection = Connect-NsxServer -vCenterServer sa-vcsa-01.vclass.local -Username administrator@vsphere.local -Password VMware1!
Invoke-NSXWebRequest -Uri $uri -Method Put -Body $body  -connection $MyConnection

Re-Deploying an EdgeGateway using RESTAPI

I am upgrading my lab from vCloud Director from 8.20 to 9.0. One of the tasks is to upgrade existing vShield Edge or Edge Gateway from 5.5.4 to 6.3.4.Although it's optional, anyhow we have to upgrade as we were moving from Isolated Port Group network backing to VXLNA backed Port Groups. As I always strive to find different ways to perform the same task, I came across this POSTMAN from which this task could also be initiated.Hereby attached is the screenshot along with how one can navigate to the edge gateway of an ORG.

As mentioned in the previous post, Login to vCD using POSTMAN you may get started following that.

Searched for ORGANIZATION named VIRTUAL SINGH, double-click on href.

Navigated to myVDC named VSINGH VDC.

Here navigating to ProviderVDC named GoldPvdc 

Finding the networking href for VDC

Finding the appropriate Edge Gateway

And Using POST method to Redploy the edge gateway.

Edge gateway redeploying initiated using REST API from Postman Client in vCloud Director.

Virtual Machine - Edit Settings - (Disk) Part 3(ii)

In my previous post, I have talked about Virtual Machine Storage Policies.
In this post, we"ll be talking about Different types of disk provisioning:-

There are 3 different types of disk provisioning:- 

• Thin Provision
• Thick Provision: i)Eager Zeroed
•                             ii) Lazy Zeroed

By default, a Virtual Machine is Thick Lazy-Zeroed in case of Block-Level storage. You can change the type of disk provisioning at the time of creating a VM in Block-Level Storage. Now I"ll talk about the three types:-

1. Thin Provisioning:- Allocation and Zeroing of blocks are done on demand upon the first write to block. The file blocks are zeroed out when blocks are allocated. File block allocation will be dynamic i.e non-contiguous. Mostly used in Cloud environments or automated infrastructures where provisioning required will be fastest.  This now defaults in case of Virtual SAN.

2. Thick Provision: i)Eager Zeroed:-  File blocks allocated are allocated and Zeroed at the time of vmdk(Disk) creation. In this blocks will be fully allocated & will have highest chances of contiguous file blocks allocation. It will be taking more time during creation.

3. Thick Provision: ii) Lazy Zeroed:-  Blocks are fully preallocated but blocks are zeroed out when each block is first written to. It will be faster than Thick Provision Eager Zeroed.  Chances will be higher of contiguous file block allocation.

There is a topic of debate when to use Thick Provision Eager Zeroed and When to use Thick Provisioned Lazy Zeroed. There's an interesting topic available Eager thick vs Lazy thick disk performance

Vembu BDR Suite : Product Review

Vembu BDR(Backup & Disaster Recovery) is a trusted solution for Small, Medium and Large Businesses reaching 100+ countries, having 4k+ partners and 60k+ Businesses. Vembu Technologies has recently released a backup and DR suite called Vembu BDR Suite v3.9.0.  Their Offerings are in 2: Vembu BDR Suite & Vembu Cloud Services. 

Vembu has a lot in their product line so covering each and every feature will be beyond this post. I"ll try to cover all the possible features in a series of blog posts.

• Vembu BDR Suite: It is a suite of four services, built into one unified web-based user interface, which includes:-

1. Vembu VMBackup: For VMs created on VMware ESXi and Microsoft Hyper-V VMbackup could be used. It supports application-aware backups without installing any agent. Apart from backup and restore of virtual machines, Vembu VMBackup also supports replicating virtual machines from one production hosts to another that's an interesting feature. 
2. Vembu ImageBackup:-Vembu ImageBackup (Part of Vembu BDR Suite) is a comprehensive data protection and disaster recovery solution designed to meet the needs of physical data centers and workstations for SMBs and Enterprises. Users can now provide their data centers and Workstations(desktops and laptops) the utmost protection they deserve with Vembu ImageBackup. It requires an agent to be installed on your machine. It uses Disk Image Backup, CBT and Application Aware Backups.
3. Vembu NetworkBackup:- It supports specific files/folder backup and application backups such as Microsoft Exchange, Microsoft SQL Server, Microsoft Active Directory, Microsoft SharePoint, Outlook, and MySQL.
4. Vembu OffsiteDR:-Vembu OffsiteDR (Part of Vembu BDR Suite) is a comprehensive data protection solution that is designed to help users restore their backups that are replicated from Vembu BDR Server to their very own data center. For this feature to work one has to deploy a Vembu Offsite DR Server.

• Vembu Cloud Services:- It is a suite of four services, built into one unified web-based user interface, which includes:-

1. Vembu CloudDR:- Vembu CloudDR is a Disaster Recovery strategy wherein users can restore their backup data directly from our Vembu Cloud Server with minimalistic downtime anywhere anytime. Users can now replicate their data from their backup server to our Vembu Cloud Server seamlessly where all your data is kept secured using AES - 256 bit encryption during rest and during replication.
2. Vembu OnlineBackup:-Vembu OnlineBackup is a comprehensive data protection solution designed to provide the best backup and recovery solutions for business data of Small and medium businesses. Vembu OnlineBackup supports specific files/folder backup and application backups such as Microsoft Exchange, Microsoft SQL Server, Microsoft Active Directory, Microsoft SharePoint, Outlook, and MySQL. It allows users to store their business data safe and secure over Vembu cloud and provides a restore anywhere/anytime model.
3. Vembu SaaSBackup:-Vembu SaaSBackup is a comprehensive data protection solution designed to meet the needs of small and medium businesses protecting their mail servers with domain and user level backups. Vembu SaaSbackup currently provides backup support to Microsoft Office 365 and Google Apps. With backup data stored over Vembu cloud, users are provided with the option of restore anywhere-anytime with the least possible downtime.
4. Vembu BDR360:-Vembu BDR360 is a centralized monitoring and management portal which provides end to end visibility to all your IT environment that is backed up(VMs, physical servers, applications etc.,) using Vembu BDR Suite products. It allows administrators to manage backups across diverse environment at ease.
   
   
   Reviews:-

• Backups have been made simpler with this. Installation is easy, Management & configuration of backups is simple with just a few clicks to initiate.
• Available as a .exe installer for Windows & .sh shell executable for Ubuntu. Whether you are running your on-prem or in Cloud Infrastructure i.e for AWS ec2 instances or for Azure VMs. Installation is same. You can check system requirements here.
• I have installed Vembu BDR on Windows Server 2012 & Ubuntu 16.04 LTS. When I started reviewing this product it was v3.8.0 and now v3.9.0 is GA so anyways I have to upgrade in the lab. I have upgraded to Windows-based Vembu BDR Server which is done with just a few clicks.
• File System:- VembuHIVE which is a vembu proprietary for BDR solutions.
• It uses PostgreSQL 9.6 database version.
• So if you are looking to migrate from one OS version to another, it will be like you have to backup of your DB and have to restore it to other O.S.
• There is a riveting comparison between free and paid edition.
• In order to activate Vembu BDR suite, the machine is required to have internet connectivity either directly or in restricted environments could be done via proxy. Every 7 days it is going to check with vembu license server for your currently added license.
  As of now, there is no option available for OFFLINE licensing of this product, but I am very sure these folks will come up near future.
• It supports integration with REST API based client means if you are looking for some automation with a tool or a script, that would be amazing. For REST-API based parameters, you may check on this link. I will be looking forward to integrating it with vRealize Automation & vRealize Orchestrator to make Day2 activities less complex.
• With the latest release they have come up with pretty much interesting feature and one of them is Pre/Post Backup commands which could be a PowerShell, Ruby, Perl script.
• The first-time configuration after installation is:-

• You have to specify the time zone in which it is installed so that reports and we console will have the same time settings.

• For registration, you have to create a unique vembu ID.

• In the snippet, above shown are the different features of VembuCloud Services such as OnlineBackup, CloudDR, OffsiteDR, BDR360, SaaSBackup.

• You have to register your product for the trial version and later you can go for the paid license. Its license is available on subscription and perpetual which has different prices for features.

• Next is you have to add your vCenter Server, ESXi for VMware environment or Hyper-V for your Microsoft environment.
• I tested and initiated a backup on Linux as well as windows VMs. It needs the most important utility to be running on your operating system i.e VMware tools in order to quiesce I/O.
• It supports Full as well as Incremental backups.
• I"ll be covering each and every feature in a series of posts for this product.

**Please note that this review was sponsored by Vembu

Encrypted password to login to vCenter in POWERCLI

$Password = Read-Host -Prompt "Enter Password" -AsSecureString
$DecodedPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($Password))
Connect-VIServer -Server vcsa-a.vclass.local -User administrator@vsphere.local -Password $DecodedPassword

Uservars.ESXiShellInteractiveTimeOut & Uservars.ESXiShellTimeOut

• By Default, both the parameters have a default value as:-
  
  i) UserVars.ESXiShellTimeOut =0
  ii) UserVars.ESXiShellInteractiveTimeOut=0
  
  Value 0 means, it is disable
• But if you are in production and had some urgent maintenance or activity enabling SSH & its timeout value could be cumbersome.
• If you have different values as per your environment, you can change the value to 0.
  
  
  
  
  
  
  
• $Compute_Cluster = Get-Cluster -Name Compute | Get-VMHost
  $Compute_Cluster | Get-AdvancedSetting -Name UserVars.ESXiShellTimeOut  | Set-AdvancedSetting -Value 0
  $Compute_Cluster | Get-AdvancedSetting -Name UserVars.ESXiShellInteractiveTimeOut  | Set-AdvancedSetting -Value 0

Turn ON/OFF SSH on All ESXi host in vCenter/Cluster

$AllEsxi = get-vmhost
$AllEsxi | get-VMhostService | where {$_.key -eq "TSM-SSH"} | Set-VMHostService -Policy ON
$Allesxi| Get-VMHostService | where {$_.key -eq "TSM-SSH"} | Start-VMHostService 

Output:-

• First, we declare a variable $AllEsxi which stores result from Get-VMhost & then we are enabling SSH on all the given hosts.
• If you want to turn OFF the SSH on all the hosts, simply replace ON with OFF and run the command.
• To turn off & stop the service on all the host.

$AllEsxi | Get-VMhostService | where {$_.key -eq "TSM-SSH"} | Set-VMHostService -Policy OFF
$Allesxi| Get-VMHostService | where {$_.key -eq "TSM-SSH"} | Stop-VMHostService 

• If you want to do it at Cluster level, then specify the cluster variable or then use the following:-
  

$Cluster_Host = Get-Cluster MyCluster | Get-VMHost

$Cluster_Host | get-VMhostService | where {$_.key -eq "TSM-SSH"} | Set-VMHostService -Policy ON

$Cluster_Host | Get-VMHostService | where {$_.key -eq "TSM-SSH"} | Start-VMHostService 

• If you want to turn ON/OFF service at cluster level, then specify the cluster variable or use the following:-

$Cluster_Host  | Get-VMhostService | where {$_.key -eq "TSM-SSH"} | Set-VMHostService -Policy OFF
$Cluster_Host  | Get-VMHostService | where {$_.key -eq "TSM-SSH"} | Stop-VMHostService 

Login in vCloud Director using REST API Client - POSTMAN

• Nowadays most of the virtualization products support REST-API.  
• I first learned to interact using rest-api is with RavelloSystems(a cloud platform by Oracle)
• The best client I have come across for it is POSTMAN client available for windows, mac & Linux. I have installed it on my MacBook & windows 10 machine. 
• So this time I wanted to interact with vCloud Director 8.2 & 9.0 using REST-API
• After a long day meeting and looking for how to log in successfully in vCD using postman client, somehow I figured it late night
• So the basic options that we use in it are GET, PUT, POST & DELETE.
• So if you have worked on HTML it makes it easy to understand the output.
• As VMware has already started in vCenter 6.5 - REST API explorer.

Steps to login in vCD using POSTMAN

• URL for vCD i.e
  https://IP or FQDN/api/sessions
• Authentication- Basic (As per snapshot no. 1)
      Username- administrator@system   :- If you are a system administrator.
      I found it difficult for the username which took me this much time. So this user can be a LDAP integrated user or Active Directory.
      Password-  As per your directory
• Once authorized with Status:200 OK .  In the AUTHORIZATION tab as highlighted, we have HEADERS tab where we will recieve a x-vcloud-authorization: token.
• Instead of using your credentials for authentication, we can use the custom header (As per snapshot no. 2 & 3).
• Now we are ready to work on this REST-API for interaction using postman client.

Virtual Machine - Edit Settings - (Disk) Part 3(i)

So after writing the series for CPU & Memory. This series will be dedicated to Settings related to vmdk i.e Virtual Machine Disk.

• The first option is the size of the virtual machine which can be in MB, GB or TB.
• Maximum Size is the size of the datastore up to which VM hard disk could expand.

VM Storage Policies in production environments is created for storage tiering if using traditional storage i.e Block-Level Storage, Virtual San or a Virtual Volume.

VM Storage policies can be created & managed from vSphere Web Client. I"ll be writing a series of post for different policies which can be created for Virtual SAN, Virtual Volume & Block level storage.

vSAN Policies with PowerCLI(FTT=1,Stripe=3, Object Space Reservation = 50%)

New-SpbmStoragePolicy -Name OBS -AnyOfRuleSets (New-SpbmRuleSet (New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.hostfailuresToTolerate") -Value 1), (New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.stripeWidth") -Value 3),(New-SpbmRule -Capability (Get-SpbmCapability -Name VSAN.proportionalCapacity)-Value 50))

Output:-

vSAN Policies with PowerCLI(FTT=1, RAID5/6)

New-SpbmStoragePolicy -Name PowerCLI-RAID5  ` -AnyOfRuleSets ` (New-SpbmRuleSet ` (New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.hostFailuresToTolerate" ) -Value 1),(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.replicaPreference" ) -Value "RAID-5/6 (Erasure Coding) - Capacity"))

Output:-

vSAN Policies with PowerCLI(Striping=3)

New-SpbmStoragePolicy -Name Stripe=3 -RuleSet(New-SpbmRuleSet -Name "striping=3" -AllOfRules @((NewSpbmRule -Capability VSAN.stripewidth 3)))

Output:-

vSAN Policies with PowerCLI (No redundancy i.e FTT=0)

New-SpbmStoragePolicy -Name FTT=0 -RuleSet(New-SpbmRuleSet -Name "NoRedundancy" -AllOfRules @((New-SpbmRule -Capability VSAN.hostFailuresToTolerate 0)))

Output:-

Check IP for Management VMkernel

esxcli network ip interface ipv4 get

Output:-

Network Port Binding for software iscsi

esxcli iscsi networkportal add -A vmhba33 -n vmk2

Output:-

Add software iscsi target using esxcli

esxcli iscsi software set -e true

Output:-

Add Target for software iscsi using esxcli

esxcli iscsi adapter discovery sendtarget add -a iscsi.vclass.local:3260 -A vmhba33

Output:-

Enable software iscsi on esxi02

Get-VMHostStorage -VMHost esxi02.vclass.local | Set-VMHostStorage –SoftwareIscsiEnabled $True

Output:-