STC vCloud Suite 5.1.x Build Guide

Posted on June 9, 2013 by michaeldmills

Having no idea how well received my vSphere 5.1 build guide was, I’ve now decided to put out the vCloud Suite build guide. Again, like the vSphere guide, it is entirely my efforts in figuring this out. I reckon there are probably some easier means but hell, I’m not at that level of skill Enough jive, let’s get busy.

Deploy the OVA of vShield Manager to the cluster that will be used for all the elements of cloud management. This cluster will not be a part of resources available to the cloud
Once deployed, power on vShield Manager and open a console. Eventually a login prompt will appear. The credentials are (admin:default). Similar to a Cisco switch, there is an enable mode. Type “enable” and then use the same password (default). There will be a # to indicate you are in enable mode
Type “setup” to launch the initial configuration wizard. Provide the IP address, subnet mask, gateway, primary and secondary DNS, and search domain (domain name). Commit the changes with a “y” and then logout using exit. Give VSM about 10 minutes to get settled and then login again (still at the console)
Log back in at the console and make sure you can ping the gateway, the vCenter for vCloud Suite, and the primary DNS server. When complete exit out and close the console. Time to move to the web interface!
Change the password for the admin account (if you like pain )
Set the NTP server and reboot to take effect
Create a service account for vShield to use to connect to vCenter (make sure to put it in the security group that is configured in SSO for login access)
SSL Certificate time! Create an A record in DNS for vShield Manager. Next, in the vShield web interface go to SSL Certificates. For Common Name, use the FQDN for vShield (thus the record you just created in DNS). The remaining fields would be filled out like with other SSL certificates within the organization. Set the Key to RSA and it should change to 2048 for size. Click Generate and then after a moment you can download the CSR file. Copy and paste the contents into the CA request web interface. The template choice is the Web Server option. In the Attributes box add this information to also allow for certification if you go to the short name and IP address for vShield. (Here is an example: san:dns=10.10.130.20) Download the certificate as Base 64 encoded
Import the Root64.cer (this was created and used for the vSphere installation)
Import the CA generated cer file for vShield and click Apply Certificate when prompted. This will reboot vShield to apply it. After reboot wait a few minutes for it to turn up the web interface
Once the web interface is accessible, check the certificate to make sure it applied and is legit. I used the short name and FQDN and it works. For whatever reason at this point it does not work with the IP address
Next connect to vCenter using the FQDN, service account credentials, and leave the box checked to make it an Enterprise user
Once that connection is complete it will show up in Solutions in the vSphere Client. Go assign the license key for it
Next we will attach it to the Lookup Service for SSO. Edit the Lookup Service and use the FQDN of the vCenter server as well as the admin@system-domain account and master password from setting up SSO prior with vSphere
With this completed, the last step is to setup a backup. An FTP/SFTP location will be required. Note that for the backup directory, use the complete path the home directory for the user created for the vShield backup jobs
With vShield Manager deployed, it is time to create the database for the vCloud Director cells. Log into the designated SQL server and create a database. Growth for the database and the log should be 10% and the database itself should start at 100mb, the log at 1mb. Set the Recovery Model (if it needs to be something besides Full) AND set the correct collation: Latin1_General_CS_AS
Run the following query to modify the database correctly:

USE [vcloud]
GO
ALTER DATABASE [vcloud] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
ALTER DATABASE [vcloud] SET ALLOW_SNAPSHOT_ISOLATION ON;
ALTER DATABASE [vcloud] SET READ_COMMITTED_SNAPSHOT ON WITH NO_WAIT;
ALTER DATABASE [vcloud] SET MULTI_USER;
GO

Create a database user for the vcloud database and set that as the default database, set db_owner and schema to dbo
Deploy two Red Hat / CentOS 6.x virtual machines to be used as vCloud cell servers. Create DNS A records for each cell for the http (web interface) and consoleproxy (connection to consoles of VMs through web interface). For SSL certificates, they will need two A records for each cell. Configure static IP addresses (/etc/sysconfig/network-scripts/ifcfg-eth0) for the management interface and set the gateway (/etc/sysconfig/network). Patch them via yum (and reboot if needed). Now shut them down and add hardware (if not already at the recommended configuration). 2 vCPU, 4GB RAM, 2 VMXNET3 network interfaces.
Power on the VMs, login, and then use SFTP to copy the vcloud installer binary to each cell (/root is fine) and chmod +x to make the binary executable
Now set the static IP for the second VMXNET3 interface (required by the vCloud installer binary). The easiest thing is to probably just cp the existing ifcfg-eth0 to ifcfg-eth1 and then modify the DEVICE= and IPADDR= (/etc/sysconfig/network-scripts/ifcfg-eth1). Note that if the second interface (typically the consoleproxy) is on a different VLAN or subnet there will need to be work done on the cells to set the correct default routes. Once this work is done restart the network service and do some test pings to the new IP
Run the binary installer but after installation do NOT run the configuration script. Additional work must be done to get the cells ready for kicking butt
Now to test mount the NFS transfer point (always test this before editing the fstab as if you mess that up, you can hose up the OS on boot. Make sure these pieces are in order before attempting the mount:

yum install nfs-utils rpcbind
/etc/init.d/rpcbind start
/etc/init.d/nfslock start
***turn those on with chkconfig

With those in place and confirmed, test the mount using the following syntax: mount -t nfs ip_address_of_nfs:/export_path /opt/vmware/vcloud-director/data/transfer
If it does look like it mounts okay check using the command: df -h to confirm
Once mounted and in good shape, edit the fstab file to mount this nfs volume on boot. Place the entry at the bottom of the fstab file on a new line and it should read like this (there are tabs between each section, only a space between the zeros) ip_address_of_nfs:/export_path /mount_point nfs defaults 0 0
Reboot and then check using df -h to make sure it mounted correctly and is accessible
Now change the owner (user and group) of the NFS mountpoint to allow vCloud to do as it pleases: chown -R vcloud.vcloud /opt/vmware/vcloud-director/data/transfer
Create a certificate keystore and initial certificate for the http (web management). Run this command on each cell: /opt/vmware/vcloud-director/jre/bin/keytool -genkey -keystore
/opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -keyalg RSA -alias http
The prompts to fill out the certificate should be completed as follows. Note that the First and Last name instance should be the FQDN of the cell:

What is your first and last name? [Unknown]:mycloud.example.com
What is the name of your organizational unit? [Unknown]:Engineering
What is the name of your organization? [Unknown]:Example Corporation
What is the name of your City or Locality? [Unknown]:Palo Alto
What is the name of your State or Province? [Unknown]:California
What is the two-letter country code for this unit? [Unknown]:US

Is CN=mycloud.example.com, OU=Engineering, O=”Example Corporation”, L=”Palo Alto”, ST=California, C=US correct?[no]:yes
Enter key password for <http> (RETURN if same as keystore password) – use the same password as the keystore for simplicity (testpassword)
Now create the certificate signing request (CSR file) with the following command:
/opt/vmware/vcloud-director/jre/bin/keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -certreq -alias http -file http.csr
Next create one for the consoleproxy (connecting to virtual machine consoles through vCloud Director web interface):
/opt/vmware/vcloud-director/jre/bin/keytool -genkey -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -keyalg RSA -alias consoleproxy
You will receive the same kinds of prompts so fill them out again, remembering to use the FQDN for the consoleproxy (the second DNS A record that was created)
Now create the certificate signing request (this time for the consoleproxy) with the following command:
/opt/vmware/vcloud-director/jre/bin/keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -certreq -alias consoleproxy -file consoleproxy.csr
Grab those CSR files from each cell using SFTP and run them through the MS CA web interface, selecting Web Server as the certificate type. Add in the following to the Attributes box to make sure FQDN, hostname, and IP are all registered: san:dns=hostname.domain.com&dns=hostname&dns=ip_address ***Download them as DER format
Rename each of the files to be either http.cer and consoleproxy.cer respectively
Copy each of those files and include the root.cer (certificate authority root cert) back onto each cell
Import the CA root certificate (and answer yes when prompted to add to the keystore): /opt/vmware/vcloud-director/jre/bin/keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -import -alias root -file root.cer
With the root certificate imported, next import the http.cer: /opt/vmware/vcloud-director/jre/bin/keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -import -alias http -file http.cer
Finally import the consoleproxy certificate: /opt/vmware/vcloud-director/jre/bin/keytool -keystore /opt/vmware/vcloud-director/certificates.ks -storetype JCEKS -storepass testpassword -import -alias consoleproxy -file consoleproxy.cer
Now do a chown to vcloud.vcloud on the certificates.ks file
Make sure before going any further you are at the console and not using ssh. For whatever reason the configure script gets funky over ssh
With the certificates imported, the next step is to run the vCloud configuration script: /opt/vmware/vcloud-director/bin/configure What this will do is let you select which IPs are used for the http piece and which is for the consoleproxy. It will require the path to the keystore, the keystore password, and information regarding the database. Also if this is being installed on a minimal install (CentOS or Redhat) do a yum install redhat-lsb to install all the needed packages.
When you run the configure script, it will prompt for the location of the certificates.ks file. Since it is in /opt/vmware/vcloud-director/, just type certificates.ks as it doesn’t need the full path. Enter the password for the keystore and then you’ll need to put in the database info. Use the FQDN for the DB server. Once the install is complete it will want to start the services so fire them up.
tail -f /opt/vmware/vcloud-director/log/cell.log so you can watch it coming up.
With the first cell up, connect to it and make sure the certificate is legit. Now it’s important for all additional cells to have access to a file that contains all the information entered during the configure script. It is kept in a response.properties file in /opt/vmware/vcloud-director/etc. Copy this file to the transfer mount so that all the other cells have access. Make sure it is owned by vcloud.vcloud and the permissions are 600
Log into each additional cell and copy the response.properties file into /root. Run the installer with the argument of the response.properties file to have it pull the info needed during the install: ./binary_installer_name -r /path_to_responses_file
After installation of all cells is complete, connect to the first cell and enter the information needed for the initial configuration wizard (license key, administrator login, etc)
Once the initial wizard is completed and the login page becomes available we move onto vCloud Director configurations. The first step is to connect it to the vCenter it will be using. After login you will be at the Home page. A service account will be needed by vCD to communicate with vCenter. Create one and add it to the correct sso_admins group
Next there is some general configuration elements for vCloud that should be performed before attaching vCenter and building stuff. Now set the mail notification settings up in the Email section of the Administration page
If SSO will be used to manage access to the System (so the administration part, not the orgs part) of vCloud, you cannot use a security group in AD as for whatever reason vCloud will not work with groups. You have to set individual users up instead. Register for the Lookup Service and use the FQDN for the Lookup Service URL and admin@system-domain for the admin credentials. Go to the Users section of the Administration page. Click the Import Users button. Put the name of the users (one per line) in this format: group@domain Note that no spaces are allowed so just use _ for everything like you’ve been doing. With the users added test the ability to connect and login using an account that was added
The alternative if SSO will not be used is to use the regular LDAP integration
Next connect to vCenter and vShield Manager using the Attach a vCenter wizard on the Home tab of the vCloud web interface
Now at this point, storage profiles need to be created to be used for the provider vdc. This is found in vCenter so pop into the vSphere Web Client, hit the home page and go into VM Storage Profiles. First, Enable storage profiles at the cluster level. Next, create a Storage Capability for each storage array that will be added to the pvdc. Next create a VM Storage Profile and add the capabilities. Now it needs to be assigned to a datastore, so head to Datastores and assign that sucker
Now the provider vdc needs to be created. Go through the creation wizard and at the storage profiles part of the wizard you will be able to select the storage profiles. Also pick the cluster and enter the credentials for the ESXi hosts to deploy the vCloud agent. Once finished check the hosts page in Monitor and Maintain to see when they are done being prepped
Once the hosts are prepared, it’s time to create an organization and then the organization vdc(‘s) below it
So at this point vCloud is ready for customization and multi-tenancy. Follow the vCloud Organization Build Guide for that piece
Next up is the Chargeback installation. For chargeback you can have multiple nodes that are then put behind their crummy software load-balancer. Likely one Windows VM will be enough. To get started, you will need a 2 vCPU, 4GB RAM, 40GB Disk, VMXNET3 VM
Sometimes there are cases where using a Template in the environment for Windows Servers will cause Chargeback to not run correctly. I recommend you build this VM from scratch
Create a service account for Chargeback and put it into the correct security group to enable it to login to vCenter
Create a SQL database for Chargeback and a local SQL account for it to use. Make the local account db_owner of the Chargeback database and set the schema to DBO. Next go back to the database, go to permissions, and make sure the SQL account has the following permissions checked:

* Alter any schema
* References
* Insert
* Select
* Delete
* Update
* Execute
* Alter any dataspace
* Create table
* Create view
* Create procedure
* Create function

Install the Microsoft Visual C++ 2005 x86 redistributable on the Chargeback Windows VM
Snapshot the VM before installing as Chargeback can be untrustworthy
Run the Chargeback installer as administrator (right click > Run as Admin). Provide the hostname for the db server (and instance name if there are multiple instances on the db server), the database name, and the local sql account that was created in the previous steps. Modify the email address if needed when presented with the load balancer option. If you are installing this into an environment large enough to require multiple Chargeback servers then this first install will be the load balancer. You can then spread that out with additional installs. Finally there is the option to install the Chargeback Manager server itself (If you were just installing the load-balancer portion don’t check this box). Checking the box will require the Server Name field to be entered, use the server’s hostname. A default administrative (local to Chargeback) account will created and you will be prompted to enter the password to be used for this account. Select the data collectors that will be installed (likely all three if this is a vCloud Suite installation). The last piece of information required to enter relates to the vCloud Director data collector. It will ask for a hostname/ip or URL for one of the vCloud cell servers. In the case of a load balancer, use the FQDN to point to vcloud for the administration page (ex: vcloud.aheadaviation.com) and provide the administrator login
After installation it will prompt to generate a default SSL certificate or to Generate your own. Click to Generate your own and a command prompt will launch. Use a password for all the key and signing request generation (you will be prompted 4 times for the same password in a row). I use testpassword like with all the VMware certificates. It will then ask you for the usual Country, State, City, Company, Division, and FQDN. You will be prompted once more for the testpassword. Then it should succeed and press any key to continue. Mind you that all this is really doing is creating the key and we have to create the csr to get the certificate from the authority
Navigate using Windows Explorer to C:\Program Files x86\VMware\VMware vCenter Chargeback\Apache2.2\conf\. Copy the openssl.cnf file to C:\Program Files x86\VMware\VMware vCenter Chargeback\Apache2.2\bin folder and replace the one that is in that folder
Copy the default.key file from C:\Program Files x86\VMware\VMware vCenter Chargeback\Apache2.2\conf\ssl\ into \bin
Open a command prompt as admin and go to the bin folder. Run the following command to generate the CSR:
openssl.exe req -new -key default.key > default.csr -config openssl.cnf
Use that CSR to get the CER file using the normal web interface method for the certificate authority server. Submit using the Web Server template. Export as a DER encoded file and rename to default.cert
Copy the default.cert file back into C:\Program Files x86\VMware\VMware vCenter Chargeback\Apache2.2\conf\ssl\ and replace the one that is in that folder
Reboot the server and it should now be square when you launch the web interface (https://fqdn/cbmui)
If all is well so far, delete the snapshot from prior to install
Time to patch Chargeback (if there are patches). Follow the instructions VMware has provided and I’d recommend snapshotting the VM before as a precaution
Chargeback now needs to be licensed (you will be prompted the first time you hit the web interface)
After licensing is complete, log in using the admin account and navigate to the settings tab. The first thing to do is setup SMTP to get notifications for alerts if there are problems with Chargeback services and also for reporting emails to be sent. In the Email Setting section, click add and put in the info for the SMTP gateway. Next set the Alert Settings on the System Health tab to have the correct email address to send alerts
Next, add the vCenter server(s) that Chargeback will be talking to. This requires the service account created earlier, the sql account credentials for the database, and the url for the db server and database name
Now add the LDAP server to allow for logins from domain users. You can use the same service account for this configuration item
Setup the Users and Roles using the LDAP security groups that were created for Chargeback. Test logins to make sure this is functioning. Note that you cannot put security groups within the security groups that are added to LDAP Groups in Chargeback and have logins be successful. For whatever reason it doesn’t like nested groups. As such, add individual users into these groups and then add them into Chargeback
Now make sure the data collectors are working and healthy. Head to the System Health tab and click the refresh button at the top right. Now address any issues with the red x’s. As a note, the vShield Manager needs the cli vshield login (admin:default) and NOT the login for the web interface. These changes can be performed on the Settings tab under Data Collectors
For proper stats collection, turn up the statistics logging level in vCenter to 3 for all 4 time intervals. Note that depending on the size of the vSphere environment this could lead to a ginormous database. As such, use caution and be considerate. There is also a VMware KB article to help with this sizing matter: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2010099

STC – vSphere 5.1 Update 1 Build Guide

Posted on May 19, 2013 by michaeldmills

As a quick note, I’m back! After joining the amazing people at Ahead, I have been busy as their new mad scientist! After spending over 80 days working straight, I am proud to say I have quite a bit of good content I can start to share (now that I know what I built works pretty darn well). As I think everyone who reads this blog knows, all of my posts are technical and I talk little of the industry. This shall not change Onwards with the heaping and unyielding blocks of text!

Here is the build guide that I have used and I have updated it for vSphere 5.1 Update 1. Two items were fixed and I am pleased by this. I also call out a few scripts that I use along the way and I’ll link them here on wordpress so that you can grab them as well. I think I can figure out how to do that

Also want to call out and say thanks to Derek Seaman as he did all the hard work with certificates!

Deploy VM from templates that will be vCenter
Power on VM and let sysprep run
After sysprep is complete, log in to kick off the activation tool (will reboot)
Next give a static IP to the VM and then patch Windows
Modify VM hardware to be the following 2 vCPU, 6GB RAM, 80GB local disk
Join server to domain
Add Application Server role to install needed .NET framework components, patch as needed
Install SQL Native Client (be mindful of your version of SQL as there is a corresponding client, ie 2008 R2 SP2)
Create a domain group (only _ is allowed for special characters) and place the users or groups that you want admin access for logging in to vCenter in this group
Connect to the DB server using SQL Management Studio
Create SSO database using the SSO_SetupTables.sql script (modify the script to give the database the correct name)
Create two users, using only _ in the names. Preferrably something like sso_user and sso_dba. Set the SSO database as the default for each. Next, expand the SSO database object to Security > Users. Right click on Users and add New User. These needs to be done for both SSO user accounts. Name them the same as the logins that were created in the prior step. The sso_dba accounts needs to be db_owner for schema and database whereas sso_user is just db_owner. These permissions are set via the same General page in the New User dialogue.
Create the vCenter and VUM databases, set growth to 10% and Simple for recovery
Create a SQL user account for vCenter and for VUM, set dbo and db_owner for both on their respective database and msdb
Run the following query to grant rights needed by the vCenter db account to grab stats:

use master
go
grant view server state to hrzvc
go

Make sure IPALL TCP Dynamic Ports is set to 0 (SQL Server Configuration Manager, requires restart of SQL Server) as it is needed for JDBC support
Mount the vSphere installer ISO and install the vSphere Single Sign On component. It would be at this point where you select if you are doing a Basic SSO install or if you want to select an HA setup. It’s a pain in the ass to repoint services so make this decision with some care. For the admin@System-Domain account, use a simple password. No crazy special characters or it breaks. Good example to use: American85! During the installation it will want the connection info for the sso database and the sso accounts. Use the FQDN for the DB server.
Moving on to certificates. Install the Visual C++ 2008 redistributable required for OpenSSL
Install OpenSSL-1.0.1e (current version at the time of this writing) and select the defaults during the install (especially the install location)
Create a local directory in C:\ called certs. This is where all certificate creation and generation will occur
Within C:\certs, extract the zip of the templates and move the .cfg files for each of the 6 services into separate directories for each service under C:\certs using the same name as the .cfg files
Modify each .cfg file to have the correct host short name, FQDN, and IP address in each of the sections
Next CSRs (certificate signing requests) must be generated, along with a key file, for each service. To do so, open a command prompt (run as admin) and navigate to C:\certs. In each directory run this command to generate the key file: C:\OpenSSL-Win32\bin\openssl.exe genrsa 2048 > rui.key
With the key file generated (again, one for each service in it’s corresponding directory), next the .csr files must be created. Again, run this command in each service directory using the cfg file for each service (ie so in the SSO directory use sso.cfg): C:\OpenSSL-Win32\bin\openssl.exe req -out rui.csr -key rui.key -new -config inventory.cfg This will generate .csr files for each service
Now the csr files need to be used to generate crt files. IE 10 doesn’t work with the 2008 R2 CA website so use Firefox. Open each CSR, copy and paste the contents in the Advanced Certificate Request option and choose the VMware-SSL option. For the download, select Base64 and rename each certificate to rui.crt (for each service, has to be the same name as the generated files)
Download the root certificate chain from the CA webpage: Choose the Base64 and download the certificate chain, giving it the name cachain.p7b
Open the cachain.p7b file and it will open a certificate management console. Expand out the folders to get to the certificate. Right click and choose export, select Base-64 as the encode method and save the file with the name of Root64.cer. Copy this file into the C:\certs directory
To create the required PKCS#12 PFX files use the following command for each service (in each folder) (note that “testpassword” is required by VMware. Do NOT try and use a custom password!). Notice that we also embed the root certificate into the PFX file (-certfile). Also, do NOT change the friendly name (-name) of “rui” to anything else. The Profile Driven storage service will likely not function and die with an unhelpful error message: C:\openssl-Win32\bin\openssl pkcs12 -export -in rui.crt -inkey rui.key -certfile C:\certs\Root64.cer -name rui -passout pass:testpassword -out rui.pfx
With the pfx files created for each service, now a JKS keystore is needed. To create the keystore, use this command (needs to be in CMD – run as administrator): “C:\Program Files\VMware\Infrastructure\jre\bin\keytool.exe” -v -importkeystore -srckeystore C:\certs\sso\rui.pfx -srcstoretype pkcs12 -srcstorepass testpassword -srcalias rui -destkeystore C:\certs\sso\root-trust.jks -deststoretype JKS -deststorepass testpassword -destkeypass testpassword
Next add the Root CA certificate to the keystore: “C:\Program Files\VMware\Infrastructure\jre\bin\keytool.exe” -v -importcert -keystore C:\certs\sso\root-trust.jks -deststoretype JKS -storepass testpassword -keypass testpassword -file C:\certs\Root64.cer -alias root-ca
The SSO service needs one additional copy of the keystore, use this command to create: copy C:\certs\sso\root-trust.jks C:\certs\sso\server-identity.jks
Now there is a batch file to run (PreStage.bat). Again run from a CMD – run as administrator. It is going to automate the prestaging of the SSL certificates into the locations the vSphere installation will look for them for all but the SSO and LookUp service
Now run the next batch file to handle SSO and LookUp service replacements: ReplaceSSOCerts_v2.cmd <SSO Server FQDN> <Certificates directory containing root-trust.jks> <SSO-Admin-Passsword> <Root64.cer directory>
Once this is completed, navigate to C:\ProgramData\VMware and create a directory called SSL
Copy your Root64.cer file into C:\ProgramData\VMware\SSL and rename it ca_certificates.crt
Compute the hash (still got that admin CMD prompt open?) for the root certificate: C:\OpenSSL-Win32\bin\openssl x509 -subject_hash_old -noout -in C:\certs\Root64.cer
If you have a single CA (no intermediary) copy your Root64.cer file into the SSL directory and rename it using the hash value from the previous step, and change the file extension to .0 (zero)
Now snapshot the VM to prevent anything stupid from happening (it happens alot)
Install the vCenter Inventory Service
Check to see if the certificate is in place (just click the lock, don’t worry about the bad request error page): https://fqdn:10443
Create the vCenter and VUM DSN (remember to use the SYSWOW64 for VUM)
Install vCenter – so this can be a bit complicated if things go wrong. Luckily you took that snapshot.
Once the install is complete make sure the services are running and the logs look clean. If so, reboot and make sure they start
Perform any Windows Updates, reboot as needed
THIS IS FIXED IN VSPHERE 5.1 UPDATE 1 Edit the registry to correct an issue as per VMware: HKLM\SYSTEM\CurrentControlSet\Services\ADAM_VMwareVCMSDS\Parameters – delete the existing Port SSL key and replace it with a DWORD(32) with the same name. Set the value to be 636 in DECIMAL and reboot
If everything is working after the reboot, delete the snapshot
Install the vSphere Client (needed for VUM)
Install the vSphere Web Client
Next install adobe flash (I know, I know, but we have little choice in the matter)
After flash is installed, launch the web client (https://hostname:9443/vsphere-client)
Login with the admin@system-domain account to adminster SSO. Navigate to Administration > Access > SSO Users and Groups
Go to the Groups tab, highlight the Administrators group, and click Add Principals
Change the context to the domain in question and search for the group that was created previously
Next, navigate to the Administration > Sign on and Discovery > Configuration
In the identity sources tab, highlight the domain and click Add to Default Domains. This will place the domain in the pane at the bottom. Highlight it there, use the up arrow to place it first in the list and click the save button
Go to the STS Certificate tab and click edit. Add the root-trust.jks file from the C:\certs\SSO\ directory. You will be prompted for the password (testpassword). Now highlight rui and click OK to add it. You will again be prompted for the password. Now reboot
Create a service account to be used for VUM and place it in the correct sso_admins group (as per a few steps back)
Install VUM: There will be a section that asks for the FQDN of vCenter, the port, and the account to use. This is where you will use the VUM service account that was created.
Once this is done, connect to vCenter using the vSphere Client (NOT web client) as configuration is required for VUM
Once logged in, install the VUM Plugin (if not already installed) and accept the SSL certificate warning (again, from the usage of 127.0.0.1)
Create a datacenter object (this will be used to attach the baseline too)
Head to the Update Manager icon on the Home page in the vSphere Client
Delete both of the default baselines that come out of the box in Baselines and Groups
Head to the Configuration tab, change the name to the FQDN for the host in the drop down menu on the Network Connectivity section (and hit apply)
On the Download Settings page click Download Now to pull down the patch metadata
On the Download Schedule page change the download task to run Sundays @ 1PM
On the Notification Check Schedule page change the task to run Sundays @ 12PM
It may also be helpful if you want notification for new patches to set that piece up but I have enough email traffic and a dedicated patch window already, regardless…
Head back to the Baselines and Groups tab and create a new Baseline (“ESXi 5.1 Baseline”) and include all VMware patches for embedded5.1.0
Go to Hosts and Clusters view, highlight the datacenter object, select the Update Manager tab and attach the newly created baseline
Now for the last SSL piece: the SSL certificates for the ESXi hosts themselves. There is a script that will need to be run and this requires the vSphere 5.1 CLI to be installed. So install that if it is not already installed
Make sure to create the A records in DNS for your ESXi hosts
Modify the bat file to contain the right info for the location and company. The certificate authority entry will also be needed as well as the certificate template to use. Lastly the path to the Certs folder needs to be in place (temp scratch space for the certificate creation)
Open (as Administrator) a vSphere CLI command prompt and run the bat file. The argument should be the FQDN for the host. This will take care of the complete certificate process and will prompt twice for the root password for the host. Afterwards the host will need a reboot. Perform this activity for each ESXi host
Enter the license info for vCenter and the hosts into vCenter
Assign the license key for vCenter
Add each ESXi host into vCenter and it will not prompt you for the SSL certificate because it’s too legit to quit baby. Also slap the proper license on that sucker when adding it in
Once all the hosts have been added patch them all in one big push
Create a cluster object (no configuration like HA or DRS) and add each ESXi host to the cluster
Now to set the vCenter configuration items. From the Home page go into vCenter Server Settings. Check both boxes for the Database Retention Policy and accept both default values of 180. Configure the SMTP setting and set the sender account to have a meaningful name (example – we have so many vCenters just use the vCenter name)
To enable alarms to send email, a powercli script will need to be run. Modify it to have the correct credentials, vCenter server hostname (or IP) and the email address(s) to send alerts too. Launch a powercli CMD prompt as administrator and run the script (again, once modified)
Host configuration is up next. For each host, rename the local datastore to hostname_Local, create a folder in the datastores view and place them all inside
Mount / Format datastores for use by the cluster
Create vDS and add first uplink for each host
Create a port group for the management VLAN (setting the right VLAN, teaming policy, etc)
Create a port group for the vMotion VLAN (again, make sure to set the right VLAN, teaming policy, etc)
Head to Hosts and Clusters > Configuration > Networking and for each host migrate the vmk0 for management to the vDS
Create a new vmk1 for vMotion for each host
Migrate the remaining nics to the vDS for each host
Enable cluster settings for HA and DRS

STC – VMware View 5.x Virtual Machine Creation Guide

Posted on January 28, 2013 by michaeldmills

This STC is a compilation of the notes and steps I used to build my parent VMs in the lab. The lab consisted of vSphere 5.0 Update 2 with VMware View 5.1.2. These settings worked fine for me but without a real production environment to test I reckon they may have some issues. I would encourage testing and customizing before using this for production.

–Virtual Machine Hardware and Customization–

**The hardware listed here is what was used in the lab**

1 vCPU
2GB RAM
LSI SAS SCSI Controller
VMXNET3 Network Adapter
24GB hard disk
Video RAM setting controlled through View

Edit the VM settings – Options tab
Set a 10,000 ms power on delay
Uncheck Logging
Check the box to boot directly into the BIOS.
Set Legacy Diskette A: to Disabled
Disable all nonessential ports under Advanced I/O

–Operating System Installation–

Install Windows 7 in the normal fashion
**Note that View does not support multiple volumes, so only use C:**
Once the operating system is installed, install VMware Tools
Patch the Operating System using Windows Update
Apply MS KB hotfix (pre sp1 – 2344941 or post sp1 – 2550978) for VMXNET3
Disable IPv6 unless needed, remove both Link Layer protocols on virtual adapter
Uninstall the IPv6 ISATAP adapter from Device Manager (hidden)
Remember to unmount the Windows 7 iso

–Optimization–

Turn off hibernation (CMD run as admin – powercfg /hibernate off)
Turn off System Protection on C:
Turn off Drive Indexing on C: (and Ignore All when the lame warning appears)
Set power options performance plan to High Performance
Set computer to NEVER sleep, turn off display, or hard drive sleep (advanced power option)
Disable Superfetch server, edit the following registry setting:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters
Set EnablePrefetcher to 0
Set EnableSuperfetch to 0
Uninstall Windows Components: Media Features, Tablet PC Components, Windows Gadgets
CMD run as admin – fsutil behavior set disablelastaccess 1
Select Windows Classic theme and Adjust for Best Performance
Disable System Maintenance Control Panel > Troubleshooting > Change Settings and select OFF under Computer Maintenance
Ensure Screen Saver is set to None
Disable Sound *if Audio will not be used* Set Personalization > Sounds Scheme to No Sounds
Disable offline files Control Panel > Sync Center > Manage offline files > Disable offline files
Speed up Menu show delay HKEY_CURRENT_USER\Control Panel\Desktop set MenuShowDelay to 1 (was 400)
Disable Windows Customer Experience Improvement Program (Control Panel > Action Center > Change Action Center, Click Customer Experience Improvement settings and click No
Disable all scheduled tasks in Task Scheduler Library > Microsoft > Windows:
Application Experience
Customer Experience Improvement
Defrag
DiskDiagnostic
Power Efficiency Diagnostics
Registry
Sideshow
WindowsBackup
Windows Error Reporting
Change TimeOutValue in the registry to be (hex, which in decimal is 190) HKLM\SYSTEM\CurrentControlSet\service\Disk

–Service to Disable–
BitLocker Drive Encryption Service
Block Level Backup Engine Service
Desktop Window Manager Session Manager
Disk Defragmenter
Diagnostic Policy Service
Home Group Listener
Home Group Provider
IP Helper – Only if IPv6 is not being used.
Microsoft iSCSI initiator service
Microsoft Software Shadow Copy Provider – Not needed since System Restore is disabled
Offline Files
Secure Socket Tunneling Protocol Service
Security Center
SSDP Discovery
Superfetch
Tablet PC Input Service
Telephony
Themes
UPnP Device Host
Volume Shadow Copy Service – Not needed since System Restore is disabled
Windows Backup
Windows Defender
Windows Error Reporting Service
Windows Firewall
Windows Search
Windows Update
WLAN AutoConfig
WWAN AutoConfig

–Final Prep–
Run Disk Cleanup
Defragment Disk
Delete Event Logs
ipconfig /release

Shut down the VM and clone it. Use the clone as the source for full / linked clones.
Correctly mark, notate, and document the clone to prevent shenanigans.

–Policy / Domain GPO Preparations–

Policies can be applied on the Parent VM directly or through group policy.
For direct policy management on the parent VM, use gpedit.msc
For domain-based policy management, use Group Policy Management (gpmc.msc)

The ADM templates will need to be imported into a domain controller if using group policy.
All ADM templates can be found on a View Connection Server: \VMware\VMware View\Server\extras\GroupPolicyFiles
Adding the templates will populate the Computer Configuration > Policies > Administrative Templates > Classic Administrative Templates
**Note that they are considered classic because they are .ADM and not .ADMX which is the new standard from Windows 2008 and newer**

If a computer account is located in a special OU that has certain Group Policy settings applied for end users of those systems,
leverage loopback policy processing to ensure Group Policies are applied in the expected and preferred fashion.

Lots o’ options for policy options from View ADM templates. Here is a sampling to get started with:
Build to Lossless
Audio Bandwidth
Frame rate, max initial and min image quality

–Pool Preparations–

**While VM is powered off (only if using NetApp and VSC 4.1+) reclaim NTFS space inside the VM**

Power on the VM. Join to the domain and place the computer object into the correct OU.
Install the View Agent (note, it installs .NET Framework components which may require additional security patches)

If the VM will use disposable disk, remove the default user TEMP and TMP variables
Disposable disk should be 1.5 times RAM + a bit more for other temp files.
Ex: 2gb RAM = 4gb disposable disk

Install any applications that would need to be used for this pool.
Run Disk Cleanup
Defragment Disk
Delete Event Logs
ipconfig /release

If the VM will be used for Full Clones, power down, convert to a template.
If the VM will be used for Linked Clones, power down, and snapshot.

**If Office 2010 is installed, rearm before shutdown. Make sure you keep a count on how many times you rearm
Office 2010 as there is a limit of 5**

Consider when to refresh desktops (scripting required for Floating Desktop pools):
After a set number of days
After the delta disk has grown to a certain percentage of the total disk size

–Notes–

To use the local storage of an ESXi host for the VM swapfile, edit the configuration of
the master image and configure the swapfile location for “Store in the host’s swapfile
datastore”.

Prepare KMS server for activating Windows 7 and Office 2010 with KMS keys. Note that
depending on whether you choose to use quickprep or sysprep, you may need to prepare your
KMS server differently. I won’t go too deep into this but quickprep does not create new
CMIDs for Windows 7 so if your KMS has not reached 25 Windows 7 clients yet, it will never
activate your pool of linked-clones whether there’s 10 or 1000 of them. This means if
your KMS server is new, and you plan to use quickprep, deploy a pool of 25 Windows 7
desktops with sysprep so you can get your KMS server to start activating because once the
KMS reaches 25 clients, it doesn’t care about the CMID anymore.

ONTAP NFS volume creation for vSphere CLI commands

Posted on January 24, 2013 by michaeldmills

So as I’m sure everyone knows with Data ONTAP, there are a number of ways to provision storage. In the case of VMware vSphere there are a number of methods for creating a Flexvol and then exporting as NFS for proper mounting by ESXi. Doing so in System Manager and the Virtual Storage Console are both supported but I find they don’t have the same behavior. I thought it best to put in some info for people to see what I do at the CLI.

As a note, in the case of snapshots, I turn them off as I’m likely scheduling snapshots using the backup and recovery component of the VSC. Also the name of the volume in this example is nfs_management. You would substitute your volume name in place

#Creates a Flexvol 200GB in size, with the US english language, thin provisioned, inside aggr0
vol create nfs_management -l en_US -s none aggr0 200g

#Turns off Update Access Time when a file is read
vol options nfs_management no_atime_update on

#Turns off snapshots
vol options nfs_management nosnap on

#Disables the default snapshot schedule
snap sched nfs_management 0 0 0

#Change the snapshot reserve space to whatever percentage x%
snap reserve nfs_management x

#Turn on deduplication for a volume and set the schedule to automated
sis on /vol/nfs_management
sis config -s auto /vol/nfs_management

#Next, to export this volume as properly, edit the /etc/exports file and add the following line for it
#In this case a subnet is used to denote which hosts can mount this export
#Note: the no_root_squash equivalent on Data ONTAP is anon=0
#Additional Note: ESXi doesn’t care about nosuid which is normally appended in an export so I removed it
/vol/nfs_management -sec=sys,rw=192.168.3.0/24,root=192.168.3.0/24,anon=0

#With this added, have ONTAP rescan the exports file and update it’s exports:
exportfs -a

Now it should be mountable by ESXi.

Data ONTAP Edge Configuration Guide

Posted on January 8, 2013 by michaeldmills

This guide will document all the steps taken to configure Data ONTAP Edge in the lab. In considering the best way to create this guide, I had considered decrying it a Standard Template Construct (STC) like many of my recent postings. However, as I have not spent the considerable amount of time with Data ONTAP Edge as I have with the other STC topics, I feel that it is best to create this guide and then after a good long period of months revisit and promote to STC.

Data ONTAP Edge is built from Data ONTAP 8.1.1 7-mode. For all intents and purposes, it is identical to ONTAP in many ways. From a configuration standpoint, this guide shall begin after the first boot of Edge. (Note that after deploying the OVA, ONTAP Edge requires a power-on operation so that an initial configuration can occur. Once this is complete, a reboot occurs and ONTAP Edge is accessible and ready for configuration.)

Some notes about the virtual hardware and the appliance itself. Vol0 cannot be reduced in size so 30GB will always be consumed. And because it is the root volume it is set to the volume guarantee and cannot be thin provisioned. RAM cannot be increased on the virtual appliance as it will halt during boot indicating the incorrect size of RAM. Each of the two CPU cores that are used based on the 2 vCPUs for Edge will stay at 100% utilization. This is how it is designed and should not be worrisome. Upgrading the virtual hardware to Hardware Version 8 appears to have no effect.

Many of the configuration steps can be used on a physical NetApp FAS or V-series system, though keep in mind a physical system is a bit more complex what with all those disks and cables and such J

To start with, login with the root account using the console feature of the vSphere Client and slap the Enter key a few times. The terminal prompt should show the hostname and it will give you some breathing room away from all the messages written to the console. If you are unfamiliar with the CLI for Data ONTAP, it can be thought of as chatty but in a good way. Akin to termmon being enabled on a Cisco device, all messages (also logged to syslog) will be generated at the console. If you are also unfamiliar with Cisco’s termmon, consider yourself brought up to speed on how they both work at a high level! You may get irritated while trying to type and having the system spew information your way but I find it to be helpful, usually. Especially if there are issues it will start barking at you from the console.

First order of business is to add in the licenses received from NetApp for the services / protocols that will be used. One thing to note: these licenses that I received are only for 90 days. However, if the license is removed and re-added, the timer resets to 90. Additionally the data remains untouched from what I can tell. During the removal of the license the protocol turns off but after entering the license again the protocol resumes. So in theory the lab can run indefinitely, just need to shut down the VMs and make sure all I/O has ceased while doing this process.

The next configuration task should be to tackle the networking. Data ONTAP Edge is unique since it is deployed as a virtual appliance. There are four network interfaces provided (e0a, e0b, e0c, e0d). Because this is a virtual appliance, the requirement for proper hardware failover and bandwidth utilization falls to the ESXi host being used. For the lab, I have chosen to use the interfaces in the following way:

e0a – Management

e0b – NFS

e0c + e0d – Unused for now

On the ESXi host, I have dedicated one vmnic to each interface in use and have created a VM Network on each with the sole purpose of hosting this traffic. Configuring the /etc/hosts and /etc/rc file is crucial for the networking configuration to persist on reboot. A common “oops” moment is to use CLI commands like ifconfig to set the networking up but forgetting to write them into /etc/hosts and /etc/rc. A reboot comes at some point and the Filer disappears from the network. “oops” indeed…

CDP does not work which is not a surprise because again Edge is running as a virtual appliance. As such, it cannot communicate directly with the switch. Enabling promiscuous mode on the virtual switch does not make a difference.

To edit these files from the CLI there is an awkward method that works but isn’t terribly intuitive. There are two commands used: rdfile and wrfile.

rdfile – can be used like the Linux “more” command to read a file. Syntax: EDGE> rdfile /etc/rc

wrfile – is used to edit a file. Syntax: EDGE> wrfile /etc/rc **Note, when this command is issued, the cursor will drop to the next line with no prompt. Paste into the terminal window the contents of the file. There is no direct editing in the sense of a normal editor like nano or vi. The wrfile is a COMPLETE replacement of the file in question. My approach is to copy the text from an editor that doesn’t modify the text structure and formatting (Textwrangler on the Mac) and paste into the terminal window. After pasting, I hit enter to go down to a blank line. Then to save and close, CTRL+C. There will be an error displayed “read: error reading standard input: Interrupted system call”. This is normal. ALWAYS check your work with rdfile afterwards to ensure the file is correct.

There are easier methods to edit these files. On way is to mount the vol0 straight from the Filer to a Windows, Mac or Linux/UNIX workstation. This allows for direct file manipulation and makes for easier management. To mount /etc/ for a Mac is easy enough:

In Finder, Go > Connect to Server and type in:

nfs://192.168.1.8/vol/vol0

To mount with a Windows workstation using CIFS (CIFS must be setup), use Windows Explorer and browse to \\ip_address\c$\etc .

Samples of the configuration files I am using:

/etc/hosts

127.0.0.1 localhost localhost-stack
127.0.10.1 localhost-10 localhost-bsd
127.0.20.1 localhost-20 localhost-sk
192.168.1.8 EDGE EDGE-e0a-mgmt
192.168.3.8 EDGE EDGE-e0b-nfs
# 0.0.0.0 EDGE-e0c
# 0.0.0.0 EDGE-e0d

/etc/rc

hostname EDGE
ifconfig e0a `hostname`-e0a-mgmt mediatype auto flowcontrol full mtusize 1500
ifconfig e0b `hostname`-e0b-nfs mediatype auto flowcontrol full mtusize 1500
route add default 192.168.1.1 1
options dns.enable on
options nis.enable off
savecore

There are additional files that are worth referencing. During the initial configuration for Data ONTAP (including Edge), a CLI command “setup” can be run which runs through a quick configuration script. Based on the inputs provided (hostname, IP address, and an assortment of others) a basic configuration can be ready to go to allow for ease of access. However, my preference is to manually configure everything as then I don’t miss any steps. Setup affects the following files (which in turn are then edited manually):

/etc/rc
/etc/exports
/etc/hosts
/etc/hosts.equiv
/etc/dgateways
/etc/nsswitch.conf
/etc/resolv.conf

Having already discussed /etc/rc and /etc/hosts, let’s move along to /etc/exports. This file represents all nfs exports and respective settings that are exported from Data ONTAP. The /etc/resolv.conf file contains each DNS server used by Data ONTAP. They are listed in the format with each entry on a separate line:

nameserver 192.168.1.4

The /est/hosts.equiv file is used to control access via rsh, ssh, and telnet to the Filer. /etc/dgateways has been deprecated and was previously used to maintain the default gateway. That is now handled by the statement in /etc/rc. The /etc/nsswitch.conf contains the order of services used for name resolution.

**Pro-tip: to seek out even more (better ha!) details for each of these files, simply use the man CLI command in front of each file name to learn more. This is just like the man command found in Linux/UNIX/Mac OSX.

Examples:

man nsswitch.conf
man rc
man resolv.conf

Understanding these configuration files will help to allow for ease of configuration and troubleshooting of any Data ONTAP system, including Edge.

Now I turn to specific OPTIONS configurations within ONTAP. This section will also be a catch-all for the remainder of configuration steps taken.

Some quick notes about the options and how Data ONTAP is configured. The options command (when typed at the command line) will list out all of the available options that are configurable.

Typing options keyword will display all the available configurable elements associated. An example of output:

There are a great deal of options and I highly suggest taking notes on them as a part of learning. The following options are what I have set for Data ONTAP Edge here in the lab. A real production Filer will require configuration customized to it’s environment.

Autosupport – I disable autosupport since this is my lab and it has no need for sending information to NetApp. On a production array I STRONGLY suggest making sure this is configured and working as it can be incredibly helpful when trouble starts.

options autosupport.enable off
options autosupport.support.enable off

Disable snapshots at the Aggregate level – I haven’t seen a need for these as they just consume space unnecessarily.

snap sched –A aggr0 0 0

Setup NTP – I can never stress how important it is to set proper time!

options timed.servers server1 server2
options timed.enable on
timezone Americas\Detroit
- Note that this uses the UNIX-style of timezone and is very particular about spelling

Disable NFS v2 – the lab is primarily being used by VMware vSphere and Red Hat / CentOS

options nfs.v2.enable off

Disable SSL v2 – sets connections to use SSL v3 only

options ssl.v2.enable off

Disable webdav – since Edge will not be used for any web serving

options webdav.enable off

Enable NFS vStorage – required to utilize the VAAI NFS plugin

options nfs.vstorage.enable on

For the most part out of the box, Data ONTAP Edge is well configured. There are certain configurations that pertain to the specific protocols that can be adjusted (NFS, iSCSI, CIFS) as well as if Edge will be joined to an Active Directory Domain. I will put together a provisioning guide that will detail the steps needed to configure and provision for each protocol. Plus if I can be verbose (no challenge there, ha!), I should be able to get three separate blog posts from it. And that helps to be more concise as well with the topics.

So at this point Data ONTAP Edge has been configured and is ready for provisioning and connectivity. The next documents will discuss provisioning storage to hosts. Further documents will cover tools and commands to use for gathering information related to Filer health, utilization, and performance.

Data ONTAP Edge Network and Performance Testing

Posted on January 1, 2013 by michaeldmills

The goal of this testing is to better understand the deployment strategy of the Data ONTAP Edge appliance. Networking plays a huge part of any good storage deployment and this would be no exception. Granted there aren’t as many options for networking Edge but the hope is to identify something that could qualify as “Best Practice”.

First a review the /etc/hosts and /etc/rc file. This file will list out the names for each interface that are referenced in the /etc/rc file for networking.

/etc/hosts:

/etc/rc:
hostname EDGE
ifconfig e0a `hostname`-e0a-mgmt mediatype auto flowcontrol full mtusize 1500
ifconfig e0b `hostname`-e0b-nfs mediatype auto flowcontrol full mtusize 1500
route add default 192.168.1.1 1
routed on
options dns.enable off
options nis.enable off
savecore

For testing networking on Data ONTAP Edge, there are several configurations that have been attempted. All tests will be based on VMware vSphere 5.0 Update 2 and will utilize VMware IOAnalyzer 1.5. Switching is provided by a stacked pair of Cisco 3750g-24ts running IOS 12.2(55) SE6 (most recent). Physical storage comes from 8 x 73GB 10k 2.5″ SAS drives in a RAID 0 via an HP P800 RAID Controller. Of note, using Firefox 17.0.1 resulted in I/O Analyzer not working correctly so Google Chrome it is!

Testing environment:

A single NFS volume mounted by one ESXi host. One IOAnalyzer will have a 50gb virtual disk located on the NFS volume. The NFS subnet is 192.168.3.x/24. The ESXi host where IOAnalyzer is running has a virtual standard switch setup with two vmnics using a single vmkernel interface for NFS. The virtual switch is setup for IP Hash and there is a port channel created on the 3750. On the ESXi host where Data ONTAP Edge is running, one vmnic is dedicated to e0a for management and one vmnic is dedicated to e0b on Edge for NFS on separate virtual standard switches.

Testing consists of running MAX_WRITE_IOPS, MAX_IOPS, and MAX_THROUGHPUT for 5 minutes to get a baseline. I believe these tests run sequential and not random so keep that in mind with the results.

Test Results:

MAX_WRITE_IOPS

Seems pretty good considering! Granted it’s RAID 0 underlying but still, I’m pleased by this. Data ONTAP Edge CPU utilization looks to be 60% to ~75%.

MAX_IOPS

Quite respectable! CPU utilization for Data ONTAP Edge ranges from 40% to ~55%.

MAX_THROUGHPUT

My understanding for what the maximums are for gigabit ethernet, this comes close to maxing out the available throughput (125MB/s would be the max).

Conclusions:

Examining the data, the MAX_IOPS test only uses about 10% of the available bandwidth while still maxing out the available IOPS. I can imagine that lab workloads will be a mixture of throughput and IOPS, but likely the limitation is going to be due to the physical disks and possibly RAID controller.

Considering the lab environment, there will be four ESXi hosts communicating with Data ONTAP Edge. A question would be if there is any benefit to using a virtual distributed switch and Load-Based Teaming on the ESXi host running Data ONTAP Edge.

I’m assembling the lab to get VMware View going so there is a good chance I’ll have the opportunity to see how far I can push Edge to the…. edge…. Ha! New Year’s humor J

Data ONTAP Edge installation

Posted on December 31, 2012 by michaeldmills

For my home lab, I have previously been using FreeNAS and was happy with the performance. My hardware platform consists of:

HP DL385 G2
2 x Dual Core Opteron 2218
32GB RAM
8 x 73GB 10k 2.5″ SAS drives (internal) setup as RAID 0 initially
HP P800 RAID Controller
6 x 1GB NIC

Don’t let the G2 moniker fool you as HP (I believe) chose G2 because of the DL385 platform was newer at the time. This server and RAM were inexpensive on eBay and when I was using it with FreeNAS 8.0.x it worked out well for NFS and iSCSI. However with the release of FreeNAS 8.3, I ran into problems accessing Volume Manager from the web interface. This led me to contemplate my storage choices for the home lab.

Sure, it would be great to pickup a FAS2040: I could run newer releases of ONTAP (8.1.x), and it’s a solid platform for the home lab. However, that can be a pricey endeavor, especially when it came time to add shelves. That led me to consider Data ONTAP Edge. I figured it would be great to try out though I wasn’t expecting how AMAZING this product is. But more on that later ha!

Data ONTAP Edge Platform Requirements
To function properly, the hardware platform must meet the following system requirements. Data ONTAP Edge requirements:

* Two dedicated physical CPU cores
* 4GB dedicated memory

* Minimum 57.5GB disk space for the Data ONTAP Edge system

Minimum server requirements:

* Quad core, or two dual-core, 64-bit Intel® x86 processors

* 2.27GHz or faster processor speed

* Greater than 4GB memory (recommend 8GB or more)

* Four or more physical disks on the server

* Single Gigabit Ethernet network

* Hardware RAID – must have a battery-backed write cache

The power management BIOS settings must be disabled on the server where you are going to install Data ONTAP Edge. If these BIOS settings are not set correctly, the virtual machine is installed, but it won’t start.

* CPU power management (pstates) must be disabled. Some CPUs may use different terminology for this setting, such as “performance states.”

* Idle states (cstates) must be disabled.

After the BIOS settings are set, they can be confirmed through the VMware vSphere client under the Configuration tab. Active Policy must be listed as Not supported. If this value is not correct, check the BIOS settings and restart the server.

In the case of the DL385 G2, I went into the BIOS and changed the power control mode from OS Control to HP Static High Performance and disabled the cstates. Looking at the host using the vSphere Client in the Configuration tab > Power Management it reports Notavailable and Notsupported respectively.

To deploy the OVA through the vSphere Client, connect to the vCenter that manages the ESXi host where Data ONTAP Edge will reside.

* Highlight the host and then File > Deploy OVF Template
* Select the OVA file and click Next
* The Template Details screen will be presented, click Next
* Accept the license agreement and click Next
* Give a name for the Edge appliance and click Next
* Make sure to select Thick Provision Lazy Zeroed for the Disk Format and click Next
* Provide the host name, ip address, subnet mask, gateway, root password for ONTAP, and how large the Managed Storage should be and click Next
* A summary screen will be provided with all selected values. Click Next to deploy Edge.

Some Notes:

* Create a DNS entry before hand and a computer object if ONTAP Edge will be configured for Active Directory
* Rule of thumb for the storage is that vol0 will be initially sized at 30GB
* Leave some space remaining in the local datastore in case ESXi is installed onto the same local datastore
* To remove false alarms, update the alarm threshold for the local datastore (due to the high space utilization)
* Additionally because of the way Data ONTAP Edge uses the underlying physical hardware, both physical CPU cores associated with the vCPUs will run at 100% utilization. My understanding is this is to help reduce any latency in response time from Edge. As such, removing the alarm for high cpu utilization for the Edge virtual machine can help to remove a false positive.

Once Data ONTAP Edge has been deployed, a quick review of the virtual hardware shows the following of note:

2 vCPU (2 socket, 1 core each)
4GB RAM
4 x LSI SAS SCSI Controllers
1 x SCSI virtual disk (SCSI ID 0:0) of the size specified during the OVA deployment
3 x IDE hard drives of sizes 1GB, 1.5GB, and 5GB respectively
4 x E1000 network adapters
Hardware Version: 7
Considered a FreeBSD 64bit OS

An initial configuration will occur once Data ONTAP Edge is powered on for the first time. After this configuration completes, Data ONTAP Edge is ready for use!

The next article that I do will discuss configuration potentials, methods for connectivity and management of Data ONTAP Edge, and storage management and provisioning.

After that, I’ll assemble an article doing performance testing using the different configurations to see which has an impact and to find out how much tuning / tweaking can be done (and makes sense!) for Edge. I’ll also see if modifying the amount of virtual hardware for RAM makes a difference (since this host has lots to spare!).

In a simple deployment testing with practically no configuration and one big NFS volume, I was able to get 21k MAX_IOPS using VMware’s IOanalyzer 1.5 fling (http://labs.vmware.com/flings/io-analyzer).

For the record, I love Data ONTAP! Such a great storage operating system!

Mike as a Service

The delivery of Mike as a solution over the internet