XenServer Host Failure

XenServer Host Failed

Find the UUID of failed server
xe host-list

Find out which VMs were running on failed server
xe vm-list resident-on= <UUID of XenServer>

Tell pool that the VMs are powered off
xe vm-reset-powerstate resident-on= <UUID of XenServer> --force --multiple

This will reset the power for the VMs that were running on this XenServer so that we can now see them in XenCenter

Steps to address a hung XenServer virtual machine:

Find the UUID of the hung virtual machines. This is done from the command line of the XenServer hosting the problem virtual machine. The command to execute is xe vm-list. Another way to find the UUID of the virtual machine is to click the ‘General’ tab of the virtual machine in XenCenter.

Once you have the UUID you need to find the domain ID of the hung virtual machine. This is done by executing the command list_domains from the XenServer hosting the problem virtual machine. You will need to match the UUID with domain ID number. The output of this command will look like this:

id |                                 uuid |  state

0 | 2fe455fe-3185-4abc-bff6-a3e9a04680b0 |     R

47 | 267227f3-a59e-dafe-b183-82210cf51ec4 |    B

59 | 298817fb-8a3e-7501-11e0-045a8aa860ff |    B

(i.e. UUID 267227f3-a59e-dafe-b183-82210cf51ec4 has a domain ID of 47)

To free the virtual machine we need to execute destroy_domain command on the domain ID. This is done by executing the command:

/opt/xensource/debug/destroy_domain –domid 47

Note: 47 is the domain ID

If the Host reports the vm in a halted state after running xe vm-list, try to reset the power.

To reset the power state run the following command:

xe vm-reset-powerstate vm=VM NAME force=true where VM Name is the server name.

Or

xe vm-shutdown vm=VMNAME force=true where  VM NAME is the server name.

Once the VM is shutdown, restart on another XenServer

xe vm-start vm=VMNAME

Sometime running commands using the XenServer command line will yield you more information about a problem that otherwise XenCenter may not reveal to you.

For example: If the VM is having issues starting; executing the command xe vm-start name-label= <Name of VM>  will likely get more information about what may be causing the issue.   

Also, sometimes a server will not report to its host after being migrated. It is either stuck in powering off (staying in yellow) or powering on.  Run this command on from the pool master for that pool.

xe-toolstack-restart

The VM should show back up. If not close out of XenCenter and reopen.

How to Setup XenServer 6.x to Auto-Start Virtual Machines

Setting the XenServer to allow Auto-Start

Gather the UUID’s of the pools you wish to auto-start.

To get the list of the pool’s on your XenServer type “xe pool-list”

Copy the UUID of the pool. If you have just one server, it will still have a pool UUID as noted in the following screenshot:

Then type the following command to set the pool or server to allow auto-start:
xe pool-param-set uuid=UUID other-config:auto_poweron=true

Note: Replacing UUID with the UUID of the XenServer or pool.

Setting the Virtual Machines to Auto-Start

Gather the UUID’s of the Virtual Machine you want to auto-start by typing: 

xe vm-list

Note: This generates a list of Virtual Machines in your pool or server and their associated UUID’s.

Copy the UUID of the Virtual Machines you want to auto-start, and type the following command for each Virtual Machine to auto-start: 

            

xe vm-param-set uuid=UUID other-config:auto_poweron=true

Note: Replace UUID with the UUID of the Virtual Machine to auto-start.

Steps to install NIC drivers on XenServer

Steps to install NIC drivers on XenServers

Check NIC driver version by running ethtool -i eth0

I created a directory in the tmp directory called iso. It does not have to be called iso this is just what I chose. I performed this using WinSCP

[root@fsxensvr14 ~]# cd /tmp/iso

Copy the NIC drivers to this new directory and then perform an ls to view the files in the directory

[root@fsxensvr14 iso]# ls

bnx2x-1.74.22.tar.gz  bnx2x.iso  bnx2x.iso.md5  bnx2x.metadata.md5

Mount the iso using the following command. You are mounting the iso to the /tmp/iso directory that you created earlier

[root@fsxensvr14 iso]# mount bnx2x.iso /tmp/iso -o loop,ro

Change directories to the /tmp/iso directory

[root@fsxensvr14 iso]# cd /tmp/iso

Perform an ls to view the files in the directory

[root@fsxensvr14 iso]# ls

bnx2x-modules-kdump-2.6.32.43-0.4.1.xs1.6.10.784.170772-1.74.22-1.i386.rpm

bnx2x-modules-xen-2.6.32.43-0.4.1.xs1.6.10.784.170772-1.74.22-1.i386.rpm

install

install.sh

XS-PACKAGES

XS-REPOSITORY

Execute install.sh by running the following command

[root@fsxensvr14 iso]# ./install.sh

Installing 'Broadcom bnx2x driver'...

Preparing...                ########################################### [100%]

   1:bnx2x-modules-xen-2.6.3########################################### [ 50%]

   2:bnx2x-modules-kdump-2.6########################################### [100%]

Pack installation successful.

Change directories to root by running the following command

[root@fsxensvr14 iso]# cd /

Unmount the /tmp/iso directory by running the following command

[root@fsxensvr14 /]# umount /tmp/iso

Clean UP !! Delete the files that you copied over to perform the installation.

Check the performance of your storage repository

Ref: http://support.citrix.com/article/CTX121634

[root@FsXENSVR109 ~]# ll /dev/disk/by-id/ | grep <SCSI ID>   NOTE: SCSI ID can be found on the 'General' tab of your SR in XenCenter

The output of this command will be something like the following:

lrwxrwxrwx 1 root root  9 Feb 12 19:16 scsi-36000d3100087d3000000000000000005 -> ../../sdb

Once you have this information you can execute the following command

[root@FsXENSVR109 ~]# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads: 248 MB in  3.06 seconds =  81.08 MB/sec
[root@FsXENSVR109 ~]# hdparm -t /dev/sdb

XenServer MultiPath Info

Useful commands discovered and used along the way. 

Check iSCSI sessions

iscsiadm -m session

Ping storage from a particular NIC IP address

ping -I xenbr8 192.168.190.10

Note: You may need to use this command to figure out where to ping from
brctl show
Show info on virtual switches

Rescan iscsi sessions

iscsiadm -m session -r 1 --rescan

Show all records in discovery database and show the targets that
were discovered from each record:
./iscsiadm -m discovery -P 1

Display session statistics:
./iscsiadm -m session -r 1 --stats

Display running sessions:
./iscsiadm -m session -P 1

Restart iscsi service
service open-iscsi restart

Check iscsi service status
service open-iscsi status

To get information about the running sessions: including the session and
device state, session ids (sid) for session mode, and some of the
negotiated parameters, run:
iscsiadm -m session -P 2

To get a Host view of the session there is the host mode.
iscsiadm -m host

Log into iSCSI targets
iscsiadm -m node --login

To log into an individual iSCSI target use:
iscsiadm -m node -l -T iqn.2001-05.com.equallogic:83bcb3401-16e0002fd0a46f3d-rhel5-test -p 172.23.10.240:3260

TFTP - cheat sheet

Here is a quick reference on how to backup/restore switch configurations...

A free TFTP server is available for download from Solarwinds website

To copy a switch configuration to a TFTP server run the following command:

copy running-configuration tftp://TFTP-IP/filename.txt

To copy a switch configuration from a TFTP server to switch run the following command:

copy TFTP://TFTP-IP/filename.txt startup-configuration

Then type:
reload