爱华网Someoneonce likened esxtopto windows task manageron steroids. When you have a look at the monitoring optionsavailable you can see why someone would think that.
Esxtop is basedon the ‘nixtopsystemtool, used to monitor running applications, services and systemprocesses. It is an interactive task manager that can be used tomonitor the smallest amount of performance metrics of the ESX(i)host.
Esxtop works onESX and it’s cousin remote esxtop or resxtop is for ESXi,although resxtop will work on ESX as well.
To run esxtopfrom ESX all you need to do is type esxtop from the commandprompt.
To run resxtop you need tosetup a vMAapplianceandthen run resxtop –server and enter the username and password whenprompted. You don’t need root access credentials to view resxtopcounters, you can use vCenter Server credentials.
The commandsfor esxtop and resxtop are basically the same so I will describethem as one.
Monitoring withresxtop collects more data than monitoring with esxtop soCPU usage may be higher. This doesn’t mean this esxtop isn’tresource intensive, quite the opposite. In fact it can use quite alot of CPU cycles when monitoring in a large environment, ifpossible limit the number of fields displayed. Fields are alsoknown as columns and entities (rows) You can also limit the CPUconsumption by locking entitiesand limiting thedisplay to a single entityusingl.
Esxtop usesworlds and groups as the entities to show CPU usage. A world is anESX Server VMkernel schedulable entity, similar to a process orthread in other operating systems. It can represent a virtualmachine, a component of the VMkernel or the service console.A group contains multiple worlds.
Countervalues
esxtopdoesn’t create performance metrics, it derives performancemetrics from raw counters exported in the VMkernel system infonodes. (VSI nodes) Esxtop can show new counterson older ESX systems if the raw counters are present in VMkernel.(i.e. 4.0 can display 4.1 counters)
Many raw counters have static values that don’t change with time.A lot of counters increment monotonically, esxtop reports the deltafor these counters for a given refresh interval. For instance thecounters for CMDS/sec, packets transmitted/sec and READS/secdisplay information captured every second.
Counternormalisation
By default counters are displayed for the group, in group view thecounters are cumulative whereas in expanded view, counters arenormalised per entity. Because of the cumulative stats, percentagedisplayed can often exceed 100%. To view expanded stats,presse.
Snapshots
esxtop takes snapshots. Snapshots are every 5 seconds by default.To display a metric it takes two snapshots and makes a comparisonof the two to display the difference. The lowest snapshot value is2 seconds. Metrics such as %USED and %RUN show the CPU occupancydelta between successive snapshots.
Manual
The manual for (r)esxtop is full of useful examples, and unlikenormal commands run on ESXi it will work if you run it from the vMAin the same way that it does from ESX.
To use justtypede>manesxtopde>orde>manresxtopde>.
Commands
Commands for (r)esxtop can be split into two distinct types,running commands and interactive commands
Running commands
These are commands that get placed at the end of the initial callof (r)esxtop.
Example
de>#de>de>esxtop-d 05de>
This commandwould set a delay on the refresh rate of 5 seconds.
Runningcommandsexamples
de>-dde>–setsdelay.
de>-ode>–sets order ofColumns.
de>-bde>–batch mode – I will explain this in further detailbelow.
Interactivecommands
These are commands that work on a key press once (r)esxtopis up and running. If in any doubt of whichcommand to use justhithforhelp.
Interactivecommandsexamples
de>cde>–CPU resourceutilisation.
de>mde>–memory resourceutilisation.
de>dde>–storage (disk) adapter resourceutilisation.
de>ude>–storage device resourceutilisation.
de>vde>–storage VM resourceutilisation.
de>fde>–displays a panel for adding or removing statistics columns to orfrom the currentpanel.
de>nde>–network resourceutilisation.
de>hde>–help.
de>ode>–displays a panel for changing the order ofstatistics.
de>ide>–interrupt resourceutilisation.
de>pde>–power resourceutilisation.
de>qde>–quit.
de>sde>–delay of updates in seconds. (can use fractionalnumbers)
de>wde>–write current setup to the config file. (esxtop4c)
From the fieldselection panel, accessible by pressing o you can move a field tothe left by pressing the corresponding uppercase letter and you canmove a field to the right by pressing the corresponding lowercaseletter. The currently active fields are shown inuppercase letters, and with an asterix to show it is selected inthe order field selection screen.
Batch mode (-b)
Batch mode is used to create a CSV file which is compatible withMicrosoft perfmon and ESXplot. For reading performance filesESXplot is quicker than perfmon. CSV compatibility requires a fixednumber of columns on every row. Because of this reason statisticsof vm (world) instances that appear after starting the batch modeare not collected. Only counters that are specified in theconfiguration file are collected. Using the -a option collects allcounters. Counters are named slightly differently to be compatiblewith perfmon.
To use batch mode select the columns you want in interactive modeand then savewithde>Wde>, thenrun
de>#de>de>esxtop-b -n 10 >esxtopfilename.csvde>
Options
de>-ade>–all.
de>-bde>–Batchmode.
de>-cde>–User defined configurationfile.
de>-dde>–Delay between statistics snapshots. (minimum 2seconds)
de>-nde>–Number of iterations before exiting.
To read thebatch mode output file is to load it in Windows perfmon.
(1) Run perfmon
(2) Type “Ctrl + L” to view log data
(3) Add the file to the “Log files” and click OK
(4) Choose the counters to show the performancedata.
Esxtop readsit’s default configuration from a .esxtopp4rc file. This filecontains 8 lines. The first 7 lines are upper and lowercase lettersto specify which fields appear, default is CPU, memory, storage,adapter, storage device, virtual machine storage, network andinterrupt. The 8th line contains other options. You can saveconfiguration files to change the default view with inesxtop.
Replay mode
Replay mode interprets data that is collected by issuing thevm-support command and plays back the information as esxtopstatistics. replay mode accepts interactivecommands until no more snapshots are collected by vm-support.Replay mode does not process the output of batchmode.
To use replaymode run
de>#de>de>vm-support-S -i 5 -d 60de>
untar the fileusing
de>#de>de>tar-xf /root/esx*.tgzde>
thenrun
de>#de>de>esxtop-R root/vm-support*de>
Commands
de>-Sde>–Snapshot mode, prompts for the delay between updates, inseconds.
de>-Rde>–Path to the vm-support collected snapshot’s directory.
CPU
CPU load average of 1.00 means full utilisation of all CPU’s. Aload of 2.00 means the host is using twice as many physical CPU’sas are currently available, likewise 0.50 means half are beingutilised.
CPU Screen
PCPU USED(%)–Physical hardware execution context. Can be a physical CPU core ifhyperthreading is unavilable or disabled or a logical CPU (LCPU) orSMT thread if hyperthreading is enabled. This displays PCPUpercentage of CPU usage when averaged over all PCPUs.
PCPU UTIL(%)-Physical CPU utilised. (real time) Indicates how much time the PCPUwas busy, in an unhalted state, in the last snapshot duration.Might differ from PCPU USED(%) due to power management technologiesor hyperthreading.
If hyperthreading is enabled these figures can be different, likewise ifthe frequency of the PCPU is changed due to power management thesefigures can also be adjusted.
As an exampleif PCPU USED(%) is 100 and PCPU UTIL(%) is 50 this is because hyperthreading is splitting the load across the two PCPUs. If you thenlook in the vSphere client you may notice that CPU usage is 100%.This is because the vSphere client will double the statistics ifhyperthreading is enabled.
In a dual core system, each PCPU is charged by the CPU schedulerhalf of the elapsed time when both PCPUs are busy.
CCPU(%)–Total CPU time as reported by ESX service console. (Not applicableto ESXi)
us – Percentage user time.
sy – Percentage system time.
id – Percentage idle time.
wa – Percentage wait time.
cs/sec – Context switches per second recorded by the ESX ServiceConsole.
CPU panel statistics(c)
Fields
ID – resource pool or VM ID of the running worlds resource poolor VM or world ID of running world.
GID – Resource pool ID of the running worlds resource pool orVM.
NAME – err… name.
NWLD – Number of members in a running worlds resource pool orVM.
%USED – CPU core cycles used.
%RUN – CPU scheduled time.
%SYS – Time spent in the ESX(i) VMkernel on behalf of theresource pool, VM or world to processor interrupts.
%WAIT – Time spent in the blocked or busy wait state.
%RDY – Time CPU is ready to run, waiting for somethingelse.
High %RDY andhigh %USED can imply CPU overcommitment.
Additional fields
%IDLE – As it says. Subtract this from %WAIT to see time waitingfor an event. WAIT-IDLE can be used to estimate guest I/O waittime.
%MLMTD (max limited – Time VMkernel didn’t run because it wouldviolate limit settings on the resource pool, VM or world limitssetting.
%SWPWT – Wait time for swap memory.
CPU ALLOC –CPU allocation. Set of CPU statistics made up ofthe following. (For a world the % are the % of one physicalCPU core)
AMIN – Attribute reservation.
AMAX – Attribute limit.
ASHRS – Attribute shares.
SUMMARY STATS– Only applies to worlds.
CPU – Which CPU esxtop was running on.
HTQ – Indicates whether a world is currently quarantined or not.(Y or N)
TIMER/s – Timer rate for this world.
%OVRLP – Time spent on behalf of a different resource pool/VM orworld while the local was scheduled. Not included in %SYS.
%CSTP – Time the vCPUS of a VM spent in the co-stopped state,waiting to be co-started. This gives an indication of theco-scheduling overhead incurred by the VM. If this value is highand the CPU Ready time is also high, this represents that the VMhas too many vCPUs. If low, then any performance problems should beattributed to other issues and not to the co-scheduling of theVM’s vCPU.
Single key displaysettings
de>ede>–expand. Displays utilisation broken down by individual worldsbelonging to a resource pool or VM. All %’s are for individualworlds of a single physicalCPU.
de>ude>–Sort by %USEDcolumn.
de>rde>–Sort by%RDY.
de>nde>–Sort by GID.(default)
de>vde>–VM instancesonly.
de>lde>–Length of NAME column.
CPU clock frequencyscaling
%USED– CPU usagewith reference to the base core frequency, i.e. the actual CPUvalue in Mhz.
%UTIL– CPUutilisation with reference to the current clock frequency.(displayed as %)
%RUN– Total CPUscheduled time. Displayed as %. If using turbo boost will showgreater than 100%.
%UTIL may bedifferent if turbo boost in enabled. To better define this, if aCPU has a base core clock speed of 1Ghz, with turbo boost is 1.5Ghzthen %USED (with turbo boost enabled) is 150%. So the current CPUspeed (%UTIL) will be that 150% displayed as a value of 100%. Nowif the current used CPU is 1Ghz then the current %UTIL will be 50%and not 100%. (as per the base core frequency) Consider this whenmonitoring these stats.
Interrupt panel (i)
VECTOR – Interrupt vector ID.
COUNT/s – Interrupts per second on CPU x
TIME/int – Average processing time per interrupt. (in microseconds)
TIME_x – Average processing time on CPU. (in micro seconds)
DEVICES – Devices that use the interrupt vector. If the interruptvector is not enabled name is in < >brackets.
Memory
The followingcounters and statistics assume a basic understanding of memorymanagement in a virtualised environment. Checkout myUnderstandingvirtual machine memoryguidefor a brief overview of memory management.
Memory screen (m)
PMEB(MB)– Machinememory statistics.
Total – Yup you guessed it, total.
COS – Amount allocated to the service console.
VMK – Machine memory being used by the ESX(i) VMkernel.
Other – Everything else.
Free – Machine memory free
VMKMEM(MB)–Statistics for VMkernel in MB.
Managed – Total amount.
Min free – Minimum amount of machine memory VMKernel aims to keepfree.
RSVD – Reserved by resource pools.
USVD – Total unreserved.
State – Values are high, soft, hard, low. (Pressure states)
COSMEM(MB)–Statistics as reported by the service console.
Free – Amount of idle memory.
Swap_t – Total swap configured.
Swap_f – Swap free.
r/s is – Rate at which memory is swapped in from disk.
w/s – Rate at which memory is swapped to disk.
NUMA(MB)–Only if running on a NUMA server.
PSHARE (MB) – Page sharing.
shared – Shared memory.
common – Across all worlds.
saving – Saved due to transparent page sharing.
SWAP(MB)
curr – Current.
target – What the ESX(i) system expects the swap usage tobe.
r/s – swapped from disk.
w/s – swapped to disk.
MEM CTL(MB)–Balloon statistics.
curr – Amount reclaimed.
target – Host attempt reclaims using the balloon driver,vmmemctl.
max – Maximum amount the host can reclaim usingvmmemctl.
Fields
AMIN – Memory reservation.
AMAX – Memory limit. A value if -1 means unlimited.
ASHRS – Memory shares.
NHN – Current home node for resource pool or VM. (NUMAonly)
NRMEM (MB) – Current amount of remote memory allocated. (NUMAonly)
N% L – Current % of memory allocated to the VM or resource poolthat’s local.
MEMSZ (MB) – Amount of phyiscal memory allocated to a resourcepool or VM.
GRANT (MB) – Guest memory mapped.
SZTGT (MB) – Amount the VMkernel wants to allocate.
TCHD (MB) – Working set estimate.
%ACTV – % guest physical memory referenced by the guest.
%ACTVS – Slow moving version of the above.
%ACTVF – Fast moving.
%ACTVN – Estimation. (This is intended for VMware useonly)
MCTL – Memory balloon drive installed or not. (Y/N)
MCTLSZ (MB) – Amount of physical memory reclaimed byballooning.
MCTLTGT (MB) – Attempts to reclaim by ballooning.
MCTLMAX (MB) – Maximum that can be reclaimed by ballooning.
SWCUR (MB) – Current swap.
Interactive
de>mde>–Sort by group mappedcolumn.
de>bde>–sort by group Memchcolumn.
de>nde>–sort by group GID column.(Default)
de>vde>–Display VM instancesonly.
de>lde>–Display length of the NAME column.
Network
The networkstats are arranged per port of a virtual switch.
PORT-IDidentifies the port and DNAME shows the virtual switch name. UPLINKindicates whether the port is an uplink. If the port is an uplink,i.e., UPLINK is ‘Y’, USED-BY shows the physical NIC name. Ifthe port is connected by a virtual NIC, i.e., UPLINK is ‘N’,USED-BY shows the port client name.
Network panel (n)
Fields
PORT-ID – Port ID.
UPLINK – Uplink enabled.(Y or N)
UP – Guess what.
SPEED – Link in MB.
FDUPLX – Full duplex.
USED-BY – VM device port user.
DTYP – Virtual network device type. (H=hub, S=switch)
DNAME – Device name.
Interactive
de>Tde>–Sort by Mbtransmitted.
de>Rde>–Sort by Mbreceived.
de>tde>–Packetstransmitted.
de>rde>–Packetsreceived.
de>Nde>–Port-ID.(default)
de>Lde>–Length of DNAME column.
Storage
Storage Panelsde>dde>- diskadapter.
de>ude>- diskdevice. (also includes NFS if ESX(i) host is 4.0 Update 2 orlater)
de>vde>- diskVM.
An I/O requestfrom an application in a virtual machine traverses through multiplelevels of queues, each associated with a resource of some sort,whether that is the guest OS, the VMkernel or the physical storage.Each queue has an associated latency.
Esxtop shows the storage statistics in three different screens; theadapter screen, device screen and vm screen.
By default data is rolled up to the highest level possiblefor each screen. On the adapter screen thestatistics are aggregated per storage adapter by default, but thecan be expanded to display data per storage channel, target,path or world using a LUN.
On the device screen statistics are aggregated per storage deviceby default and on the VM screen, statisticsareaggregatedon a per-groupbasis. One VM hasonecorrespondinggroup,so they areequivalentto per-vmstatistics. Use interactive command V to showonlystatisticsrelated toVMs.
Queue Statistics
AQLEN – The storage adapter queue depth.
LQLEN – The LUN queue depth.
WQLEN – The World queue depth.
ACTV – The number of commands in the ESX Server VMKernel that arecurrently active. QUED The number of commands queued.
LOAD – The ratio of the sum of VMKernel active commands andVMKernel queued commands to the queue depth.
%USD – The percentage of queue depth used by ESX Server VMKernelactive commands.
%USD = ACTV /QLEN * 100%
I/O throughputstatisticsCMDS/s– Number of commands issued per second.
READS/s – Number of read commands issued persecond.
WRITES/s – Number of write commands issued per second.
MBREAD/s – MB reads per second.
MBWRTN/s – MB written per second.
I/O latencies
I/O latencies are measured per SCSI command so it is not affectedby the refresh interval. Reported latencies are average values forall the SCSI commands issued within the refresh interval window.Reported average latencies can be different on different screens,(adapter, LUN, VM) since each screen accounts for different groupof I/O’s.
Latency statistics
This group of counters report latency values.These are under the labels GAVG, KAVG and DAVG.GAVG is the sum of DAVG and KAVG.
KAVG+DAVG=GAVG
GAVG –round-trip latency that the guest sees for all IO requests sent tothe virtual storage device.(should be under 25)
KAVG – latencies due to the ESX Kernel’s command. should besmall in comparison to DAVG DAVG latency seen at the device driverlevel. includes the roundtrip time between the HBA and the storage.(should be 2 or less)
QAVG – average queue latency. QAVG is part of KAVG (should bezero)
Storage adapter
CID – Channel ID.
TID – Target ID.
LID – LUN ID.
Interactive
e–Expand/rollup storage adapterstatistics.
de>pde>–Same asebutdoesn’t roll up to adapterstatistics.
de>ade>–Expand/rollup storage channelstatistics.
de>tde>–Expand/rollup storage targetstatistics.
de>rde>–Sort byREAD/s.
de>wde>–Sort byWRITES/s.
de>Rde>–sort byMBREADS/s.
de>Tde>–Sort by MBWRTN/s.
