Monitor CPU Consumption using vmstat

In addition to the UNIX ps command, other OS utilities can be used to monitor CPU consumption. The most common UNIX utility is the vmstat utility. While the ps command shows the instantaneous CPU consumption of an individual process, vmstat shows the total CPU consumption for all of the CPUs on your database server.
Note: This course does not cover UNIX tools such as top and glance because they are not available on all UNIX platforms. These tools also can be used to monitor CPU consumption. The vmstat command provides valuable information about the state of our database server. There are five columns in vmstat that are important for you to learn:

The `vmstat` (virtual memory statistics) utility is a command-line tool available on Unix and Linux systems that provides a snapshot of system performance. It reports information about processes, memory, paging, block I/O, traps, and importantly, CPU activity. By using `vmstat`, you can monitor CPU consumption in real-time or over specified intervals, helping you diagnose performance issues and understand system behavior.
Using `vmstat` to Monitor CPU Consumption

1. Basic Usage: Running `vmstat` without any arguments displays a one-time report summarizing system activity since the last reboot:
   

   vmstat
   

   
   However, to effectively monitor CPU consumption, you should run `vmstat` with an interval, which updates the statistics at regular intervals.
2. Real-Time Monitoring: To monitor CPU usage at regular intervals, use the following syntax:
   

   vmstat [interval] [count]
   

   • `interval`: Time in seconds between each report.
   • `count`: Total number of reports to generate.
   
   For example, to display statistics every 1 second indefinitely:
   

   vmstat 1
   

   
   Or to display statistics every 2 seconds for a total of 10 updates:
   

   vmstat 2 10
   

3. Understanding the Output: The `vmstat` output is divided into several sections. The CPU-related statistics are found under the `cpu` section. Here's a sample output:
   

   procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
    2  0      0 102400  20480 307200    0    0     1     2  100  200 15 10 70  5  0
   

   
   Key CPU Columns:
   • `us` (User Time): Percentage of CPU time spent on user processes.
   • `sy` (System Time): Percentage of CPU time spent on kernel processes.
   • `id` (Idle Time): Percentage of CPU time spent idle.
   • `wa` (Wait Time): Percentage of CPU time waiting for I/O operations to complete.
   • `st` (Stolen Time): Percentage of CPU time stolen from a virtual machine (applicable in virtualized environments).
4. Interpreting CPU Usage"
   
   • High `us` Value: Indicates heavy usage by user-level applications.
   • High `sy` Value: Suggests significant CPU time consumed by system processes, possibly due to kernel operations.
   • Low `id` Value: Means the CPU is heavily utilized; low idle time.
   • High `wa` Value: Implies the CPU is waiting on I/O operations, which could be a bottleneck.
   • High `st` Value: In virtualized setups, a high stolen time may indicate the host system is overcommitted.
5. Practical Monitoring Steps:
   1. Step 1: Start monitoring with a 1-second interval:
      

      vmstat 1
      

   2. Step 2: Observe the `cpu` section, focusing on `us`, `sy`, `id`, and `wa` columns.
   3. Step 3: Identify patterns or spikes in CPU usage:
      • Consistently High `us` + `sy`: CPU-bound applications or processes.
      • Increasing `wa`: Potential I/O bottlenecks affecting CPU performance.
   4. Step 4: Press `Ctrl + C` to stop monitoring when done.

• Including Timestamps"
  Add the `-t` option to include a timestamp with each report:
  

  vmstat -t 1
  

• Extended Output Width: For a more readable output with wider columns, use the `-w` option:
  

  vmstat -w 1
  

• Monitoring Specific CPUs (If Supported)
  On systems that support per-CPU statistics, you can use:
  

  vmstat -P ALL 1
  

• `top` or `htop`: For a dynamic, real-time view of running processes and their CPU usage.
• `mpstat`: To display CPU statistics per processor.
• `sar`: For historical data collection and reporting.

Automated Monitoring and Logging
To log `vmstat` output to a file for later analysis:

vmstat 5 288 > vmstat_log.txt

This command collects data every 5 seconds for 24 minutes (288 intervals) and writes it to `vmstat_log.txt`.

You can create scripts that parse `vmstat` output and trigger alerts when CPU usage exceeds thresholds:

#!/bin/bash
threshold=80
while true; do
  cpu_idle=$(vmstat 1 2 | tail -1 | awk '{print $15}')
  cpu_usage=$((100 - cpu_idle))
  if [ "$cpu_usage" -gt "$threshold" ]; then
    echo "High CPU usage detected: $cpu_usage%"
    # Add notification logic here (email, SMS, etc.)
  fi
  sleep 5
done

• Snapshot vs. Continuous Monitoring: `vmstat` provides snapshots of CPU usage, which is useful for trend analysis but may miss transient spikes.
• Data Interpretation: Requires understanding of system behavior to accurately interpret the data.
• Virtual Environments: In virtualized systems, CPU statistics may be influenced by the hypervisor's scheduling.

Conclusion
By leveraging `vmstat`, you gain valuable insights into your system's CPU consumption patterns. Regular monitoring can help you:

• Identify Performance Bottlenecks: Spot processes or system activities that are consuming excessive CPU resources.
• Optimize System Performance: Make informed decisions about load balancing, process scheduling, and resource allocation.
• Proactive Troubleshooting: Detect issues before they impact system performance or availability.

Remember: While `vmstat` is a powerful tool, it is most effective when used in conjunction with other monitoring utilities and when you have a clear understanding of your system's normal operating parameters.

When you see 100% CPU usage in conjunction with run queue, (r) values greater than the number of CPUs, you have identified a CPU bottleneck.
A CPU (sy + us) at 100% does not necessarily mean you have a CPU problem. The OS is designed to make the CPUs perform at their top capacity and it is not uncommon to see busy CPUs that do not have a CPU bottleneck. Be sure to check this against the run queue values.

• Remedying CPU problems The only was to remedy an overloaded CPU is to:
  1. Reduce CPU consumption
     
  2. Add additional CPUs
     
  3. Upgrade the existing CPUs to faster processors
  
  The next lesson demonstrates techniques for changing CPU dispatching priorities.

First, click the Quiz link below to try interpreting the output from a vmstat session.
vmstat Output - Quiz