Smartctl: Auditing Storage Media Health, Predicting NVMe Drive Degradation, and Automating Hardware Telemetry in Production

Standard diagnostic tools like top, df, or iostat are blind to this kind of decay. They only see what the operating system sees, treating modern storage drives as simple black boxes that accept data and return results. When a drive begins to fail physically, it does not always announce itself with a fatal crash; more often, it retries reads hundreds of times behind the scenes, introduces massive latency spikes, and slowly corrupts data while pretending everything is normal.

To uncover what is actually happening, you have to bypass the operating system's file layers and speak directly to the microchip embedded inside the drive itself. The gold standard for this physical interrogation is smartctl, the command-line powerhouse from the open-source Smartmontools Official Project Documentation suite.

When you need an immediate answer to whether a suspect drive is on the verge of death, one simple command cuts through the noise:

smartctl -i -H /dev/nvme0n1

Running this command against a primary NVMe solid-state drive produces a swift identification profile alongside the drive controller’s overall health verdict, as documented in the smartctl(8) Manual:

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.13-production] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZQL21T9HCJR-00A07
Serial Number:                      S64VNY0T104921
Firmware Version:                   GDC7302Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,920,383,410,176 [1.92 TB]
Unallocated NVM Capacity:           0
Controller Busy Time:               1,420
Power Cycles:                       18
Power On Hours:                     14,892
Unsafe Shutdowns:                   4
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

In less than a second, smartctl retrieves the drive's serial number, total power-on hours, unsafe power cuts, and an overarching assessment test result: PASSED. But while a quick health check is an essential first triage step, surface-level health indicators can sometimes give a false sense of security. True storage resilience requires understanding how to read the deeper telemetry hidden within your drives.

What S.M.A.R.T. Does in Plain English

Every modern hard disk drive (HDD), solid-state drive (SSD), and Non-Volatile Memory Express (NVMe) card comes equipped with Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.). Think of S.M.A.R.T. as an onboard flight data recorder and medical monitor combined.

The drive's internal microcontroller continuously tracks electrical health, surface defects, operating temperatures, and flash cell endurance. Crucially, smartctl can also instruct the drive to run autonomous background self-tests. These diagnostic routines scan the physical magnetic platters or silicon flash blocks directly at the hardware level, without needing to unmount filesystems or interrupt live user applications.

Essential Flags and Quick Reference

The smartctl utility uses structured flags to translate protocol-specific telemetry across ATA, SCSI, and NVMe hardware into human-readable diagnostics:

5 Real-World Production Scenarios

Scenario 1: Running Non-Disruptive Background Self-Tests on Busy Storage Nodes

The Operational Context

An enterprise SAS hard drive operating inside a distributed storage cluster begins throwing intermittent slow-request warnings. You need to verify whether the magnetic surface of the disk platters or the mechanical read heads are degrading, but you cannot afford to take the storage daemon offline or disrupt live data traffic.

The Command Sequence

First, instruct the drive controller to initiate an extended background self-test across its entire storage surface:

smartctl -t long /dev/sdb

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.13-production] (local build)
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 380 minutes for test to complete.
Test will complete after Tue Aug 18 04:34:08 2026 UTC
Use smartctl -X to abort test.

Once the estimated testing window has elapsed, query the drive’s internal self-test history log:

smartctl -l selftest /dev/sdb

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     18492         0x004a8f10
# 2  Short offline       Completed without error       00%     18450         -
# 3  Extended offline    Completed without error       00%     17810         -
# 4  Short offline       Completed without error       00%     17200         -

Analytical Breakdown of Output

• Sending command: "Execute SMART Extended...: The host operating system dispatches a diagnostic request across the storage bus directly to the drive’s onboard microcontroller. Under the Linux Kernel Storage Subsystem Documentation, the operating system does not execute the scan; the drive manages the test independently during idle execution cycles.
• Please wait 380 minutes...: The drive calculates how long it needs to read every single physical sector. Normal application reads and writes take absolute priority—if a database query hits the disk during the test, the drive pauses its self-test, services the database, and resumes testing seamlessly.
• Num 1 Extended offline Completed: read failure Remaining 90%: Test entry #1 failed almost immediately upon encountering an unrecoverable physical read defect.
• LBA_of_first_error 0x004a8f10: The exact physical Logical Block Address where the read-channel hardware failed to recover data using built-in error-correcting codes.

Next Actions for the Systems Engineer

Because the drive suffered an unrecoverable read failure during non-destructive background polling, the physical media is actively decaying. Mark the storage daemon out of service in your cluster management console, allow the cluster to re-balance its data across healthy disks, and schedule an immediate physical replacement.

Scenario 2: Auditing NVMe Wear Endurance and Thermal Throttling on Database Servers

The Operational Context

A high-throughput PostgreSQL primary server handling immense write-ahead log traffic begins experiencing sudden, unexplained transaction latency spikes. You need to establish whether the NVMe solid-state drive is burning through its silicon write endurance or suffering from severe thermal throttling under heavy load.

The Command Sequence

Run a comprehensive diagnostic audit targeting the drive’s standardized NVMe health log:

smartctl -a -d nvme /dev/nvme1n1

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.13-production] (local build)
=== START OF SMART DATA SECTION ===
SMART / Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        74 Celsius
Available Spare:                    11%
Available Spare Threshold:          10%
Percentage Used:                    96%
Data Units Read:                    4,891,204,112 [2.50 PB]
Data Units Written:                 8,124,551,908 [4.15 PB]
Host Read Commands:                 194,120,441
Host Write Commands:                412,981,004
Controller Busy Time:               18,401 minutes
Power Cycles:                       42
Power On Hours:                     21,490
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    18
Error Information Log Entries:      18
Warning  Comp. Temperature Time:    142 minutes
Critical Comp. Temperature Time:    0 minutes
Temperature Sensor 1:               74 Celsius
Temperature Sensor 2:               88 Celsius
Thermal Management T1 Trans Count:  2412
Thermal Management T2 Trans Count:  0
Thermal Management T1 Total Time:   18420

Analytical Breakdown of Output

• SMART / Health Information (NVMe Log 0x02): Unlike legacy SATA disks with vendor-specific attribute numbers, modern NVMe devices follow the unified NVM Express Base Specification, making all metric definitions consistent across manufacturers.
• Critical Warning: 0x00: A bitmask status indicator. A non-zero value here signals imminent hardware emergencies (such as spare memory running out or drive reliability becoming compromised).
• Available Spare: 11% (Threshold: 10%): Solid-state drives keep a hidden pool of reserve flash memory to replace worn-out cells. This drive has burned through almost all its reserves; if it drops just 1% further, the drive will trigger a hardware warning alert.
• Percentage Used: 96%: An enterprise endurance measurement. At 96%, the drive has consumed nearly 100% of its rated write lifecycle after processing 4.15 petabytes of written data.
• Temperature: 74 Celsius & Sensor 2: 88 Celsius: Sensor 1 measures the overall composite drive temperature, while Sensor 2 exposes the internal Flash Translation Layer controller chip, which is running dangerously hot at 88°C.
• Thermal Management T1 Trans Count: 2412: The drive has entered heavy thermal throttling 2,412 separate times, intentionally slowing down its processing speed to prevent silicon melting—explaining your database's intermittent latency spikes.

Next Actions for the Systems Engineer

This NVMe drive is suffering from both severe overheating and near-total flash memory exhaustion. First, inspect server chassis fan curves and airflow paths to resolve the 88°C thermal bottleneck. Second, trigger an immediate live failover to your secondary database replica and replace /dev/nvme1n1 before its spare pool drops to zero and locks the drive into a read-only state.

Scenario 3: Distinguishing Between Decaying Disks and Faulty Cables

The Operational Context

A virtualization host running multiple virtual machines reports that storage tasks are stalling for over two minutes at a time. You must determine whether the underlying SATA storage disk is physically failing or whether the connection cable between the motherboard and backplane has degraded.

The Command Sequence

Query the drive's complete ATA attribute matrix:

smartctl -A /dev/sdc

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.13-production] (local build)
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   084   084   010    Pre-fail  Always       -       1288
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24810
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       34
194 Temperature_Celsius     0x0022   068   052   000    Old_age   Always       -       32 (Min/Max 18/48)
197 Current_Pending_Sector  0x0012   092   092   000    Old_age   Always       -       48
198 Offline_Uncorrectable   0x0010   090   090   000    Old_age   Always       -       16
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

Analytical Breakdown of Output

• Understanding the Table Columns:
• VALUE: A normalized health score calculated by the drive firmware (typically starting at 100 or 200 and counting downwards toward zero).
• WORST: The lowest score the drive has ever recorded in its operating lifespan.
• THRESH: The failure threshold. If VALUE drops below THRESH, the drive declares an official hardware failure.
• RAW_VALUE: The raw, un-normalized measurement (such as sector count, hours, or temperature).
• ID# 5 Reallocated_Sector_Ct (Raw: 1288): The drive has retired 1,288 damaged physical sectors and remapped their storage addresses to healthy spare sectors.
• ID# 197 Current_Pending_Sector (Raw: 48): The drive has found 48 unstable sectors waiting to be tested and reallocated on the next write attempt.
• ID# 198 Offline_Uncorrectable (Raw: 16): Background scans encountered 16 sectors that could not be read or corrected.
• ID# 199 UDMA_CRC_Error_Count (Raw: 0): The data transmission path between the drive and the motherboard is flawless. When this counter climbs, the culprit is almost always a damaged, loose, or unshielded SATA cable, rather than the drive itself.

Next Actions for the Systems Engineer

Because UDMA_CRC_Error_Count is zero, you can definitively rule out cabling issues. The drive is suffering from progressive physical degradation (Reallocated and Current_Pending sectors are both dangerously high). Live-migrate your virtual machines to another host immediately and replace the failing drive.

Scenario 4: Exporting Drive Telemetry Directly to Prometheus and Alertmanager

The Operational Context

In modern data centres housing hundreds or thousands of servers, checking drive health by hand in a terminal is impossible. You need to extract structured, machine-readable S.M.A.R.T. metrics automatically to power custom alerts via the Prometheus Node Exporter Architecture.

The Command Sequence

Invoke smartctl with its structured JSON output flag targeting your NVMe device:

smartctl -a --json=g /dev/nvme0n1

{
  "json_format_version": [1, 0],
  "smartctl": {
    "version": [7, 4],
    "svn_revision": "5530",
    "platform_info": "x86_64-linux-6.6.13-production",
    "build_info": "(local build)",
    "argv": ["smartctl", "-a", "--json=g", "/dev/nvme0n1"],
    "exit_status": 0
  },
  "device": {
    "name": "/dev/nvme0n1",
    "info_name": "/dev/nvme0n1",
    "type": "nvme",
    "protocol": "NVMe"
  },
  "smart_status": {
    "passed": true
  },
  "nvme_smart_health_information_log": {
    "critical_warning": 0,
    "temperature": 48,
    "available_spare": 100,
    "available_spare_threshold": 10,
    "percentage_used": 12,
    "data_units_read": 14209124,
    "data_units_written": 29810441,
    "host_reads": 451290123,
    "host_writes": 918230192,
    "controller_busy_time": 412,
    "power_cycles": 12,
    "power_on_hours": 3410,
    "unsafe_shutdowns": 1,
    "media_errors": 0,
    "num_err_log_entries": 0
  },
  "temperature": {
    "current": 48
  }
}

Analytical Breakdown of Output

• "json_format_version": [1, 0]: Follows an explicit version schema, ensuring your parsing scripts won't break when updating smartmontools packages in the future.
• "smartctl.exit_status": 0: An integer bitmask indicating overall execution results:
• Bit 0: Command-line parsing error.
• Bit 1: Device open or identification failure.
• Bit 2: S.M.A.R.T. command failed or returned no data.
• Bit 3: Drive reported a FAILING pre-failure state.
• Bit 4: Drive attributes or error logs contain signs of physical decay.
• "nvme_smart_health_information_log": A typed, structured JSON object that eliminates the need for brittle regex parsing across different drive manufacturers.

Next Actions for the Systems Engineer

Deploy a lightweight shell script scheduled via a systemd timer or cron job to parse this JSON data into Prometheus metrics:

#!/usr/bin/env bash
# Generates Prometheus node-exporter metrics from smartctl JSON
METRIC_FILE="/var/lib/node_exporter/textfile_collector/smart_nvme.prom"
JSON=$(smartctl -a --json=g /dev/nvme0n1)

PERCENT_USED=$(echo "$JSON" | jq '.nvme_smart_health_information_log.percentage_used // 0')
SPARE=$(echo "$JSON" | jq '.nvme_smart_health_information_log.available_spare // 0')
MEDIA_ERRORS=$(echo "$JSON" | jq '.nvme_smart_health_information_log.media_errors // 0')

cat <<EOF > "${METRIC_FILE}.$$"
# HELP node_disk_nvme_percentage_used Percentage of drive lifetime consumed
# TYPE node_disk_nvme_percentage_used gauge
node_disk_nvme_percentage_used{device="nvme0n1"} $PERCENT_USED

# HELP node_disk_nvme_available_spare Remaining reserve flash capacity
# TYPE node_disk_nvme_available_spare gauge
node_disk_nvme_available_spare{device="nvme0n1"} $SPARE

# HELP node_disk_nvme_media_errors Media and data integrity error count
# TYPE node_disk_nvme_media_errors counter
node_disk_nvme_media_errors{device="nvme0n1"} $MEDIA_ERRORS
EOF

mv "${METRIC_FILE}.$$" "$METRIC_FILE"

Configure your Prometheus Alertmanager rules to notify your engineering team whenever node_disk_nvme_available_spare < 20 or node_disk_nvme_media_errors > 0.

Scenario 5: Inspecting Physical Disks Hidden Behind Hardware RAID Controllers

The Operational Context

You are troubleshooting an enterprise rack server equipped with an LSI MegaRAID hardware controller. The operating system only sees a single aggregated virtual disk (/dev/sda). When you run a standard smartctl command against /dev/sda, it returns only generic controller info. You must bypass the RAID abstraction layer to inspect the health of each physical SAS drive inside the enclosure.

The Command Sequence

Instruct smartctl to route its diagnostic queries through the MegaRAID controller directly to physical drive slot 0:

smartctl -a -d megaraid,0 /dev/sda

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.13-production] (local build)
=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST1200MM0088
Revision:             0003
Compliance:           SPC-4
User Capacity:        1,200,243,695,616 bytes [1.20 TB]
Logical block size:   512 bytes
Rotation Rate:        10000 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c50085a129ef
Serial number:        0008S4E2
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Tue Aug 18 01:22:10 2026 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     38 C
Drive Trip Temperature:        65 C

Manufactured in week 42 of year 2020
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  24
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles: 1210
Elements in grown defect list: 3

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites   corrected invocations   [10^9 bytes]  errors
read:          0        0          0         0          0      84102.190           0
write:         0        0          0         0          0      49102.941           0
verify:        0        0          0         0          0      12901.000           0

Non-medium error count:        0

Analytical Breakdown of Output

• -d megaraid,0 /dev/sda: Instructs smartctl to use SCSI generic passthrough calls to speak directly with physical enclosure drive slot 0.
• Transport protocol: SAS (SPL-3): Confirms that smartctl is communicating directly in native Serial Attached SCSI protocol with the target drive.
• Elements in grown defect list: 3: In enterprise SAS disks, the "Grown Defect List" is the equivalent of SATA Reallocated Sectors. A small number (3) indicates minor normal wear, which is acceptable so long as uncorrected errors remain zero.
• Error counter log ... Total uncorrected errors: 0: The drive's internal read channels have processed over 84 terabytes of data without a single fatal read failure.

Next Actions for the Systems Engineer

Iterate through all remaining physical drive slots (megaraid,1, megaraid,2, and so on) using a quick shell loop to locate any failing disks in the array:

for slot in {0..7}; do
  echo "=== AUDITING PHYSICAL ENCLOSURE SLOT $slot ==="
  smartctl -H -d "megaraid,$slot" /dev/sda | grep -E "SMART Health Status|result"
done

Once a failing physical slot is identified, use your RAID controller utility (such as storcli /c0/eall/s<slot> start locate) to illuminate the chassis LED indicator before pulling the faulty disk from the server rack.

Common Traps and Misconceptions

1. Misinterpreting Raw Attribute Values on SATA Disks

A frequent pitfall is treating the RAW_VALUE column of ATA attributes as a standard integer. On many Seagate and Western Digital drives, Attribute 1 (Raw_Read_Error_Rate) or Attribute 7 (Seek_Error_Rate) packs multiple internal metrics into a single 64-bit integer. Seeing a raw number like 1844201948 does not mean the drive has suffered 1.8 billion errors; it means the firmware has combined total operation counts and error counts into a single number.

• Best Practice: Always look at the normalized VALUE, WORST, and THRESH columns first. Never declare a SATA disk dead solely based on high raw numbers in attributes 1 or 7 unless the normalized score drops below its threshold, or uncorrectable sectors are confirmed in attributes 5 and 197.

2. Running Extended Self-Tests During Peak Production Workloads

Although S.M.A.R.T. self-tests run in the background during idle moments, budget SATA SSDs and entry-level enterprise drives can experience internal controller congestion. Under intense random write workloads, running smartctl -t long can overwhelm the drive's microcontroller, causing host read and write requests to queue beyond the operating system's default timeout window.

• Best Practice: Schedule extended self-tests during planned maintenance windows. If storage latency spikes while a test is active, cancel the diagnostic immediately: bash smartctl -X /dev/sdX

3. Guessing RAID and Passthrough Flags

Specifying the wrong device type flag (such as forcing -d sat on a native SAS disk or passing -d scsi to an NVMe drive inside a USB enclosure) can cause bus resets. In rare cases involving buggy controller firmware, malformed diagnostic commands can lock up the host controller, requiring a hard reboot.

• Best Practice: Use smartctl --scan or consult the ArchLinux S.M.A.R.T. Documentation to let smartctl auto-detect the correct device types before manually configuring -d flags: bash smartctl --scan

Today's Takeaway

You do not need to wait for a 2am pager alert to find out if your storage hardware is failing. Open your terminal right now and run sudo smartctl -i -H /dev/nvme0n1 (or your primary /dev/sda drive path) to get an immediate, definitive health verdict on your machine. Then, configure the background smartd daemon in /etc/smartd.conf to run short daily checks and send automated email alerts the moment a bad sector appears. Taking five minutes to establish proactive hardware monitoring today is the difference between a routine, stress-free drive swap and a catastrophic data loss emergency tomorrow.