H100 Thermal Throttling Starts at a Margin, Not at 87 °C

Google’s troubleshooting page for A3 series virtual machines says performance degradation can follow when GPUs consistently exceed 87 °C under load. NVIDIA’s H100 PCIe product brief uses the same 87 °C for a different job: the thermal qualification temperature, averaged over the GPU’s internal sensors. That brief prints no absolute temperature for thermal throttling at all. It writes each threshold as a distance from a limit, and the distance counts down through zero.

Slowdown is a margin, not 87 degrees

The brief lists a thermal qualification temperature for the GPU, TAVG = 87 °C, and a separate one for the HBM. The three limits beneath them are written as TLIMIT, which the brief defines as the current distance in degrees C from the software slowdown event. Those values sit in Table 1 beside the other figures used here.

Source: NVIDIA, “NVIDIA H100 PCIe GPU Product Brief,” PB-11133-001_v02 (November 2022), Tables 4 and 5.

The nvidia-smi manual explains how to read the signed value. T.Limit is a margin to the maximum operating temperature and not an absolute reading, and a value of 0 °C or lower indicates that the GPU may optimize its clock based on thermal conditions. The manual separates two reductions. Software thermal capping lowers clocks below the requested value when temperature is above the maximum operating temperature. Hardware thermal slowdown cuts core clocks by a factor of 2 or more, and the brief lists its 50% clock slowdown as the hardware slowdown step.

The manual also says the hardware clock adjustment can only be triggered by the maximum GPU temperature. TAVG averages every internal sensor, so the 87 °C qualification point and the slowdown trigger are measured on different bases, and a single hot spot spends margin that the average hides. An alarm written as a fixed 87 °C on the temperature.gpu reading therefore carries an assumption that neither document supplies. The signal that settles the question is a clock reason. Google’s page tells operators to query clocks_throttle_reasons.hw_slowdown and to read Active as throttling from high temperatures.

Source: NVIDIA, “nvidia-smi: NVIDIA System Management Interface program,” clock event reasons and temperature sections; Google Cloud, “Troubleshoot GPU VMs,” section “Monitor GPU thermal throttling.”

Two limits on scope apply. The brief covers the PCIe card, which uses a passive heat sink that needs system airflow and has a maximum thermal design power of 350 W, so its margin is spent by the server’s inlet air as much as by the card, and the per-rack air cooling limit decides how much of that air arrives. Google’s page does not say which sensor its 87 °C refers to, and A3 series machines are not the PCIe card the brief describes.

Five degrees from clock cuts to shutdown

The window is narrow. The 50% hardware slowdown sits 2 °C past the point where clock optimization may begin, and shutdown sits 5 °C past it.

A heat path with a given thermal resistance in K/W converts each added watt into a fixed temperature rise, so a load increase eats that 2 °C in proportion to the resistance and not in proportion to the nameplate.

Runtime ratio = nominal clock ÷ actual clock = 1 ÷ 0.5 = 2. If throughput scaled one-for-one with core clock, a 10-hour compute-bound job would take 20 hours. Work that does not scale with core clock loses less, so 2 is a ceiling and not a typical value.

The software step is gentler and arrives earlier. Meta’s Llama 3 paper reports a diurnal 1-2% throughput variation on the 405B training run and attributes it to higher mid-day temperatures affecting GPU dynamic voltage and frequency scaling. The passage gives no temperatures and no cooling design.

Minutes per day = 1,440 × 0.01 to 1,440 × 0.02 = 14.4 to 28.8. That treats the full swing as a deficit for 24 hours, so it is an upper bound.

Source: Llama Team, AI @ Meta, “The Llama 3 Herd of Models,” arXiv:2407.21783v3 (23 November 2024), section 3.3.4.

Nothing opened here ties that variation to a particular T.Limit value, so it cannot be placed on the brief’s ladder. The manual keeps separate counters for software thermal slowdown and hardware thermal slowdown, and the hardware flag marks the 50% step, not this smaller loss. A fleet that watches only the hardware flag would miss the first kind of loss. The same Meta section notes that even a single straggler can slow thousands of other GPUs, which puts the cost of a synchronous job at the hottest GPU rather than the mean.

Arrhenius predicts 8.8×, ASHRAE’s field curve 1.7×

The lifetime side has no threshold, only a slope, and two published slopes disagree. ASHRAE TC9.9’s storage white paper reproduces the server failure-rate X-factor curve from the 2011 thermal guidelines and reads it as a rise of about 70% from 20 to 45 °C of inlet air.

That is a ratio of 1.7. The paper sets it against 8.8× from prediction models such as MIL-217 and Telcordia, which use an Arrhenius model with 0.7 eV activation energy.

Source: ASHRAE TC9.9, “Data Center Storage Equipment – Thermal Guidelines, Issues, and Best Practices” (2015), section 4.5 and Figure 24.

The Arrhenius figure can be reproduced. Acceleration factor = exp((Ea ÷ k) × (1 ÷ T1 − 1 ÷ T2)), with Ea = 0.7 eV, k = 8.617 × 10⁻⁵ eV/K, T1 = 293.15 K (20 °C) and T2 = 318.15 K (45 °C). Ea ÷ k = 8,123 K and (1 ÷ 293.15 − 1 ÷ 318.15) = 2.681 × 10⁻⁴ K⁻¹, so the exponent is 2.178 and the factor is 8.82. At T2 = 303.15 K (30 °C) the exponent is 0.914 and the factor is 2.49. The result is a model value for one thermally activated mechanism held at constant temperature.

ASHRAE’s power equipment white paper puts the curve at 1.4 for continuous operation at 30 °C, where the model gives 2.49. The two measure different things. The curve is indexed to inlet air for whole servers, and the storage paper notes that components inside a storage array chassis typically run well above inlet temperature. A change at the inlet is therefore not the same change at a die, and an aggregate curve that mixes many components is likely to be shallower than a single-mechanism model, which is an inference and not a statement from either paper.

Source: ASHRAE TC9.9, “Data Center Power Equipment Thermal Guidelines and Best Practices” (2016), section on reliability and life expectancy of rack PDUs.

That 2016 paper adds a point that matters for excursions: expected failure rate is a time-weighted average over the temperature history, not the value at the highest excursion. On that basis a short excursion costs little failure rate, and a margin held low for months costs more. No document opened here gives a GPU failure rate against die temperature, so the throttling thresholds cannot be read as lifetime thresholds in either direction. They are performance protections set by the vendor.

Field data from liquid-cooled GPUs shows operation far from those thresholds. The Delta system study from the University of Illinois reports mean temperatures of 40 °C for A100 and 37 °C for H100 at average utilization, with maximums up to 49 °C. The same study attributes GPU “fallen off the bus” errors to loose connections or contact failure caused by thermal cycles, a mechanism tied to temperature swings rather than to a steady level.

Source: Cui et al., “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs,” arXiv:2503.11901 (SC ’25), cooling and temperature paragraph and the section on GPU fallen off the bus errors.

Which number answers which question

Table 1. Stated thermal values and what each one triggers (°C, sorted by value)

* TLIMIT is a distance from the software slowdown event, so negative values lie beyond it. The brief prints no absolute temperature for these three rows.

Source: NVIDIA, “NVIDIA H100 PCIe GPU Product Brief,” PB-11133-001_v02 (November 2022); NVIDIA, nvidia-smi manual; Google Cloud, “Troubleshoot GPU VMs”

The first clock action, the 50% step and shutdown sit within 5 °C of each other on the sensor that matters, while the lifetime slope is shallow and spread over years. A proposal that quotes one GPU temperature limit should be asked which sensor, which product and which of the events in the table it names. Setting alarm thresholds, margins and airflow for a specific system is work for a qualified engineer using the vendor’s own documentation.

References

• NVIDIA / NVIDIA H100 PCIe GPU Product Brief, PB-11133-001_v02 / November 2022 / https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/h100/PB-11133-001_v01.pdf
• NVIDIA / nvidia-smi: NVIDIA System Management Interface program (manual) / undated, page as opened / https://docs.nvidia.com/deploy/nvidia-smi/index.html
• Google Cloud / Troubleshoot GPU VMs / page footer “Last updated 2026-10-08 UTC” / https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-gpus
• Llama Team, AI @ Meta / The Llama 3 Herd of Models, arXiv:2407.21783v3 / 23 November 2024 / https://arxiv.org/pdf/2407.21783
• ASHRAE TC9.9 / Data Center Storage Equipment – Thermal Guidelines, Issues, and Best Practices / 2015 / https://www.ashrae.org/file%20library/technical%20resources/bookstore/ashrae_storage_white_paper_2015.pdf
• ASHRAE TC9.9 / Data Center Power Equipment Thermal Guidelines and Best Practices / 2016 / https://www.ashrae.org/File%20Library/Technical%20Resources/Bookstore/ASHRAE_TC0909_Power_White_Paper_22_June_2016_REVISED.pdf
• Cui et al., University of Illinois / Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs, arXiv:2503.11901 / SC ’25, November 2025 / https://arxiv.org/pdf/2503.11901