Recently leaked data allows us for the first time to estimate the carbon emissions from training OpenAI’s GPT-4
With recent news alerting us that global average temperatures keep rising [1] it’s important to remind ourselves that most human activities have a carbon footprint that contributes towards global warming and other climate change. This is also true for digital technology in general and AI in particular. This article serves as a reminder of this, as it estimates the carbon emissions of training OpenAI’s large language model GPT-4.
To make such estimates, we need to know:
- How much electricity was used to train GPT-4
- The carbon intensity of the electricity, i.e. the carbon footprint of generating 1 KWh of electricity
Let’s dive right in.
Let’s first estimate GPT-4’s energy consumption. According to unverified information leaks, GPT-4 was trained on about 25,000 Nvidia A100 GPUs for 90–100 days [2].
Let’s assume the GPUs were installed in Nvidia HGX servers which can host 8 GPUs each, meaning 25,000 / 8 = 3,125 servers were needed.
One way to estimate the electricity consumption from this information, is to consider the thermal design power (TDP) of an Nvidia HGX server. TDP, denoted in watts, expresses the power consumption of a piece of hardware under maximum theoretical load [11], ie the actual power consumption may differ.
Unfortunately, Nvidia does not disclose this information, so let’s instead use the TDP of the similar Nvidia DGX server, which is 6.5 kW [3]. So, if an Nvidia DGX server runs at full power for 1 hour, it will have consumed 6.5 KWh according to the TDP.
Recall that it’s estimated that it took 90–100 days to train GPT-4. That’s 90 or 100 * 24 = 2,160 to 2,600 hours per server. If we assume the servers…