A research team from DeepMind and Google proposes a new approach involving ML (machine learning) techniques to decrease the energy consumption of HVAC (heating, ventilation and air conditioning).
Their work was presented in the research paper “Controlling Commercial Cooling Systems Using Reinforcement Learning” released on December 12, 2022.
The team employed a RL (reinforcement learning) algorithm for HVAC controlling systems of two commercial buildings. The experiments were conducted in partnership with Trane Technologies and final tests showed an improvement in energy consumption decrease of HVAC from 9 to 13%.
The climate change problem
Climate change encompasses global warming and long-term shifts in temperature and other climate conditions, leading to the rise of sea levels, and more frequent extreme weather events such as heat waves, droughts, and hurricanes. One of the causes of climate change is the increase of CO2 emissions due to human activity.
The effects of HVAC usage
A significant percentage of global CO2 emissions are due to the systems of HVAC. For example, space cooling alone accounts for around 10% of the world’s total electricity demand.
HVAC is used to control the temperature, humidity, and air quality in a building. These systems are designed to provide a comfortable indoor environment for people and are typically found in homes, offices, and other commercial buildings. They are an important part of building design and operation, as they can have a significant impact on energy consumption and the comfort of the occupants.
How significant is their effects?
HVAC systems contribute significantly to the operational cost and the global CO2 emissions. Most of the CO2 emissions produced worldwide are attributable to HVAC. For instance, almost 10% of the entire global electricity demand is accounted for by space cooling alone.
Therefore, improving HVAC systems with machine learning algorithms like RL may be a powerful technique for mitigating climate change in the future.
A new approach to cooling systems
The experiments in this paper were conducted on chiller plant buildings. The water side of the cooling system is responsible for cooling down and heating up the water, so that it can be used to cool the air.
The building temperature is impacted by the sunlight, convection from the outdoor air, heat from the grounds, air movements and equipment heat. These are highly variable and are challenging for the model. Normally, they are solved using physics models, but they require modeling of the building.
In this paper’s approach the research team uses sensors and RL (reinforcement learning) algorithms to gather and control the real time state of the building.
RL algorithms
RL is a type of ML that involves training an agent to take actions in an environment in order to maximize a reward.
The system can be thought of as a finite state machine. At each state the system uses the transition function to take the best possible action that moves the system onto another state. After reaching the terminal state the system gets the returns that are calculated from the discounted rewards.
In the context of this paper, the state is the array of sensor measurements containing temperature, water-flow rate, and equipment status information. The potential action the system can take on at each state includes turning the equipment on/off, adjusting the differential pressure of chilled water and altering the number of water pumps.
The model is an ensemble of neural networks (NN) as in the following graph:
Methodology & results
The team first had to define the environment and the system states, each state being defined as a numerical array of sensor measurement values such as temperatures, water flow rate, and equipment status.
Next, they defined the reward function that the agent had to optimize. This might involve minimizing the difference between the desired temperature and the actual temperature, or minimizing the energy consumption of the cooling system.
Finally, the agent was trained using RL techniques to take actions that maximize the reward by trial and error, adjusting its actions based on the outcomes of those actions.
Once the agent was trained, it could be used to control the commercial cooling system in real-time, by taking actions to keep the temperature within a desired range while minimizing energy consumption. The RL algorithm was able to demonstrate around 9% and 13% savings during live testing in two real world facilities.
Conclusions
This research paper shows the advantages of using RL algorithms to continuously monitor and optimize HVAC systems in real-time.
For large commercial buildings the cooling system is responsible for high energy consumption. Therefore, the ML approach can have a significant impact in decreasing energy consumption.
One of the main advantages of the RL approach for controlling HVAC is that it does not require a detailed and comprehensive physics based model in order to be operational.
Further research is needed to improve the system. New approaches may involve:
- improving data efficiency by adding additional domain specific inductive biases to the model
- using more realistic simulations using different facilities
- using more accurate sensors
- using more powerful function approximators such as deep neural networks that can help improve the performance of the RL algorithms.
- using other RL algorithms
Learn more:
- Research paper: “Controlling Commercial Cooling Systems Using Reinforcement Learning“ (on arXiv)
- Related research: “Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning” (in Elsevier)
