While everyone is looking to ’s bright future, the rapid growth has the potential to do more harm than good.
Copyright by insidehpc.com
is data-driven and heavily dependent on computing power. Depending on the complexity of the or model at hand, can involve a staggering amount of data, requiring massive computational resources. Given the sizable energy requirements of modern tensor processing hardware, this results in enormously high power consumption.
Since 2012, the amount of computing power used for research has been doubling every 3.4 months, according to OpenAI researchers Dario Amodei and Danny Hernandez.
This equates to an estimated 300,000-fold increase from 2012 to 2018, far outpacing Moore’s Law, which states that the overall processing power for computers will double every two years.
And as the world’s demand for such technology continues to grow, so does the industry’s energy consumption. In an environmentally hostile chain reaction, rapidly increasing computational needs will unavoidably escalate carbon costs.
By its nature, is extremely compute-intensive. Deep learning is based on neural networks that are comprised of multiple layers, with manifold parameters that can number in the billions. The greater the network depth, the greater the compute complexity, which requires high-performance computational power and longer training times. Canadian researchers, Victor Schmidt et al. report that state-of-the-art neural architectures are frequently trained on multiple GPUs for weeks, or even months, to beat existing achievements.
At present, the vast majority of research is focused on achieving the highest levels of accuracy, without much concern paid to computational or energy efficiency. In fact, competition for accuracy in the community is robust, with numerous leaderboards tracking which system is performing a given task the best. Regardless of whether the leader board is tracking programs for or language comprehension, accuracy is by far the most important metric of success.
But as the world’s attention has shifted to climate change, the field of is beginning to take note of its carbon cost. Research done at the Allen Institute for by Roy Schwartz et al. raises the question of whether efficiency, alongside accuracy, should become an important factor in research, and suggests that scientists ought to deliberate if the massive computational power needed for expensive processing of models, colossal amounts of training data, or huge numbers of experiments is justified by the degree of improvement in accuracy.
Research by the University of Massachusetts (Strubell et al., 2019) demonstrates the unsustainable costs of . It analyzed the computational requirements for a neural architecture search for machine translation and language modeling. The model ran for a total of 979 million training steps, and took 10 hours to train for 300,000 steps on one TPUv2 core, equating to 274,120 hours on eight P100 GPUs.
The estimated carbon cost of training the model was 626,155 lbs of carbon dioxide emissions, which is comparable to the amount produced by 125 round-trip flights from New York to Beijing. Given the significant impact that ever-expanding research could have on the environment, it is critical that the field of starts to weigh sustainability against utility.
How organizations can power sustainable
Information and Communications Technology (ICT) already accounts for approximately four per cent of worldwide carbon emissions, according to The Shift Project research, and its contribution to greenhouse gas emissions is 60 per cent higher than the aviation industry.
As more enterprises and organizations turn to and applications in an effort to drive innovation, there is a corresponding increase in demand for cloud optimized data centre facilities. If Anders Andrae, senior researcher at Huawei, is right in his prediction that by 2025 data centers will account for 33 per cent of global ICT electricity consumption, the sustainability of is a conversation that green-minded organizations desperately need to start having.
There are positive steps companies can take to minimize their carbon footprint whilst still accessing cutting-edge supercomputing to drive their innovations. Given that and applications consume an enormous amount of energy, companies need to ensure that the data centers housing those applications can efficiently handle the high-density compute involved, at industrial scale. Many corporate data centers simply are not equipped to handle these demands. According to a survey by Science Direct, out of 100 data centers, 61 per cent were operating with systems running at their lowest efficiency. […]