The human brain operates with a “grow and prune” strategy, initially starting off with a massive amount of neural connections and then pruning away the unused connections over time.
Copyright by www.unite.ai
Recently, a team of researches has applied this approach to systems and found that it could substantially reduce the amount of energy required to train an .
A team of researchers from Princeton University recently created a new method of training systems. This new training method seems able to meet or surpass the industry standards for accuracy, but it’s able to accomplish this while consuming much less computational power, and therefore less energy, than traditional models. Over the course of two different papers, the Princeton researchers demonstrated how to grow a network by adding neurons and connections to it. The unused connections were then pruned away over time, leaving just the most effective and efficient portions of the model.
Niraj Jha, professor of Electrical Engineering at Princeton, explained to Princeton news that the model developed by the researchers operates on a “row-and-prune paradigm”. Jha explained that a human’s brain is the most complex it ever will be at around three years of age, and after this point the brain begins trimming away unneeded synaptic connections. The result is that the fully developed brain is able to carry out all the extraordinarily complex tasks we do every day, but it uses about half of all the synapses it had at its peak. Jha and the other researchers mimicked this strategy to enhance the training of .
“Our approach is what we call a grow-and-prune paradigm. It’s similar to what a brain does from when we are a baby to when we are a toddler. In its third year, the human brain starts snipping away connections between brain cells. This process continues into adulthood, so that the fully developed brain operates at roughly half its synaptic peak. The adult brain is specialized to whatever training we’ve provided it. It’s not as good for general-purpose learning as a toddler brain.”
Thanks to the growing and pruning technique, equally good predictions can be made about patterns in data using just a fraction of the computational power that was previously required. Researchers are aiming to find methods of reducing energy consumption and computational cost, as doing so is key to bringing to small devices like phones and smartwatches. Reducing the amount of energy consumed by algorithms can also help the industry reduce its carbon footprint. Xiaoliang Dai, the first author on the papers, explained that the models need to be trained locally due to transmission to the cloud requiring a lot of energy.
During the course of the first study, The researchers tried to develop a neural network creation tool that they could use to engineer neural networks and recreate some of the highest performing networks from scratch. he tool was called NeST (Neural network Synthesis Tool), and when it is provided with just a few neurons and connections it rapidly increases in complexity by adding more neurons to the network. Once the network meets a selected benchmark it begins pruning itself over time. While previous network models have used pruning techniques, the method engineered by the Princeton researchers was the first to take a network and simulate stages of development, going from “baby” to “toddler” and finally to “adult brain”. […]