Unlike humans, who can recognise an object from an image after seeing one or two examples, AI deep learning models need to see tens of thousands of images. Imagine attempting to learn and recognise every item in your environment that way and you can begin to understand why AI software consumes so much computing power and requires such large training datasets. As an example, whilst working on Google’s AutoML software, researchers were using as many as 800 graphics chips running in unison to train their powerful algorithms.
If neural networks could be training to run tasks based on a smaller dataset, it would complete change the whole paradigm, Charles Bergan, vice president of engineering at Qualcomm, told the audience at MIT Technology Review’s EmTech China conference earlier this week.
The resource intensive process of feeding huge volumes of data to train algorithms would be rendered obsolete if neural networks were capable of “one-shot learning,” Bergan said.
In theory this sounds great, but it could have knock-on implications for the hardware industry – who as a whole are focusing all their R&D efforts on developing more powerful processors designed solely to run today’s data-heavy AI algorithms. On the plus side, it would mean far more efficient machine learning, with speed-to-market increasing dramatically.
Whilst neural networks that can be built with small datasets are not yet a reality, research is already underway on how we can make algorithms smaller without sacrificing on accuracy, chief scientist at Nvidia, Bill Dally, said at the conference.
Alluding to the research, Dally explained that their researchers are using a process called network pruning to make a neural network more efficient by removing the neurons that don’t contribute directly to output.
“There are ways of training that can reduce the complexity of training by huge amounts,” Dally said.