For decades we’ve been teaching machines how to classify and determine things in the world, giving them many examples to draw from. But what if they could teach themselves?
Machine learning, which is a part of artificial intelligence, has long been about supervised learning where machines are given labelled data or examples to learn from when carrying out certain tasks such as classifying an object or predicting future outcomes.
Going a step above that is unsupervised learning, where there are no labels and the machine has to observe the data, make sense of it and provide an outcome. It’s this kind of learning that’s of interest to many data scientists and AI researchers – mostly because in the real world, data doesn’t usually come in neat, clearly labelled packages.
“To really get to the next level of performance in different applications like language and so on, it’s going to be very challenging to use these annotations. Sometimes we just don’t have the luxury of having users label things for us,” says Adam Coates, director of Baidu Silicon Valley AI Lab.
“For example, if I ask you to do something very complicated for me in a very natural language request, it’s often not the case that we can get someone to read that request and then tell us what it means in a simpler way – it’s just too complicated and expensive.
“So one hope in deep learning research is that for some of these situations where we can’t get tags or we can’t get labels, that it will be able to learn unsupervised – what we think humans spend a lot of time doing. The deep learning algorithm can look at a bunch of text or audio or images and learn to understand and to make sense of them without necessarily being told what is the right answer.”
Deep learning is artificial neural networks that are made up of many hidden layers between the input and output. Currently, it can do unsupervised learning but only to the extent of pre-training or detecting good features to assist in supervised learning.
“What machines are currently not really able to do properly is unsupervised learning. A lot of us have been working on this for a long time, but I don’t think we have the solution for it yet,” says Facebook’s director of AI research, Yann LeCun.
“The problem with unsupervised learning is that we don’t even have good principles to base it on. There are a lot of proposals for different ways, theoretical principles when it comes to the underlying mathematics to base unsupervised learning on, but I believe we don’t have the answer yet.”
Tools like Google’s open source Word2Vec, however, is one example of unsupervised learning that works, LeCun says. Developed by Tomas Mikolov, who is now a research scientist at Facebook, and others, the tool doesn’t need labelled data to learn the vector representation of a word for predicting other words in a sentence.
The reason unsupervised learning works in that case, LeCun says, is because there is a small number of words predicted to follow on from other words. Whereas with images and video, the number of possibilities are much greater, making it difficult to apply unsupervised learning.
“What if I take a movie and I show you a couple seconds of that movie and ask you a second from now what is it going to look like? An infinite number of things could happen in a second that you can’t predict. So it’s very difficult to train a system to predict what the next frame is going to be.
“Right now if you ask it to do this what it’ll do is put some sort of average picture of all the things that could happen and that doesn’t look very good. So that’s why I’m talking about the principles.
“Some people claim that, and I kind of agree with this, the kind of principle in which unsupervised learning should be based is prediction. So as humans, we are good prediction machines. You know what the world is going to look like if you move your head a little to the left. You’ve got a good model of the world where you’ve seen this many times and you know what it would look like.
“You’ve trained yourself by observing the world, and by doing that you’ve extracted some sort of underlying structure of it. We would like to develop machines that can do this, just observe and learn a lot about the world.”
The AI research team at Facebook published a paper last month that is a step closer to doing unsupervised learning on image recognition. They built a deep learning system, trained in an unsupervised manner, which can automatically create its own images of objects and animals from photographs. The images created looked realistic to human evaluators 40 per cent of the time.
Baidu’s Coates says in the next few years more progress will be made in developing unsupervised learning to produce better results and further advance artificial intelligence.
“We don’t have really big success stories of this technology yet, but I am hopeful in the next few years we will be able to make progress on that. It’s an open research area,” he says.
“But I think for the next couple of years there is still more room to run with the current deep learning technology. I think we can still make a tonne of impact and make some really amazing technology based on supervised learning over the next couple of years,” he adds.
Next page: Risk of AI turning into a sci-fi nightmare?