The NLP Revolution

The Lex Fridman's MIT deep learning course has started again. I knew about this last year, but didn't get to see any of the videos, but am determined, as part of my New Year resolutions for 2019, to make an effort to watch them. (I am still hoping for another tranche of Andrew Ng courses.) 

Deep learning has been around for a while now and, of course, neural nets have been around for decades. Deep Learning just involves bigger neural nets and more data. Obviously, there is more to it than that, Deep Reinforcement Learning, for instance, is quite a different beast in a lot of ways, but it would be easy to feel that the Peak of Inflated Expectations had been reached. It would be easy to feel that if it were for what actually happened in 2018, which was the NLP revolution.

I have to admit that I was only vaguely aware of this. I had seen a few mention of it and hints about things around it, but I have not been paying the assiduous attention to the AI newsletters to which I subscribe. Again, that's something I can't let go in 2019. Various improved machine translation systems have been around for a while and they do seem to be getting better as far as I can see from translating Wikipedia pages. Certainly, the translations, if not quite perfect, are much smoother and more idiomatic than was the case only a year or two.

But the hot new thing is Google BERT. I hadn't even heard of it until last night, but apparently it created a splash back in November. It is open-sourced and already other groups have incorporated it and related ideas into their own systems. This is one reason that everyone has been excited about digital assistants. These have been around for a while now, but are coming on, I understand, in leaps and bounds. See, for instance, the leaderboard for "the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks" on which BERT is now behind a Microsoft project. BERT reached human-level on the Allen Institute for Artificial Inteligence SWAG dataset, although it wasn't specifically optimised for the dataset. A problem with a lot of A(G)I problems is producing a system that works well for more than one problem. With Kaggle competitions and their like, there is a danger of overoptimising to the particular problem at hand. It's not that the system just memorises the training set, but that it is tuned to recognise just fish, say, and then has to be retrained expensively to recognise anything else. But with AlphaZero and BERT, we are definitely seeing transfer learning at some significant level. 

I do need to start exercising Google Assistant a lot harder. How good is it now? Now, I'd like to have a play around with these kinds of systems. A problem is the sheer amount of compute needed. I have a six-year old laptop. Yes, I need a new one, but how far can one get really get with our a fairlygrunty GPU or even a TPU? It would be nice to apply an AlphaZero-like system to something like FreeCell. But do I have the compute leaving aside whether there is an API for AisleRiot Solitaire?

A interesting facet of the NLP revolution is that there had been concern that had performance on various Deep Learning tasks had been plateauing, such as in computer vision or self-driving cars, but then we see sudden breakthroughs as also with AlphaGo/AlphaZero. But it is also clear that the possibility of rapid or even very rapid progress over short timescales is also possible. It is 2019 and I still think there is no reason to think that beyond 2025, we might see the emergence of true AGI. That 03:14:06 UTC, 19 January 2038 Singularity seems still very much on the cards.