2019: Spring

It is not long ago that people wre claiming - I myself was claiming - that go was solved because go programs could play at at moderate club player level. After all, it would take several years to get to moderate club player level for most people. It was not long ago that people were claiming that OK, machines could now play a good game of go, but people could still kick their ass at games like Civilization, which involved hidden information and randonmess and long term strategies in a highly dynamic and heterogeneous environment, much less something like StarCraft II. which has all that plus is played in real-time.

But now we have AlphaStar. It is not quite true to say StarCraft II is solved, but the program can play and win at the grandmaster level against humans under competition conditions. There are ways of making things harder, such a relying just on what the camera can see on the screen or playing with different races on different boards (although humans tend to specialise in one race, I understand) It is not as though AlphaStar is not going to get better. As with those mid-2010s go program, AlphaStar can play a good game of StarCraft II and not, this case, just a good game, but a game at elite level.  

Is this a bigger deal than AlphaGo or AlphaZero? Arguably, it is because it is more obvious what you might do with an AlphaStar-style system. What about self-driving cars? Or financial trading? Or weather forecasting and climate prediction (an example DeepMind state in their blog article)? Obviously, for myself, I am interested professionally in anomaly detection in APIs. As Lex Fridman points out in his lecture I am currently watching on Deep Reinforcement, it is not quite right to say that DRL is not supervised learning. It is supervised in the sense that someone has assigned a reward function. In traditional supervised learning, that's how many predictions the system got right. With a DRL that could still be the case (did it predict or postdict the weather correctly?). but usually the reward is something related to winning the game. Of course, this doesn't have to just be one point for a win. AlphaStar has a league table and lots of different variants specialised for certain strategies. A backgammon or poker or trading sysyem would care not about winning as such, but how much money it makes. A problem for my AlphaAPImetrics is what exactly is an anomaly? I need to think about this. Perhaps there is some way of defining a reward in the system. 

So, now that we have AlphaStar, does that mean that all of the Atari games are solved? What of backgammon, bridge, poker? Is AlphaCiv now just a weekend hack for undergraduates? Exactly how much raw comouting power would you need for an AlphaCiv. AlphaStar trained for something like 200 years. Clearly, there is still a lot of improvements yet possible to reach human-level in terms of speed and efficiency of learning. Of course, that might not matter. What would happen if you turned AlphaStar up to max? It makes a comparable number of moves per second to a human player, but that's partly because it is deliberately throttled to avoid the agent winning simply because it can coordinate large numbers of units at superhuman speeds. Of course, there is no guarantee than a(n un)Friendly AGI would do that. 

Deep Learning is a sexy name, but it is really just using bigger neural nets and more data. You couldn't do that in the 1980s or 1950s. OK, there are some tweaks in terms of the architecture to get good performance out of the system, but there can be feeling that just getting better at face recognition is an ability with diminishing returns. There are only so many situations in which you have large collections of labelled training data. DRL on the other hand does seem more "magic", if there aree domains to which it can be applied. People might get bored with just deep learning, but combining AlphaStar and BERT suggests to me that there might be more some mileage in the AI hype cycle yet. As I said before, A(G)I isn't going to stop being a good idea. It's a question of finding the right problem to solve. We can already see some of those questions. Bu what we need is a system capable of getting a first in maths at Oxford. That's a DRL problem. It will be interesting to see how long it is before that one is achieved. And as long as the research dollars go into that kind of problem, I don't think we will see an AI autumn.

Comments

[John B]

A colleague was using neural nets for weather forecasting in the mid '90s. They never got much traction, though back propagation techniques are part of the key solvers.

I'm not surprised computer games can be brute-forced, as the game's difficulty is set by the designer introducing rules and complexity to a level a player can't handle at the turn rate. Real Time Strategy games could always be won by micro-managing, but its the real-time aspect that makes it a challenge. Turn based games might seem a fairer playing field, but again what Civ player doesn't take shortcuts.

AIs vs games are just exposing the rules of the designers. AIs vs the real world is much more interesting, as you can't argue that its the laws of physics that are being revealed, as they layers of complexity and randomness hide them so well

[Paul M. Cray]

As the Weasel has indicated, there were people trying to do stuff with neural nets for hydroinformatics in the late 1990s/early 2000s and not getting very far, but that was probably a combination of Moore's law, inadequate tools and ill-defined problems.

As you indicate, the physics or real-world problems are in, a way, much more interesting because you can't cheat nature and to the extent that you can, it is telling you something interesting and useful.

I'm certainly though to see to what AlphaStar-like systems can be games like "Civilization". Would it just be a case of having enough raw compute to train the system?