In 2022, Meta AI revealed CICERO, “an AI system that beat 90% of human players at Diplomacy. Unlike Chess, Go, etc., Diplomacy requires multiplayer collaboration – and backstabbing!” Meta AI researchers provide an overview of their breakthrough in natural language processing (NLP) and building trust, cooperation, deception, and strategic reasoning into an NLP system.
Building Trust, Cooperation, Deception, and Strategic Reasoning into Natural Language AI Systems
Diplomacy AI “thread-of-threads” from META AI researchers
AI Pub generated a 9/9 thread based on the release of Meta’s CICERO – an AI system that beat 90% of human players at Diplomacy: “Meta AI presents CICERO — the first AI to achieve human-level performance in Diplomacy, a strategy game which requires building trust, negotiating and cooperating with multiple players.” Meta researchers p
// Diplomacy AI thread-of-threads //
OODA Loop Sponsor Yesterday, Meta AI revealed CICERO, an AI system that beat 90% of human players at Diplomacy.
Unlike Chess, Go, etc., Diplomacy requires multiplayer collaboration – and backstabbing!
A CICERO thread-of-threads from Meta AI researchers:
1/9 pic.twitter.com/z66SWRwbAy
— AI Pub (@ai__pub) November 23, 2022
From the thread
- @polynoamial explains some of the inherent difficulties of AI playing Diplomacy: “Diplomacy is about building trust in an environment that encourages players to not trust anyone… must account for the risk that players might lie, and that players might doubt your honesty.”
-
@adamlerer on why CICERO is a huge breakthrough: “[CICERO] addresses 2 [questions]:
- How to integrate planning & RL ([reinforcement learning], eg. AlphaZero) with dialogue (eg. GPT-3)
- How to learn and plan in settings with both competition and cooperation, where agents must understand [human intentions]”
- For decades, Diplomacy has been viewed as a near-impossible grand challenge in AI due to the complex interplay of strategy, theory of mind, and natural language involved...we present CICERO, the world’s first human-level full-press Diplomacy AI.
- Our paper describing a human-level AI for Diplomacy was published in Science. This is the first human-level AI for a game requiring cooperation through *natural language*. Really proud of what we built and excited to finally share it.
-
@ml_perception with some of the training & implementation details behind CICERO. CICERO passed for human in 40 anonymous online games, and finished in the top 10% of players!
An example of persuasion from our new Diplomacy AI agent Cicero, published in Science today. The bot (Italy) successfully convinces France that it’s not planning to attack. pic.twitter.com/kzgr9YxvuM
— Noam Brown (@polynoamial) November 22, 2022
Why Diplomacy?: A Grand Challenge for AI
From Meta AI:
A game about people rather than pieces: Diplomacy is a seven-player board game that can be described as a combination of the board game Risk, the card game poker, and the TV show Survivor. Unlike many board games where you just need to be the best at moving pieces around the board, Diplomacy has a cooperative component — the only way to win is by working with other players to capture as much territory as possible. This coordination is achieved through natural language negotiation that occurs before every move in the game.
Why Diplomacy? Advancements in AI gameplay have long served as benchmarks for progress in AI. For decades, researchers have been building simplified variants of Diplomacy gameplay agents without natural language communication capabilities. But no one has ever attempted to build an AI agent that can negotiate with open-ended dialogue. Achieving this grand challenge of AI has simply been beyond the capabilities of what’s existed in AI, until now.
The making of a breakthrough: Strategic Reasoning + Natural Language: Within CICERO is a language model integrated with strategic reasoning algorithms that control the generation of dialogue. Our advancements in strategic reasoning and natural language processing together enable effective cooperation with humans — by giving CICERO the ability to understand what other players are trying to achieve, negotiate a plan, suggest shared goals and communicate with strategic intent.
Human-level play in the game of Diplomacy by combining language models with strategic reasoning
From the Science article published by the Meta AI research team:
AI masters Diplomacy: The game Diplomacy has been a major challenge for artificial intelligence (AI). Unlike other competitive games that AI has recently mastered, such as chess, Go, and poker, Diplomacy cannot be solved purely through self-play; it requires the development of an agent to understand other players’ motivations and perspectives and to use natural language to negotiate complex shared plans.
The Meta Fundamental AI Research Diplomacy Team (FAIR) et al. developed an agent that is able to play the full natural language form of the game and demonstrates performance well above the human average in an online Diplomacy league. The present work has far-reaching implications for the development of cooperative AI and language models for communication with people, even when interactions involve a mixture of aligned and competing interests.
Abstract: Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge.
We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
What Next?
Playing a part in the metaverse and beyond: Gaming has long been a playground for pushing the limits of AI. Every iteration takes us deeper into more immersive types of AI, with new ways to interact in the physical and virtual worlds. As we build for the metaverse, we see potential for technology like CICERO’s to power cooperative agents that can help people learn new skills, negotiate on their behalf, create more immersive social and gaming experiences and connect more meaningfully with those around them…by open sourcing CICERO’s code…[which is available at GitHub].
Take a deeper dive into CICERO’s Diplomacy gameplay: Visit the Meta AI Gameplay page for the basics of Diplomacy, to learn about CICERO’s performance in live games and to view a collection of explainers, including a clip of CICERO gameplay with 3x Diplomacy World Champion Andrew Goff and Meta AI researchers.
Future directions: While CICERO is only capable of playing Diplomacy, the technology behind this achievement is relevant to many real world applications. Controlling natural language generation via planning and RL, could, for example, ease communication barriers between humans and AI-powered agents. For instance, today’s AI assistants excel at simple question-answering tasks, like telling you the weather, but what if they could maintain a long-term conversation with the goal of teaching you a new skill? Alternatively, imagine a video game in which the non player characters (NPCs) could plan and converse like people do — understanding your motivations and adapting the conversation accordingly — to help you on your quest of storming the castle.
Additional Resources
AI Discipline Interdependence: There are concerns about uncontrolled AI growth, with many experts calling for robust AI governance. Both positive and negative impacts of AI need assessment. See: Using AI for Competitive Advantage in Business.
Benefits of Automation and New Technology: Automation, AI, robotics, and Robotic Process Automation are improving business efficiency. New sensors, especially quantum ones, are revolutionizing sectors like healthcare and national security. Advanced WiFi, cellular, and space-based communication technologies are enhancing distributed work capabilities. See: Advanced Automation and New Technologies
Emerging NLP Approaches: While Big Data remains vital, there’s a growing need for efficient small data analysis, especially with potential chip shortages. Cost reductions in training AI models offer promising prospects for business disruptions. Breakthroughs in unsupervised learning could be especially transformative. See: What Leaders Should Know About NLP
