An Introduction to Learning Through Simulation

Who's this guy

Chief Executive Officer

I'VE DONE MORE

Chief Technology Officer at Yoyo

Payment facilitator for small businesses

Video Game Development Engineer at Playful, Ensemble, Newtoy (Zynga with Friends)

AOE, Lucky's tale, Creativerse, Oculus SDK (v0 through v1)

Minor OSS contributor

DevIL, ResIL, fog, Emscripten (Web Assembly)

We've come a long way

What Are World Models?

An internal simulator that learns to predict what happens next

Captures state, transitions, and visual output

Reinforcement Learning

• Q‑Learning (Watkins & Dayan, 1992) — “Q-Learning”, Machine Learning, 8(3–4):279-292. This paper proves convergence of the Q-learning algorithm under appropriate conditions. 

• Playing Atari with Deep Reinforcement Learning (Mnih et al., 2013) — Introduces the DQN algorithm (a deep neural-network variant of Q-learning) that can learn control policies directly from raw pixels.

• Deep Reinforcement Learning with Double Q‑Learning (van Hasselt et al., 2016) — A deep RL variant addressing over-estimation in Q-learning via Double Q-learning.

VPT: Learning from Watching

Generative

Why not just teach machines by doing?

Dreaming is a safer, faster path to learning

why simulate?

❌ Real-World Learning

• Slow (robot training)

• Risky (self-driving)

• Expensive (physical trials)

✅ Dreaming (Simulation)

• Fast iterations

• Completely safe

• Cost-effective

Why Do We Need World Models?

Timeline & Milestones

Understanding the Basics

Observation

Encoder

Latent State

Decoder

Under the Hood: Architecture

Key Components

Showcase: NES Agents

Gathering the Data

NES emulator in Python

Because... why not? 😁

• Origin & Adoption — Introduced in 1975, affordable & efficient
   
• Significance in NES — Pivotal for game development in the platform
   
• Lasting Legacy — remains a symbol of the pioneering era of gaming

The Heart of Retro Gaming: The 6502 Chip

6502 Instruction Set

• NES 6502 instruction set: The heart of classic gaming.
   
• The PPU: a chip so mysterious even Nintendo had to guess what it was doing half the time.
   
• If you're writing a NES emulator and it works the first time... you didn't emulate it.

NES Emulation Basics

The NES Architecture

A more familiar picture

Hard awakening...

But we found our way...

Training Progress:
Dreams get better

Like RL agents, world models improve over time

Demo

Strengths & Limitations

The Road Ahead

Conclusions

Questions?

Links of Interest

• RL - https://www.youtube.com/watch?v=MXpNf0fqI6E
• VPT - https://www.youtube.com/watch?v=9axsVfOEdGQ
• RNN - https://www.youtube.com/watch?v=9HaKyGNhMvY
• https://slides.com/raulg-roagomez