How Game Theory Works: The Math of Strategic Decision Making

What's the best move when the best move depends on what everyone else does? Game theory answers that with a fixed point, not a formula, and the fixed point is often worse than everyone hoped for.

By Petrus Sheya

July 19, 2026 · 6 min read

What's the best move when the best move depends on what everyone else does?

Normal optimization doesn't help here. You can't just maximize your outcome, because your outcome isn't only yours to control. Your opponent reacts to you. You react to them. They react to your reaction.

It's a hall of mirrors. You guess what they'll do. They guess what you'll guess. Where does it end?

That's the question game theory answers. And the answer isn't a bigger formula. It's a different kind of question entirely: is there a point where nobody wants to change their mind?


Why you can't just optimize alone

Picture two people on a seesaw, each shifting their weight to react to the other. Push too hard, the other side pushes back. Push too little, you tip.

Eventually, if both keep adjusting, the seesaw settles. Not because nobody is pushing, but because both sides are pushing exactly hard enough to cancel out.

That settled position is the whole idea behind game theory. We call it an equilibrium. Not the best possible outcome. Just the one where nobody, acting alone, wants to move.

Let's make this concrete with an actual game.


A concrete trap: the Prisoner's Dilemma

Two people get arrested together. Each can stay silent (cooperate with their partner) or rat the other out (defect). Neither knows what the other will choose.

If both stay silent, they both get a light sentence. If both talk, they both get a medium sentence. But if one talks and the other doesn't, the talker walks free and the silent one gets crushed.

We write the outcomes as a payoff to each player, higher is better:

Click any cell to see both payoffs. Toggle best responses to see which cell neither player wants to leave.

B: CooperateB: DefectA: CooperateA: Defect3, 30, 55, 01, 1
Payoff to A
Payoff to B
Nash equilibrium?

Click through the four cells. Then hit "Show best responses." Watch what happens: no matter what B does, A does better by defecting. No matter what A does, B does better by defecting too.

So both defect. Even though both cooperating would have paid more.

That's the trap. Each player is making the individually smart choice. The result is collectively dumb. This is the single most important idea in the whole subject: rational choices by each player don't add up to the best outcome for the group.


What makes a point an equilibrium

Here's the idea in plain English before we name it: a choice is stable if, given what everyone else is doing, you'd make the same choice again.

We call this a best response. Player A's best response to "B cooperates" is whatever gives A the highest payoff, given that B cooperates. Same for B.

When every player is simultaneously playing a best response to everyone else, nobody has a reason to switch. That's a Nash equilibrium, named after John Nash. We write it as: for every player ii,

ui(si,si)ui(si,si)for all siu_i(s_i^*, s_{-i}^*) \geq u_i(s_i, s_{-i}^*) \quad \text{for all } s_i

In words: your actual payoff, given what everyone else is doing, is at least as good as any other choice you could have made instead.

In the Prisoner's Dilemma, (Defect, Defect) is the only cell where both arrows point inward at once. That's why it's stable, and why it's a trap.


When there's no stable choice at all

But not every game has a nice, stable pure choice. Sometimes the only way to be unpredictable is to actually be unpredictable.

Take Matching Pennies. Player A picks Heads or Tails. Player B, without seeing A's choice, also picks Heads or Tails. If they match, B wins. If they don't, A wins.

Try to find a stable pure choice here. You can't. If A always plays Heads, B just always plays Heads too, and wins every time. But then A should switch to Tails. But then B should switch to Tails. It never settles... unless someone randomizes.

Drag p, the chance A plays Heads. Watch B's two expected payoffs cross, only p = 0.5 leaves B with no better option.

00.250.50.751-101p = 0.5EB(Heads)EB(Tails)
EB(Heads)0.00
EB(Tails)0.00
B's best responseIndifferent

Drag pp, the probability A plays Heads. Notice B's two expected payoffs, one for playing Heads, one for playing Tails, form straight lines that cross exactly once.

Away from that crossing point, B has a strict best response and would happily exploit it. Only at p=0.5p = 0.5 is B indifferent between both choices. That's the only value of pp where A isn't handing B a free win.

This is a mixed strategy equilibrium: instead of picking one action, each player picks a probability that makes the other player indifferent. Nobody can improve by responding differently, because there's nothing left to exploit.


Repetition changes everything

The Prisoner's Dilemma looks hopeless: rational players always end up defecting. But that's only true if you play once and never see each other again.

What if you play the same opponent over and over?

Now your current move can be punished or rewarded in the next round. A simple strategy called tit-for-tat captures this: cooperate on the first move, then just copy whatever your opponent did last time.

Player A always mirrors your last move (tit-for-tat). Pick B's strategy, press Play, and watch scores diverge over 20 rounds.

051015200255075100A (Tit-for-Tat)B (Always Defect)
Round0 / 20
A's move
B's move
Score A / B0 / 0

Pick an opponent strategy and press Play. Notice the pattern: against Always Cooperate, tit-for-tat cooperates forever and both scores climb steadily. Against Always Defect, tit-for-tat gets burned exactly once, then defects forever after, capping the damage. Against Random, the scores wobble but tit-for-tat never falls far behind.

Repetition turns a one-shot trap into a relationship. The threat of future retaliation is enough to make cooperation the smart move, even though cooperation alone was never smart in a single round.


Equilibrium with continuous choices

So far, every choice has been discrete: cooperate or defect, Heads or Tails. But real strategic decisions are often continuous, like how much to produce, how much to charge, how much to bid.

Consider two companies competing on how much of a product to make. Call their quantities qAq_A and qBq_B. The more they make combined, the lower the price falls, so more output for one firm is bad news for the other's profit.

Each firm has a reaction curve, a rule for its profit-maximizing output given what the rival makes:

qB=acqA2q_B^* = \frac{a - c - q_A}{2}

Drag firm A's output. Firm B always produces its profit-maximizing response, watch the point slide until both curves cross.

00151530304545606075759090q_Aq_BNash (30, 30)B's reaction curveA's reaction curve
B's best response40.0
Profit A400
Profit B1600
At equilibrium?No

Drag firm A's output. Firm B always slides to its own best response, tracked by the point on the chart. Notice it moves along B's reaction curve, but only lands where A's reaction curve also crosses does the point stop being able to improve for either side.

That crossing point is the Nash equilibrium of the game, here at qA=qB=30q_A = q_B = 30. Move A away from it, B reacts, and the system doesn't rest until both are back at that intersection.


The short version

Whenever your best move depends on someone else's move, optimizing alone doesn't work. You need to find the point where every player is simultaneously doing their best, given what everyone else is doing. That point is a Nash equilibrium.

Sometimes it's a single stable choice, even a bad one, like both prisoners defecting. Sometimes there's no stable pure choice, and the only equilibrium is to randomize in just the right way. Sometimes repetition changes the incentives entirely, turning a trap into cooperation. And sometimes the choices are continuous, and equilibrium is just where two reaction curves cross.

In every case, the math is the same question: is there a point where nobody, acting alone, wants to change their mind?


All visualizations are interactive React components running entirely in your browser, built with plain SVG and no external libraries. The repeated-game simulator uses requestAnimationFrame to step through rounds; everything else responds instantly to drags and clicks.