encoder - How does a ball mouse know the direction?

Foreach axis on a mechanical (ball) mouse there is an IR transmitter and receiver pair with a wheel that looks like having only simple slits.

How is the direction calculated?

Answer

There's not just an IR transmitter/receiver pair, there are two of them. Between transmitters and receivers there's a slotted wheel, which, when rotating, causes a pulse train in the receiver. (The light from the transmitter is blocked, can pass, blocked again, and so on.)

The trick is how the two receivers are placed, namely in quadrature.

This means that the pulses of one receiver precede the pulses of the other by a number of degrees (ideally 90°). If the wheel turn the other way the same pulses now lag the others.

Notice that on a rising edge of channel A the B channel is at a high level when turning one way, and low when turning the other way.

edit (about absolute encoders)
I wasn't completely satisfied with my reply to JGord's comment (some inaccuracies), hence this reprise
The system described above is known as an incremental encoder because it detects relative changes, from one position to the next. Over a full rotation the codes are repeated a number of times, so you can't know your absolute position just by looking at the code.
To overcome this there exist absolute encoders. Instead of two channels in quadrature they have a lot more channels creating a unique pattern for each rotation position. A 10 channel encoder can tell \$2^{10}\$ or 1024 different positions apart. Shaft encoders in robots are even more accurate.

The specific pattern is typical of Gray coding.

about Gray coding
Ordinary binary has the disadvantage that code transitions may create erroneous codes during the transition. Take for instance the change from 0111 (7) to 1000 (8). If the leftmost bit is a bit faster than the others you will see for a moment 1111 (15), which is totally off.
Gray code overcomes this by rearranging the codes so that there's only 1 bit changing at a time.

Absolute encoders won't help you to find the absolute mouse position, however, because the wheel rotates several times while you're moving the mouse. The "unique" pattern will repeat every few mm and isn't so unique after all. Besides, it's always possible to move the mouse when the computer is off, or you can lift the mouse and put it down again a bit further. Both actions will go undetected.

---

Further reading
"Control Shaft Encoders" \$-\$ Circuit Cellar issue 250, May 2011, p.28 ff