The adoption and abandonment of first names through time is a fascinating phenomenon that may shed light on social dynamics and the forces that determine cultural taste in general. Here we show that baby name dynamics is governed almost solely by deterministic forces, even though the emerging abundance statistics resembles the one obtained from a pure drift model. Exogenous events are shown to affect the name dynamics very rarely, and most of the year-to-year fluctuations around the deterministic trend may be attributed solely to demographic noise. We suggest that the rise and fall of a name reflect an "infection" process with delay and memory. The symmetry between adoption and abandonment speed emerges from our model without further assumptions.