Search engines look at a variety of signals for ranking purposes, as well as spam detection purposes. This includes a site review to determine if it's over optimized, and can result in the rankings for a site being dropped to a lower position than they may otherwise occupy.