A clustering engine is a search engine which automatically groups search results into related topics. This can help search engines focus results on certain terms.
For instance, the phrase "George Bush" and displayed the clusters that relate to that search. At the top of the list were variations on the president's name but as you went down the list, you could see broader keywords that were clearly relevant to the term "George Bush". The phrase "miserable failure" was not apparent, but it highlight that the term "idiot" clearly appeared in clusters matching the president.
The well known clustering engines are Clusty,Vivisimo, and AltaVista uses a clustering engine called Prisma.