With the increase in collaboration among researchers of various disciplines, changing the research topic or working on multiple topics is not an unusual behavior. Several comprehensive efforts have been made for predicting, quantifying, and studying the researcher's impact. The question, that how the change in the field of interest over time or working in more than one topics can influence the scientific impact, remains unanswered. In this research, we study the effect of topic drift on the scientific impact of an author. We apply Author Conference Topic (ACT) model to extract topic distribution of individual authors who are working on multiple topics to compare and analyze with authors who work on a single topic. We analyze the productivity of the authors on the basis of publication count, citation count and h-index. We find that authors who stick to one topic, produce a higher impact and gain more attention. To further strengthen our results we gather the h-index of top-ranked authors working on one topic and top-ranked authors working on multiple topics and examine whether there are similar trends in their progress. The results show an evidence of significant impact of topic drift on career choices of researchers.