Through technological advancements in various areas of our lives, Conversational Agents progressed in their human-likeness. In the field of HCI, however, the use of conversational fillers (e.g., “um,” “uh,” etc.) by Conversational Agents have not been fully explored in an experimental setting. We observed the effects on user perceptions of Intelligence, Human-likeness and Likability of Conversational Agents by a 2 x 2 experimental design. From the results of 26 total participants, we concluded that 1) the use of fillers by Conversational Agents are perceived as less intelligent and less likable in task-oriented conversations, 2) and the fillers did not have any statistically significant change in perception of human-likeness. However, further examination showed that users reported filler-speaking agents as more entertaining for social-oriented conversations. With these findings, we discuss design implications for voice-based Conversational Agents.