In this paper, we address the problem of processing semantic data streams. The semantic annotation of sensor data is one of the solutions to the heterogeneous nature of sensor data streams. Existing systems for publishing semantic streaming data collect stream data and transmit the semantic streaming data to query engines regardless of the queries registered in the query engines. As a large number of sensing devices become available, there is an increasing amount of the stream data, resulting in the performance degradation of a query engine. To remedy this problem, we propose a query-adaptive method of filtering semantic streams. The proposed method filters out sensors and semantic streaming data, which are not related with queries registered in a semantic stream query engine. The approach fairly reduces the data size necessary to answer semantic stream queries and consequently improves the performance of the query processing. To demonstrate the efficiency of our proposal, we present extensive experimental performance evaluations under a variety of sensor streams and query types. Experimental results show that the proposed method dramatically improves the performance of query processing compared to a non-filtering approach.