This paper presents a new approach for modeling the normal behaviors and detecting the abnormal behaviors. The approach consists of several main steps. First, using a detection method, the foreground and background regions are separated. Then, the busy-idle rates are defined as the behavioral features and, based on these features, a behavioral model is extracted for each pixel block. In the following, spectral clustering is used to classify the normal behaviors on the condition that a set of normal data is provided. In the classification process, the pixel blocks with similar behaviors are grouped together. A behavioral model is defined for each group of the blocks with similar behaviors. The behavioral model adopted in this paper is Hidden Markov Model. The results of the obtained classification and normal behaviors are used to detect the abnormal behaviors; i.e., based on the normal-behavior model for each cluster, if the observation sequence probability given by the normal behavior model is lower than the threshold level, the pixel block is identified as the region in which the abnormal behaviors happened. The experimental results obtained from video data confirm the efficiency, accuracy, and speed of the approach adopted in this paper.