Removing salt and pepper noise is an active research area in image processing. In this paper, a two-phase method is proposed for removing salt and pepper noise while preserving edges and fine details. In the first phase, noise candidate pixels are detected which are likely to be contaminated by noise. In the second phase, only noise candidate pixels are restored using adaptive median filter. In terms of noise detection, a two-stage method is utilized. At first, a thresholding is applied on the image to initial estimation of the noise candidate pixels. Since some pixels in the image may be similar to the salt and pepper noise, these pixels are mistakenly identified as noise. Hence, in the second step of the noise detection, the pixon-based segmentation is used to identify the salt and pepper noise pixels more accurately. Pixon is the neighboring pixels with similar gray levels. The proposed method was evaluated on several noisy images, and the results show the accuracy of the proposed method in salt and pepper noise removal and outperforms to several existing methods.