Precise segmentation of colon polyps in gastrointestinal endoscopy images holds significant clinical value. However, the traditional segmentation methods often struggle with capturing enough fine details and rely on large-scale data heavily, leading to poor performance when addressing complex polyp morphologies. Although Segment Anything Model (SAM) has notable progress in natural image segmentation, the ideal effect in polyp segmentation task cannot be achieved by SAM methods due to domain differences between natural and medical images. To address this issue, a lightweight fine-tuning method based on SAM architecture was proposed, named Segment Anything Model for Colon Polyps (SAMCP). In this method, a streamlined adapter module focusing on channel-dimension information was introduced, a joint loss function was simplified using Dice and Intersection over Union (IoU), and parameters of the original image encoder and prompt encoder were frozen during training to enhance polyp segmentation performance with low training cost. Experimental results on three public datasets comparing SAMCP with nine advanced methods demonstrate that SAMCP outperforms other SAM methods. Specifically, SAMCP improves the Dice and IoU values by 56.7% and 84.5%, respectively, on the Kvasir-SEG dataset, by 46.0% and 86.0%, respectively, on the CVC-ClinicDB, and by 95.3% and 122.2%, respectively, on the CVC-ColonDB dataset, surpassing the current best performance of SAM-based methods. With the introduction of point-based prompts, even with a single click, SAMCP can also outperform other SAM-based methods. The above validates that SAMCP performs well in handling complex shapes and local details, providing physicians with more precise segmentation guidance.