Peptide sequencing using tandem mass spectrometry for proteolytically digested peptides (referred to as peptide identification) is a foundational technology in proteomics research. Current de novo peptide sequencing algorithms face challenges in identifying phosphopeptides accurately, which are of significant biological importance. The primary reason is the complex fragmentation patterns induced by phosphorylation, the frequent occurrence of neutral loss peaks, and the low abundance of phosphopeptides' mass spectrum in conventional mass spectrometric data. To address these issues, a Transformer and Gated Recurrent Unit (GRU)-based de novo sequencing algorithm for phosphopeptides was proposed, namely TGNovo. A spectrum graph was introduced in TGNovo to model the mass differences between peaks explicitly, guiding the Transformer encoder to capture spectral features. The Transformer module and the GRU module jointly model the association between spectral and amino acid sequence features and the dependencies among spectral peaks and amino acids, respectively, working in concert to achieve peptide reconstruction. Compared to the fully Transformer-based de novo sequencing algorithm Casanovo, TGNovo fully utilizes prior spectral information through the spectrum graph and GRU module, enhancing the model's ability to model spectrum graph. In evaluations of phosphopeptide fragments across species, TGNovo outperforms Casanovo with average improvements of 16.5 percentage points in peptide-level recall and 37.1 percentage points in amino acid-level recall. Additionally, experimental results on an immune peptide dataset show that TGNovo-identified high-confidence antigenic peptides cover 86% of the database search results.