As pedestrians are non-rigid objects, effective invariant representation of their visual features is the key to improving recognition performance. In natural visual scenes, moving pedestrians often undergo changes in scale, background, and pose, which creates obstacles for existing techniques for extracting these irregular features. The issue was addressed by exploring the problem of invariant recognition of moving pedestrians based on the neural structural characteristics of mammalian retinas, and a Moving Pedestrian Detection Neural Network (MPDNN) was proposed for visual scenes. MPDNN was composed of two neural modules: the presynaptic network and the postsynaptic network. The presynaptic network was used to perceive low-level visual motion cues representing the moving object and extract the object’s binarized visual information, and the postsynaptic network was utilized to take advantage of the sparse invariant response properties in the biological visual system and use the invariant relationship between large concave and convex regions of the object’s contour after continuous shape changes, then, stably changed visual features were encoded from low-level motion cues to build invariant representations of pedestrians. Experimental results show that MPDNN achieves a 96.96% cross-domain detection accuracy on the public datasets CUHK Avenue and EPFL, which is 4.52 percentage points higher than the SOTA (State of the Art) model; MPDNN demonstrates good robustness on scale and motion posture variation datasets, with accuracy of 89.48% and 91.45%, respectively. The effectiveness of the biological invariant object recognition mechanism in moving pedestrian detection was validated by the above experimental results.