Federated learning， as an important research direction in the field of artificial intelligence， constructs global models through distributed collaborative training under the premise of ensuring data privacy， providing a new paradigm for joint modeling in privacy-sensitive scenarios. However， the existing methods are challenged by performance degradation， training oscillation， and slow convergence when faced with heterogeneous client data. Therefore， a federated learning algorithm based on prototype clustering and Fisher information matrix adaptive weighted fusion， named FedPFA， was proposed. First， the prototypes uploaded by clients were clustered by the server to mitigate data distribution discrepancies and enhance the consistency and generalization capability of the global model. Subsequently， differentiated weights were assigned to the prototypes by incorporating the trace estimated values of the Fisher information matrix from each client， thereby highlighting the contributions of high-quality clients and suppressing interference from noisy or inadequately trained clients during the fusion process. Experimental results on the MNIST， Fashion-MNIST， and CIFAR-10 datasets demonstrate that， under the setting of 20 clients and a participation rate of 0.3， FedPFA achieves accuracy improvements of 19.64， 26.14， and 16.15 percentage points， respectively， compared to the Federated Global prediction Header （FedGH） algorithm. It can be seen that FedPFA ensures performance enhancement， and improves the convergence speed and stability of the global model significantly， exhibiting strong robustness and practical application value on various datasets.