Accurately distinguishing GlomeruloNephritis （GN） subtypes is crucial for clinical decision-making， however， GN automated diagnosis faces challenges in modeling the intrinsic relationships between non-consecutive Transmission Electron Microscopy （TEM） images within a bag and utilizing the fine-grained expert knowledge in pathology reports effectively. Therefore， an image-text multimodal model， CLIP-HaRe （CLIP framework with HTA and RME）， was proposed， comprising a Hierarchical Transformer Aggregation （HTA） module and a Report Memory Enhancement （RME） module. In HTA module， a hierarchical aggregation strategy “sub-image-instance-bag” was utilized， in which a self-attention mechanism was employed to model the semantic relationships among non-consecutive TEM images， and max-pooling was used to extract key lesion features. In RME module， the pathology report memory banks were constructed using modern Hopfield network， the class prompts generated by a Large Language Model （LLM） were employed for query， and fine-grained textual knowledge was integrated from the reports， thereby enhancing the disease-specific representation capability of the prompts. Experimental results in the disease classification task on a constructed dataset of three GN subtypes show that the proposed model achieves an accuracy of 0.773 7， a precision of 0.781 6， a recall of 0.772 3， an F1 score of 0.772 7， and an Area Under the receiver operating Characteristic curve （AUC） of 0.930 8. These metrics represent improvements of 11.01， 10.68， 10.69， 11.04， and 9.85 percentage points， respectively， over a baseline method Linear-Probe. CLIP-HaRe enables the automated classification of GN subtypes by fusing image and text information， with its significant performance enhancement over the baseline， validating its potential for application in assisting clinical diagnosis.