In recent years, depression, as a prevalent mental health disorder has drawn increasing attention. With the advance of AI technology, automatic and objective diagnosis methods emerge by observing signals like electroencephalogram (EEG) signals, faces and behaviors. In the present paper, we propose gait analysis as a non-invasive method for depression detection. In this study, we propose a prior-prompt-based graph convolution network (PP-GCN) for depression recognition through gait that integrates skeleton and text modalities. Different from the conventional single-modal methods in the present study, we utilize prior knowledge and angle features. We innovatively introduce Generative Action Prompt (GAP), leveraging a pre-trained large language model to generate motion descriptions for different body parts, thereby providing prior knowledge for depression recognition. Additionally, considering the subtle gait feature variations in individuals with depression, we further incorporate a joint-angle-based representation strategy to capture fine-grained variations in movements. Experimental results demonstrate that the proposed model outperforms existing skeleton-based approaches on a large-scale dataset which contains over 25,000 gait sequences from nearly 300 volunteers named D-Gait, achieving excellent performance.