The diversity of multi modal pest and disease data sets leads to low average accuracy of the model. Based on this paper， a multi modal pest and disease detection model based on deep learning is designed. Firstly， multi modal data such as HD image， audio and video are collected， and existing resources are integrated to form a data set. Secondly， data enhancement is carried out to improve the generalization ability of the model and reduce the risk of overfitting. Finally， MobileNet in deep learning is selected as the backbone network to capture key features， and the model training incorporates multi modal data and significantly enhances detection performance through dynamic weight allocation and attention mechanisms. The experimental results show that after 60 iterations， the mAP of the design model reaches 0.87， which is far higher than the 0.72 and 0.79 of the model in literature [1] and [2]. It is proved that the design model can rapidly converge and accurately detect a variety of pests and diseases.