Objective In the current ECG diagnosis system, the expert system lacks self-learning mechanism and the neural network system does not have interpretability for its black-box characteristic. Therefore, in this paper we propose an auto diagnosis system with the advantages of both expert system and network system. Methods There are several modules in our system, including feature extraction module, diagnosis matrix module and diagnosis inference system. First, we extract semantic features from ECG wave data, then, we combine this with diagnosis matrix to compute the probability of each disease the patient may have. Finally, we diagnose the disease by comparing with a threshold value. In the experiment, we use disease records which never used before to test our system. Results In the experiment, we apply our algorithm to 1200 patients’records from the PhysioBank database, and the average accuracy is 95.2%. Conclusions The algorithm based on semantic features has a high diagnosis accuracy with the advantages of the accuracy of the neural network and the interpretability of expert system.