Objective Aiming at solving the problem of clinical squeaking, the mechanism of clinical squeaking in ceramic-on-ceramic total hip arthoplasty (THA) is explored. Methods A finite element model containing human pelvis, ceramic hip endoprosthesis and a femur is established. Dynamic responses of the system under friction are recorded by the complex eigenvalue method. Then the cause of squeaking is investigated by comparing the vibration characteristics of the components. Results Increase in friction coefficient leads to mode couplings and gives rise to unstable modes of the prosthesis system. The bending and torsional vibration of the femoral component mainly contributes to the unstable mode shapes of the system. Similar mode shapes are found in modal analysis of the femoral component, and the corresponding frequencies are similar as well. In addition, the unstable modes at frequency of about 1250Hz, 2400Hz and 3930Hz have larger damping ratio. The femoral component reveals two bending resonances around the unstable modal frequency of 1250Hz and 2400Hz, and a torsional resonance around unstable modal frequency of 3930Hz, which are the sources of the self-excited vibration of the ceramic hip endoprosthesis system . Conclusions Mode coupling may be the candidate mechanism of self-excited oscillation of the prosthesis system. Pelvis is involved in the noise emitting and causes unstable modes of the system as well.