Voice, as a convenient and efficient way of information delivery, has a significant advantage over the conventional keyboard-based input methods, especially on small mobile devices such as smartphones and smartwatches. However, the human voice could often be exposed to the public, which allows an attacker to quickly collect sound samples of targeted victims and further launch voice impersonation attacks to spoof those voicebased applications. In this paper, we propose the design and implementation of a robust software-only voice impersonation defense system, which is tailored for mobile platforms and can be easily integrated with existing off-the-shelf smart devices. In our system, we explore magnetic field emitted from loudspeakers as the essential characteristic for detecting machine-based voice impersonation attacks. Furthermore, we use a state-of-the-art automatic speaker verification system to defend against human imitation attacks. Our advanced sensory data processing technique achieves fast authentication speed (6.1 s) which makes it suitable for mobile platforms. Finally, our evaluation results show that our system achieves significantly high accuracy (100%) and low (0%) equal error rates (EERs) in detecting the machinebased voice impersonation attack on smartphones.