Abstract
Controllable speech synthesis for low-resource languages remains difficult, especially when users require explicit emotion or dialect control without training data. We propose a practical pipeline that achieves zero-shot controllable generation by combining voice conversion (VC) with F0-contour transfer. The system uses a Diffusion-Transformer (DiT) VC backbone and a BigVGAN vocoder, while prosody is manipulated at inference time: RMVPE extracts frame-level F0; an Auto-F0-Adjust module normalizes pitch range across speakers; optional semitone pitch-shifting modulates expressivity; and dynamic time warping aligns target and source contours. The approach preserves linguistic content and timbre from the source while injecting target prosody (emotion or dialectal intonation), and it requires no additional acoustic training. Evaluated on Kazakh speech, we report mean-opinion-scores (MOS) ≥ 4.1, ABX perceptual identification rates of 77–83% for target emotion/dialect cues, and Pearson correlations between converted and reference F0 contours of 0.75–0.82. The system operates with a real-time factor of ~0.4 for short utterances and ~0.7 for longer ones, supporting interactive use. Our contributions are: (1) a deployable, training-free VC+prosody transfer pipeline; (2) an explicit F0 control interface (Auto-F0-Adjust + semitone shifting) that generalizes across speakers; and (3) an evaluation protocol combining MOS, ABX, and F0-correlation to quantify controllability. This work demonstrates a realistic path to emotion/dialect-controlled synthesis in low-resource settings.