This is an echo problem. The agent's audio plays through the speaker, the microphone picks it up, and the model treats it as the caller speaking.
Try these fixes in order. First, turn on echo cancellation and noise suppression in your audio capture or telephony layer. Second, lower the speaker volume or use a headset when testing. Third, mute or ignore the incoming audio while the agent is speaking. This is simple and reliable, but it stops callers from interrupting. Fourth, tune turn detection by raising the voice activity threshold and allowing slightly longer silence before a turn ends. Always test on a real speakerphone, since laptop tests hide the issue. Check the realtime API docs for current turn detection options.