Speaking through my phone to an outside person is way too loud on the doorbell

When I speak through my phone to an outside person, the doorbell is way too loud on the doorbell. The whole street can hear and I want to turn it down. I have tried changing the “ringer” setting within the general settings tab but this only changes the sound sound of the doorbell. As soon as I answer with my voice it goes back to being extremely loud. Hope you can help with this. Thanks in advance.