I’ve listened to several podcasts, including a couple where the hosts attempt to remove silences using tools such as Audacity’s “Detect Silence.” This can remove long periods of silence between words and sentences where hosts are attempting to think of ideas or if something happens.
However, if you make the final time too short, it makes the conversation sound downright unrealistic. If you’re trying to think of something, you want the listener to hear a pause as you attempt to consider something. If a disconnect happens, or someone fails to respond within a reasonable time, you want the listener to hear time pass before you say so. It makes the conversation sound more realistic.
I recommend thresholding silence to somewhere between 1 and 2 seconds. This is a realistic pause in a normal conversation without being overly large. If you compress a pause to less than a second, then the conversation including the pause can in some cases come off as unrealistic (particularly if the now-removed pause was commented on).
So, don’t eliminate all pauses, try to keep some in your recordings for better results, and a more natural-sounding conversation.