When this happens, do the browsers go instantly to the error screen, or do they try connecting for some amount of time before returning an error?
The first thing I would check would be the apache logs. Check error_log to see if it is reporting anything during those times, and also check access_log to see if there is weird traffic at the times when this is happening.