Vacuum fluctuations of a scalar field in a rectangular waveguide