Both Susan Saint James and Nancy Walker left the show at the end of the 1975-1976 season. Saint James because of a contract dispute, and Walker because she had just signed a contract with ABC.
The slight modification of the interior set of the McMillan's home provided a continuity problem. In season five's, "Aftershock", the mystery centered around a skeleton found in their brick fireplace which supposedly had been put there when the home was built. However, in at least the first two seasons of the show, their fireplace was of different construction and not made of brick, thereby belying the idea the skeleton could have been there all along.