The building used for the setting of the hospital (externally) is actually a residential property on Scarborough's south cliff esplanade. It's split into flats and cottages.
The house used for Dr Ormerod and Dr Wetherill's home is a residential property situated directly opposite St Mary's Church in Scarborough and has panoramic views of the North Bay from its garden which can be seen in an episode where they have a garden party.
Robert Daws (Dr. Gordon Ormerod) & Wendy Craig (Matron) also worked together on episode 5.5, Death in Paradise: Lost Identity (2016), of Death in Paradise (2011) as John Green / Marcus Knight & Mary Goodman respectively.