According to William Henry Cooper's account of the town, the house at 4 The Quay was once the Harbour Master's House; originally it had no entrance directly from the Quay, access instead being from the lane behind, until Harbour Master Thomas Kewish had a flight of stairs built so residents could reach the house from the quayside. The adjacent Harbour Master's Office on the site was rebuilt in 2000. The house sits among Castletown's row of harbourfront properties overlooking the historic quay.