The earliest written records tell us that the area was part of the manor of Wyks Bishop, held by the Bishops of Norwich from the 13th century but we do not know whether or not they had a house on the site. During the reign of Henry VIII the manor was surrendered to the crown, and shortly afterwards granted to Sir John Jermy. There followed a succession of lords of the manor before the title was acquired by John Cobbold in 1812.