how many people lived in a castle