The Duke of Sussex was adamant that he would not move back to the UK with his family due to security concerns.