Andrew Lincoln always worried that his Love Actually character was “creepy”