Lily Allen and David Harbour have their own take on those breakup rumours