Paul Hollywood says his former Bake Off colleagues "abandoned" the show