Peerzada said the rumours had not personally affected him, but they had caused concern among his fans