“I believe that if you were not in disgrace with the Court, and had not been overshadowed by that cloud for years past, a letter de cachet would have sent me to some fortress indefinitely.”