Machine reading comprehension (MRC) is a type of question answering mechanism in which a computer reads documents and answers related questions. The accuracies of recent MRC systems surpass those of humans. However, most MRC systems exhibit significant performance deteriorations when domains are changed. Hence, we propose a self-training framework for MRC. The proposed framework is composed of a pseudo-answer extractor, a pseudo-question generator, and an MRC system. In the source domain, components are pretrained using an MRC training dataset. In the target domain, the performances of the pseudo-question generator and MRC system is improved through a mutual self-training scheme. During the mutual self-training, the pseudo-question generator provides new training data to the MRC system and obtains rewards from the MRC system for reinforcement learning. In experiments with a Wikipedia domain (source domain) and civil affair domain (target domain), an MRC system based on the proposed self-training scheme demonstrates better performances than that based on automatic data augmentation.