Climatic changes during glacial periods have had a major influence on the recent evolutionary history of living organisms, even in the warm temperate forests, where the land was not covered with ice sheets. We investigated the phylogeographical patterns of the weevil, Rhynchaenus dorsoplanatus (Roelofs) (Coleoptera: Curculionidae), a host-specific leaf miner of Castanopsis (Fagaceae) that grows in the broadleaved evergreen forests of Japan. We examined 2343 bp of mitochondrial DNA (mtDNA) for 171 individuals collected from new infested leaves of Castanopsis at 55 localities in Japan. Spatial analysis of molecular variance showed a significant difference (φ;_CT = 0.14, P < 0.001) in genetic structure of the leaf-mining R. dorsoplanatus between the southwestern and the northeastern areas within the main islands of Japan. We also observed a large gap between the southwestern and northeastern clades (φ;_CT = 0.79, P < 0.001) with respect to the mtDNA sequences of the seed-boring Cuivulio hilgendorfi (Harold) (Coleoptera: Curculionidae), which shares the same host plant species. The demographic population expansion was detected and the beginning time of the expansion was dated to the end of the last glacial period in both weevil species. The congruent phylogeographical patterns observed in the leaf-mining and seed-boring weevil species reinforce the robustness of the deduced glacial and postglacial history of Castanopsis-associated organisms. These data suggest that the host-specific parasitic insects and their associated Castanopsis forests were likely restricted to separate southwestern and northeastern refugia within the main islands of Japan during repeated glacial periods in the Quaternary.