Models of hybrid zone dynamics incorporate different patterns of hybrid fitness relative to parental species fitness. An important but understudied source of variation underlying these fitness differences is the environment. We investigated the performance of two willow species and their F₁, F₂, and backcross hybrids using a common-garden experiment with six replicated gardens that differed in soil moisture. Aboveground biomass, catkin production, seed production per catkin, and seed germination rate were significantly different among genetic classes. For aboveground biomass and catkin production, hybrids generally had intermediate or inferior performance compared to parent species. Salix eriocephala had the highest performance for all performance measures, but in two gardens F₁ plants had superior or equal performance for aboveground biomass and female catkin production. Salix eriocephala and backcrosses to S. eriocephala had the highest numbers of filled seeds per catkin and the highest estimates of total fitness in all gardens. Measures of filled seeds per catkin and germination rate tend to support the model of endogenous hybrid unfitness, and these two measures had major effects on estimates of total seed production per catkin. We also estimated how the two willow species differ genetically in these fitness measures using line cross analysis. We found a complex genetic architecture underlying the fitness differences between species that involved additive, dominance, and epistatic genetic effects for all fitness measures. The environment was important in the expression of these genetic differences, because the type of epistasis differed among the gardens for aboveground biomass and for female catkin production. These findings suggest that fine-scale environmental variation can have a significant impact on hybrid fitness in hybrid zones where parents and hybrids are widely interspersed.