Dilute palladium-in-gold alloys have potential as efficient oxidation catalysts; controlling the Pd surface distribution is critical. Here, the activity for CO oxidation catalyzed by robust dilute Pd-in-Au nanoparticles supported on raspberry-colloid-templated (RCT) silica depends on the pretreatment and gas environment. The activities of oxygen-pretreated catalysts are different in light-off studies versus after long-term use. Transient increases in activity are also induced by flowing CO/He at 553 K. Altogether, these results indicate changes in Pd distribution at the surface induced by reactive gases and that light-off studies alone are not adequate for evaluation of alloy catalyst performance. Kinetic studies show evidence of both isolated and multiple Pd atoms. A dual-site mechanism is operative over Pd0.02Au0.98 RCT-SiO2, whereas a single-site mechanism governs reaction over Pd0.10Au0.90 RCT-SiO2. The distinct mechanisms suggest that tuning the ratio of isolated to clustered Pd sites is possible, underscoring the importance of characterization under reaction conditions.