Consider the problem of obtaining inferential guarantees on the excess risk in non-parametric regression. We seek a method that requires only black-box access to the regression procedure via a limited number of refits, and allows for data heterogeneity and non-i.i.d. behavior. Under these constraints, we develop a method for black-box validation; it exploits Rademacher residual symmetrization, in the spirit of the wild bootstrap, to construct a synthetic dataset with artificially inflated noise levels. By refitting once on the synthetic dataset, we obtain fully data-dependent non-asymptotic upper bounds on the excess risk without needing hold-out samples or cross-validation. We illustrate the procedure in plug-and-play inverse imaging and photometric redshift prediction in astronomy.