Visual working memory performance degrades both as the number of memorized items (set size) increases and as the retention interval (delay duration) lengthens. However, whether these two primary constraints operate independently or interact during maintenance—and whether memory representations systematically drift toward canonical feature values over time as predicted by stable attractor network models—has remained unresolved. To address methodological limitations and inconsistencies across prior studies, we conducted six new delayed estimation experiments varying set size and delay duration for color and orientation stimuli and reanalyzed seven previously published datasets using a unified analytical framework that separates response variability, stimulus-specific biases, and swap (intrusion) errors. Across datasets, recall variability increased with both longer retention intervals and larger set sizes, and crucially these factors interacted such that the degrading effect of delay duration was amplified at higher memory loads. By contrast, while systematic stimulus-specific biases were reliably present, bias amplitude did not vary as a function of set size or retention interval. These findings support diffusion accounts of working memory maintenance in which random noise accumulates at a load-dependent rate, while challenging stable attractor accounts of systematic drift during retention.