When multiple items are held in working memory or encountered across time, recalled features are systematically biased toward (attraction) or away from (repulsion) competing representations. Existing accounts typically link the direction of these biases to absolute encoding or maintenance noise in the target representation. Here, we propose and test a normative “demixing” model in which the brain attempts to disentangle overlapping neural signals from competing items. A key prediction of this model is that the direction of memory biases is determined not by absolute noise alone, but by the relative distribution of noise across competing representations. Across four continuous color-report experiments that independently manipulated the noise levels of target and non-target items using color mosaics, we show that targets with the same absolute noise level can be either attracted to or repelled from competing items depending on their relative noise configuration. These findings establish relative representational noise as a fundamental determinant of attraction and repulsion biases in visual working memory.