In recent months, all kinds of image-generating models got the spot-light, opening many possibilities for further research direction, and from the commercial side, many teams will be able to start experimenting and building products on top of them. A sub-area of image generation that picked the most interest in the eye of the public is text-to-image models, most notably Stable Diffusion and MidJourney. Open sourcing Stable Diffusion and free tier of MidJourney allowed many product teams to start building on top of them with little to no resources. However, applying any pre-trained model without proper testing and experimentation creates unknown risks for companies and teams using them. In this paper, we are demonstrating what might happen if such models are used without additional filtering and testing through bias detection. 

Bias Analysis in Stable Diffusion and MidJourney Models

In this paper, we propose a novel gender bias detection method by utilizing attention map for transformer-based models. We 1) give an intuitive gender bias judgement method by comparing the different relation degree between the genders and the occupation according to the attention scores, 2) design a gender bias detector by modifying the attention module, 3) insert the gender bias detector into different positions of the model to present the internal gender bias flow, and 4) draw the consistent gender bias conclusion by scanning the entire Wikipedia, a BERT pretraining dataset. We observe that 1) the attention matrices, Wq and Wk introduce much more gender bias than other modules (including the embedding layer) and 2) the bias degree changes periodically inside of the model (attention matrix Q, K, V, and the remaining part of the attention layer (including the fully-connected layer, the residual connection, and the layer normalization module) enhance the gender bias while the averaged attentions reduces the bias).

Yueying Liang

Papers

Detecting Gender Bias in Transformer-based Models: A Case Study on BERT.