Transformer Architecture Explained

Mathematically:

where $d_k$ is the dimensionality of the Key vectors, and the scaling factor $\sqrt{d_k}$ stabilizes gradients.