Peptide-MHC complexes are central components of the immune system, and understanding the mechanism behind stable peptide-MHC binding will aid the development of immunotherapies. While MHC binding is mostly influenced by the identity of the so-called anchor positions of the peptide, secondary interactions from non-anchor positions are known to play a role in complex stability. However, current HLA-binding prediction methods lack an atomistic analysis of the major conformational states of the system, and might underestimate the impact of secondary interactions. In this work, we present an atomically-detailed analysis of peptide-MHC binding that can reveal the contributions of any interaction towards stability. We propose a simulation framework that uses both umbrella sampling and adaptive sampling to generate a Markov state model (MSM) for a peptide from SARS-CoV (QFKDNVILL), bound to one of the most prevalent MHC receptors in humans (HLA-A*24:02). While our model reaffirms the importance of the anchor positions of the peptide in establishing stable interactions for binding, our model also reveals the underestimated importance of position 4 (p4), a non-anchor position. We confirmed our results by simulating the impact of specific peptide mutations, and validated these predictions through competitive binding assays. Remarkably, by comparing the MSM of the wild-type system with those of the D4A and D4P mutations, our modeling reveals stark differences in unbinding pathways. The analysis presented here can be applied to any peptide-MHC complex of interest with a 3D model as input, representing an important step towards comprehensive and accurate modeling of the MHC class I pathway.