SHAP

SHAP (SHapley Additive exPlanations) is the game-theoretic method for explaining any machine learning model's output: it assigns a mathematically optimal Shapley value to each feature for every prediction.

SHAP is a powerful, unified framework for model interpretability, connecting optimal credit allocation to local explanations. It uses classic Shapley values from cooperative game theory to fairly distribute the prediction's difference (from the baseline average) among the input features, providing a solid theoretical foundation. The technology is model-agnostic but offers highly optimized implementations like Tree SHAP for complex models (e.g., XGBoost, LightGBM), making it computationally feasible. SHAP is essential for achieving model transparency, conducting robust error analysis, and ensuring regulatory compliance in critical domains like finance and healthcare.

https://shap.readthedocs.io/en/latest/