The increased interest in solar energy as a renewable energy source brings new challenges to the seamless operation of the power grid due to the inherent intermittent availability. Therefore, high precision forecasts are needed to successfully integrate the growing capacity into the grid. However, forecasting the yield from solar panels is difficult due to limited data availability. Instead, GHI is forecasted. Several deep learning approaches have been proposed throughout the years, however deep learning is often considered as black-box and therefore disregarded by decision makers. In this paper we propose the Probabilistic Solar Irradiance Transformer (ProSIT), a novel end-to-end deep learning architecture for interpretable probabilistic multi-horizon forecasting of GHI. To learn both long and short-term temporal dependencies across the entire input sequence ProSIT uses several complex components to encode the high-dimensional feature space such as bi-directional recurrent networks, temporal multi-head self attention layers, and temporal convolutional layers. ProSIT also features residual connections and gating mechanisms to suppress superfluous components ad hoc. We conduct several experiments benchmarking the performance of the model, and demonstrate that ProSIT achieves state-of-the-art performance on two real-world datasets.