Generative models are probabilistic models which aim at approximating the process by which a given dataset is generated. They are well suited to the unsupervised learning setup and constitute intuitive and powerful models providing an interpretable representation of the data. However, learning even simple generative models can be challenging in the traditional Maximum Likelihood framework, due to the inflexibility of the training objective used. Motivated by its topological properties, we will show in this thesis how methods based on Optimal Transport can overcome these difficulties and offer competitive alternatives. Firstly, we show that training generative models that combine both discrete and continuous latent variables can be significantly more effective when using Optimal Transport methods. Such intuitive models are highly motivated by the structure of many real-world datasets but remain hard to train with the most common likelihood-based method, often resulting in the collapse of the discrete latent variables. Secondly, we propose a novel approach based on Optimal Transport to training models with fully Markovian deep-latent hierarchies. Probabilistic models with deep latent-variable structures have powerful modelling capacity, but common approaches often fail to leverage deep-latent hierarchies without complex inference and optimisation schemes. Our method successfully leverages the whole hierarchy of the models and shows competitive generative performance while learning smooth latent manifolds through every layers of the latent hierarchy. Finally, we introduce a new training objective to improve the learning of interpretable and disentangled representation of the data. Our method achieves competitive disentanglement relative to state-of-the-art techniques whilst improving the reconstruction and generation performances of the models.