Deep generative modelling is becoming increasingly popular and influential. The applications of this technology are wide-ranging, from photo-editing, speech synthesis to drug discovery. In this thesis, we analyze and improve the flexibility of two types of generative models: normalizing flows and diffusion models. Specifically, in the first part of this thesis we aim to make normalizing flows more expressive by inventing new ways to construct invertible convolutional layers. We shall see that different constructions can be chosen, originating from linear algebra and Fourier analysis. Further, we explore methods to define normalizing flows and diffusion models for discrete spaces. We find new model formulations that can be optimized successfully. Certain variants of these new models have a practical by-product: they can be effectively applied to lossless compression. Furthermore, we shall see that one of these new discrete diffusion models connects a number of well-known generative models. It bridges the gap between discrete diffusion, autoregressive models, and masked language models. Finally, we design a normalizing flow and diffusion model for molecule generation in 3D. To model the discrete atom types, we will incorporate our new techniques to operate on discrete spaces. In addition, since molecules live in physical space, we will demonstrate that it is important to consider the Euclidean symmetries of the positional information.