The world is becoming increasingly interconnected, which makes events occurring at one time and location trigger new events, in the future and other locations. As a result, such sequences of events have a contagious character. The class of mutually exciting point processes, also known as multivariate Hawkes processes, is a natural contender to model the contagious behavior of such sequences of events. While much research has been conducted on the univariate, self-exciting point processes, there are certain unexplored theoretical aspects of the multivariate, mutually exciting point processes which are addressed in this thesis. First, we characterize the distribution of the point process in a general, non-Markovian setting. This characterization is obtained by exploiting the branching representation to derive a fixed-point relationship. We further derive convergence results and the asymptotic tail behavior. Second, we consider the compound multivariate Hawkes process, motivated by the insurance industry to model risk processes. We derive a Large Deviations Principle and use it to derive properties of the probability of ruin as well as exceedance probabilities. Furthermore, we develop an importance sampling estimator to evaluate the typically small probabilities and then prove its optimality. Finally, we consider the special case where the process exhibits the Markov property. This gives a more explicit characterization of the distribution of the point process in terms of a system of ordinary differential equations. We then reveal a nested structure within the computations of moments, yielding fast computational means to find joint- and cross-moments up to arbitrary order.