Gravitational-wave emission from extreme-mass-ratio inspirals (EMRIs) is expected to be a key source for the Laser Interferometer Space Antenna (LISA), a future space-based gravitational-wave detector. In this thesis, we detail an approach to model these systems through a perturbative method known as gravitational self-force theory. Accurate EMRI science requires us to go to second order in perturbation theory, which introduces a number of obstacles. One major problem that we focus on ameliorating in this thesis is the strong divergence encountered on the worldline of the small object. This divergence creates a severe computational cost in numerical simulations and hinders the rapid calculations that are required for waveform generation for LISA. However, building on previous work by Pound [Phys. Rev. D 95, 104056 (2017)], we develop a class of “highly regular” gauges with a weaker singularity structure. We calculate all orders of the metric perturbations required for numerical implementation and generate fully covariant and generic coordinate-expansion expressions for the metric perturbations in this class of gauges. Not only will the weaker divergences enable quicker numerical calculations, they also allow us to rigorously derive a pointlike second-order stress-energy tensor for the small object. We demonstrate that the form of this second-order stress-energy tensor is valid in any smoothly related gauge and, using a specific distributional definition, also valid in a widely used gauge in self-force calculations, the Lorenz gauge. This stress-energy tensor can then be used as part of the source when solving for the full, physical fields at second order and we outline how this can be done through the introduction of a counter term that cancels the most singular part of the second-order source in the Lorenz gauge. Finally, we present the calculation of the gauge vector required to transform from the Lorenz gauge to the highly regular gauge and provide it in mode-decomposed form for the case of quasicircular orbits in Schwarzschild spacetime. While this work is motivated by EMRIs, much of the work in this thesis is valid for a small object in any vacuum background spacetime with an external lengthscale much larger than the size of the small object.