The significance of distributed optimization algorithms manifests through growing interest in various application domains. It finds its use in Big Data analytics, distributed machine learning, distributed control, vehicle networks and smart grid, inter alia. This thesis focuses on primal and dual distributed convex optimization methods. First, it introduces a general algorithmic framework of first and second order methods of primal type, that utilize the concepts of sparsified communications and computations across a connected graph of working nodes. Besides several already existing methods, the framework also includes novel variants that utilize unidirectional communication. Although there have been a remarkable amount of theory and theoretical advances in this field, practical evaluations of methods on real data and practical large scale and High Performance Computing (HPC) systems are of much smaller volume. Therefore, we developed the implementations and performed various numerical evaluations of the proposed methods in an actual, parallel programming environment. The implementations were developed using the Message Passing Interface (MPI) and tested on a High Performance Computing cluster. These empirical evaluations result with very useful insights and guidelines regarding performance and highlights the most importantcommunication-computational tradeoffs in a real execution environment. As there exists a wide set of machine learning algorithms that can be viewed as optimization problems, distributed optimization plays a significant role in this area. The thesis also presents an algorithm for convex clustering, based on the dual method Alternating Direction Method of Multipliers (ADMM), that relies on COMPS Superscalar (COMPSs) framework for parallelization. We provide results of extensive numerical evaluations of the algorithm on a HPC cluster environment, to demonstrate the high degree of scalability and efficiency of the method, compared to existing alternative solvers for convex clustering. Theprogram code for the developed algorithms is open-source and available in thecorresponding repositories.