This thesis focuses on improving accuracy and assessing robustness of deep learning for medical image analysis. Part I focuses on developing and evaluating techniques to improve accuracy of deep-learning-based algorithms trained using fully-labeled, weakly-labeled, and partially-labeled data. Part II focuses on assessing robustness of deep learning algorithms to adversarial perturbations.