This thesis presents several novel approaches to achieving label-efficient learning in computer vision. By demonstrating the effectiveness of various algorithms that use a reduced set of labels or even zero labels, we have addressed the main research question across six typical computer vision challenges. Our research highlights the importance of leveraging unlabeled data, exploring alternative learning paradigms, boosting label efficiency by augmentation, and carefully designing network structures to achieve label efficiency. Our results suggest that label-efficient learning has the potential to significantly reduce the amount of labeled data required for training computer vision models, opening up new possibilities for real-world applications. This thesis also presents practical implications for the development of such applications. By reducing the reliance on labeled data, our approach can significantly lower the cost and time required for data annotation. As a result, it becomes easier to develop computer vision solutions for a wide range of industries. However, our study is not without limitations. Further research is needed to explore the full potential of label-efficient learning, including utilizing multi-modalities and Large Language Models, to achieve self-evolution without reliance on labels. Nonetheless, our research provides valuable insights and a promising direction for future work in this area.