logo

How to use open source foundation models

How to use open source foundation models

The rise of off-the-shelf and foundation models has enabled AI teams to fine-tune existing models and pre-label data faster and more accurately — in short, significantly accelerating AI development. 


Open source foundation models are large pre-trained artificial intelligence models that are trained on a broad dataset that can be used for different tasks with minimal fine-tuning. These models serve as a base for training different kinds of task-specific AI models with adaptability to specific requirements.


Benefits of open source foundation models

Building specialized AI applications typically requires a large amount of effort and funding, but open source foundation models make it more accessible to create specialized AI applications.


Open source foundation models are also released with the code that implements the underlying machine learning algorithm, and in most cases with the data that it was trained on. This means that open source foundational models typically have full reproducibility, and users can review, modify, or contribute to the model which ensures accuracy, transparency, and reliability through peer review.


In addition, while the AI and ML industry tends to move fast, it is still an emerging technology. Through the sharing of information and innovation through open source foundation models, the number of publicly available resources for machine learning and AI building applications keeps increasing.


How to use open source foundation models to overcome AI challenges

Many challenges come with building AI that can be solved by open source foundation models. See the following list for more details.


Overcoming small datasets

One of the primary stumbling blocks of building specialized AI applications is that typically, the amount of data available for these use cases isn’t nearly enough to train a custom model to the required performance level. The lack of sufficient training data can significantly delay an AI project. By leveraging the right foundation model, however, teams can circumvent this issue.


Foundation models are powerful because they are trained on internet-level data, making them ideal for transfer and few-shot learning. By giving the right model just a few labeled examples for your use case, you can create a baseline ML model. For instance, a team working on a plant pathology use case could fine-tune an image classification model such as YOLOv8 on just a few hundred labeled images of healthy and diseased crop samples. The foundation model provides general visual understanding, which is then specialized for your specific task.


Reducing engineering requirements with pre-built capabilities

Foundation models can dramatically reduce the engineering effort required to build AI systems, enabling teams to skip many time-consuming tasks and focus their energy on the end application. Here are a few key ways that these models streamline the development process:


  • Minimal fine-tuning needed for data enrichment: Foundation models can often achieve good zero-shot performance on common use cases. For specialized applications, this performance can translate into a quick and effective data enrichment solution.

  • Accessible through APIs: Many foundation models are available via developer APIs. This means that teams can skip training models entirely and simply call the APIs. Even for teams fine-tuning LLMs, APIs can still reduce much of the engineering lift.

  • Multi-task capabilities: Models like GPT-4 have strong capabilities across many tasks like classification, summarization, and translation, so they can accelerate your AI development regardless of your specialized/niche use case requirements.


Foundation models encapsulate a tremendous amount of solved tasks in a reusable package. AI teams spend less time rebuilding common capabilities from scratch. With proven architectures, features, and performance, they can focus on rapidly building and delivering end applications. The productivity boost enables faster iteration and innovation.


Pre-labeling with open source foundation models

Labeling even a relatively small amount of data for AI development can be slow and expensive if done by humans, especially when domain experts are involved in specialized use cases. 


Leading AI teams have proven that pre-labeling your data — that is, using a model or program to take a first pass at a labeling task — can significantly cut labeling time and expenses. Teams that use a powerful and accurate model for pre-labeling can realize 50% or more in labeling costs. With the advent of foundation models, such high-level pre-labeling workflow is accessible to any AI team, not just those with large budgets and advanced architectures.


Where to find open source foundation models

Most open source foundation models are deep learning models. The reason is neural networks benefit from large datasets and sophisticated architecture that learn and encompass multiple different parameters. It's typically unfeasible to replicate the full training of such models because of the large amount of both code and data.


For example, an example of one of the largest ever released open-source foundation models is YaLM 100B which was trained on 1.7 TB of text for 65 days on a pool of 800 high-grade A100 graphic cards.


Ultimately, GitHub is the largest open-source model repository, but Hugging Face has also become an excellent resource for finding open source foundation models, although they have a large focus on natural language processing (NLP) applications. They have their open proprietary models available for use as well that are easy to use and access through a call from their Python library.


Framework-specific collections such as TensorFlow Hub and PyTorch Hub are good resources, in addition to Model Zoo for a larger, broader collection of frameworks, platforms, and applications.


Deploying open source foundation models

In order to utilize and productionize open source foundation models to develop AI applications, the first step is to download the model library to your development and production environment. From there you can simply use the foundation model directly for inference, or use transfer learning on the foundation model in order to extend its trainable parameters and fine-tune for your specific AI application.


When using an open source foundation model, either by using the original model or a fine-tuned model, something to keep in mind is that a continuous retraining pipeline is often not needed due to the nature of the model.


Final thoughts on how to use open source foundation models

AI has transformed every single industry, but productionizing and building specialized, practical AI applications is still a challenge. Open source foundation models have contributed heavily towards remedying this challenge and expanding the accessibility of building AI applications.


Learn more about how you can leverage the foundation models above for AI development quickly and easily with Labelbox Foundry. Explore and experiment with a variety of foundation models, evaluate the performance of your data, and leverage the best one for your use case.