Higher-quality data results in higher-quality models. However, obtaining high-quality data can be challenging. Every ML engineer or labeling operations manager often faces a recurring challenge – how to complete large, complex training data projects in a systematic way.
There are so many variables and decision trees, that it can be challenging to envision the most efficient way to structure, label, and QA your projects. While teams have had to traditionally rely on weekly team meetings or other manual methods, we built Workflows – the ultimate orchestration tool for your training data cycles.
Custom workflows can help optimize how labeled data gets reviewed across multiple tasks and reviewers. Workflows is a new feature that allows teams the flexibility to tailor their review workflows for faster iteration cycles.
Teams can set up dynamic workflows based on attributes, such as annotation type or labeler, to reduce costs and increase labeling and review throughput, quality, and efficiency.
At a glance, Workflows allows you to easily understand the state of your data within your labeling operations pipeline. Rather than relying on manual spreadsheets or ad-hoc methods, you have a systematic way of understanding how many data rows are ready for model training, how many are currently in-review, how many are being re-labeled, and how many haven’t been labeled yet.
By using Workflows, teams have more granular control over how data rows get reviewed – teams can use Workflows to customize their review pipeline to drive efficiency and automation into their review process.
Designed to help save your team labeling time and cost, Workflows enables you to:
By default, there are three tasks that appear when you create a new project:
You can have up to 10 review tasks within a workflow.
How it works with a project’s Overview page
The new overview page provides teams with a holistic view of where your data rows are in the labeling pipeline.
At a glance, you’re able to understand which data rows can be used downstream. For a deeper dive, you can click on a stage to view all data rows in the filtered state in the Data Rows tab.
Batches are a collection of data rows that are queued from Catalog and added to your labeling project. They are critical in enabling faster data-centric iterations and in helping unlock active learning workflows to improve label or model errors.
In the Data Rows tab, you can filter by batch or annotation to send subsets of your data to 'Rework' in bulk. You can also send bulk data rows to different stages of your workflow.
Based on project progress or ad-hoc needs, you can create a new review task. This allows you to modify your workflow based on feedback from a Labeling Operations Manager or based on your team’s labeling performance.
Another reason why teams might want to create an ad-hoc review task is based on subject-matter expert review. Teams can add a review step and set parameters to maximize what gets reviewed at this stage in order to maximize the reviewer’s time and cost.
Teams can also send a data row to a specific review task from the Data Row browser for ad-hoc review. For example, if you find some poorly labeled data rows that you want to send to re-work, you can do so directly from the Data Row browser.
When reviewing your data rows, you can easily approve or reject data rows based on if it's a Benchmark-based project or a Consensus-level batch.
Approving a data row: Data rows that have been approved will be automatically moved to the next review task in your queue. Data rows that are in the "Done" step give you a real-time, high-level view of how much of your data is ready to incorporate into your model.
Rejecting a data row: Data rows that have been rejected will be automatically sent to “Rework” for correction. You can mark the issue and leave a comment as to why the label was rejected – allowing your labelers to understand and correct any labeling errors.
How to approve / reject data rows for a Benchmark-based project
How to approve / reject data rows for a Consensus-based project
With large complex projects, it can be hard to manage and keep track of a Data Row’s lifecycle.
A multi-step review workflow means that a Data Row can travel between different stages of the workflow. You can quickly do spot checks with the audit log to better understand how a Data Row has ended up in a particular stage of your workflow. This helps ensure that no Data Row has skipped an important step of the review process and can give you a clear picture of how a Data Row has ended up from “to Label” to “Done”.
This ability to track the journey of a Data Row through a review cycle can help teams make more informed decisions on how to make edits to their review workflow for future iterations.
What replaces thumbs up / thumbs down voting?
For all new projects, thumbs up / thumbs down voting will be replaced by the approve / reject workflow. In review tasks, the following actions can be taken:
Approve: Reviewer will approve a data row
Reject: Reviewer will reject a data row
Approve: Reviewer will choose a winner label to approve the data row
Reject: Reviewer will reject a data row, essentially rejecting all labels on that data row
You can also use the 'Move to step' functionality for ad-hoc review.
How do you keep track of the journey of a Data Row?
Workflows allows you to see the journey of your Data Rows as they move from 'To label' to 'Done' through your customized review steps.
With the audit log, admins can view the entire set of events that happened on a Data Row after it was labeled, allowing you to understand the complete journey of each Data Row. This can be especially helpful for when teams need to investigate the review history of a particular data row or a set of Data Rows.
How do you know when a data row is ready to be used for model training?
Data Rows that appear in the 'Done' stage of your workflow can be used for model training. The 'Done' stage signifies that the Data Row has passed through all necessary review steps, giving you a real-time, high-level view of how much of your data is ready to be used for model training.
How do you ensure that you’re only reviewing the most important assets?
This will likely depend on your particular workflow, but teams can use filters in the Data Rows tab to surface specific data rows of interest for review.
Multi-step review workflows can give teams the flexibility to review Data Rows at a specific step of the review process. Rather than having to review or sort through all of your data rows, the Data Row tab gives teams a holistic look into all the review steps within a project’s workflow
You can read more about this in the Data Rows tab guide.
Can I use the Rework task instead of Delete-and-requeue?
'Delete and requeue' will still be supported for a little while longer. Eventually, the 'Rework' step of your Workflow will replace Delete-and-reqeueue. Labelbox is currently working on parity between the asynchronous functionality in Rework tasks and 'Delete and requeue'.