Queue Customization

Queue Prioritization

When setting up a project or configuring an existing project, there are certain types of workflows (real-time, active learning, etc.) where queue prioritization and customization becomes necessary. Labelbox allows you to programmatically prioritize assets with a score and set a custom consensus level in order to fulfill complex workflows necessary in many use cases.

In order to set prioritization for data rows, the necessary Labeling Parameter Overrides need to be set on a project (all required fields):

  • Data Row ID: the ID of the data row you wish to set priority for
  • Priority: a score (number) between 1 and 9007199254740992
  • Number of Labels: the total number of times a data row should be labeled
Queue Details
Prioritization

When prioritization is set for one or more data rows, the queue will be “split” between all of the overridden data rows and the rest of the queue, which will be sorted by the data row creation time with the first data row in that list getting a prioritization score of n + 1, where n is the maximum of the prioritized data rows.

For example, if you have data rows A, B, C, D, E (in the order they were created by, with the oldest first), and set prioritization of 1 for E, and 2 for C, the queue will be structured as follows (Data Row ID: Priority):

E: 1 C: 2 A: 3 B: 4 D: 5

Another example of prioritization, with the same data rows but setting 1 for E, and 5 for D, the queue will be structured as follows:

E: 1 D: 5 A: 6 B: 7 C: 8

When setting the queue prioritization of an active queue, the priority for a given data row will be updated, but in accordance with the current prioritization of the queue. An example being, given the following queue:

A: 1 B: 2 C: 3 D: 4 E: 5

If you update the priority of D to 1, the queue will be structured as follows:

A: 1 D: 1 B: 2 C: 3 E: 5

When setting queue prioritization on an active queue, and without the queue rebuilding, the given data rows will have their priority updated, and if there’s a collision (as in above, where two data rows have a priority of 1), they will be ordered lexicographically.

Number of Labels

For workflows where you want a data row to be labeled a different number of times than the rest of the project, you can set a custom number of labels to be created for each data row via GraphQL as well.

For example, with a queue of A, B, C, D, E, with a consensus of 100% data rows labeled 1 time, if you were to give D a priority of 1 and want it labeled 3 times, the queue would be structured as follows (Data Row ID: Priority / Number of Labels):

D: 13 A: 21 B: 31 C: 41 E: 51

Setting Queue Prioritization

The following GraphQL mutation will set up prioritization for two different data rows.

Success will return true if the request is properly received and processed, otherwise it will return an error.

NoteIf you submit a data row or rows that do not exist, but the request is still properly received and processed, it will still return true.
Make sure to use a DataRow ID and not your DataRow's external ID
mutation SetLabelingParameterOverrides {
project(where: { id: "<PROJECT_ID>" }) {
setLabelingParameterOverrides(data: [
{
dataRow: { id: "<DATA_ROW_ID_ONE>" },
priority: 1,
numLabels: 2
},
{
dataRow: { id: "<DATA_ROW_ID_TWO>" },
priority: 2,
numLabels: 3
}
]) {
success
}
}
}

Unsetting Queue Prioritization

The following GraphQL mutation will unset the prioritization for the given data rows, and will result in rebuilding the labeling queue.

mutation UnsetLabelingParameterOverrides {
project(where: { id: "<PROJECT_ID>" }) {
unsetLabelingParameterOverrides(data: [
{ dataRowId: "<DATA_ROW_ID_ONE>" },
{ dataRowId: "<DATA_ROW_ID_TWO>" }]) {
success
}
}
}

Fetching Labeling Parameter Overrides

You can use this query to read the existing labeling parameter overrides you created.

query PullLabelingParameters{
project(where:{id:"<PROJECT_ID>"}){
labelingParameterOverrides{
id
priority
}
}
}


How did we do?