Questions on task assignment

[jenahwang]

I have two questions.

Q1: Do the fields for automatic_assignment config specification (e.g. "labels per instance") only hold when the "on" value is True? If this is not turned on basically it will show all the available tasks in the order they are presented in the data?

Q2: In a testing set up I created a data file for 3 instances. I annotated them successfully, and it brought up the post annotation blurb (basically, "you are done" and redirect). Then I added another data file to the mix. It seems that it shows the 3 instances I annotated, then brings up the post annotation blurb even though there were 12 other datapoints to annotate (in the new data file). Weirdly enough if I press on move forward button, it goes to the next data file.

So to see what would happen, I removed from data_file specifications the first data file with 3 instances. When I log in, it shows me the post annotation blurb first. It is not until I press the "move forward" button that it assigns me the 12 instances from the new data file.

So then, I'm not sure how to go about the annotation. Once an annotator finishes the assigned files, and I want to assign more, is there a way to stop from that post annotation blurb showing up first? I hope my description here is making sense.

[Jiaxin-Pei]

Hi @jenahwang,

Q1: Yes. If you turn it off, each annotator will be able to see all the instances in the data files.

Q2: Could you describe your task in more detail? Is it in-house annotation or crowdsourcing? Usually, each annotator is assigned with the same amount of instances so that it's more balanced.

[jenahwang]

Q1: Great. thank you for the clarification.

Q2: it is for a controlled annotation setting. We don't have constraints on how many instances each annotator should have, rather we want to assign certain sets of instances to the annotator, and add more as the need arises.

I think what I am realizing is that possibly my best bet is to simply clear out the annotators annotated files each time when the annotator is done with a batch, and treat any new instances as a subsequent "batches". So I think we are okay with the Q2 I asked.

Actually, is there a way to assign annotators to particular data files? We are doing expert annotation, and we need data assigned matching to the expertise. So ideally when the user logs in, they will only see instances randomly assigned from a particular data file. Right now I think our way of handling this will be to spawn off several instances of the tool, but I am wondering if theres a better way to achieve this that I do not know about.

[Jiaxin-Pei]

@jenahwang Thanks a lot for providing more details on this!

We currently do not support user-based instance allocation so the current workaround is to have multiple potato instances. We have an example project that is in a similar annotation setting.

But we are also super happy to support group-based task assignments! I think this is an important feature. Are you using in-house annotators or crowd workers? Or more specifically, do you know who are the annotators? If this is true, I think we only need a simple mapping from annotator IDs to the files/tasks. Things might be tricker for crowd workers as it will then depend on the setup of the crowdsourcing platform.

Let me know!

[jenahwang]

Somewhere in between. We are sourcing from Upwork. So yes, we do know ahead of time who will be working for us. But because they aren't our "in house" annotators, we do not have an id ahead of time (Upwork doesn't work via IDs) so we were going to let them choose their own.

I will check out the example project you sent me! Thank you

[jenahwang]

After talking with the IT team, I believe my best bet would be to run multiple instances of the tool. Thanks for all the help here! I am happy to consider this issue closed.