How to train a task with soft label?

[chawannut157]

I'm quite new to Jiant and trying to implement knowledge distillation method.
https://arxiv.org/abs/1904.09482

Basically given soft labels (One sample): Y = [0.3,0.4,0.3] and logit(X) = [0.2,0.3,0.5].
The costs function can be KL divergence, cross entropy etc.

Can you suggest how to extend the framework to support such task?
First, it seems like I need to adjust this part which should not be too hard

jiant/jiant/proj/main/modeling/taskmodels.py

Lines 173 to 179 in de5437a

	logits = self.head(unpooled=encoder_output.unpooled, spans=batch.spans)
	if compute_loss:
	loss_fct = nn.CrossEntropyLoss()
	loss = loss_fct(logits.view(-1, self.head.num_labels), batch.label_id.view(-1),)
	return LogitsAndLossOutput(logits=logits, loss=loss, other=encoder_output.other)
	else:
	return LogitsOutput(logits=logits, other=encoder_output.other)

Then I need to change batch.label_id.view(-1) to support the format of soft labels.
Can you guide on this one? Is there a similar task that already do this?

[sleepinyourhat]

My impression is that the hard label format is baked in in a few places, so you'll have to do some significant investigation on your own to make this change, and it's out of scope for what we currently support. That said, @zphang might be able say more.