Benchmark experiments for knowledge distillation

These docs are outdated! Please check out https://docs.titanml.co for the latest information on the TitanML platform. If there's anything that's not covered there, please contact us on our discord.

On this page you'll find a few examples of knowledge distillation experiments you can run with public HuggingFace models for different use-cases. For any of these experiments, you can substitute the model, dataset or both with a path to a suitable local folder if you want to try using your own models/datasets.

If you want to try running each experiment with a different model, we have a small selection of sample models for each task here.

Question Answering on SQuAD v2

iris distil \
	--model TitanML/Electra-Large-SQUADV2 \
	--dataset squad_v2 \
	--task question_answering \
	--name my_test_squadv2 \
	--has-negative
	

Note that since this experiment uses SQuAD v2, using the flag --has_negative is not necessary. However, any other dataset containing questions which are not answerable from context must be passed to iris distil with the flag.

Remember you can always use the abbreviated iris distil arguments as listed here; this goes for any task, and applies to both local and remote models/datasets. E.g.

iris distil -m TitanML/Electra-Large-SQUADV2 -d squad_v2 -t question_answering -hn -n my_test_squad

Sequence classification with GLUE MRPC

This is the same as the example we used here.

iris distil \
	--model TitanML/Electra-Large-MRPC \
	--dataset glue \
	--task sequence_classification \
	--subset mrpc \
	-tf sentence1 \
	-tf sentence2 \
	-nl 2 \
	--name my_test_mrpc
	

Remember you can skip the subset argument if you're not using a dataset (like GLUE) with subsets!

Token Classification with conll2003

conll2003 has 9 token labels as shown below; pass each one to iris distil in the form {index}:{label}.

iris distil \
        --model TitanML/Electra-Large-CONLL2003 \
        --dataset conll2003 \
        --subset conll2003 \
        --task token_classification \
        -ln 0:O \
        -ln 1:B-PER -ln 2:I-PER \
        -ln 3:B-ORG -ln 4:I-ORG \
        -ln 5:B-LOC -ln 6:I-LOC \
        -ln 7:B-MISC -ln 8:I-MISC \
        --name my_test_conll