Marcin Sowański
supervisor: Artur Janicki
Machine translation models in some use cases present translation abilities close to human translation. If translated text is short, as short as one sentence, and the context of previous sentences is not particularly needed, we can expect that the translation done by machine will be as good as translation done by humans. Such assumption (short sentences without context) is met in the domain of virtual assistants. Commands used for communication with virtual assistants (such as Bixby) typically are between 1 and 10 words and context is rarely needed. Machine translation models are therefore used to automatically translate testing and training corpora of virtual assistants. In a typical pipeline the corpus is translated by machine and then evaluated and fixed by a human language expert. Such pipelines, apart from bringing cost reduction, is at the same time wasting a lot of human creative potential because tasks are repetitive and boring. In this work we present ways of measuring whether the process in which a human evaluator works is using their creativity enough. This knowledge can be used to design better processes that can increase human satisfaction from work and at the same time help improve translation at the same time.