Cross-lingual prosody evaluation through transfer learning

Abstract: The importance of automatic prosody assessment has been acknowledged, as it provides relevant information about the speaker, the languages, the pragmatics and paralinguistics of speech. Nevertheless, available data sets are often insufficient for the tasks aimed at, namely when it comes to the usage of DNNs, which require great amounts of data. Transfer learning is a state-of-the-art technique being used for several tasks and proven to be very informative, as training with diverse data sets and testing on distinct ones can assure robustness and cross-lingual analysis. In this case, we investigate whether transfer learning can be applied to L2 learners, either exclusively in learning contexts or in e-health ones too, which is a very challenging task.The main data set we use was built for an intonation imitation task with native speakers of Portuguese, whose assessment relied on a DTW algorithm only. Building upon previous work, a nativeness assessment task of L2 speakers of English, we applied a DNN for feature extraction. We have considered features with no temporal dependency and features with temporal dependency, corresponding to an LSTM layer.

Tags: