X-Transformer: A Green Self-attention Based Machine Translation Model

With the exponential growth of data and the increasing demand for computational resources, it is crucial to develop energy-efficient and environmentally friendly deep learning models in AI research. This paper proposes a novel machine translation model designed to address the growing concern on energy consumption. The proposed model named X-Transformer, which refined from the state-of-the-art Transformer model in three aspects. First, the model parameter of the encoder is compressed. Second, the encoder structure is modified by adopting two layers of the self-attention mechanism consecutively and reducing the point-wise feed forward layer to help the model understand the semantic structure of sentences precisely. Third, we streamline the decoder model size, while maintaining the accuracy. Through experiments, we demonstrate the effectiveness of the green X-Transformer in achieving significant training time saving and performance upgrading. The X-Transformer reaches the state-of-the-art result of 46.63 and 55.63 points in the BiLingual Evaluation Understudy (BLEU) metric of the World Machine Translation (WMT), from 2014, using the English–German and English–French translation corpora, thus outperforming the Transformer model with 19 and 18 BLEU points, respectively. The heat maps of the X-Transformer reach token-level precision (i.e., token-to-token attention), while the Transformer model remains at the sentence level (i.e., token-to-sentence attention). In addition, the X-Transformer demonstrates significantly shorter training time, requiring only one-third of that of the original transformer.