Skip to content
DSML.KZ
Publications

Talk · Scientific School Dynamics of Complex Networks and their Applications · September 13, 2021

Long-context Transformers: A survey

Atabay Ziyaden, Amir Yelenov, Alexandr Pak

Amir Yelenov
Open via DOI

Abstract

Current Transformer architecture falls behind in terms of the computational complexity of its attention mechanism, namely being O(N2). That problem leads to this model being unable to efficiently work with long sequences. There were a lot of attempts which tries to deal with this problem. In this paper, we considered the most up-to-date methods and how these methods compare to each other in different tasks. We address their limitations and how well they perform, and which ones are most versatile and could be used as a base for future works.

Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applicationslong sequence modelingefficient transformerstransformer models

More by Amir Yelenov