Transformer 模型是自然語言處理領(lǐng)域中的一種最先進(jìn)的模型，它使用注意力機(jī)制來處理任意長度的輸入文本，并且可以學(xué)習(xí)語言模式和規(guī)律。近年來，隨著計(jì)算能力的不斷提高，Transformer 模型已經(jīng)被廣泛應(yīng)用于機(jī)器翻譯、文本生成、文本分類等任務(wù)中。同時，Transformer 模型也被應(yīng)用于代碼生成任務(wù)中，通過訓(xùn)練模型來生成高質(zhì)量的代碼。

Transformer作為提升NLP效率的黑魔法，在自然語言處理領(lǐng)域發(fā)生了巨大的影響，如當(dāng)下火熱的GPT-3 和 lang-8 等，就是基于Transformer架構(gòu)構(gòu)建的大語言模型。近年來，研究者們在注意力機(jī)制方面進(jìn)行了廣泛的研究。其中，一些研究者提出了基于注意力機(jī)制的新方法，如自適應(yīng)的注意力機(jī)制和全局注意力機(jī)制。這些新方法為 Transformer 模型的改進(jìn)提供了新的思路。同時，為了更好地利用計(jì)算資源，研究者們提出了許多模型壓縮方法，如剪枝、量化和蒸餾等。這些方法可以幫助研究者們更好地利用已有的 Transformer 模型，并提高模型的性能和效率。

Transformer 模型的最新進(jìn)展表明，它已經(jīng)成為自然語言處理領(lǐng)域中最重要的模型之一，并且在許多任務(wù)中取得了最先進(jìn)的性能。

關(guān)于Transformer模型的頂會論文列表如下（由于篇幅關(guān)系，本篇只展現(xiàn)部分頂會論文，可復(fù)制文末鏈接，直達(dá)頂會會議列表，查看所有論文）

1.Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer

2.Towards Long-delayed Sparsity: Learning a Better Transformer through Reward Redistribution

3.HDFormer: High-order Directed Transformer for 3D Human Pose Estimation

4.CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation

5.Learning Attention from Attention: Efficient Self-refinement Transformer for Face Super-resolution

6.FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformer

7.Towards Incremental NER Data Augmentation via Syntactic-aware Insertion Transformer

8.Neighborhood Attention Transformer

9.EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

10.RGB no more: Minimally-decoded JPEG Vision Transformers

11.BiFormer: Vision Transformer with Bi-Level Routing Attention

12.Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

13.DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

14.OneFormer: One Transformer to Rule Universal Image Segmentation

15.Graph Transformer GANs for Graph-Constrained House Generation

16.DeepVecFont-v2: Exploiting Transformers to Synthesize Vector Fonts with Higher Quality

17.Vision Transformers are Parameter-Efficient Audio-Visual Learners

18.In-context Reinforcement Learning with Algorithm Distillation

19.Language Modelling with Pixels

20.A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

21.Relational Attention: Generalizing Transformers for Graph-Structured Tasks

22.Encoding Recurrence into Transformers

23.Specformer: Spectral Graph Neural Networks Meet Transformers

24.MaskViT: Masked Visual Pre-Training for Video Prediction

25.Efficient Attention via Control Variates

26.What Do Self-Supervised Vision Transformers Learn?

27.Are More Layers Beneficial to Graph Transformers?

28.User Retention-oriented Recommendation with Decision Transformer

29.Tracing Knowledge Instead of Patterns: Stable Knowledge Tracing with Diagnostic Transformer

30.MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters

31.Compact Transformer Tracker with Correlative Masked Modeling

32.Learning Progressive Modality-shared Transformers for Effective Visible-Infrared Person Re-identification

33.An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

34.Vision Transformers Are Good Mask Auto-Labelers

35.Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention During Vision Transformer Inference

36.Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

37.Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation

38.Burstormer: Burst Image Restoration and Enhancement Transformer

39.Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers

40.Recurrent Vision Transformers for Object Detection with Event Cameras

41.Q-DETR: An Efficient Low-Bit Quantized Detection Transformer

42.Feature Shrinkage Pyramid for Camouflaged Object Detection with Transformers

43.A Light Touch Approach to Teaching Transformers Multi-view Geometry

44.DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting

45.Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

46.CompletionFormer: Depth Completion with Convolutions and Vision Transformers

47.Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localization

48.Vision Transformer with Super Token Sampling

49.POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery

50.Supervised Masked Knowledge Distillation for Few-Shot Transformers

復(fù)制鏈接，直達(dá)頂會頁面：https://www.aminer.cn/conf

如何使用ChatPaper？

為了讓更多科研人更高效的獲取文獻(xiàn)知識，AMiner基于GLM-130B大模型能力，開發(fā)了Chatpaper，幫助科研人快速提高檢索、閱讀論文效率，獲取最新領(lǐng)域研究動態(tài)，讓科研工作更加游刃有余。

ChatPaper是一款集檢索、閱讀、知識問答于一體的對話式私有知識庫，AMiner希望通過技術(shù)的力量，讓大家更加高效地獲取知識。

ChatPaper：https://www.aminer.cn/chat/g