awesome-graph-transformer

This repository contains a list of papers on the Graph Transformers; we categorize them based on their detailed techniques.

We will try to make this list updated. If you found any error or any missed paper, please don't hesitate to open an issue or pull request.

Structural Encoding / Postional Encoding for Graph Transformers

Spectral Positional Encoding

1. Rethinking Graph Transformers with Spectral Attention. NeurIPS 2021. [paper]
2. A Generalization of Transformer Networks to Graphs. AAAI workshop 2021. [paper]
3. Transformers Meet Directed Graphs. ICML 2023. [paper]

Other Structure-aware Encoding

1. Do Transformers Really Perform Bad for Graph Representation? NeurIPS 2021. [paper]
2. Graph Neural Networks with Learnable Structural and Positional Representations. ICLR 2022. [paper]
3. GRPE: Relative Positional Encoding for Graph Transformer. ICLR 2022 Workshop MLDD [paper]
4. Global Self-Attention as a Replacement for Graph Convolution. KDD 2022. [paper]
5. Pure Transformers are Powerful Graph Learners. NeurIPS 2022. [paper]
6. Are More Layers Beneficial to Graph Transformers? ICLR 2023. [paper]
7. Graph Inductive Biases in Transformers without Message Passing. ICML 2023. [paper]
8. Rethinking Structural Encodings: Adaptive Graph Transformer for Node Classification Task. WWW 2023. [paper]
9. HINormer: Representation Learning On Heterogeneous Information Networks with Graph Transformer. WWW 2023. [paper]
10. Graph Propagation Transformer for Graph Representation Learning. IJCAI 2023. [paper]
11. LGI-GT: Graph Transformers with Local and Global Operators Interleaving. IJCAI 2023. [paper]
12. On Structural Expressive Power of Graph Transformers. KDD 2023. [paper]
13. LPFormer: An Adaptive Graph Transformer for Link Prediction. Arxiv 2024. [paper]
14. Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers. Arxiv 2024. [paper]

Graph Neural Networks as Structural Encoder

1. GraphiT: Encoding Graph Structure in Transformers. arXiv 2021. [paper]
2. Representing Long-Range Context for Graph Neural Networks with Global Attention. NeurIPS 2021. [paper]
3. Structure-Aware Transformer for Graph Representation Learning. ICML 2022. [paper]
4. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. [paper]

Scalability of Graph Transformers (Graph Transformers on Large Graphs)

Transformers with Sampling

1. A Self-Attention Network based Node Embedding Model. ECML-PKDD 2020. [paper]
2. Heterogeneous Graph Transformer. WWW 2020. [paper]
3. Gophormer: Ego-Graph Transformer for Node Classification. arXiv 2021. [paper]
4. Coarformer: Transformer for large graph via graph coarsening. Openreview 2021. [paper]
5. Hierarchical Graph Transformer with Adaptive Node Sampling. NeurIPS 2022. [paper]
6. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs. ICLR 2023. [paper]
7. Hierarchical Transformer for Scalable Graph Learning. IJCAI 2023. [paper]
8. LPFormer: An Adaptive Graph Transformer for Link Prediction. Arxiv 2024. [paper]

Transformers with Adapted Attention

1. From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers. ICML 2022. [paper]
2. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. [paper]
3. Deformable Graph Transformer. arXiv 2022. [paper]
4. NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification. NeurIPS 2022. [paper]
5. DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion. ICLR 2023. [paper]
6. GOAT: A Global Transformer on Large-scale Graphs. ICML 2023. [paper]
7. EXPHORMER: Sparse Transformers for Graphs. ICML 2023. [paper]
8. KDLGT: A Linear Graph Transformer Framework via Kernel Decomposition Approach. IJCAI 2023. [paper]
9. Gapformer: Graph Transformer with Graph Pooling for Node Classification. IJCAI 2023. [paper]
10. Polynormer: Polynomial-Expressive Graph Transformer in Linear Time. ICLR 2024. [paper]

Applications of Graph Transformers (Molecules, Texts, Knowledge Graphs, Recommendation, Medical, Traffic, etc)

1. Modeling Graph Structure in Transformer for Better AMR-to-Text Generation. EMNLP 2019. [paper]
2. Heterogeneous Graph Transformer for Graph-to-Sequence Learning. ACL 2020. [paper]
3. Molecule Attention Transformer. arXiv 2020. [paper]
4. Interpretable Rumor Detection in Microblogs by Attending to User Interactions. AAAI 2020. [paper]
5. Graph Transformer for Graph-to-Sequence Learning. AAAI 2020. [paper]
6. Self-Supervised Graph Transformer on Large-Scale Molecular Data. NeurIPS 2020. [paper]
7. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS 2020. [paper]
8. Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs. TextGraphs 2021. [paper]
9. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. NeurIPS 2021. [paper]
10. Systematic Generalization with Edge Transformers. NeurIPS 2021. [paper]
11. Mesh Graphormer. ICCV 2021. [paper]
12. Relative Molecule Self-Attention Transformer. arXiv 2021. [paper]
13. Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification. IJCAI 2021. [paper]
14. A graph-transformer for whole slide image classification. TMI 2023. [paper]
15. Neighbour Interaction based Click-Through Rate Prediction via Graph-masked Transformer. arXiv 2022. [paper]
16. Equivariant Transformers for Neural Network based Molecular Potentials. ICLR 2022. [paper]
17. Periodic Graph Transformers for Crystal Material Property Prediction. NeurIPS 2022. [pper]
18. Brain Network Transformer. NeurIPS 2022. [paper]
19. Multi-Relational Graph Transformer for Automatic Short Answer Grading. NAACL-HLT 2022. [paper]
20. Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries. KDD 2022. [paper]
21. Relphormer: Relational Graph Transformer for Knowledge Graph Representations. arXiv 2022. [paper]
22. CoAtGIN: Marrying Convolution and Attention for Graph-based Molecule Property Prediction. BIBM 2022. [paper]
23. GRIP: Graph Representation of Immune Repertoire Using Graph Neural Network and Transformer. AAAI 2023. [paper]
24. MulGT: Multi-Task Graph-Transformer with Task-Aware Knowledge Injection and Domain Knowledge-Driven Pooling for Whole Slide Image Analysis. AAAI 2023. [paper]
25. Inductive Graph Transformer for Delivery Time Estimation. WSDM 2023. [paper]
26. LightGT: A Light Graph Transformer for Multimedia Recommendation. SIGIR 2023. [paper]
27. Graph Transformer for Recommendation. SIGIR 2023. [paper]
28. DiGress: Discrete Denoising diffusion for graph generation. ICLR 2023. [paper]
29. Relational Attention: Generalizing Transformers for Graph-Structured Tasks. ICLR 2023. [paper]
30. Heterformer: A Transformer Architecture for Node Representation Learning on Heterogeneous Text-Rich Networks. arXiv 2022. [paper]
31. Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks. ICLR 2023. [paper]
32. Single-Cell Multimodal Prediction via Transformers. CIKM 2023. [paper]
33. Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation. arXiv 2023. [paper]
34. A Retrieve-and-Read Framework for Knowledge Graph Link Prediction. CIKM 2023. [paper]
35. GTMGC: Using Graph Transformer to Predict Molecule’s Ground-State Conformation. ICLR 2024. [paper]
36. Graph Transformers on EHRs: Better Representation Improves Downstream Performance. ICLR 2024. [paper]

Pre-training with Graph Transformers

1. Self-supervised graph transformer on large-scale molecular data. NeurIPS 2020. [paper]
2. Graph-Bert: Only Attention is Needed for Learning Graph Representations. arXiv 2020. [paper]
3. Pre-training Graph Transformer with Multimodal Side Information for Recommendation. MM 2021. [paper]
4. Graph Masked Autoencoders with Transformers. arXiv 2022. [paper]

Survey

1. Transformer for Graphs: An Overview from Architecture Perspective. arXiv 2022. [paper]
2. A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective. arXiv 2022. [paper]
3. Attending to Graph Transformers. arXiv 2022. [paper]

Neural Architecture Search (NAS) for Graph Transformers

1. AutoGT: Automated Graph Transformer Architecture Search. ICLR 2023. [paper]

Benchmarks and Analyses

1. Transformers Generalize DeepSets and Can be Extended to Graphs & Hypergraphs. NeurIPS 2021. [paper]
2. Universal Graph Transformer Self-Attention Networks. WWW 2022. [paper]
3. Long Range Graph Benchmark. NeurIPS 2022. [paper]
4. Rethinking the Expressive Power of GNNs via Graph Biconnectivity. ICLR 2023. [paper]
5. On the Connection Between MPNN and Graph Transformer. ICML 2023. [paper]
6. A Generalization of ViT/MLP-Mixer to Graphs. ICML 2023. [paper]
7. Demystifying Oversmoothing in Attention-Based Graph Neural Networks. NeurIPS 2023. [paper]
8. Transformers vs. Message Passing GNNs: Distinguished in Uniform. ICLR 2024. [paper]