Publications

Towards understanding how transformer perform multi-step reasoning with matching operation

Zhiwei Wang, Yunji Wang, Zhongwang Zhang, Zhangchen Zhou, Hui Jin, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Yaoyu Zhang, Zhi-Qin John Xu, 2025.

Submitted to The Forty-second International Conference on Machine Learning (ICML 2025).

We propose a buffer mechanism and found evidence that supports such mechanism being employed by language models during the reasoning process. We propose a method to enhance the model’s reasoning capability, significantly improving data utilization efficiency in logical reasoning datasets.

Download [pdf].

Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers

Brian K Chen, Tianyang Hu, Hui Jin, Hwee Kuan Lee, Kenji Kawaguchi, 2025.

Published in The Forty-First International Conference on Machine Learning (ICML 2024).

We find a way to convert the prompts into the model weights by introducing an extra bias term into the attention module.

Download [pdf].

How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training

Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, Huajun Chen, 2025.

Submitted to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025) .

We analyze how LLMs learn new knowledge through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing.

Download [pdf].

Characterizing the Spectrum of the NTK via a Power Series Expansion

Michael Murray, Hui Jin, Benjamin Bowman, Guido Montufar, 2022.

Published in The Eleventh International Conference on Learning Representations (ICLR 2023).

Download [pdf].

Learning curves for Gaussian process regression with power-law priors and targets

Hui Jin, Pradeep Kr Banerjee, Guido Montúfar, 2021.

Published in The Tenth International Conference on Learning Representations (ICLR 2022).

Workshop version presented at Workshop on Bayesian Deep Learning NeurIPS, 2021.

Download [pdf], [slide].

Implicit bias of gradient descent for mean squared error regression with wide neural networks

Hui Jin, Guido Montúfar, 2020.

Journal of Machine Learning Research JMLR 24(137):1-97, 2023. Repo GitHub.

Download [pdf], [slide].

Noisy Subgraph Isomorphisms on Multiplex Networks

Hui Jin, Xie He, Yanghui Wang, Hao Li, Andrea L Bertozzi, 2019.

Published in 2019 IEEE International Conference on Big Data (Big Data).

Download [pdf].