本講相關(guān)學(xué)習(xí)資料：

Robert Huben, “How does GPT-3 spend its 175B parameters?”?-?https://aizi.substack.com/p/how-does-gpt-3-spend-its-175b-parameters

Anthropic, “In-context Learning and Induction Heads”?深入探索了大語言模型 in-context learning 能力的來源 -?https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html

最近的 RedPajama 項(xiàng)目中嘗試“復(fù)現(xiàn)”了LLaMA的訓(xùn)練數(shù)據(jù)集 -?https://together.ai/blog/redpajama

Yao Fu, “How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources”?為何要在訓(xùn)練中包括代碼數(shù)據(jù)， GPT 模型家族譜系圖， alignment tax 等內(nèi)容 - https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1

Open Assistant數(shù)據(jù)集?https://huggingface.co/datasets/OpenAssistant/oasst1

Anthropic: Constitutional AI?https://www.anthropic.com/index/claudes-constitution

OPT優(yōu)化的血淚史?https://arxiv.org/pdf/2205.01068.pdf

模型inference優(yōu)化的手段?https://lilianweng.github.io/posts/2023-01-10-inference-optimization/