來源：https://e2eml.school/transformers.html#softmax
中英雙語版，由各類翻譯程序和少量自己理解的意思做中文注釋

相關(guān)文章匯總在文集：Transformers from Scratch（中文注釋）
--------------------------------------------------------------------------------------------------------------------

Congratulations on making it this far. You can stop if you want.

恭喜你走到了這一步。如果你愿意，你可以停下來。

The selective-second-order-with-skips model is a useful way to think about what transformers do, at least in the decoder side.

帶跳過的選擇性二階模型是一種有用的方法，可以思考transformers的作用，至少在解碼器端是這樣。

It captures, to a first approximation, what generative language models like OpenAI's GPT-3 are doing.

它捕捉到了像OpenAI的GPT-3這樣的生成語言模型在做什么。

It doesn't tell the complete story, but it represents the central thrust of it.
它沒有講述完整的故事，但它代表了它的核心。

The next sections cover more of the gap between this intuitive explanation and how transformers are implemented.

下一節(jié)將介紹這一直觀解釋與transformers如何實(shí)現(xiàn)之間的更多差距。

These are largely driven by three practical considerations.
這主要是由三個(gè)實(shí)際考慮因素驅(qū)動(dòng)的。

1、Computers are especially good at matrix multiplications.

1、 計(jì)算機(jī)特別擅長矩陣乘法。

There is an entire industry around building computer hardware specifically for fast matrix multiplications.?

整個(gè)行業(yè)都圍繞著專門用于快速矩陣乘法的計(jì)算機(jī)硬件。

Any computation that can be expressed as a matrix multiplication can be made shockingly efficient.?

任何可以表示為矩陣乘法的計(jì)算都可以變得非常高效。

It's a bullet train.?

這是一列子彈頭火車。

If you can get your baggage into it, it will get you where you want to go real fast.

如果你能把行李放進(jìn)去，它會(huì)讓你很快到達(dá)你想去的地方。

2、Each step needs to be differentiable.?

2、 每個(gè)步驟都需要是可微的。

So far we've just been working with toy examples, and have had the luxury of hand-picking all the transition probabilities and mask values—the model parameters.

到目前為止，我們只是在研究玩具的例子，并且已經(jīng)有了手工挑選所有轉(zhuǎn)換概率和模型參數(shù)的掩碼值的奢侈體驗(yàn)。

In practice, these have to be learned via backpropagation, which depends on each computation step being differentiable.?

在實(shí)踐中，這些必須通過反向傳播來學(xué)習(xí)，這取決于每個(gè)計(jì)算步驟是可微的。

This means that for any small change in a parameter, we can calculate the corresponding change in the model error or loss.

這意味著，對(duì)于參數(shù)的任何微小變化，我們都可以計(jì)算模型誤差或損失的相應(yīng)變化。

3、The gradient needs to be smooth and well conditioned.?

3、 斜率需要平滑且條件良好。

The combination of all the derivatives for all the parameters is the loss gradient.?

所有參數(shù)的所有導(dǎo)數(shù)的組合就是損耗梯度。

In practice, getting backpropagation to behave well requires gradients that are smooth, that is, the slope doesn’t change very quickly as you make small steps in any direction.?

在實(shí)踐中，要使反向傳播表現(xiàn)良好，需要平滑的梯度，也就是說，當(dāng)你在任何方向上邁出小步時(shí)，斜率都不會(huì)很快改變。

They also behave much better when the gradient is well conditioned, that is, it’s not radically larger in one direction than another.?

當(dāng)梯度條件良好時(shí)，它們也表現(xiàn)得更好，也就是說，它在一個(gè)方向上不會(huì)比另一個(gè)方向大得多。

If you picture a loss function as a landscape, The Grand Canyon would be a poorly conditioned one.

如果你把損失函數(shù)想象成一個(gè)景觀，大峽谷將是一個(gè)條件很差的峽谷。

Depending on whether you are traveling along the bottom, or up the side, you will have very different slopes to travel.

根據(jù)您是沿著底部還是向上行駛，您將有非常不同的斜坡要行駛。

By contrast, the rolling hills of the classic Windows screensaver would have a well conditioned gradient.

相比之下，經(jīng)典Windows屏幕保護(hù)程序的起伏山丘會(huì)有一個(gè)良好的漸變。

If the science of architecting neural networks is creating differentiable building blocks, the art of them is stacking the pieces in such a way that the gradient doesn’t change too quickly and is roughly of the same magnitude in every direction.
如果構(gòu)建神經(jīng)網(wǎng)絡(luò)的科學(xué)是創(chuàng)建可區(qū)分的構(gòu)建塊，那么它們的藝術(shù)就是以這樣的方式堆疊這些塊，即梯度不會(huì)變化太快，并且在每個(gè)方向上都大致相同。