up主微信公眾號pythonEducation

http://ucanalytics.com/blogs/credit-scorecards-model-validation-part-6/參考

There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.

– Albert Einstein

A Commentary on Curiosity?

生活只有兩種方式。 一個好像沒有什么是奇跡。 另一個好像一切都是奇跡。

- 艾爾伯特愛因斯坦

好奇心評論

Advanced Analytics Professional: An Unbiased Observer – by Roopam

I think the best way to appreciate and enjoy the trivial is to travel. When I say trivial, it includes doorknobs, posters, letterboxes, graffiti and everything we never bother to turn our heads for in our own city. I experienced the same last week while traveling with my wife across Florence and Tuscany. I think one’s level of awareness and curiosity goes up many-fold while traveling. In Florence, we stayed at a lovely bed-and-breakfast named Fiorenza. The breakfast was good and the people even better. There we met this amicable family from the UK with a year old baby named Owen and his 7-year-old sister Kyra. Owen and Kyra were playing hide and seek while having their breakfast. Kyra hid behind the same chair repeatedly and jumped out to reveal herself to her younger brother. Owen was pleasantly surprised every time during this process. All humans are born curious. However, they lose it as they grow older and get familiar with things. The phenomenon could be the reason why we never turn our heads for the trivial in our own city.

我認為欣賞和享受瑣事的最佳方式是旅行。當我說瑣碎的時候，它包括門把手，海報，信箱，涂鴉以及我們從未在我們自己的城市中轉過頭來做的一切。上周我與妻子一起在佛羅倫薩和托斯卡納旅行時經(jīng)歷了同樣的經(jīng)歷。我認為一個人的意識水平和好奇心在旅行時會增加很多倍。在佛羅倫薩，我們住在一個可愛的住宿加早餐，名為Fiorenza。早餐很好，人們甚至更好。在那里，我們遇到了這個來自英國的友好家庭，一個名叫Owen的嬰兒和他7歲的妹妹Kyra。歐文和凱拉在吃早餐時玩捉迷藏。凱拉反復躲在同一把椅子后面，跳出來向她的弟弟透露自己。歐文在這個過程中每次都感到驚喜。所有人都天生好奇。然而，隨著年齡的增長和熟悉事物，他們會失去它。這種現(xiàn)象可能是我們永遠不會為自己城市中的瑣事而煩惱的原因。

Curiosity and Data Science Career

Being curious and aware requires constant energy and effort. Perhaps, humans have the natural tendency to slip into a low energy state. Nonetheless, this is particularly dangerous for analysts since their job requires finding meaning in something that seems mundane to others. In my opinion, the biggest challenge for analytics is not the sophistication of statistical algorithms and enhancement of computing power, but for its practitioners to stay curious and constantly ask questions. Zen Buddhists try to achieve cosmic awareness by living in the moment. If that is too difficult, I would recommend that treat your job like a wonderful travel destination and be a good tourist – curious and aware.

Ok, so that was a bit of a detour from our original discussion on scorecards. However, there are a couple of reasons for telling you the above: primarily, to tell you why I was late in posting this part of the series. Secondly, I would like us to have a discussion on the importance and challenges of being curious at work and life in general. I already have a few examples in mind i.e. Louis Pasteur and Edward Lorenz but that is for later.

Now, let’s continue with the topic for this part i.e. model evaluation.

好奇心與數(shù)據(jù)科學事業(yè)
充滿好奇和意識需要不斷的精力和努力。也許，人類有自然傾向于陷入低能量狀態(tài)。盡管如此，這對分析師來說尤其危險，因為他們的工作需要在對他人而言看似平凡的事情中找到意義。在我看來，分析的最大挑戰(zhàn)不是統(tǒng)計算法的復雜性和計算能力的提高，而是讓其從業(yè)者保持好奇并不斷提出問題。禪宗佛教徒試圖通過生活在當下來實現(xiàn)宇宙意識。如果這太難了，我建議把你的工作當作一個很棒的旅游目的地，做個好游客 - 好奇又有意識。

好的，所以這與我們對記分卡的原始討論有點迂回。但是，有幾個原因告訴你上面的內容：主要是告訴你為什么我在發(fā)布這個系列的這一部分時遲到了。其次，我希望我們討論一般對工作和生活充滿好奇的重要性和挑戰(zhàn)。我已經(jīng)有一些例子，即路易斯巴斯德和愛德華洛倫茲，但這是為了以后。

現(xiàn)在，讓我們繼續(xù)討論這個部分的主題，即模型評估。

Model Validation & Evaluation

Model Evaluation & Validation: the test of the pudding is in the eating – by Roopam

When I was in high school, I joined a cricket academy during the summer vacations. Cricket is a game quite similar to baseball. I shall use baseball terminology in parenthesises for everyone to understand. The design of the training camp was to train for about a month followed by a full game with kids at same skill-level from another club. There was this tall and lean kid with us in the camp; he was the star bowler (pitcher) throughout during the training sessions. He used to bowl (pitch) some of the best Yorkers (curve balls). We were quite sure he would outperform everyone in the game. We ask him to open the bowling, his first bowl went for a six (home run) followed by several more. Maybe it was a mix match pressure, expectations, and the crowd but his performance was an absolute disaster. Later the coach told us what happened was not unusual and he had seen this several times before. At higher levels, the game is played not on the ground but the space between the ears. Clearly, he was referring to players’ presence of mind and temperament.

當我在高中時，我在暑假期間加入了板球學院。 Cricket是一款與棒球非常相似的游戲。我將在括號中使用棒球術語，讓每個人都能理解。訓練營的設計是訓練大約一個月，然后與來自另一個俱樂部的相同技能水平的孩子進行完整的比賽。在營地里有一個高大瘦弱的孩子和我們在一起;在訓練期間，他一直是明星投手（投手）。他過去常常把一些最好的Yorkers（曲線球）弄成一團糟。我們非常肯定他會在游戲中勝過每個人。我們要求他打開保齡球，他的第一個碗去了六個（本壘打），然后是幾個。也許這是混合比賽壓力，期望和人群，但他的表現(xiàn)是絕對的災難。后來教練告訴我們發(fā)生的事情并不罕見，他以前曾多次見過這件事。在更高的級別，游戲不是在地面上播放，而是在耳朵之間的空間播放。顯然，他指的是球員的思想和氣質。

Sampling Strategy for Model Validation

As the famous saying goes,?the test of the pudding is in the eating.?One could be a star on the training fields but a complete flop in the match situation. The same is true for an analytical model as well. A model, after going through a round of training (Part 5 of the series) goes through a several rounds of testing.

1. Out of sample test:?remember?article 2, where we have divided our sample into the training and the test sample. The first level of testing happens on the holdout or test sample. The test sample needs to perform as well as the training sample. Let us come back to this in the next section when I will discuss the measures for performance and ROC curve.

2. Out of time sample test:?since the model was built on a sample of the portfolio with reasonable vintage (refer to Part 2), the analyst would like to test the performance of a more recent portfolio. The number of bad borrowers (90+ DPD) in this out of time sample will be certainly less but the overall trend of good/bad ratio against scores will still be a good indicator for model performance. Additionally, the analyst could relax the condition for bad loans and consider 30+ DPD as bad. Again, the overall trend should match the scorecard estimations.

3. On field test:?this is where the test of the pudding is; the analyst needs to be completely aware of any credit policy changes that the bank has gone through since the scorecard is developed and more importantly, the impact the changes will have on the scorecard. Always remember not every policy change will influence the scorecard – a good business understanding and a bit of common sense really help here. A regular monitoring and accordingly calibrating the scorecard is a good way to keep it updated.

正如俗名所說，布丁的考驗就在于吃。一個人可能是訓練場上的明星，但在比賽情況下完全失敗了。對于分析模型也是如此。經(jīng)過一輪訓練（系列的第5部分）后，模型經(jīng)過了幾輪測試。

1.train VS test樣品外測試：記住第2條，我們將樣品分成培訓和測試樣品。第一級測試發(fā)生在保持或測試樣本上。測試樣本需要與訓練樣本一樣好。讓我們在下一節(jié)回到這一點，我將討論性能和ROC曲線的措施。

2.OOT超時樣本測試：由于該模型是基于合理年份的投資組合樣本（參見第2部分），因此分析師希望測試最近投資組合的表現(xiàn)。在這段時間樣本中，不良借款人（90+ DPD）的數(shù)量肯定會減少，但是對比分的好/壞比率的整體趨勢仍將是模型表現(xiàn)的良好指標。此外，分析師可以放松不良貸款的條件，并認為30+ DPD是壞的。同樣，整體趨勢應該與記分卡估計相匹配。

3.政策變化對模型影響大

場景測試：這是布丁測試的地方;分析師需要完全了解銀行自開發(fā)記分卡以來所經(jīng)歷的任何信貸政策變化，更重要的是，變更將對記分卡產(chǎn)生的影響。永遠記住不是每個政策變化都會影響記分卡 - 良好的商業(yè)理解和一些常識在這里真的很有幫助。定期監(jiān)控并相應地校準記分卡是保持更新的好方法。

Performance Tests for Model Validation

There are several ways to test the performance of the scorecard such as confusion matrix, KS statistics, Gini and area under ROC curve (AUROC) etc. The KS statistics is widely used metric in scorecards development. However, I personally prefer the AUROC to the others. I must add the Gini is a variant of the AUROC. The reason for my liking of the AUROC could be my formal training in Physics and engineering. I think it is a more holistic measure and lets the analyst visually analyze the model performance. I prefer graph and visual statistics any day to raw numbers.

有幾種方法可以測試記分卡的性能，例如混淆矩陣，KS統(tǒng)計，基尼系數(shù)和ROC曲線下面積（AUROC）等.KS統(tǒng)計量是記分卡開發(fā)中廣泛使用的度量標準。 但是，我個人更喜歡AUROC和其他人。 我必須添加Gini是AUROC的變種。 我喜歡AUROC的原因可能是我在物理和工程方面的正式培訓。 我認為這是一個更全面的衡量標準，讓分析師可以直觀地分析模型的表現(xiàn)。 我更喜歡圖形和視覺統(tǒng)計數(shù)據(jù)，以及原始數(shù)字。

ROC Curve: for Credit Scorecard Model Validation and Evaluation – by Roopam

The adjacent graph shows a ROC. The two axes on the curve are true and false positive rates. As expected, the plot informs about the level of prediction for the model. A perfect model will perfectly segregate good and bad cases. Hence, you will get 100% true positives in the beginning (i.e. absolute lift) as shown with the green curve in the graph. However, like anything in life perfection does not exist. As they say – If it is too good to be true it probably is. On the other extreme is a worthless model, curve marked in red. Anything close to or below the red curve is as good as tossing a coin, then why to bother with the effort to build a model. Finally, a typical scorecard ROC will look like the blue curve. The AUROC for a usual credit-scoring model is within 70 to 85, higher the better. However, for some fraud and insurance models, a slightly above 60 is an acceptable ROC. Again, analysts should be sure about the business benefits from the scorecard before finalizing the ROC. A simple cost-benefit analysis helps significantly before finalizing the model and reporting it to the top management.

相鄰的圖表顯示了ROC。曲線上的兩個軸是真實和誤報率。正如預期的那樣，該圖表通知了該模型的預測水平。一個完美的模型將完美地隔離好的和壞的案件。因此，您將在開始時獲得100％真實的正數(shù)（即絕對提升），如圖中的綠色曲線所示。但是，生活中的任何事物都不存在完美。正如他們所說 - 如果真是太好了，那可能就是這樣。另一個極端是一個毫無價值的模型，曲線標記為紅色。任何靠近或低于紅色曲線的東西都和投擲硬幣一樣好，那么為什么要費心去打造一個模型。最后，典型的記分卡ROC看起來像藍色曲線。通常的信用評分模型的AUROC在70到85之間，越高越好。但是，對于某些欺詐和保險模式，略高于60的是可接受的ROC。同樣，分析師應該在最終確定ROC之前確保記分卡的業(yè)務收益。在最終確定模型并將其報告給最高管理層之前，簡單的成本效益分析可以顯著提供幫助。

Sign-off Note

I hope after reading this, you will pick up your camera and visit that unexplored nook at the corner of the street – and be ready for some wonderful surprises!

References1. Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring – Naeem Siddiqi 2. Credit Scoring for Risk Managers: The Handbook for Lenders – Elizabeth Mays and Niall Lynas

博主網(wǎng)校主頁：http://dwz.date/bwes