Mutual Fund Rating – Performance evaluation or statistical optimism?

The skills of mutual fund managers are evaluated by comparing their portfolio risks and returns. This risk reward relation is quantified using Sharpe ratio that compares average returns for the investment period with the standard deviation (risk) of these returns. In Pakistan, mutual fund industry has grown substantially in the last decade and consequently warranted an independent opinion on the performance of fund managers to facilitate investors. The local rating agencies provide a mutual fund rating (also called star ratings) employing quantitative comparison of returns based on net asset value and their standard deviations. Sharpe ratio is no doubt an important benchmark for assessing performance but there are some fundamental limitations that cannot be ignored especially when it is used for critical investment decisions.
The rating agencies claim this quantitative measure to be free of estimation biases. Their methodologies describe the alleged superiority of this ratio but never disclose the limitations that will make use of this methodology questionable. It is worth noting that Nobel Prize winning economist William Sharpe in 1994 acknowledged the estimation issues in use of Sharpe ratio that he developed some thirty years earlier. Therefore, it is important for investors, fund managers and regulators to understand the inherent deficiencies in this methodology to evaluate the opinions that are based on application of this technique.
The basic problem with Sharpe ratio lies in the use of standard deviation as measure of risk. If the risk parameter is not a true representative of risk, the resulting ratio could give us misleading conclusion about the performance. Even if we ignore the statistical details, standard deviation in practice becomes controversial for at least two reasons. First, when we evaluate Sharpe ratio, apparently it will penalise managers for taking high risks with low returns. However, the anatomy of this ratio presents a different picture. Standard deviation represents the distance of each return from the mean value for the investment period. This implies that positive deviations from mean are treated similar to negative deviation. The positive deviations are penalised on the perception that there was an equal possibility of these becoming negative thus increasing the risk. Isn’t this ridiculous that while fund managers are enjoying bonuses on large positive deviations, their performance is being penalised by the Sharpe ratio? Similarly, in dynamic investment strategies where portfolios are continuously rebalanced, standard deviation cannot capture time varying volatility of returns.
Second, unlike returns, standard deviation is an estimate and not directly observable and calculated from a time series of returns using ending net asset value. Our local rating firms propose use of ending monthly net asset values for a year to calculate returns and standard deviation. This estimate is not a measure of risk for the fund manager unless it is representative of all possible time series of returns. Therefore, standard deviation can be a meaningful measure if underlying returns are stationary and they follow a normal distribution. The stationarity refers to the property where returns have constant skewness and kurtosis. For fund managers, this implies that they have consistent investment style throughout their investment period. The fund managers who demonstrate their skills by exploiting anomalies to yield high returns with little investment will make underlying returns non stationary making Sharpe style variation inappropriate. Therefore, given the dynamic nature of markets and increase in skills, investment style cannot remain constant and we cannot expect statistical characteristics to remain stable challenging the use of Sharpe ratio for performance evaluation.
The normal distribution of returns is the only situation where Sharpe style performance valuation will work. A simple violation to normal distribution will be an investment strategy that yields small returns for successive periods with occasional but large losses. The Sharpe ratio will overstate the performance in all periods before the loss has realised. Hence, normal distribution violations will be violated if volatility is varying over time. The last decade has witnessed extreme turbulence in asset returns with volatility clustering and speculative valuations. In such investment environment it is redundant to believe that normal distribution will prevail and standard deviation will reflect the risk for mutual funds. This discussion provides insight into the basic limitations of using one single measure for assessing the performance of funds managers. These limitations are relevant for practitioners as none of it is theoretical, rather these are the issues related to applying a theoretical concept to practice. The severity is magnified when rating agencies base their opinions on a technique which is fundamentally flawed in practice. The possible solution could be to develop appropriate statistical techniques that correlate with the returns distributions in a turbulent economy rather than assuming a heavenly Gaussian distribution. The central bank should be diligent about such flaws otherwise performance evaluation ratings will be no more than statistical optimism.

The author holds a PhD in Quantitative Finance from Paris Dauphine. He is Associate Professor of Finance at Lahore School of Economics and provides consultancy on risk management through Synergistic Financial Advisors.