Ranking the Rankers: Conclusion

    Finding the Best NHL Draft Rankings

    “Ranking the Rankers” is a 7-piece series that quantifies past NHL draft rankings from various prognosticators based on player performance to date.

    Part 7: Conclusion

    Your favourite hockey team just drafted a teenager. He, presumably, is good at hockey. How can you be sure? Because this set of draft rankings from [insert writer / scout / website here] says so.

    This series attempts to determine which set of draft rankings is best at predicting future NHL success (and which one you should hope your team follows on draft day).

    The final piece in the series compares the performance of the actual draft results and the final rankings of Bob McKenzie, Corey Pronman, ISS Hockey, and Craig Button.

    Methodology Review

    Check out Part 1 for a detailed summary of:

    • How player performance is measured (career Point Shares/60 minutes of TOI)
    • How draft rankings are gathered (first round only for 2010-2013)
    • How scout performance is measured (absolute difference between draft ranking and that player’s career PS/60 rank among their draft class peers)
    • Limitations (Point Shares and TOI sample size)

    Bob McKenzie’s lists are based on the consensus opinion of a group of scouts that he trusts most.

    Draft rankings from 2010, 2011, 2012, and 2013 are used for all sources except Craig Button. I could not find “Craig’s List” from 2010 despite an epic Google search quest.

    How to Interpret the Following Charts

    The smaller the total absolute value for each year, the better that set of draft rankings performed. By comparing each prognosticator’s average performance from 2010-2013, we get an idea of whose NHL draft rankings are best at predicting future NHL success.

    Yearly Scout Performance

    Hover over the legend labels to highlight each source.

    ISS Hockey wins the 2010 NHL draft with the lowest “Total Average Value of 1st Round Draft Misses” score. The Waterloo-based company hit on the draft class-leading Vladimir Tarasenko earliest (at 4), avoided a common Jack Campbell miss, and uniquely hit on late-firsts Brock Nelson and Jason Zucker. A great year, especially compared to their analyst competition (read: not team scouts and GMs).

    Craig Button wins the 2011 draft by a landslide. Modest hits are littered throughout the entirety of the round, with only one or two notable misses. Button was highest on Nikita Kucherov (one of two sources to have him in the first round) at 17, who leads the 2011 draft crop in career Point Shares/60 minutes. John Gibson was a unique hit – nobody else had the goalie in their top-30. He also correctly avoided Duncan Siemens and Tyler Biggs, both big, common misses. A superb year for the sometimes controversial analyst.

    Bob McKenzie wins the 2012 draft. No big common misses are avoided. Though not uncommon hits, McKenzie placed Filip Forsberg, Alex Galchenyuk, Matt Dumba, and Olli Maatta within 3 spots of their performance rank. Multiple other single-digit hits are present. Steady.

    Corey Pronman wins the 2013 draft by a slight margin. The aforementioned TOI sample size limitations put an asterisk on this win for now, however. Andre Burakovsky looks like a nice hit. Pronman was highest on the Capitals forward at 13th overall.

    Team scouts and general managers (proxied by the actual NHL draft results) failed to win any individual year in terms of draft miss scores. Interesting.

    Draft miss scores decrease as we progress to more recent draft years. I suspect this occurs because later-round picks tend to get their chance to establish themselves later than first-round picks. Top-30 talents that don’t live up to the hype then drop in the PS/60 rankings as they are outperformed by the draft steals.

    Average Scout Performance

    Hover over the legend labels to highlight each source.

    Looking at the three-year sample size (I couldn’t find Button’s 2010 rankings), Craig Button and Bob McKenzie come out on top. The 2010-2013 sample winners are Bob McKenzie and ISS Hockey. Depending on how Button would have fared in 2010, there could be one, two, or three overall winners.

    Outperforming the actual NHL draft order appears common. Team scouts and general managers often fell into the trap of picking non-scoring, shutdown-type defensemen early.

    Bob McKenzie is good. This is not surprising. You would think his results would be similar to the team scouts and GMs because they are based on NHL scout opinions, but nope. He’s better. He must draw from a smart group of scouts.

    Craig Button is good, which surprised me. Just as good as McKenzie, maybe even better. These results suggest that maybe he doesn’t deserve the flak he gets for his “controversial” picks. This year, for example, Button was one of the first on the Logan Brown in the top-10 bandwagon (he went 11th overall to the Senators). That seemed like a hot take for page views kind of thing a few months ago to some. I’m guilty and a big portion of my Twitter follows are as well. We should reconsider our stances.

    ISS Hockey performing well makes sense. They are a huge, profit-based organization with five head scouts and multiple regional scouts. They get paid to do well, and they do.

    This whole series was started on the hypothesis that Corey Pronman could identify talent in today’s NHL better than most scouts. I subscribed to ESPN Insider prior to the 2014 NHL Draft, where Pronman was high on the ultra-skilled William Nylander and Nikolaj Ehlers, both successful so far. He also seemed to be in the “don’t draft goalies and physicality-reliant defensemen (e.g. Griffin Reinhart) early” camp.

    Though Pronman didn’t perform well in this series, my hypothesis still may be true in 2016. He had some bad luck with Alexander Khokhlachev, who he ranked high and is still stuck in the AHL despite stellar stats. 2010 was also the first draft he covered in his current role. 2014 was the first with a substantial scouting budget.

    I expect his 2014 and 2015 results to be much better. Mathew Barzal, Travis Konecny, Kyle Connor, and Daniel Sprong all look terrific so far and Pronman was much higher on those players than most of the field. The apparent contradiction of his results and my hypothesis may just be bad timing plus a natural progression of gaining experience.