In the advanced stats on bbref Phillips graded out as an above average defender last year. I think you underrate how good he is on ball - he's lightning quick with great feet. I'd bet he graded out well on KenPom too defensively. Not that I place much stock in those generally, but there is some evidence other than the eye test.
Van Leer was below average.
http://www.sports-reference.com/cbb/players/terrence-phillips-1.html