That's half the reason why there's such a contrast between his on/off numbers, because the Cavs are that weak defensively (they're a poor defensive unit even when he's on court). Even Love has a better on/off court defensive rating than LeBron.
The problem with that is it's only on shots defended (no other strand of defense), where teams are more likely to test the weaker defenders on the Cavs rather than him (Thompson*, Love, Frye, Korver and Shumpert all defend more shots per minute than LeBron - with Kyrie close behind). He's been extremely inconsistent with rotations and stopping players from driving this season, which largely can be put down to leading the league in minutes (behind Lowry). You can't expect him to be a dominant force defensively, all things considered.
Sleeping on LeBron's defense, this season? I think LeBron's the one who's been doing the sleeping on defense. He's certainly had a few purple patches, but his defense hasn't been consistent by any reasonable measure this season.