Its used a lot in science though. You wouldn't be able to get through Korean high school without knowing the Hanja. The biggest challenge in Korean is the hierarchy. Its almost like there are different languages for speaking with older people, younger people, your family, your friends etc.
We aren't talking about getting through a Korean high school though. We are talking about language learning difficulty from an English speaker's perspective. And the fact of the matter is, Hanja are rarely used by average people.
I am a Korean American and even though I am not fluent in the langauge, I can confirm that the way you talk to a friend is not the way you would address a parent or an elder. It's used to show and give respect to the elders and I feel that is one of the reasons why Korean can appear to be challenging to people.