I've learned Mandarin and Spanish up to intermediate, and I've learned bits and pieces of Korean, Thai, and Bulgarian... And yeah, language learning difficulty is very real. I was shocked the first time I was in Mexico that I spoke better Spanish after 6 weeks than I did Mandarin after a year...
That being said, what I find is the biggest hurdle to learning a language is immersive context. If you're in a situation where you're exposed to "comprehensible input" in a consistent manner, and you're motivated to learn and use your new skills, then you will learn. If, however, the place you're at is accommodating to non-speakers, the tendency to rely on English will win out.
As an example, I lived a few months in a non-touristy part of Thailand (Issan) where I really needed Thai just to order at the restaurant. So I learned the (really hard) alphabet and (equally hard) language up to a functional level. Then I moved to Chiang Mai, which has plenty of tourists and thus plenty of Thais who speak English, and I lost most of it. It was embarrassing.
Some people can get by with discipline instead of being forced to speak their target language. I spent a month in Spain and forced myself to communicate only in Spanish, even if the people I met knew English or French (the two other languages I knew better back then). I even deliberately chose touristic information in Spanish instead of the two other, including audio guides and stuff. The funny thing is, I have no idea how easier it would have been with English and/or French, since I didn't even ask if people spoke other languages than Spanish!
But I agree that people without discipline should jump in the water and go to a place they simply can't rely on their native language.