I would have wholeheartedly agreed with you up until the point where I hit the more complex grammar. I haven't had time to really study Japanese for a couple of years, but the early grammar is so simple that I still remember almost all of the grammar I learned. Later grammar kicked my ass, though, as did trying to learn the pronunciations of each kanji.
One of my Japanese friends described the difficulty levels of Japanese and English as being sort of like this. Japanese starts out extremely easy, but you hit a point where it increases dramatically, while English starts out incredibly difficult but gets easier once you get the ridiculous crap out of the way.
Something else I just thought of: Japanese could be considered harder to learn for people that learn languages by reading anything and everything they're capable of (as I think I do). Kanji makes it much more difficult to just pick up a book and try to read and gain as much as possible out of it.