Cool! I wouldn't consider Japanese difficult just because of the Kanji (which is the writing system containing about 50,000 unique characters, of which only 2,000 are commonly used). The grammar is fairly simple and the other two syllabaries are relatively easy to memorize. It takes children growing up in Japan about 6 years (starting at age 6) to learn around 1,000 Kanji characters so it's expected to take a while to learn regardless. Speaking the language is MUCH easier, though, from what I know.
I would have wholeheartedly agreed with you up until the point where I hit the more complex grammar. I haven't had time to really study Japanese for a couple of years, but the early grammar is so simple that I still remember almost all of the grammar I learned. Later grammar kicked my ass, though, as did trying to learn the pronunciations of each kanji.
One of my Japanese friends described the difficulty levels of Japanese and English as being sort of like this. Japanese starts out extremely easy, but you hit a point where it increases dramatically, while English starts out incredibly difficult but gets easier once you get the ridiculous crap out of the way.
Something else I just thought of: Japanese could be considered harder to learn for people that learn languages by reading anything and everything they're capable of (as I think I do). Kanji makes it much more difficult to just pick up a book and try to read and gain as much as possible out of it.
Speaking and listening is actually very easy, remembering all the kanji is what really makes the language hard. Also their onomatopoeia's aren't very intuitive.