No, Spanish is not that difficult to learn. Like Moura above said, it's considered by most native English people to be an "easy" language to learn, but then most English people are monolingual and have no real concept of what is required to truly master a second language. As with any foreign language, you have to acquire a whole new grammar and vocabulary and true mastery will take years, regardless of natural ability.
This said, you've clearly learned English to a high standard, and English is far 'messier' than Spanish. Spanish has phonetic pronounciation, which means you can pronounce any word purely from how it is written, and a lot of words will be similar to their Portuguese equivalents. The hardest part of Spanish for me is the little suffixes that add shade of meaning to a noun, e.g. -ito, -azo, though you probably have these in Portuguese too.
My advice would be not to worry too much about what other people think, and get stuck in. You may want to stick to learning Peninsular Spanish though, as Latin American Spanish uses different nouns and grammar in some cases, and this can be confusing for a beginner.