The one year I tried Photoshop, I did not find it difficult to learn, per se, I found it difficult to manage all the choices each tool gave me. I found I was never done with fixing my photo because there was always another way I could tweak. The next upgrade, I went back to PSE.
Personally, I think the current versions of PSE are so great, I don't feel like I'm missing out by going back to PSE.
My advice? Don't be afraid of giving Photoshop a try, though. Once you know PSE, the transition won't be that difficult.