There is also no reason to beleive that a higher power doesn't exist. You belief stands on nothing as well.
That's not how logic works. You don't just believe in things because there's no evidence that it doesn't exist. There's not evidence that Big Foot doesn't exist either, but that doesn't mean a person is justified in believing it exists. Now, that's not to say that its impossible for there to be a Big Foot, but a person would be foolish to assume that it does without good evidence.
For ANY claim, logical people disbelieve it until it's shown that the claim is likely to be true. That's not saying they know or believe it to be a false claim, but they don't accept it as true. I hope you can see the difference between those two positions. For some reason it seemed to elude you when discussing atheism.
And alot of this depends on the particular claim. If I say I have a bag of doritos in my pantry... you might be inclined to take my word for it, as doritos are popular and it's something people tend to put in a pantry. It's not that far fetched of a claim. But when talking about a supernatural entity that doesn't manifest in the real world, except when it created the universe billions of years ago ... that's a little more extraordinary, and is going to require a bit more evidence for someone who's skeptically minded to accept as true or likely to be true.
*edit for your edit*
Plus since when doid natural occurrences become an end all to the being existing?
Again as I say the deist beleif and a beleif that the being interacts with the universe sometimes still stands.
What else IS existing except a manifestation in the natural world? If this entity interacts with the universe today, then those interactions should be in the realm of scientific investigation, no?