"Physical reality isn't a theory; it is what you can directly perceive right now through your senses."
In other words: you have no more direct access to it than to the things you claim are "just the brain spinning stories".
You say "directly" but that is simply not true. We have no direct access to anything, certainly not the physical world. "Through the senses" is Indirectly, not directly. And then the information is processed by the brain...
The mind making up things is a real problem, I agree, but this problem is present in our perception of the physical world as well.
We have no idea what it would even look like without the mind spinning narratives and interpreting signals.