If you have a verified email address, it is not currently possible to change your email address without demonstrating you have access to the original email address; the account management portal has had two-factor authentication like this for quite some time.
We could add mobile verification to that, but like I said, our current policy (which is in line with most other services out there) is that if you have access to the email address of the account holder, you are the account holder. It also does not sit well with me personally that we would be required to collect your phone number, or that you need a smart phone, in order to benefit from a core security measure.
That said, if you don't have your email verified, yes, this is a big problem. We will revisit the scope of this problem in the future to see if we can't shift more players to have verified emails.