I doubt that you can redefine brackets in LaTeX. I don't think it is a PGF math limitation, but a LaTeX limitation. Brackets are used to delimit command options in LaTeX. In PGF math expressions, brackets are used to access a particular element of an array, but this is just the usual LaTeX command option notation. Perhaps you could change the definition using catcodes, but that seems like a dangerous thing to do.
You are more likely to have success modifying MathObject behavior than changing bracket behavior in TikZ and PGF.
You are more likely to have success modifying MathObject behavior than changing bracket behavior in TikZ and PGF.