در مورد این فرمول کوچکتر کردن کمکی نمیکند. به نظرم اگر آن را از جای مناسب بشکنید بهتر باشد:
\documentclass{article}
\usepackage{amsmath}
\usepackage{xepersian}
\settextfont{XB Niloofar}
\begin{document}
\begin{equation}
\begin{aligned}
&\left( \begin{matrix}
\psi _{1}^{\text{i}{{\text{n}}^{\xi }}} & \psi _{2}^{\text{i}{{\text{n}}^{\xi }}} & \psi _{3}^{\text{i}{{\text{n}}^{\xi }}} \\
\left\langle {{\phi }_{1}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{1}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{1}}|{{\psi }_{3}} \right\rangle \\
\left\langle {{\phi }_{2}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{2}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{2}}|{{\psi }_{3}} \right\rangle \\
\left\langle {{\phi }_{3}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{3}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{3}}|{{\psi }_{3}} \right\rangle \\
\end{matrix} \right)=\\
&{{\left( \begin{matrix}
\xi _{1}^{\text{i}{{\text{n}}^{\phi }}} & \xi _{2}^{\text{i}{{\text{n}}^{\phi }}} & \xi _{3}^{\text{i}{{\text{n}}^{\phi }}} \\
\text{ }\left\langle {{\phi }_{1}}|{{\xi }_{1}} \right\rangle & \left\langle {{\phi }_{1}}|{{\xi }_{2}} \right\rangle & \left\langle {{\phi }_{1}}|{{\xi }_{3}} \right\rangle \\
\left\langle {{\phi }_{2}}|{{\xi }_{1}} \right\rangle & \left\langle {{\phi }_{2}}|{{\xi }_{2}} \right\rangle & \left\langle {{\phi }_{2}}|{{\xi }_{3}} \right\rangle \\
\left\langle {{\phi }_{3}}|{{\xi }_{1}} \right\rangle & \left\langle {{\phi }_{3}}|{{\xi }_{2}} \right\rangle & \left\langle {{\phi }_{3}}|{{\xi }_{3}} \right\rangle \\
\end{matrix} \right)}^{\dagger }}.{{\left( \begin{matrix}
\psi _{1}^{\text{in}\phi } & \psi _{2}^{\text{in}\phi } & \psi _{3}^{\text{in}\phi } \\
\left\langle {{\phi }_{1}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{1}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{1}}|{{\psi }_{3}} \right\rangle \\
\left\langle {{\phi }_{2}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{2}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{2}}|{{\psi }_{3}} \right\rangle \\
\left\langle {{\phi }_{3}}|{{\psi }_{1}} \right\rangle & \left\langle {{\phi }_{3}}|{{\psi }_{2}} \right\rangle & \left\langle {{\phi }_{3}}|{{\psi }_{3}} \right\rangle \\
\end{matrix} \right)}^{{}}}
\end{aligned}
\end{equation}
\end{document}
برای این کار از محیط aligned
از بسته amsmath
استفاده کردهام.
نتیجه: