This study focuses on investigating protection challenges in microgrids and presenting a novel approach for the optimal coordination of overcurrent relays using the Grey Wolf Optimization (GWO) algorithm. Due to the presence of distributed generation (DG) sources and the capability of operating in both grid-connected and islanded modes, microgrids experience variations in short-circuit levels and bidirectional power and fault-current flows, which can adversely affect the proper operation of conventional protection relays. To address this issue, the coordination problem between primary and backup relays is formulated as an optimization problem, and the GWO algorithm, inspired by the hierarchical structure and cooperative hunting behavior of grey wolves, is employed to determine the optimal values of the relay pickup current setting (ISET) and time multiplier setting (TMS). The test system considered in this study is the standard IEEE 30-bus system. Simulation results demonstrate that the proposed method significantly reduces relay operating times, prevents interference between primary and backup relay operations, and maintains proper protection coordination under various operating conditions. A comparison with other metaheuristic algorithms, including the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO), indicates that the GWO algorithm provides faster convergence, greater stability, and more accurate solutions. Overall, the findings demonstrate the high effectiveness of the GWO algorithm in enhancing the reliability and improving the protection performance of microgrids. Therefore, it can be considered an effective approach for the design and operation of future microgrid protection systems. Future studies are recommended to investigate hybrid approaches combining GWO with other swarm intelligence techniques and to evaluate its performance in larger microgrids under load and distributed-generation uncertainties.