Abstract

This paper investigates a downlink cooperative non-orthogonal multiple access (NOMA) system assisted by a multi-antenna amplify-and-forward (AF) relay. Unlike conventional single-antenna relay configurations, the considered framework jointly exploits relay diversity and direct transmission links between the base station (BS) and users. Under independent Nakagami-$m$ fading channels, closed-form expressions for the outage probability (OP) and ergodic capacity (EC) of both users are derived for scenarios with and without direct BS--user links. The analytical formulation explicitly captures the effects of the number of relay antennas, fading severity, and power allocation coefficients on system performance. For the ergodic capacity analysis, an exact integral representation combined with a Gaussian--Chebyshev quadrature approach is developed to efficiently evaluate the performance under the max-SINR selection criterion. The analytical results are verified through Monte Carlo simulations and compared with orthogonal multiple access (OMA) benchmarks. Numerical results demonstrate that increasing the number of relay antennas significantly improves reliability due to enhanced spatial diversity. Moreover, the NOMA scheme achieves superior outage performance for the far user compared with OMA, while the ergodic capacity of the near user exhibits a pronounced gain in the moderate-to-high SNR regime. These findings confirm the effectiveness of multi-antenna cooperative relaying in improving both reliability and spectral efficiency.

Key words: Cooperative non-orthogonal multiple access (NOMA); Multi-antenna AF relaying; Spatial diversity; Nakagami-m fading; Outage probability; Ergodic capacity.