As an industrial by-product, phosphogypsum has become a promising candidate for reducing environmental pollution and improving resource utilization efficiency. However, the mechanical properties and durability of phosphogypsum-based materials are limited, necessitating modification to enhance their overall performance. To improve the performance of phosphogypsum-based composite concrete (PGC), nanosilica (NS) was incorporated into the mixture to produce NS-modified PGC (PGC-NS). The results indicated that PGC-NS with 1.5% NS content (PGC-1.5NS) exhibited excellent performance, achieving a 28-day compressive strength of 63.8 MPa and a softening coefficient of 0.98. Fourier transform infrared spectroscopy and thermogravimetric analysis revealed that the incorporation of NS promoted the hydration reaction of PGC-NS, leading to the formation of more C-S-H gel. Sustainability analysis showed that the energy consumption and carbon emissions of PGC-1.5NS were only 34% and 28% of those of C50 concrete, respectively, highlighting its significant environmental and low-carbon benefits. The findings provide new insights for the modification of phosphogypsum-based composite concrete and offer guidance for promoting the green development of concrete.