This dissertation studies the extension of sparse optimization techniques to the numerical solution of partial differential equations for applications in scientific computing, in particular many-particle systems that are governed by a differential equation. Sparse optimization techniques have attracted much attention due to their substantial computational efficiency and feasibility for large-scale problems such as image processing, compressed sensing, and machine learning. In this dissertation, $\ell^1$-minimization scheme has been studied for the solutions of elliptic and parabolic differential equations. Theoretical considerations for the effectiveness of the scheme, such as the sparsity properties, completeness, consistency, and the asymptotic behavior of the solutions are analyzed.