In this dissertation, three new algorithms for three distinct problems are proposed. The three distinct problems considered here have applications to stochastic modeling, compressive sensing, and numerical solutions of partial differential equations. A common aspect of these problems is that to obtain accurate results require an ever increasing number of unknown variables. Since the proposed algorithms are more efficient than the state of the art methods used for these problems, the use of these new algorithms allows one to compute solutions to these problems with substantially higher accuracy. In addition, some theoretical analysis is provided relating to the investigated problems and the proposed algorithms.