A REVIEW OF OPTIMIZATION TECHNIQUES IN ARTIFICIAL NETWORKS.

Previous studies showthat neural networks are implemented on various problemssuch as regression and classification successfully. Optimization process of a neural network is highly in accordance with unconstrained optimization theory and many efforts have been done to accelerate this process. Particularly, various algorithms have been developed by numerical optimization theory to speed up neural network optimization. Additionally, conventional innovative approaches like variable learning rate or momentum lead to a considerable improvement in this regard. In this paper, first, we introduced optimization techniques in neural networks and then, compared the methods in three problems of parity-n, alphabetic fonts, and Monk. We attempted to conduct a study at a large scale on performance of the presented algorithms and identify their probable advantages.