I would like more information about the mathematical foundations of vulnerability and exploit development.online sources or books in the right direction will be helpful.
I would read up on static program analysis
Static program analysis is the analysis of computer software that is performed without actually executing programs (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code.
Dynamic program analysis is the analysis of computer software that is performed by executing programs on a real or virtual processor. For dynamic program analysis to be effective, the target program must be executed with sufficient test inputs to produce interesting behavior
In computer science, abstract interpretation is a theory of sound approximation of the semantics of computer programs, based on monotonic functions over ordered sets, especially lattices. It can be viewed as a partial execution of a computer program which gains information about its semantics (e.g. control-flow, data-flow) without performing all the calculations.
In computer science, symbolic execution (also symbolic evaluation) refers to the analysis of programs by tracking symbolic rather than actual values, a case of abstract interpretation. The field of symbolic simulation applies the same concept to hardware. Symbolic computation applies the concept to the analysis of mathematical expressions. Symbolic execution is used to reason about all the inputs that take the same path through a program.
In mathematics and computer science, computer algebra, also called symbolic computation or algebraic computation is a scientific area that refers to the study and development of algorithms and software for manipulating mathematical expressions and other mathematical objects
In computer science, a simulation is a computation of the execution of some appropriately modelled state-transition system. Typically this process models the complete state of the system at individual points in a discrete linear time frame, computing each state sequentially from its predecessor.
In computer science, model checking aka property checking refers to the following problem: Given a model of a system, exhaustively and automatically check whether this model meets a given specification.
might want to read System Assurance: Beyond Detecting Vulnerabilities.
Rolf probably has a ton of really good input on this subject. Read about his advice here
As it turns out, somebody asked roughly the same question on reddit about a year ago and I posted a rather extensive answer to it, and I have continued to edit it in the meantime.