I was lately curious about an iterative approach that would solve maths equations containing composed functions with contraints.

For example, if I have the following equation:

$$ f(g(h(w))) = 0 text{, with } w = begin{pmatrix}

a_{11} & 0 & ldots & a_{1n}\

0 & a_{22} & ldots & a_{2n}\

vdots & vdots & ddots & vdots\

0 & 0 &ldots & a_{nn}

end{pmatrix}

$$

Along additional constraints on the $3$ functions like

$ f < g $;

$ h > 2 * g $; and

$ f, g,h $ not constant

**The goal is to find the $3$ functions expressions given a specific matrix $w$ and the constraints.**

What approach would be the most convenient to find a solution or solution**s** to this problem ? I was personally thinking about using reinforcement learning (machine learning) where each time a solutions is chosen, a positive or negative reward will be attributed to the solution generator.

Thank you