Zubov's Method for Stochastic Control Systems

We consider a controlled stochastic system with an a.s. locally exponentially controllable compact set. Our aim is to characterize the set of points which can be driven by a suitable control to this set with some prescribed probability. We show that a generalization of Zubov's method leads to this characterization and can be used as basis for numerical computations.


INTRODUCTION
Zubov's method (Zubov, 1964) is a general procedure for deterministic systems of ODEs which allows to characterize the domain of attraction of an asymptotically stable fixed point and an associated Lyapunov function on this domain by the solution of a suitable partial differential equation, the Zubov equation (see f.e.(Khalil, 1996) for an account of the various developments of this method).
A typical difficulty in the application of this method is the existence of a regular solution to the Zubov equation, which was overcome in (Camilli et al., 2001) by using a suitable notion of weak solution, the Crandall-Lions viscosity solution.The use of weak solutions allows the extension of this method to perturbed and controlled systems, see (Grüne, 2002), Chapter VII for an overview.
Using this framework, in (Camilli and Loreti, 2004), (Camilli and Grüne, 2003) the Zubov method was applied to (uncontrolled) Ito stochastic differential equations obtaining a characterization of the points which are attracted with any prescribed probability to the fixed point.
In control theoretic applications it is interesting to consider the so-called asymptotic controllability problem, i.e. the possibility of asymptotically driving a nonlinear system to a desired target by a suitable choice of the control law.Whereas in the deterministic case there is huge literature about this problem (see f.e.(Sontag, 1999)), in the stochastic case it seems to be less considered, also because it request some degeneration of the stochastic part which makes it difficult to handle with classical stochastic techniques.In (Grüne and Wirth, 2000) Zubov's method was extended to this problem for deterministic systems and in this paper we apply this method to stochastic control systems, proceeding in two steps: In the first step in Section 2 we introduce a suitable optimal control problem associated with the stochastic system.We show that a suitable level set of the corresponding value function v gives the set of initial points for which there exists a control driving the stochastic system to the locally controllable set with positive probability.The value function is characterized as the unique viscosity solution of the Zubov equation, which is the Hamilton-Jacobi-Bellman of the control problem.
In the second step in Section 3 we introduce a discount factor δ > 0 and pass to the limit for δ → 0 + .We show that the set of points controllable to the fixed point with probability p ∈ [0, 1] is given by the subset of R N where the sequence v δ converges to 1 − p.The sequence v δ converges to a l.s.c.v 0 which is a supersolution of a Hamilton-Jacobi-Bellman related to an ergodic control problem.In this respect the Zubov equation with positive discount factor can be seen as a regularization of the limit ergodic control problem which gives the appropriate characterization.
Finally, in Section 4 we describe an example where the previous objects are calculated numerically.

ZUBOV'S EQUATION AND POSSIBLE NULL-CONTROLLABILITY
We fix a probability space (Ω, F, F t , P), where {F t } t≥0 is a right continuous increasing filtration, and consider the controlled stochastic differential equation (1) where α(t), the control applied to the system, is a progressively measurable process having values in a compact set A ⊂ R M .We denote by A the set of the admissible control laws α(t).Solutions corresponding to an initial value x and a control law α ∈ A will be denoted by X(t, x, α) (or X(t) if there is no ambiguity).
We assume that the functions b : R N × A → R N , σ : R N ×A → R N ×M are continuous and bounded on R N × A and Lipschitz in x uniformly with respect to a ∈ A and that 0 ∈ A.
Moreover we assume that there exists a set ∆ ⊂ R N locally a.s.exponentially null-controllable, i.e. there exist r, λ positive and a finite random variable β such that for any x ∈ B(∆, r) = {x ∈ R N : d(x, ∆) ≤ r}, there exists α ∈ A for which d(X(t, x, α), ∆) ≤ βe −λt a.s.for any t > 0. (2) In this section we study the domain of possible null-controllability i.e., the set of points x for which it is possible to design a control law α such that the corresponding trajectory X(t, x, α) is attracted with positive probability to ∆.
We introduce a control problem associated to the dynamics in the following way.We consider for x ∈ R N and α ∈ A the cost functional Proof: Note that by definition 0 ≤ v ≤ 1 and v(x) > 0 for x ∈ ∆.We claim that C is the set of the points x ∈ R N for which there exists α ∈ A such that E[exp(−t(x, α))] > 0, where and therefore v(x) = 1.
If x ∈ C, by the previous claim there exists α such that P[t(x, α) < +∞] > 0. Set τ = t(x, α) and take T and K sufficiently large in such a way where M g and L g are respectively an upper bound and the Lipschitz constant of g.
We have obtained a link between C and v.In the next two propositions we study these objects in order to get a PDE characterization of v.
ii) C is open, connected, weakly positive forward invariant (i.e.there exists α ∈ A such that the inequality P[X(t, x, α) ∈ C for any t] > 0 holds.) iii Proof: The proof is a straightforward generalization of the proofs of the corresponding results in (Camilli and Loreti, 2004).
Remark 2.3.Note that if C does not coincide with all R N , the weakly forward invariance property requires some degeneration of the diffusion part of (1) on ∂C, see f.e.(Bardi and Goatin, 1999).
The typical example we have in mind is a deterministic system driven by a stochastic force, i.e. a coupled system see e.g.(Colonius et al., 1996) for examples of such systems.Note that for systems of this class the diffusion for the overall process ) is naturally degenerate.
Set Σ(x, a) = σ(x, a)σ t (x, a) for any a ∈ A and consider the generator of the Markov process associated to the stochastic differential equation Proposition 2.4.v is continuous on R N and a viscosity solution of Zubov's equation Proof: The only point is to prove that v is continuous on R N .Then a standard application of the dynamic programming principle shows that v is a viscosity solution of (5) (see f.e.(Yong and Zhou, 1999), (Fleming and Soner, 1993)). (xn,α) ] → 1 for n → +∞ and hence v is continuous on the boundary of C.
To prove that v is continuous on the interior of C, it is sufficient to show that v is continuous in B(∆, r) since outside g is strictly positive and we can use the argument in (Lions, 1983, part I), Theorem II.2.
Fix x, y ∈ B(∆, r) and > 0. Let b be such that Take T in such a way that L g b exp(−λT )/λ < /4, where λ as in (2), and let α be a control satisfying (2) and g(X(t,x,α),α(t))dt ] + 8 and δ sufficiently small in such a way that The next theorem gives the characterization of C through the Zubov equation (5).
Theorem 2.5.The value function v is the unique bounded, continuous viscosity solution of (5) which is null on ∆.
Proof: We show that if w is a continuous viscosity subsolution of ( 5) such that w(x) ≤ 0 for x ∈ ∆, then w ≤ v in R N .Using a standard comparison theorem (see f.e.(Fleming and Soner, 1993)), the only problem is the vanishing of g on ∆.Therefore we first prove that w ≤ v in B(∆, r) using (2), we then obtain the result in all R N by applying the comparison result in R N \ B(∆, r).
Fix > 0 and let δ > 0 be such that if d(z, ∆) ≤ δ, then w(z), v(z) ≤ .For x ∈ B(∆, r) by the dynamic programming principle we can find α ∈ A satisfying (2) and such that Therefore we have where Set B k = {β ≤ K} and take T and K sufficiently large in such a way that 2M e −g δ T ≤ , 2M P[B c k ] ≤ and, recalling (2 k ] + 2 ≤ 4 , and thus w ≤ v in B(∆, r) since was arbitrary.
By a similar argument we can prove that if u is a continuous viscosity supersolution of ( 5) such that u(x) ≥ 0 for x ∈ ∆, then u ≥ v in R N .
Remark 2.6.The function v is a stochastic control Lyapunov function for the system in the sense that for any x ∈ C \ ∆ and any t > 0.

CONTROLLABILITY DOMAINS
In this section we are interested in the set D p of points x ∈ R N which are asymptotically controllable to the set ∆ with probability arbitrarily close to a given p ∈ [0, 1], i.e., We require a slightly stronger stability condition, namely that besides (2) it also holds that for any x ∈ B(∆, r) there exists a control α ∈ A such that E[d(X(t, x, α), ∆) q ] ≤ M e −µt for any t > 0 (7) for some q ∈ (0, 1] and positive constants M , µ.This assumption is motivated by the uncontrolled linear case where ( 7) is a consequence of (2).
We consider a family of value functions depending in the discount factor on a positive parameter δ The main result of this section is Theorem 3.1.
Proof: The proof is split in three steps.
Since g is Lipschitz continuous in x uniformly in a and g(x, a) = 0 for any (x, a) ∈ ∆ × A, we have g(x, a) ≤ min{L g x , M g } ≤ C q x q for any q ∈ (0, 1] and corresponding constant C q .Let α be a control satisfying (7).Then for any δ, by the Lipschitz continuity of g, (2) and ( 7) we get where t(x, a) is defined as in (4).
The proof of the claim is very similar to the one of Lemma 3.2 in (Camilli and Grüne, 2003), so we just sketch it.Let α ∈ A be such that sup (x,α) ] + and T 0 such that exp(−δT ) ≤ for T > T 0 .Hence for T > T 0 To obtain the other inequality in (9), take α ∈ A, T sufficiently large and δ small such that and, for t < T , e −δt ≥ 1 − .Then Since is arbitrary, it follows that lim inf For any α ∈ A, we have (x,α) ] and therefore by Claim 2, lim inf Now fix > 0, δ > 0 and take T sufficiently large such that exp(−δM g T ) ≤ .By the dynamic programming principle, for any α ∈ A we have δg(X(s),α(s))ds dt+ e − T ∧t(x,α) 0 δg(X(t),α(t))dt v(X(T ∧ t(x, α))}.
(10) Now using Claim 1 and recalling that 0 ≤ v δ ≤ 1 we estimate the second term in the right hand side of ( 10) by

A NUMERICAL EXAMPLE
We illustrate our results by a stochastic version of a creditworthiness model from (Grüne et al., 2005) given by In this model k = x 1 is the capital stock of an economic agent, B = x 2 is the debt, j = α is the rate of investment, H is the external finance premium and f is the agent's net income.The goal of the economic agent is to steer the system to the set {x 2 ≤ 0}, i.e., to reduce the debt to 0. Extending H to negative values of x 2 via H(x 1 , x 2 ) = θx 2 one easily sees that for the deterministic model controllability to {x 2 ≤ 0} becomes equivalent to controllability to ∆ = {x 2 ≤ −1/2}, and that for the stochastic model this set ∆ satisfies our assumptions.

Using the parameters
2 we have numerically computed the solution v δ for the corresponding Zubov equation with δ = 10 −4 using the scheme described in (Camilli and Grüne, 2003) extended to the controlled case (see (Camilli and Falcone, 1995) for more detailed information).For the numerical solution we used the time step h = 0.05 and an adaptive grid (see (Grüne, 2004)) covering the domain Ω = [0, 2] × [−1/2, 3].For the control values we used the set A = [0, 0.25].
As boundary conditions for the outflowing trajectories we used v δ = 1 on the upper boundary and v δ = 0 for the lower boundary, on the left boundary no trajectories can exit.On the right boundary we did not impose boundary conditions (since it does not seem reasonable to define this as either "inside" or "outside").Instead we imposed a state constraint by projecting all trajectories exiting to the right back to Ω.We should remark that the effect of these conditions has to be taken into account in the interpretation of the results.Figure 1 show the numerical results for σ = 0, 0.1 and 0.5 (top to bottom).In order to improve the visibility, we have excluded the values for x 1 = 0 from the figures (observe that for x 1 = 0 and x 2 > 0 it is impossible to control the system to ∆, hence we obtain v δ ≈ 1 in this case).