The Directed Subdifferential of DC Functions

The space of directed sets is a Banach space in which convex compact subsets of Rn are embedded. Each directed set is visualized as a (nonconvex) subset of Rn, which is comprised of a convex, a concave and a mixed-type part. Following an idea of A. Rubinov, the directed subdifferential of a difference of convex (DC) functions is defined as the directed difference of the corresponding embedded convex subdifferentials. Its visualization is called the Rubinov subdifferential. The latter contains the Dini-Hadamard subdifferential as its convex part, the Dini-Hadamard superdifferential as its concave part, and its convex hull equals the Michel-Penot subdifferential. Hence, the Rubinov subdifferential contains less critical points in general than the Michel-Penot subdifferential, while the sharp necessary and sufficient optimality conditions in terms of the Dini-Hadamard subdifferential are recovered by the convex part of the directed subdifferential. Furthermore, the directed subdifferential could distinguish between points that are candidates for a maximum and those for a minimum. It also allows to easily detect ascent and descent directions from its visualization. Seven out of eight axioms that A. Ioffe demanded for a subdifferential are satisfied as well as the sum rule with equality.


Introduction
In [1,2] a linear normed and partially ordered space is introduced, in which the convex cone of all nonempty convex compact sets in R n is embedded and thus a way to subtract convex sets is opened.This space of so-called "directed sets" is a Banach and a Riesz space for dimension n ≥ 2 and a Banach lattice for n = 1.It is defined without equivalence classes as the difference in [25,28,31].Our embedding is more involved than the one using only support functions [13], but provides a visualization of DC sets as usually non-convex sets in R n with attached normal directions.In this space one can calculate "directed differences" of convex compact sets and visualize them as non-convex sets in R n .
In 2000 the late A. Rubinov suggested us to apply the directed differences of convex sets in order to define a subdifferential of DC (differences of convex) functions.It took us a long time to convince ourselves that the existence of such a new subdifferential is justified among the variety of already known subdifferentials, convex and non-convex ones.The visualization of the new subdifferential helped us to see its interesting properties and relations to other known subdifferentials.
The paper is organized as follows: After some basic notations below, we recall several well-known definitions of subdifferentials.In Section 3 we provide some basic definitions and facts on the Banach space of directed sets and in Section 4 we define the directed subdifferential and the Rubinov subdifferential and give some examples.In Section 5 we describe the relations of the Rubinov subdifferential to well-known subdifferentials, and in Section 6 we discuss conditions for optimality, saddle points, ascent and descent directions.

Basic Notations
Let C(R n ) be the set of all convex, compact, nonempty subsets of R n .We denote by S n the unit sphere in R n , and by cl(A), co(A) the closure and the convex hull of the set A respectively.The following operations in C(R n ) are well-known: which coincides with the subdifferential of the support function.Some known definitions of differences of two sets are listed below: • geometric (Hadwiger-Pontryagin) difference [10,27], sometimes called starshaped difference This difference has the property that A − * A = {0}, but may often be empty.
2 Preliminaries -Some Known Subdifferentials We recall first definitions of subdifferentials which are always convex.The classical convex (Moreau/Rockafellar) subdifferential of a convex function f : The vector s ∈ ∂f (x) is called (convex) subgradient of f at x.This subdifferential is a convex, compact and nonempty set for convex f : R n → R (see e.g.[29]), and its support function is the directional derivative where the directional derivative of f at x in direction l is defined as In [18, Chapter 2, Section 1.5], the following list of axioms for subdifferentials for Lipschitz functions is given.
, if f and g coincide in a neighborhood of x or as Rockafellar proposed: coincides with the classical convex subdifferential.
Naturally, the classical convex subdifferential satisfies the above axioms.In addition it fulfills the following stronger form of (SD 6 ) for convex functions g, h : R n → R and x ∈ R n , sometimes called Moreau-Rockafellar theorem or the Sum Rule (cf.[29,Theorem 23.8]): This stronger equality is not fulfilled for many other subdifferentials of non-convex fuctions.
We now define some other convex subdifferentials.The Dini-Hadamard subdifferential studied e.g. in [3,26,16,12] and in [9, Section III.4] (under the name "Penot subdifferential"), sometimes called the radial subdifferential or Gâteaux-subdifferential, is defined for a function f : R n → R that has directional derivatives in x ∈ R n for all directions l ∈ S n−1 : This definition is identical to (2) for convex functions, but the directional derivative is not necessarily a convex function w.r.t.l for a non-convex function.
If f : R n → R is a DC function with f = g − h, it is observed in [12], [9, Chapter III, Proposition 4.1], that the Dini-Hadamard subdifferential equals the geometric difference of the two convex subdifferentials, i.e.
To introduce the Michel-Penot subdifferential [21], we recall that the Michel-Penot directional derivative of a function f : and the Michel-Penot subdifferential of f at x is The following connection between the Michel-Penot subdifferential and the Demyanov difference follows from [7, Theorem 6.1] for any DC function f = g − h (g, h convex), and l, x ∈ R n : and the Michel-Penot subdifferential calculates as The most well-known subdifferential of a non-convex function is probably the Clarke's subdifferential (cf.[4]), which is also a convex set.For f : R n → R and l, x ∈ R n , the Clarke directional derivative of f at x in direction l is the limit: The Clarke subdifferential is defined as As it is well-known, cf.e.g., [5,7], and they are equal in the case of a convex function f .These inclusions may be strict as it is shown in the examples in Section 4. Now, we recall definitions of some non-convex subdifferentials.The most famous nonconvex subdifferential is the (basic/lower) subdifferential of Mordukhovich, [22], [24, Definition 1.77], ∂ M f (x), which is equivalent to the approximate subdifferential of Ioffe in finite dimensions [14,15,17], [24,Theorem 3.59] and may be defined as Recall that a vector v is a proximal subgradient of f at x, if for some ε > 0 As it is well-known, the Mordukhovich subdifferential is a compact in R n , and the Clarke subdifferential is its (closed) convex hull (see e.g., [15], [24, Theorem 3.57]).
Finally, we mention the quasidifferential of Demyanov-Rubinov [9, Chapter III, Section 2] of DC functions, defined as an element of a linear normed space of equivalence classes generated by pairs of convex sets, following the approach of Rådström in [28].For the DC function f (x) = g(x) − h(x), its quasidifferential is generated by the pair [∂g(x), −∂h(x)].A tool to get rid of the non-uniqueness of the pairs, keeping other positive features of this approach, is the space of directed sets.
To define our directed subdifferential of a DC function as an element of this space, we need some background on directed sets, presented in the next section.

Directed Sets
Extending the parametrization of convex compacts via their support functions, we define a directed set as a pair of mappings that associates to each unit direction a (n−1)-dimensional directed set ("directed supporting face") and a scalar (the value of the "support function") in this direction.This method enables us to use recursive constructions.
The definition of directed sets is inductive in the dimension n ≥ 1 and based on the notion of directed interval for n = 1.Our definition of directed intervals is based on support functions, and is equivalent to the generalized and directed intervals ( [19,20]).
Linear combinations of directed intervals are calculated as linear combinations of vectors: We visualize a directed interval − −− → [α, β] attaching to every its end points the corresponding unit normal vector, l 1 = −1 to the point α and l 2 = 1 to β.The directed interval is proper when α ≤ β (then the normals are pointing outwards) and improper otherwise (with normals pointing inwards).Some proper and improper intervals are visualized in Example 3.2.
Example 3.2 One proper and one improper interval, obtained by subtraction of directed intervals are shown here: Another improper interval as well as the embedded scalar 1 are visualized below: Motivated by describing the convex, compact, nonempty set A for each direction l ∈ S n−1 by its (n − 1)-dimensional supporting face and the value of the support function, the linear normed space D(R n ) of the directed sets in R n is constructed inductively.
The set of all directed sets in R n is denoted by The linear operations are defined recursively, on the two components of the directed sets , where ×n is the natural projection, R n,l is a fixed rotation for every l ∈ S n−1 satisfying We introduce a multiplication of a matrix M ∈ R m×n for a difference of two embedded convex compacts The visualization for a directed set in D(R n ) consists of three parts: the convex, the concave and the mixed type parts.We recall their definitions here.
A are defined by: The mixed-type part M n ( − → A ) is defined recursively and collects all reprojected points from the visualization of the "support faces" which are not elements of the convex or concave part.
The visualization V n : D(R n ) ⇒ R n is the union of the three parts with the reprojection Q n,l (y Except the degenerate case when V n ( − → A ) consists of a single point, at least one of the convex and the concave part of A is empty.It may happen that both of them are empty and the visualization consists only of the mixed-type part.In the one-dimensional case the mixed-type part is empty, and in the non-degenerate case exactly one of The visualization of the difference of directed sets is strongly related to other differences.The following equalities are known for [2]: Example 3.5 The visualization of the inverse in Fig. 1 comprises the pointwise negations of the boundary points, keeping the corresponding normal directions l.Note that if the set A is symmetric with respect the origin, then the visualizations of − → A and − − → A coincide, and the only way to distinguish between them is to add the normal vectors to the visualisation.
The operations addition, scalar multiplication commute with the embedding and the visualisation, cf.[2].Namely, for A, B ∈ C(R n ) and λ ≥ 0: The directed subdifferential The following definition was suggested to us by A. M. Rubinov.
We call its visualisation the Rubinov subdifferential: The vector s ∈ ∂ R f (x) is called Rubinov subgradient of f at x.
We note that the directed subdifferential is well-defined for DC functions, i.e. ∂ R f (x) does not depend on the specific representation f = g − h, which may be easily checked, using the property ( 16) of the embedding J n and the fact that D(R n ) is a vector space in which the cancellation law holds.Also, the Rubinov subdifferential is always a nonempty compact, not necessarily convex set.
The following properties of the directed subdifferential for DC functions are established (note the stronger versions of (SD 6 ) and (SD 7 ) being also valid for negative λ): Proposition 4.2 Let f : R n → R be a DC-function and x ∈ R n .Then, the directed subdifferential of f at x fulfills: (SD ′ 6 ) Let n = n 1 + n 2 and x = (x (1) , x (2) with the projection Π i : R n → R n i and Π i (x) = x (i) for x = (x (1) , x (2) ), i = 1, 2.
Moreover, the sum rule holds for each DC function f: Proof: We prove here only (SD 3 ), part of (SD ′ 7 ) and (SR): In (SD ′ 7 ) we consider only f . In (SR) with f = g − h, f = g − h, it follows the DC representation (g + g) − (h + h) for the sum.From (SR) in Section 2 and (16) follows the sum rule for the directed subdifferential.
We note that the sum rule (SR) can only be proved for the directed subdifferential and not for the Rubinov one, since the linear structure of the directed sets space D(R n ) is essential in the proof.For the subdifferential of Rubinov, the left-hand side is only a subset of the right-hand side, since the visualization of the sum of two directed sets is only a subset of the Minkowski sum of their visualizations, which follows easily from the definitions.
The following relations follow from ( 7), ( 9), (15), and Definitions 3.4 and 4.1, and clarify the place of the Rubinov subdifferential among other known subdifferentials: Directly from [5, Lemma 2.3] with the corresponding fact for the Michel-Penot subdifferential, it follows that the Rubinov subdifferential generalizes the Gâteaux differential.[33])  By (SD 6 ), the convex subdifferentials of g, h are For this function, the Dini-Hadamard subdifferential is empty, and the Michel-Penot subdifferential coincides with Clarke's one.
In this example, we have the following inclusions of the subdifferentials: The Mordukhovich subdifferential of this function coincides with the Treiman one and consists of two parallel edges of the unit square: The directed subdifferential comprises only a mixed-type part (its positive and negative part are empty), as one can see in Figure 2.For this example, the union of the Mordukhovich lower and upper subdifferential (cf.[23], [24, Definition 1.78]) in Fig. 3 gives the Rubinov subdifferential which consists of the boundary of the square The Mordukhovich subdifferential in the previous example was a subset of the Rubinov one, while in the next example the opposite inclusion holds.
Here, we have the following inclusions of the subdifferentials: As it is known, the Mordukhovich subdifferential contains the extremal points (0, 0) and (0, 1) of Clarke subdifferential (cf.Fig. 4), thus The following conjecture may be derived from the last examples: If the Clarke subdifferential coincides with Michel-Penot one, then the Rubinov subdifferential coincides with the

Optimality conditions, descent and ascent directions
Here we translate some known necessary and sufficient conditions for an extremum into the language of directed subdifferential.In the case when the Dini-Hadamard subdifferential is non-empty, these conditions are equivalent to those known for it.If it is empty (which means that the considered point is not a minimizer), we provide a simple criterion which says if a given direction is a direction of ascent or descent.
Let f : R n → R, l ∈ S n−1 .We first recall some definitions.If there exists ε > 0 such that f (x + tl) < f (x) for t ∈ (0, ε], then l is called direction of descent at x.In the case of f (x + tl) > f (x) for t ∈ (0, ε], l is called direction of ascent at x.We call the point x a strict saddle point, if there exist a direction of ascent with a positive directional derivative and a direction of descent with a negative directional derivative at x. Clearly, if all directions are direction of ascent at x, it is a strct minimum, and similarly for a strict maximum.Next, we state well-known necessary conditions for an extremum of a DC function in terms of the directed subdifferential. Proof: (i) If f = g − h has a local minimum in x, then necessarily 0 R n ∈ ∂g(x) − * ∂h(x), cf.[26,8,12] and [9, Chapter V, Theorem 3.1].But the geometric difference coincides with the positive part of the directed subdifferential − → ∂ f (x) = J n (∂g(x)) − J n (∂h(x)).(ii) Similarly, 0 R n ∈ ∂h(x) − * ∂g(x) is a necessary condition for a local maximum of f at x.The geometric difference could be retranslated as negative part of the visualization of − → ∂ f (x).(iii) Since there exists l 1 , l 2 ∈ S n−1 with f ′ (x; l 1 ) < 0 and f ′ (x; l 2 ) > 0, the origin could not lie in the positive or negative part of the visualization.Hence, it must lie in the mixed-type part of the directed subdifferential.
It is known that if f : R n → R is directional differentiable for l ∈ S n−1 , then • if f ′ (x; l) < 0, then l is a direction of descent, • if f ′ (x; l) > 0, then l is a direction of ascent.
The following simple criterion helps to distinguish between directions of ascent and descent by the sign of the second component of the directed subdifferential (its "support" function).
Proof: For f = g − h we have due to the embedding and ( 12) that Clearly, the sign of the directional derivative determines whether l is a direction of ascent or descent.
On the visualization, this may be interpreted as follows: if for a given l there is a boundary point z of the directed subdifferential with corresponding normal direction l such that l, z < 0, then l is a descent direction, and similarly for an ascent direction.
Proof: The proof is similar to the one of Proposition 5.1 and uses the sufficient condition for optimality "0 R n ∈ int ∂g(x) − * ∂h(x) " mentioned above.
For a further reference to ascent and descent directions, see e.g., [9, Section V.1].
Let us now go back to the examples discussed in the last section.
Example 5.5 Consider the funtion f from Example 4.5.As mentioned before, the Dini-Hadamard subdifferential at x = 0 is empty, thus the origin is not a minimum point (also the Dini-Hadamard of −f (x) is empty, so it is not a maximum).Clearly, the zero is outside the non-convex subdifferentials of Mordukhovich, Rubinov and Treiman, but is inside the subdifferentials of Clarke and Michel-Penot.A closer look at the visualization of the directed subdifferential (Fig. 2) enables to conclude that the directions (1, 0), (−1, 0) are ascent directions, and (0, 1), (0, −1) are descent directions.Checking the values of f ′ (x; l) in these directions we conclude that the origin is a strict saddle point.In the next example, the criterion in Proposition 5.3 together with Propositions 5.1 and 5.4 are used to test the optimality conditions.
The function and the directed subdifferential are plotted in Fig. 5-8 for r ∈ {0.5, 1.0, 1.25, 1.5}.5 that the origin is a global minimum.For this function, the sufficient condition for a minimum in Proposition 5.4 is fulfilled: and all directions are directions of ascent (all the normals are pointing from the corresponding boundary points away from the origin).
and all directions are directions of descent (all the normals are pointing from the corresponding boundary points towards the origin).

Conclusions
The directed subdifferential is a directed (oriented) set, and not a usual set in R n .Its non-oriented visualization, the Rubinov subdifferential, is a compact connected (generally non-convex) subset of R n with its boundary parametrized by the (normal) directions of the unit sphere in R n : to each (unit) direction l there is a unique connected "support face" lying in a hyperplane with normal l.
As we saw in the previous section, the information of the orientation of the normals (the sign of the "support function") enables to identify directions of descent or ascent.In the future research we intend to extend the calculus rules for the new subdifferential, and to introduce and study directed/Rubinov normal and tangent cones.We hope that the investigation of the relations between the directed/Rubinov subdifferential and the subdifferentials of Mordukhovich and Treiman may indicate whether it is possible to extend the definition of the Rubinov subdifferential beyond the case of DC functions.Furthermore, the property (SD 8 ) should be proved.

Corollary 4 . 4
Let f = g − h be a DC function with Gâteaux differentiable functions g and h, then the Rubinov subdifferential consists only of the gradient of f .The exact relation between the Rubinov subdifferential and the non-convex subdifferentials of Mordukhovich and Treiman is not yet clear to us.The following examples show that none of the subdifferentials of Rubinov and Mordukhovich is necessarily a subset of the other.

Proposition 5 . 1
Let f : R n → R be a DC-function and x ∈ R n .Then

Proposition 5 . 4
Let f : R n → R be a DC-function with f = g − h and x ∈ R n .

Example 5 . 6
We consider the function f (x) from Example 4.6.Here, zero is outside the Dini-Hadamard, Rubinov, Michel-Penot and Treiman subdifferentials, but belongs to the subdifferentials of Mordukhovich and Clarke.Thus the latter two subdifferentials deliver critical points for the origin being no extremum.

Figure 6 .
Here, the necessary condition for the origin being a local minimum in Proposition 5.1 is satisfied: