On Discrete Time Reversibility modulo State Renaming and its Applications

Time reversibility plays an important role in the analysis of continuous and discrete time Markov chains (DTMCs). Specifically, the computation of the stationary distribution of a reversible Markov chain has been proved to be very efficient and does not require the solution of the system of global balance equations. A DTMC is reversible when the processes at forward and reversed time are probabilistically indistinguishable. In this paper we introduce the concept of ρ-reversibility, i.e., a notion of reversibility modulo a renaming of the states, and we contrast it with the previous definition of dynamic reversibility especially with respect to the assumptions on the state renaming function. We discuss the applications of discrete time reversibility in the embedded and uniformized chains of continuous time processes.


INTRODUCTION
Markov chains have been widely used to study the performances of computer hardware and telecommunication systems as well as software architectures.Markov models can be designed at a very low-level by mapping each state of the process to a state of the system and then by setting the transition rates/probabilities.However, this method is often time consuming and prone to errors due to the complexity of the considered systems.For these reasons, in the past decades, several formalisms have been developed to allow one to specify a stochastic model in a relative compact way by using important features such as the compositionality and/or the hierarchical approach (see, e.g., [4,11]).Nevertheless, compact descriptions do not necessary allow for efficient analysis because the stochastic process underlying even a simple high-level model may have a very high state space cardinality [14].If we consider the steady-state analysis, the problem of deriving the stationary probability distribution requires to solve a set of linear equations (called global balance equations, GBE) that has an order of |S|, where S is the state space of the chain.
The analysis of Markov chains, both at continuous and discrete time, can sometimes exploit some properties of the stochastic process that allow a numerical or analytical tractability of some performance indices' computation.For instance matrix geometrics/analytics methods take advantage of repeated patterns in the Markov chain process description [12], while lumping [7,5] exploits the similarity in the stochastic behaviour of some states to reduce the state space.
In this paper we focus on the techniques based on the idea of reversible processes (see, e.g., [6,16,3,9,10]).A Markov chain X(t) is reversible when it is stochastically indistinguishable from the stochastic process at reversed time X(τ − t) for all τ .The computation of the stationary probabilities of reversible processes is extremely efficient since the system of GBE needs not to be solved and an explicit formula for each unnormalized state probability can be given in terms of the Markov chain's transition rates/probabilities.A remarkable example of reversible model is the set of M/M/1 queues with state dependent service rate.In this case the ratio between the stationary probabilities of state n > 0 and state 0 can be derived as the product of the transitions from 0 to n (λ n in case of constant arrival rate λ) divided by the product of the transition rates from n to 0 ( n i=1 µ(i), if µ(i) is the service rate in state i).
Contribution.In this paper we consider a notion of reversibility according to which a discrete time Markov chain X(t) and its reversed X(τ − t) are probabilistically indistinguishable modulo an arbitrary renaming ρ of the states (ρ-reversibility).We show that this class of models enjoys properties that are very similar to those of the standard reversibility [6] and hence the notion of ρ-reversibility allows for an efficient computation of the stationary distribution.Similarly to [6,16] and in contrast to [3], we show that we can decide this property solely on the analysis of the forward process without the need to derive X(τ − t).We discuss the properties of the renaming function ρ with attention to what has been previously done in the literature.Finally, we discuss the properties of reversibility modulo renaming both at continuous and discrete time and prove that a CTMC is ρ-reversible if and only if every corresponding uniformized chain is also ρ-reversible.This result does not hold for the embedded chain.We introduce the notion of almost ρ-reversible CTMC as a CTMC whose embedded chain is ρ-reversible and show that the conditions for almost ρreversibility are weaker than those required by the notion of ρ-reversibility at continuous time studied in [10].
Related work.In [6] the author deeply discusses the notion of time reversibility both for DTMCs and CTMCs, and several applications are illustrated.In particular the author is interested in a sound discussion of the so call product-form property of the Markov models.In [16,2] the authors introduce the notion of dynamic reversibility to study physical systems such as the growth and decrescence of two dimensional crystals.We discuss our contribution with respect to such work in Section 3.1.In [9] we introduced the definition of autoreversibility that allows one to exploit the symmetrical structures of a class of CTMCs to derive the steady-state probabilities in an efficient way.Similarly to what is proposed here, a relation between a state and its inverse is taken in consideration.The idea of using a general permutation of states for comparing forward and reversed processes at continuous time is discussed in [10].However, notice that the formulations of ρ-reversibility in continuous and discrete time differ for some peculiar aspects that we discuss in Section 4.
Structure of the paper.The paper is structured as follows.Section 2 introduces the notation and the definitions of reversible and reversed DTMC.Here, we also prove a new result to characterize reversed DTMCs.In Section 3 we introduce the notion of ρ-reversible DTMC and prove its properties.We also discuss the differences with the previous definition of dynamic reversibility.Section 4 discusses the properties of ρ-reversibility both at continuous and discrete time and states the relationships between a CTMC and the corresponding ρ-reversible uniformized and embedded discrete chains.Section 5 concludes the paper.

THEORETICAL BACKGROUND
In this section we review some aspects of the theory of discrete time Markov chains which will be required in the sequel (see [13] for more details).
Preliminaries.A stochastic process X(t) in discrete time is a sequence of random variables taking values in a countable state space S for t ∈ Z.If (X(t1), X(t2), . . ., X(tn) has the same distribution as (X(t1 + τ ), X(t2 + τ ), . . ., X(tn + τ ) for all t1, t2, . . ., tn, τ ∈ Z then the stochastic process X(t) is said to be stationary.A Discrete Time Markov Chain (DTMC) is a stochastic process X(t) in discrete time, i.e., with t ∈ Z, such that for all t1 < t2 < • • • tn < tn+1 the joint distribution of (X(t1), X(t2), . . ., X(tn), X(tn+1)) is such that In other words, for a Markov chain its past evolution until the present state does non influence the conditional (on both past and present states) probability distribution of future behaviour.A Markov chain is time homogeneous if the conditional probability P (X(t + 1) = j | X(t) = i) does not depend upon t, and we use the notation for i, j ∈ S. The value pij denotes the probability that the chain, whenever in state i, next makes a transition into state j, and is referred to as one-step transition probability.The square matrix P = (pij)i,j∈S is called one-step transistion matrix, and since when leaving state i the chain must move to one of the states j ∈ S, each row sums to one (e.g., forms a probability distribution): for each i ∈ S j∈S pij = 1.
The n-step transition probabilities of the chain are given by Henceforth, we assume the ergodicity of the DTMCs that we study.A chain satisfying all these assumptions admits a unique stationary distribution, that is the unique vector π of positive numbers πi with i ∈ S such that: Each πi can be interpreted as the average proportion of time spent by the chain X(t) in state i.A Markov chain with a stationary distribution is said to be in steady state.
Time reversibility.The analysis of an ergodic DTMC with stationary distribution can be greatly simplified if it satisfies the property that when the direction of time is reversed the behaviour of the chain remains the same.
Consider an ergodic DTMC in steady state, X(t) with t ∈ Z, defined on a state space S with transition probability matrix P and stationary distribution π, and suppose that starting at some time we trace the sequence of states going backwards in time.We denote by X R (t) the reversed process of X(t).It turns out that X R (t) is also a stationary CTMC.
A reversible process is stationary.For a stationary Markov chain there exist simple necessary and sufficient conditions for reversibility expressed in terms of the equilibrium distribution π and the transition probabilities pij.These conditions are given in Proposition 1 and are called the detailed balance equations; they should be contrasted with the equilibrium equations, which are sometimes called the global balance equations.
Proposition 1 (Detailed balance equations [6]).A stationary DTMC with state space S and transition probability matrix P is reversible if and only if there exists a collection of positive real numbers πi, i ∈ S, summing to unity that satisfy the detailed balance equations: πipij = πjpji for all i, j ∈ S. (2) In that case, πi is the stationary probability of state i.
Equation ( 2) can be interpreted as stating that, for all states i and j, the flow from state i to state j is equal to that from state j to state i.If we can find nonnegative numbers, summing to 1, which satisfy Equation ( 2), then it follows that the Markov chain is reversible and the numbers represent its stationary probabilities.This is so since if that are exactly the conditions in (1).Since the stationary probabilities πi are the unique solution of the above system of equations, it follows that xi = πi for all i ∈ S.
An important property of reversible DTMCs is the Kolmogorov's criterion which states that the reversibility of a Markov chain can be established directly from its transition probabilities.
Proposition 2 (Kolmogorov's criterion [13]).A stationary DTMC with state space S and transition probability matrix P is reversible if and only if starting in state i, any path back to i has the same probability as the reversed path, for all i ∈ S, i.e., for every finite sequence of states i1, i2, . . . in ∈ S, Note that this criterion does not require the knowledge of the stationary distribution.
Reversed process.The transition probabilities of the reversed process X R (t) of a DTMC X(t) can always be defined in terms of the stationary distribution of the process X(t) as stated by the following proposition.
Proposition 3 (Reversed probabilities [13]).Given a stationary DTMC X(t) with state space S, transition probability matrix P and stationary distribution π, the reversed process X R (t) is an ergodic DTMC with the same state space S, the same stationary distribution π and transition probability matrix P R defined by: Observe that we can replace in Equation ( 4) any non-trivial solution of the GBE.Roughly speaking, we can say that the knowledge of the reversed process' transition probabilities allows for an efficient computation of the invariant measure of the process and vice versa the latter allows for an efficient definition of the reversed process.The converse of Proposition 3 also holds.
Proposition 4 (Reversed process's equations [13]).Consider a stationary DTMC X(t) with state space S and transition probability matrix P. If we can find a transition probability matrix P R and a stationary distribution π such that πipij = πjp R ji (5) then P R is the transition probability matrix of the reversed process X R (t) and π is stationary distribution of both X(t) and X R (t).
The importance of this proposition is that we can sometimes guess at the nature of the reversed chain and then use the set of Equations ( 5) to obtain both the stationary probabilities and the p R ij .Following the lines of [3] developped for CTMC, we prove that the Kolmogorov's criteria can be generalized in order to encompass non-reversible DTMCs.The Discrete Time Generalized (DTG) Kolmogorov's criterion is stated below.
Proposition 5 (DTG-Kolmogorov's criterion).A stationary DTMC with state space S and transition probability matrix P has reversed chain with transition matrix P R if and only if for every finite sequence i1, i2, . . . in ∈ S, Proof.Consider finite sequence of states i1, i2, . . . in ∈ S. By Proposition 4, that, by simplifying, yields Consider an arbitrary state i0 ∈ S as a reference state and i ∈ S.
be a chain of one-step transitions in the forward process X(t).We prove: where Ci 0 ∈ R + .We prove that πi is well-defined.Indeed, By hypothesis, we have: Moreover, considering the one- From the previous two equations we obtain: Figure 1: A ρ-reversible discrete time Markov chain Hence: where Ci 0 ∈ R + , is well-defined.
In order to prove that this is the stationary probability of state i ∈ S we show that it satisfies the system of global balance equations for i.Indeed, πi = j∈S πjpji , which can be written as: By Proposition 4 this is equal to j∈S p R ij = 1 hence the equation above is an identity.Now let i, j ∈ S such that pji > 0. Then Hence, by Proposition 4, P R is the transition probability matrix of X R (t).

REVERSIBILITY UNDER STATE PERMU-TATION
Many stochastic chains are not reversible, however they may be reversible under some state permutation.In this section we generalize the notion of reversibility for DTMC and introduce a novel notion named ρ-reversibility.
Hereafter, a renaming ρ over the state space of a Markov chain is a bijection on S. For a Markov chain X(t) with state space S we denote by ρ(X)(t) the same process where the state names are changed according to ρ.More formally, let P and π be the transition probability matrix and the stationary distribution of X(t), P and π be the transition probability matrix and the stationary distribution of ρ(X)(t).It holds that, for all i, j ∈ S, pij = p ρ(i)ρ(j) and πi = π ρ(i) .
We introduce the notion of ρ-reversibility.
Since, X(t) and X R (t) have the same stationary distribution π, it follows that if X(t) is ρ-reversible then πi = π ρ(i) for all i ∈ S.
Example 1.Consider the DTMC X(t) depicted in Figure 1 and the corresponding reversed chain depicted in Figure 2. It is easy to see that X(t) is ρ-reversible with the renaming ρ defined by: Indeed, one can easly see that X(t) and ρ(X R )(t) are probabilistically indistinguishable.
Example 2. Consider the DTMC X(t) depicted in Figure 3.One can easly prove that X(t) is ρ-reversible under the renaming ρ defined by: Analogously to Proposition 1 above, we prove necessary and sufficient conditions for ρ-reversibilty based on the existence of the solution of the linear system called ρ-detailed balance equations.The proof follows the lines in [6].

Proposition 6 (ρ-detailed balance equations).
Let X(t) be an ergodic DTMC with state space S and transition probability matrix P. Let ρ be a renaming on S. X(t) is ρ-reversible if and only if there exists a collection of positive numbers πi, i ∈ S, summing to unity that satisfy the following system of ρ-detailed balance equations: If such a solution π exists then it is the stationary distribution of both X(t) and ρ(X R )(t) and πi = π ρ(i) for all i ∈ S.  Proof.Let P be the transition probability matrix of ρ(X R )(t).We prove that if X(t) is ρ-reversible then it satisfies the system of ρ-detailed balance equations (8) where π is the stationary distribution of X(t).Indeed, observe that by Definition 2 of ρ-reversibility, p ρ(j)ρ(i) = p R ji .Moreover, since X(t) and ρ(X R )(t) are probabilistically identical we have p ρ(j)ρ(i) = p ρ(j)ρ(i) and hence, p R ji = p ρ(j)ρ(i) = p ρ(j)ρ(i) .Then, by Proposition 3 we obtain the desired result.Now suppose that there exists a collection of positive numbers πi, i ∈ S, summing to unity that satisfy the system of ρ-detailed balance equations.Then π is the stationary distribution of X(t).This is proved by substituting the definition if πi given by Equation (8) in the system of global balance equations of X(t).Hence, we have: that divided by πi gives: Since is a bijection, it this reduces to the identity 1 = 1.
Let us now prove that the ρ-detailed balance equations imply the fact that X(t) is ρ-reversible.In fact, by Proposition 3 i.e., X(t) and ρ(X R )(t) are probabilistically identical.
Corollary 1.Let X(t) be an ergodic DTMC with state space S, transition probability matrix P and stationary distribution π.Let ρ be a renaming on the state space S. If the transition probabilities of X(t) satisfy the following system of equations: By applying the Kolmogorov's criterion we obtain the following characterization of ρ-reversibility.Proposition 7. Let X(t) be an ergodic DTMC with state space S and transition probability matrix P, and ρ be a renaming on S. X(t) is ρ-reversible if and only if for every finite sequence i1, i2, . . . in ∈ S, Proof.Consider finite sequence of states i1, i2, . . . in ∈ S. By Proposition 6, that, by simplifying, yields .
Conversely, observe that, since X(t) is irreducible, for all j, k ∈ S we can find a chain of one-step transitions.From the hypothesis that X(t) and ρ(X R )(t) are probabilistically identical, there is also a chain Consider an arbitrary state i0 ∈ S as a reference state and i ∈ S.
be two chains of one-step transitions in X(t).We prove that: where Ci 0 ∈ R + .We prove that πi is well-defined.Indeed, By hypothesis, we have: Moreover, considering the one- From the previous two equations we obtain: Hence: where Ci 0 ∈ R + , is well-defined.
In order to prove that this is the stationary probability of state i ∈ S we show that it satisfies the system of global balance equations for i.Indeed, πi = j∈S πjpji , which can be written as: that is an identity.Now let i, j ∈ S such that pji > 0. Then Hence, by Proposition 6, X(t) is ρ-reversible.
The stationary probability distribution of a ρ-reversible DTMC can be computed as follows.
Proposition 8. Let X(t) be an ergodic DTMC with state space S and transition probability matrix P, ρ be a renaming on S, and i0, i1, i2, . . . in = i ∈ S be a finite sequence of states.If X(t) is ρ-reversible then for all i ∈ S, where i0 ∈ S is an arbitrary reference state and Ci 0 ∈ R + .
Proof.See the proof of Proposition 7.

On the properties of function ρ
In this section we discuss some properties of the renaming function ρ for ρ-reversible DTMCs.In particular we show that a DTMC may be reversible modulo more than a single state renaming ρ.Recall that a permutation ρ on a set S admits a unique decomposition into cycles of different states: Cycles of length one are called fixed points and those of length two transpositions.If ρ only consists of transpositions and fixed points then it is called involution, i.e., for all i ∈ S we have ρ(ρ(i)) = i.
If a DTMC is ρ-reversible then it allows for a very efficient derivation of its stationary state probabilities.It is important to note that the existence of a bijection ρ with respect to which the process is ρ-reversible does not imply that ρ is unique.In case of multiple candidates for ρ the choice is arbitrary although it may be desirable to use the one whose cycles are longest in order to reduce the number of states to analyse (since all the states belonging to the same cycle share the same stationary probability) as shown by the following example.
Example 3. Let us consider the DTMC depicted in Figure 1 and notice that the renaming under which X(t) and ρ(X R )(t) are probabilistically identical is not unique.In this example X(t) is also ρ -reversible where ρ is defined by: The renaming ρ given in Example 1 in an involution, while ρ has longer cycles in its definition and thus shows immediately that the stationary distributions of states i k (resp.j k ) are identical for k = 1, . . ., 4.
In [16,2] the authors use the idea of dynamic reversibility to study physical systems by Markov chains.In these cases, the authors consider only involutions for the state renaming in the mapping between the forward and the reversed chain.So we think that understanding if using permutations instead of involutions effectively extends the class of tractable Markov chains is an intriguing problem.In many cases (see e.g., Examples 1 and 3), although a permutation of states may be found there exists also an involution and hence the benefit of using ρ with longer cycle are only computational.In Example 4 we present a ρ-reversible DTMC for which does not exist any involution ρ such that ρ (X R (t)) = X(t).As a consequence we show that using general permutations instead of simple involutions enlarges the class of models which are analysable with the theory of reversibility modulo state renaming.5. Observe that the forward and the reversed processes are identical with ρ defined by the cycles (1, 2, 3, 4); (5,6,7,8); (9); (10).Let us prove that there cannot exist ρ such that ρ is formed by cycles of length 1 or 2. Observe state 10, and notice that if ρ exists then ρ (10) = 10 since the residence time in the forward and reversed chain must be distributed identically and the self-loop with probability 5/6 characterises only state 10.
Observe the transition 10 → 5 in the forward DTMC.This implies two cases: either 1) ρ (5) = 6 ∧ ρ (6) = 5 or 2) ρ (5) = 8 and ρ (8) = 5.Assume case 1).Then we should find a node (which mimics node 1) in the forward process that has an incoming arc from 6 with probability 1/6 and from 5 with probability 1/3.It is easy to see that such a node does not exist.Assume case 2).Then, we should be able to find a node (which mimics node 4) that has an outgoing transition to 8 with probability 2/3 and to 5 with probability 1/3.It is easy to see that also this case is impossible.

FROM CTMC TO DTMC
In the previous section we have considered stochastic processes that are discrete in both time and space, and that satisfy the Markov property.Here, we consider Continuous Time Markov Chains (CTMCs) and study the relation between the notion of ρ-reversibility defined in this setting [10] and the one presented above for DTMCs.We recall that the two main approaches to transform a CTMC into a DTMC are the uniformisation and the definition of the embedded chain (see, e.g., [15]).A CTMC is fully characterised by its infinitesimal generator matrix Q where qij is the transition rate from state i to j for i = j while the diagonal elements qii = −qi are defined as the negative sum of the non-diagonal elements of each row.For ergodic chains, the steady-state distribution π * is the unique vector of positive numbers π * i with i ∈ S, summing to unit and satisfying system of the global balance equations: In [9,10] we have introduced the counterparts of Propositions 6 and 7 for CTMCs which are stated below (noting that further conditions on the residence times are required with respect to the discrete case).Proposition 9. Let X(t) be an ergodic CTMC and ρ be a renaming on its state space S. Then X(t) is ρ-reversible if and only if there exists a collection of positive real number πi summing to unity such that πiqij = πjq ρ(j)ρ(i) and qi = q ρ(i) for every state i.If such a collection πi exists, then it is the stationary distribution of X(t).
Proposition 10.Let X(t) be an ergodic CTMC and ρ be a bijection on its state space S. Then X(t) is ρ-reversible if and only if for every finite sequence i1, i2, . . ., in ∈ S, and qi = q ρ(i) for every state i ∈ S.
Let X(t) be a CTMC such that there exists a positive real number φ for which qi ≤ φ for all i ∈ S. We define the DTMC X U (t) with the same state space of X(t) by uniformisation, i.e., we consider ν = max{qi, i ∈ S} and define the transition probabilities of X U (t) as follows: The steady-state distribution of X U (t) and X(t) are the same and sometimes it may be more convenient to study the uniformised chain rather then that at continuous time.From the prospective of ρ-reversibility, the following result holds: Proposition 11.Let X(t) be an ergodic CTMC and let X U (t) be a DTMC obtained by uniformisation.Then X(t) is ρ-reversible if and only if X U (t) is ρ-reversible.
Proof.⇒) If X(t) is ρ-reversible then its stationary solution satisfies the detailed balance equation according to Proposition 9 for all state i = j.By definition of uniformisation, also the DTMC satisfies the corresponding equations for i = j.However, in the DTMC we have introduced new transitions, namely the self-loops, and we have to check that for each state we have: πipii = πip ρ(i)ρ(i) , which is true if and only if p ρ(i)ρ(i) = pii.This is true since, by Proposition 9, i and ρ(i) have the same residence time distribution.⇐) If X U (t) is ρ-reversible then the detailed balance equations of Proposition 6 hold for all the state transitions i → j.If i = j, clearly the detailed balance equations hold also for the CTMC.It remains to prove that the residence time distribution for state i and ρ(i) are the same.Indeed, let us take the transition i → i in the DTMC and observe that the corresponding detailed balance equation implies pii = p ρ(i)ρ(i) that, by the definition of uniformisation, implies qi = q ρ(i) in X(t).
Another possible way of analyzing a CTMC X(t) is throughout the corresponding embedded Markov chain X E (t).If we consider the Markov process only at the moments upon which the state of the system changes, and we number these instances 0, 1, 2, etc., then we get a DTMC.This Markov chain has the transition probabilities pij for i, j ∈ S as: and pii = 0.If π is the steady-state distribution of the DTMC one may derive the distribution π * of the corresponding CTMC as: In some cases, although the CTMC X(t) is not ρ-reversible, its embedded chain X E (t) is.In this case we say that X(t) is almost ρ-reversible.We can decide if X(t) is almost ρreversible using the following proposition.
Proposition 12.An ergodic CTMC X(t) is almost ρreversible if and only if for every finite sequence of states it holds that: Proof.The proof is trivial and follows from the definition of embedded DTMC and Proposition 7.
Notice that if a CTMC is ρ-reversible then it is also almost ρ-reversible but the opposite is not true as shown by Example 5.The stationary distribution π * of X(t) satisfies the following set of detailed balance equations: γ = λ + µ and according to the permutation defined as: Note that, under the assumption γ = λ + µ, for all the states s it holds qs = q (s) and, by exploiting the regularity of the process, we have to check the following cycles: Therefore, the CTMC is ρ-reversible and we have πn1 = πn2 for all n ≥ 1.However, notice that the CTMC is almost ρreversible without requiring the condition γ = λ + µ.We can therefore apply the detailed balance equations for almost ρreversible CTMCs to straightforwardly derive the invariant measure of the process (see e.g. for π12 and π11):

CONCLUSION
In this paper we have proposed a theory of reversibility modulo a renaming of states for discrete time Markov chains.We have shown that the class of ρ-reversible DTMCs enjoys a high numerical tractability since the unnormalized steadystate distribution can be derived as product of transition probabilities instead as solution of the system of linear equations GBEs.We have shown that in contrast to [16,2,9] the state renaming needs not to be an involution and moreover that assuming general bijections allows us to efficiently study models that cannot be analysed by involutions.Finally, we have used the results of ρ-reversibility for DTMC to give the definition of almost ρ-reversibility for CTMCs that allows us to enlarge the class of CTMCs that we are able to solve without computing the solution of the GBEs with respect to the previous class [10].
We leave as future work the investigation of the relations among ρ-reversibility and product-forms.In fact, if a CTMC is ρ-reversible then its steady-state distribution can be expressed as a ratio between two products of rates (see Proposition 8) and this clearly simplifies the task of obtaining a product-form solution for the model.Nevertheless, coherently with the product-form theory developed in the literature (see, e.g., [6,3,1,8]), the formulation of conditions on the isolated components is desirable so that one has not to construct the whole joint process.In other words, we are interested in finding sufficient conditions under which the composition of high level stochastic models (e.g., Markovian process algebra components) originates a joint model which is ρ-reversible for some renaming ρ.

xipij = xjpji and i∈S xi = 1 then
summing over i yields i∈S xipij = xj i∈S pji = xj and i∈S xi = 1

Figure 5 :
Figure 5: Reversed chain of the DTMC in Figure 4

Example 4 .
Let us consider the DTMC depicted by Figure 4 whose reversed is shown in Figure
whose inverse is itself and hence the conditions of Proposition 10 are satisfied.whose product of the rates satisfies the conditions of Proposition 10.
closed form expression of the unnormalized steady-state distribution can be readily derived for n > 0: n ∈ Z and i, j ∈ S, denoting the probability that a process in state i will be in state j after n additional transitions.A state j is said to be accessible from state i if p n ij > 0 for some n ≥ 0. Two states i and j accessible to each other are said to communicate and to belong to the same equivalence class.A Markov chain is said to be irreducible if there is only one equivalence class, i.e., if all states communicate with each other.State i is said to have a period d ≥ 1 if d is the greatest integer such that p n ii = 0 if n is not a multiple of d.A state with period 1 is said to be aperiodic.A Markov chain is said to be aperiodic if all states are aperiodic.The recurrence time Ti of a Markov chain is defined as Ti = min{n ≥ 1 | Xn = i given X0 = i} (notice that Ti is a random variable.)Astatei is said to be recurrent ifP (Ti < ∞) = 1; otherwise, itis called transient.The mean recurrence time Mi of state i is defined as Mi = E[Ti].A recurrent state i is called positive recurrent if Mi < ∞.A Markov chain is called positive recurrent if all of its states are positive recurrent.A time homogeneous DTMC is said ergodic if it is irreducible, aperiodic and positive recurrent.