Section: New Results

Ergodic theory for controlled Markov chains with stationary inputs

Consider a stochastic process $𝐗$ on a finite state space $X=\{1,\cdots ,d\}$. It is conditionally Markov, given a real-valued `input process' $\zeta$. This is assumed to be small, which is modeled through the scaling, ${\zeta }_t=\epsilon {\zeta }_t^1,0\le \epsilon \le 1,$ where ${\zeta }^1$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\zeta$:

• A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain ${𝐗}^{•}$ obtained with $\zeta \equiv 0$. The triple $(𝐗,{𝐗}^{•},\zeta )$ is a jointly stationary process satisfying $𝖯\{X\left(t\right)\ne X^{•}\left(t\right)\}=O\left(\epsilon \right).$ Moreover, a second-order Taylor-series approximation is obtained:

  $𝖯\{X\left(t\right)=i\}=𝖯\{X^{•}\left(t\right)=i\}+{\epsilon }^2\varrho \left(i\right)+o\left({\epsilon }^2\right),1\le i\le d,$

  with an explicit formula for the vector $\varrho \in {\Re }^d$.

• For any $m\ge 1$ and any function $f:\{1,\cdots ,d\}\times \Re \to {\Re }^m$, the stationary stochastic process $Y\left(t\right)=f\left(X\right(t),\zeta (t\left)\right)$ has a power spectral density ${\text{S}}_f$ that admits a second order Taylor series expansion: A function ${{\text{S}}^{\text{(2)}}}_f:[-\pi ,\pi ]\to C^{m\times m}$ is constructed such that

  ${\text{S}}_f\left(\theta \right)={\text{S}}_f^{•}\left(\theta \right)+{\epsilon }^2{{\text{S}}^{\text{(2)}}}_f\left(\theta \right)+o\left({\epsilon }^2\right),\theta \in [-\pi ,\pi ].$

  An explicit formula for the function ${{\text{S}}^{\text{(2)}}}_f$ is obtained, based in part on the bounds in (i).

The results are illustrated using a version of the timing channel of Anantharam and Verdu.