The Truncated Exponential Distribution (TED)

To apply the method of Section 5.3.2 to the doubly-bounded case, we need to choose the right surrogate distribution. A clue to the proper surrogate density can be found by observing the marginal distribution of an element of the input data for UMS samples as $N$ becomes large, as we saw in Figures 4.1, and 5.7. Mathematically, the surrogate density should also be the maximum entropy density under the applicable moment constraints. This was true of the Gaussian and exponential densities corresponding to Figures 4.1, and 5.7, respectively. To get a clue, we repeated the experiment of Figure 5.7, but with the double bound on the input data. Figure 7.1 shows the result, in which the marginal of $x_2$ has a shape that could be approximated by a truncated exponential distribution (with positive exponent).

Figure: Left: manifold sampling results using rejection sampling for $N=8$. Right, histogram of $x_2$.

In fact, for data bounded to the interval $[0,1]$, the truncated exponential distribution is the maximum entropy distribution under mean (first moment) constraints ([48], page 186). Let's take a closer look at this distribution. For data $x$ in the interval $[0,1]$ and exponent parameter $\alpha$ (which can be positive or negative), the uni-variate truncated exponential distribution is

$$p(x; \alpha) = C_\alpha \; e^{\alpha x}, \; \; 0\leq x \leq 1,$$ (7.2)

where $C_\alpha = \left( \frac{\alpha}{e^{\alpha} - 1}\right).$ The mean of this density is given by

$$\lambda(\alpha) = \int_{0}^1 \; x\; p(x; \alpha) \; {\rm d} x = \frac{e^{\alpha}}{e^{\alpha} - 1}-\frac{1}{\alpha}$$ (7.3)

and the entropy is given by

$$H_{db}(\alpha) = -\log \left( \frac{\alpha}{e^{\alpha } - 1}\right) -\alpha \lambda(\alpha),$$ (7.4)

where “db" indicates the doubly-bounded case. The multi-variate TED is

$$\log p({\bf x}) = \sum_{i=1}^N \log C_{\alpha_i} + \alpha_i x_i.$$ (7.5)

Thus, the entropy of the multivariate density of $N$ independent truncated exponentials is therefore, for input data constrained to the unit interval $[0,1]$ is

$$H_{db}( \mbox{\boldmath$\alpha$} ) = \sum_{i=1}^N \left\{ -\log \left( \frac{\alpha_i}{e^{\alpha_i} - 1}\right) -\alpha_i \lambda_i\right\},$$ (7.6)

where $\alpha$$=[\alpha_1, \alpha_2 \ldots \alpha_N]$ and $\lambda_i$ is computed from $\alpha_i$ using (7.3).