
Gibbs Sampler (Gibbs)
Acceptance Rate | The acceptance rate is 1. |
Applications | This is a widely applicable, general-purpose algorithm that generally requires conjugacy. |
Difficulty | This algorithm is difficult for a beginner when the conditional distribution must be specified. Otherwise, it is fully automatic. |
Final Algorithm? | Yes. |
Proposal | Componentwise. |
Gibbs sampling was introduced by Turchin (1971), and later by brothers Geman and Geman (1984). The Geman brothers named the algorithm after the physicist J. W. Gibbs, some eight decades after his death, in reference to an analogy between the sampling algorithm and statistical physics. Geman and Geman introduced Gibbs sampling in the context of image restoration.
Gibbs has two algorithm specifications: FC
accepts a user-specified function to calculate full-conditionals and MWG
defaults to NULL
and otherwise accepts a numeric vector that indicates which parameters are updated with Metropolis-within-Gibbs. FC
accepts two arguments, parm
and Data
, just like the Model specification function, and returns the full vector of parameters parm
. Each iteration, the full-conditional distributions are completed first, then MWG updates, if any.
In its basic version, Gibbs sampling is a special case of the Metropolis-Hastings algorithm. However, in its extended versions, Gibbs sampling can be considered a general framework for sampling from a large set of variables by sampling each variable (or in some cases, each group of variables) in turn, and can incorporate the Metropolis-Hastings algorithm (such as Metropolis-within-Gibbs or similar methods such as Slice sampling) to implement one or more of the sampling steps. Currently, Metropolis-within-Gibbs is included.
Gibbs sampling is applicable when the joint distribution is not known explicitly or is difficult to sample from directly, but the conditional distribution of each variable is known and easy to sample from. A Gibbs sampler generates a draw from the distribution of each parameter or variable in turn, conditional on the current values of the other parameters or variables. Therefore, a Gibbs sampler is a componentwise algorithm. As a simplified example, given a joint probability distribution p(θ1, θ2 | y), a Gibbs sampler would draw p(θ1 | θ2, y), then p(θ2 | θ1, y).
For a user to determine each conditional distribution, the joint distribution must be known first, then all parameters are held constant except the current parameter of interest, and the equation is simplified to the conditional distribution.
There are numerous variations of Gibbs sampling, such as blocked Gibbs sampling, collapsed Gibbs sampling, and ordered overrelaxation.
The advantage of Gibbs sampling to other MCMC algorithms is that it is often more efficient when it is appropriate, due to a 100% acceptance rate. The disadvantages are that a Gibbs sampler is appropriate only with conjugate distributions and low correlations between parameters, and therefore Gibbs sampling is less generally applicable than other MCMC algorithms. Since Gibbs is not adaptive, it is suitable as a final algorithm.
References
- Geman S and Geman D (1984). "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images." IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6), 721-741.
- Turchin, VF (1971). "On the Computation of Multidimensional Integrals by the Monte Carlo Method", Theory of Probablility and its Applications, 16(4), 720-724.



