ENTROPY INEQUALITY SPANS A PATH OF GIBBS STATES
Let X be a finite set. Let m : X –> R be a “probabilistic
measure” on points, i.e. let m(x) be non-negative for every x in X,
and let SUM(m) = Sum( m(x) | x in X) = 1 (geometrically speaking
m is a point of the simplex which has X as its set of vertices).
Then the Shannon entropy is, by definition,
s(m) = (-1) * Sum( m(x)*log(m(x)) | x in X )
= I(-log o m)
where I is the integral w.r. to m, and “o” is the symbol of the
composition of function.
Also 0*log(0) is assumed to be 0 (hence entropy s is a continuous
function of m).
Let u : X –> R be the constant function: u(x) = 1/|X|.
Then u is a probabilistic “measure” in X, we can call it uniform.
A well known elementary but fundamental inequality states:
s(m) <= s(u) = log|X|
for every “probabilistic measure” m in X.
We will prove this inequality by connecting measures m and u
by an entire path of measures in X for which entropy will
increase smoothly. The intermediate states (measures) will
be the Gibbs states obtained from m by increasing temperature
(so to speak). It’s only natural that higher temperature
leads to higher entropy (as we see it on this net all the time).
Gibbs states for finite systems
Consider a finite set X and an arbitrary real function
U : X –> R;
X is called the space of configurations, and U is called
the energy function. We also define the “partition function”
Z = Z(T), where T > 0 is called temperature:
Z = Sum( exp(-b*U(x)) | x in X )
where b = 1/T (sorry for that :-).
Finally, the Gibbs state m_T at temperature T is a “measure”
given by formula:
m_T = (exp o (-b*U)) / Z
m_T(x) = exp(-b*U(x)) / Z for every x in X.
Obviously, SUM(m_T) = 1 hence m_T is probabilistic.
When temperature rises then m_T converges to the uniform
lim(m_T(x) | T –> infinity) = 1/|X|.
For the sake of completeness let’s add that:
| 0 if U(x) is not minimal
lim(m_T(x) | T –> +0) = <
| 1/|X_min| if U(x) is minimal
where X_min is the set of all x in X for which
energy U(x) is minimal in X.
given f,g: X –> R let
<f,g> = Sum( f(x)*g(x) | x in X)
I_T(f) = <f,m_T>
(hence, for entropy, s(m_T) = -I_T(log o m_T).
It’s easy to see that:
dZ/db = – < U, exp(-b*U) >
d(logZ)/db = – I_T(U)
dm_T/db = – (U – I_T(U)) * m_T
Furthermore, the Shannon entropy of a Gibbs state is easily
seen to be:
s(m_T) = log(Z) + b*I_T(U)
Thus it’s once again easy (apply Lagrange inequality) that:
“MAIN THEOREM” : ds(m_T)/db = b*((I_T(U))^2 – I_T(U^2)) <= 0
for every T>0.
Corollary 1. d(s(m_T))/dT >= 0 for every T>0.
Corollary 2. s(m_Q) <= s(m_T) whenever 0 < Q <= T.
Corollary 3. log(|X_min|) <= s(m_T) <= log(|X|) for every T>0.
Shannon entropy for general finite probabilistic spaces
Let n : X –> R be an arbitrary positive “measure”, i.e.
let n(x) > 0 for every x in X, and let SUM(n) = 1.
Define U : X –> R as follows:
U(x) = -log(n(x)) for every x in X.
Then the “measure” n and the Gibbs state m_1 (at
temperature 1) coincide: m_1 = n. Thus we immediately
obtain from Corollary 3 above that:
THEOREM. s(n) <= log(|X|) for every “probabilistic measure”
n in X.
(For all of them, not just for the strictly positive ones.)
An attempt to demystify Gibbs States
It is enlighting and interesting to have the notions of physics
help us to do pure mathematics. Such things happen at least
since Archimedes. We get more motivated and get a better understanding.
Or do we? Would the language of physics obscure our, simple after all,
inequality about entropy.
Indeed, we do not need physics to connect any given “probabilistic
measure” with the uniform one by a smooth path. Let
Z_T = SUM( m^b ) = Sum( (m(x))^b | x in X )
for every T>0 and b = 1/T. Then m_T : X –> R defined by:
m_T = m^b/Z_T
is a “probabilistic measure”. Obviously, m_1 = m and lim m_T
is the uniform measure when T –> infinity.
No Gibbs states. Actually these m_T measures are the same
Gibbs states as earlier but we don’t need to know it. Hence so far
so pure, as an “ultramathematician” would say.
Now let’s try again to prove that the entropy s(m_T) increases
when the temperature T does:
s(m_T) = log(Z_T) – b * SUM( m_T * (log o m) )
= log(Z_T) – b * I_T(log o m)
etc. We can continue just as we did earlier. But if we don’t
recognize the role played by U = -log o m then, well, then
we will understand less than before, e.g. our simple proof will
look ad hoc and like a mess. Hence:
CONCLUSION. The formalism of the Gibbs states adds to our
understanding of the notion of entropy in general (outside
of physics, in the realm of pure mathematics).
Enjoy (or hate 🙂