# Entropy inequality via Gibbs states

· blog
Authors

Wlodzimierz Holsztynski

ENTROPY INEQUALITY SPANS A PATH OF GIBBS STATES

Let        X  be a finite set.  Let  m : X –> R  be a “probabilistic
measure” on points, i.e. let  m(x)  be non-negative for every  x  in X,
and let  SUM(m) = Sum( m(x) | x in X) = 1  (geometrically speaking
m  is a point of the simplex which has  X  as its set of vertices).
Then the Shannon entropy is, by definition,

s(m)  =  (-1) * Sum( m(x)*log(m(x)) | x in X )

=  I(-log o m)

where  I  is the integral w.r. to m,  and  “o”  is the symbol of the
composition of function.

Also  0*log(0)  is assumed to be 0        (hence entropy        s  is a continuous
function of        m).

Let        u : X –> R be the constant function:  u(x) = 1/|X|.
Then         u  is a probabilistic “measure” in X, we can call it uniform.
A well known elementary but fundamental inequality states:

s(m) <= s(u) = log|X|

for every “probabilistic measure”  m  in X.

We will prove this inequality by connecting measures  m  and  u
by an entire path of measures in  X  for which entropy will
increase smoothly.  The intermediate states (measures) will
be the Gibbs states obtained from  m  by increasing temperature
(so to speak).  It’s only natural that higher temperature
leads to higher entropy  (as we see it on this net all the time).

Gibbs states for finite systems
——————————-

Consider a finite set  X  and an arbitrary real function

U : X –> R;

X  is called the space of configurations, and  U  is called
the energy function.  We also define the “partition function”
Z = Z(T),  where  T > 0  is called temperature:

Z  =  Sum( exp(-b*U(x)) | x in X )

where  b = 1/T  (sorry for that :-).

Finally, the Gibbs state  m_T  at temperature  T  is a “measure”
given by formula:

m_T  =        (exp o (-b*U)) / Z

i.e.

m_T(x)  =        exp(-b*U(x)) / Z               for every  x in X.

Obviously,  SUM(m_T) = 1  hence  m_T  is probabilistic.

When temperature rises then        m_T  converges to the uniform
state  u:

lim(m_T(x) |  T –> infinity)  =  1/|X|.

For the sake of completeness let’s add that:
_
|        0  if  U(x) is not minimal
lim(m_T(x) | T –> +0)        =  <
|        1/|X_min|  if  U(x) is minimal
–

where  X_min is the set of all  x  in  X  for which
energy  U(x)  is minimal in X.

*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*

Notation:

given  f,g: X –> R  let

<f,g>  =  Sum( f(x)*g(x) | x in X)

and

I_T(f) =  <f,m_T>

(hence, for entropy,  s(m_T) = -I_T(log o m_T).

*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*-=-=-=-=-*

It’s easy to see that:

dZ/db           =  – < U, exp(-b*U) >

d(logZ)/db = – I_T(U)

dm_T/db    = – (U – I_T(U)) * m_T

Furthermore, the Shannon entropy of a Gibbs state is easily
seen to be:

s(m_T)        =  log(Z) + b*I_T(U)

Thus it’s once again easy (apply Lagrange inequality) that:

“MAIN THEOREM” :   ds(m_T)/db  =  b*((I_T(U))^2 – I_T(U^2)) <= 0
for every  T>0.

Corollary 1.  d(s(m_T))/dT >= 0   for every        T>0.

Corollary 2.  s(m_Q) <= s(m_T)   whenever  0 < Q <= T.

Corollary 3.  log(|X_min|) <= s(m_T) <= log(|X|)  for every        T>0.

Shannon entropy for general finite probabilistic spaces
——————————————————-

Let        n : X –> R be an arbitrary positive “measure”, i.e.
let        n(x) > 0  for every  x        in  X,        and let  SUM(n) = 1.
Define  U : X –> R        as follows:

U(x) = -log(n(x))    for every        x in X.

Then  the “measure”  n  and the Gibbs state  m_1  (at
temperature 1) coincide:  m_1 = n.        Thus we immediately
obtain from        Corollary 3  above that:

THEOREM.  s(n) <= log(|X|)  for every “probabilistic measure”
n  in  X.

(For all of them, not just for the strictly positive ones.)

An attempt to demystify Gibbs States
————————————

It is enlighting and interesting to have the notions of physics
help us to do pure mathematics.  Such things happen at least
since Archimedes.  We get more motivated and get a better understanding.
Or do we?  Would the language of physics obscure our, simple after all,
inequality about entropy.

Indeed, we do not need physics to connect any given “probabilistic
measure” with the uniform one by a smooth path.  Let

Z_T =  SUM( m^b )  =  Sum( (m(x))^b | x in X )

for every  T>0  and  b = 1/T.  Then  m_T : X –> R  defined by:

m_T  =  m^b/Z_T

is a “probabilistic measure”.  Obviously,  m_1 = m  and  lim m_T
is the uniform measure when  T –> infinity.

No Gibbs states.  Actually these  m_T  measures are the same
Gibbs states as earlier but we don’t need to know it.  Hence so far
so pure,  as an “ultramathematician” would say.

Now let’s try again to prove that the entropy  s(m_T)  increases
when the temperature  T  does:

s(m_T)  =  log(Z_T)  –  b * SUM( m_T * (log o m) )

=  log(Z_T)  –  b * I_T(log o m)

etc.  We can continue just as we did earlier.  But if we don’t
recognize the role played by   U = -log o m   then, well, then
we will understand less than before,  e.g.  our simple proof will
look  ad hoc  and  like a mess.  Hence:

CONCLUSION. The formalism of the Gibbs states adds to our
understanding of the notion of entropy in general (outside
of physics, in the realm of pure mathematics).

Enjoy (or hate 🙂

Wlodek

Advertisements