Entropy inequality via Gibbs states

· blog

Wlodzimierz Holsztynski


   Let        X  be a finite set.  Let  m : X –> R  be a “probabilistic
   measure” on points, i.e. let  m(x)  be non-negative for every  x  in X,
   and let  SUM(m) = Sum( m(x) | x in X) = 1  (geometrically speaking
   m  is a point of the simplex which has  X  as its set of vertices).
   Then the Shannon entropy is, by definition,

        s(m)  =  (-1) * Sum( m(x)*log(m(x)) | x in X )

             =  I(-log o m)

   where  I  is the integral w.r. to m,  and  “o”  is the symbol of the
   composition of function.

   Also  0*log(0)  is assumed to be 0        (hence entropy        s  is a continuous
   function of        m).

   Let        u : X –> R be the constant function:  u(x) = 1/|X|.
   Then         u  is a probabilistic “measure” in X, we can call it uniform.
   A well known elementary but fundamental inequality states:

        s(m) <= s(u) = log|X|

   for every “probabilistic measure”  m  in X.

   We will prove this inequality by connecting measures  m  and  u
   by an entire path of measures in  X  for which entropy will
   increase smoothly.  The intermediate states (measures) will
   be the Gibbs states obtained from  m  by increasing temperature
   (so to speak).  It’s only natural that higher temperature
   leads to higher entropy  (as we see it on this net all the time).

   Gibbs states for finite systems

   Consider a finite set  X  and an arbitrary real function

                        U : X –> R;

   X  is called the space of configurations, and  U  is called
   the energy function.  We also define the “partition function”
   Z = Z(T),  where  T > 0  is called temperature:

        Z  =  Sum( exp(-b*U(x)) | x in X )

   where  b = 1/T  (sorry for that :-).

   Finally, the Gibbs state  m_T  at temperature  T  is a “measure”
   given by formula:

                m_T  =        (exp o (-b*U)) / Z


             m_T(x)  =        exp(-b*U(x)) / Z               for every  x in X.

   Obviously,  SUM(m_T) = 1  hence  m_T  is probabilistic.

   When temperature rises then        m_T  converges to the uniform
   state  u:

        lim(m_T(x) |  T –> infinity)  =  1/|X|.

   For the sake of completeness let’s add that:
                                    |        0  if  U(x) is not minimal
        lim(m_T(x) | T –> +0)        =  <
                                    |        1/|X_min|  if  U(x) is minimal

   where  X_min is the set of all  x  in  X  for which
   energy  U(x)  is minimal in X.



        given  f,g: X –> R  let

                <f,g>  =  Sum( f(x)*g(x) | x in X)


                I_T(f) =  <f,m_T>

   (hence, for entropy,  s(m_T) = -I_T(log o m_T).


   It’s easy to see that:

        dZ/db           =  – < U, exp(-b*U) >

        d(logZ)/db = – I_T(U)

        dm_T/db    = – (U – I_T(U)) * m_T

   Furthermore, the Shannon entropy of a Gibbs state is easily
   seen to be:

        s(m_T)        =  log(Z) + b*I_T(U)

   Thus it’s once again easy (apply Lagrange inequality) that:

   “MAIN THEOREM” :   ds(m_T)/db  =  b*((I_T(U))^2 – I_T(U^2)) <= 0
   for every  T>0.

   Corollary 1.  d(s(m_T))/dT >= 0   for every        T>0.

   Corollary 2.  s(m_Q) <= s(m_T)   whenever  0 < Q <= T.

   Corollary 3.  log(|X_min|) <= s(m_T) <= log(|X|)  for every        T>0.

   Shannon entropy for general finite probabilistic spaces

   Let        n : X –> R be an arbitrary positive “measure”, i.e.
   let        n(x) > 0  for every  x        in  X,        and let  SUM(n) = 1.
   Define  U : X –> R        as follows:

        U(x) = -log(n(x))    for every        x in X.

   Then  the “measure”  n  and the Gibbs state  m_1  (at
   temperature 1) coincide:  m_1 = n.        Thus we immediately
   obtain from        Corollary 3  above that:

   THEOREM.  s(n) <= log(|X|)  for every “probabilistic measure”
   n  in  X.

   (For all of them, not just for the strictly positive ones.)

   An attempt to demystify Gibbs States

   It is enlighting and interesting to have the notions of physics
   help us to do pure mathematics.  Such things happen at least
   since Archimedes.  We get more motivated and get a better understanding.
   Or do we?  Would the language of physics obscure our, simple after all,
   inequality about entropy.

   Indeed, we do not need physics to connect any given “probabilistic
   measure” with the uniform one by a smooth path.  Let

        Z_T =  SUM( m^b )  =  Sum( (m(x))^b | x in X )

   for every  T>0  and  b = 1/T.  Then  m_T : X –> R  defined by:

         m_T  =  m^b/Z_T

   is a “probabilistic measure”.  Obviously,  m_1 = m  and  lim m_T
   is the uniform measure when  T –> infinity.

   No Gibbs states.  Actually these  m_T  measures are the same
   Gibbs states as earlier but we don’t need to know it.  Hence so far
   so pure,  as an “ultramathematician” would say.

   Now let’s try again to prove that the entropy  s(m_T)  increases
   when the temperature  T  does:

      s(m_T)  =  log(Z_T)  –  b * SUM( m_T * (log o m) )

             =  log(Z_T)  –  b * I_T(log o m)

   etc.  We can continue just as we did earlier.  But if we don’t
   recognize the role played by   U = -log o m   then, well, then
   we will understand less than before,  e.g.  our simple proof will
   look  ad hoc  and  like a mess.  Hence:

   CONCLUSION. The formalism of the Gibbs states adds to our
   understanding of the notion of entropy in general (outside
   of physics, in the realm of pure mathematics).

Enjoy (or hate 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

  1. WhoWheWha (in progress)
  2. The sharpest minds ever
  3. California – State – USA – Knol Authors – Knols – Education
  4. Money — economy (part 1)
  5. “california rains”
  6. Pairs, Kuratowski pairs, and reality
  7. Number Theory — congruences
  8. Matematyka – esej
  9. Congruence x^2 ≡ -1 mod p (Euler), and a super-Wilson theorem.
  10. Dabanese – dictionary (1)
  11. Dabanese syntax
  12. Dabanese — a general introduction
  13. Tentative dabanese dictionary (1)
  14. Fractions and diophantine approximations, I
  15. Mathematics — index
  16. Reflections on mathematics (migma)
  17. Special characters & HTML strings
  18. Infinitude of primes – 1
  19. Reference
  20. Geometric progression
  21. Ergänzungssatz: x^2 ≡ 2 mod p; and more.
  22. Primes in arithmetic progressions (part I)
  23. Euler introduction to Gauss quadratic reciprocity
  24. Trigonometry of a triangle–sinus & cosinus
  25. Wlodzimierz Holsztynski, knolog
  26. Tichonov product
  27. Factorization in semigroups
  28. Nowe knole
  29. Knole w języku polskim
  30. Metric spaces — introduction
  31. Topology — compact spaces II
  32. Topology–singular spaces
  33. the last summer concert in san jose
  34. California in poetry
  35. threeway
  36. san jose blues
  37. san francisco blues
  38. open your arcs
  39. Tom Wachtel, [I was running out of …]
  40. Tom Wachtel, [on the sand at…]
  41. “september”
  42. Topological sequences and convergence
  43. Topology–Arkhangelskii nets
  44. Topology–compact spaces I
  45. Topological product of two spaces. Hausdorff spaces.
  46. Trigonometry
  47. Total logarithmic series
  48. “affinity”
  49. [close your eyes…]
  50. “San Jose”
  51. willow girl
  52. heaven california
  53. “phase transitions”
  54. “california?”
  55. “dimensions”
  56. bachelor life
  57. “spring in california”
  58. [day -]
  59. Singular product of two spaces. Double limit.
  60. Art of Agreeing — painless tax
  61. The Birkhoff lattice of topologies
  62. Left topologies
  63. Topology — Kolmogorov axiom
  64. Topological subbases and bases
  65. Complexity of sorting
  66. ∞-Metrics
  67. Even perfect numbers and Mersenne primes
  68. Fermat sequence base b
  69. Harmonic series & Euler’s gamma
  70. 4-Baroque numbers with utmost 4 different prime divisors
  71. 3-Baroque numbers with utmost 3 different prime divisors
  72. Mathematical notation
  73. Baroque numbers – 2
  74. Number theory — ideals in Z, and the greatest common divisor
  75. Baroque numbers – 1
  76. Infinitude of primes – 2
  77. Number theory — Gothic numbers
  78. log and exp–a constructive and an axiomatic approaches
  79. Aleksandrov 2-point space
  80. Topology — short-short introduction
  81. Art of Agreeing — patent law
  82. Topology — the closure operation
  83. Integration of monotone functions
  84. The ground level properties of integral
  85. Metric spaces universal for 2-point spaces
  86. Products of bounded primes
  87. Sequences of pairwise coprime integers
  88. Mathematics — right triangles
  89. Art of Agreeing — United Nations
  90. Mathematics — two definitions
  91. Government wa(steful wa)ys
  92. Art of Agreeing — index
  93. Metric universality of (R^n d_m) — part 2
  94. Iso-graphs of metric maps into R (part 1)
  95. Metric universality of (R^n d_m) — part 1
  96. Metric spaces universal for 3- and 4-point spaces
  97. Number theory–units, composites, and primes
  98. Social life & energy saving
  99. Topological spaces and continuous mappings. Isolated points.
  100. Topological weight
  101. Mathematics — Euclid-Heron area of a triangle
  102. Connected spaces
  103. Art of Agreeing — marriage versus law & government
  104. Art of Agreeing — introduction
  105. Dabanese — index
  106. Linear orders in topological spaces
  107. Topological cuts and miscuts
  108. Closed sets. T1-spaces.
  109. Topology — the interior operation
  110. Mathematics — triangles
  111. Continuity of the piecewise continuous functions
  112. Topological subspaces
  113. Art of Agreement — medical insurance
  114. Art of Agreeing — business & stock market
%d bloggers like this: