Introduction to coding theory (CMU, Spr 2010)

April 24, 2010

Lecture 24 summary

Filed under: Lecture summary — Venkat Guruswami @ 10:08 pm

We discussed the message passing algorithm for decoding LDPC codes based on (d_v,d_c)-regular graphs on the binary erasure channel, and derived an expression for threshold erasure probability for which the algorithm guarantees vanishing bit error probability. We then turned to the binary symmetric channel, and discussed Gallager’s “Algorithm A” and derived the recurrence equation for the decay of the bit error probability. We briefly discussed Gallager’s “Algorithm B” as well, where a variable node flips its value if more than a certain cut-off number (typically majority after a few iterations) of its neighboring check nodes suggest that the node flips its value. We mentioned the values of the threshold crossover probability for some small values of d_v and d_c.

During lecture, the question of the speed of convergence of the bit error probability (BER) to zero was asked. The answer I guessed turns out to be correct: if we run the algorithm for \Omega(\log n) iterations which is smaller than the girth of the graph, for Algorithm A the BER is at most 1/n^{\beta} for some \beta > 0, and for Algorithm B for d_v > 3 with an optimized cut-off for flipping, the BER is at most 2^{-n^{\gamma}} for some \gamma > 0.

We do not plan to have notes for this segment of the course.¬† I can, however, point you to an introductory survey I wrote (upon which the lectures are loosely based), or Gallager’s remarkable Ph.D. thesis which can be downloaded here (the decoding algorithms we covered are discussed in Chapter 4). A thorough treatment of the latest developments in the subject of iterative and belief propagation decoding algorithms can be found in Richardson and Urbanke’s comprehensive book Modern Coding Theory.



  1. How is this related / is it related to maximum a posteriori inference in Markov Random Fields (there is also belief propagation algorithms there — though with no theoretical guarantees at all)?

    Comment by Ali Sinop — April 28, 2010 @ 8:15 am | Reply

    • I think these are all intimately related. Here is a good survey on Factor graphs and the sum-product algorithm and how this framework unifies similar algorithms that have been developed and used in several different application areas.

      Comment by Venkat Guruswami — April 28, 2010 @ 3:31 pm | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: