Introduction to coding theory (CMU, Spr 2010)

Wrap up

Venkat Guruswami — Mon, 03 May 2010 23:07:13 +0000

Thanks once again to all of you who took the course and hung in there for the semester! I certainly had a great time with the course, and we covered a lot of ground despite the few Friday classes that had to be cancelled.

I have entered the grades on the electronic web form and I think you should be able to access it soon.

If you have not done so already, please fill in the course evaluation at my.cmu.edu under Academics. You probably also got an email with instructions on where to fill this out.

There was some request for notes on the 3-query LDCs we covered in the final lecture. I may not be able to write full blown notes, but I’ll try to expand the lecture summary for that post with some details of the construction and analysis when I find time.

Lecture 26 summary

Venkat Guruswami — Fri, 30 Apr 2010 22:17:04 +0000

We mentioned two topics which were introduced to coding theory by theoretical computer science: local testing and local decoding of codes. These and related topics (such as PCPs and applications of locally decodable codes in complexity and cryptography) have been intensively researched in the last 10-15 years, with several breakthroughs occurring in recent years.

We focused on local (unique) decoding of codes for the lecture. We saw how Hadamard codes can be locally decoded using just two queries. However, their encoding length for a message of length is . We then saw the higher degree generalization of Hadamard codes, where the message is interpreted as a degree homogeneous multilinear polynomial (i.e., all terms have degree exactly ). This gave us codes of encoding length , and we discussed a -query local decoding algorithm. This was based on interpolating the restriction of the multilinear polynomial on a line in a random direction. Thus for any constant , we got codes that are locally decodable using queries that have encoding length .

We then turned to the ingenious 3-query locally decodable code (LDC) construction due to Yekhanin. In keeping with the theme of our initial constructions, we presented a polynomial view of these codes, where the messages are again interpreted as homegeneous multilinear polynomials of certain degree (say ) but only a carefully chosen subset of all possible monomials are allowed. (This actually reduces the rate compared to our earlier construction, but the big gain is that one is able to locally decode using only three queries instead of about queries!) Our description is based on a variant of Yekhanin’s construction that was discovered by Raghavendra and subsequently presented by Gopalan as polynomial based codes.

For every such that is prime (such a prime is called a Mersenne prime), we gave a construction of -query LDCs of encoding length . Since very large Mersenne primes are known, we get 3-query LDCs of encoding length less than . We presented a 3-query algorithm and proved its correctness assuming the stated properties of the “matching sets” used in the construction, and then explained how to construct families of such subsets of of size .

Notes on list decoding folded RS codes

Venkat Guruswami — Fri, 30 Apr 2010 21:43:45 +0000

Notes for the lectures on achieving the optimal trade-off between rate and list decoding radius via folded Reed-Solomon codes are now posted on the course webpage. Notes 7,8 on Reed-Solomon unique decoding, GMD decoding, and expander codes have also been edited.

Lecture 25 summary

Venkat Guruswami — Thu, 29 Apr 2010 01:51:48 +0000

We discussed irregular LDPC codes, and characterized their rate and erasure correction capability (via the message passing algorithm discussed in the previous lecture) in terms of the degree distribution of the edges. Specifically, let (resp. ) is the fraction of edges incident on degree variable (resp. check) nodes, an define the generating functions and . Then the rate of the LDPC code is given by

Also if for every , and some constant 0' title='\alpha > 0' class='latex' />, we argued why the message passing algorithm succeeds with high probability on for any constant .

We then argued how the distributions

and

(perhaps truncated to a finite series) enables achieving capacity of — we can achieve a rate with decoding complexity (since the average variable node degree is ).

This result is from the paper Efficient erasure correcting codes. Further details, including extensions to BSC and AWGN channels, and the martingale argument for the concentration of the performance around that of the average code in the ensemble, can be found in the paper The capacity of low-density parity-check codes under message-passing decoding.

The last quarter of the lecture was devoted a recap of the main topics covered in the course.

Lecture 24 summary

Venkat Guruswami — Sun, 25 Apr 2010 03:08:11 +0000

We discussed the message passing algorithm for decoding LDPC codes based on -regular graphs on the binary erasure channel, and derived an expression for threshold erasure probability for which the algorithm guarantees vanishing bit error probability. We then turned to the binary symmetric channel, and discussed Gallager’s “Algorithm A” and derived the recurrence equation for the decay of the bit error probability. We briefly discussed Gallager’s “Algorithm B” as well, where a variable node flips its value if more than a certain cut-off number (typically majority after a few iterations) of its neighboring check nodes suggest that the node flips its value. We mentioned the values of the threshold crossover probability for some small values of and .

During lecture, the question of the speed of convergence of the bit error probability (BER) to zero was asked. The answer I guessed turns out to be correct: if we run the algorithm for iterations which is smaller than the girth of the graph, for Algorithm A the BER is at most for some 0' title='\beta > 0' class='latex' />, and for Algorithm B for 3' title='d_v > 3' class='latex' /> with an optimized cut-off for flipping, the BER is at most for some 0' title='\gamma > 0' class='latex' />.

We do not plan to have notes for this segment of the course. I can, however, point you to an introductory survey I wrote (upon which the lectures are loosely based), or Gallager’s remarkable Ph.D. thesis which can be downloaded here (the decoding algorithms we covered are discussed in Chapter 4). A thorough treatment of the latest developments in the subject of iterative and belief propagation decoding algorithms can be found in Richardson and Urbanke’s comprehensive book Modern Coding Theory.

List-decodability of random linear codes

Venkat Guruswami — Thu, 22 Apr 2010 14:46:55 +0000

In our discussion on random coding arguments to show the existence of list-decodable codes, we showed that a random -ary code of rate was -list decodable w.h.p. For random linear codes over , the result (or rather proof) was weaker, and only guaranteed a rate of . I had mentioned that this discrepancy in list-size between linear and general codes was recently resolved, showing that a random linear code of rate is -list decodable w.h.p as well.

I’ll speak about this result in the ACO seminar today. The paper can be downloaded here.

Lecture 23 summary

Venkat Guruswami — Wed, 21 Apr 2010 20:29:18 +0000

We completed the discussion of the rate vs. list decoding radius trade-off achieved by folded Reed-Solomon codes and multivariate interpolation based decoding, and discussed its complexity and list-size bounds, as well as alphabet size. We highlighted the powerful list recovery property offered by folded RS codes, where having up to possible choices for each codeword position does not affect the ability to correct with agreement (where is the rate), and we can “absorb” the effect of into a somewhat larger alphabet size and decoding complexity. This feature is invaluable in using folded RS codes as outer codes in concatenation schemes, as we saw in two results:

Binary codes which are list-decodable up to the Zyablov radius (earlier we saw to unique decode up to half the Zyablov radius using GMD decoding)
Construction of codes of rate over an alphabet of size that are list-decodable up to a fraction of errors. The alphabet size is not far from the optimal bound of , and nicely combines ideas from the algebraic coding and expander decoding parts of the course.

We then wrapped up our discussion of list decoding by mentioning some of the big questions that still remain open, especially in constructing binary codes with near-optimal (or even better than currently known) trade-offs.

We discussed the framework of message-passing algorithms for LDPC codes, which will be the subject of the next lecture or two. We will mostly follow the description in this survey, but will not get too deep into the material.

Lecture 22 summary

Venkat Guruswami — Wed, 14 Apr 2010 19:59:19 +0000

We discussed how folded Reed-Solomon codes can be used to approach the optimal trade-off between rate and list decoding radius, specifically list decoding in polynomial time from a fraction of errors with rate for any desired constant 0' title='\epsilon > 0' class='latex' />.

We presented an algorithm for list decoding folded Reed-Solomon codes (with folding parameter ) when the agreement fraction is more than . This was based on the extension of the Welch-Berlekamp algorithm to higher order interpolation (in variables). Unfortunately, this result falls well short of our desired target, and in particular is meaningless for 1/s' title='R > 1/s' class='latex' />.

We then saw how to run the -variate algorithm on a folded RS code with folding parameter s' title='m > s' class='latex' />, to list decode when the agreement fraction is more than . Picking large and , say and , then enables list decoding from agreement fraction . We will revisit this final statement briefly at the beginning of the next lecture, and also comment on the complexity of the algorithm, bound on list-size, and alphabet size of the codes.

Notes for this lecture may not be immediately available, but you can refer to the original paper Explicit codes achieving list decoding capacity: Error-correction with optimal redundancy or Chapter 6 of the survey Algorithmic results for list decoding. Both of these are tailored to list decode even from the (in general) smaller agreement fraction and use higher degrees for the ‘s in the polynomial as well as multiple zeroes at the interpolation points. In the lecture, however, we were content, for sake of simplicity and because it suffices to approach agreement fraction , with restricting to be linear in the ‘s.

A reminder that we will have NO lecture this Friday (April 16) due to Spring Carnival.

Notes for lectures 18-21

Venkat Guruswami — Wed, 14 Apr 2010 02:51:00 +0000

Drafts of the notes for the lectures up till last Friday are now posted on the course webpage. I plan to proofread and make necessary edits to portions of the notes (for lecture 15 and later) in the next couple of weeks or so. But the current versions should already be useful if you need a refresher on something we covered in lecture, or as reference for working on the problem set.

Lecture 21 summary

Venkat Guruswami — Fri, 09 Apr 2010 19:54:39 +0000

Today we completed the description and analysis of the multiplicities based weighted polynomial reconstruction algorithm which immediately yielded an algorithm for list decoding Reed-Solomon codes up to the Johnson radius of for rate . We discussed the utility of weights in exploiting “soft” information available during decoding (eg. from decoding inner codes in a concatenation scheme, or from a demodulator which “rounds” analog signals to digital values). We saw simple consequences for list decoding binary concatenated codes, and in particular how to list-decode from a fraction of errors with rate and list-size . While the rate is positive for every 0' title='\gamma > 0' class='latex' />, it is far from the optimal bound. (We will soon see how this can be improved substantially by using codes with more powerful list-decoding properties than Reed-Solomon codes at the outer level.) Finally we defined (a version of) folded Reed-Solomon codes (we will give a list decoding algorithm for these next lecture).

We will have notes for this week’s lecture available soon, but the material covered this week has also been written about in several surveys on list decoding (some of which are listed on the course webpage). Here are a couple of pointers, which also discuss the details of list decoding folded RS codes which we will cover next week (though we will use a somewhat simpler presentation with weaker bounds in our lectures):

Algorithmic results for list decoding (Chapter 4 covers RS list decoding)
Bridging Shannon and Hamming: List Error-Correction with Optimal Rate (again Section 4 briefly discusses list decoding RS codes)