Jekyll2019-03-02T00:43:05-08:00https://markkm.com/feed.xmlMark Kim-MulgrewMark Kim-Mulgrew is a software engineer, writer, and musician based in Seattle, WA.{"name"=>"", "avatar"=>nil, "bio"=>"", "location"=>nil, "email"=>nil, "links"=>[{"label"=>"Comments", "icon"=>"fas fa-fw fa-envelope-square", "url"=>"mailto:comments@markkm.com"}, {"label"=>"Twitter", "icon"=>"fab fa-fw fa-twitter-square", "url"=>"https://twitter.com/markkimmulgrew"}, {"label"=>"Goodreads", "icon"=>"fab fa-goodreads", "url"=>"https://goodreads.com/markkm"}]}Elias M. Stein, 1931-20182018-12-29T06:00:00-08:002018-12-29T06:00:00-08:00https://markkm.com/blog/elias-stein-1931-2018<p>Elias Menachem Stein, one of the most influential mathematicians of the post-World War II era, <a href="https://www.math.princeton.edu/news/elias-m-stein-1931-2018">died on December 23</a> at the age of 87.</p>
<p>During a career that spanned over six decades, Stein revolutionized the field of harmonic analysis through first-rate research, exposition, and mentoring. Stein’s research work was an embodiment of what <a href="https://en.wikipedia.org/wiki/Antoni_Zygmund">Zygmund</a> called “<a href="http://www.ams.org/notices/201205/rtx120500641p.pdf">mathematical imperialism</a>,” pushing the boundaries of harmonic analysis far beyond its initial focus on the Fourier series and establishing it as an essential tool in the study of partial differential equations, representation theory, several complex variables, ergodic theory, functional analysis, geometric measure theory, and number theory.</p>
<p>Not only was Stein a key innovator in harmonic analysis, he also fostered a large community of harmonic analysts with his tireless mentoring. By his retirement in 2012, Stein had <a href="http://www.genealogy.ams.org/id.php?id=6454">over fifty doctoral students</a>, many of whom went on to become <a href="https://en.wikipedia.org/wiki/Charles_Fefferman">leaders</a> <a href="https://en.wikipedia.org/wiki/Terence_Tao">of</a> <a href="https://en.wikipedia.org/wiki/Robert_Strichartz">the</a> <a href="https://en.wikipedia.org/wiki/William_Beckner_(mathematician)">field</a>. <script type="math/tex">{}_{}</script> Stein, together with over five hundred academic descendants, has fundamentally changed the way mathematical analysis is practiced, establishing the influential Calderón–Zygmund–Stein school of analysis. <a name="footnote-1-origin"></a><a href="#footnote-1">[1]</a></p>
<p>Beyond his research activities, Stein sought to enrich the mathematics community at large through <a href="https://press.princeton.edu/titles/1977.html">his</a> <a href="https://press.princeton.edu/titles/1976.html">legendary</a> <a href="https://press.princeton.edu/titles/1978.html">expository</a> <a href="https://press.princeton.edu/titles/5390.html">works</a>, providing students and non-experts access to techniques previously available only to a small group of experts. In the 2000s, he pushed to share the core ideas of the field with an even wider audience by penning a <a href="https://en.wikipedia.org/wiki/Princeton_Lectures_in_Analysis">now-famous four-volume series on mathematical analysis</a>, the first-ever widely-used series of textbooks at the advanced undergraduate and beginning graduate level that presents the many subdisciplines of mathematical analysis as a unified whole. <a name="footnote-2-origin"></a><a href="#footnote-2">[2]</a></p>
<p>Although I have met Stein only a few times in person <a name="footnote-3-origin"></a><a href="#footnote-3">[3]</a>, I can say without hesitation that he has been the single most influential person in my professional life thus far. It was a student of Stein who first encouraged me to study mathematics seriously, and it was Stein’s summer program that helped me decide the overarching direction of my graduate studies. I learned much of the mathematics I know from Stein’s books <a name="footnote-4-origin"></a><a href="#footnote-4">[4]</a>, and it is through Stein’s writing that I developed my interest in science and technical writing. Even computer science was a subject I first encountered while attempting to study Stein’s works more broadly. Stein’s influence is pervasive in everything I have done professionally, and what I have learned from Stein and his works will no doubt continue to serve as a profound inspiration throughout my career.</p>
<p><a name="footnote-1"></a><strong>[1]</strong> Zygmund was Stein’s advisor, and <a href="https://en.wikipedia.org/wiki/Alberto_Calder%C3%B3n">Calderón</a> was a student of Zygmund. The name “Calderón–Zygmund–Stein theory” typically refers to the collection of mathematical techniques centered around <a href="https://en.wikipedia.org/wiki/Singular_integral">singular integrals</a>, <a href="https://en.wikipedia.org/wiki/Maximal_function">maximal functions</a>, <a href="https://en.wikipedia.org/wiki/Pseudo-differential_operator">pseudodifferential operators</a>, and <a href="https://en.wikipedia.org/wiki/Oscillatory_integral_operator">oscillatory integrals</a>. <a href="#footnote-1-origin"><i class="fas fa-level-up-alt"></i></a></p>
<p><a name="footnote-2"></a><strong>[2]</strong> <a href="https://en.wikipedia.org/wiki/%C3%89l%C3%A9ments_de_math%C3%A9matique">Bourbaki</a> de-emphasized analytic approaches to mathematics, <a href="https://en.wikipedia.org/wiki/Treatise_on_analysis">Dieudonné</a> was never widely adopted, and <a href="https://bookstore.ams.org/simon-set">Simon</a> will likely be regarded as references rather than expository works suitable for beginning students. <a href="#footnote-2-origin"><i class="fas fa-level-up-alt"></i></a></p>
<p><a name="footnote-3"></a><strong>[3]</strong> Once, Stein told a story of having to read the Harry Potter series to hang out with his grandchildren. I asked how he could possibly have time to read such a massive series with all the work he is doing. He grinned and said, well, I read the last volume and Wikipedia-ed the rest.<a href="#footnote-3-origin"><i class="fas fa-level-up-alt"></i></a></p>
<p><a name="footnote-4"></a><strong>[4]</strong> I thought I had eight books by Stein, but I can only find seven in my bookshelf. Bonus points for guessing what they are!<a href="#footnote-4-origin"><i class="fas fa-level-up-alt"></i></a></p>{"name"=>"", "avatar"=>nil, "bio"=>"", "location"=>nil, "email"=>nil, "links"=>[{"label"=>"Comments", "icon"=>"fas fa-fw fa-envelope-square", "url"=>"mailto:comments@markkm.com"}, {"label"=>"Twitter", "icon"=>"fab fa-fw fa-twitter-square", "url"=>"https://twitter.com/markkimmulgrew"}, {"label"=>"Goodreads", "icon"=>"fab fa-goodreads", "url"=>"https://goodreads.com/markkm"}]}An obituary of Elias M. Stein, one of the most influential mathematicians of the post-World War II era.Reservoir sampling: who discovered Algorithm R?2018-10-28T03:00:00-07:002018-10-28T03:00:00-07:00https://markkm.com/blog/reservoir-sampling<p><strong><em>Reservoir sampling</em></strong> is a class of algorithms for sampling from streaming data, i.e., a sequence of data that we can access only once. Presented below is <strong><em>Algorithm R</em></strong>, the first well-known reservoir sampling algorithm.</p>
<p class="notice"><strong>Reservoir sampling – Algorithm R.</strong> We wish to sample <script type="math/tex">m</script> elements of <script type="math/tex">L</script> with equal probability. To this end, we define a size-<script type="math/tex">m</script> set <script type="math/tex">R</script> (“reservoir”) and initialize it by setting <script type="math/tex">R[i] = x_i</script> for <script type="math/tex">0 \leq i \leq m-1</script>. For each index <script type="math/tex">i \geq m</script>, we perform a simple random sampling on the set of integers between <script type="math/tex">0</script> and <script type="math/tex">i</script> to choose <script type="math/tex">k</script>. If <script type="math/tex">0 \leq k \leq m-1</script>, then we set <script type="math/tex">R[k] = x_i</script>; otherwise, we leave <script type="math/tex">R</script> unchanged. <script type="math/tex">R</script> is the desired sample, once we go through all elements in <script type="math/tex">L</script>.</p>
<p>In Volume 2, Section 3.4.1 of <em>The Art of Computer Programming</em>, Knuth attributes Algorithm R to Alan G. Waterman. He, however, does not provide a reference, and there appears to be little information available on the matter. I sent a letter of inquiry to Knuth and received the following reply:</p>
<blockquote>
<p>28 Dec 2017</p>
<p>to mark kim</p>
<p>from don knuth</p>
<p>Alan Waterman wrote me in the 70s, with detailed comments and suggestions that I incorporated into the second edition of Volume 2—which I was happy to complete in 1975 or 1976. (Bad typesetting took over, and I had to develop <script type="math/tex">\TeX</script> before that book could be published, finally, in 1981.)</p>
<p>The first edition had an inferior Algorithm 3.4.2R dating from 1962; Alan was responding to it.</p>
<p>My books contain lots of unpublished work that I happen to learn about in various ways. But if the author(s) do(es) publish, I try to give a citation.</p>
<p>I’m unaware that Alan ever did write a paper about this (or any of his other contributions that are cited elsewhere in the second edition). If you learn of any appropriate references, please let me know … and you will deserve a reward of 0x$1.00.</p>
<p>All I remember is that I was tickled pink to receive a letter from a real Harvard statistician who actually enjoyed my self-taught attempts at exposition of statistical methods…</p>
<p>Enclosed is a copy of the letter that survives in my files.</p>
<p>-don</p>
</blockquote>
<figure style="width: 500px" class="align-center">
<img src="https://markkm.com/assets/images/reservoir-1.jpg" alt="knuth letter on reservoir sampling" />
</figure>
<p>Here are the relevant sections of Waterman’s 1975 letter to Knuth:</p>
<blockquote>
<p>Dr. Alan G. Waterman</p>
<p>March 24, 1975</p>
<p>Dear Prof. Knuth:</p>
<p>I have some remarks on Chapter 3 of your book, The Art of Computer Programming, on Random Numbers.</p>
<p>[omitted]</p>
<p>There is a better algorithm than reservoir sampling, section 3.4.2, when the length of the file is unknown. Bring in the first <script type="math/tex">n</script> records, then for each <script type="math/tex">k > n</script>, skip over the <script type="math/tex">k</script>th record with probability <script type="math/tex">(k-n)/k</script>, and replace the <script type="math/tex">i</script>th item in the sample by the <script type="math/tex">k</script>th record with probability <script type="math/tex">1/k</script>, for each <script type="math/tex">i \leq n</script>. If it is necessary to pass the records to a reservoir (the text is not clear on this point), the replacements may be done in an internal table of indices to the reservoir.</p>
<p>[omitted]</p>
<p>Fourth, your improved reservoir algorithm (why oh why didn’t I think of it?) will replace the old one in Section 3.4.2.</p>
</blockquote>
<figure style="width: 500px" class="align-center">
<img src="https://markkm.com/assets/images/reservoir-2.jpg" alt="waterman letter on reservoir sampling 1" />
</figure>
<figure style="width: 500px" class="align-center">
<img src="https://markkm.com/assets/images/reservoir-3.jpg" alt="waterman letter on reservoir sampling 2" />
</figure>
<p>All in all, Algorithm R was known to Knuth and Waterman by 1975, and to a wider audience by 1981, when the second edition of <em>The Art of Computer Programming</em> volume 2 was published.</p>
<p>Meanwhile, <a href="http://www.stats.uwo.ca/faculty/mcleod/">A. I. McLeod</a> and <a href="http://www.stats.uwo.ca/faculty/bellhouse/">D. R. Bellhouse</a> appear to have discovered Algorithm R independently, as [<a href="http://www.stats.uwo.ca/faculty/aim/vita/pdf/McLeod83.pdf">McLeod–Bellhouse 1983</a>] presents Algorithm R without citing Knuth or Waterman.</p>
<p>Before the publication of Waterman’s algorithm, [<a href="https://www.jstor.org/stable/2281647?seq=1#page_scan_tab_contents">Fan–Muller–Rezucha 1962</a>] described a similar, but not quite the same, algorithm:</p>
<blockquote>
<p>Procedure of Method 4:</p>
<p>For each <script type="math/tex">t</script>, <script type="math/tex">t = 1, 2, \cdots, n</script>, obtain <script type="math/tex">r_t</script>, associate it with the item and its label <script type="math/tex">I_t</script>, and place these pieces of information in the reservoir. For each <script type="math/tex">t</script>, <script type="math/tex">t > n</script>, place <script type="math/tex">r_t</script>, the item, and its label <script type="math/tex">I_t</script> in the reservoir if <script type="math/tex">r_t</script> is equal to or less than the maximum value of <script type="math/tex">r</script> of a specified subset of values of <script type="math/tex">r</script> in the reservoir. The subset of values of <script type="math/tex">r</script> used in the comparison consists of the <script type="math/tex">D</script> smallest distinct values of <script type="math/tex">r</script> in the reservoir at the time of the comparison, where <script type="math/tex">D = n</script> if all the <script type="math/tex">r</script>’s in this comparison subset are distinct; otherwise <script type="math/tex">% <![CDATA[
D < n %]]></script>.</p>
<p>When <script type="math/tex">t = N</script>, i.e., all the items have been inspected, stage one is completed.</p>
<p>Stage 2. Search the reservoir and select the <script type="math/tex">I_t</script>’s associated with the subset of the <script type="math/tex">n</script> smallest values of <script type="math/tex">r</script> in the reservoir Difficulty can occur only when the values of the <script type="math/tex">r</script>’s are not distinct. In this case it is possible that there will be one or more items in the reservoir with a value of <script type="math/tex">r</script> equal to the largest value, say <script type="math/tex">r_u</script>, of the comparison subset upon completion of stage 1. Let <script type="math/tex">L</script> denote the number of items in this comparison subset upon completion of stage 1 which have values of <script type="math/tex">r</script> less than <script type="math/tex">r_u</script> and let <script type="math/tex">M</script> be the number of items in the reservoir which have a value of <script type="math/tex">r</script> equal to <script type="math/tex">r_u</script>. To satisfy the original requirement of obtaining a random sample of exactly <script type="math/tex">n</script> distinct items it will be necessary to utilize an additional selection procedure to select <script type="math/tex">M'</script> distinct items out of <script type="math/tex">M</script> such that <script type="math/tex">L+M' = n</script>. This selection can be accomplished by using, for example, Method 1.</p>
</blockquote>
<p>While it might be reasonable to say that the reservoir algorithm paradigm was discovered by Fan, Muller, and Rezucha, it seems unlikely that they were aware of Algorithm R before Knuth published Waterman’s algorithm.</p>
<p>As far as I can tell, most citations of Algorithm R credit Waterman. Curiously, however, the <a href="https://en.wikipedia.org/w/index.php?title=Reservoir_sampling&oldid=813383641">Wikipedia article on reservoir sampling</a> makes no mention of Waterman, crediting instead <a href="http://chancellor.olemiss.edu/meet-jeff/">J. S. Vitter</a> via [<a href="https://dl.acm.org/citation.cfm?id=198435">Vitter 1985</a>]. But then, the Vitter paper cites Waterman:</p>
<blockquote>
<p>Algorithm R (which is is [sic] a reservoir algorithm due to Alan Waterman) works as follows: When the <script type="math/tex">(t+1)</script>st record in the file is being processed, for <script type="math/tex">t \geq n</script>, the <script type="math/tex">n</script> candidates form a random sample of the first <script type="math/tex">t</script> records. The <script type="math/tex">(t+1)</script>st record has a <script type="math/tex">n/(t+1)</script> chance of being in a random sample of size <script type="math/tex">n</script> of the first <script type="math/tex">t+1</script> records, and so it is made a candidate with probability <script type="math/tex">n/(t+1)</script>. The candidate it replaces is chosen randomly from the <script type="math/tex">n</script> candidates. It is easy to see that the resulting set of <script type="math/tex">n</script> candidates forms a random sample of the first <script type="math/tex">t+1</script> records.</p>
</blockquote>
<p>This is probably why high school history teachers tell their students not to use Wikipedia for their essay homework.</p>{"name"=>"", "avatar"=>nil, "bio"=>"", "location"=>nil, "email"=>nil, "links"=>[{"label"=>"Comments", "icon"=>"fas fa-fw fa-envelope-square", "url"=>"mailto:comments@markkm.com"}, {"label"=>"Twitter", "icon"=>"fab fa-fw fa-twitter-square", "url"=>"https://twitter.com/markkimmulgrew"}, {"label"=>"Goodreads", "icon"=>"fab fa-goodreads", "url"=>"https://goodreads.com/markkm"}]}We study the historical origin of Algorithm R, the first reservoir sampling algorithm.