atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[iWeb Hosting]]></title>
  <link href="http://iwebhosting.github.io/atom.xml" rel="self"/>
  <link href="http://iwebhosting.github.io/"/>
  <updated>2013-06-28T17:22:00+01:00</updated>
  <id>http://iwebhosting.github.io/</id>
  <author>
    <name><![CDATA[iWeb Hosting]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Forcing Exim Local Delivery on Ubuntu (For Development)]]></title>
    <link href="http://iwebhosting.github.io/blog/2013/06/06/forcing-exim-local-delivery-on-ubuntu-for-development/"/>
    <updated>2013-06-06T17:16:00+01:00</updated>
    <id>http://iwebhosting.github.io/blog/2013/06/06/forcing-exim-local-delivery-on-ubuntu-for-development</id>
    <content type="html"><![CDATA[<p>Sometimes you want to run development code against your production database before it goes live, as a test, but want to make sure that there are no side effects from this process.</p>

<p>Recently a developer at iWeb wanted to test some email code&rsquo;s interaction with a real mail server, without potentially mailing thousands of real customers. Mocks wouldn&rsquo;t be suitable because the tests cross the API boundary; it&rsquo;s not enough to know it&rsquo;s called, they needed to see what it <em>did</em>.</p>

<!--more-->


<h4>Installing Exim</h4>

<p>If you don&rsquo;t already have Exim installed, just install <code>exim4-daemon-light</code>. When Debconf asks you for the configuration type, you want &ldquo;Local Delivery Only&rdquo;:</p>

<p><img src="http://17k.co.uk/s/09207fc6.png" alt="Local delivery" /></p>

<p>Select what you want for the next three dialogs, and you <em>do</em> want to split configuration into small files:</p>

<p><img src="http://17k.co.uk/s/91ffcd4b.png" alt="Small files, please" /></p>

<p>Now, create the file <code>/etc/exim4/conf.d/router/050_force_local_delivery</code> with the following contents:</p>

<pre><code>forced:
  debug_print = "R: forced for $local_part@$domain"
  driver = redirect
  allow_fail
  allow_defer
  data = "iweb" # change this to a local user on your machine
</code></pre>

<p>Update your config and restart Exim:</p>

<pre><code># update-exim4.conf
# /etc/init.d/exim4 restart
</code></pre>

<p>Putting it in as &ldquo;050&rdquo; means that this will fire before the <code>nonlocal</code> router, which is in <code>200_exim4-config_primary</code>. If you see the error &ldquo;Mailing to remote domains not supported&rdquo;, check that your file is named correctly and that you ran <code>update-exim4.conf</code> and restarted Exim.</p>

<p>And then give it a test:</p>

<pre><code># mail anyone@anywhere.com &lt; /dev/null
No message, no subject; hope that's ok
# tail -3 /var/log/exim4/mainlog
2013-06-06 17:36:32 1UkdAu-0000b5-3y &lt;= root@test.iweb U=root P=local S=338
2013-06-06 17:36:32 1UkdAu-0000b5-3y =&gt; iweb &lt;anyone@anywhere.com&gt; R=local_user T=maildir_home
2013-06-06 17:36:32 1UkdAu-0000b5-3y Completed
</code></pre>

<p>&nbsp;</p>

<p>If you&rsquo;ve found this useful, please let me know on Twitter via <a href="http://twitter.com/insom">@insom</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Moving Varnish caching logic into PHP with the cURL VMOD]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/11/20/moving-varnish-caching-logic-into-php-with-the-curl-vmod/"/>
    <updated>2012-11-20T14:46:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/11/20/moving-varnish-caching-logic-into-php-with-the-curl-vmod</id>
    <content type="html"><![CDATA[<p>As part of our general quest to provide <a href="http://iwebhosting.github.io/blog/2012/09/11/diagnosing-magento-speed-issues-with-strace/">fast Magento hosting</a>, we&rsquo;ve
spent some time exploring new features in <a href="https://www.varnish-cache.org/releases/varnish-cache-3.0.0">Varnish 3.0</a>. VMOD support means that there&rsquo;s a supported way of extending VCL with modules in their own name
spaces. Some of the modules from <a href="https://www.varnish-cache.org/vmods">the VMOD list</a> looked really
interesting; particularly <a href="https://github.com/zephirworks/libvmod-redis">Redis</a>, <a href="https://github.com/sodabrew/libvmod-memcached">Memcached</a> and
<a href="https://github.com/varnish/libvmod-curl/">cURL</a>.</p>

<p>In this post I&rsquo;m going to use the <a href="https://github.com/varnish/libvmod-curl/">cURL</a> VMOD to decide whether to show
a page from the cache or not, by deciding if the user is logged in. This is the
kind of thing you <em>could</em> do with VCL if you were happy to set custom cookies
alongside your apps own session cookie, but as you want to make more complex
decisions I thought it would be interesting to push this into PHP.</p>

<!--more-->


<h4>Building the cURL VMOD</h4>

<p>On Ubuntu Precise, install the dependencies:</p>

<pre><code># apt-get install varnish php5 apache2 libvarnishapi-dev varnish-dbg \
build-essential libcurl4-openssl-dev git automake libtool autoconf \
libpcre3-dev
</code></pre>

<p>Grab the varnish source, this is required to build the VMOD. We also have to
build (but not install) it:</p>

<pre><code>$ apt-get source varnish
$ cd varnish*
$ ./configure
$ make
</code></pre>

<p>Check the VMOD source code out and generate the configure file:</p>

<pre><code>$ git clone git://github.com/varnish/libvmod-curl.git
$ cd libvmod-curl
$ ./autogen.sh
$ ./configure VARNISHSRC=$HOME/varnish-3.0.2/
$ make
# make install
</code></pre>

<p>On my machine it put everything in the right place, and I was able to just put
<code>import curl;</code> into my default.vcl and start using it right away.</p>

<h4>Creating a simple PHP test application</h4>

<p>Let&rsquo;s create a small app with a page which is expensive to generate, but can be cached for users who haven&rsquo;t stored any state in their session.</p>

<figure class='code'><figcaption><span>&#8220;slow.php&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="cp">&lt;?php</span>
</span><span class='line'><span class="nb">session_start</span><span class="p">();</span>
</span><span class='line'><span class="nb">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">);</span>
</span><span class='line'><span class="nb">print_r</span><span class="p">(</span><span class="nv">$_SESSION</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>


<p>The above file, if we didn&rsquo;t override Varnish, would never be cachable because
the <a href="http://php.net/session_start">session_start</a>() call will emit a <code>Set-Cookie</code> header, and Varnish
will not normally cache a response which sets cookies.</p>

<p>Anonymous users should always see an empty array, and users with state should
always see a &ldquo;YES&rdquo;.</p>

<figure class='code'><figcaption><span>&#8220;start_session.php&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="cp">&lt;?php</span>
</span><span class='line'><span class="nb">session_start</span><span class="p">();</span>
</span><span class='line'><span class="nv">$_SESSION</span><span class="p">[</span><span class="s1">&#39;started&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;YES&#39;</span><span class="p">;</span>
</span><span class='line'><span class="nb">session_write_close</span><span class="p">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>This will set the above &ldquo;YES&rdquo;. Once this is visited, we expect that we&rsquo;ll see
an uncached version of the site.</p>

<p>As of right now visiting <code>/slow.php</code> should take 5 seconds, reloading it will
take 5 seconds (and still show a blank array), then visiting
<code>/start_session.php</code> will be fast and finally visiting <code>/slow.php</code> will take
5 seconds, and show &ldquo;YES&rdquo;. This is because, right now, nothing is cached.</p>

<h4>Just enough VCL to let PHP make the decision</h4>

<figure class='code'><figcaption><span>&#8220;default.vcl&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="n">import</span> <span class="n">curl</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="n">backend</span> <span class="k">default</span> <span class="p">{</span>
</span><span class='line'>    <span class="p">.</span><span class="n">host</span> <span class="o">=</span> <span class="s">&quot;127.0.0.1&quot;</span><span class="p">;</span>
</span><span class='line'>    <span class="p">.</span><span class="n">port</span> <span class="o">=</span> <span class="s">&quot;80&quot;</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="n">sub</span> <span class="n">vcl_recv</span> <span class="p">{</span>
</span><span class='line'>        <span class="cm">/* The next 5 lines take the cookie line and normalise it to just</span>
</span><span class='line'><span class="cm">         * the session_id */</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span> <span class="o">=</span> <span class="s">&quot;;&quot;</span> <span class="o">+</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">cookie</span><span class="p">;</span>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span> <span class="o">=</span> <span class="n">regsuball</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">,</span> <span class="s">&quot;; +&quot;</span><span class="p">,</span> <span class="s">&quot;;&quot;</span><span class="p">);</span>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span> <span class="o">=</span> <span class="n">regsuball</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">,</span> <span class="s">&quot;;(PHPSESSID)=&quot;</span><span class="p">,</span> <span class="s">&quot;; &quot;</span><span class="p">);</span>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span> <span class="o">=</span> <span class="n">regsuball</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">,</span> <span class="s">&quot;;[^ ][^;]*&quot;</span><span class="p">,</span> <span class="s">&quot;&quot;</span><span class="p">);</span>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span> <span class="o">=</span> <span class="n">regsuball</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">,</span> <span class="s">&quot;^[; ]+|[; ]+$&quot;</span><span class="p">,</span> <span class="s">&quot;&quot;</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">curl</span><span class="p">.</span><span class="n">fetch</span><span class="p">(</span><span class="s">&quot;http://localhost/decider.php?c=&quot;</span> <span class="o">+</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="n">remove</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">cookie</span><span class="p">;</span>
</span><span class='line'>        <span class="n">set</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">session</span> <span class="o">=</span> <span class="n">curl</span><span class="p">.</span><span class="n">header</span><span class="p">(</span><span class="s">&quot;X-Started&quot;</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">if</span> <span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">session</span> <span class="o">!=</span> <span class="s">&quot;yes&quot;</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                <span class="n">remove</span> <span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">cookie</span><span class="p">;</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="n">sub</span> <span class="n">vcl_hit</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">session</span> <span class="o">==</span> <span class="s">&quot;yes&quot;</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                <span class="k">return</span> <span class="p">(</span><span class="n">pass</span><span class="p">);</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="n">sub</span> <span class="n">vcl_fetch</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">x</span><span class="o">-</span><span class="n">session</span> <span class="o">!=</span> <span class="s">&quot;yes&quot;</span> <span class="o">&amp;&amp;</span> <span class="n">req</span><span class="p">.</span><span class="n">url</span> <span class="o">!~</span> <span class="s">&quot;add.php&quot;</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                <span class="n">remove</span> <span class="n">beresp</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">set</span><span class="o">-</span><span class="n">cookie</span><span class="p">;</span>
</span><span class='line'>                <span class="n">set</span> <span class="n">beresp</span><span class="p">.</span><span class="n">ttl</span> <span class="o">=</span> <span class="mi">5</span><span class="n">m</span><span class="p">;</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>The above temporarily adds an <code>X-Cookie</code> header which will just contain the PHP
session ID. We&rsquo;re going to pass this to our <code>decider.php</code> script, and it will
pass back an <code>X-Started</code> header saying whether we consider this session to be
&ldquo;properly&rdquo; started. We do all of this in the <code>vcl_recv</code> step, immediately as
the request comes in to Varnish.</p>

<p><code>vcl_hit</code> will check if we consider the session to be started and if so will
issue a pass from now on; this is the code that stops cached pages being
served to people with &ldquo;proper&rdquo; sessions.</p>

<p><code>vcl_fetch</code> will now look at the response and, if <code>X-Session</code> is not &ldquo;yes&rdquo;,
will strip off the <code>Set-Cookie</code> and set the ttl, allowing it to be cached.</p>

<p>The only unpleasant part in all of this is where we check if the URL is
<code>add.php</code>, on line 35. We have to allow the cookie through from this
script so it can start our session.</p>

<h3>Our example &lsquo;decider&rsquo;</h3>

<p>Finally, the contents of our example decider file:</p>

<figure class='code'><figcaption><span>&#8220;decider.php&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="cp">&lt;?php</span>
</span><span class='line'><span class="nb">session_start</span><span class="p">();</span>
</span><span class='line'><span class="nv">$sessionid</span> <span class="o">=</span> <span class="nv">$_GET</span><span class="p">[</span><span class="s1">&#39;c&#39;</span><span class="p">];</span>
</span><span class='line'><span class="cm">/* The next line is horribly Ubuntu-specific and should just serve as</span>
</span><span class='line'><span class="cm"> * a demonstration. */</span>
</span><span class='line'><span class="nv">$b</span> <span class="o">=</span> <span class="nb">file_get_contents</span><span class="p">(</span><span class="s1">&#39;/var/lib/php5/sess_&#39;</span> <span class="o">.</span> <span class="nv">$sessionid</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">);</span>
</span><span class='line'><span class="nb">session_decode</span><span class="p">(</span><span class="nv">$b</span><span class="p">);</span> <span class="cm">/* This loads the session data into $_SESSION */</span>
</span><span class='line'><span class="k">if</span><span class="p">(</span><span class="nv">$_SESSION</span><span class="p">[</span><span class="s1">&#39;started&#39;</span><span class="p">])</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">header</span><span class="p">(</span><span class="s1">&#39;X-Started: yes&#39;</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Basically, we will only consider a session to have started fully if the
<code>started</code> key in the session exists. The presence of the session cookie
is not enough.</p>

<p>On an e-commerce store you could decide that a session isn&rsquo;t &ldquo;proper&rdquo; until
the user is logged in, or has an item in their cart, allowing almost all of
your visitors to continue to see cached versions of the site.</p>

<p>Some large PHP packages (<em>cough</em> Magento) are very free with sending cookies
and starting sessions, but the common advice is to override this with
lots of logic in the VCL about what can start a session. Moving this logic
back into PHP seems like a natural step.</p>

<p>The speed of your decider is critical; it&rsquo;s hit for every request. In
production you probably want to have logic above our <code>vcl_recv</code> block
which short-circuits it for images and other statics. It will probably
not be acceptible to create, say, a new <code>Mage</code> object from PHP in order
to check if the page could be served from cache, given the current session.</p>

<p>&nbsp;</p>

<p>If you&rsquo;ve found this useful, please let me know on Twitter via <a href="http://twitter.com/insom">@insom</a>. I
plan on following this post up with a Magento-specific example of how to
build a fast store by bypassing much of Magento&rsquo;s logic on starting sessions.</p>

<p><em>That said</em> a colleague has an alternative pure-PHP solution to Magento&rsquo;s
specific session starting behaviour, but the example will hopefully show how
the above can be applied to other non-trivial web applications.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Diagnosing Magento speed issues with strace]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/09/11/diagnosing-magento-speed-issues-with-strace/"/>
    <updated>2012-09-11T10:28:00+01:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/09/11/diagnosing-magento-speed-issues-with-strace</id>
    <content type="html"><![CDATA[<p>When your application depends on a remote service, and that service is either
down or slow, it will directly affect your own application&rsquo;s performance.</p>

<p><img src="http://iwebhosting.github.io/assets/magento-slow.png" alt="Slow Website" /></p>

<p>There are tools targeted directly at developers, like <a href="http://www.xdebug.org/">Xdebug</a> and
<a href="http://kcachegrind.sourceforge.net/html/Home.html">KCachegrind</a> but traditional sysadmin tools also have a place and provide a
lot of information that language-level debugging tools omit.</p>

<!--more-->


<p>Because tools like <a href="http://linux.die.net/man/1/gdb">gdb</a>, <a href="http://linux.die.net/man/1/time">time</a> and <a href="http://linux.die.net/man/1/strace">strace</a> are designed to work on
single processes, the first thing to do is get your Magento page running as a
separate process, outside of Apache or any other web server. All Magento wants
to know is the HTTP Host header and the URI that has been called. You can pass
these in via the standard <a href="http://www.ietf.org/rfc/rfc3875">cgi-bin</a> environment variables. Running just
<code>php</code> on its own will fire up an interpreter, just like if it was called as a
cgi-bin and similar to how it run a script when called through a web server.</p>

<p>Just change to the top of your Magento tree; typically your <code>public_html</code> or
equivalent folder and run a single request:</p>

<pre><code>$ HTTP_HOST=www.website.com REQUEST_URI=/slow-page.html php index.php
</code></pre>

<p>You should get a screenful of generated HTML. We don&rsquo;t care about this too
much, so in future we&rsquo;ll add <code>&gt;/dev/null</code> to suppress the output of the script.</p>

<p>The simplest thing we can do is run <a href="http://linux.die.net/man/1/time">time</a> &ndash; this will give us at least the
amount of CPU seconds spent in user and system, and the &lsquo;wall clock time&rsquo; of
how long the process took to run. On modern Linux systems it also gives the
percentage CPU used, the peak amount of RAM consumed and the amount of
pagefaults.</p>

<pre><code>$ HTTP_HOST=www.website.com REQUEST_URI=/slow-page.html time php index.php &gt;/dev/null

0.82user 0.24system 0:01.73elapsed 60%CPU (0avgtext+0avgdata 403520maxresident)k
</code></pre>

<p>This one page took 1.7 seconds to render completely, 0.8 of those seconds were
used by PHP using the CPU for something, and 0.24 were spent in system, meaning
that the Linux kernel was doing work on your behalf. The remaining 0.67 seconds
were spent waiting for &hellip; something. In most cases the something is a response
from the network or from the disks. (The response from the network can include
things like MySQL, which is effectively a network connection, even if it&rsquo;s on
the same machine). This request also took 403Mb of RAM to run, including shared
memory.</p>

<p>In an ideal world, each request would spend as close to 100% of its time in
user as possible, and none of it waiting or in system, so let&rsquo;s try and track
down where those missing milliseconds are going. For this, we&rsquo;ll use
<a href="http://linux.die.net/man/1/strace">strace</a>.</p>

<p>In brief, strace will print out each of the system calls that your process
makes. A system call is where control is passed from user to system, basically
whenever you ask the kernel to do something for you, like open a file, send
data over a socket, or write something to a file descriptor. It also includes a
lot of noisy stuff like getting the time of day, allocating RAM and blocking on
mutexes, so we&rsquo;ll apply a filter to strace to just get the system calls we&rsquo;re
likely to be interested in. You can do this with the <code>-e</code> parameter, and full
documentation is in the <a href="http://linux.die.net/man/1/strace">man page</a>.</p>

<pre><code>$ HTTP_HOST=www.website.com REQUEST_URI=/slow-page.html strace -e trace=sendto,connect,open,write php index.php &gt;/dev/null
</code></pre>

<p>There will be a <em>lot</em> of output, so here&rsquo;s a quick guide to figuring out what&rsquo;s
going on.</p>

<p>If you see a lot of lines like this:</p>

<pre><code>open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Model/Product/Media/Config.php", O_RDONLY) = 6
open("/home/website/www.website.com/public_html/app/code/core/Mage/Media/Model/Image/Config/Interface.php", O_RDONLY) = 6
open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Helper/Image.php", O_RDONLY) = 6
open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Model/Product/Image.php", O_RDONLY) = 6
</code></pre>

<p>Then you aren&rsquo;t using compilation. This means that Magento is having to open
hundreds (797 in my test) of little files, and that&rsquo;s contributing to your time
spent in system. For our purposes though, having compilation off makes things
easier to debug, so for the next run turn it off, if it isn&rsquo;t off already. Look
for any places where the output appears to pause. In my test, I saw a sendto()
call block. Adding <code>-T</code> to strace gives you the relative time spend in each
system call, at the end of the line.</p>

<figure class='code'><figcaption><span>&#8220;Long strace(1) output&#8221; </span></figcaption>
<div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>open("/home/website/www.website.com/public_html/app/design/front-end/iweb/default/template/catalog/product/view/media.phtml", O_RDONLY) = 6 &lt;0.000027>
</span><span class='line'>open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Model/Product/Media/Config.php", O_RDONLY) = 6 &lt;0.000020>
</span><span class='line'>open("/home/website/www.website.com/public_html/app/code/core/Mage/Media/Model/Image/Config/Interface.php", O_RDONLY) = 6 &lt;0.000018>
</span><span class='line'>open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Helper/Image.php", O_RDONLY) = 6 &lt;0.000019>
</span><span class='line'>open("/home/website/www.website.com/public_html/app/code/core/Mage/Catalog/Model/Product/Image.php", O_RDONLY) = 6 &lt;0.000019>
</span><span class='line'>open("/home/website/www.website.com/public_html/media/catalog/product/c/a/camden_brown_2.png", O_RDONLY) = 6 &lt;0.000018>
</span><span class='line'>open("/home/website/www.website.com/public_html/media/catalog/product/c/a/camden_brown.png", O_RDONLY) = 6 &lt;0.000017>
</span><span class='line'>open("/home/website/www.website.com/public_html/lib/Varien/Image.php", O_RDONLY) = 6 &lt;0.000025>
</span><span class='line'>open("/home/website/www.website.com/public_html/lib/Varien/Image/Adapter.php", O_RDONLY) = 6 &lt;0.000018>
</span><span class='line'>open("/home/website/www.website.com/public_html/lib/Varien/Image/Adapter/Gd2.php", O_RDONLY) = 6 &lt;0.000018>
</span><span class='line'>open("/home/website/www.website.com/public_html/lib/Varien/Image/Adapter/Abstract.php", O_RDONLY) = 6 &lt;0.000017>
</span><span class='line'>open("/home/website/www.website.com/public_html/media/catalog/product/c/a/camden_brown.png", O_RDONLY) = 6 &lt;0.000023>
</span><span class='line'>sendto(6, "\24\0\0\0\26\0\1\3\305\tOP\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 &lt;0.000026>
</span><span class='line'>connect(6, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) &lt;0.000025>
</span><span class='line'>connect(6, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) &lt;0.000531>
</span><span class='line'>open("/etc/host.conf", O_RDONLY)        = 6 &lt;0.000019>
</span><span class='line'>open("/etc/resolv.conf", O_RDONLY)      = 6 &lt;0.000017>
</span><span class='line'>open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 6 &lt;0.000017>
</span><span class='line'>connect(6, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("10.7.0.197")}, 16) = -1 EINPROGRESS (Operation now in progress) &lt;0.000091>
</span><span class='line'>sendto(6, "GET /media/catalog/product/c/a/c"..., 60, MSG_DONTWAIT, NULL, 0) = 60 &lt;0.000167>
</span><span class='line'>sendto(6, "Host: www.website.com\r\n", 27, MSG_DONTWAIT, NULL, 0) = 27 &lt;0.000041>
</span><span class='line'>sendto(6, "\r\n", 2, MSG_DONTWAIT, NULL, 0) = 2 &lt;0.000192></span></code></pre></td></tr></table></div></figure>


<p>There&rsquo;s a lot of information here, but taking it line by line:</p>

<ul>
<li><p>1: The <code>Catalog_Product_Media</code> template is loaded. This is in a custom skin,
so it&rsquo;s a good place to look for inadvertent changes that might affect speed.</p></li>
<li><p>2-5: Dependencies of this template are loaded by the Magento autoloader,
<code>Catalog_Model_Product_Media_Config</code> through to <code>Catalog_Model_Product_Image</code>.</p></li>
<li><p>6-7: Loading source images from the media folder.</p></li>
<li><p>8-11: Creating <code>Varien_Image</code> objects, loading the classes in.</p></li>
<li><p>13-18: Doing a lookup of the website&rsquo;s own name, in nscd and then in
/etc/hosts. Because the site is defined in /etc/hosts no DNS lookup is done,
but it would happen here if that wasn&rsquo;t the case. It would look like this:</p></li>
</ul>


<figure class='code'><figcaption><span>&#8220;Example DNS request&#8221; </span></figcaption>
<div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("91.208.170.133")}, 16) = 0 &lt;0.000021>
</span><span class='line'>sendto(6, "\3140\1\0\0\1\0\0\0\0\0\0\3www\vwebsite\3com"..., 37, MSG_NOSIGNAL, NULL, 0) = 37 &lt;0.000060>
</span><span class='line'>connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("91.208.170.133")}, 16) = 0 &lt;0.000015>
</span><span class='line'>sendto(6, "&\312\1\0\0\1\0\0\0\0\0\0\3www\vwebsite\3com"..., 37, MSG_NOSIGNAL, NULL, 0) = 37 &lt;0.000037>
</span><span class='line'>connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("91.208.170.133")}, 16) = 0 &lt;0.000015>
</span><span class='line'>sendto(6, "\23m\1\0\0\1\0\0\0\0\0\0\3www\vwebsite\3com"..., 37, MSG_NOSIGNAL, NULL, 0) = 37 &lt;0.000037></span></code></pre></td></tr></table></div></figure>


<ul>
<li><p>19: Creating a connection to the web-server (ourselves in this case).</p></li>
<li><p>20-22: Making a request for an image that we have locally.</p></li>
</ul>


<p>At this point the site will block, waiting for the web-server to respond with
an image. If, as in this case, it&rsquo;s the local server, you may not even notice
because it&rsquo;s so quick, but if your site is under load, or the images are being
loaded from a CDN there could be a significant delay introduced. Let&rsquo;s find the
code that caused this &ndash; the best place to start looking is the .phtml file
nearest to the network request.</p>

<p>I can&rsquo;t show the full file, but a careful read through of the file shows that
we&rsquo;re passing a URL to a function expecting a file page. Due to PHP&rsquo;s <a href="http://php.net/manual/en/wrappers.php">fopen()
wrappers</a> you can actually do this, and your code will run, and you may not
even realise the resources that are being consumed behind the scenes.</p>

<figure class='code'><figcaption><span>&#8220;Accidental fopen() wrapper usage&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='php'><span class='line'><span class="cp">&lt;?php</span>
</span><span class='line'><span class="nv">$img</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span>
</span><span class='line'>  <span class="s1">&#39;zoomImg&#39;</span>       <span class="o">=&gt;</span> <span class="nv">$_image</span><span class="o">-&gt;</span><span class="na">getData</span><span class="p">(</span><span class="s1">&#39;url&#39;</span><span class="p">),</span>
</span><span class='line'>  <span class="s1">&#39;zoomImgPath&#39;</span><span class="o">=&gt;</span> <span class="nv">$_image</span><span class="o">-&gt;</span><span class="na">getData</span><span class="p">(</span><span class="s1">&#39;path&#39;</span><span class="p">),</span>
</span><span class='line'><span class="p">);</span>
</span><span class='line'><span class="nv">$img</span><span class="p">[</span><span class="s1">&#39;size&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="nb">getimagesize</span><span class="p">(</span><span class="nv">$img</span><span class="p">[</span><span class="s1">&#39;zoomImg&#39;</span><span class="p">]);</span>
</span></code></pre></td></tr></table></div></figure>


<p>Changing that last <code>zoomImg</code> to <code>zoomImgPath</code> resolves the issue &ndash; the results
on the front-end are the same, but no extra requests are made to the
web-server. This frees up more connections for legitimate users and has removed
a potential source for latency.</p>

<p>As with any micro-benchmark like this, it&rsquo;s important to check that the gains
that you&rsquo;re making are significant in the end, and also very important to check
that in your production environment you see an improvement and no regressions.
In this case, you would probably run a <a href="http://www.euperia.com/linux/tools-and-utilities/speed-testing-your-website-with-siege-part-one/720">siege</a> against the site before and
after the change to verify that the improvements are valid when running under
an actual web server.</p>

<p>If you&rsquo;ve found this useful, please let me know on Twitter via <a href="http://twitter.com/insom">@insom</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Magento to WebSockets, via Redis, Node and Juggernaut]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/02/24/magento-to-websockets/"/>
    <updated>2012-02-24T23:02:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/02/24/magento-to-websockets</id>
    <content type="html"><![CDATA[<p>I tried to squeeze more hipster web development tech into the title, but we&rsquo;ll
have to just make do with the above.</p>

<p>As our developers are building <a href="http://www.magentocommerce.com/">Magento</a> sites, I though it would be useful
to get to know it better. We know Magento from an operations perspective; it
likes a lot of IOs and CPU, runs well with ephemeral data stored on RAM disk,
needs a tuned APC etc., but I like to understand as much of the stack that
I look after as possible.</p>

<!--more-->


<p>I&rsquo;m a recovered PHP developer, so this wasn&rsquo;t too hard. Magento&rsquo;s source is
very OO and the PEAR-style autoloader means that files are mostly in obvious
places. We&rsquo;re going to define <code>Iweb_Redis_Observer</code> &ndash; this will appear in
<code>app/code/local/Iweb/Redis/Observer.php</code>. We&rsquo;re also going to define an
<code>Iweb_Redis</code> plugin, so the configs will live in
<code>app/etc/modules/Iweb_Redis.xml</code> and <code>app/code/local/Iweb/Redis/etc/config.xml</code>.</p>

<p>When adding passive hooks into Magento (logging, for example) you can use the
<a href="http://www.magentocommerce.com/wiki/5_-_modules_and_development/0_-_module_development_in_magento/customizing_magento_using_event-observer_method">Event/Observer method documented on the Magento wiki</a> &ndash; various
parts of the core fire events without having a tight binding to what is
receiving them. Classes register their interest in events via XML files. There
are a lot of XML files in Magento.</p>

<p>File /Users/aaron/Documents/Repo/iwebhosting-octopress/source/downloads/code/magento/Iweb_Redis.xml could not be found</p>

<p>File /Users/aaron/Documents/Repo/iwebhosting-octopress/source/downloads/code/magento/config.xml could not be found</p>

<p>File /Users/aaron/Documents/Repo/iwebhosting-octopress/source/downloads/code/magento/Observer.php could not be found</p>

<p>Simple, isn&rsquo;t it? We&rsquo;ll also need <a href="https://github.com/colinmollenhour/credis">Credis</a> put somewhere the autoloader can
find it. Cloning <code>git://github.com/colinmollenhour/credis.git</code> into
<code>app/code/local/Credis</code> will do it.</p>

<p>Then <a href="https://github.com/maccman/juggernaut/blob/master/README.md">install Juggernaut</a>, install Redis (<code>apt-get install redis-server</code>)
and visit <a href="http://your.server:8080/">http://your.server:8080/</a> for the basic Juggernaut console:</p>

<p><img src="http://o7.no/whqDZi" alt="Juggernaut Console" /></p>

<p>If you&rsquo;ve found this useful or have any comments, please let me know on Twitter
via <a href="http://twitter.com/insom">@insom</a>. (If you&rsquo;re looking for <a href="http://www.iweb.co.uk/">specialists in Magento</a>, I should point
you towards <a href="http://www.iwebsolutions.co.uk/">our parent company</a>).</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Lunchtime Project: Apache Space Core Module]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/02/08/lunchtime-project-apache-space-core-module/"/>
    <updated>2012-02-08T21:08:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/02/08/lunchtime-project-apache-space-core-module</id>
    <content type="html"><![CDATA[<p>Show and tell time!</p>

<p><a href="http://twitter.com/DazWorrall">Darren</a> mentioned in passing how we should return quotes in the HTTP
responses for our new product. <a href="http://keithdevens.com/weblog/archive/2002/Jul/20/FuturamaQuotesHTTPHeaders">Slashdot already does this</a> with
Futurama quotes, and it didn&rsquo;t take long to add support for the main WSGI
application, but in the interests of security the public-facing website doesn&rsquo;t
run any dynamic languages; no Python, no PHP.</p>

<!--more-->


<p>I investigated using <a href="http://httpd.apache.org/docs/current/mod/mod_headers.html">mod_headers</a> and the <a href="http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritemap">RewriteMap</a> functionality of
<a href="http://httpd.apache.org/docs/current/mod/mod_rewrite.html">mod_rewrite</a>, but even if that <em>would</em> work the random selection built
into RewriteMap requires that all of the quotes be on one line. Unfortunately,
in our case <a href="http://theportalwiki.com/wiki/Core_voice_lines#Space_core_lines">the source material</a> is 88 lines long.</p>

<p>So, on a box with just Apache, there&rsquo;s just one reasonable choice: learn how to
write Apache modules. <a href="http://stderr.net/apache/rpaf/">mod_rpaf</a> is just about the simplest module I
know of. It tweaks the inbound headers with code like this:</p>

<figure class='code'><figcaption><span>&#8220;Add X-Forwarded-For&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">if</span> <span class="p">(</span><span class="n">is_in_array</span><span class="p">(</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">connection</span><span class="o">-&gt;</span><span class="n">remote_ip</span><span class="p">,</span> <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">proxy_ips</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">if</span> <span class="p">(</span><span class="n">fwdvalue</span> <span class="o">=</span> <span class="n">apr_table_get</span><span class="p">(</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">headers_in</span><span class="p">,</span> <span class="s">&quot;X-Forwarded-For&quot;</span><span class="p">))</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">apr_array_header_t</span> <span class="o">*</span><span class="n">arr</span> <span class="o">=</span> <span class="n">apr_array_make</span><span class="p">(</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">pool</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">));</span>
</span></code></pre></td></tr></table></div></figure>


<p>Building and installing a standalone module is incredibly simple if you have
the apache2.2-dev package installed (or equivalent for your distribution):</p>

<pre><code>apxs2 -i -a -c *.c
echo "LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf.so" &gt;&gt; /etc/apache2/httpd.conf
</code></pre>

<p>Lets look at the major parts:</p>

<figure class='code'><figcaption><span>&#8220;Boilerplate Config&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="n">module</span> <span class="n">AP_MODULE_DECLARE_DATA</span> <span class="n">rpaf_module</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">enable</span><span class="p">;</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">sethostname</span><span class="p">;</span>
</span><span class='line'>    <span class="n">apr_array_header_t</span> <span class="o">*</span><span class="n">proxy_ips</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span> <span class="n">rpaf_server_cfg</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">static</span> <span class="kt">void</span> <span class="o">*</span><span class="nf">rpaf_create_server_cfg</span><span class="p">(</span><span class="n">apr_pool_t</span> <span class="o">*</span><span class="n">p</span><span class="p">,</span> <span class="n">server_rec</span> <span class="o">*</span><span class="n">s</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">rpaf_server_cfg</span> <span class="o">*</span><span class="n">cfg</span> <span class="o">=</span> <span class="p">(</span><span class="n">rpaf_server_cfg</span> <span class="o">*</span><span class="p">)</span><span class="n">apr_pcalloc</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">rpaf_server_cfg</span><span class="p">));</span>
</span><span class='line'>    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">cfg</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">proxy_ips</span> <span class="o">=</span> <span class="n">apr_array_make</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="kt">char</span> <span class="o">*</span><span class="p">));</span>
</span><span class='line'>    <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">enable</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>    <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">sethostname</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span><span class="n">cfg</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Seems reasonable, we define a one-off struct, allocate it in the server config
callback and set some defaults. The RPAF code uses and apr_array, but they&rsquo;re
not the easiest thing to iterate over, according to some <a href="http://dev.ariel-networks.com/apr/apr-tutorial/html/apr-tutorial-19.html">APR tutorial</a> that
I found. However, a <a href="http://dev.ariel-networks.com/apr/apr-tutorial/html/apr-tutorial-19.html#ss19.4">ring</a> looks ideal: we can loop over it any random amount
and be confident that we won&rsquo;t walk off of the end, so we don&rsquo;t even need to
keep a count of how many items are in it. (Though, we do)</p>

<figure class='code'><figcaption><span>&#8220;Initialisation, Callbacks&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">static</span> <span class="k">const</span> <span class="n">command_rec</span> <span class="n">rpaf_cmds</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// SNIP - Aaron</span>
</span><span class='line'>    <span class="n">AP_INIT_ITERATE</span><span class="p">(</span>
</span><span class='line'>                    <span class="s">&quot;RPAFproxy_ips&quot;</span><span class="p">,</span>
</span><span class='line'>                    <span class="n">rpaf_set_proxy_ip</span><span class="p">,</span>
</span><span class='line'>                    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>                    <span class="n">RSRC_CONF</span><span class="p">,</span>
</span><span class='line'>                    <span class="s">&quot;IP(s) of Proxy server setting X-Forwarded-For header&quot;</span>
</span><span class='line'>                    <span class="p">),</span>
</span><span class='line'>    <span class="p">{</span> <span class="nb">NULL</span> <span class="p">}</span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="k">static</span> <span class="kt">void</span> <span class="nf">register_hooks</span><span class="p">(</span><span class="n">apr_pool_t</span> <span class="o">*</span><span class="n">p</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">ap_hook_post_read_request</span><span class="p">(</span><span class="n">change_remote_ip</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">APR_HOOK_MIDDLE</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="n">module</span> <span class="n">AP_MODULE_DECLARE_DATA</span> <span class="n">rpaf_module</span> <span class="o">=</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">STANDARD20_MODULE_STUFF</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="n">rpaf_create_server_cfg</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="n">rpaf_cmds</span><span class="p">,</span>
</span><span class='line'>    <span class="n">register_hooks</span><span class="p">,</span>
</span><span class='line'><span class="p">};</span>
</span></code></pre></td></tr></table></div></figure>


<p>Okay, everything&rsquo;s still reasonable with this, we&rsquo;re providing an array of
configuration directives and respective callbacks, a function that gets called
to register some hooks, and a struct that gets read when the module is
<code>dlopen()</code>ed pointing to these things.</p>

<p>Our versions of these:</p>

<figure class='code'><figcaption><span>&#8220;Ring structure and Config structure&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">typedef</span> <span class="k">struct</span> <span class="n">_quote_t</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">APR_RING_ENTRY</span><span class="p">(</span><span class="n">_quote_t</span><span class="p">)</span> <span class="n">link</span><span class="p">;</span>
</span><span class='line'>    <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">quote</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span> <span class="n">quote_t</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">typedef</span> <span class="k">struct</span> <span class="n">_quote_ring_t</span> <span class="n">quote_ring_t</span><span class="p">;</span>
</span><span class='line'><span class="n">APR_RING_HEAD</span><span class="p">(</span><span class="n">_quote_ring_t</span><span class="p">,</span> <span class="n">_quote_t</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'><span class="n">module</span> <span class="n">AP_MODULE_DECLARE_DATA</span> <span class="n">space_core_module</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">quote_ring_t</span> <span class="o">*</span><span class="n">ring</span><span class="p">;</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">count</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span> <span class="n">space_core_server_cfg</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="k">static</span> <span class="kt">void</span> <span class="o">*</span><span class="nf">space_core_create_server_cfg</span><span class="p">(</span><span class="n">apr_pool_t</span> <span class="o">*</span><span class="n">p</span><span class="p">,</span> <span class="n">server_rec</span> <span class="o">*</span><span class="n">s</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="n">cfg</span> <span class="o">=</span> <span class="p">(</span><span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="p">)</span><span class="n">apr_pcalloc</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">space_core_server_cfg</span><span class="p">));</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">cfg</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">ring</span> <span class="o">=</span> <span class="n">apr_palloc</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">quote_ring_t</span><span class="p">));</span>
</span><span class='line'>    <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">count</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span><span class='line'>    <span class="n">APR_RING_INIT</span><span class="p">(</span><span class="n">cfg</span><span class="o">-&gt;</span><span class="n">ring</span><span class="p">,</span> <span class="n">_quote_t</span><span class="p">,</span> <span class="n">link</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span><span class="n">cfg</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>quote_ring_t</code> will be the container for a bunch of <code>quote_t</code>s which themselves
include a pointer to their next link and a string (for the quote).
<code>space_core_server_cfg</code> will be just a count of the number of quotes and the
ring structure itself. Suitable macro hackery is going on under <code>APR_RING_INIT</code>
&ndash; I don&rsquo;t even <em>want</em> to know what.</p>

<figure class='code'><figcaption><span>&#8220;Configuration and Initialisation&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">static</span> <span class="k">const</span> <span class="n">command_rec</span> <span class="n">space_core_cmds</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">AP_INIT_TAKE1</span><span class="p">(</span>
</span><span class='line'>                    <span class="s">&quot;SpaceCoreFile&quot;</span><span class="p">,</span>
</span><span class='line'>                    <span class="n">space_core_file</span><span class="p">,</span>
</span><span class='line'>                    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>                    <span class="n">RSRC_CONF</span><span class="p">,</span>
</span><span class='line'>                    <span class="s">&quot;Location of Quotes File&quot;</span>
</span><span class='line'>                    <span class="p">),</span>
</span><span class='line'>    <span class="p">{</span> <span class="nb">NULL</span> <span class="p">}</span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="k">static</span> <span class="kt">void</span> <span class="nf">register_hooks</span><span class="p">(</span><span class="n">apr_pool_t</span> <span class="o">*</span><span class="n">p</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">ap_hook_fixups</span><span class="p">(</span><span class="n">ap_headers_fixup</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="n">APR_HOOK_LAST</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="n">module</span> <span class="n">AP_MODULE_DECLARE_DATA</span> <span class="n">space_core_module</span> <span class="o">=</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">STANDARD20_MODULE_STUFF</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="n">space_core_create_server_cfg</span><span class="p">,</span>
</span><span class='line'>    <span class="nb">NULL</span><span class="p">,</span>
</span><span class='line'>    <span class="n">space_core_cmds</span><span class="p">,</span>
</span><span class='line'>    <span class="n">register_hooks</span><span class="p">,</span>
</span><span class='line'><span class="p">};</span>
</span></code></pre></td></tr></table></div></figure>


<p>This is a fairly straight swap: lots of things and renamed to <code>space_core</code>, the
hook we register is <code>ap_hook_fixups</code>, a suitable place to manipulate the
response. The possible hooks are documented <a href="http://httpd.apache.org/docs/2.0/developer/modules.html#messy">in this guide</a>. We defined
just one parameter <code>SpaceCoreFile</code> which should be the absolute filename of the
quotes file.</p>

<p>We&rsquo;re using <code>AP_INIT_TAKE1</code> instead of <code>AP_INIT_ITERATE</code>; this is because
mod_rpaf takes a list of IPs, but we really just want one filename. This is
expanded on <a href="http://www.apachetutor.org/dev/config">in this Apache tutorial</a> (wow, remember webpages like this?).</p>

<h2>The good stuff!</h2>

<p>Most of this has been fairly boilerplate, this is where the (admittedly small)
magic happens:</p>

<figure class='code'><figcaption><span>&#8220;Load the File in&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">static</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="nf">space_core_file</span><span class="p">(</span><span class="n">cmd_parms</span> <span class="o">*</span><span class="n">cmd</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">dummy</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">server_rec</span> <span class="o">*</span><span class="n">s</span> <span class="o">=</span> <span class="n">cmd</span><span class="o">-&gt;</span><span class="n">server</span><span class="p">;</span>
</span><span class='line'>    <span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="n">cfg</span> <span class="o">=</span> <span class="p">(</span><span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="p">)</span><span class="n">ap_get_module_config</span><span class="p">(</span><span class="n">s</span><span class="o">-&gt;</span><span class="n">module_config</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">space_core_module</span><span class="p">);</span>
</span><span class='line'>    <span class="kt">char</span> <span class="n">buf</span><span class="p">[</span><span class="mi">1024</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="mi">0</span><span class="p">};</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">apr_file_t</span> <span class="o">*</span><span class="n">fd</span><span class="p">;</span>
</span><span class='line'>    <span class="n">apr_file_open</span><span class="p">(</span><span class="o">&amp;</span><span class="n">fd</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">APR_READ</span><span class="p">,</span> <span class="n">APR_OS_DEFAULT</span><span class="p">,</span> <span class="n">cmd</span><span class="o">-&gt;</span><span class="n">pool</span><span class="p">);</span>
</span><span class='line'>    <span class="k">while</span><span class="p">(</span><span class="n">APR_EOF</span> <span class="o">!=</span> <span class="n">apr_file_gets</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="mi">1024</span><span class="p">,</span> <span class="n">fd</span><span class="p">))</span> <span class="p">{</span>
</span><span class='line'>        <span class="k">if</span><span class="p">(</span><span class="n">strlen</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="n">buf</span><span class="p">[</span><span class="n">strlen</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="sc">&#39;\0&#39;</span><span class="p">;</span>
</span><span class='line'>            <span class="n">quote_t</span> <span class="o">*</span><span class="n">elem</span> <span class="o">=</span> <span class="n">apr_palloc</span><span class="p">(</span><span class="n">cmd</span><span class="o">-&gt;</span><span class="n">pool</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">quote_t</span><span class="p">));</span>
</span><span class='line'>            <span class="n">elem</span><span class="o">-&gt;</span><span class="n">quote</span> <span class="o">=</span> <span class="n">apr_pstrdup</span><span class="p">(</span><span class="n">cmd</span><span class="o">-&gt;</span><span class="n">pool</span><span class="p">,</span> <span class="n">buf</span><span class="p">);</span>
</span><span class='line'>            <span class="n">APR_RING_INSERT_TAIL</span><span class="p">(</span><span class="n">cfg</span><span class="o">-&gt;</span><span class="n">ring</span><span class="p">,</span> <span class="n">elem</span><span class="p">,</span> <span class="n">_quote_t</span><span class="p">,</span> <span class="n">link</span><span class="p">);</span>
</span><span class='line'>            <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">count</span><span class="o">++</span><span class="p">;</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>We&rsquo;ll create a staticly allocated buffer, filled with NULs. We&rsquo;ll trust that
the filename is good, because I don&rsquo;t check any return values. <code>apr_file_gets</code>
is a portable version of <code>fgets</code> which respects maxium lengths.</p>

<p>If the line has more than one character in it (including the newline, so
essentially: if it&rsquo;s not blank) then we strip off the trailing newline,
duplicate the string and add it to a fresh <code>quote_t</code> structure.</p>

<p><code>APR_RING_INSERT_TAIL</code> does some more magic macro work to expand the ring with
the new element, and we increment the quote count.</p>

<p>Pulling a quote out should be a simple case of picking a random number,
advancing through the ring a few times and then setting it in the response:</p>

<figure class='code'><figcaption><span>&#8220;Set the Header&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='c'><span class='line'><span class="k">static</span> <span class="kt">int</span> <span class="nf">ap_headers_fixup</span><span class="p">(</span><span class="n">request_rec</span> <span class="o">*</span><span class="n">r</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="n">cfg</span> <span class="o">=</span> <span class="p">(</span><span class="n">space_core_server_cfg</span> <span class="o">*</span><span class="p">)</span><span class="n">ap_get_module_config</span><span class="p">(</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">server</span><span class="o">-&gt;</span><span class="n">module_config</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">space_core_module</span><span class="p">);</span>
</span><span class='line'>    <span class="kt">int</span> <span class="n">rand</span> <span class="o">=</span> <span class="n">random</span><span class="p">()</span> <span class="o">%</span> <span class="n">cfg</span><span class="o">-&gt;</span><span class="n">count</span><span class="p">;</span>
</span><span class='line'>    <span class="n">quote_t</span><span class="o">*</span> <span class="n">elem</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">for</span><span class="p">(</span><span class="n">elem</span> <span class="o">=</span> <span class="n">APR_RING_FIRST</span><span class="p">(</span><span class="n">cfg</span><span class="o">-&gt;</span><span class="n">ring</span><span class="p">);</span> <span class="n">elem</span> <span class="o">&amp;&amp;</span> <span class="n">rand</span><span class="p">;</span> <span class="n">elem</span> <span class="o">=</span> <span class="n">APR_RING_NEXT</span><span class="p">(</span><span class="n">elem</span><span class="p">,</span> <span class="n">link</span><span class="p">))</span>
</span><span class='line'>        <span class="n">rand</span><span class="o">--</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">if</span><span class="p">(</span><span class="n">elem</span><span class="p">)</span>
</span><span class='line'>        <span class="n">apr_table_set</span><span class="p">(</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">headers_out</span><span class="p">,</span> <span class="s">&quot;X-Space-Core&quot;</span><span class="p">,</span> <span class="n">elem</span><span class="o">-&gt;</span><span class="n">quote</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="n">DECLINED</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>And that&rsquo;s it! There&rsquo;s a <a href="https://gist.github.com/1774452">gist with the finished code</a> with the whole
file, and I&rsquo;ve packaged it up (for fun) <a href="https://launchpad.net/~bradya/+archive/space-core/">as a .deb for Ubuntu Lucid</a> &ndash; just</p>

<pre><code>apt-get install libapache2-mod-space-core
a2enmod space_core
/etc/init.d/apache2 reload
</code></pre>

<p>and you&rsquo;re done!</p>

<pre><code>$ curl -I http://localhost/
HTTP/1.1 200 OK
Date: Wed, 08 Feb 2012 22:17:04 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Space-Core: What’s your favorite thing about space? Mine is space.
Last-Modified: Tue, 07 Feb 2012 22:18:12 GMT
ETag: "124bde-b1-4b867279581c1"
</code></pre>

<p>This could easily be rounded out as a generate quotes module with a few more
configuration directives: if you&rsquo;ve found this useful, please let me know
on Twitter via <a href="http://twitter.com/insom">@insom</a></p>

<p><img src="http://i0.kym-cdn.com/photos/images/original/000/118/802/halolz-dot-com-portal2-nyancatspacecore.gif" alt="Nyan Core" /></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Using BGP to serve High-Availability DNS]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/01/27/using-bgp-to-serve-high-availability-dns/"/>
    <updated>2012-01-27T22:48:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/01/27/using-bgp-to-serve-high-availability-dns</id>
    <content type="html"><![CDATA[<p>Routing protocols, like BGP and OSPF, can be used for more than just
establishing connectivity. The best-known example of this is probably using BGP
to blackhole DoS origins, <a href="http://packetlife.net/blog/2009/jul/6/remotely-triggered-black-hole-rtbh-routing/">Remotely Triggered Blackhole Routing</a>.</p>

<p>Another use is to provide failover at layer 3, as an alternative to services
like <a href="http://www.linux-ha.org/wiki/Main_Page">Heartbeat</a>. This post will explain how to use BGP to provide a highly
available recursive DNS service.</p>

<!--more-->


<h4> Rationale</h4>

<p>Why HA DNS? DNS is already able to handle outages if you specify more than one
name server. Unfortunately, while true, in practice this breaks down. There can
be a significant timeout before rolling over to the next resolver, a timeout
which in many cases is longer than other application-level ones.</p>

<p>The default on Ubuntu is 5 seconds, the same value as Varnish&rsquo;s default &lsquo;first
byte&rsquo; timeout- in the simplest case this means that all database connections,
reverse lookups or connections to web services could add 5 seconds <em>each</em> to
the rendering time of a page. Even if you avoid the request failing, it will be
significantly slower than if all of your resolvers were available.</p>

<h4> Set up</h4>

<p>For this example, there will be two machines serving DNS in an active/active
set up. In the event that one of the machines fails then both sets of IPs will
be handled by the single machine left. <a href="http://www.iweb.co.uk/">iWeb</a> deploys Ubuntu almost
exclusively, so you&rsquo;ll need to mentally translate if you&rsquo;re using another
distribution.</p>

<pre><code>192.168.0.0/29:
  192.168.0.1 - Border Router
  192.168.0.2 - Machine A's Real IP
  192.168.0.3 - Machine B's Real IP
  192.168.0.4 - First Virtual IP
  192.168.0.5 - Second Virtual IP
192.168.0.8/29:
  192.168.0.9 - Border Router (Area 0)
  192.168.0.10 - A Client
</code></pre>

<h4>Network Bit</h4>

<p>Configure the networking like below. Machine B is the same as Machine A except
for its real IP; replace 192.168.0.2 with 192.168.0.3.</p>

<figure class='code'><figcaption><span>&#8220;Machine A -</span><a href='http://iwebhosting.github.io/etc/network/interfaces"'>link</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>auto lo
</span><span class='line'>iface lo inet loopback
</span><span class='line'>
</span><span class='line'>auto lo:0
</span><span class='line'>iface lo:0 inet static
</span><span class='line'>address 192.168.0.4
</span><span class='line'>netmask 255.255.255.255
</span><span class='line'>
</span><span class='line'>auto lo:1
</span><span class='line'>iface lo:1 inet static
</span><span class='line'>address 192.168.0.5
</span><span class='line'>netmask 255.255.255.255
</span><span class='line'>
</span><span class='line'>auto eth0
</span><span class='line'>iface eth0 inet static
</span><span class='line'>address 192.168.0.2
</span><span class='line'>netmask 255.255.255.248
</span><span class='line'>gateway 192.168.0.1</span></code></pre></td></tr></table></div></figure>


<p>As both of the machines&#8217; <code>eth0</code> interfaces are in the same LAN they will both
clash by sending out ARP responses for the same IPs. We can tell the kernel to
only answer ARP requests which match the interface the request came in from:</p>

<figure class='code'><figcaption><span>&#8220;/etc/sysctl.conf&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>net.ipv4.conf.eth0.arp_ignore <span class="o">=</span> 1
</span><span class='line'>net.ipv4.conf.eth0.arp_announce <span class="o">=</span> 2
</span></code></pre></td></tr></table></div></figure>


<p>Reload with <code>sysctl -p</code>.</p>

<h4>DNS Bit</h4>

<p>The basic DNS install is uneventful: install the daemons and <em>make sure</em> to
lock down the recursive resolver to your network. I&rsquo;m assuming that you
want to serve all of the hypothetical 192.168.0.0/24 block.</p>

<pre><code>apt-get install bind9
vi /etc/bind/named.conf.options
# add the line
#      allow-query-cache { 192.168.0.0/24; }; 
# inside the options stanza
/etc/init.d/bind9 restart
</code></pre>

<h4>BGP Bit</h4>

<p>We&rsquo;re going to use <a href="http://code.google.com/p/exabgp/">exabgp</a> by <a href="http://thomas.mangin.com/">Thomas Mangin</a> on the DNS server side &ndash;
it&rsquo;s a pure-Python BGP implementation that doesn&rsquo;t depend on too much. On the
router we&rsquo;re going to use <a href="http://www.quagga.net/">Quagga</a>, though the syntax should basically be
identical for IOS.</p>

<p>It&rsquo;s not packaged <em>for Lucid</em>, so we&rsquo;ll install using <code>pip</code>.</p>

<pre><code>apt-get install python-pip
pip install exabgp
</code></pre>

<p>That puts it in <code>/usr/local/bin/exabgp</code>, we&rsquo;ll need to supply our own config
file and be responsible for launching it. Here&rsquo;s an example <code>upstart</code> job:</p>

<figure class='code'> <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>start on runlevel <span class="o">[</span>2345<span class="o">]</span>
</span><span class='line'>stop on runlevel <span class="o">[</span>!2345<span class="o">]</span>
</span><span class='line'>respawn
</span><span class='line'><span class="nb">exec</span> /usr/local/bin/exabgp /etc/bgp.conf
</span></code></pre></td></tr></table></div></figure>


<p><em>Update:</em> ExaBGP is, infact, packaged in Oneiric and Precise, though it&rsquo;s the
previous stable release.</p>

<blockquote class="twitter-tweet"><p>Ubuntu users will find ExaBGP on Oneiric <a href="http://t.co/3pXamdn3" title="http://packages.ubuntu.com/hu/oneiric/net/exabgp">packages.ubuntu.com/hu/oneiric/net…</a></p>&mdash; ExaBGP (@exabgp) <a href="https://twitter.com/exabgp/status/148911146781523968" data-datetime="2011-12-19T23:42:38+00:00">December 19, 2011</a></blockquote>


<script src="http://iwebhosting.github.io//platform.twitter.com/widgets.js" charset="utf-8"></script>


<p>Our upstream router advertises its connected routes into OSPF area 0. Because
it has 192.168.0.1/29 as an interface it will advertise this whole network, and we
won&rsquo;t need to redistribute BGP into OSPF. Something like the below Quagga
config will do. This is not a complete Quagga config, you should follow another
guide to get basic routing up and running.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>router bgp 64512
</span><span class='line'> bgp router-id 192.168.0.1
</span><span class='line'> neighbor 192.168.0.2 remote-as 64512
</span><span class='line'> neighbor 192.168.0.2 timers 5 10
</span><span class='line'> neighbor 192.168.0.3 remote-as 64512
</span><span class='line'> neighbor 192.168.0.3 timers 5 10 
</span><span class='line'>!
</span><span class='line'>router ospf
</span><span class='line'> ospf router-id 192.168.0.9
</span><span class='line'> redistribute connected metric 60
</span><span class='line'> network 192.168.0.8/29 area 0.0.0.0
</span><span class='line'>!</span></code></pre></td></tr></table></div></figure>


<p>AS numbers above 64511 are reserved for private use, so you can use this freely.</p>

<p>This will advertise 192.168.0.0/29 into area 0 unconditionally. We have set up
both of the &lsquo;real&rsquo; IPs as iBGP peers, and set very short values for the
keep-alive and hold-time timers. We will advertise ourselves every five seconds,
and if we don&rsquo;t hear from <code>exabgp</code> every 10 seconds we will consider that
path to be dead.</p>

<p>The defaults are 60 and 180 seconds respectively, but that would lead to a
noticable outage, though of course YMMV.</p>

<p>For the <code>exabgp</code> side, use a config like this for Machine A:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>neighbor 192.168.0.2 {
</span><span class='line'>    router-id 192.168.0.2;
</span><span class='line'>    local-address 192.168.0.2;
</span><span class='line'>    local-as 64512;
</span><span class='line'>    peer-as 64512;
</span><span class='line'>    hold-time 5;
</span><span class='line'>    static {
</span><span class='line'>        route 192.168.0.4/32 next-hop 192.168.0.2 local-preference 200;
</span><span class='line'>        route 192.168.0.5/32 next-hop 192.168.0.2 local-preference 150;
</span><span class='line'>    }
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>And this for Machine B:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>neighbor 192.168.0.3 {
</span><span class='line'>    router-id 192.168.0.3;
</span><span class='line'>    local-address 192.168.0.3;
</span><span class='line'>    local-as 64512;
</span><span class='line'>    peer-as 64512;
</span><span class='line'>    hold-time 5;
</span><span class='line'>    static {
</span><span class='line'>        route 192.168.0.4/32 next-hop 192.168.0.3 local-preference 150;
</span><span class='line'>        route 192.168.0.5/32 next-hop 192.168.0.3 local-preference 200;
</span><span class='line'>    }
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>We&rsquo;re not learning any routes, we&rsquo;re exclusively advertising routes. Machine A
will advertise both routes, with a higher priority for the first VIP, and
Machine B will do the same with a higher priority for the second VIP. Local
preference seems to be more effective than weight or metrics for bullying iBGP
into doing what you want.</p>

<p>On each machine, start the BGP process and watch tcpdump while you ping the
virtual IPs.</p>

<pre><code>start bgp
tcpdump -ni any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
23:44:59.389054 IP 192.168.0.10 &gt; 192.168.0.4: ICMP echo request, id 40574, seq 1, length 64
23:44:59.389256 IP 192.168.0.4 &gt; 192.168.0.10: ICMP echo reply, id 40574, seq 1, length 64
</code></pre>

<p>You should see the first VIP pings appear on Machine A, and the second VIP
pings on Machine B. Keep those pings going.</p>

<p>By stopping the BGP service on either box you should see the hand-over happen
with no missed pings. If you force a failure of one of the machines, it will be
within seconds:</p>

<pre><code>iptables -I INPUT -i eth0 -j DROP
</code></pre>

<p>then:</p>

<pre><code>PING 91.208.170.131 (91.208.170.131) 56(84) bytes of data.
64 bytes from 91.208.170.131: icmp_seq=1 ttl=63 time=1.02 ms
64 bytes from 91.208.170.131: icmp_seq=2 ttl=63 time=0.890 ms
64 bytes from 91.208.170.131: icmp_seq=3 ttl=63 time=4.23 ms
[ Four missed pings ]
64 bytes from 91.208.170.131: icmp_seq=8 ttl=63 time=0.956 ms
64 bytes from 91.208.170.131: icmp_seq=9 ttl=63 time=1.94 ms
</code></pre>

<p>Hopefully this has been instructive. If you have any comments, please direct
them to my Twitter account, <a href="http://twitter.com/insom">@insom</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Where we are]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/01/23/where-we-are/"/>
    <updated>2012-01-23T15:41:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/01/23/where-we-are</id>
    <content type="html"><![CDATA[<p>A mini <a href="http://en.wikipedia.org/wiki/State_of_the_Union_address">state of the union</a>, if you like. I thought it would be worthwhile to outline where we are at the moment, and touch a little on whats coming up.</p>

<!--more-->


<p>iWeb has been in business since 1995 designing and developing websites &ndash; mainly in PHP &ndash; and over time the hosting side of business has grown from a few sites running on a handful of servers into a multi datacenter operation, spanning hundreds of virtual servers.</p>

<p>Managing an infrastructure that large by hand isn&rsquo;t particularly feasible, so we have been using <a href="http://puppetlabs.com/">Puppet</a> for our configuration management, driven by our custom built CMDB/inventory system, which is a <a href="https://www.djangoproject.com/">Django</a> web app. It&rsquo;s about 18 months old now, and it&rsquo;s been instrumental in allowing us to scale our infrastructure &ndash; we&rsquo;ve moved away from shared hosting (one of the motivators to increase our automation), and deployed a lot of new machines, all managed by puppet. We&rsquo;ve learned a lot, and are planning a &lsquo;version 2&rsquo; of the web app this year, which I&rsquo;m sure we&rsquo;ll be blogging about.</p>

<p>In terms of monitoring, we&rsquo;ve experimented a lot since I started here. It&rsquo;s a <a href="https://twitter.com/#!/search/%23monitoringsucks">hot topic</a> everywhere at the moment, and iWeb is just the same. Early on we used <a href="http://munin-monitoring.org/">Munin</a>, which we had trouble scaling &ndash; no so much with the number of hosts, but with the data resolution we were looking for. We switched to <a href="http://collectd.org/">collectd</a> and its been great &ndash; extremely lightweight, even with a 10 second resolution. It has a number of <a href="http://collectd.org/wiki/index.php/Category:Generic_Plugins">generic plugins</a>, which are fantastic for collecting all sorts of application specific metrics (like <a href="http://collectd.org/wiki/index.php/Plugin:Tail/Config#Exim">exim stats</a>, for example). While its proved great at collecting metrics, collected doesn&rsquo;t provide any UI &ndash; instead it ships with a number of <a href="http://collectd.org/wiki/index.php/Category:Callback_write">write plugins</a> which can be used to persist data (or not, as we&rsquo;ll soon see…). Having used Munin we were used to dealing with RRD&rsquo;s, so we ran with those, and cooked up a simple <a href="http://flask.pocoo.org/">Flask</a> based <a href="https://github.com/iwebhosting/collectd-flask">web interface</a> to display them (though we actually used <a href="http://kenny.belitzky.com/projects/collectd-web">collectd-web</a> to render the graphs). For a time we used it&rsquo;s notification system to power our Icinga alerts, but we stopped that in the end &ndash; our alerting is a whole other blog post :)</p>

<p>Lately we&rsquo;ve actually been sending all our collectd metrics into <a href="http://graphite.wikidot.com/">Graphite</a> &ndash; which is a fantastic tool for combining and visualising metrics. We&rsquo;re also able to collect additional metrics not suited to collectd (the data isn&rsquo;t available for collection every n seconds, or it&rsquo;s expensive to collect that data every n seconds). We may even turn off our RRD generation completely in the near future, Graphite has given us many new ways to visualise what&rsquo;s happening (or what has happened) on our infrastructure. Because we in operations largely know the software stacks running on our customers machines, we are able to combine application metrics with system metrics in a way we haven&rsquo;t been able to before. I suspect this will lead to blog posts :)</p>

<p>Our alerting is one of the areas thats least changed in my time here, at least in terms of the technology. Alert states are maintained by <a href="https://www.icinga.org/">Icinga</a>, and we have written an app to aggregate that data, display interesting state changes and if necessary alert the on call personnel that something is broken. This is an area we have targeted for improvement this year &ndash; there&rsquo;s data locked up in here that can be useful to customers, their project managers and developers. It will be fun coming up with new ways to expose it.</p>

<p>iWeb has a popular <a href="http://www.iweb-ftp.co.uk/">ftp service</a> which has been running for a few years now. It&rsquo;s due a bit of a refresh &ndash; not just from a design point of view, but the current system has grown to the point where is poses a number of engineering challenges, particularly around data storage. We&rsquo;ve been working on the new service since just before Christmas and will be ready to launch soon &ndash; it&rsquo;s particularly exciting for us in operations as &ndash; owing to these engineering challenges &ndash; we are developing the whole stack ourselves, and have designed the components parts to be the platform upon which we will launch a number of new hosted services this year. Not only is the technology interesting here, but the process as well &ndash; we&rsquo;ve built a continuous deployment workflow so that we can release features and fixes as fast as possible. As Aaron mentioned, we&rsquo;ll be blogging about this a bit more in the near future.</p>

<p>So, thats a little bit about where we are, and where we&rsquo;re heading at the moment.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Hubot HTTP Daemon Support]]></title>
    <link href="http://iwebhosting.github.io/blog/2012/01/21/hubot-http-daemon-support/"/>
    <updated>2012-01-21T17:02:00+00:00</updated>
    <id>http://iwebhosting.github.io/blog/2012/01/21/hubot-http-daemon-support</id>
    <content type="html"><![CDATA[<p><a href="http://hubot.github.com/">Hubot</a>, Github&rsquo;s excellent CoffeeScript-based chat bot recently gained
support for running an internal HTTP daemon. Previously, this had to be done by
hand by plugins which expected to receive events from outside sources, like in
this post on <a href="http://tomb.io/posts/hubot-ci-and-deploying/">using Hubot for CI and Deploying</a>, which inspired Darren&rsquo;s
<a href="https://gist.github.com/1494013">fleshed out take</a>.</p>

<!--more-->


<p>This functionality hasn&rsquo;t yet made it into a release yet, and we also use XMPP
internally (instead of Campfire), so we had to make a few tweaks to deploy our
bot on Heroku. Being new to CoffeeScript and NPM (<em>and</em> Heroku) it took some
trial and error, but I just <code>git rebase</code>d it away so it looks like I got it
right first time!</p>

<p>I changed the dependency blob in the JSON to this:</p>

<figure class='code'><figcaption><span>&#8220;Depedencies to run from Hubot and XMPP master&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'>  <span class="s2">&quot;dependencies&quot;</span><span class="o">:</span> <span class="p">{</span>
</span><span class='line'>    <span class="s2">&quot;hubot-xmpp&quot;</span><span class="o">:</span> <span class="s2">&quot;git://github.com/markstory/hubot-xmpp.git&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="s2">&quot;hubot&quot;</span><span class="o">:</span> <span class="s2">&quot;git://github.com/github/hubot&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="s2">&quot;hubot-scripts&quot;</span><span class="o">:</span> <span class="s2">&quot;2.0.2&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="s2">&quot;optparse&quot;</span><span class="o">:</span> <span class="s2">&quot;1.0.3&quot;</span>
</span><span class='line'>  <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Because <a href="https://github.com/github/hubot/blob/master/src/templates/README.md">the HTTP support</a> requires the <code>PORT</code> environment variable (which
Heroku supplies to web apps) you&rsquo;ll need to change your <code>Procfile</code> from app to
web</p>

<pre><code>web: bin/hubot -a xmpp -n Hubot
</code></pre>

<p>which also means changing your scale command to</p>

<pre><code>heroku ps:scale web=1
</code></pre>

<p>There&rsquo;s some <a href="https://github.com/github/hubot/blob/master/src/scripts/httpd.coffee">example HTTP calls on master</a>, and (unlike the above
approaches) Hubot also includes Connect&rsquo;s <a href="http://www.senchalabs.org/connect/middleware-bodyParser.html">bodyParser</a> which Does the Right
Thing&trade; with POST bodies, whether they are URL encoded forms or JSON.</p>

<div><script src='https://gist.github.com/1633778.js'></script>
<noscript><pre><code></code></pre></noscript></div>


<p>The above is enough for us to use Hubot like a puppet, posting the room and the
message and having it repeat it in the room:</p>

<figure class='code'><figcaption><span>&#8220;Make Hubot speak&#8221; </span></figcaption>
 <div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>curl -d <span class="nv">room</span><span class="o">=</span>yourroom@conference.iweb.co.uk <span class="se">\</span>
</span><span class='line'>  -d <span class="nv">message</span><span class="o">=</span>Hello http://appname.herokuapp.com/hubot/say
</span></code></pre></td></tr></table></div></figure>


<p>So far, I&rsquo;ve mostly been using it to fake the bot being more clever than it is</p>

<pre><code>&lt;Hubot&gt; [CAKE] Upstairs, via @jellis
</code></pre>

<p>but with our CI workflow (upcoming post!) it will start to be more genuinely
useful.</p>
]]></content>
  </entry>
  
</feed>