Leigh Brenecki urn:uuid:46EACC73-1BFB-484C-88EB-D9F828734035 2021-09-16T10:14:01Z I made a fake COVID-19 vaccine certificate https://leigh.net.au/writing/fake-covid-pass/ 2021-09-15T00:00:00+0930 2021-09-15T00:00:00+0930 <p>Here it is. It took me about five minutes.</p> <p><img alt="" src="/assets/articles/fake-covid-pass/fronts.png" /></p> <p><em>Left to right: the forgery I created, my genuine COVID-19 certificate, and the <a href="https://blog.yaakov.online/forging-covid-19-vaccination-certificates-is-childs-play/">forgery Yaakov created</a>.</em></p> <h1 id="how-apple-wallet-passes-work-and-how-i-forged-mine">How Apple Wallet passes work, and how I forged mine</h1> <p>This is the technical bit. If you're just interested in what this means and how it could be fixed, skip to <em>How the government could fix the problem</em>. </p> <p>An Apple Wallet pass is just a Zip file with some images, a JSON file containing the text to display, and a signature. (If you're double-vaxxed, you can open up your pass in the Wallet app, copy it to a Mac using the share button and AirDrop, rename the <code>.pkpass</code> extension to <code>.zip</code>, open it up, and have a look at what's inside.)</p> <p><img alt="" src="/assets/articles/fake-covid-pass/cert-innards.png" /></p> <p><em>The contents of my (genuine) COVID-19 vaccine certificate</em></p> <p>About that signature: you might be thinking that'd stop forgeries, but that's not why it's there. <strong>There's no way to tell <em>which developer ID signed a pass</em> from within Apple Wallet;</strong> the app doesn't show you which developer ID was used to sign a pass, and there's no restriction or validation of the names and logos and whatever else on a pass.</p> <p>That means, if you have an Apple Developer Program membership (or a spare $150), you can grab your COVID-19 vaccine certificate, change the name to whatever you please, re-sign it with your own key pair, and hey presto! My friend <a href="https://blog.yaakov.online/forging-covid-19-vaccination-certificates-is-childs-play/">Yaakov</a> does have an ADP membership, and I mentioned this experiment to him in a conversation. Five minutes later, <em>while we were still talking</em>, he had a forgery.</p> <p>The thing is, I don't have an ADP membership, and I still managed to create a forgery. </p> <p>That's because there are multiple third-party apps that let you create your own pass with whatever information you want; they'll then generate the pass with <em>their</em> key pair. These apps have legitimate uses—I use them to add loyalty cards that aren't natively available for Apple Wallet. This also means that the bar for forging this passes is <em>even lower</em>; you can do it without even knowing what "JSON" means.</p> <p>I'm not going to tell you which app I used. I want to prompt discussion about how this system could be improved, but I don't want to make it even easier than it already is for the anti-vaxxers (especially since it's already hilariously easy).</p> <h1 id="is-it-a-pixel-perfect-forgery">Is it a pixel-perfect forgery?</h1> <p>Not quite. The big blob of terms and conditions on the back is missing, because the app I used didn't let me add more than four fields on the back, and if you put them side by side you can see that the logo is <em>slightly</em> smaller for reasons I'm not quite sure of.</p> <p>That said, if I wanted to pour a bit more time into this I could probably find an app that would let me create a pixel perfect forgery. If I wanted to drop $150 on signing up for the Apple Developer Program, I also could do what Yaakov did and just alter the name and ID numbers on my genuine pass, and re-sign it using my own key pair—<strong>his <em>was</em> a pixel perfect forgery</strong>.</p> <p><img alt="" src="/assets/articles/fake-covid-pass/backs.png" /></p> <p><em>Left to right: the "back side" of the forgery I created, my genuine COVID-19 certificate, and the forgery Yaakov created.</em></p> <p><a href="https://youtu.be/0JB1wbJV6tg">Nine News' Kate Creedon reported on fake vaccination certificates</a> just like the ones Yaakov and I were able to produce being sold on Telegram. At the time her segment went to air, those were probably produced using <a href="https://twitter.com/wabzqem/status/1427899005304938497?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1427899005304938497%7Ctwgr%5E%7Ctwcon%5Es1_&amp;ref_url=https%3A%2F%2Fwww.abc.net.au%2Fnews%2Fscience%2F2021-08-23%2Fcovid-19-vaccine-certificates-forged-in-10-minutes%2F100390578">the flaw that developer Richard Nelson found in August</a> and demonstrated in the same segment.</p> <p>(As an aside, those "security measures" Hank Jongen mentioned in that clip? Aside from being forgeable themselves, they're also not present in the Apple Wallet cards we're talking about here, just the Medicare app.)</p> <p>Assuming the government has fixed that flaw, the professional forgers have probably switched to the method Yaakov used; <a href="https://mobile.twitter.com/wabzqem/status/1437683489113993216">Richard says they're now selling for around $270</a>, which would still let them buy a new ADP membership for only one or a few passes, while still making a profit.</p> <p>Meanwhile, given that the method I used to make my own forgery has almost zero financial <em>or</em> technical barrier to entry, there's probably anti-vaxxers already using it for themselves and their family and friends.</p> <h1 id="can-apple-fix-this">Can Apple fix this?</h1> <p>Maybe. Apple could revoke the signing keys of every app that lets you produce arbitrary Apple Wallet passes, or insist that the makers of such apps do some sort of filtering to prevent things that look like they could be a COVID-19 pass, or add filtering to Apple Wallet to only allow things that look like they could be a COVID-19 pass if they're signed by a limited allowlist of keys.</p> <p>They could also try to proactively hunt down every key pair that's being used in this way and revoke them. They're probably disinclined to spend significant amounts of engineering and administrative time to clean up after the Australian government once again not doing an IT project properly, but they <em>could</em>, I guess.</p> <p>What Apple can't do is proactively tell when a key pair is being used to sign fake passes. The key pair is issued before the pass is created, and the process doesn't contact Apple's servers at any point from then on.</p> <h1 id="how-the-government-could-have-done-it-better">How the government could have done it better</h1> <p>The government have focused on things like flashy animations (easy to forge) and IDs (how do you even check they're not just random numbers?), forgetting one key principle: <strong>if someone <em>has</em> a device, they <em>have complete control over</em> that device.</strong> The best they can do is make forgeries slightly harder. Meanwhile, the overworked customer service staff that are going to be tasked with checking thousands of these a day won't be looking for much more than a green rectangle with a tick in it. They're in an arms race, they're moving slowly, and there's money to be made in beating them; they're not gonna win.</p> <p>Instead, they should introduce a second device that the verifier <em>can</em> trust. EU vaccination certificates, just like SA and NSW drivers' licenses, have a barcode which someone else can scan using a government issued app on their own phone to verify.</p> <p>There's a chain of trust here. If the verifier has their own phone, and they trust the government that published the app, and the app store they downloaded it from, they can scan the barcode and trust the result, even though the pass holder's device isn't trustworthy.</p> <p>There are multiple ways of doing this under the hood. They could use public key cryptography like the EU vaccine passports, which means verification still works without a data connection (or if the servers go down). Or they could use a token fetched from a server by the pass holder's phone and verified with that same server by the verifier's phone, which makes revoking bad certificates easier.</p> <p>There are pros and cons to each, but at the end of the day, they just need <em>something</em> that doesn't require trusting the pass holder's device; until they do that they'll always have a security hole big enough to drive a Sydney furniture removalist's truck through.</p> <h1 id="should-the-government-have-used-blockchain">Should the government have used blockchain?</h1> <p>No. Don't be silly.</p> <h1 id="will-you-give-or-sell-me-a-fake-certificate">Will you give or sell me a fake certificate?</h1> <p>No. Get vaccinated.</p> <h1 id="in-closing">In closing</h1> <p>Please, <em>please</em> get vaccinated if you haven't already. All of the vaccines available in Australia are very safe, and effective in reducing your likelihood of getting COVID, going to hospital, dying, and passing it on to others. Vaccination is one of the most powerful tools we have to keep ourselves, our loved ones, and people we'll never meet safe. The other powerful tool is lockdowns, but they're far more disruptive to our daily lives <em>and</em> losing their effectiveness with the Delta strain.</p> <p>But the vaccines aren't perfect, and not everyone can get vaxxed. To keep ourselves safe we also need the people around us to get vaxxed. So as we move towards a life where proof of vaccination is going to be required for lots of things, it's worth being mindful that so far, the current vaccine cards are difficult to verify and easy to forge—and maybe <a href="https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=-1&amp;gen=0&amp;ps=0">write to your local MP</a> and ask them to fix that.</p> Leigh Brenecki Using TypeScript 4.0's variadic tuple types to typecheck your path into an object https://leigh.net.au/writing/ts-variadic-tuple-pathinto/ 2020-09-23T00:00:00+0930 2020-09-23T00:00:00+0930 <p>There's two generic types that I've wanted to be able to implement in TypeScript for ages. Here's an example of them in use:</p> <div class="highlight"><pre><span></span><span class="kd">type</span> <span class="nx">FormData</span> <span class="o">=</span> <span class="p">{</span> <span class="nx">name</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">address</span><span class="o">:</span> <span class="p">{</span><span class="nx">street</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">suburb</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">postcode</span>: <span class="kt">number</span><span class="p">},</span> <span class="p">}</span> <span class="kd">type</span> <span class="nx">PathsIntoFormData</span> <span class="o">=</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;] | [&#39;address&#39;, &#39;street&#39; | &#39;suburb&#39; | &#39;postcode&#39;]</span> <span class="kd">type</span> <span class="nx">Street</span> <span class="o">=</span> <span class="nx">TypeAtPath</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;address&#39;</span><span class="p">,</span> <span class="s1">&#39;street&#39;</span><span class="p">]</span><span class="o">&gt;</span> <span class="c1">// = string</span> </pre></div> <p>Here, we have <code>PathInto&lt;T&gt;</code>, which returns a union of tuples, representing all possible "paths" of attribute access into nested levels of objects. If you can access the <code>name</code> property on <code>T</code>, then <code>['name']</code> will be a member of <code>PathInto&lt;T&gt;</code>; if you can access <code>address.street</code> on T, <code>['address', 'street']</code> will be; and so on.</p> <p>We also have <code>TypeAtPath&lt;T, P&gt;</code> which takes one such tuple type as a parameter and resolves to the type you'd get as a result. So, for instance, if you access <code>address.street</code> on <code>T</code> and get a <code>string</code> result, then <code>TypeAtPath&lt;T, ['address', 'street']&gt;</code> will resolve to <code>string</code>.</p> <p>Where might you want to use this? Form and state management libraries.</p> <h1 id="what-problem-do-these-types-solve">What problem do these types solve?</h1> <p>You might want to use types like this if you're building a form or state management library.</p> <p>As an example, let's say we're using <a href="https://formik.org/docs/api/useField">the <code>useField()</code> hook</a> in the ever-popular Formik:</p> <div class="highlight"><pre><span></span><span class="kd">const</span> <span class="p">[</span><span class="nx">field</span><span class="p">,</span> <span class="nx">meta</span><span class="p">,</span> <span class="nx">helpers</span><span class="p">]</span> <span class="o">=</span> <span class="nx">useField</span><span class="o">&lt;</span><span class="kt">number</span><span class="o">&gt;</span><span class="p">(</span><span class="s1">&#39;address.postcode&#39;</span><span class="p">)</span> </pre></div> <p>This is fine until we try to send something to the Northern Territory, whose postcodes start at 0800, and we lose the leading zero; or we try sending to the UK where postcodes can have letters in them. So we update our <code>FormData</code> type... and then we find every occurrence of <code>number</code> in reference to that particular field, and replace it.</p> <p>Instead, we could have written <code>useField&lt;FormData['address']['postcode']&gt;('address.postcode')</code>—that would have solved this issue! That repetition is unsightly, though, and it's only a matter of time until developers start omitting the annotation, leaving it to fall back to the default of <code>any</code>. There's also the potential for typos in the <code>'address.postcode'</code> string, which isn't checked.</p> <p>Instead, let's imagine a different API:</p> <div class="highlight"><pre><span></span><span class="c1">// type definition of our new useField</span> <span class="kd">function</span> <span class="nx">useField</span><span class="o">&lt;</span><span class="nx">P</span> <span class="k">extends</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="nx">path</span>: <span class="kt">P</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">FormField</span><span class="o">&lt;</span><span class="nx">TypeAtPath</span><span class="o">&lt;</span><span class="nx">P</span><span class="o">&gt;&gt;</span> <span class="c1">// and a usage example</span> <span class="kd">const</span> <span class="nx">field</span> <span class="o">=</span> <span class="nx">useField</span><span class="p">([</span><span class="s1">&#39;address&#39;</span><span class="p">,</span> <span class="s1">&#39;postcode&#39;</span><span class="p">])</span> <span class="c1">// field will be of type FormField&lt;number&gt;</span> </pre></div> <p>Here, we've used <code>PathInto&lt;T&gt;</code> to limit the possible arguments; if you try to call <code>useField(['address', 'postcoed'])</code> you'll get an error. TypeScript can also easily infer the type, so there's no repetition. We've also used <code>TypeAtPath&lt;T, P&gt;</code> to tell our return type what sort of field it is.</p> <p>(I'm hand-waving away providing <code>FormData</code> itself to our form library's type signatures—Formik naturally can't hard-code <code>FormData</code> into its type signatures. A slightly different API could, but this article is long enough already without my going into the perils of React context.)</p> <p>It's been <em>sort of possible</em> to implement <code>PathInto</code> and <code>TypeAtPath</code> in TypeScript for a while now, but it certainly wasn't easy or pretty. It involved a big, long chain of <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#conditional-types">conditional types</a>, one for each level of nesting you wanted to support, and plenty of arcane type system shenanigans at each level to actually construct the value you wanted. I've done it in a past project; the type definition for a <code>PathInto&lt;T&gt;</code> ends up being around 50 lines long, <em>plus</em> some extra private type definitions it uses internally, and even that only works for four levels of nesting.</p> <p>Happily, TypeScript creator Anders Heljsberg <a href="https://github.com/microsoft/TypeScript/pull/39094">added variadic tuple types</a> to TypeScript 4.0, so now we can define both of these types fairly easily—<strong>and</strong> use them with as many levels of nesting as we like!</p> <h1 id="defining-pathinto">Defining <code>PathInto</code></h1> <p>Our new <code>PathInto</code> type looks like this:</p> <div class="highlight"><pre><span></span> <span class="kd">type</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="o">&gt;</span> <span class="o">=</span> <span class="o">|</span> <span class="p">[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="o">|</span> <span class="p">{</span> <span class="p">[</span><span class="nx">K</span> <span class="k">in</span> <span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span><span class="o">:</span> <span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span> <span class="k">extends</span> <span class="nx">object</span> <span class="o">?</span> <span class="p">[</span><span class="nx">K</span><span class="p">,</span> <span class="p">...</span><span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">}[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> </pre></div> <p>Before we break this down: I've tried to strike a balance between brevity and clarity, but we're using a <em>lot</em> of TypeScript's more advanced type system features at once here. If there's a feature you haven't come across enough to fully understand yet, <a href="https://mariusschulz.com/blog">odds are that Marius Schulz has an excellent explainer on it</a>.</p> <p>With that said, let's dive in.</p> <h2 id="single-level-types">Single-level types</h2> <p>First, we have <code>[keyof T]</code>. <code>keyof T</code> (without the square brackets) is the <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#index-types">index type</a> of <code>T</code>, which is the union of all its keys. For example, <code>keyof {a: string, b: number}</code> will be the type <code>'a' | 'b'</code>. By putting it in square brackets we turn it into a 1-element tuple. At this point, for objects that are only a single layer of nesting deep, we're done!</p> <div class="highlight"><pre><span></span><span class="kd">type</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="o">&gt;</span> <span class="o">=</span> <span class="p">[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="kd">type</span> <span class="nx">PathIntoFormData</span> <span class="o">=</span> <span class="nx">PathIntoPart1</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;]</span> </pre></div> <h2 id="filtering-out-the-non-objects">Filtering out the non-objects</h2> <p>Next we have this bit: <code>{[K in keyof T]: ...}</code>, which is a <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#mapped-types">mapped type</a> over the keys of <code>T</code>. It creates a new object type, where the key types are whatever is in <code>keyof T</code>, and the values are the right-hand side (which we've replaced with <code>...</code> for now); <code>K</code> in the right-hand side is substituted for each key. For example, if we had <code>{[K in keyof T]: K}</code>, and <code>T</code> was <code>{a: string, b: number}</code>, the resulting type would be <code>{a: 'a', b: 'b'}</code>.</p> <p>On the right-hand side of our mapped type, we have <code>T[K] extends object ? ... : never</code>. This is a <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#conditional-types">conditional type</a>, which works similarly to the JavaScript ternary operator. If <code>T[K]</code> is an object type, we'll get the left-hand side (again, replaced with <code>...</code>); if not, we get the right-hand side—the <code>never</code> type.</p> <div class="highlight"><pre><span></span><span class="kd">type</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="o">&gt;</span> <span class="o">=</span> <span class="o">|</span> <span class="p">[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="o">|</span> <span class="p">{</span> <span class="p">[</span><span class="nx">K</span> <span class="k">in</span> <span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span><span class="o">:</span> <span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span> <span class="k">extends</span> <span class="nx">object</span> <span class="o">?</span> <span class="p">...</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">}</span> <span class="kd">type</span> <span class="nx">PathIntoFormData</span> <span class="o">=</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;] | {address: ...}</span> </pre></div> <h2 id="recursing-with-the-variadic-tuple-type">Recursing with the variadic tuple type</h2> <p>Inside <em>this</em>, we have our variadic tuple type! <code>[K, ...PathInto&lt;T[K]&gt;]</code> is a tuple type that has <code>K</code> as its first element. Variadic tuple types work like the spread operator: <code>PathInto&lt;T[K]&gt;</code> gives us a tuple type, and then its <em>contents</em> is filled into the top-level tuple type.</p> <div class="highlight"><pre><span></span><span class="kd">type</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="o">&gt;</span> <span class="o">=</span> <span class="o">|</span> <span class="p">[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="o">|</span> <span class="p">{</span> <span class="p">[</span><span class="nx">K</span> <span class="k">in</span> <span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span><span class="o">:</span> <span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span> <span class="k">extends</span> <span class="nx">object</span> <span class="o">?</span> <span class="p">[</span><span class="nx">K</span><span class="p">,</span> <span class="p">...</span><span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">}</span> <span class="kd">type</span> <span class="nx">PathsIntoFormData</span> <span class="o">=</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;] | {address: [&#39;address&#39;, ...PathInto&lt;FormData[&#39;Address&#39;]&gt;]}</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;] | {address: [&#39;address&#39;, &#39;street&#39; | &#39;suburb&#39; | &#39;postcode&#39;]}</span> </pre></div> <h2 id="extracting-values-from-our-mapped-type">Extracting values from our mapped type</h2> <p>The last bit we need to deal with is that <code>{address: ... }</code> bit. We don't want the object, we just want what's inside it. </p> <p>To get one specific key, we can write <code>{...}['b']</code>, but in some cases there'd be more than one key. Fortunately, indexing by a union of keys gives us a union of value types; for example, <code>{x: T1, y: T2}['x' | 'y']</code> gives you <code>T1 | T2</code>. We already have access to the union of all possible keys in our mapped type, so we can write <code>{...}[keyof T]</code>, and we'll get the union of all our 2-layer tuple types!</p> <h2 id="all-done">All done!</h2> <p>Now we just combine the two, and we get the result we wanted. And because our 2-layer tuple got filled in by applying <code>PathInto</code> recursively, the same type will work for 3-layer nested types, or 4-layer, or as many layers as you like.</p> <div class="highlight"><pre><span></span> <span class="kd">type</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="o">&gt;</span> <span class="o">=</span> <span class="o">|</span> <span class="p">[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="o">|</span> <span class="p">{</span> <span class="p">[</span><span class="nx">K</span> <span class="k">in</span> <span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span><span class="o">:</span> <span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span> <span class="k">extends</span> <span class="nx">object</span> <span class="o">?</span> <span class="p">[</span><span class="nx">K</span><span class="p">,</span> <span class="p">...</span><span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">T</span><span class="p">[</span><span class="nx">K</span><span class="p">]</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">}[</span><span class="nx">keyof</span> <span class="nx">T</span><span class="p">]</span> <span class="kd">type</span> <span class="nx">PathsIntoFormData</span> <span class="o">=</span> <span class="nx">PathInto</span><span class="o">&lt;</span><span class="nx">FormData</span><span class="o">&gt;</span> <span class="c1">// = [&#39;name&#39; | &#39;address&#39;] | [&#39;address&#39;, &#39;street&#39; | &#39;suburb&#39; | &#39;postcode&#39;]</span> </pre></div> <h1 id="defining-typeatpath">Defining <code>TypeAtPath</code></h1> <p>Here's my second type, <code>TypeAtPath</code>. A walkthrough of exactly what's going on here is coming in Part 2 of this post—but since you've stuck with me this far, you can have the type definition itself early. (You'll notice we're using most of the same tricks here, but there's a couple of new ones—one of them is a hack working around a <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-4-1-beta/#recursive-conditional-types">restriction that'll go away in TypeScript 4.1</a> anyway.)</p> <div class="highlight"><pre><span></span><span class="kd">type</span> <span class="nx">TypeAtPath</span><span class="o">&lt;</span><span class="nx">T</span> <span class="k">extends</span> <span class="nx">object</span><span class="p">,</span> <span class="nx">P</span> <span class="k">extends</span> <span class="nx">any</span><span class="p">[]</span><span class="o">&gt;</span> <span class="o">=</span> <span class="nx">P</span> <span class="k">extends</span> <span class="p">[</span><span class="nx">infer</span> <span class="nx">U</span><span class="p">]</span> <span class="o">?</span> <span class="p">(</span><span class="nx">U</span> <span class="k">extends</span> <span class="nx">keyof</span> <span class="nx">T</span> <span class="o">?</span> <span class="nx">T</span><span class="p">[</span><span class="nx">U</span><span class="p">]</span> <span class="o">:</span> <span class="nx">never</span><span class="p">)</span> <span class="o">:</span> <span class="nx">P</span> <span class="k">extends</span> <span class="p">[</span><span class="nx">infer</span> <span class="nx">U</span><span class="p">,</span> <span class="p">...</span><span class="nx">infer</span> <span class="nx">R</span><span class="p">]</span> <span class="o">?</span> <span class="p">(</span><span class="nx">U</span> <span class="k">extends</span> <span class="nx">keyof</span> <span class="nx">T</span> <span class="o">?</span> <span class="p">(</span> <span class="nx">T</span><span class="p">[</span><span class="nx">U</span><span class="p">]</span> <span class="k">extends</span> <span class="nx">infer</span> <span class="nx">U2</span> <span class="o">?</span> <span class="p">(</span> <span class="nx">U2</span> <span class="k">extends</span> <span class="nx">object</span> <span class="o">?</span> <span class="p">[</span><span class="nx">TypeAtPath</span><span class="o">&lt;</span><span class="nx">U2</span><span class="p">,</span> <span class="nx">R</span><span class="o">&gt;</span><span class="p">]</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">)</span> <span class="o">:</span> <span class="nx">never</span> <span class="p">)</span> <span class="o">:</span> <span class="nx">never</span><span class="p">)[</span><span class="mf">0</span><span class="p">]</span> <span class="o">:</span> <span class="nx">any</span> </pre></div> Leigh Brenecki Changing the default branch for new Git repositories https://leigh.net.au/writing/git-init-main/ 2020-06-09T00:00:00+0930 2020-06-09T00:00:00+0930 <p>The software community has been moving away from master-and-slave metaphors—slowly, and not without resistance—for some time now, but one that's mostly escaped scrutiny until recently is Git's default <code>master</code> branch.</p> <p>As far as I could tell, most of the discussions on the topic I could find online seemed to consist mostly of white people arguing with other white people about how non-white people do or should feel. <a href="https://twitter.com/ExcitedLeigh/status/1264727682744381440">When I recently tweeted about this</a>, the two Black folks who responded both said that despite not being paired with the term "slave", the "master" branch does make them feel uncomfortable.</p> <p>Separately, I learned that—contrary to what white dudes, summoned to the replies like flies to honey, will tell you whenever this comes up—<a href="https://mail.gnome.org/archives/desktop-devel-list/2019-May/msg00066.html">the term's usage in Git seems to have its origin in a master-and-slave metaphor</a> used in BitKeeper.</p> <p>It's also just not a very good name. <code>main</code> is a good one-size-fits-all replacement name, but other alternatives are more descriptive: for instance, if you're building a hosted service, and you run automated deploys when your main branch changes, you might call it <code>production</code>.</p> <p>In any case, I'm far from the only person noticing this! <a href="https://dev.to/horus_kol/renaming-your-master-branch-2a37">Fellow Adelaidean Stuart Jones</a> and <a href="https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx">prominent Microsoftie Scott Hanselman</a> both published posts on this exact topic yesterday alone. Both posts cover renaming <code>master</code> to something different in an existing repo, but new repos will still default to <code>master</code>.</p> <p>Git doesn't have a configuration option to override this default, but there is a workaround!</p> <h1 id="how-to-do-it">How to do it</h1> <ol> <li>Create a new directory somewhere. For this example, I'll use <code>~/.config/git/template</code>.</li> <li>Inside that, create a new file called <code>HEAD</code>, with the contents <code>ref: refs/heads/main</code>. (You can substitute <code>main</code> here for whatever you'd like your new default to be.)</li> <li>Set Git up to use your new template: <code>git config --global init.templateDir ~/.config/git/template</code></li> </ol> <h1 id="what-it-does">What it does</h1> <p>A lot of Git commands let you use hooks to extend their behaviour, but <code>git init</code> isn't one of them. Instead, <a href="https://git-scm.com/docs/git-init#_template_directory"><code>git init</code> lets you set a <strong>template directory</strong></a>, the contents of which are copied over the top of the newly-created <code>.git</code> directory once initialisation is done.</p> <p>(Template directories are also used when you <code>git clone</code> an existing repository, but the <code>HEAD</code> file gets overwritten afterwards. This means our change doesn't affect clones, only brand new repositories, which is what we want.)</p> <p>The second part of the puzzle is <strong><a href="https://git-scm.com/book/en/v2/Git-Internals-Git-References">Git references</a></strong>, which are almost always referred to by the abbreviation <strong>ref</strong>. A ref is a name that points to a commit, and they're stored as text files with the ref name as the file name, and the commit hash as the file's contents.</p> <p>Branches and tags are both kinds of ref! Open any Git repository's <code>.git</code> folder, and you'll find that in <code>.git/refs/heads</code> there is one file named for each branch. Open one of those files, and you'll see the commit hash of the newest commit on that branch (which is called the branch's "head", hence the directory name). The <code>.git/refs/tags</code> directory contains all of the repository's tags, stored in the same manner.</p> <p>These are all ordinary refs, but there's another kind of ref called a <strong>symbolic ref</strong>. Instead of pointing to a commit, symbolic refs point to another ref, which points to a commit. (If you're familiar with symbolic links, symbolic refs are the Git equivalent, which is why they're called that.) Symbolic refs are text files too; instead of the commit hash, their content is <code>ref:</code> followed by the ref that they point to.</p> <p>At this point, you can probably tell where we're going! There is a special ref in a git repository called <code>HEAD</code>, which points to the commit that's currently checked out. It's usually a symbolic ref, pointing to the currently checked out branch.</p> <p>The final puzzle piece is this: When you run <code>git init</code>, Git doesn't actually create the master branch at that point. It just creates the <code>HEAD</code> symbolic ref, deliberately pointing it to a branch that doesn't exist yet. When you create your first commit, <code>git commit</code> notices that <code>HEAD</code> is pointing to a non-existent branch, and creates it. In our workaround, we just replace <code>HEAD</code> so that it points to a non-existent branch with a different name.</p> Leigh Brenecki Men are the worst https://leigh.net.au/writing/20200122-men/ 2020-01-22T00:00:00+0930 2020-01-22T00:00:00+0930 <p>Men are the worst.</p> <p>This sentence might make you angry, especially if you're a man. If it doesn't, my hope is that it will by the time you're done reading this.</p> <p>I’m not a man. I’m non-binary, approximately feminine within some margin of error. But I was assigned male at birth, and so for the last 25 years, I've been trying to find my place in masculinity. Now that I've realised my place is outside it, I have some parting thoughts.</p> <p><strong>Masculinity is narrow.</strong> In our culture, being a man restricts so much of your life: appearance, dress, careers, hobbies, sexuality, emotional literacy, emotional intimacy, platonic affection. It bars you from asking for help. It prevents you from showing emotion, be it sadness or joy. It tells you that you are superior if, and only if, you prove you’re manly enough by obeying its restrictions, and it encourages enforcing those restrictions and exploiting that superiority with mockery and violence.</p> <p>When men are called out on bad behaviour, a common response is that <strong>"boys will be boys"</strong>. It is in the nature of men to harass, to creep, to be violent, to exclude, to treat others as lesser than them. It is innate. It is unchangeable. It can’t be helped, or so it is said.</p> <p>If it can’t be helped, the rational solution is to lock all men up for the betterment of society, so it’s just as well that this is complete nonsense.</p> <p>If you are a man, this defence should insult you; it is saying that you are not capable of being better. But that is not the truth. Masculinity as it is now is not the way things must be. Men can create a new, better, safer masculinity, <strong>one that is fearless in its love and kindness and inclusion</strong>.</p> <p>Traditional masculinity is exclusively and aggressively heterosexual, so gay and bi men are excluded from it, and they suffer for that exclusion. Outside its confines, many of them have created a masculinity that is richer, more diverse, and far more interesting. Tear down those confines, and you simultaneously welcome them in and let yourself out.</p> <p>My boyfriend wrote this, on the topic of toxic masculinity amongst trans men (a topic that is itself not my place to discuss):</p> <blockquote> <p>We need to carve out a healthy masculinity space. Occupy it, exemplify it, use our voices to protect others, and show that the place exists. We need cis men to join us, rather than us joining them in toxicity.</p> </blockquote> <p>In finding their way to masculinity, some trans men arrive at one that is not traditional masculinity, but that is healthier and kinder. One of those men says "we need cis men to join us", but cis men themselves need that space more than many of them realise.</p> <p>If you are a cisgender man, your journey is different. You can find your way to masculinity—the masculinity that culture guides you to—on autopilot. But maybe you shouldn’t. To an extent, trans people’s journeys to our gender identities are conscious and deliberate, out of necessity. Maybe yours should be conscious and deliberate too.</p> <p>There are also positive forms of traditional masculinity, men that see themselves as traditionally masculine, but that reject the toxic parts of that identity. They reject the gate-keeping of masculinity, and they reject that long list of important parts of life that traditional masculinity sees as weakness. They reject the idea that those unlike them are lesser, and they treat women, non-binary people, and non-conforming men as equals worthy of respect.</p> <p>Create a masculinity that is so much broader, more welcoming, and more inclusive than the one we have. Create a masculinity where you can be you, and where others can be themselves.</p> <p>Men are the worst, and that should make you angry, because that fact is hurting you and everyone around you. But this is not immutable, and you can be part of the solution, and perhaps you already are!</p> <p>So treat masculinity like a campsite, or perhaps like it’s a buggy legacy codebase you inherited from the generations before you. <strong>Leave masculinity better than you found it.</strong></p> <p><em>This is a written adaptation of <a href="https://youtu.be/eqQUepwTHjA?t=1997">my Linux.conf.au 2020 lightning talk of the same title</a>.</em></p> Leigh Brenecki We don't need Moment.js just to display dates and times anymore https://leigh.net.au/writing/20180913-momentjs-intl/ 2018-09-13T00:00:00+0930 2018-09-13T00:00:00+0930 <p>Moment.js is a great library, but it’s still something extra for your users’ browsers to download, parse and run when they visit your site! Then, once you’ve downloaded all the code for all its myriad features, it’s often just turning an ISO date-time string from some API into the right timezone and format for the user. Fortunately we can do that, and do it better, with nothing but the features built in to modern browsers and IE 11!</p> <h1 id="parsing-iso-dates">Parsing ISO dates</h1> <p>In modern browsers and IE 9+, the <code>Date</code> constructor (as well as <code>Date.parse()</code>) supports <a href="http://www.ecma-international.org/ecma-262/5.1/#sec-">a restricted subset of the ISO date format</a>, with separators being mandatory. (For instance, <code>2017-04-26T19:30:00+09:30</code> works in browsers, but <code>20170426T193000+0930</code> doesn’t.) In practice, this restricted subset is exactly what most APIs will send, so the following Just Works:</p> <div class="highlight"><pre><span></span><span class="k">const</span> <span class="n">valueFromApi</span> <span class="o">=</span> <span class="s2">&quot;2017-04-26T19:30:00+09:30&quot;</span> <span class="k">const</span> <span class="n">myDate</span> <span class="o">=</span> <span class="n">new</span> <span class="n">Date</span><span class="p">(</span><span class="n">valueFromApi</span><span class="p">)</span> </pre></div> <h1 id="displaying-to-the-user">Displaying to the user</h1> <p>This is where Moment.js used to shine. Way back when, all we got from the browser was <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toDateString"><code>Date.prototype.toDateString()</code></a> and friends, which returned “a human readable form in American English”. In contrast, Moment.js has <a href="https://momentjs.com/docs/#/displaying/format/">a comprehensive format string syntax</a> that lets us mix and match date components whichever way we want. The built in <code>Date</code> object still doesn’t have anything like that, but modern browsers and IE 11 have something arguably better: <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat"><code>Intl.DateTimeFormat</code></a>. Here’s an example:</p> <div class="highlight"><pre><span></span><span class="k">const</span> <span class="n">longFmt</span> <span class="o">=</span> <span class="n">new</span> <span class="n">Intl</span><span class="o">.</span><span class="n">DateTimeFormat</span><span class="p">(</span><span class="n">undefined</span><span class="p">,</span> <span class="p">{</span> <span class="n">weekday</span><span class="p">:</span> <span class="s1">&#39;long&#39;</span><span class="p">,</span> <span class="n">year</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">month</span><span class="p">:</span> <span class="s1">&#39;long&#39;</span><span class="p">,</span> <span class="n">day</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">hour</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">minute</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">timeZoneName</span><span class="p">:</span> <span class="s1">&#39;long&#39;</span><span class="p">})</span> <span class="k">const</span> <span class="n">shortFmt</span> <span class="o">=</span> <span class="n">new</span> <span class="n">Intl</span><span class="o">.</span><span class="n">DateTimeFormat</span><span class="p">(</span><span class="n">undefined</span><span class="p">,</span> <span class="p">{</span> <span class="n">year</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">month</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">day</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">hour</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">,</span> <span class="n">minute</span><span class="p">:</span> <span class="s1">&#39;numeric&#39;</span><span class="p">})</span> <span class="n">console</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">longFmt</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">myDate</span><span class="p">))</span> <span class="o">//</span> <span class="n">Wednesday</span><span class="p">,</span> <span class="mi">26</span> <span class="n">April</span> <span class="mi">2017</span><span class="p">,</span> <span class="mi">7</span><span class="p">:</span><span class="mi">30</span> <span class="n">pm</span> <span class="n">Australian</span> <span class="n">Central</span> <span class="n">Standard</span> <span class="n">Time</span> <span class="o">//</span> <span class="p">(</span><span class="n">on</span> <span class="n">my</span> <span class="n">system</span><span class="p">,</span> <span class="n">yours</span> <span class="n">will</span> <span class="n">probably</span> <span class="n">be</span> <span class="n">different</span><span class="p">)</span> <span class="n">console</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">shortFmt</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">myDate</span><span class="p">))</span> <span class="o">//</span> <span class="mi">26</span><span class="o">/</span><span class="mi">04</span><span class="o">/</span><span class="mi">2017</span><span class="p">,</span> <span class="mi">7</span><span class="p">:</span><span class="mi">30</span> <span class="n">pm</span> </pre></div> <p>This lets you precisely control how dates are formatted—4- or 2-digit years, text or numeric months, omitting components, and so on—<strong>but still use the component ordering and separators your users are expecting</strong>, depending on the locale they chose when they first set up their computer or phone! (You can also specify a specific locale string like <code>'en-AU'</code>, or an IANA timezone to convert to like <code>'Australia/Adelaide'</code>, if you’re confident you know what the user wants to see better than the browser does.)</p> <p>By changing your date display code to use <code>Intl</code> instead of Moment.js, you’re not only saving a few kilobytes of JS, but you’ll also show your users dates that are easier for them to read, wherever they are in the world.</p> Leigh Brenecki It is possible to undo a HSTS deployment... sort of https://leigh.net.au/writing/20180114-hsts/ 2018-01-14T00:00:00+0930 2018-01-14T00:00:00+0930 <p>Once you start serving your web app or website over HTTPS, one really easy way to improve its security is to use HTTP Strict Transport Security, or HSTS for short. All you need to do is send a response header that looks like this:</p> <div class="highlight"><pre><span></span><span class="err">Strict-Transport-Security: max-age=31536000</span> </pre></div> <p>Most modern browsers will remember this header, and will always automatically upgrade every request to the same domain to HTTPS, for the number of seconds in the max-age parameter (in this case, one year).</p> <p>This is great! Normally, when someone types your website into their address bar, unless they explicitly include the <code>https://</code> at the start, the browser will make a plain HTTP request first. Even if the very first thing you do whenever you get a HTTP request is redirect it to HTTPS, an attacker could intercept that first redirect and make it point to a server under their control instead. With HSTS, everything goes over HTTPS from the start, so they don't get a chance.</p> <p>The one caveat with deploying HSTS, or so the common advice goes, is that there's no going back. If you discover that you need to go back to HTTP, browsers that have already seen the HSTS header will keep requesting HTTPS versions of your site, and will refuse to use plain HTTP; that's the whole point. So, make sure you test everything on HTTPS, and start off with small max-age values, until you're sure everything works.</p> <p>This is good advice, and you should definitely follow it, but what if it's too late? What if you discover that part of your site doesn't work with HTTPS, but you've already sent out HSTS headers with a 1-year expiry to visitors? The solution looks something like this:</p> <div class="highlight"><pre><span></span><span class="kr">HTTP</span><span class="o">/</span><span class="m">1.1</span> <span class="m">302</span> <span class="ne">Found</span> <span class="na">Location</span><span class="o">:</span> <span class="l">http://example.com/foo</span> <span class="na">Strict-Transport-Security</span><span class="o">:</span> <span class="l">max-age=0</span> </pre></div> <p>It turns out that you can change the max-age parameter in a Strict-Transport-Security header, and it'll override whatever the browser has stored. You can even set it to zero, which will tell the browser that HSTS no longer applies to this site. By doing that and then redirecting to HTTP, you're essentially undoing HSTS in a way that's invisible to the user.</p> <p>Of course, if you totally lose the ability to serve HTTPS at all for some reason, this still won't help you. In fact, if you're stuck in this scenario there's likely no way out, because it's impossible for a browser to tell the difference between "the real server can't serve HTTPS anymore and wants me to use HTTP" and "there's a man-in-the-middle attacker that's modifying traffic and wants me to use HTTP". That said, given the abundance of certificate authorities that will sell you cheap domain-verified certificates, as well as Let's Encrypt which will give them to you for free, you're unlikely to be in this situation—even if your server won't let you set it up this way, you can put a reverse proxy like nginx in front of it and configure the redirect there.</p> <p>In the more likely scenario where you've rolled out HTTPS, rolled out HSTS, then discovered that there's a part of your site that uses some third-party JavaScript or iframe that's HTTP-only and can't be replaced, you're all set. Of course, you should still serve as much as you can over HTTPS—especially things like admin panels for CMSes or anything else where users have to log in—and you should do whatever you can to get the third-party provider to support HTTPS too, but at least your site won't be broken for all your repeat visitors in the meantime.</p> Leigh Brenecki Yes, a sex toy recording audio without consent is still a big deal https://leigh.net.au/writing/20171203-sex-toy-audio-consent/ 2017-12-03T00:00:00+0930 2017-12-03T00:00:00+0930 <p>A few weeks ago, <a href="https://www.reddit.com/r/sex/comments/7bmi3i/psa_lovense_remote_control_vibrator_app_recording/">a post popped up in Reddit's /r/sex community claiming that the Lovense Android app was taking audio recordings</a> when the company's "connected" sex toys were in use, and saving them to the device SD card. This was quickly corroborated by other Android-using Lovense owners, and then quickly spread through the media like wildfire, with a heavy dose of speculation on whether the app sent this sound data to Lovense. A few days later, <a href="https://www.reddit.com/r/sex/comments/7bmi3i/psa_lovense_remote_control_vibrator_app_recording/dpm4050/">the company chimed in</a>:</p> <blockquote> <p>Regarding the sound file in question, it has already been confirmed that this is a minor bug - a temporary file that is created when someone uses the Sound Control feature. Your concern is completely understandable. But rest assured, no information or data is sent to our servers.</p> </blockquote> <p>In the interim, RenderMan posted on the website of <em>Internet of Dongs</em>, his sex toy security auditing project, in an article entitled <a href="https://internetofdon.gs/2017/11/11/destroying-the-reddit-lovense-remote-vuln-conspiracy/">Destroying the Reddit Lovense Remote Vuln Conspiracy</a> (emphasis in original):</p> <blockquote> <p>Given that the function the user was using the app, the "Ambient Sound" function, I do wonder about them. Presuming they know that the louder the sounds around them, the more the device vibrated, I would ask a very simple question: <strong>"How did you think that the app knew what the ambient sound levels were without using the microphone?".</strong> To my knowledge, phones are not telepathic (yet. I bet Google's has a project trying to do that right now).</p> <p>Android permissions are not that granular. Granting an app a permission for the microphone and camera allows any part of the app to use those functions should it want to. You cannot (though it would be nice to) have granular control of when you want to allow permission in an app or not.</p> </blockquote> <p>Putting aside RenderMan's insistence on repeatedly belittling a layperson for not being as clever as he is, I have a few issues with this response.</p> <h2 id="this-isnt-how-this-feature-should-work">This isn't how this feature should work</h2> <p>RenderMan asks above, in so many words, "well, what did you think this feature would do?" I would have expected that the app records sound into a short buffer of some kind, perhaps does a bit of post-processing, calculates the amplitude over a short period of that data, and sends that to the toy to update its vibration intensity. I wouldn't have expected the app to store sound data for more than a few seconds, if that, and I certainly wouldn't expect it to store an entire session of sound data and leave it lying around on disk afterwards.</p> <p>A user with less technical knowledge might not express it in the same way, but would probably expect the same; after all, the app doesn't need to know what the ambient sound intensity was ten seconds or one minute or ten minutes ago; it only needs to know what it is right now, so why would it keep that data around?</p> <p>RenderMan <em>sort of</em> acknowledges that the file didn't need to be written to disk:</p> <blockquote> <p>While this file could be stored in RAM, it is much easier and more efficient to stream it to disk for temporary storage (500mb of RAM is more useful than 500mb of disk). This makes sense, especially when it was clear that the file was meant to be purged once it was no longer needed.</p> </blockquote> <p>The thing is, the file didn't need to be 500MB either; a few seconds of uncompressed audio is only a few hundred kilobytes.</p> <h2 id="recording-people-having-sex-is-still-kind-of-a-big-deal">Recording people having sex is still kind of a big deal</h2> <p>So we know, both from RenderMan's analysis and Lovense's response, that the data was never sent anywhere, it just stayed on disk until it was overwritten by the next use of that feature. That's better than the alternative, but the fact remains that the app stored a recording of its users without their consent.</p> <p>That would be a problem in any case, but this is an app whose primary purpose is use during sexual activity. Not only are plenty of people self-conscious about the sounds they make during sex, but recordings of people having sex or masturbating—even though they're perfectly normal things most adults do, and even when those recordings are made or shared without consent—can ruin careers and lives. (Fortunately, these recordings are just sound with no video, which probably makes them less desirable in the unscrupulous parts of the Internet where these things are shared should they leak, but there are other parts of the Lovense app that <em>do</em> use video.)</p> <p>As soon as those recordings are made, they're a liability, and making them without telling the user is careless at best. They might not get sent to Lovense's servers, but they're still lying around on disk, unbeknownst to the phone's owner, waiting to be discovered by a "friend" that thinks shoulder-surfing someone's PIN code and browsing through an unattended phone is okay, or exfiltrated by a third-party app that's been granted unrestricted disk access.</p> <p>RenderMan comes so close to realising why people are so bothered by this—yet so, so far:</p> <blockquote> <p>Authors Note: I can't imagine the usefulness of Ambient sound recordings of users IoD usage would be of any use to vendors. Have you ever stepped back and listened to people have sex? People make some of the most ridiculous sounds mid coitus, so it's only useful for comedy purposes I think</p> </blockquote> <p>This isn't, as Lovense suggested, a "minor bug".</p> <h2 id="it-wasnt-malevolent-but-that-doesnt-make-it-okay">It wasn't malevolent, but that doesn't make it okay</h2> <p>RenderMan correctly points out that Lovense have nothing to gain from collecting this data, and there's no reason to accuse them of maliciously spying on their users. In reality, there's probably something like this floating around in Lovense's codebase:</p> <div class="highlight"><pre><span></span><span class="c1">// TODO: only store what we need,</span> <span class="c1">// rather than writing everything to disk</span> </pre></div> <p>People make mistakes. Software developers, despite what we like to think about ourselves, are particularly adept at it. Lovense, to their credit, fixed the bug as soon as it was brought to their attention.</p> <p>Connected sex toys, though, are something that users are putting a lot of trust in—literally putting them in, on or around the most sensitive and private parts of their bodies, then connecting them to a network shared by half the world's population—meanwhile the tech industry more broadly seems to be bent on proving itself unworthy of public trust. PR departments like Lovense's downplaying a problem with their products is par for the course, but when independent security researchers do the same, then proceed to belittle users that have concerns, it really doesn't help that perception.</p> Leigh Brenecki Efficiently fetching updated rows with Postgres https://leigh.net.au/writing/20170921-fetching-postgres-updates/ 2017-09-21T00:00:00+0930 2017-09-21T00:00:00+0930 <p>Let's say you've got a web app that's currently fetching all its data from an API, or doing everything on the server with server-rendered views, but you want to start storing all your users' data locally. Maybe you're making a native mobile app, or you want to build a Progressive Web App like all the cool kids.</p> <p>You don't necessarily care about synchronising offline edits or anything like that; not yet, anyway. All you want to do is store a read-only copy of your users' data on their devices, so you don't have to hit the network and make them wait every time they launch your app, then periodically fetch whatever's changed from the server.</p> <h2 id="the-simple-but-wrong-solution">The simple (but wrong) solution</h2> <p>The obvious thing to do here is to just store a last-modified timestamp on each row, making it <code>DEFAULT</code> to <code>now()</code> and updating it in an <code>ON UPDATE</code> trigger. Then, when the client wants to fetch everything that's changed on the server, it sends an if-modified-since timestamp of the last time it checked, and the server sends back every row newer than that timestamp.</p> <p>This will work in testing, and if each row is only normally exposed to a single human user, it'll work in production most of the time. But there's a race condition here:</p> <ol> <li>Server A opens a transaction and writes Row A, firing our trigger and updating the last-modified timestamp.</li> <li>Server B receives an update request from Client B, and selects all updated rows.</li> <li>Server A commits.</li> </ol> <p>Server B won't see Row A, because it wasn't committed yet. Next time Client B fetches an update, though, it'll send an if-modified-since timestamp newer than Row A's last-modified timestamp, so it'll <strong>never</strong> see Row A. (At least, not until Row A gets updated next.)</p> <p>Note that Server A and Server B don't need to be distinct <em>physical</em> servers; they could be two separate worker processes, threads or coroutines on the same machine.</p> <h2 id="barking-up-the-wrong-tree">Barking up the wrong tree</h2> <p>What if, instead of our client sending the timestamp of the last time it checked, it sends the newest last-modified timestamp it already has? That solves the scenario above, but it doesn't solve the race condition. Consider:</p> <ol> <li>Server A writes Row A.</li> <li>Server B writes Row B, and commits.</li> <li>Server C receives an update request from Client C, and selects all updated rows.</li> <li>Server A commits.</li> </ol> <p>In this scenario, Client C will never see Row A, because it's already got a copy of Row B which is newer.</p> <h2 id="transaction-ids-to-the-rescue">Transaction IDs to the rescue</h2> <p>Fortunately, Postgres has functionality that can help us out here. Let's use a to-do list as an example:</p> <div class="highlight"><pre><span></span><span class="k">create</span> <span class="k">table</span> <span class="n">todo_items</span> <span class="p">(</span> <span class="n">id</span> <span class="nb">serial</span> <span class="k">primary</span> <span class="k">key</span><span class="p">,</span> <span class="n">done</span> <span class="n">bool</span> <span class="k">not</span> <span class="k">null</span> <span class="k">default</span> <span class="k">false</span><span class="p">,</span> <span class="n">name</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span> <span class="k">not</span> <span class="k">null</span><span class="p">,</span> <span class="n">last_updated_txid</span> <span class="nb">bigint</span> <span class="k">not</span> <span class="k">null</span> <span class="p">);</span> <span class="k">create</span> <span class="k">function</span> <span class="n">add_txid</span><span class="p">()</span> <span class="k">returns</span> <span class="k">trigger</span> <span class="k">as</span> <span class="s1">&#39;</span> <span class="s1"> begin</span> <span class="s1"> new.last_updated_txid := txid_current();</span> <span class="s1"> return new;</span> <span class="s1"> end</span> <span class="s1">&#39;</span> <span class="k">language</span> <span class="s1">&#39;plpgsql&#39;</span><span class="p">;</span> <span class="k">create</span> <span class="k">trigger</span> <span class="n">todo_items_txid</span> <span class="k">before</span> <span class="k">insert</span> <span class="k">or</span> <span class="k">update</span> <span class="k">on</span> <span class="n">todo_items</span> <span class="k">for</span> <span class="k">each</span> <span class="k">row</span> <span class="k">execute</span> <span class="k">procedure</span> <span class="n">add_txid</span><span class="p">();</span> </pre></div> <p>This is basically the same strategy as we were using before, except instead of storing a timestamp on each row, we're storing <a href="https://www.postgresql.org/docs/devel/static/functions-info.html#functions-txid-snapshot">the ID of the transaction we're currently in</a>, which is a 64-bit integer that increases monotonically and is guaranteed to never wrap throughout the life of a Postgres installation. (Internally, Postgres uses a 32-bit ID which <em>does</em> wrap, but it tracks each wrap in order to provide a larger number with that guarantee to us, the user.)</p> <p>The important bit happens when we go to fetch updated data. Along with all of our updated rows, we also send back to the client the output of this query:</p> <div class="highlight"><pre><span></span><span class="k">select</span> <span class="n">txid_snapshot_xmin</span><span class="p">(</span><span class="n">txid_current_snapshot</span><span class="p">());</span> </pre></div> <p>This gives us the current <code>xmin</code>—the ID of the <strong>oldest transaction that is currently active</strong>. Every transaction with an ID smaller than this number is guaranteed to already be either committed or rolled back, so when our client sends that value back to us on its next fetch, we know we're definitely giving it every change they don't already have. We can see this behaviour by opening multiple Postgres client sessions side by side:</p> <div class="highlight"><pre><span></span><span class="c1">-- Session A</span> <span class="k">insert</span> <span class="k">into</span> <span class="n">todo_items</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">values</span> <span class="p">(</span><span class="s1">&#39;Buy some milk&#39;</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;Walk the dog&#39;</span><span class="p">);</span> <span class="c1">-- Session B</span> <span class="k">begin</span><span class="p">;</span> <span class="k">insert</span> <span class="k">into</span> <span class="n">todo_items</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">values</span> <span class="p">(</span><span class="s1">&#39;Clean the kitty litter&#39;</span><span class="p">);</span> <span class="k">update</span> <span class="n">todo_items</span> <span class="k">set</span> <span class="n">done</span> <span class="o">=</span> <span class="k">true</span> <span class="k">where</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span> <span class="c1">-- Session C</span> <span class="k">begin</span><span class="p">;</span> <span class="k">insert</span> <span class="k">into</span> <span class="n">todo_items</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">values</span> <span class="p">(</span><span class="s1">&#39;Write a blog post&#39;</span><span class="p">);</span> <span class="k">commit</span><span class="p">;</span> <span class="c1">-- Session A</span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">todo_items</span><span class="p">;</span> <span class="c1">-- +------+--------+-------------------+---------------------+</span> <span class="c1">-- | id | done | name | last_updated_txid |</span> <span class="c1">-- |------+--------+-------------------+---------------------|</span> <span class="c1">-- | 1 | False | Buy some milk | 3751 |</span> <span class="c1">-- | 2 | False | Walk the dog | 3751 |</span> <span class="c1">-- | 4 | False | Write a blog post | 3753 |</span> <span class="c1">-- +------+--------+-------------------+---------------------+</span> <span class="k">select</span> <span class="n">txid_snapshot_xmin</span><span class="p">(</span><span class="n">txid_current_snapshot</span><span class="p">());</span> <span class="c1">-- 3752</span> <span class="c1">-- Session B</span> <span class="k">commit</span><span class="p">;</span> <span class="c1">-- Then, when Client A wants to update its data...</span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">todo_items</span> <span class="k">where</span> <span class="n">last_updated_txid</span> <span class="o">&gt;=</span> <span class="mi">3752</span><span class="p">;</span> <span class="c1">-- +------+--------+------------------------+---------------------+</span> <span class="c1">-- | id | done | name | last_updated_txid |</span> <span class="c1">-- |------+--------+------------------------+---------------------|</span> <span class="c1">-- | 3 | False | Clean the kitty litter | 3752 |</span> <span class="c1">-- | 2 | True | Walk the dog | 3752 |</span> <span class="c1">-- | 4 | False | Write a blog post | 3753 |</span> <span class="c1">-- +------+--------+------------------------+---------------------+</span> </pre></div> <p>This is effectively playing out the second scenario we described above, and you can see that the changes we made in Session B are successfully retrieved.</p> <p>Note that the "Write a blog post" entry gets sent to the client twice; where the timestamp method makes data invisible to clients in this edge case, this one sends data the client already has, which is much less undesirable.</p> <p>For the general case, we are still only sending updated rows, as shown by the "Buy some milk" row.</p> <h2 id="caveats">Caveats</h2> <ul> <li>We're dealing with 64-bit integers here. If your client is written in JavaScript, which doesn't have an integer type and can't precisely store the entire space of 64-bit integers, you might want to stringify your transaction IDs on the server side.</li> <li>If you move your database to a different installation, your transaction IDs are unlikely to match up. You could choose a constant to add to IDs on the new system, large enough that the old installation will never overlap before it's decommissioned, but you'll still need to plan how you're going to cut over from one to the other carefully.</li> <li>In the same vein, if you're doing anything involving more than one Postgres installation, you'll want to check how transaction IDs work on your setup.</li> <li>If you have jobs that keep transactions open for a long time, all of your clients will keep fetching duplicate data until that transaction is closed. Long-running transactions on a production server are best avoided anyway, so you might want to refactor them away or move them to a read replica if you can.</li> </ul> <h2 id="whats-this-like-in-production">What's this like in production?</h2> <p>I don't know, you tell me! I've got some project ideas that I want to try building as PWAs that store a cache of user data locally in IndexedDB, and this is probably how I'll sync them. For now, this is just an idea I've had bouncing around in my head that I wanted to get onto paper.</p> <hr /> <p>Thanks to <a href="https://twitter.com/AzMoo">Matt Magin</a> for his feedback on this idea, and initially pointing out to me why the timestamp solution wouldn't work, and to <a href="https://twitter.com/nic_and_chips">Nic Crouch</a> for his feedback and insight on a draft version of this article.</p> Leigh Brenecki "Trust me, it's secure" https://leigh.net.au/writing/20131012-trust-me-its-secure/ 2013-10-12T00:00:00+0930 2013-10-12T00:00:00+0930 <p>There's a lot of sites—including some that want your credit card data, and even some operated by financial institutions—that are <em>technically</em> using HTTPS, but are still jeopardising users' security.</p> <p>Some of them have a form to collect sensitive data which is served on a non-secure page, but submits to a secure page. Others collect sensitive data on a secure page, but that secure page is presented inside an iframe on a non-secure page. There are two huge security problems with this.</p> <p>The first problem with this is that <strong>the user doesn't know it's secure</strong>. Sure, they can use their browser's web developer tools to inspect the form or iframe element, but most users won't know how to do that, and the users that do know will probably trust you less for the fact that they have to.</p> <p>Many of these sites feature prominent messages saying "THIS SITE IS SECURE" and containing clip-art pictures of padlocks; these are meaningless, and web-savvy users know that. (In fact, those web-savvy users will, once again, probably trust you less for it.) <em>These sites are telling their users "trust me, it's secure", without actually proving it.</em></p> <p>The second problem is the big one: <strong>the insecure page itself is a weak link</strong>. In an ideal situation, all of the data is sent over a secure connection; our system is secure against passive interception. Let's say our user, Alice, is trying to buy something from Bob's Online Store using her favourite cafe's open WiFi. Eve is sitting the next table over, sniffing wi-fi traffic. Our system foils her for now.</p> <p>But, if we were always dealing with ideal situations, <em>we wouldn't need HTTPS in the first place</em>. Later, at another cafe, Alice decides to make another purchase. She connects to the cafe's wifi, and once again fills in her credit card details. This time, Mallory is sitting across the room with an espresso, a copy of the Sunday Mail, and a fake access point running all traffic through something like <a href="http://www.thoughtcrime.org/software/sslstrip/">sslstrip</a>, which modifies the insecure page in-flight, so that the links pointing to the secure page now point elsewhere. Alice doesn't realise that the form has been tampered with, and will now submit to an insecure URL, until she starts seeing strange charges on her credit card days later.</p> <h2 id="one-page-isnt-enough">One page isn't enough</h2> <p>The easy way to protect users from this sort of thing is to properly secure the data entry page, and tell them to look for the padlock (or, in the case of a site with Extended Validation, the "Green Bar"). But there's still a problem: the other pages that the user clicks through to get there. If they're not secured, our attacker can still launch her HTTPS stripping attack earlier in the process. Users are rarely diligent enough to check the address bar every time, particularly with the complacency that comes with a familiar site. (While the Big Four Australian banks all have encrypted internet banking login pages, all but Commonwealth fall down at this point.)</p> <p>The only real solution, then, is to secure every page on the website, from the homepage the user sees when they type the address in onwards. But we're still not safe yet! Users generally type URLs in without a scheme (e.g. typing in <code>commbank.com.au</code>, instead of <code>https://commbank.com.au</code>), and browsers will interpret that as a HTTP URL. Commonwealth's servers, like almost every other HTTPS-only site, will respond by redirecting the browser to HTTPS, but since the redirection itself happens over HTTP, it can still be intercepted.</p> <p>Sadly, there's no foolproof way to ensure that all requests to your site are sent through HTTPS from the get-go, but we can come close with HTTP Strict Transport Security. Sites that use HSTS send a response header that tells the browser "this domain operates over HTTPS only", and from that point onwards browsers automatically rewrite all requested URLs for that domain to HTTPS before the request is sent, regardless of whether they came from a user's typing, a link, or an embedded resource in another page.</p> <p>Despite how easy implementing HSTS is—for nginx, just add the single line below to your server block—not even Commonwealth are using it.</p> <div class="highlight"><pre><span></span><span class="k">add_header</span> <span class="s">Strict-Transport-Security</span> <span class="s">max-age=31536000</span><span class="p">;</span> </pre></div> <hr /> <p><em>Edited on 9 July 2017:</em></p> <p>Since this article was written, all of the Big Four banks now serve their entire sites over HTTPS. Commonwealth and Westpac now also send Strict-Transport-Security headers; NAB and ANZ are still vulnerable to the attack described above. In addition, Web browsers now ship with a "HSTS preload" list built in, which thwarts this attack even on the first time a user visits a site; of the Big Four, only Commonwealth is on that list.</p> Leigh Brenecki Using Apache as a PHP development server https://leigh.net.au/writing/20130606-apache-php-dev-server/ 2013-06-06T00:00:00+0930 2013-06-06T00:00:00+0930 <p>I'm mostly a Python web developer, but PHP is <em>the</em> most popular runtime environment for web applications, and this makes it somewhat unavoidable.</p> <p>When you're writing a web app in Python (or, really, most other languages), you test your code in a development server. It runs in the foreground in a Terminal window, and displays error messages rather than writing them to a log file, so you don't have to go digging around for them.</p> <p>Since version 5.4, PHP can do this too. Unfortunately, if you're working on an app that uses a previous PHP version, or that is inextricably dependent on .htaccess files or some other Apache feature, this doesn't help you.</p> <p>There are ways to run Apache-dependent PHP apps in development; on Mac OS X they usually involve adding virtual hosts to your global Apache instance, or installing <a href="https://web.archive.org/web/20150226200708/http://mamp.info/">MAMP</a>. Neither of these give us the ability to conveniently start and stop our server with one command, or to have our logs conveniently appear as the output of that command. (MAMP also requires you to poke around in the settings of a GUI app to change your <code>DocumentRoot</code> every time you want to switch from one project to another.)</p> <p>So, if we can't use a development server instead of Apache, let's make Apache act like a development server.</p> <p>We'll use the Apache server that comes with OS X Mountain Lion, since the one from Homebrew inexplicably segfaults when you use PHP with it. But, we'll write our own configuration file for it. This is the bare minimum you need in an Apache config file for it to start up and serve static files:</p> <div class="highlight"><pre><span></span><span class="nb">Listen</span> <span class="m">8000</span> <span class="nb">ErrorLog</span> <span class="s2">&quot;|cat&quot;</span> <span class="nb">PidFile</span> <span class="sx">/full/path/to/a/directory/httpd.pid</span> <span class="nb">LockFile</span> <span class="sx">/full/path/to/a/directory/accept.lock</span> <span class="nb">DocumentRoot</span> <span class="sx">/full/path/to/your/project</span> <span class="nb">ServerName</span> localhost </pre></div> <p>Save that as httpd.conf. Some things to note here:</p> <ul> <li>We're serving up the site on port 8000, not 80, so we don't have to run Apache as root. If you have Web Sharing turned on in System Preferences, this also ensures we can fire up our Apache instance without interfering with the Web Sharing one.</li> <li>When the value for ErrorLog starts with a |, errors are piped out to the given command instead of written to a file. Since cat just copies standard input to standard output, we'll see errors in the terminal.</li> <li>All of the paths have to be absolute; relative paths won't work. For PidFile and LockFile, the directory must exist but the files must not.</li> </ul> <p>This will serve up static files, but that's about it; for other stuff, we need plugins.</p> <p>Open the httpd.conf that comes with Mac OS X, located at /etc/apache2/httpd.conf. Copy the lines that start with LoadModule into your custom httpd.conf. You may want to go through <a href="https://web.archive.org/web/20150226200708/http://httpd.apache.org/docs/2.2/mod/directives.html">each directive in your .htaccess files</a> and determine which modules you need, you may want to see what your production server is running and install those, or you could just copy them all.</p> <p><strong>Then, comment out <code>userdir_module</code></strong> if it's present (it was third from the bottom on my install). For some reason unknown to me, when this module is enabled at the same time as PHP, PHP somehow gets disabled and PHP scripts will be served up as static files instead of executed.</p> <p>Last but not least, we'll add PHP itself. Mac OS X ships with PHP 5.3, but I want PHP 5.2, so I'll install it from Homebrew. (Note that if you don't already have MySQL and PostgreSQL installed, you'll have to either install them or remove the appropriate flags from brew install.)</p> <div class="highlight"><pre><span></span>$ brew tap homebrew/dupes $ brew tap josegonzalez/php $ brew install php52 --with-psql --with-mysql </pre></div> <p>Now, tell Apache about your PHP installation:</p> <div class="highlight"><pre><span></span><span class="nb">LoadModule</span> php5_module <span class="sx">/usr/local/opt/php52/libexec/apache2/libphp5.so</span> <span class="nb">AddType</span> application/x-httpd-php .php <span class="nb">AddType</span> application/x-httpd-php-source .phps <span class="nb">DirectoryIndex</span> index.html index.php </pre></div> <p>Last but not least, run Apache in the foreground. (You'll notice that it really likes its absolute paths.)</p> <div class="highlight"><pre><span></span>$ httpd -f <span class="nv">$PWD</span>/httpd.conf -D FOREGROUND </pre></div> <p>You should now have a functioning Apache + PHP installation running right inside your terminal, separate from the OS's one.</p> Leigh Brenecki Not having a license does the exact opposite to what you think it does https://leigh.net.au/writing/20130324-not-having-a-license/ 2013-03-24T00:00:00+0930 2013-03-24T00:00:00+0930 <p>There's a lot of "open source" projects on GitHub and other places that don't have a license attached to them. If I had to guess why, I think the people sharing these projects don't put a license on because they don't care what people do with their code. They're sharing it with the world, the world can do what it wants with it.</p> <p>But that's not how copyright works - in effect, they're doing the exact opposite. Computer code is automatically copyrighted, meaning nobody can do anything with it without getting permission from the author. When you attach a license to your code, you're giving the world permission to use it.</p> <p>This means, code without a license can't legally be used. Not long ago at work, I spent several days completely reimplementing from scratch something that someone had already done, because they didn't have a license on their repo, so we couldn't legally use it. They'd put it up on GitHub, so they'd obviously <em>intended</em> for people to use it, but nobody could.</p> <p>If you have open source code without a license, it's time to add one. If you're not sure which, <a href="http://www.tldrlegal.com/">tl;drLegal</a> is a great site that summarises the differences between all your options. If you want anyone to be able to use your code to do anything with no restrictions at all, use the <a href="http://www.tldrlegal.com/license/do-wtf-you-want-to-public-license-v2">WTFPL</a> or <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0</a>.</p> <hr /> <p><strong>Update:</strong> It's been pointed out (thanks @lonetwin!) that there's a <a href="http://programmers.stackexchange.com/questions/26548/what-is-the-default-software-license/26652#26652">clause in Github's T&amp;Cs that sounds like it gives all public GitHub projects an 'implicit' or 'default' license</a>. However, there's a few things to note about this. One is that it sounds a lot like this clause is more so that if you use someone else's copyrighted work, they can't <em>sue GitHub</em>, but it still might not stop them suing you. Another is that it only says others can "view and fork", not edit, alter, distribute, use in derivative works, and all that other junk that proper licenses usually talk about. A third is that this is an agreement between the other user and GitHub—not the other user and you—which might be important. With all that in mind, I'd steer clear of making any assumptions about that clause without proper legal advice (which this isn't, because I'm not a lawyer).</p> Leigh Brenecki