Michael Melanson

Up to speed on the new features of Javascript

2019-11-21T14:31:29+00:00

A few years ago, many great new features were added to Javascript in the ES5, ES6 and ES7 language versions that make it a much better language for building large web applications and services. If you're someone who learned Javascript a few years ago but haven't learned these new features, this article will help get you up to speed.

A short history of Javascript</h2>
Javascript was created for the Netscape browser in 1995. The first version of the interpreter was written in only about two weeks, and so it was a quick and dirty implementation intended to be used for small interactive features in web pages.
In the next year it was standardized as "ECMAScript" (named after the standards body "European Computer Manufacturers Association"). This is where the name "ES6" comes from, but otherwise the name ECMAScript is rarely used. The language spread and became one of the most widely used languages in the world. A few attempts were made to evolve the language (ECMAScript 2-4) that never really caught on.
But once developers started using Javascript for single-page applications (web applications where all the routing between pages is handled in the browser, not by requesting new HTML pages from the server) in about 2005-2008, people started getting frustrated by its limitations. But it's hard to change a language that's used so widely.
This led to languages such as CoffeeScript (released in 2009) where developers could write code in a more sophisticated language, which was then compiled to Javascript so it could be run in a browser. This acted as a way to prototype new language features, and many parts of CoffeeScript were eventually rolled back into Javascript in ES5 (2015) and ES6 (2016). The latest version is ES7, released in 2017.
In the rest of this article I'm going to talk about many of the new features in these most recent releases of the Javascript language.

New variable bindings (let</code> and const</code>)</h2> In old Javascript there was only one way to declare a variable – by using var</code>. var name = "Michael"; console.log(name); // prints "Michael" </code></pre> var</code> still exists of course, but ES6 added two new binding types (ways of declaring variables): let</code> and const</code>. The difference between them is that let</code> can be reassigned to it, and const</code> can't: const name = "Michael"; name = "Bob"; // error let name = "Michael"; name = "Bob"; // okay </code></pre> But, it's important to understand that the const</code> binding doesn't make things immutable (unable to be changed at all). It just means that you can't reassign it to an entirely new value. You can, however, mutate the value it's pointing to – you can change characters in a string, or fields in an object like this: const michael = { name: "Michael", age: 34 }; // it's my birthday! michael.age = 35; </code></pre> What's the difference between let</code> and var</code>?</h3> If both let</code> and var</code> can be changed, why create a whole new binding type? The difference between them is that let</code> doesn't cause the variable to be hoisted. What's hoisting? In most programming languages, if you declare a variable in the middle of a scope then you can access that variable only later in the scope. If this were another language, this would throw an error about message</code> not being defined: console.log(message); var message = "Hello world!"; </code></pre> But Javascript is different; that snippet won't throw an error. What do you think that will print? The answer is that it will print the value undefined</code>. On the first line, message</code> exists as a variable in that scope even though it's before the var</code> statement. It just has the value undefined</code>. Why does it exist? Because the var</code> binding hoisted the variable to the top of the scope as if it were declared at the very top of the scope. Then the var</code> statement itself just assigns it the value "Hello world"</code>. It exactly the same as if you wrote this: var message = undefined; console.log(message); message = 'Hello world!'; </code></pre> But this doesn't happen with either let</code> or const</code>. You can only reference one of these bindings after they are declared. So let</code> is like var</code> except that it's not hoisted. tl;dr My recommendation is that there's no reason to use var</code> anymore. You should also prefer to use const</code> unless you actually need to reassign a variable. Array and object matching (destructuring)</h2> The next two features – destructuring, and object shorthand – are probably the new features I use the most. Destructuring is a syntax for declaring local variables and assigning them to fields from an object. It's best explained with an example: const michael = { name: "Michael", age: 34 }; const { name } = michael; // (1) console.log(name); // prints 'Michael' </code></pre> The second line there, marked (1)</code>, destructures the michael</code> variable, creating a variable name</code> initialized to michael.name</code>. It means the same thing as this: const name = michael.name; </code></pre> So far this seems not too useful. But you can do this for multiple fields at the same time: const { name, age } = michael; console.log(name, 'is', age, 'years old'); // print 'Michael is 34 years old' </code></pre> You can also do this on function parameters: function printBirthday({name, age}) { console.log(name, 'is', age, 'years old'); } printBirthday(michael); // prints 'Michael is 34 years old' </code></pre> The other feature of destructuring is that you can also use the spread operator (...</code>) to create an object with all the remaining fields in the object that you haven't mentioned: function printBirthday({name, age, ...rest}) { console.log(name, 'is', age, 'years old'); console.log("Here's some more things about them:", rest); } const michael = { name: "Michael", age: 34, city: "Ottawa" }; printBirthday(michael); // prints "Michael is 34 years old" and "Here's some more things about them: { city: 'Ottawa' }" </code></pre> Notice that it only showed the city</code> field, because that was the only field in the input object other than the name</code> and age</code> that were pulled out specifically. Destructuring is one of those features that when it's described like this it sounds kind of interesting. But once you start using it, it becomes a natural way of working with object values and you use it all the time. Shorthands for creating objects</h2> If destructuring is for taking apart objects, the flip side is the shorthand for putting them together. The first is a notation for creating fields bound to local variables: const name = "Michael"; const age = 34; const michael = { name, age }; // (1) console.log(michael.name, 'is', michael.age, 'years old'); </code></pre> On the line marked (1)</code>, it creates an object with two fields, name</code> and age</code>, and assigns them to the values from variables of the same name. It's the same thing as writing this: const michael = { name: name, age: age }; </code></pre> That's nifty and convenient and avoids extra noise giving the same name twice if the variables are the same name. I often use it to build up return objects like this: function someFunction() { const one = "first value"; const two = "second value"; return { one, two }; } </code></pre> This lets you deal with each field separately, then the return line just packages them up together into an object with the right shape. You can also use the spread operator we saw earlier when creating objects. This copies all the fields from one object into the new object. One place I use this a lot is when creating objects in test cases, where you have some fields that are required and some are specific to the test. // at the top of the file const defaultPerson = { name: "Michael", age: 34 }; // the test function it('people are adults when they turn 18', () => { const person = { ...defaultPerson, age: 18 }; // (1) expect(isAdult(person)).toEqual(true); }); </code></pre> On the line marked (1)</code>, we create a new person that's like the default one except that it has a different age</code>. Any other fields, like the name: "Michael"</code> field here, will have the value from defaultPerson</code>. If you want you can spread in multiple objects. That's perfectly okay – it'll just do any of the assignments in order: const person = { ...defaultPerson, age: 18, ...overrides }; </code></pre> In this case it will copy the fields from defaultPerson</code>, then assign age</code> to be 18</code>, then copy all the fields from overrides</code>. Arrow functions</h2> It's always been possible to write simple anonymous functions in Javascript and people have used this to write callbacks, for example. But one of the awkward things about Javascript is that when you do this, the this</code> variable is special. Its value isn't passed as part of the closure (the variables captured from the scope where a function is defined), but rather is set when the function is called. (The technical term for this is that this</code> is dynamically scoped but everywhere else in Javascript things are lexically scoped.) This difference led to lots of confusion and hacks where you'd have to write myFunction.bind(this)</code> to create a new version of the function where this</code> had a particular value no matter who calls it. The solution to this was to create a new way of writing inline functions, usually called "arrow functions" because of the =></code> operator you use to define them. They look like this: const printBirthday = ({name, age}) => { console.log(name, 'is', age, 'years old'); }; printBirthday(michael); // prints 'Michael is 34 years old' </code></pre> This does the same thing as an example earlier in the section on destructuring. It creates an arrow function, then assigns it to a variable printBirthday</code> which can then be called just like any function. If you use it inside a class, then the function will have the this</code> value you expect. Promises and async</code> / await</code></h2> I'm saving the best for last here because Promises, and the async</code> and await</code> syntax that came with them, are probably the single most significant change in Javascript. But first I have to go back a bit and talk about how the Javascript runtime works. Aside: How the Javascript runtime works.</h3> Javascript runs in a single threaded environment. It can only be running one line of code at a time – anything else needs to wait. There's no way for a web page to be running two pieces of Javascript code at the same time. (For the pedantic readers: I'm ignoring WebWorkers here, which is a whole other topic that you can read about if you're interested.) This has the great benefit of avoiding all the problems that you run into when you have a multithreaded environment – where two processors might be running different code in the same memory at the same time – meaning your code can be much simpler and easier to reason about. But the drawback is that if you want to stay responsive you can never block the main thread for more than a few milliseconds. Web browsers want to run at 60 frames per second, which gives you 16 milliseconds (1000 / 16) to work before you start delaying the next frame. If this happens then the interface will start to feel unresponsive or "janky". To make it appear like we can do more than one thing, we have to interleave pieces of work – instead of waiting, we need to be able to give up the processor to let something else run. You can see this interleaving if you open the "Performance" tab in your browser's development tools. If you click around a page you'll see something like this: See how the top bar is sliced up with many different bars? These are pieces of work being scheduled on the main thread. This interleaving is done with *event handlers – *callback functions that get invoked when interesting events happen. If you want to know when a button gets clicked, you can set a click handler.  <div> <button onClick={onButtonClicked}>Click me</button> </div> </code></pre> // in your Javascript function onButtonClicked() { console.log("You clicked the button!"); } </code></pre> This means that you don't need to write some loop to like while (!button.clicked()) { ... }</code> to wait for a click. You can set the handler, then give up the processor for other work. If a click happens, the browser knows how to let you know so you can respond to it. The same is true for network requests, by the way. When you make a request, it may take a long time (up to a few seconds) for a result to come back. Rather than waiting for a response, you can set an event handler on the request to say "call this function when a result comes back". Then you can give up the main thread so other work can happen. Callback hell</h3> This all works great for simple cases. The problem is what happens when you start building larger applications. Sometimes you want to build code that does a sequence of steps. You want to do something, then wait for an event, then do another thing. You also need error handlers for lots of asynchronous operations. Very quickly, you end up with very deeply nested code that looks like this: doOneThing(function() { doSecondThing(function() { console.log("All done!"); }, function() { console.log("An error happened on the second thing!"); }); }, function() { console.log("An error happened on the first thing!"); }); </code></pre> Imagine this being several layers deep. It becomes an unmanageable, tangled mess of callback functions. People called this "callback hell". How promises work</h3> Promises help with callback hell by giving you a way of representing some work you want to happen in the future once a value is available. A promise is just a Javascript object that you can call .then(...)</code> on giving it a function. It will call that function when the promise is resolved if the work completes successfully. You can also call .catch(...)</code> on it to give it a function you want called if the promise is rejected because the work failed. Both of these functions – then</code> and catch</code> – each return promises themselves. This means that they can be chained: fetch('/products.json') .then(({products}) => console.log("These are the products in your store:", products)) .then(() => console.log("All done!")) .catch((error) => console.log("An error occurred:", error)); </code></pre> This lets you flatten out what would be nested nested callbacks into a sequence of steps. It's also composable – you can combine together promises in a natural sort of way. Let's say we wanted to hide the error handling logic in the catch</code> step. You could do this by writing a new function that will fetch the products, and also handles errors if it fails: function fetchProducts() { return fetch('/products.json') .then((result) => { console.log("All done!"); return result; }) .catch((error) => console.log("An error occurred fetching the products:", error)); } </code></pre> This function returns a promise that resolves once the fetch occurs – and it also prints a message and handles the error. Now someone can call it and chain on their own work to use the products that were just fetched: fetchProducts() .then(({products}) => console.log("These are the products in your store:", products)); </code></pre> Being able to create higher level operations like this is one of the most powerful parts of promises. It was very hard to do with just callbacks. async</code> and await</code></h3> As great as promises are all by themselves, writing the list of then</code> and catch</code> clauses is still awkward and hard to read, especially if you need to do loops. You need to be sure that you chain promises together properly and never leave them 'dangling'. The next step was to add syntax to Javascript to make promises a core part of the language so asynchronous code is easier to write and reads more like synchronous code. The async</code> and await</code> keywords work together. You put async</code> on a function where you want to use await</code>. Inside an async</code> function, you can use await</code> to essentially wrap everything after that line in a then</code> clause. You can also use a try</code>/catch</code> block to catch rejected promises the same way as any other exception in Javascript. Let's rewrite the fetchProducts</code> function above again to use this: async function fetchProducts() { try { const result = await fetch('/products.json'); console.log("All done!"); return result; } catch(error) { console.log("An error occurred fetching the products:", error); } } </code></pre> Then you can use it like this: const {products} = await fetchProducts(); console.log("These are the products in your store:", products); </code></pre> In this way you can almost pretend that you're writing regular code and forget that it will run asynchronously. But it will. This code won't actually wait for the result to come back. It will break and give up the processor and let other things use the processor. Then, when a result eventually comes back from the server (or an error occurs) your code will pick up where it left off. The await</code> statement works properly even in the case of loops, if</code> conditions, and so on. More reading</h2> I hope this article was helpful to you! I certainly haven't covered all the new features in Javascript. This is just a few I think are the most important but there's more, including classes, the module system, template variables, and many more! If you want to learn more about them I suggest going to es6-features.org</a> where you'll find a list of all the features. If you want more details about any of the jargon terms I've used in this article, I suggest googling for mdn <term></code>. This will point you to the Mozilla Developer Network's documentation, which is a great resource that has all the details about every part of Javascript.

Running on the treadmill 2017-10-03T02:44:00+00:00 If there’s one thing that’s always true about software development, it’s that it just never sits still. Just when you think you’re on top of things, a new framework comes along and suddenly you feel out-of-date and you’re worried all the cool kids will laugh at you. But there’s some bits of evergreen advice I can give. Knowing vs. knowing</h4> No one knows everything. No one is an expert on every single Javascript library out there. Most of them are crap anyway. Realize that there's many levels of ‘knowing’ something – Awareness: “I’ve heard the word before.”</li> Familiarity: “I’ve read some stuff and know the general idea behind it.”</li> Tinkerer: “I’ve played around with it or made something small.”</li> Experienced: “I’ve shipped a real product that uses it.”</li> Expert: “I know it inside and out and can make it do things the author never intended.”</li> </ul> What do you mean when you say you 'know' something? Are you sure that's what other people mean? Set unrealistic expectations about how many things you can be an expert on. One thing? You’re doing well. And you can say you ‘know’ something long before you work your way through all of them and definitely when you’re experienced. You might imagine that if someone else says they ‘know’ something that they’re an expert, but then you look at your own skill set and think “I still have to check the docs every five minutes and Stack Overflow several times a day”. We all do. That’s okay. Tick tock.</h4> If you squint a bit, new developments follow a metronome pattern. There’s a tick phase where people have a new problem of some kind. They solve it by adding or replacing a thing in their system’s tech stack. Then there’s a tock phase where people look at that solution and try to do the same thing only better: merge two things together, make it faster, give it a better developer experience, more repeatable, whatever. It’s okay to ignore the tick phase altogether. When you start a new system, start at the last tock and skip from there to the next tock when it comes along. Usually at that point there’s usually a clear advantage to moving from one tock to the next. You don’t have to stay on the bleeding edge. But even then make sure it makes sense for you and your customers compared to everything else you could be spending time working on. It’s rarely necessary to use a tick technology unless it’s a specific competitive advantage for you. You might want to anyway if that’s fun for you, but you don’t have to. Mind the mess.</h4> Don’t let ideologues fool you. Building software in the real world is never as neat and tidy as we’d like. Real software always has dust bunnies under the couches. When you decide to adopt a new technology, you do not have to adopt it 100%. Invest in it where it makes sense to, leave things as-is when there’s no reason to change it. Are you using fancy newfangled client-rendered React but there’s some dusty corner of your system where things are plugging along with server-rendered JSP templates? Relax, don’t worry about it. If you have to come back to that dusty corner, perhaps because it’s producing bugs or because you need to modify it for some new work, then think about updating it then. But if it’s quietly doing its thing? Let it be, focus on more important things. Code is code.</h4> Never forget that no matter what framework you’re using, how it’s built, or how it’s delivered to your users, the same fundamentals apply. It’s all just code. Optimize for being able to come back later, understand the code easily, and change it without breaking other stuff. Look out for each other.</h4> Do you feel like you can never keep up-to-date with all the new stuff? Being a software developer can feel like you’re always on a treadmill going just a bit too fast. You’re always worried you’re one misstep away from face-planting. Take a breath. If you’re even reading posts like this one and thinking these thoughts, you’re already doing plenty. Do a good day’s work, leave your code in a better place than it was yesterday, then go home. Spend time with your family and friends, read a book, do some Judo 🥋. Whatever helps give you perspective about how insane our industry can be. Most importantly, realize that everyone else around you feels insecure about this too. Be kind and helpful to each other as we all muddle through this together. Transactions: the limits of isolation 2014-12-01T02:37:00+00:00 At Toushay, one of our major feature was 'digital signage' — lists of images and videos we'd display on the screens of tables. We ran into a tricky issue: our customers were noticing that, very rarely, these digital signage playlists were being subtly corrupted. After our customers had been moving items around in the list, things would go awry. Sometimes an image would be lost, other times one would be duplicated. It happened only a few times, and in each case we were able to manually correct it. But it was very painful – and trust-breaking – for them when it did. Clearly something was wrong with how we were handling this data. The problem was that it wasn’t reproducible on a developer’s machine, it didn’t show up in our integration tests, and occurred only very rarely in production. I was tasked with figuring out the source of the problem and solving it once and for all. At first, everything worked okay on my machine and I couldn’t reproduce it. On a hunch that it may only occur on a database under high load, I tried hammering my local database with as much load as I could manage with this command: mysqlslap -a --auto-generate-sql-execute-number=10000 --concurrency 140 </code></pre> That command spins up 140 MySQL clients, and each one hits the database with 10,000 random queries as fast as they can. That keeps the server busy for several minutes with much higher load than our production servers will ever see. With that command running, I was able to reproduce the problem fairly consistently: duplicated images, missing images, the whole bit. Clearly, the extra database load was causing the different requests to step on each other’s toes. But how? The code to move an item in the list basically works like this: Read in the list of images.</li> Check that the item we’re moving is in the list and at the position we expect. This is a sanity check to ensure the UI isn’t running on stale data.</li> Move the item to the new position.</li> Save the list to the database.</li> </ol> This block of code is run within a database transaction (I checked to be sure), which should isolate it from other transactions. Yet, the behaviour could only be caused by two requests colliding with each other something like this: Request A</th> Request B</th></tr></thead> Read in the list of images.</td> </td></tr> Check that the item we’re moving is in the list and at the position we expect.</td> </td></tr> </td> Read in the list of images.</td></tr> </td> Check that the item we’re moving is in the list and at the position we expect.</td></tr> Move the item.</td> </td></tr> Save the list to the database.</td> </td></tr> </td> Move the item.</td></tr> </td> Save the list to the database.</td></tr> </tbody></table> But how can this be? Transactions are one of the key features of any relational data store and, as we all learned in school, transactions provide ACID guarantees. Shouldn’t the Isolation guarantee prevent two transactions from interfering with each other like that? By digging deeper, I realized what was going on. One key fact that is not well known about transactions, or is easily forgotten, is this: by default, databases do not provide full Isolation guarantees. How isolated is isolated?</h4> What exactly the Isolation guarantee in the SQL standard means based on what they call “read phenomena”. There are three types of phenomena: Dirty reads – If another transaction writes, but does not commit, during your transaction, is it possible that you will see their data?</li> Non-repeatable reads – If you read the same row twice, is it possible that you might get different data the second time?</li> Phantom reads – If you read a collection of rows twice, is it possible that different rows will be returned the second time?</li> </ul> In the SQL standard, there are four levels of transactional isolation based on which of these phenomena they prevent (from weakest to strongest): Read Uncommitted – A transaction can see the results of another transaction’s INSERT</code> or UPDATE</code> queries before they are committed. This allows all read phenomena.</li> Read Committed – A transaction is guaranteed to only see rows that are committed. This prevents dirty reads but allows the other phenomena.</li> Repeatable Read – A transaction is gets a snapshot of all rows it reads during the transaction. This prevents dirty reads and non-repeatable reads, but allows phantom reads.</li> Serializable – A transaction gets a “range lock” any time it queries a collection of rows, guaranteeing that it sees a snapshot of that data. This prevents phantom reads as well as dirty reads and non-repeatable reads.</li> </ul> The default isolation level, at least for PostgreSQL, MySQL and Oracle, is Read Committed and so that’s what we were using in our application. You probably are in your application to, unless you’ve specifically changed the isolation level. If that’s the case, you can’t assume that because you read in some data that someone can’t change it underneath you before you commit. The reason they don’t use a higher default level is that both Repeatable Read and Serializable both introduce cases where a transaction cannot be completed and must be aborted (race conditions). In these cases, the DBMS aborts the transaction, rolls it back, and returns an error. When this happens, the application has to know how to retry the transaction. In some cases it may be difficult to retry a transaction if it had side effects on the application’s side: talking to external services or other data stores. In these cases, the application would have to use something like a two-phase commit protocol</a>, or else manually roll back the side effects. The solution</h4> The corruption we were seeing was the result of non-repeatable reads and phantom reads: we were reading a range of images, doing some checks on them, then modifying that range and writing them out. In between the reads and writes, there’s a window where another transaction could sneak in and insert or remove an image, or rearrange the list. Any of those cases could cause corruption of the list when you make your changes and overwrite the new changes. There are two solutions to this: either you use row locks (probably a shared lock, but possibly exclusive) to prevent any concurrent access to the rows, or you use stricter transaction isolation. In our case, we need Serializable isolation to prevent the phenomena we were seeing. The problem with both of these is that you take a performance hit. We checked what impact enabling Serializable isolation globally (by setting the MySQL server variable tx_isolation = SERIALIZABLE</code>) would have, and the results were… mixed. In some cases, the effect was negligible; in other cases, it raised the response time by up to 10x. And in all cases, it made the response time much more variable. So switching to Serializable everywhere was a no-go. We also didn’t want to use row locks: it’s easy to miss a case where a lock is necessary, and need to be used properly to prevent deadlocks. They’re also often stricter than we need, which may incur a greater performance hit than is necessary. In the end, we switched to Serializable isolation on the particular operations where the stricter isolation is necessary. In all the rest, we continue to use the default Read Committed isolation. Lessons learned</h4> This was a very subtle and interesting bug. It came about from a fairly serious misunderstanding of how transactions work. But it’s an easy one to make: it’s natural to think of transactions as some sort of magic that protects you from concurrency issues. Going into this, my own mental model was that transactions gave something like Serializable isolation by default. I’m sure I’m not alone there, which is why I wrote this blog post. Here the key lesson you can take away from this: be sure to understand your database’s documentation in detail. I got all the information in this post from the MySQL</a> and PostgreSQL</a> documentation. It also showed us that there’s a gap in our testing methodology: the tests are run on an unloaded database, and transaction isolation issues don’t appear until you have multiple competing transactions. When you’re doing integration testing, consider running mysqlslap</code>, or similar, in the background. This should increase the chances that isolation problems will occasionally cause your tests to fail. Using Git's interactive rebase 2013-06-17T02:24:00+00:00 Interactive rebase is a one-stop shop for revising history: combining, removing and reordering commits. It’s one of those things where you learn it, and afterwards wonder how you got by without knowing it. I use it almost every day when preparing a topic branch for review. If I have to make changes during review, I commit them as incremental patches, then squash them with an interactive rebase before I commit. Update (June 2019): This article was originally written in 2013. You know how it feels to come back to something you did a long time ago and all you can do is see everything wrong with it? Well I just did that and, long story short, spent a while editing it and updating it so it doesn't make me cringe quite as much. Cheers! </blockquote> In this post, I’ll explain a bit about how it works. Let’s forget about topic branches right now, and just assume you’re working directly on the master. Let's say you’ve been working a bit on a new feature, and if you're anything like me if you run git log</code> you'll find you've ended up with history that looks like this: master</code> Fix a bug when you foo the bar</li> Fix typo</li> Implement the remainder of feature XYZ</li> Oops, forgot to add this file</li> Implement part of feature XYZ</li> origin/master</code>(Other commits already pushed to the upstream repo)</li> </ul> There's five local commits here, but only two of them are actual commits. The rest are fixing errors or bugs in code that no one else has seen. Leaving them in the history would just clutter it up. It's best for each commit to represent a complete bit of work so it's easier to understand by reviewers now, and by other developers later if they have to go back into the history. This is where "interactive rebase" comes in. It's a tool in Git to let you edit the recent history in almost any way you could want. The best way to understand it is to actually use it. Since we’ve added five commits here, we start by running a rebase with the -i</code> ("interactive") flag: git rebase -i HEAD~5 </code></pre> This will open up your editor – the same one you'd use to write a commit message – and at the top you'll see something that looks like this: pick 39a2bef Implement part of feature XYZ pick 4bc987e Oops, forgot to add this file pick f9282bc Implement the remainder of feature XYZ pick 2c5dd33 Fix typo pick a524b5c Fix a bug when you foo the bar </code></pre> On each line, you have the command (pick</code> here), the commit ID, and the description. Below that, there's some instructions about what's going on including a list of commands you can use: pick</code> — leave the commit as-is</li> reword</code> — change the commit summary only (this is a recent addition)</li> edit</code> — change the commit contents, much like doing a git reset --mixed</code></li> squash</code> — keep the changes in this commit, but merge it with the commit on the previous line, and append the commit messages and allow the message to be edited.</li> fixup</code> — keep the changes, like with squash but throw away the commit message instead of appending them.</li> </ul> You can also delete a line to discard a commit entirely, or reorder them. The idea is that you edit this file so it shows the history you wish you created, then write the file and close your editor. Let’s change it so it looks like this: pick 39a2bef Implement part of feature XYZ fixup 4bc987e Oops, forgot to add this file pick f9282bc Implement the remainder of feature XYZ fixup 2c5dd33 Fix typo fixup a524b5c Fix a bug when you foo the bar </code></pre> All I've done here is change the command “pick</code>” to “fixup</code>” for all the mistake commits I made. This will roll them in with the commits above them so that, when we close the editor, Git will rewrite the commit history so it looks like this: master</code> Implement the remainder of feature XYZ</li> Implement part of feature XYZ</li> origin/master</code>(Other commits already pushed to the upstream repo)</li> </ul> It's important to understand here that all the changes from the commits are still there! They've just been incorporated into the previous commit. So, for example the changes in the "Oops, forgot to add this file" commit, presumably to add the missing file, is now part of the previous "Implement part of feature XYZ" commit where it belongs. As I said, I use this almost daily to prepare a patch set for review. After I rebase, I prepare patch emails with git format-patch -2 --attach</code>, then send them with git send-email 000[12]* --to=mailinglist@somewhere.com</code>. Update (June 2019): I'm leaving that last paragraph in for posterity. I can't believe that used to be my workflow! If I were writing this today I would have talked about posting a pull request with the changes. </blockquote> I hope that helps you out. Interactive Rebase is a very useful feature of Git that I haven’t seen in other source control systems. It can be a bit confusing at first but once you get the hang of it, it's a very handy tool making the history clean and easy to understand for reviewers and for anyone who may look at the history later.

Request A</strong></th>	Request B</strong></th></tr></thead>
Read in the list of images.</td>	</td></tr>
Check that the item we’re moving is in the list and at the position we expect.</td>	</td></tr>
</td>	Read in the list of images.</td></tr>
</td>	Check that the item we’re moving is in the list and at the position we expect.</td></tr>
Move the item.</td>	</td></tr>
Save the list to the database.</td>	</td></tr>
</td>	Move the item.</td></tr>
</td>	Save the list to the database.</td></tr> </tbody></table> But how can this be? Transactions are one of the key features of any relational data store and, as we all learned in school, transactions provide ACID guarantees. Shouldn’t the Isolation guarantee prevent two transactions from interfering with each other like that?</p> By digging deeper, I realized what was going on. One key fact that is not well known about transactions, or is easily forgotten, is this: by default, databases do not provide full Isolation guarantees.</p> How isolated is isolated?</h4> What exactly the Isolation guarantee in the SQL standard means based on what they call “read phenomena”. There are three types of phenomena:</p> Dirty reads</strong> – If another transaction writes, but does not commit, during your transaction, is it possible that you will see their data?</li> Non-repeatable reads</strong> – If you read the same row twice, is it possible that you might get different data the second time?</li> Phantom reads</strong> – If you read a collection of rows twice, is it possible that different rows will be returned the second time?</li> </ul> In the SQL standard, there are four levels of transactional isolation based on which of these phenomena they prevent (from weakest to strongest):</p> Read Uncommitted</strong> – A transaction can see the results of another transaction’s `INSERT</code> or UPDATE</code> queries before they are committed. This allows all read phenomena.</li>` `Read Committed</strong> – A transaction is guaranteed to only see rows that are committed. This prevents dirty reads</em> but allows the other phenomena.</li>` `Repeatable Read</strong> – A transaction is gets a snapshot of all rows it reads during the transaction. This prevents dirty reads</em> and non-repeatable reads</em>, but allows phantom reads</em>.</li>` Serializable</strong> – A transaction gets a “range lock” any time it queries a collection of rows, guaranteeing that it sees a snapshot of that data. This prevents phantom reads</em> as well as dirty reads</em> and non-repeatable reads</em>.</li> </ul> The default isolation level, at least for PostgreSQL, MySQL and Oracle, is Read Committed and so that’s what we were using in our application. You probably are in your application to, unless you’ve specifically changed the isolation level. If that’s the case, you can’t assume that because you read in some data that someone can’t change it underneath you before you commit.</p> The reason they don’t use a higher default level is that both Repeatable Read and Serializable both introduce cases where a transaction cannot be completed and must be aborted (race conditions). In these cases, the DBMS aborts the transaction, rolls it back, and returns an error.</p> When this happens, the application has to know how to retry the transaction. In some cases it may be difficult to retry a transaction if it had side effects on the application’s side: talking to external services or other data stores. In these cases, the application would have to use something like a two-phase commit protocol</a>, or else manually roll back the side effects.</p> The solution</h4> The corruption we were seeing was the result of non-repeatable reads and phantom reads: we were reading a range of images, doing some checks on them, then modifying that range and writing them out. In between the reads and writes, there’s a window where another transaction could sneak in and insert or remove an image, or rearrange the list. Any of those cases could cause corruption of the list when you make your changes and overwrite the new changes.</p> There are two solutions to this: either you use row locks (probably a shared lock, but possibly exclusive) to prevent any concurrent access to the rows, or you use stricter transaction isolation. In our case, we need Serializable isolation to prevent the phenomena we were seeing.</p> The problem with both of these is that you take a performance hit. We checked what impact enabling Serializable isolation globally (by setting the MySQL server variable tx_isolation = SERIALIZABLE</code>) would have, and the results were… mixed. In some cases, the effect was negligible; in other cases, it raised the response time by up to 10x. And in all cases, it made the response time much more variable. So switching to Serializable everywhere was a no-go.</p> We also didn’t want to use row locks: it’s easy to miss a case where a lock is necessary, and need to be used properly to prevent deadlocks. They’re also often stricter than we need, which may incur a greater performance hit than is necessary.</p> In the end, we switched to Serializable isolation on the particular operations where the stricter isolation is necessary. In all the rest, we continue to use the default Read Committed isolation.</p> Lessons learned</h4> This was a very subtle and interesting bug. It came about from a fairly serious misunderstanding of how transactions work. But it’s an easy one to make: it’s natural to think of transactions as some sort of magic that protects you from concurrency issues. Going into this, my own mental model was that transactions gave something like Serializable isolation by default. I’m sure I’m not alone there, which is why I wrote this blog post.</p> Here the key lesson you can take away from this: be sure to understand your database’s documentation in detail. I got all the information in this post from the MySQL</a> and PostgreSQL</a> documentation.</p> It also showed us that there’s a gap in our testing methodology: the tests are run on an unloaded database, and transaction isolation issues don’t appear until you have multiple competing transactions. When you’re doing integration testing, consider running mysqlslap</code>, or similar, in the background. This should increase the chances that isolation problems will occasionally cause your tests to fail.</p> Using Git's interactive rebase 2013-06-17T02:24:00+00:00 Interactive rebase is a one-stop shop for revising history: combining, removing and reordering commits. It’s one of those things where you learn it, and afterwards wonder how you got by without knowing it. I use it almost every day when preparing a topic branch for review. If I have to make changes during review, I commit them as incremental patches, then squash them with an interactive rebase before I commit.</p> Update (June 2019):</strong> This article was originally written in 2013. You know how it feels to come back to something you did a long time ago and all you can do is see everything wrong</em> with it? Well I just did that and, long story short, spent a while editing it and updating it so it doesn't make me cringe quite as much. Cheers!</p> </blockquote> In this post, I’ll explain a bit about how it works. Let’s forget about topic branches right now, and just assume you’re working directly on the master.</p> Let's say you’ve been working a bit on a new feature, and if you're anything like me if you run git log</code> you'll find you've ended up with history that looks like this:</p> master</code> Fix a bug when you foo the bar</li> Fix typo</li> Implement the remainder of feature XYZ</li> Oops, forgot to add this file</li> Implement part of feature XYZ</li> origin/master</code>(Other commits already pushed to the upstream repo)</em></li> </ul> There's five local commits here, but only two of them are actual commits. The rest are fixing errors or bugs in code that no one else has seen. Leaving them in the history would just clutter it up. It's best for each commit to represent a complete bit of work so it's easier to understand by reviewers now, and by other developers later if they have to go back into the history.</p> This is where "interactive rebase" comes in. It's a tool in Git to let you edit the recent history in almost any way you could want. The best way to understand it is to actually use it. Since we’ve added five commits here, we start by running a rebase with the -i</code> ("interactive") flag:</p> git rebase -i HEAD~5 </code></pre> This will open up your editor – the same one you'd use to write a commit message – and at the top you'll see something that looks like this:</p> pick 39a2bef Implement part of feature XYZ pick 4bc987e Oops, forgot to add this file pick f9282bc Implement the remainder of feature XYZ pick 2c5dd33 Fix typo pick a524b5c Fix a bug when you foo the bar </code></pre> On each line, you have the command (pick</code> here), the commit ID, and the description. Below that, there's some instructions about what's going on including a list of commands you can use:</p> pick</code> — leave the commit as-is</li> reword</code> — change the commit summary only (this is a recent addition)</li> edit</code> — change the commit contents, much like doing a git reset --mixed</code></li> squash</code> — keep the changes in this commit, but merge it with the commit on the previous line, and append the commit messages and allow the message to be edited.</li> fixup</code> — keep the changes, like with squash but throw away the commit message instead of appending them.</li> </ul> You can also delete a line to discard a commit entirely, or reorder them. The idea is that you edit this file so it shows the history you wish you created, then write the file and close your editor. Let’s change it so it looks like this:</p> pick 39a2bef Implement part of feature XYZ fixup 4bc987e Oops, forgot to add this file pick f9282bc Implement the remainder of feature XYZ fixup 2c5dd33 Fix typo fixup a524b5c Fix a bug when you foo the bar </code></pre> All I've done here is change the command “pick</code>” to “fixup</code>” for all the mistake commits I made. This will roll them in with the commits above them so that, when we close the editor, Git will rewrite the commit history so it looks like this:</p> master</code> Implement the remainder of feature XYZ</li> Implement part of feature XYZ</li> origin/master</code>(Other commits already pushed to the upstream repo)</em></li> </ul> It's important to understand here that all the changes from the commits are still there! They've just been incorporated into the previous commit. So, for example the changes in the "Oops, forgot to add this file" commit, presumably to add the missing file, is now part of the previous "Implement part of feature XYZ" commit where it belongs.</p> As I said, I use this almost daily to prepare a patch set for review. After I rebase, I prepare patch emails with git format-patch -2 --attach</code>, then send them with git send-email 000[12]* --to=mailinglist@somewhere.com</code>.</p> Update (June 2019):</strong> I'm leaving that last paragraph in for posterity. I can't believe that used to be my workflow! If I were writing this today I would have talked about posting a pull request with the changes.</p> </blockquote> I hope that helps you out. Interactive Rebase is a very useful feature of Git that I haven’t seen in other source control systems. It can be a bit confusing at first but once you get the hang of it, it's a very handy tool making the history clean and easy to understand for reviewers and for anyone who may look at the history later.</p>