I haven’t written much about the mechanics of my actual work in a while since many of the platforms and frameworks I used at Stripe are proprietary, making it difficult to write about them and share code in a useful manner. My new company, Vori, is a startup and we take heavy advantage of open source software, so I can more easily write about my work and share code.
Vori uses TypeORM to interact with our Postgres database. We were using an older version, 0.2.x, but upgraded to 0.3.x in May of this year as we integrated NestJS into our backend.
Version 0.3.0 introduced multiple breaking changes that made it impossible to install the new version without significant changes to our code base. My initial upgrade attempt resulted in a few hundred TypeScript errors.
A simple find-and-replace would not fix these errors but, while at Stripe, I learned about codemods. Codemods are scripts that modify the codebase itself. Stripe used codemods to migrate from Flow to TypeScript, adjust how our Ruby package layout, and more that I’ve since forgotten. I never used codemods at Stripe, but I was happy to learn more about them for this project.
Let me first explain why the codemods were necessary in a little more detail.
Normally, when a method signature or name changes for a library, I might resolve the issues with find-and-replace or a regex. In this case the signature change removed part of a union type.
Let’s examine the findOne() method used to load a single entity from the database. In 0.2.x, this method had three different signatures we made use of:
/** Adapted for brevity from:
* https://github.com/typeorm/typeorm/blob/0.2.45/src/repository/Repository.ts#L363-L376.
* https://github.com/typeorm/typeorm/blob/0.2.45/src/find-options/FindOneOptions.ts
*/
// Load by ID
findOne(id: string)
// Load by matching different fields
// Example: findOne({ deleted_at: IsNull() }) ==> SELECT * FROM <table> WHERE deleted_at IS NULL
findOne(conditions: FindConditions)
// Load with a robust number of options such as selecting specific fields, loading relations, or changing sort order
// Example: findOne({ where: { deleted_at: IsNull() }, relations: ['store'] }) ==> (Same WHERE clause as above, and with a JOIN to the `stores` table)
findOne(options: FindOneOptions)
Here are the equivalents for 0.3.0:
// Adapted for brevity from https://github.com/typeorm/typeorm/blob/0.3.0/src/repository/Repository.ts#L524-L540.
// Equivalent to the 0.2.x counterpart
async findOne(options: FindOneOptions)
// FindOptionsWhere (0.3.x) is equivalent to FindConditions (0.2.x)
async findOneBy(where: FindOptionsWhere)
Find-and-replace with a regex can detect findOne(id) calls:
findOne\((\w+)\)
We can easily replace this with findOne({ where: { id: $1 } }) to use the new method signature.
Discerning between FindConditions and FindOneOptions, across multiple lines due to Prettier formatting is beyond my abilities and would be error-prone. We had the same problem for other methods.
We could have brute-forced this problem and divided the manual work amongst the team of six backend engineers; however, this would have been disruptive to higher-priority work and, honestly, boring and tedious.
I decided to build a more automated approach that would allow me to work on this off-and-on in a repeatable fashion. Since this work is not my highest priority, I needed to be able to drop it for a few days and not worry about redoing code modifications impacted by merge conflicts. Thus, codemods!
Pretty much every search I ran related to “JS codemod” or “TypeScript codemod” led me to jscodeshift. I found it confusing and lacking in examples. I still do, but I at least know enough now to quickly modify our codebase.
Essentially jscodeshift offers a jQuery-like API to navigate the abstract syntax tree (AST). The best way to begin to understand AST is to use a visualizer. I alternated between the two below. The latter occasionally showed more detail that helped me write more accurate code queries.
After a bit of trial-and-error, I created this codemod to correct the calls to findOne:
/**
* Updates calls to findOne.
*
* Replace findOne(FindConditions) with findOneBy(FindOptionsWhere).
*
* @note Use a regex to update findOne(string).
*/
module.exports = function (fileInfo, api, options) {
const j = api.jscodeshift;
const root = j(fileInfo.source);
// Find all calls to findOne
return root
.find(j.CallExpression)
.filter((p) => p?.node?.callee?.property?.name === 'findOne')
.replaceWith((nodePath) => {
const { node } = nodePath;
// Ignore non-TypeORM methods
// NOTE: findOne(id) was fixed by find-and-replace.
let optionsArgument = node.arguments[0];
if (optionsArgument.type !== j.ObjectExpression.toString()) {
return node;
}
// If the argument object has a `where` key, the object is a
// `FindOneOptions` and is compatible with the new signature.
// No changes are needed.
let whereProperty = optionsArgument.properties.find(
(p) => p.key.name === 'where'
);
// If the argument does NOT have a `where` key the object is a
// `FindConditions`, and we need to change the method from
// `findOne` to `findOneBy`.
if (!whereProperty) {
node.callee.property.name = 'findOneBy';
}
return node;
})
.toSource();
};
I ultimately wrote codemods to update the following query-related methods:
countfindfindOneI also wrote a couple to centralize our usage of the global-scoped getManager and getRepository methods which are deprecated.
These codemods are available at https://github.com/clintonb/codemods. I’m not sure how many others are using TypeORM 0.2.x but, hopefully, this makes the upgrade to 0.3.x a little less onerous. I wish these had been provided by the TypeORM maintainers to decrease the friction around upgrades.
Codemods resolved the majority of the few hundred type check errors that popped up after the upgrade, saving us at least a developer-week’s worth of effort. I resolved the last couple dozen manually since it was faster than writing more codemods.
Nothing broke in production! Our test coverage isn’t great, but type checking offered great assurance that the codemods worked. Increased type safety motivated the TypeORM maintainers to make the signature changes in the first place! I was prepared to quickly rollback the release in the event of an incident, but a rollback was not needed.
I’m happy to have a new tool at my disposal for large scale changes. I’m still a relative novice, but it was quite fun to play with these codemods that I had heard so much about at Stripe, but never used. I’m already thinking through the codemods we can use to help us migrate from type-graphql to nestjs/graphql.
If anyone has a tool that combines the AST with type checking, please share!
If you are a library maintainer making a breaking change, consider including codemods to decrease the friction around upgrading. It’s extra work, but saves duplicated work by consumers.