Safeguarding changes using update plans
At the company I’m currently working at, a.k.a. re:cap, we have some manual data imports that are business-critical, e.g., a customer-provided CSV file that we want to import into our database and that affects business operations. A simple pattern I came to love to make this process fast & safe is using update plans.
What are update plans?
Update plans are a way to preview changes before they are applied.
If you know Terraform, you already know update plans. Terraform is a tool that is used to manage infrastructure using a declarative programming language. Changing infrastructure is extremely delicate, remove one line of code, and the production server could be deleted, increasing the avg. heart rate of the team by 50 BPM. To safeguard our servers and heart rates, an infrastructure update is typically done with the following steps:
- The developer changes the infrastructure code as they desire, e.g., increasing the memory of a server from 8 to 16 GB.
- The developer executes the command "terraform plan". This will compare the wanted changes to what is currently active. The result is a plan on “how to move the current state to the desired state”.
- The developer verifies that the changes look sound.
- The developer executes "terraform apply" to actually apply the changes.
The key safeguard is that Terraform doesn’t directly apply the changes but first creates a plan to transition the current state to the desired one, allowing the user to review and approve the plan. This ensures that the developer can verify the changes are as intended, catching any undesired effects early.1
Some more examples:
- Checkout pages are basically an update plan - they show you what you’ll buy, the amount of money to be deducted, where to send it, etc. One can read through all this before hitting the "buy" (or rather "apply") button.
- A git diff is an update plan - it shows you the changes that you are about to commit.
- A file deletion dialog is an update plan - it often shows how many files will be deleted and asks you to confirm.
Update plans for database changes
Not every update is as important as updating your infrastructure and needs an explicit plan & apply step. But the pattern to split important changes into a plan & apply step is useful not only in infrastructure management but in all cases where one can preview what the final changes will look like. Database updates fit this very well, and I will show you how this can look like.
An example: say you are working on the system for a supermarket chain and the prices for each product are shown on a tiny screen that is controlled by a centralized server, which gets the prices from a database. The prices are updated quite often, but sadly still in a manual way as the supermarkets’ fast paced environment never allowed them to address tech debt. To update the prices, someone has to upload a CSV file with two columns: product_id, price. Let’s model this with an update plan to make sure that the price for frozen pizza is always correct. I’ll use Go in the following example, because it's a great language.
First, we define our product data model: a simple ID & price combination. Before people scream that one should never use a float for a monetary value, we will use an int that represents "money" in cents.2
type Product struct {
ID string
Price int
}Then we define how our ProductPlan is supposed to look. Our update plan has:
- Added - products that are only in the new dataset and thus will be added.
- NoChanges - products that are both in the old and new datasets, but the price did not change.
- Updated - products that are both in the old and new datasets, but the price changed.
- Removed - products that are only in the old dataset and thus will be removed.
type ProductPair struct {
OldProduct Product
NewProduct Product
}
type ProductUpdatePlan struct {
Added []Product
NoChanges []Product
Updated []ProductPair
Deleted []Product
}The CreatePlan function then can look as follows. The cool thing here is that it’s a pure function; it has no side effects and doesn’t even return an error. It’s very easy to test this.
func CreatePlan(
oldProducts []Product, newProducts []Product,
) ProductUpdatePlan {
plan := ProductUpdatePlan{}
oldProductMap := make(map[string]Product)
for _, product := range oldProducts {
oldProductMap[product.ID] = product
}
newProductMap := make(map[string]Product)
for _, product := range newProducts {
oldProduct, ok := oldProductMap[product.ID]
if !ok {
plan.Added = append(plan.Added, product)
} else if oldProduct.Price != product.Price {
plan.Updated = append(plan.Updated, ProductPair{
OldProduct: oldProduct,
NewProduct: product,
})
} else {
plan.NoChanges = append(plan.NoChanges, product)
}
newProductMap[product.ID] = product
}
for _, product := range oldProducts {
_, ok := newProductMap[product.ID]
if !ok {
plan.Deleted = append(plan.Deleted, product)
}
}
return plan
}To actually apply these changes, we define an interface that will represent the database. We keep it simple and do not use any batch updates/inserts here.
type Repo interface {
InsertProduct(product Product) error
UpdateProduct(product Product) error
DeleteProduct(product Product) error
GetProducts() ([]Product, error)
Transaction(func(repo Repo) error) error
}
func (pup ProductUpdatePlan) Apply(repo Repo) error {
return repo.Transaction(func(repo Repo) error {
for _, product := range pup.Added {
err := repo.InsertProduct(product)
if err != nil {
return err
}
}
for _, productPair := range pup.Updated {
err := repo.UpdateProduct(productPair.NewProduct)
if err != nil {
return err
}
}
for _, product := range pup.Deleted {
err := repo.DeleteProduct(product)
if err != nil {
return err
}
}
return nil
})
}The overall flow could look like this - this could be the function an endpoint would call; the preview flag controls if it’s just a preview or the actual import.
func UpdateProducts(
repo repo, csv string, preview bool
) (ProductUpdatePlan, error) {
// Implementing parseCSV is left as an exercise for the reader.
products, err := parseCSV(csv)
if err != nil {
return ProductUpdatePlan{}, err
}
oldProducts, err := repo.GetProducts()
if err != nil {
return ProductUpdatePlan{}, err
}
plan := CreatePlan(oldProducts, products)
if preview {
return plan, nil
}
return plan, plan.Apply(repo)
}With this, we can offer a user experience that feels safe and predictable. The user can clearly see how their changes will interact with the system.
Interactive example
To have an excuse for building this blog with React.js, here is an interactive toy example of how this could look. You can change the provided CSV data, click "Preview changes" and see the update plan. You can then apply the update plan to the database by clicking "Apply changes". The products and CSV data are seeded with some sensible data. There is no error handling, you have to rely on the update plan for the safe guarding ;).