Encrypt Personal Data

As previously noted the schema created by the phx.gen.auth generator leaves email addresses stored as plaintext:

mix-phx-gen-auth-people-table-email-plaintext

It's disappointing to us that this is the default and many devs will naively think this is "OK".

It's Never "OK" to Store Personal Data as `plaintext`

It might be tempting to store personal data as human-readable text during development to make it easier to debug, but we urge everyone to resist this temptation! Data breaches happen every day. Breaches can destroy the reputation of a company. Thankfully, the EU now has stronger laws in the form of the General Data Protection Regulation (GDPR), which carry heavy fines for companies that improperly store personal data.

Most software engineers don't consider the law(s) around data protection but none of us have any excuse to ignore them even in basic projects. The good news is: protecting personal data is quite straightforward.

Why Encrypt Now?

We know this might feel like a deeply technical step to have so early on in the creation of the auth App. Why don't we just skip this and focus on the interface people will see? We feel that getting the data privacy/security right from the start is essential for building a robust and therefore trustworthy app. We will get to the interface design in the next chapter so if you prefer that part, feel free to speed-read/run this and only return to it when you need a deeper understanding.

How To Encrypt Sensitive Data?

When we first started using Elixir in 2016 we explored the topic of encrypting personal data in depth and wrote a comprehensive guide: dwyl/phoenix-ecto-encryption-example We highly recommend following step-by-step guide to understand this in detail. We aren't going to duplicate/repeat any of the theory here. Rather we are going to focus on the practice using the fields package we created to implement transparent encryption.

Using `Fields` to Automatically Encrypt Personal Data

Following the instructions in the fields repo, open the mix.exs file and locate the defp deps do section and add fields to the list:

{:fields, "~> 2.10.3"},

Save the mix.exs file and run:

mix deps.get

Create Environment Variables

fields expects an `environment variable

e.g. using the phx.gen.secret command:

mix phx.gen.secret

You should see output similar to:

XGYVueuky4DT0s0ks2MPzHpucyl+9e/uY4UusEfgyR2qeNApzoYGSH+Y55cfDj1Y

Export this key in your terminal with the following command:

export ENCRYPTION_KEYS=XGYVueuky4DT0s0ks2MPzHpucyl+9e/uY4UusEfgyR2qeNApzoYGSH+Y55cfDj1Y

Run the phx.gen.secret command again and export it as SECRET_KEY_BASE, e.g:

export SECRET_KEY_BASE=GLH2S6EU0eZt+GSEmb5wEtonWO847hsQ9fck0APr4VgXEdp9EKfni2WO61z0DMOF

We use an .env file on localhost:

export ENCRYPTION_KEYS=XGYVueuky4DT0s0ks2MPzHpucyl+9e/uY4UusEfgyR2qeNApzoYGSH+Y55cfDj1Y
export SECRET_KEY_BASE=GLH2S6EU0eZt+GSEmb5wEtonWO847hsQ9fck0APr4VgXEdp9EKfni2WO61z0DMOF

See: .env_sample for a sample including all the recommended/required environment variables for running the auth app.

Now to the interesting part!

Update `people.email` Data Type

The phx.gen.auth generator created the people schema with the :email field defined as a :string:

field :email, :string

See: lib/auth/accounts/person.ex#L6

Create Migration File

Using Ecto.Migration run the following command to create a migration file with a descriptive name:

mix ecto.gen.migration modify_people_email_string_binary

You should see output similar to:

* creating priv/repo/migrations/20230309145958_modify_people_email_string_binary.exs

Open the file in your editor. You should see:

defmodule Auth.Repo.Migrations.ModifyPeopleEmailStringBinary do
  use Ecto.Migration

  def change do

  end
end

Update it to include the alter statement:

defmodule Auth.Repo.Migrations.ModifyPeopleEmailStringBinary do
  use Ecto.Migration

  def change do
    alter table(:people) do
      remove :email
      add :email, :binary
      add ...
    end

    alter table(:people_tokens) do
      remove :sent_to
      add :sent_to, :binary
    end
  end
end

Note: we tried using Ecto.Migration.modify/3 to modify the field type but got the error:

15:06:13.691 [info] alter table people
** (Postgrex.Error) ERROR 42804 (datatype_mismatch) column "email" cannot be cast automatically to type bytea

    hint: You might need to specify "USING email::bytea".

Given that this is a new project and there is no data in the DB, we decided it was easier to remove and re-add the email column/field.

Save the migration file and run:

mix ecto.reset

You should see output similar to the following:

15:13:29.989 [info] == Migrated 20230226095715 in 0.0s

15:13:30.057 [info] == Running 20230309145958 Auth.Repo.Migrations.ModifyPeopleEmailStringBinary.change/0 forward

15:13:30.058 [info] alter table people

15:13:30.059 [info] alter table people_tokens

15:13:30.059 [info] == Migrated 20230309145958 in 0.0s

### Update person.email

Open lib/auth/accounts/person.ex and replace the line:

field :email, :string

With:

field :email, Fields.EmailEncrypted
field :email_hash, Fields.EmailHash

Sadly, this essential privacy/security enhancement was not quite as straightforward as we had hoped and had quite a few ramifications. The boilerplate code generated by phx.gen.auth was relying on person.email being plaintext so we ended up having to make several changes.

Rather than repeating all of these here which will take up a lot of space and duplicate the code unessessarily, we recommend you read through the git commit: #2bbba99

In a follow-up section we will be removing most of this code because we don't want to use links in emails to verify people. Instead we will be using one-time-numeric codes.

Following the changes made in: #2bbba99

If we now run a query to view the data in the people table:

SELECT id, email, inserted_at FROM people;

We see that the email is now a binary blob, i.e: encrypted:

This cannot be read by anyone who does not have the encryption key. So if the database is somehow compromised, it's useless to the attacker.

Registration and Login still works as expected:

But now the personal data captured in registration is stored encrypted at rest the way it should be. 🔐

Now we can get on with building auth!

dwyl book