Estuary
Estuary Flow is a platform for creating real-time data pipelines at scale. It combines the intuitive interface of an ELT service with an event-driven runtime and a variety of open-source connectors.
You can use Flow to migrate your data from Firestore to the Postgres database in your Supabase project. You do this by building a real-time pipeline that captures data from Firestore and materializes (loads) that data to Postgres. Once created, the pipeline backfills all your historical data from Firestore and continues to process new data events in real time.
Prerequisites#
Before you begin, you'll need:
An Estuary account.
For your Firestore database:
- A Google service account with read access to your Firestore database, via roles/datastore.viewer. You can assign this role when you create the service account, or add it to an existing service account.
- A generated JSON service account key for the service account.
A Supabase project
Step 1: Capture your Firestore data#
You'll start by creating a capture, a task in Flow that connects to your data source system: in this case, Firestore. This process will create one or more data collections, backed by a real-time data lake.
Go to the Flow web application at dashboard.estuary.dev and sign in using the credentials provided by your Estuary account manager.
Click the Captures tab and choose New Capture.
Locate and select the Google Firestore card.
A form appears with the properties required for a Firestore capture.
Set a name for your capture.
Click inside the Name field to generate a drop-down menu of available prefixes and select one (likely, this will be the name of your organization). Append a unique capture name after the
/to create the full name, for example,acmeCo/myFirestoreCapture.Fill out the required properties for Firestore.
- Database: Flow can autodetect the database name, but you may optionally specify it here. This is helpful if the service account used has access to multiple Firebase projects. Your database name usually follows the format
projects/$PROJECTID/databases/(default). - Credentials: The JSON service account key created per the prerequisites.
- Database: Flow can autodetect the database name, but you may optionally specify it here. This is helpful if the service account used has access to multiple Firebase projects. Your database name usually follows the format
Click Discover Endpoint.
Flow uses the provided configuration to initiate a connection with Firestore. It generates a capture specification and details of the collections that it will create, once published.
Use the Specification Editor to view the JSON schemas for each collection and make sure they are formatted correctly for your needs. If they're not, you can edit them.
Click Save and publish.
You'll see a notification when the capture publishes successfully.
The data currently in your Firestore database has been captured, and future updates to it will be captured continuously.
Click Materialize Collections to continue.
Step 2: Materialize your collections to Postgres#
Next, you'll add a Postgres materialization to connect the captured collections to tables in your Supabase Postgres database.
On the Create Materialization page, search for and select the PostgreSQL tile.
A form appears with the properties required for a Postgres materialization.
Choose a unique name for your materialization like you did when naming your capture; for example,
acmeCo/mySupabaseMaterialization.Fill out the required properties for PostgreSQL. You can find most of these in Supabase by going to the Settings section and clicking Database.
- Address: Format at
<host>:<port>. - User: Usually, this is
postgres. - Password: The password you set when you created your Supabase project.
- Address: Format at
Scroll down to view the Collection Selector and fill in the Table field for each collection.
The collections you just created have already been selected, but you must provide names for the tables to which they'll be materialized.
Click Discover Endpoint.
Flow uses the provided configuration to initiate a connection to your Supabase Postgres database and generate a specification.
Click Save and Publish. You'll see a notification when the full materialization publishes successfully.
Your Firestore collections are copied to tables in Supabase. As long as you leave the capture and materialation running, any changes to the Firestore data will be reflected in Supabase in milliseconds.
Resources#
For more information, visit the Flow docs. In particular: