When a company needs to migrate data between Firestore databases —whether due to a project change, creating a development environment, or a backup strategy— the process seems simple: copy and paste. However, the reality is far more complex. Firestore does not store information in a flat structure; documents can contain nested subcollections, which in turn can have further subcollections. Ignoring this hierarchy leads to incomplete transfers that compromise data integrity. In this comprehensive guide, you will learn how to transfer Firestore data between databases while preserving the entire structure, including subcollections of any depth, from a technical and business perspective.
At Q2BSTUDIO, as a software and technology development company, we understand that data migration is not just a technical move but a strategic decision affecting business continuity. Whether you need custom software applications with Firestore as a backend or want to integrate artificial intelligence and cybersecurity solutions into your cloud infrastructure, mastering these transfers is essential.
The subcollection challenge
When you fetch a Firestore document, the SDK does not automatically bring its subcollections. For example, a 'Categories' collection contains documents with fields like 'name', but it may also have a 'Subcategories' subcollection, and each subcategory may have another 'Items' subcollection. If you only transfer top-level documents, you lose all nested content. This is the most common and costly mistake.
Technical prerequisites
Before starting, ensure you have Node.js (v12 or higher), the firebase-admin package installed, and two Firebase/Google Cloud projects: source and destination. You will also need to generate service account keys from the Firebase console (Project Settings > Service Accounts) and assign appropriate IAM roles: for source, Cloud Datastore Viewer (read-only); for destination, Cloud Datastore User (write). Following the principle of least privilege protects your infrastructure and facilitates auditing.
Basic script: flat transfer
An initial script that iterates over a collection and writes documents to the destination database works for flat data but ignores subcollections. The typical code uses admin.firestore(), gets the collection snapshot, iterates through documents, and uses batch.set() to write in batches of 500 operations (Firestore limit). However, this does not recurse into subcollections. For a complete transfer, we need a recursive approach.
Recursive transfer with subcollections
The key is to use listCollections() on each source document to discover all subcollections. Then, for each subcollection, call the same collection transfer function again, and so on until reaching the deepest levels. This method ensures the entire hierarchy is preserved. For each document, besides writing its data, you iterate over the subcollections found and transfer them following the same batch logic.
Production-ready complete script
A robust script must handle: reading all subcollections recursively, batch writes respecting the 500-operation limit, progress logging, error handling, and configurable collection paths. Additionally, it should run in two passes: first transfer the current collection's documents, then for each document list and transfer its subcollections. This avoids concurrency issues and ensures all base documents exist before inserting their descendants.
Security best practices
Never include service account keys in the repository. Use environment variables (SOURCE_CREDENTIALS, DEST_CREDENTIALS) and add *.key.json files to .gitignore. Rotate keys every 90 days and revoke old ones immediately. Enable auditing via Cloud Audit Logs to monitor access. At Q2BSTUDIO we apply these measures in all our cybersecurity and cloud projects.
Troubleshooting common issues
If 'Permission denied' appears, verify that the IAM roles are correct. If the batch exceeds 500 operations, the script must split into exact batches (counter modulo 500). For very large datasets, consider using Firestore's native export/import via Cloud Storage: gcloud firestore export and import. Another option is the 'Firestore Import/Export' extension from the Firebase Extensions marketplace, which offers a GUI and scheduled backups, though with less control.
Alternatives for large volumes
Export using gcloud is atomic, fast, and handles composite indexes, but requires a Cloud Storage bucket and incurs storage and traffic costs. It is ideal for full project migrations. The Firebase extension is more user-friendly for teams without command-line experience. For environments already using cloud AWS/Azure, integration with each provider's data transfer services can be an option.
The business value of a correct migration
An incomplete data transfer can cause application failures, loss of critical information, and missed deadlines. For a company developing custom software applications, data integrity is a fundamental non-functional requirement. Moreover, when incorporating artificial intelligence capabilities or AI agents, the quality of training data directly depends on reliable migrations. That is why at Q2BSTUDIO we integrate these practices into our AI and Business Intelligence (Power BI) services, ensuring data flows correctly between environments.
Automation and AI agents
With the adoption of AI agents capable of orchestrating migration processes, companies can reduce human risk and accelerate transfers. For example, an agent can schedule script execution, monitor logs, and notify the team if an error occurs. At Q2BSTUDIO we develop automation solutions that include these agents, as part of our process automation offering.
Conclusion
Transferring Firestore data between databases is not a trivial task. It requires understanding the hierarchical structure, implementing recursion, managing write limits, and applying security measures. With the right approach, your company can perform reliable migrations, preserving every document and subcollection. At Q2BSTUDIO we offer consulting and development to help you through this process, from planning to execution, integrating cloud, AI, and cybersecurity according to your needs.
