Introduction Snowflake Openflow is a platform for integrating and processing data that allows you to design, test, and deploy data pipelines. Running it locally facilitates development and testing before moving to production. This article explains how to set up the openflow-localforge environment using Docker Compose and associated tools, with practical tips and best practices.
About Q2BSTUDIO Q2BSTUDIO is a software development company specialized in custom applications and custom software. We offer artificial intelligence solutions, AI for businesses, AI agents, cybersecurity, AWS and Azure cloud services, and business intelligence services such as Power BI to improve decision-making. If you need integration between Snowflake, local pipelines, and custom solutions, our team can help you implement, secure, and optimize processes.
Repository The complete openflow-localforge project is available at https://github.com/Snowflake-Labs/openflow-local-forge. Clone the repository and navigate to the project folder.
Prepare the environment Create the necessary directories and export the OPENFLOW_LOCALFORGE_HOME variable pointing to the cloned folder. Copy .env.example to .env and edit it with your local values. Keep .env out of version control to protect secrets.
Prerequisites Install Docker and Docker Compose with at least 8 GB of RAM allocated to the engine. Install Snowflake CLI to manage connections and Snowpark Container Services (SPCS). Install LocalStack to simulate AWS services locally, jq to process JSON, and Git for version control of flows.
System requirements Memory: 8 GB minimum (16 GB recommended), 5 GB free disk space, Linux, macOS, or Windows with WSL2, and a Snowflake account with SPCS privileges. Configure a connection profile in ~/.snowflake/config.toml with the account, user, database, schema, warehouse, and role parameters, or use private key authentication for production environments.
SPCS authentication and image download Use Snowflake CLI to authenticate to the SPCS registry: run snow spcs image-registry login and get the URL with snow spcs image-registry url. List images in the snowflake_images collection and retrieve the most recent paths for runtime-extensions and runtime-server using jq. Add the obtained paths to the .env file as OPENFLOW_RUNTIME_EXTENSION_IMAGE and OPENFLOW_RUNTIME_SERVER_IMAGE, and verify that you can docker pull those images.
Additional configuration Generate a secure properties key to encrypt sensitive values, for example with xxd over /dev/urandom, and store properties.key outside version control. Copy login-identity-providers.xml.example to login-identity-providers.xml and use the gen_password.py script to generate the secure password to insert into the local authentication configuration.
Docker Compose services description LocalStack simulates S3, SQS, Lambda, and other AWS services. The runtime-extensions service copies libraries and extensions to the shared volume. The runtime-server service runs the main Openflow runtime, and NiFi exposes the HTTPS interface on port 8443 and mounts volumes for configuration, flows, and state.
Start the environment From the project root, run docker compose up -d and validate the status with docker compose ps. Follow the runtime-server logs with docker compose logs -f runtime-server until you see that the application has started correctly. If a message indicating Application Started appears, it means the NiFi server is ready.
Configure LocalStack and test data Give execution permissions to the setup-localstack.sh script and run it to create a local S3 bucket and upload sample data such as WillShakespeare.zip. Verify LocalStack health with curl https://localhost:4566/health and list the S3 contents with aws --endpoint-url=https://localhost:4566 s3 ls s3://sample-data/
Web interface access Access the NiFi interface at https://localhost:8443/nifi, accept the self-signed certificate, and log in with the credentials defined in login-identity-providers.xml. You will find a canvas with example flows, for example Simple Flow adapted to use S3 with LocalStack.
Flow version control Flows are stored in the flows folder mounted as a Git repository. Initialize the repository with git init, add the files, and make commits. Optionally connect a remote and push the main branch to maintain versions and collaborations.
Troubleshooting common issues For memory errors, increase the memory allocated to Docker and monitor with docker stats. If there are port conflicts, use lsof -i :8443 or lsof -i :4566 to identify processes and free the ports or change the ports in docker-compose.yml. For extension issues, make sure you are authenticated to the SPCS registry and docker pull the images again. For SPCS authentication issues, validate the connection with snow connection test --connection and run snow spcs image-registry login again.
Best practices Never include sensitive data in the repository. Use the .env.example template and keep .env in .gitignore. Rotate encryption keys in production environments, limit network access for real deployments, and use environment-specific configurations by copying .env.example to dev, staging, and prod.
Optimization and performance Reserve adequate memory for Docker (8 GB minimum), use SSD disks to improve I/O, monitor resources with docker stats, and periodically clean unused volumes to keep the environment agile.
Cleanup When you finish, stop services with docker compose down, and if you want to remove volumes, add the -v option. To free up space, run docker system prune with caution.
Next steps Explore the built-in processors for data transformation, connect external systems via JDBC and REST, implement custom processors with the extension framework, and configure monitoring and alerts. For production deployments, consider BYOC or using SPCS in Snowflake.
Resources openflow-localforge repository at https://github.com/Snowflake-Labs/openflow-local-forge, Snowflake Openflow documentation at https://docs.snowflake.com, and Snowpark Container Services in the official documentation, as well as LocalStack and Docker Compose guides.
Conclusion Running Snowflake Openflow locally with LocalStack and Docker Compose offers a consistent and secure environment for developing data pipelines without cloud costs. If you need support adapting Openflow to your needs, integrating artificial intelligence solutions, AI agents, business intelligence services, or deploying custom applications and custom software in secure AWS or Azure cloud environments, contact Q2BSTUDIO for personalized consulting in artificial intelligence, cybersecurity, and cloud solutions.
Contact Q2BSTUDIO helps transform ideas into real solutions through custom application development, custom software, artificial intelligence, cybersecurity, AWS and Azure cloud services, business intelligence services, AI agents, and Power BI to improve the visibility and value of data in your company.
