Designing a Production Data Platform: From Requirements to Architecture
Try This First
Test your knowledge before reading. Don't worry if you get it wrong — that's part of learning.
Key Insights
- Master the full lifecycle of data platform design: requirements gathering, architecture methodology, a fintech case study, data mesh decentralization, open source vs managed decision framework, budget-friendly stacks, and evolving a platform into a data product.
From Requirements to Architecture: A Design Methodology
Designing a production data platform requires a systematic approach that starts with business requirements and ends with a concrete technology architecture. The methodology has five phases:
- Phase 1 — Requirements Gathering: Identify data sources, consumer personas, latency SLAs, volume estimates, compliance constraints (GDPR, SOX, SOC2), and team skill sets. Interview stakeholders across engineering, analytics, ML, and business teams.
- Phase 2 — Logical Architecture: Define the data flow: ingestion → storage → transformation → serving. Choose architectural patterns (medallion, data mesh, lambda/kappa) based on requirements. Map data domains to ownership boundaries.
- Phase 3 — Technology Selection: Evaluate tools against a weighted decision matrix. Factors: team expertise, operational maturity, scale requirements, budget, vendor lock-in risk, and community health.
- Phase 4 — Implementation Plan: Define a phased rollout. Start with a minimal viable platform serving one use case, then add capabilities incrementally. Plan for data migration, schema evolution, and cutover strategies.
- Phase 5 — Operations and Governance: Establish monitoring, alerting, cost governance, data cataloging, and access control. Define SLOs for freshness, quality, and availability. Run a post-mortem after every incident.
Case Study: Building a Data Platform for a Fintech Startup
Let's walk through a realistic scenario. A fintech startup processes 5M transactions/day, needs real-time fraud detection, daily regulatory reporting, and ML-driven credit scoring. Here's how the design unfolds:
| Requirement | Choice | Rationale |
|---|---|---|
| Real-time fraud detection | Kafka + Flink | Sub-second latency, exactly-once semantics, stateful processing |
| Daily regulatory reporting | dbt + Snowflake | SQL-first, tested, auditable transformations |
| ML credit scoring | Feast + Databricks | Feature store for training/serving consistency |
| Orchestration | Dagster | Software-defined assets, dbt integration, partitioned backfills |
| Storage | S3 + Iceberg | Open table format, ACID transactions, schema evolution |
| Data quality | Great Expectations + dbt tests | Expectations at ingestion, tests at transformation |
| Observability | Monte Carlo + OpenLineage | Automated lineage, freshness monitoring, incident management |
Key design decisions: Use a medallion architecture (Bronze for raw, Silver for cleaned, Gold for aggregated) with Iceberg as the table format. Run batch pipelines for regulatory reporting on Snowflake (SQL-friendly for auditors), and streaming pipelines for fraud detection on Flink. Dagster orchestrates both worlds.
Data Mesh: Decentralized Ownership at Scale
As the organization grows beyond a single data team, the centralized platform model breaks down. Data mesh, proposed by Zhamak Dehghani, offers an alternative: domain teams own their data as products, while a shared infrastructure layer provides cross-domain interoperability.
- Domain Ownership: Each business domain (payments, risk, marketing, compliance) owns its data — from ingestion to serving. The domain team decides schema, quality standards, and access policies.
- Data as a Product: Each domain exposes its data as a well-documented, versioned, and discoverable product with SLOs. Consumers subscribe to data products rather than requesting one-off exports.
- Self-Serve Infrastructure: A shared platform team provides the infrastructure (storage, compute, catalog, lineage) that domains use to build and serve their data products. This includes data discovery (DataHub/Amundsen), lineage (OpenLineage), and access control.
- Federated Governance: Global standards (naming conventions, security policies, catalog metadata) are defined centrally but enforced locally. Each domain implements standards within its own data product boundaries.
- dbt Mesh: Enables decentralized dbt projects where each domain owns its models and contracts, while sharing cross-project references via a public model API. This is a practical implementation of the data mesh principle for the transformation layer.
Open Source vs Managed: A Decision Framework
| Factor | Choose Open Source | Choose Managed |
|---|---|---|
| Team size | 5+ platform engineers | < 5 engineers |
| Operating budget | Engineering time > vendor cost | Vendor cost > engineering time |
| Customization needs | High (custom connectors, transforms) | Standard patterns only |
| Compliance requirements | Data residency, air-gapped | SOC2/HIPAA certified vendor |
| Scalability horizon | Predictable, gradual growth | Rapid, unpredictable scaling |
Most organizations use a hybrid approach: managed services for commodity capabilities (data warehouse, message broker) and open source for differentiation (transformation logic, custom connectors, ML pipelines).
Building a Data Platform on a Budget
Not every organization can afford Snowflake + Fivetran + Monte Carlo. Here's a cost-effective open-source stack that scales from startup to mid-market:
- Ingestion: Airbyte (open source) + Kafka (self-managed or Confluent Cloud free tier). Airbyte has 300+ connectors for sources and destinations.
- Storage: S3/MinIO + Apache Iceberg. MinIO provides S3-compatible storage on-premises or in any cloud. Iceberg provides table management, ACID, and time travel.
- Transformation: dbt Core (free) + DuckDB for local development. dbt Cloud is optional for CI/CD and scheduling — use Dagster instead.
- Orchestration: Dagster (open source). The Dagster+ managed service is optional; the OSS version is production-ready.
- Data Quality: Great Expectations (open source) + dbt tests (built into dbt Core). Both are free and widely adopted.
- Observability: OpenLineage (free) + Elementary (free dbt-native observability) + custom monitoring with Prometheus/Grafana.
- BI: Metabase or Apache Superset (both open source). Good enough for internal dashboards and ad-hoc querying.
This stack costs <$500/month for infrastructure at startup scale and can handle terabytes of data. As you grow, you can replace components with managed alternatives without rewriting your pipeline logic — because the core transformation code (dbt models, Python assets) is portable.
From Data Platform to Data Product
The final evolution is treating the data platform itself as a product. This means:
- Developer Experience: Provide self-service tools for data ingestion (web UI or API), transformation templates, and CI/CD pipelines. Data engineers should not be a bottleneck for adding new data sources.
- Documentation and Discovery: Every dataset has a description, owner, freshness SLO, quality score, and lineage. Use a data catalog (DataHub, Amundsen, or Atlan) to make data discoverable.
- Cost Transparency: Show each team their storage and compute costs. Chargeback or showback models incentivize efficient usage. Monitor cost per dataset and alert on anomalies.
- Feedback Loops: Collect feedback from data consumers through surveys, usage analytics, and regular meetings. Treat the platform backlog like any product backlog — prioritize by impact.
The ultimate measure of a data platform is not its technology stack but its time-to-insight: how long does it take a data consumer to go from a question to a trusted answer? A well-designed platform reduces this from weeks to minutes.
Article Metadata
Review with Spaced Repetition
Add this lesson's 7 flashcards to your SM-2 study queue. They will appear when due in the Study Queue.
Feynman Concept Cards
Master each building block: read the ELI5, explore the analogy, work the example, find your gaps, teach it back, build it.
Data Mesh is a architecture concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Data Mesh as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Data Mesh in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Data Mesh should include: 1. The core components of data mesh 2. How they interact 3. Expected outcomes or outputs
DataOps is a best practices concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Lakehouse Architecture is a architecture concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Lakehouse Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Lakehouse Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Lakehouse Architecture should include: 1. The core components of lakehouse architecture 2. How they interact 3. Expected outcomes or outputs
Design is a specialized concept in the Markets pillar.
Analogy
Example
Find Gaps
Explain Design as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Design in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Design should include: 1. The core components of design 2. How they interact 3. Expected outcomes or outputs
Feynman Synthesis — Prove You Understand
1. The One-Pager
Explain this lesson's core idea to a smart 15-year-old. No jargon allowed.
2. The Gap Map
List 3 things you are still unsure about. Be specific.
Knowledge Check
Test your understanding of this lesson.
Flashcards
Space = flip · 1-4 = grade · Swipe on mobile