DataOps Glossary
Key Insights
- Key DataOps and data engineering terminology.
Key terms used across DataOps, data engineering, and data science — as referenced in AcaciaFund's research and learning materials.
- DataOps
- Automated, process-oriented methodology applying Agile, DevOps, and statistical process control to data pipeline lifecycle management.
- ELT vs ETL
- Extract-Load-Transform: data is extracted and loaded raw into the warehouse, then transformed in-place. Contrasts with classic ETL where transformation happens before loading.
- Medallion Architecture
- Bronze (raw) → Silver (cleaned) → Gold (aggregated) data layering pattern popularized by Databricks for lakehouse implementations.
- Pipeline Observability
- Real-time monitoring of data pipelines — row counts, schema drift, latency, error rates — to detect and diagnose failures quickly.
- Data Contract
- Formal agreement between data producers and consumers specifying schema, semantics, quality SLOs, and ownership.
- dbt
- Data build tool — SQL-first transformation framework. Models are SELECT statements; dbt handles materialization, testing, and docs.
- Great Expectations
- Open-source Python framework for defining, testing, and documenting data quality expectations.
- SQI (Signal Quality Index)
- AcaciaFund's composite metric: source authority × freshness × consensus × relevance, normalized to [0,1].
- CDC (Change Data Capture)
- Technique for capturing row-level changes in databases, enabling real-time data replication without bulk loads.
- DAG (Directed Acyclic Graph)
- Acyclic graph of tasks with dependencies — the fundamental scheduling unit in Airflow, Dagster, and Prefect.
- MCP (Model Context Protocol)
- Emerging protocol for AI agents to interact with data tools and APIs in a standardized way, enabling agent-operated data pipelines.
- Feature Store
- Centralized repository for ML features enabling consistent computation, sharing, and serving across training and inference.
- Data Lineage
- End-to-end tracking of data from source through transformations to final consumption, enabling debugging, auditing, and impact analysis.
- Schema-on-Read
- Data lake paradigm where schema is applied at query time rather than ingest time, enabling flexible data storage.
Article Metadata
Further Reading
Databricks Blog
Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog
Apache Kafka
Kafka documentation, KIPs, and ecosystem updates
Apache Flink
Flink documentation and release notes
Apache Iceberg
Iceberg table format — specs, REST catalog, performance
dbt Blog
dbt Labs engineering blog — analytics engineering, Semantic Layer
Dagster Blog
Dagster orchestration — software-defined assets, IO managers
Feynman Concept Cards
Master each concept: read the ELI5, explore analogies, work examples, and teach it back.
Extract-Load-Transform is a foundations concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Extract-Load-Transform as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Load-Transform in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Load-Transform should include: 1. The core components of elt 2. How they interact 3. Expected outcomes or outputs
DataOps is a best practices concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Extract-Transform-Load is a foundations concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Extract-Transform-Load as if teaching a colleague who is new to foundations. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Extract-Transform-Load in a real-world foundations scenario. Walk through your design decisions.
Show solution
A diagram for Extract-Transform-Load should include: 1. The core components of etl 2. How they interact 3. Expected outcomes or outputs
ELT Pipeline Architecture is a architecture concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain ELT Pipeline Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates ELT Pipeline Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for ELT Pipeline Architecture should include: 1. The core components of elt pipeline 2. How they interact 3. Expected outcomes or outputs
Research is a specialized concept in the Markets pillar.
Analogy
Example
Find Gaps
Explain Research as if teaching a colleague who is new to specialized. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Research in a real-world specialized scenario. Walk through your design decisions.
Show solution
A diagram for Research should include: 1. The core components of research 2. How they interact 3. Expected outcomes or outputs