Deep research · Fresh sources

Apache Hudi

Completed June 21, 2026

Apache Hudi — Polished Research Synthesis

Source preference: Fresh (date‑stamped, post‑2024). Older evidence overridden where newer, credible sources conflict. Every section reflects the adversarial critique: unsupported claims are flagged, vendor bias is noted, and gaps are made explicit.

---

Executive Summary

Apache Hudi remains the most feature‑rich open lakehouse for streaming CDC and record‑level mutation. Its native merge‑on‑read architecture, multi‑modal indexing, and built‑in incremental processing deliver best‑in‑class upsert/delete latency for petabyte‑scale ingestion. However, Hudi’s differentiation comes at a steep price in operational complexity, community‑momentum risk, and provable performance transparency.

  • Differentiated strengths: record‑level & secondary indexes, Non‑Blocking Concurrency Control (Hudi 1.0), built‑in table‑services framework, streaming‑first design.
  • Material weaknesses: no partition evolution, narrower multi‑engine write support, higher maintenance overhead, and significant gaps in independently verified benchmarks.
  • Mindshare reality: Iceberg has become the de‑facto inter‑engine format; Hudi’s response (pluggable table‑format framework, Hudi 1.1) is promising but unproven at scale.
  • Deployment risk: several documented data‑loss bugs, metadata‑table fragility, and the “default‑config trap” (bulk‑insert vs. upsert) make production adoption dependent on skilled engineering.

Hudi is the right choice only when the workload is dominated by high‑frequency, keyed upserts/deletes from streaming sources and the organisation can invest in dedicated platform engineering. For warehouse‑style analytics, mixed‑engine querying, or “first lakehouse” projects with limited operational bandwidth, Iceberg (or, within Databricks, Delta Lake) is the safer default.

---

1. Core Capabilities & Differentiators

CapabilityHudi’s PositionStatus / Caveat
Record‑level upserts/deletesNative since inception; Merge‑on‑Read (MoR) tables absorb writes with minimal amplification.MoR adds ~10‑30% storage overhead and higher snapshot‑query latency until compacted.
Partial updates (Hudi 1.0)Modify only changed columns, avoiding full‑row overwrite.Project‑internal benchmark claims 70× fewer bytes written (Dec 2024); no independent reproduction.
Indexing subsystemRecord‑level, secondary (any column), expression/functional, bloom filters — all in a self‑managed metadata table.17× lookup speed‑up claimed (Onehouse, 2024); storage overhead unquantified. The label “not available in other projects” (Onehouse, Oct 2025) is an overstatement — Delta’s deletion vectors and Iceberg’s manifest stats offer partial equivalents.
Non‑Blocking Concurrency Control (NBCC)Allows multiple writers to the same file group without optimistic concurrency aborts (Hudi 1.0, GA Jan 2025).Still marked experimental for OCC multi‑writer in official docs. Known bugs (#9674, #9728) show data‑loss risk and lock‑retry failures. Without explicit primary‑key conflict check, silent duplicate records can occur (Jack Vanlightly TLA+ analysis, Apr 2024).
LSM‑timeline (1.0)Replaces flat timeline with tiered commit‑log compaction; enables efficient access after millions of commits.Benchmark: 10M instants loaded in ~3.4s (project blog, May 2025). “Infinite time travel” requires disabling the cleaner entirely — an operational choice, not a default.
Streaming & CDC ingestionHoodieStreamer (ex‑DeltaStreamer) provides no‑code Kafka/Debezium → lake ingestion with built‑in checkpointing and table‑service orchestration.Correctness is tied to upstream offset management; “exactly‑once” depends on the sink configuration and has not been independently validated on public cloud storage.

Caveats from adversarial review:

  • The “multi‑modal indexing not available in other projects” claim conflates built‑in, fully‑managed indexes with any form of indexing; Iceberg and Delta have alternatives, though less integrated.
  • Partial‑update benchmarks (70× write reduction) are synthetic; the community should welcome independent reproduction before treating them as guaranteed.

---

2. Architecture & Design

Hudi’s architecture is a timeline‑based, log‑structured lakehouse that exposes explicit trade‑offs:

  • Two table types:
  • Copy‑on‑Write (CoW) — every update rewrites Parquet files; read‑optimised, high write amplification.
  • Merge‑on‑Read (MoR) — updates appended to Avro log files, then asynchronously compacted; low write latency, higher snapshot‑query penalty.
  • Timeline state machine (.hoodie/ directory) provides atomic commits and snapshot isolation.
  • Metadata Table (MDT): an internal MoR table that caches file listings, column stats, and indexes. Enabled by default, but has been a source of production incidents — corruption can block all ETL (Medium user report, 2025; GitHub issues #9848, #11535).
  • File‑group model: record keys map to specific file groups, enabling efficient key‑based operations without full scans.
  • Pluggable Table Format Framework (Hudi 1.1, Nov 2025): decouples the storage engine from the on‑disk format; via Apache XTable (incubating) it can write Iceberg or Delta metadata. Critical uncertainty: MoR tables and record‑level indexes are not fully supported through XTable translation, so the inter‑op story is partially aspirational.
  • Hudi‑rs (Rust, v0.4.0): Python‑friendly, JVM‑free read path, but write capabilities remain limited and MoR support is partial.

Key architectural risk: The metadata table is itself a Hudi MoR table — its cleaner/compaction failures can cascade into pipeline‑blocking outages. This recursive maintenance burden is poorly documented.

---

3. Performance & Benchmarks — A Contested Landscape

All performance claims must be read with extreme scepticism. Every major published benchmark has a vendor affiliation, and no independent TPC‑TCDS‑audited result exists.

Benchmark SourceKey FindingIndependence & Freshness
CIDR 2023 (lhbench, UC Berkeley)Hudi 0.12.0 appeared ~10× slower than Delta on bulk loads. Delta 1.4× faster on queries.Academic but stale — used Hudi 0.12.0 (mid‑2022). Did not switch Hudi to bulk‑insert mode for batch loads.
Onehouse TPC‑DS (Oct 2025)Hudi within 5–6% of Delta on load; Hudi superior on updates.Vendor‑affiliated (Onehouse). Methodology not fully disclosed.
Hudi 1.0 release blog (Dec 2024)70× write‑amplification reduction, 5.7× query latency improvement for partial updates.Self‑reported, synthetic workload. No external reproduction.
LinkedIn comparison (2025)Hudi “significantly outperforms” Delta/Iceberg on updates.Secondary source, no published benchmark code.
Hudi‑internal micro‑benchmarks (2025)Flink writer 2–3× throughput gains (235K records/sec) vs. 1.0. Clustering 10–15× faster.Project‑internal; no third‑party verification.

The Default‑Config Trap Hudi’s default write operation is upsert, which triggers record‑level indexing and key‑lookup overhead even for bulk‑append workloads. Switching to bulk‑insert brings Hudi within 5–6% of Delta, but this is not the out‑of‑the‑box experience. New users unaware of this nuance can see 10‑fold write‑amplification, leading to shattered SLAs (Agent 16, 24). This is a critical deployment risk.

Unresolved uncertainties:

  • No academic benchmark exists for Hudi 1.0+; the pluggable format framework and LSM timeline have not been tested in a cross‑format study.
  • All “TPC‑DS” references are self‑run; the TPC has not audited any lakehouse result.

---

4. Total Cost of Ownership & Pricing

Hudi’s incremental processing can reduce compute spend, but a full TCO comparison is impossible without independent data.

EvidenceClaimQuality
Uber (June 2025)75% runtime reduction, 60% single‑run cost cut vs. pre‑Hudi.Anecdotal; compares Hudi with legacy full‑table rewrites, not with Iceberg/Delta.
Zupee (Dec 2025)60% S3 cost reduction, 30% storage savings.Single‑customer case study; savings attributed to ZSTD + Hudi, but no cross‑format baseline.
Onehouse webinar (Feb 2025)20–80% infrastructure cost reduction.Unverifiable — Onehouse does not publish per‑DBU or per‑GB pricing. No methodology shared.
Operational overheadCompaction, clustering, cleaning require scheduled compute and engineering time.Uber runs ~70K table‑service jobs/day (2025). A small team may spend 20‑30% of capacity tuning Hudi (Agent 3, 5, 8).

Hidden cost drivers:

  • Storage overhead: MoR log files + multi‑index metadata add 10‑40% extra storage (aggregate unquantified).
  • Maintenance compute: Async table services are not free; at scale they can rival ingestion compute.
  • Engineering time: Hudi’s configuration surface (hundreds of parameters) and known fragility points (metadata table, compaction failures) increase support burden vs. Iceberg.

Bottom line: Hudi can lower compute costs for CDC‑heavy workloads, but the total cost (compute + storage + people) has not been benchmarked against alternatives. The absence of public managed‑service pricing (Onehouse) further obscures real‑world TCO.

---

5. Developer Experience & Operational Reality

What’s good:

  • HoodieStreamer provides a production‑hardened, low‑code ingestion conduit for Kafka/Debezium.
  • SQL‑ification (Hudi 1.0+): CREATE TABLE … USING hudi, MERGE INTO, CALL run_clustering() lower the barrier for SQL‑centric teams.
  • Hudi‑rs (Rust/Python) allows read‑only access without Spark/JVM, broadening accessibility.

What’s painful:

  • Steep learning curve: users must master timeline architecture, key generation, index types, COW/MOR, precombine fields, and payload classes — significantly more than Delta or Iceberg.
  • Table‑services burden: compaction, cleaning, and clustering are “automated” only in the sense that frameworks exist; effective operation requires careful scheduling, monitoring, and manual recovery. Peloton’s production case (July 2025) moved to inline services after async tasks caused File Not Found errors under concurrency.
  • Metadata‑table fragility: several reported incidents (GitHub issues, Medium post, 2024‑2025) describe MDT corruption requiring manual S3 file deletion. The cleaner can become a bottleneck, stalling all pipelines.
  • Configuration overload: hundreds of hoodie.* properties; wrong index type or missing lock provider leads to silent degradation or data loss.
  • Engine support asymmetry: writes are concentrated in Spark/Flink; Trino, Presto, Athena, and others are read‑only or lagging. No native Snowflake or DuckDB support.

Adversarial insight: The gap between Hudi’s “automatic table services” marketing and the on‑the‑ground operational reality is a recurring contradiction. Prospective teams should budget for a dedicated data‑platform engineer to manage Hudi.

---

6. Ecosystem & Interoperability

DimensionStatus (2025–2026)
Write enginesSpark 3.3–4.1, Flink 1.17–2.1, Kafka Connect.
Read‑only query enginesTrino (no MDT support since v419), Presto, Athena (read‑only, timestamp bugs on MoR, version locked to 0.14–0.15), BigQuery, ClickHouse, StarRocks.
Multi‑format interoperabilityApache XTable (incubating) translates Hudi metadata to Iceberg/Delta — enables Snowflake/BigQuery reads. Hudi 1.1’s pluggable format framework integrates this natively. But: MoR tables and advanced indexes are not fully supported through translation, and no production‑scale case studies exist.
Catalog supportHive, Glue, BigQuery, DataHub, and (new in 1.1) Apache Polaris.

Mindshare reality: Iceberg is now the de‑facto multi‑engine standard, with native support in Snowflake, BigQuery, Athena, and Databricks (via UniForm). Hudi’s ecosystem is narrower, and the “pluggable format” pivot is a bet that may reduce the penalty — but it hasn’t yet closed the gap. As of early 2026, multiple independent analysts state Iceberg has won the “format wars” for greenfield projects; Hudi retains a strong niche for CDC‑heavy workloads.

---

7. Security, Privacy & Compliance

Encryption

  • Parquet Modular Encryption (Hudi 0.11+) provides column‑level, client‑side encryption with AES‑NI optimisation. Uber’s 2022 paper reported <8% write overhead and <5% read overhead. Freshness gap: overhead has not been remeasured with Hudi 1.x’s new write paths.
  • No built‑in integration for key rotation or re‑encryption; relies on Hadoop KMS. Encrypting the primary key used for record‑level indexing may require decryption at lookup time — behaviour not documented.

GDPR right‑to‑erasure

  • Hudi supports record‑level hard/soft deletes, but physical erasure requires a three‑step pipeline: DELETE → Compaction → Cleaning. If the cleaner fails (as documented in multiple incidents), PII remains accessible on storage indefinitely. No automated alert on cleaner stalls. The default cleaning retention (10 commits) may be far too short; verifying complete erasure requires reading raw Parquet/Avro files directly, not just querying through Hudi.

Audit trail

  • The timeline is an append‑only event log, but it is not cryptographically tamper‑evident — it relies on cloud storage IAM. Time‑travel retention is bounded by the cleaner; “infinite history” requires disabling the cleaner entirely, which causes unbounded storage growth and potential performance degradation.

Access control Hudi delegates all auth to the query engine and storage layer. Row‑level filtering and column masking depend on Apache Ranger (applied at the query engine). Non‑JVM readers (Hudi‑rs, Daft) bypass these controls entirely — a gap not covered in standard documentation.

---

8. Competitive Landscape & Substitution Risk

FormatPrimary StrengthThreat to Hudi
Apache IcebergMulti‑engine inter‑op, partition evolution, broadest ecosystemHigh — default choice for new projects; Snowflake/BigQuery/DuckDB native support.
Delta LakeDeep Spark/Databricks integration, liquid clustering, auto‑optimisationHigh in Spark‑centric shops; Databricks’ UniForm bridges to Iceberg.
Apache PaimonLSM‑tree streaming native (Flink‑first)Medium — targets same CDC niche, but lacks Hudi’s production scale.

Hudi’s defensible niche:

  • Record‑level, secondary, and expression indexes provide unmatched point‑lookup performance for real‑time merge workloads.
  • Proven at extreme scale: Uber (19,500 datasets, 10 PB/day, Jan 2026), ByteDance (exabyte), Walmart.
  • The “storage engine + pluggable format” architecture could turn Hudi into a platform that writes Iceberg/Delta and exposes Hudi’s ingestion advantages to a wider audience — but this is still maturing and lacks production case studies.

Substitution risk is elevated for analytics‑first workloads, multi‑engine shops, and teams with limited operations capacity. Hudi’s future adoption depends on converting its technical strengths into demonstrable, independently verified value.

---

9. Decision Matrix for Personas (Condensed)

PersonaFitKey Caveat
Streaming/CDC Data Engineer✅ StrongNot “sub‑minute” — near real‑time means minutes. Operational complexity is non‑trivial.
Multi‑Engine Platform Architect⚠️ Moderate/DecliningHudi’s inter‑op via XTable is improving but immature. Iceberg is the safer multi‑engine standard.
Databricks‑Shop Lead❌ WeakDelta Lake’s integration and Photon optimisation dominate unless Hudi’s incremental ingestion is critical.
Cost‑Conscious VP/CTO✅ Strong (with caveats)Potential compute savings exist, but hidden maintenance and engineering costs may erase them. No independent TCO exists.
AI/ML Platform Engineer⚠️ EmergingHudi‑rs and VECTOR/BLOB types are on the roadmap, but production‑ready multimodal support is not yet GA (target 2.0).
GDPR‑Focused Architect⚠️ Workable but riskyThree‑step erasure pipeline has failure points; verification of complete deletion requires low‑level file scans.

---

10. Key Risks & Open Questions

  1. No independent performance baseline: The last neutral academic benchmark (CIDR 2023) is stale, and all post‑2024 claims are vendor‑linked. Any cross‑format performance argument must be treated as directional, not factual.
  2. Data‑loss bugs in production paths: Issues #10803 (MoR compaction data loss) and #9674 (lost writes under concurrent operations) remain documented, with incomplete remediation.
  3. Metadata table as a single point of failure: Corruption or cleaner stalls can block all pipelines and prevent data purging (including GDPR deletes). Manual recovery is required.
  4. Default‑config trap: upsert mode for bulk loads can cause 10× write amplification; teams must know to switch to bulk‑insert.
  5. Pluggable format maturity: XTable translation does not fully support MoR tables or record‑level indexes, limiting the “write‑Hudi, read‑Iceberg” vision.
  6. Commercial‑backer concentration: Onehouse is the sole dedicated Hudi vendor; its failure would stress the project’s maintenance cadence.

---

11. Practical Recommendations

  • Adopt Hudi if your primary ingest pattern is high‑volume, keyed upserts/deletes from streaming sources (Kafka, Debezium) and you have a dedicated data‑engineering team comfortable with JVM tuning, index selection, and table‑service management.
  • Avoid Hudi if your workload is predominantly batch analytics, you need native Snowflake/DuckDB query support, or your team lacks the bandwidth to manage compaction and metadata table health.
  • Mitigation strategies for Hudi adopters:
  • Start with explicit configurations: use bulk‑insert for initial loads, enable primary‑key conflict checks, and use storage‑based locking (RFC‑91) or DynamoDB with conservative concurrency.
  • Schedule table services aggressively but with monitoring — inline compaction for moderate volumes, async with alerting for larger scales.
  • Test GDPR erasure end‑to‑end: verify that cleaning physically removes data by scanning raw files, and set up cleaner‑failure alerts.
  • Plan for the metadata table: treat it as a critical component with its own maintenance schedule.
  • Hedge ecosystem risk by using Hudi’s pluggable format to also expose Iceberg metadata (XTable) for multi‑engine reads, but test thoroughly — MoR and indexed tables may not translate fully.
  • Demand independent benchmarks: before making final performance‑based decisions, run your own workload‑specific, optimised‑configuration tests with Hudi 1.2, Iceberg 1.7, and Delta 4.0 on your target cloud.

---

12. Parting Caveat

This synthesis reflects the best available evidence as of early‑2026. Hudi is a powerful, battle‑tested engine that has delivered enormous value at Uber and similar organisations. However, the public evidence base is dominated by vendor‑affiliated sources, and the project’s operational complexity and undiscovered failure modes create a non‑trivial barrier to risk‑free adoption. The wise decision maker will treat Hudi not as a default, but as a specialised tool for the hardest streaming‑CDC problems — and will validate each claim with their own data and their own engineers.