← к выдаче
разработка

Data Engineer

Commit Offshore21 дн. назад

Эта же роль в других локациях (2)

  • —
  • —

Коротко

Owns the company data lake for financial events on S3 + Snowflake/Databricks, including lakehouse architecture, CDC streaming, data layout, retention · Needs: 3 years in data engineering; lakehouse architecture (bronze/silver/gold, Iceberg/Delta/Hudi); TB+ scale optimization; Snowflake/Databricks/BigQuery; Kafka +

Описание от работодателя

We are looking for a Mid Data Engineer in Kraków, Poland who will own the company data lake — the system of record for millions of financial events a day (bets, wallet movements, live odds) across 12M+ active users. You decide how that data lands, is stored, retained, and governed on S3 + Snowflake/Databricks, so analytics, finance, and regulators all see accurate, reconciled data with zero drift from source. Domain: Regulated iGaming / wallet & ledger data. Audit-heavy: regulators, finance and analytics all consume the same tables. Millions of financial events per day, terabyte-plus scale. What you'll be doing: Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution. Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events. Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake. Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion. Guarantee correctness: freshness SLAs, drift detection, reconciliation against the source wallet and ledger systems. Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails. Monitor ingestion health, data anomalies, and cloud storage/compute spend. Must-have: 3 years' experience in hands-on delivery within that architecture — pipelines, ingestion, models, monitoring Lakehouse architecture — bronze/silver/gold layering, an open table format (Iceberg, Delta, or Hudi), schema evolution. Data layout & query optimization at TB+ scale — partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake. Cloud lakehouse/DWH in production — Snowflake, Databricks, or BigQuery. CDC & streaming ingestion — Kafka + Debezium or equivalent; late, duplicate and out-of-order events. Strong SQL and data modeling — enough relational grounding to reason about the OLTP systems you capture from. Critical for financial ledgers. Correctness — freshness SLAs, drift detection, reconciliation against source wallet/ledger systems. Governance — catalogs, lineage, row/column access control, PII masking, retention, GDPR deletion. Cloud object storage — S3 or GCS, plus storage tiering and archival. Python and an orchestrator — Airflow or Dagster, as tools. Location & work model: Kraków, Poland. Hybrid — 2 days per week from the office .