Available for Data Engineering opportunities

Biplov Singh

Data Engineer & Apache Airflow Contributor specializing in building scalable batch & real-time streaming pipelines, distributed processing architectures, and robust cloud data platforms.

2 Merged PRs Apache Airflow Core
Stream & Batch Distributed Lakehouse Stack
IIIT Lucknow M.Sc. Data Science ('27)
01. About

Engineering Background

I specialize in designing, scaling, and maintaining high-throughput data architectures and distributed processing platforms.

My focus lies in building resilient, automated data workflows. As an open-source contributor to Apache Airflow, I standardise provider testing frameworks (Snowflake, Teradata). I construct real-time streaming pipelines using Apache Kafka and Spark Structured Streaming, manage distributed compute clusters with Spark and Hadoop, and architect modern Data Lakehouses with Apache Iceberg.

Currently pursuing my M.Sc. in Data Science at IIIT Lucknow (2025–2027), I bridge the gap between raw data streams and analytical engines using Terraform, Docker, PostgreSQL, and AWS.

Open Source Apache Airflow Contributor Merged Snowflake & Teradata provider test refactors
Education M.Sc. Data Science @ IIIT Lucknow Focus: Distributed Big Data Systems (2025–2027)
Location Jamshedpur, Jharkhand, India
Availability Open for Roles & Consultations

Experience & Education

Teaching Assistant IIIT Lucknow Aug 2025 – Dec 2025 • Lucknow, UP

Assisted undergraduate coursework in data structures, programming, and data science. Led weekly hands-on lab sessions and evaluated technical projects.

M.Sc. in Data Science IIIT Lucknow 2025 – 2027 (Expected)

Specializing in distributed computing architectures, real-time data streaming, cloud infrastructure, and large-scale data engineering.

B.Sc. in Computer Application & Cyber Security Jharkhand Raksha Shakti University 2022 – 2025 • CGPA: 7.92

Rigorous grounding in systems architecture, networking, database management, systems security, and algorithmic problem solving.

02. Stack

Technologies & Toolkit

Core Languages

Python SQL (PostgreSQL, MySQL) C / C++ Bash / Shell

Orchestration & Streaming

Apache Airflow Apache Spark / PySpark Apache Kafka Hadoop (Hive, Pig, Flume)

Databases & Storage

Apache Iceberg / Parquet PostgreSQL & PostGIS Redis In-Memory Cache MongoDB

Infrastructure & DevOps

Docker & Compose Kubernetes Terraform (IaC) & AWS Azure Git & CI/CD (pytest)
03. Work

Open Source & Projects

OPEN SOURCE // 01 Apache Airflow

Apache Airflow Core Repository

Open Source Contributor • Workflow Orchestration

Contributed code modifications to the official core repository of Apache Airflow, standardizing provider testing infrastructure, eliminating legacy testing dependencies, and validating CI/CD pipeline stability.

Refactored provider test suites for Snowflake (#68898 Merged) and Teradata (#68190 Merged) modules.
Eliminated legacy unittest imports in favor of modern pytest conventions, fixtures, and parameterization.
Validated pipeline stability across 80+ continuous integration matrix checks on multi-platform runner environments.
Architecture & Contribution Flow
Provider Test Modules Snowflake & Teradata
→
Pytest Refactoring Modern fixtures & assertions
→
Apache Breeze Environment Local containerized validation
→
80+ CI Runner Checks Multi-platform (AMD/ARM)
→
Upstream Merged apache/airflow core
Results: Standardized provider test harnesses, removed deprecated unittest imports, and successfully merged both PRs (#68898 and #68190) into the Apache Airflow main branch.
Key Takeaway: Navigating a 35,000+ star open-source project taught rigorous adherence to backwards-compatibility standards, clean mock isolation, and deep CI automation hygiene.
PROJECT // 02 Streaming

Real-Time Ride Analytics Pipeline

Kafka & PySpark Structured Streaming

An event-driven streaming pipeline processing replayed NYC taxi event streams at 100+ events/sec, calculating geospatial surge pricing, windowed aggregations, and dual storage persistence.

Ingests high-frequency ride events from replayed NYC taxi records via partitioned Apache Kafka topics.
Applies 5-minute tumbling windows and 2-minute watermarks in PySpark Structured Streaming to handle out-of-order records.
Integrates PostGIS spatial geofencing for dynamic zone-based surge multipliers.
Dual storage persistence: Redis for sub-5ms low-latency cache queries and AWS S3 Apache Iceberg for lakehouse analytics.
Architecture & Data Flow
NYC Taxi Stream Replayed producer (100+ ev/s)
→
Apache Kafka Partitioned brokers
→
PySpark Streaming 5m tumbling • 2m watermarks
→
PostGIS Engine Geospatial surge index
→
Dual Sinks: Redis + Iceberg Hot cache & S3 Lakehouse
Results: Sustained 100+ simulated events/second with sub-second stream processing latency, delivering sub-5ms dashboard queries from Redis and durable Parquet snapshot commits to S3 Apache Iceberg.
Key Takeaway: Watermark tuning is vital for balancing stream freshness against late-arriving event completeness. Next iteration: adopting AWS MSK Serverless and automating Iceberg compaction via dbt-core.
PROJECT // 03 Foundations & Batch

Website Clickstream Trend Analysis

Hadoop Big Data Pipeline • Distributed Batch Foundations

An end-to-end distributed data pipeline demonstrating core big data engineering foundations: collecting, ingesting, and aggregating high-volume clickstream logs using Hadoop, Apache Flume, Pig, and Hive.

Ingests clickstream web logs continuously via configured Apache Flume source-channel-sink agents.
Cleans raw semi-structured access logs and filters corrupted records using Apache Pig Latin scripts.
Structures analytical tables and aggregates user session trends in Apache Hive with partitioned schemas.
Fully containerized distributed system environment managed via Docker for reproducible benchmarking.
Architecture & Data Flow
Raw Web Logs High-volume access stream
→
Apache Flume Agent spooling ingestion
→
HDFS Storage Distributed file system
→
Apache Pig ETL & corrupt filtering
→
Apache Hive Partitioned analytical tables
Results: Successfully parsed multi-gigabyte log datasets with 100% schema enforcement and invalid record isolation; partitioned Hive tables reduced analytical scan times by 65%.
Key Takeaway: Established core intuition for HDFS block replication, MapReduce execution stages, and YARN resource negotiation, providing a foundational baseline for modern distributed engines like Spark and Trino.
04. Contact

Get In Touch

I am open to new data engineering opportunities, project collaborations, and discussions on distributed architectures.