Data Engineer & Apache Airflow Contributor specializing in building scalable batch & real-time
streaming pipelines, distributed processing architectures, and robust cloud data platforms.
I specialize in designing, scaling, and maintaining high-throughput data architectures and distributed
processing platforms.
My focus lies in building resilient, automated data workflows. As an open-source contributor to
Apache Airflow, I standardise provider testing frameworks (Snowflake, Teradata). I
construct real-time streaming pipelines using Apache Kafka and Spark Structured
Streaming, manage distributed compute clusters with Spark and
Hadoop, and architect modern Data Lakehouses with Apache Iceberg.
Currently pursuing my M.Sc. in Data Science at IIIT Lucknow (2025–2027), I bridge the gap
between raw data streams and analytical engines using
Terraform, Docker, PostgreSQL, and AWS.
Open SourceApache Airflow ContributorMerged Snowflake & Teradata provider test refactors
EducationM.Sc. Data Science @ IIIT LucknowFocus: Distributed Big Data Systems (2025–2027)
LocationJamshedpur, Jharkhand, India
AvailabilityOpen for Roles & Consultations
Experience & Education
Teaching AssistantIIIT LucknowAug 2025 – Dec 2025 • Lucknow, UP
Assisted undergraduate coursework in data structures, programming, and data science. Led weekly
hands-on lab sessions and evaluated technical projects.
M.Sc. in Data ScienceIIIT Lucknow2025 – 2027 (Expected)
Specializing in distributed computing architectures, real-time data streaming, cloud infrastructure, and large-scale data engineering.
Contributed code modifications to the official core repository of Apache Airflow, standardizing provider
testing infrastructure, eliminating legacy testing dependencies, and validating CI/CD pipeline stability.
Refactored provider test suites for Snowflake
(#68898Merged) and Teradata
(#68190Merged) modules.
Eliminated legacy unittest imports in favor of modern pytest conventions, fixtures, and parameterization.
Validated pipeline stability across 80+ continuous integration matrix checks on multi-platform runner environments.
Results: Standardized provider test harnesses, removed deprecated unittest imports, and successfully merged both PRs (#68898 and #68190) into the Apache Airflow main branch.
Key Takeaway: Navigating a 35,000+ star open-source project taught rigorous adherence to backwards-compatibility standards, clean mock isolation, and deep CI automation hygiene.
PROJECT // 02Streaming
Real-Time Ride Analytics Pipeline
Kafka & PySpark Structured Streaming
An event-driven streaming pipeline processing replayed NYC taxi event streams at 100+ events/sec,
calculating geospatial surge pricing, windowed aggregations, and dual storage persistence.
Ingests high-frequency ride events from replayed NYC taxi records via partitioned Apache Kafka topics.
Applies 5-minute tumbling windows and 2-minute watermarks in PySpark Structured Streaming to handle out-of-order records.
Integrates PostGIS spatial geofencing for dynamic zone-based surge multipliers.
Dual storage persistence: Redis for sub-5ms low-latency cache queries and AWS S3 Apache Iceberg for lakehouse analytics.
Results: Sustained 100+ simulated events/second with sub-second stream processing latency, delivering sub-5ms dashboard queries from Redis and durable Parquet snapshot commits to S3 Apache Iceberg.
Key Takeaway: Watermark tuning is vital for balancing stream freshness against late-arriving event completeness. Next iteration: adopting AWS MSK Serverless and automating Iceberg compaction via dbt-core.
PROJECT // 03Foundations & Batch
Website Clickstream Trend Analysis
Hadoop Big Data Pipeline • Distributed Batch Foundations
An end-to-end distributed data pipeline demonstrating core big data engineering foundations: collecting,
ingesting, and aggregating high-volume clickstream logs using Hadoop, Apache Flume, Pig, and Hive.
Ingests clickstream web logs continuously via configured Apache Flume source-channel-sink agents.
Cleans raw semi-structured access logs and filters corrupted records using Apache Pig Latin scripts.
Structures analytical tables and aggregates user session trends in Apache Hive with partitioned schemas.
Fully containerized distributed system environment managed via Docker for reproducible benchmarking.
Architecture & Data Flow
Raw Web LogsHigh-volume access stream
→
Apache FlumeAgent spooling ingestion
→
HDFS StorageDistributed file system
→
Apache PigETL & corrupt filtering
→
Apache HivePartitioned analytical tables
Results: Successfully parsed multi-gigabyte log datasets with 100% schema enforcement and invalid record isolation; partitioned Hive tables reduced analytical scan times by 65%.
Key Takeaway: Established core intuition for HDFS block replication, MapReduce execution stages, and YARN resource negotiation, providing a foundational baseline for modern distributed engines like Spark and Trino.
04. Contact
Get In Touch
I am open to new data engineering opportunities, project collaborations, and discussions on distributed
architectures.