WeatherETL
End-to-end serverless weather analytics pipeline on AWS with live Superset dashboards
Overview
WeatherETL is a fully automated, cloud-native data engineering pipeline that ingests live weather data from the Open-Meteo public API and delivers it through a production-grade AWS infrastructure into a queryable PostgreSQL database and visual Superset dashboard.
The pipeline is orchestrated by AWS EventBridge, which fires a scheduled trigger to invoke a Lambda function. The Lambda function fetches current weather readings, stages the raw JSON into S3, transforms the data, and loads it into a self-managed PostgreSQL instance running on an EC2 server — demonstrating an IaaS pattern rather than a fully managed RDS approach. Apache Superset, containerized via Docker Compose on the same EC2 host, connects to PostgreSQL and serves live charts accessible from a browser.
CloudWatch collects Lambda execution logs and EC2 system metrics, while IAM roles are scoped to least-privilege policies to restrict each service to only the permissions it needs. The project was designed to showcase the full ETL lifecycle — extraction, staging, transformation, loading, and visualization — entirely within the AWS ecosystem.
Pipeline Flow
1. EventBridge Trigger
Scheduled cron rule fires periodically and invokes the Lambda function without any manual intervention.
2. Lambda — Extract
Python Lambda function calls the Open-Meteo API, retrieves hourly weather data, and stages raw JSON to an S3 bucket.
3. S3 — Stage
Raw payloads are persisted in S3 as a durable staging layer, enabling replay and audit without re-calling the upstream API.
4. EC2 / PostgreSQL — Transform & Load
Lambda reads from S3, normalizes the schema, and upserts rows into PostgreSQL running on EC2 via a secure connection.
5. Docker Superset — Visualize
Apache Superset in Docker Compose connects to PostgreSQL and renders interactive charts for trend analysis and historical comparison.
Key Features
- Fully automated serverless ingestion via EventBridge + Lambda
- S3 raw data staging for durability and replay
- Self-managed PostgreSQL on EC2 (IaaS pattern)
- Docker Compose deployment of Apache Superset
- CloudWatch logs and metrics for observability
- IAM least-privilege roles scoped per service
- Schema normalization in Python transform step
- Upsert logic prevents duplicate rows on reruns
Screenshot
View the full source code, infrastructure setup, and deployment instructions on GitHub.
View on GitHub