04 · Cloud & Data Engineering
Public health reporting, from upload to approval.
A reporting workflow that tracks required datasets, preserves corrections, and connects each archived report to the versions a manager approved.
The problem
Which files are ready, and which output was approved?
Recurring reporting brings together files from multiple programs. A valid file can arrive while another dataset is missing. A correction can replace an earlier submission, and a manager can be looking at an outdated version when approving the report.
The project makes those conditions explicit. File validation, cycle completeness, and manager approval each have their own state.
My contribution
Translate reporting rules into a reviewable workflow.
This portfolio implementation was developed with AI-assisted coding. The work covers the reporting rules, state transitions, analyst and manager screens, failure scenarios, and operating documentation. The demonstration uses synthetic records and does not include employer code or production data.
The resulting workflow connects dataset requirements to version history, human decisions, and the exact reports retained in the archive.
How it works
Define the reporting cycle
A manager sets the reporting dates and required datasets. Missing submissions remain visible until every current dataset is valid.
Validate incoming data
The shared Python validator checks required fields, identifiers, reporting dates, and controlled values. Safe formatting changes are applied; ambiguous values produce explicit errors.
Correct the source
An analyst uses row-level feedback to correct the CSV. Each upload receives a new version. Earlier results remain in history, and a failed replacement blocks approval.
Review the current versions
A manager inspects validation results and aggregate reports, then approves or rejects with a reason. A revision check prevents approval from an outdated screen.
Archive the approved reports
Approval locks the versions and records their checksums. Archiving verifies the report bytes and preserves the selected versions in a manifest.
Demonstrated outcome
Two required datasets. Three submissions. One approved archive.
In the browser walkthrough, the first valid upload left the cycle at 1 of 2 datasets complete. After the second dataset arrived, the manager rejected the cycle with a reason. A replacement became version 2 while version 1 remained in history.
Approval locked program A version 1 and program B version 2. The archive retained those exact versions and their SHA-256 checksums. Repeating archive verification returned the same archive.
Engineering decisions
Make changes and failures traceable.
- Current version determines readiness
- A failed replacement blocks review even when an older file passed. Late validation of an older upload cannot make it current again.
- Serialize state changes
- SQLite write transactions protect version allocation and manager decisions. Review requests include the revision the manager saw.
- Bind approval to report content
- The approval snapshot records versions and hashes. Missing or changed report files stop the archive operation.
- Give retries a stable destination
- The archive uses the approval identifier as its destination and writes the manifest last. After a storage failure, the cycle remains approved and the operation can be retried.
Verification
Exercise the paths that can change the result.
Thirty backend tests passed, including required-dataset completeness, failed replacements, stale approvals, role restrictions, concurrent version allocation, legacy-state migration, and archive recovery. Two repository checks, lint, and the frontend production build also passed.
The browser walkthrough covered configuration, uploads, aggregate review, rejection, replacement, approval, and archive verification. Invalid CSV and storage-failure scenarios were checked through automated API tests. These results describe software behavior; no reporting-time savings or production health outcomes were measured in this demonstration.
AWS architecture and next work
Carry the same reporting rules into cloud processing.
The repository includes an AWS architecture and infrastructure definitions using S3, EventBridge, Step Functions, Lambda, and an intended PostgreSQL state layer. The local workflow shares the CSV validation logic used by the Lambda code.
Connecting cloud events to PostgreSQL, binding submissions to immutable S3 versions, and integrating enterprise identity remain future work. The local role selector simulates permissions. Worker crash recovery, retention automation, and external-agency delivery are also outside the demonstrated scope.
Approved cycles are locked; later corrections require a new cycle. The archive and database are separate storage systems, so committed Archived status is the completion signal. Docker image validation was unavailable because the local engine was stopped.
What this project demonstrates
Data engineering that accounts for the people reviewing the output: explicit requirements, actionable validation, preserved history, and a record of what was approved.