Description
I have two scraper/API jobs that are supposed to continuously pull data from two different websites and write it into a PostgreSQL database hosted on AWS RDS (Aurora/RDS instance called "fincrawl", running in us-east-2 / Ohio). The database has not received any new data in about a year, and I need an experienced engineer to diagnose and fix the pipeline so it's reliably populating the DB again — plus set up monitoring so this doesn't silently break again. Environment: Database: AWS RDS PostgreSQL instance ("fincrawl") in us-east-2 Compute: An EC2 instance ("FinCrawl") in the same VPC runs the two data-pull jobs/APIs Security groups: rds-ec2-1 (DB side) and ec2-rds-1 (EC2 side) control traffic between the two Access: I can provide AWS console access (or scoped IAM credentials), SSH/EC2 Instance Connect access to the EC2 instance, and DBeaver is already set up on my end for direct DB inspection What I need done: Diagnose why the two ingestion jobs stopped writing to the database — could be a crashed process, a cron/scheduler failure, a broken scraper due to a website layout change, an expired credential/API key, a network/security group misconfiguration between EC2 and RDS, disk/storage issue on the instance, or something else entirely Fix whatever is broken so both jobs reliably pull data from their respective source websites and write it into the correct tables in the fincrawl database Verify data is actually landing correctly — check for gaps, duplicate rows, or malformed da