ETL Jobs

I have millions of raw records that need contact-level enrichment and, until now, I have been relying on Melissa. Licensing costs, however, have become prohibitive, so I’m searching for a markedly cheaper way to append email addresses, phone numbers, postal addresses, and, where possible, full names—without sacrificing accuracy. Here’s what I need: • A clear proposal for a new enrichment workflow or alternative provider that keeps per-record costs low while maintaining dependable match rates. • A small pilot run (10–50 k records) so I can verify accuracy, error handling, and throughput before we scale. • A documented pipeline—scripted in Python, R, or a lightweight ETL tool of your choice—showing how to push millions of records throu...

For Project 40648634 - AI Commodity Intelligence Platform Workflow you need: Collect → Analyze → Compare → Score → Challenge → Recommend → Risk-check → Present. Research only, no auto-trading. Client approves/rejects. Stack for Phase 1: GitHub + Codespaces (free allowance) + Python + SQLite + Streamlit + OpenAI API + Scheduler + GitHub Secrets. Using Codex as agentic engineer. What I will build: 1. Ingest: Weather (Open-Meteo/NOAA), Gov Reports (USDA WASDE/FAO PDF scraper + LLM summary), News RSS, Prices (Yahoo/FRED). Modular for satellite/NDVI later. 2. Core: SQLite schema (raw_data, evidence, scores), ETL, Scoring 0-100 [30% Supply Stress, 25% Demand, 20% News Sentiment, 15% Price Anomaly, 10% Risk], Anomaly detection. 3. Challenge/Risk-check: 2+ ...

Our production datalake pipeline on Hadoop is misbehaving: Spark and Flink jobs crash intermittently, completed stages fail to commit their results to the Hive tables, and overall throughput has slowed to a crawl. I need someone to dive in, trace the failures, and leave me with a clean, fully functioning data flow. You will have direct access to the existing Spark and Flink code, YARN cluster dashboards, Hive metastore, and any relevant logs. The immediate goal is to identify and correct the root causes of the job failures and the missing Hive writes, then tune the pipeline so it can keep up with daily load without time-outs or excessive retries. Acceptance criteria – once you are done: • All scheduled Spark and Flink jobs finish successfully when triggered manually and by t...

AWS Data Engineer Needed — 1:1 Mentor/Tutor to Teach AWS Data Engineering Category: Data Engineering / Cloud Computing / Tutoring & Mentoring Description: I'm looking to build strong AWS Data Engineering skills from the ground up and want an experienced AWS Data Engineer to act as a personal tutor/mentor — covering core concepts, tools, and hands-on skills needed to work as a data engineer on AWS. What I'm looking for: 1:1 sessions (video call) covering AWS data engineering fundamentals and hands-on labs Coverage of core services: S3, Glue, Redshift, Athena, Kinesis, Lambda, EMR, Step Functions, RDS/DynamoDB ETL/ELT pipeline design, data lake vs. data warehouse concepts Python/PySpark for data engineering, SQL for analytics Guidance on real-world project work...

I have a production oracle instance that has grown beyond 100 GB and it now needs to live on SQL ,Server postgres sql . The task covers everything from preparing the source, mapping the schema, transferring the data, and validating that every object—tables, views, functions, and constraints—behaves exactly the same once it lands in SQL Server. Here is what I expect to walk away with: • A fully migrated SQL Server database, ready for cut-over • A repeatable migration script or SSIS/ADF/DMS pipeline so I can re-run the process for future refreshes • A concise migration report highlighting any datatype conversions, performance tweaks, and objects that required special handling • Verification scripts or queries that prove row counts, checksums, and refe...

I need a complete data-warehouse solution for Dual Specialist Hospital (Medicare) in the United States. The core objective is to pull our patient health records into a single, well-structured repository and make that information immediately useful for data analytics and reporting. Clinical staff, quality-improvement teams, and executives all expect fast insight without wading through multiple systems or static spreadsheets. The build must support: • Real-time data analysis so new encounters, labs, and vitals are visible in minutes, not days. • Customizable reporting templates our analysts can adjust without developer help. • Data visualization tools that present trends clearly for non-technical decision-makers. I’m open to the platform you recommend—wheth...

We are looking for an experienced KNIME developer to build and test a structured KNIME workflow based on an already-defined process and workflow diagram. The workflow will: read specific worksheets from one Excel workbook; validate and filter project/bidder data; scan a folder structure containing bidder working files; identify and classify files based on filename/path rules; group and pivot the results; join the folder-scan results with the Excel register; apply deterministic readiness rules; produce a final multi-sheet Excel output report. The workflow must handle 1 to 10 bidders dynamically, without fixed branches for each bidder. We will provide: the workflow logic and block diagram; a detailed build guidance document; a populated Excel test workbook; sample folder/file structures...