AI compatibility
Refactoring a legacy ETL pipeline is squarely in AI's wheelhouse — with a human sign-off before it touches production.
AI can handle this.
Average across 1 submission.
The honest read
This is a well-scoped coding task with crisp success criteria: the output either processes 40 GB correctly in under 2 hours or it doesn't. An AI code agent can handle the Perl-to-Python translation, add validation and logging scaffolding, and produce benchmarking hooks — but a human engineer must review the fixed-width field definitions, validate output correctness against known-good data, and own the production deployment decision.
Aggregated across 1 submission.
The five dimensions
Repeatability
HighETL refactoring follows a well-worn pattern: parse the existing logic, map it to idiomatic Python, add error handling and logging layers. The structural work is highly repeatable and doesn't require novel judgment each time.
Ambiguity Tolerance
MediumThe performance target (40 GB in under 2 hours) and feature requirements (row validation, rejection handling, logging, benchmarking) are concrete. However, the fixed-width field schema, rejection policy semantics, and acceptable data quality thresholds are not specified and will require clarification or inference from the source script.
Data & Tool Availability
MediumThe agent needs the 800-line Perl script, the fixed-width field layout specification, sample input files, and PostgreSQL schema details. If these are provided, the agent can proceed; without them, it can only produce a structural scaffold that requires significant human gap-filling.
Error Cost
MediumA buggy ETL pipeline silently corrupting or dropping rows in a 40 GB production dataset is a serious but recoverable problem — the original data files still exist. The risk is real but bounded, and a proper test harness against known-good output mitigates it substantially before deployment.
Human Judgment Required
MediumDecisions about what constitutes a 'malformed row,' how aggressively to reject versus coerce data, and whether the refactored output is semantically equivalent to the original all require a domain-knowledgeable human to validate. The coding mechanics are automatable; the correctness sign-off is not.
What an agent would need
- The full 800-line Perl source script provided as input
- Fixed-width field layout specification or column definitions for the mainframe export format
- PostgreSQL target schema (table definitions, types, constraints)
- Sample input files including known-malformed rows to validate rejection logic
- Defined acceptance criteria for row rejection (e.g., null key fields, out-of-range values, encoding errors)
Best-matched agent type
The kind of agent this work would call for if it were a fit. For this task, it isn't.
Run your own fit check
Get a calibrated read on your specific task in under a minute.