AI Quality Assurance Analyst
Full Time $90,000 - $100,000 / yearJob Overview
We are seeking an AI Quality Assurance (QA) Analyst to support the testing, validation, and continuous improvement of AI-driven solutions within our mortgage platform. This role ensures that AI models, automation tools, and system integrations function accurately, compliantly, and efficiently across the loan lifecycle.
The ideal candidate has experience in mortgage operations (e.g., underwriting, processing, or closing) combined with strong QA/testing skills and an interest in AI technologies.
The ideal candidate has experience in mortgage operations (e.g., underwriting, processing, or closing) combined with strong QA/testing skills and an interest in AI technologies.
Responsibilities and Duties:
- Validate AI-driven outputs related to loan processing, underwriting, document recognition, and decisioning.
- Design and execute test cases for AI models, automation workflows, and system enhancements.
- Identify, document, track defects, inconsistencies, and performance issues.
- Perform regression, functional, and user acceptance testing (UAT) for mortgage systems.
- Collaborate with product, engineering, and data science teams to improve AI accuracy and reliability.
- Analyze AI model behavior and flag potential risks, biases, or compliance concerns.
- Ensure alignment with mortgage industry regulations and investor guidelines (e.g., Non-QM with the future agency).
- Support data validation efforts, including OCR (optical character recognition) and document classification tools.
- Monitor production performance and provide feedback for ongoing AI model training and optimization.
- Assist in developing QA standards and best practices for AI implementations.
Education and Experience:
- 3+ years of experience in QA, software testing, or mortgage operations
- Strong understanding of the mortgage lifecycle, including origination, processing, underwriting, and closing
- Experience with Loan Origination Systems (LOS), such as Vesta, LauraMac, or similar platforms
- Familiarity with QA methodologies, including test planning, execution, and defect tracking
- Strong analytical and problem-solving skills with a high level of attention to detail
- Ability to analyze system behavior and validate expected versus actual results
Preferred Qualifications:
- Experience with AI/ML tools, automation platforms, or intelligent document processing
- Understanding of OCR, NLP, or decisioning engines
- Knowledge of mortgage compliance and regulatory requirements
- Experience with SQL, data validation, or basic scripting is a plus
- Exposure to Agile or Scrum development environments
- Ability to work in a hybrid office setting in Bloomington, MN
Key Competencies:
- AI & QA Testing Expertise: Designs and executes test cases to validate AI models, automation workflows, and system functionality.
- Analytical & Critical Thinking: Evaluates AI outputs, identifies discrepancies, and assesses model behavior for accuracy and risk.
- Mortgage Domain Knowledge: Applies understanding of loan lifecycle, documentation, and regulatory requirements to testing scenarios.
- Attention to Detail & Data Validation: Ensures precision in testing, defect tracking, and validation of OCR and document classification outputs.
- Cross-Functional Collaboration: Works effectively with product, engineering, and data science teams to improve AI performance.
How This Role Demonstrates Our Values:
- Integrity: Ensures AI-driven decisions and outputs are accurate, compliant, and aligned with regulatory and investor requirements.
- Collaboration: Partners with cross-functional teams to enhance AI solutions and support seamless system performance.
- Excellence: Maintains high standards in testing, validation, and continuous improvement of AI-driven processes.
- Critical Curiosity: Investigates AI behavior, identifies potential risks or biases, and seeks opportunities to improve model accuracy and efficiency.