Automated Data Integration: Scaling Client Onboarding

Background

At Kanarys, our ability to deliver valuable workforce insights to clients was critically dependent on clean, well-structured HRIS data. When I joined, the company relied on a single team member dedicated to cleaning and loading HRIS files, a process that took anywhere from 1-5 days per client depending on data quality and organizational size. Enterprise clients with complex data structures often required extensive back-and-forth communication about issues, creating a severe operational bottleneck. To better understand this critical process, I trained on and began personally handling file processing alongside the specialist, giving me firsthand insight into the inefficiencies and challenges that were frustrating both our team and clients.

By the time I implemented this solution, I had transitioned to a full-time Senior Product Manager role, giving me the organizational authority and broader scope to drive this cross-functional initiative.

Challenge

The HRIS data collection and processing challenges included:

• Manual Process Inefficiency: A substantial portion of operational time was consumed by manually processing HRIS files from clients.

• Data Format Inconsistencies: Despite providing templates, client data would arrive with inconsistent formatting (e.g., "woman" vs. "female"), incompatible date formats, and calculation requirements.

• Data Quality Issues: Missing information (job titles, departments, managers) was often discovered late in the process, requiring multiple client interactions to resolve.

• Technical Debt: Each data issue discovered required custom handling, creating a growing backlog of one-off solutions.

• Client Experience Impact: The back-and-forth to resolve data issues created friction and delayed value delivery to clients.

  
  

Solution Approach

I identified two parallel solution paths that would address these challenges:

• Self-Service Upload Tool: Create a client-facing tool that would validate, transform, and standardize data during the upload process.

• Direct HRIS Integrations: Implement a scalable integration strategy to connect directly with clients' HRIS systems.

  
  

Self-Service Upload Tool Development

• User Flow Design: Created a user-friendly flow for clients to download a template, populate it, and upload it to the platform.

• Technical Implementation: Used SQL to design the data transformation pipeline, developing custom queries to handle data validation and standardization processes.

• Database Architecture: Collaborated with engineering to optimize the database schema for efficient ETL processing, ensuring scalability for enterprise-level data volumes.

• Validation Engine: Developed a system to identify common formatting issues, missing data, and inconsistencies using both front-end validations and back-end SQL-based rules.

• Mapping Interface: Built an intuitive mapping tool that identified similar terms (e.g., "female" = "woman") and asked clients to confirm matches, storing these mappings for future automation.

• Error Handling: Implemented a process that allowed clients to export specific errors for correction while proceeding with the rest of the data upload.

• Version Control: Added functionality for clients to overwrite their file if they needed to update information, with SQL-based change tracking to maintain data integrity.

  
  

HRIS Integration Strategy

• Build vs. Buy Analysis: Led a comprehensive technical evaluation and competitive analysis to determine whether to build custom integrations or leverage existing solutions, including creating a decision matrix weighted on technical requirements, resource constraints, and time-to-market considerations.

• Vendor Selection: Selected Merge, a SOC-2 compliant third-party integration platform specializing in HR system connections, after evaluating multiple integration providers based on API capabilities, security compliance, and supported HRIS systems.

• Technical Integration: Worked closely with engineering to architect the integration between our database and Merge's API, designing the data mapping and transformation logic to ensure seamless data flow.

• Embedded Experience: Implemented an iframe to embed the integration tool directly into our platform's workflow, collaborating with UI/UX to create a cohesive user experience.

• Database Schema Optimization: Used SQL to create optimized database structures that could efficiently store and process the incoming integration data from various HRIS systems.

• Security Protocol: Developed documentation and processes to help clients navigate security approval processes with their IT teams, including detailed data flow diagrams and security compliance information.

• Historical Data Retrieval: Configured the integration to access historical HRIS data, writing custom SQL queries to enable more comprehensive year-over-year analysis and trend identification.

  
  

Team Leadership and Training

• Customer Success Team Training: Developed and delivered comprehensive training for the Customer Success team on both the self-service tool and integration process, ensuring they could effectively support clients through these new workflows.

• Pilot Program Management: Personally set up and managed the first client integrations with Merge while the embedded solution was still being developed, providing hands-on implementation to gather real-world insights.

• Cross-functional Coordination: Aligned engineering, customer success, and sales teams on messaging, capabilities, and support processes for the new solutions.

  
  

Results

Operational Efficiency

• Reduced Processing Time: Decreased HRIS data processing time from 1-5 days to minutes.

• Resource Reallocation: Redirected team resources (including myself and the dedicated specialist) from manual data processing to higher-value client engagement activities.

• Scalability Improvement: Created capacity to handle significantly more client onboardings without proportional staffing increases.

• Bottleneck Elimination: Removed dependency on specialized staff for data processing, preventing delays when team members were unavailable or at capacity.

  
  

Data Quality

• Standardized Formatting: Ensured consistent data formatting across all client implementations.

• Error Reduction: Significantly reduced data errors through real-time validation during the upload process.

• Historical Data Access: Gained access to client historical data, enabling more robust year-over-year analysis.

  
  

Client Experience

• Reduced Setup Time: Decreased HRIS setup time from days or weeks to as little as 10 minutes with direct integrations.

• Self-Service Empowerment: Enabled clients to manage their own data uploads and corrections without waiting for our team.

• Transparent Process: Provided clear visibility into data issues and resolution paths for clients.

  
  

Business Impact

• Onboarding Acceleration: Dramatically decreased the time from contract signing to first value delivery.

• Contract Expansion: Enabled more comprehensive year-over-year analysis capabilities that supported contract renewals and expansions.

• Platform Differentiation: Created a competitive advantage through seamless data integration compared to more manual competitors.

  
  

Skills Demonstrated

Technical Skills Leveraged

• SQL Database Management: Wrote and optimized queries for data validation, transformation, and integration

• ETL Process Design: Architected the Extract, Transform, Load workflows for HRIS data

• Systems Integration: Evaluated and implemented third-party integration solutions

• Data Modeling: Designed database schemas to accommodate diverse client data structures

• Business Intelligence: Connected processed data to visualization and reporting systems

  
  

Project Management Skills Applied

• Technical Project Leadership: Led cross-functional team through complex technical implementation

• Knowledge Transfer: Trained customer success team on technical processes and troubleshooting

• Requirements Gathering: Translated business needs into technical specifications

• Vendor Management: Evaluated, selected, and managed integration with third-party provider

• Change Management: Guided internal teams and clients through process transformation

• Stakeholder Communication: Bridged technical details with business value for executive buy-in

  
  

Lessons Learned

• Automation Early and Often: Identifying and automating manual processes early creates compounding benefits for both clients and internal teams.

• Build vs. Buy Importance: The decision to partner with Merge rather than build custom integrations saved significant development resources while providing a superior solution.

• User-Centered Design: Focusing on the client experience during data uploads and transformations dramatically improved adoption and satisfaction.

• Error Prevention vs. Correction: Building proactive error prevention into the workflow was far more efficient than reactive error correction.

• Security Considerations: Integrations requiring security approval need clear documentation and stakeholder management strategies to navigate client approval processes.

• Technical Understanding Drives Product Management: My hands-on experience with SQL and database management was essential for effectively leading the technical aspects of this project and communicating requirements to engineering.

  
  

By transforming our approach to HRIS data processing, we not only improved operational efficiency but fundamentally enhanced the client experience, reduced time-to-value, and enabled more sophisticated data analysis capabilities across our product portfolio.