- INDUSTRY: Education
- Client: Evisions
- Location: Houstan,Texas,USA
- Completed Date: 20-05-2023
Executive Summary
A provider of higher education software sought to streamline the process of gathering and reporting institutional data for IPEDS (Integrated Postsecondary Education Data System) surveys. The existing manual process was time-consuming, resource-intensive, and prone to errors. To address these challenges, they developed DORIS (Data Orchestration and Repository Integration System), a centralized data platform. This case study examines the challenges faced, the DORIS solution, the technologies used, and the resulting benefits.
Challenge
The organization assists institutions with IPEDS surveys using their in-house built application. However, the process of updating and maintaining custom data connectors and aggregators for each client whenever IPEDS survey requirements changed was highly inefficient. Key challenges included:
Manual Data Gathering: Gathering raw institutional data for IPEDS surveys was a manual, time-consuming process.
Data Inconsistency: Maintaining data consistency across various client systems and data sources was difficult.
Resource Intensive: The process required significant IT resources to update and manage data connectors.
Lack of a Single Source of Truth: Data was scattered across multiple systems, making it difficult to ensure accuracy and integrity.
Solution
OktaBytes collaborated with the organization to develop DORIS, a cloud-based data platform designed to centralize and streamline IPEDS data management. The solution included:
-
Data Ingestion: Automated data ingestion from on-premises databases (Oracle, etc.) and cloud sources into AWS S3.
-
Pipeline Management: Tools for creating, editing, and managing data pipelines for efficient data flow.
-
Data Source Configuration: Ability to connect to various data sources (Banner, Colleague, CSV) and manage existing connections.
-
Data Mapping and Transformation: Features for mapping source data to the Evisions data model and applying data transformations.
-
Data Validation and Cleansing: Mechanisms for identifying and resolving data inconsistencies and errors.
-
Single Source of Truth: A centralized data repository for storing and managing IPEDS-related data.
-
DORIS-IRIS Integration: Seamless integration with their survey reporting application for generating survey reports.
Tech Stack
Typescript
JavaScript
NodeJS
React Js
Python
PostgreSQL
AWS Services
Lambda
API Gateway
S3
RDS
Athena
StepFunctions
CloudWatch
Glue
Cognito
IAM
CodePipeline
CodeBuild
Impact
-
Streamlined IPEDS Reporting: Automated data gathering and processing significantly reduced the time and effort required for IPEDS surveys.
-
Improved Data Accuracy: Centralized data management and validation ensured data consistency and accuracy.
-
Reduced IT Burden: Automated processes reduced the workload on IT staff for maintaining data connectors.
-
Enhanced Data Governance: Established a reusable data model and data governance practices.
-
Single Source of Truth: Provided a centralized repository for all IPEDS-related data, improving data integrity.
-
Simplified Data Access: Enabled both internal and external users to access data without complex point-to-point integrations.
Conclusion
The DORIS project successfully addressed the challenges associated with IPEDS data management by creating a centralized, automated, and efficient data platform. By leveraging AWS cloud services and a robust architecture, the organization significantly improved the IPEDS reporting process, reduced IT overhead, and enhanced data accuracy and governance. This project demonstrates the power of cloud-based solutions in streamlining complex data management tasks and enabling data-driven decision-making in higher education.
