Dao Duy Dai_

Overview

This project transformed manual SQL scripts and scheduled queries into an automated, scalable data pipeline capable of processing billions of events daily across multiple mobile games. The solution addresses challenges of scaling data operations, standardizing schemas across different games, and maintaining cost efficiency in BigQuery.

Key Innovations

- Configuration-Driven Models: YAML-based model generation reduced 5,000+ lines of SQL to 300 lines of config
- Cost Optimization: Strategic partitioning, clustering, and FARM_FINGERPRINT for efficient joins
- OBT Approach: One-Big-Table strategy with pivot columns for flexible, cost-efficient analytics
- Custom Hooks: Pre-hooks for data quality validation and post-hooks for optimized incremental updates
- Orchestration: Airflow-based workflow management for multiple game projects

Results

- 60% reduction in BigQuery costs
- Processing 2B+ events daily with 45-minute end-to-end pipeline
- 450+ automated data quality tests
- Scalable architecture supporting 50+ event types across multiple games

Read the full case study: Building a Cost-Optimized Gaming Analytics Pipeline with dbt and BigQuery

Developed during the VPBank Technology Hackathon 2024, the Customer Lifetime Value Optimization project focuses on automating data processing and enhancing marketing strategies. By leveraging AWS services, the project integrates customer data collection, machine learning models, personalized product recommendations, sentiment analysis, and omnichannel marketing optimization. Key Objectives:
1. Streamline Data Ingestion: Design an ETL pipeline to integrate data from all VPBank sources.
2. Empower Personalized Recommendations: Develop algorithms to learn customer preferences and recommend suitable products.
3. Optimize Omnichannel Marketing: Implement a feedback loop to refine algorithms for the best engagement channels and times.
4. Understand Customer Concerns: Use sentiment analysis on feedback data to identify barriers to customer satisfaction.

Project Phases:
1. Data Collection and ETL Pipeline Development: Customer data is collected and integrated using AWS services like S3, RDS, and Lambda. This includes extracting, transforming, and loading data into a centralized storage system.
2. Machine Learning Models for Customer Segmentation and CLV Prediction: Models developed in SageMaker segment customers based on behavior and predict Customer Lifetime Value (CLV), helping understand different customer segments.
3. Personalized Product Recommendations: Algorithms recommend products based on customer characteristics, integrating results into the client interface using a resilient API.
4. Sentiment Analysis and Customer Feedback: AWS Comprehend performs sentiment analysis on feedback from various channels to understand and improve customer experience.
5. Omnichannel Marketing Optimization: AWS Kinesis processes real-time data, while Amazon Pinpoint manages targeted marketing campaigns across multiple channels.
6. Data Visualization and Insights: AWS QuickSight creates dashboards to monitor customer interactions and campaign performance, optimizing marketing efforts.



Techniques
The core of this project involves:
- Data Collection and ETL Pipeline Development: Utilizing AWS CloudFormation, S3, RDS, and Lambda to automate the extraction, transformation, and loading of customer data.
- Machine Learning Models: Implementing models in SageMaker for customer segmentation, CLV prediction, and personalized product recommendations.
- API Integration: Utilizing AWS Lambda and Bedrock to generate and deliver personalized email content.
- Sentiment Analysis: Employing AWS Comprehend for analyzing customer feedback to improve customer engagement strategies.
- Omnichannel Optimization: Using AWS Kinesis for real-time data processing and Amazon Pinpoint for multi-channel marketing campaigns.
- Data Visualization: Leveraging AWS QuickSight to create dashboards for monitoring and optimizing marketing efforts.

Role and Responsibilities
As the lead developer, my responsibilities included:
- Designing and implementing the ETL pipeline using various AWS services.
- Developing and testing machine learning models in SageMaker for effective customer segmentation and product recommendations.
- Creating APIs with AWS Lambda and Bedrock for generating personalized marketing content.
- Integrating sentiment analysis using AWS Comprehend to derive actionable insights from customer feedback.
- Utilizing AWS Kinesis and Pinpoint to optimize omnichannel marketing strategies.
- Creating and managing data visualization dashboards with AWS QuickSight to monitor campaign performance.

Outcome
The project successfully automated the customer data processing pipeline and deployed machine learning models for enhanced personalized marketing strategies. As a result, our team was recognized as a finalist in the VPBank Technology Hackathon 2024 competition.

Through this project, I gained extensive experience in integrating various AWS services to build a scalable and efficient customer lifetime value optimization system.

Techniques

The core of this project involves:

- Prototyping: Rapidly developing initial versions of the recommendation system to explore different approaches and refine our methods.
- Model Testing: Evaluating various recommendation models to ensure accuracy and effectiveness in personalizing customer interactions.
- API Development: Using Flask, a micro web framework written in Python, to build and deploy APIs that serve the recommendations to the client's systems.
- Database Management: Handling large datasets with PostgreSQL to store and manage user data, which is crucial for generating accurate recommendations.
- Linux: Utilizing Linux-based servers to host and run our applications, ensuring stability and performance.
- Workflow Management: Leveraging Apache Airflow for task management and orchestrating the machine learning workflow.

Role and Responsibilities

As the lead developer, my responsibilities include:

- Building the end-to-end system architecture.
- Developing and testing multiple recommendation models to find the most effective approach.
- Implementing the models into a functional API.
- Managing the PostgreSQL database to ensure optimal performance and reliability.
- Overseeing the deployment of our Flask-based API on the client's Linux servers.
- Managing the machine learning workflow with Apache Airflow.

Outcome

The recommendation model has been fully developed and deployed. However, the ongoing phase of monitoring real-world usage and conducting A/B testing to fine-tune the model has not yet commenced. This next stage is crucial for adjusting the system to effectively meet real-world demands and improve the personalization of product recommendations.

Through this project, I have already gained valuable insights into the development of recommendation systems, including data processing, model building, and deployment strategies.

Overview

This project transformed manual SQL scripts and scheduled queries into an automated, scalable data pipeline capable of processing billions of events daily across multiple mobile games. The solution addresses challenges of scaling data operations, standardizing schemas across different games, and maintaining cost efficiency in BigQuery.

Key Innovations

- Configuration-Driven Models: YAML-based model generation reduced 5,000+ lines of SQL to 300 lines of config
- Cost Optimization: Strategic partitioning, clustering, and FARM_FINGERPRINT for efficient joins
- OBT Approach: One-Big-Table strategy with pivot columns for flexible, cost-efficient analytics
- Custom Hooks: Pre-hooks for data quality validation and post-hooks for optimized incremental updates
- Orchestration: Airflow-based workflow management for multiple game projects

Results

- 60% reduction in BigQuery costs
- Processing 2B+ events daily with 45-minute end-to-end pipeline
- 450+ automated data quality tests
- Scalable architecture supporting 50+ event types across multiple games

Read the full case study: Building a Cost-Optimized Gaming Analytics Pipeline with dbt and BigQuery

Developed during the VPBank Technology Hackathon 2024, the Customer Lifetime Value Optimization project focuses on automating data processing and enhancing marketing strategies. By leveraging AWS services, the project integrates customer data collection, machine learning models, personalized product recommendations, sentiment analysis, and omnichannel marketing optimization. Key Objectives:
1. Streamline Data Ingestion: Design an ETL pipeline to integrate data from all VPBank sources.
2. Empower Personalized Recommendations: Develop algorithms to learn customer preferences and recommend suitable products.
3. Optimize Omnichannel Marketing: Implement a feedback loop to refine algorithms for the best engagement channels and times.
4. Understand Customer Concerns: Use sentiment analysis on feedback data to identify barriers to customer satisfaction.

Project Phases:
1. Data Collection and ETL Pipeline Development: Customer data is collected and integrated using AWS services like S3, RDS, and Lambda. This includes extracting, transforming, and loading data into a centralized storage system.
2. Machine Learning Models for Customer Segmentation and CLV Prediction: Models developed in SageMaker segment customers based on behavior and predict Customer Lifetime Value (CLV), helping understand different customer segments.
3. Personalized Product Recommendations: Algorithms recommend products based on customer characteristics, integrating results into the client interface using a resilient API.
4. Sentiment Analysis and Customer Feedback: AWS Comprehend performs sentiment analysis on feedback from various channels to understand and improve customer experience.
5. Omnichannel Marketing Optimization: AWS Kinesis processes real-time data, while Amazon Pinpoint manages targeted marketing campaigns across multiple channels.
6. Data Visualization and Insights: AWS QuickSight creates dashboards to monitor customer interactions and campaign performance, optimizing marketing efforts.



Techniques
The core of this project involves:
- Data Collection and ETL Pipeline Development: Utilizing AWS CloudFormation, S3, RDS, and Lambda to automate the extraction, transformation, and loading of customer data.
- Machine Learning Models: Implementing models in SageMaker for customer segmentation, CLV prediction, and personalized product recommendations.
- API Integration: Utilizing AWS Lambda and Bedrock to generate and deliver personalized email content.
- Sentiment Analysis: Employing AWS Comprehend for analyzing customer feedback to improve customer engagement strategies.
- Omnichannel Optimization: Using AWS Kinesis for real-time data processing and Amazon Pinpoint for multi-channel marketing campaigns.
- Data Visualization: Leveraging AWS QuickSight to create dashboards for monitoring and optimizing marketing efforts.

Role and Responsibilities
As the lead developer, my responsibilities included:
- Designing and implementing the ETL pipeline using various AWS services.
- Developing and testing machine learning models in SageMaker for effective customer segmentation and product recommendations.
- Creating APIs with AWS Lambda and Bedrock for generating personalized marketing content.
- Integrating sentiment analysis using AWS Comprehend to derive actionable insights from customer feedback.
- Utilizing AWS Kinesis and Pinpoint to optimize omnichannel marketing strategies.
- Creating and managing data visualization dashboards with AWS QuickSight to monitor campaign performance.

Outcome
The project successfully automated the customer data processing pipeline and deployed machine learning models for enhanced personalized marketing strategies. As a result, our team was recognized as a finalist in the VPBank Technology Hackathon 2024 competition.

Through this project, I gained extensive experience in integrating various AWS services to build a scalable and efficient customer lifetime value optimization system.

Techniques

The core of this project involves:

- Prototyping: Rapidly developing initial versions of the recommendation system to explore different approaches and refine our methods.
- Model Testing: Evaluating various recommendation models to ensure accuracy and effectiveness in personalizing customer interactions.
- API Development: Using Flask, a micro web framework written in Python, to build and deploy APIs that serve the recommendations to the client's systems.
- Database Management: Handling large datasets with PostgreSQL to store and manage user data, which is crucial for generating accurate recommendations.
- Linux: Utilizing Linux-based servers to host and run our applications, ensuring stability and performance.
- Workflow Management: Leveraging Apache Airflow for task management and orchestrating the machine learning workflow.

Role and Responsibilities

As the lead developer, my responsibilities include:

- Building the end-to-end system architecture.
- Developing and testing multiple recommendation models to find the most effective approach.
- Implementing the models into a functional API.
- Managing the PostgreSQL database to ensure optimal performance and reliability.
- Overseeing the deployment of our Flask-based API on the client's Linux servers.
- Managing the machine learning workflow with Apache Airflow.

Outcome

The recommendation model has been fully developed and deployed. However, the ongoing phase of monitoring real-world usage and conducting A/B testing to fine-tune the model has not yet commenced. This next stage is crucial for adjusting the system to effectively meet real-world demands and improve the personalization of product recommendations.

Through this project, I have already gained valuable insights into the development of recommendation systems, including data processing, model building, and deployment strategies.