Data Acquisition Platform for Product Development Insights

A successfully implemented scaling, optimization, and extension of data acquisition system as well as the development of AWS CloudWatch dashboards for better visibility of the system itself.

Our client is a tech start-up that uses artificial intelligence to gain insights into real-time consumer demands and provide manufacturers and beauty brands with analytics and predictions about what beauty products they should launch, what risks and challenges they may face, and helps identify emerging opportunities. 

PROJECT OUTLINE

Before embarking on the platform development, Quintagroup experts had a few introductory calls with the client in order to dive deeper into their requirements and to better understand their objectives. As soon as our team gained a crystal clear understanding of the product vision, we built out a product roadmap.

The client company was interested in scaling, optimization, and extension of their data acquisition platform. The project's goals were:

  • to gain performance insights for the data acquisition platform enabling our client to make data-driven decisions and optimize resources;
  • to update and add features to the system, which would enhance the performance of a system itself and ramp up data sets.

QUINTAGROUP SOLUTION

Quintagroup’s team of seasoned professionals analyzed our client's current data acquisition approaches and strategies to better understand how to improve the performance and stability of a system.

Our team built out several Cloudwatch dashboards on AWS that log a set of metrics to track the performance of the acquisition platform. Having analyzed the metrics, we added features to the Python-based system that optimized its performance and increased the amount of data being acquired and processed.

With our solution, data collection takes place in 3 stages:

  1. Pulling data from target accounts.
    Every 2 hours, we grab pictures and videos that have been posted recently by the key target Instagram accounts.
  2. Tagging media with products.
    We aim to tag all the media that comes into our system with products that are contained, shown in, or referenced by the image or video. In this process are involved both automated and human means. When the product ID model can not identify the product, it passes to a manual tagging tool. Once tagged. It gets stored.
  3. Pulling comments from tagged media items.
    Every 40 minutes, <48 hrs old comments on the media item are taken from Instagram via private API and are saved in the comments pool in the client's database.

TECH STACK

Backend: Python, Django, Celery, RabbitMQ, PostgreSQL

AWS: Lambda, CloudWatch, Step Functions

BUSINESS OUTCOME

As a result of our partnership, the client company managed to enhance the performance, visibility, and stability of their system. Quintagroup has developed the AWS Cloudwatch dashboard, optimized the process of data acquisition, and updated the system for better and smooth functioning.

With the solution Quintagroup has delivered, our client is now able to easily get insights in the beauty area, specifically to predict and measure the demand for products before they launch and generally open up new horizons in this industry.

Are you in need of product development? Our team of experts is ready to help you implement your ideas in real life. Contact us today to get started on a successful project.

Connect with our experts Let's talk