Project - Scalable Product Feed Infrastructure

Overview

At DPG Media, I worked in a team responsible for the software that generates product widgets displayed on editorial websites such as Ouders van Nu and VT Wonen. These widgets aggregate products from multiple sources — Bol.com, Tweakers, Amazon, and product feeds from advertisers.

My work focused on simplifying the existing infrastructure by phasing out the legacy Offer Manager and Affiliate Redirect Service, and replacing them with modern, maintainable alternatives.

Key Contributions

Channable Product Adapter

DPG Media adopted the Channable platform to manage product feeds. I contributed significantly to implementing Channable as a new Product Adapter. Unlike the Bol.com or Amazon integrations — which fetch product data in real time via API — Channable products are stored locally.

One of the biggest challenges was handling the sheer volume of incoming product feeds. I designed a queue-based processing architecture, documented with detailed flowcharts, which was approved by the team and subsequently implemented collaboratively. This approach ensures large feeds are ingested reliably without overwhelming the system.

Because products are now stored locally, a search capability was needed. I built a product search feature using Postgres full-text search, enabling fast and relevant results across the locally stored catalog.

Bol.com Marketing API Migration

The existing Bol.com adapter suffered from persistent "Rate Limit Exceeded" errors that had caused production issues for a long time. I rebuilt the adapter from scratch using the new Bol.com Marketing API, which resolved the rate-limiting problems and provided smoother, more reliable product data synchronization.

Infrastructure Simplification

Contributed to phasing out two legacy services — the Offer Manager and the Affiliate Redirect Service — consolidating their responsibilities into the new architecture and reducing overall system complexity.