How a Search Engine with 2.7 Million Products Works: Lessons from gega.bg

How a Search Engine with 2.7 Million Products Works: Lessons from gega.bg

gega.bg brings together products and deals from dozens of Bulgarian online stores. The database holds about 2.7 million items. Here is how the site is built and what we have learned while running it.

Where the products come from

The stores take part in affiliate networks such as ProfitShare and CJ and provide their products as feeds – XML, CSV or via an API. Every network has its own format, and even stores within the same network fill in the fields differently.

So each source has its own parser that translates the data into one common format: name, price, old price, image, brand, category, store and link. The rest of the site works only with that common format.

Categories

Every store has its own categories – "Women's trainers", "Shoes › Women › Sport", "Trainers for women". Put together, they become chaos. So each store is mapped to a main category of the site (Fashion, Electronics, Kids and so on), and its subcategories are arranged beneath it.

Search

The first version searched with the classic "contains this text" (LIKE '%text%'). With millions of rows, that means the database scans every product for every search. With a full-text index (FULLTEXT), the database searches words in a pre-built index and the result comes back in a fraction of a second. For this, the tables were converted to InnoDB.

Maintenance

Affiliate programmes change constantly: stores join and leave, feeds change address, one feed is split into several. So we regularly compare the list of active stores in the networks with what the site downloads, remove inactive ones and add new ones.

Lessons for any data-heavy project

  1. One common format. However many sources there are, all are translated into one data model.
  2. Do not trust incoming data. Encodings, empty prices, duplicates and broken images are daily reality – the checks belong in the parser.
  3. Indexes for the queries you actually run. Not indexes "just in case", but for the search and lists the site shows.
  4. Lightweight pages. Cards with the essentials, images loaded on demand and minimal JavaScript.
  5. Document the process. Which store comes from where and what was changed – otherwise in a year nobody remembers.

If you need product and price import from suppliers, price comparison or a data-heavy catalogue, see web-based software or get in touch.

← All articles

Have an idea for a website, app or system?

Tell us briefly what you need – we will get back to you for a short call and a clear quote.