Home/Methods

E-commerce · Pricing

How to monitor competitor prices for commodity products

To monitor competitor prices, keep a list that maps each of your products to the same product on competitor sites, read the price from the page's structured data every night, store every reading in an append-only history, and compare only fresh, plausible prices against yours. Matching the right product and keeping the feed alive are harder than reading the price. A price change based on the comparison is a decision to test, not a proven gain.

Updated 2026-10-11

What competitor price monitoring is for

Competitor price monitoring is the regular, automated reading of the prices other shops charge for the same products you sell, so that you know when you are undercut and where you have room to charge more. It matters most for commodity products: generic goods such as home and garden items, where shoppers compare the identical item across several retailers and choose largely on price. For products that are not the same item as the competitor's, a direct price comparison says much less.

A monitor answers three questions per product: what is the cheapest live competitor price, how far is ours from it, and is that gap a problem or an opportunity. We built two versions. One is a small standalone watchlist for ad hoc checks. The other feeds a nightly verdict engine for a home-and-garden store and produces a repricing plan.

Step 1: map each product to the same product elsewhere

The mapping between your product and the competitor's product page is the foundation. Every later number depends on it. A mapping is a row with your product, the competitor's product URL and the last price read.

When we revisited the home-and-garden store's engine, the mapping was already in good shape: 396 of 479 products (83%) had at least one competitor URL. Of those URLs, 94% pointed directly at retailers' product pages; the other 6% were ad redirect links, which are not product pages and should be discarded.

We also tried to build mappings automatically by matching product images against a product database. Recall was 15%: most products were not found, partly because that database did not cover the main marketplace in the niche. We kept image matching only as a supplement for products with no mapping. A human-reviewed URL list beat clever matching by a wide margin.

Step 2: read the price from structured data, not from the visible page

The visible HTML is the most fragile place to read a price. The order we use is: the platform's own product endpoint where one exists (on Shopify, the /products/{handle}.js file), then the JSON-LD Product block's offers.price and availability, then the product:price:amount or og:price meta tags, then an itemprop="price" attribute. Most Shopify, WooCommerce and similar shops publish JSON-LD, because search engines use it for rich results.

Normalise the number before storing it. In Romanian and many European formats, 1.234,56 means one thousand two hundred and thirty-four point five six, with the dot as a thousands separator and the comma as the decimal mark. A naive parser will reject it or read the wrong value. Strip the currency text first, then decide which symbol is the decimal mark.

Give every URL a hard time limit. Our nightly re-scrape stops waiting on any single URL after 18 seconds, because a few hanging DNS lookups were enough to stall a whole run.

Step 3: store every reading, and feed the engine before it runs

Store prices in an append-only history: one new row per reading, never an update of the old one. Append-only history answers „what was the competitor's price last Tuesday?” and lets you see a price drop as a change, not as an overwritten value. Our watchlist flags two events on each check: a competitor price that dropped since the last reading, and a competitor price below ours.

Schedule the scrape before whatever consumes it. Our re-scrape runs every night, an hour before the verdict engine, so the engine always reads prices from the same night.

Measure the success rate per run and treat it as the health signal. Our re-scrape reads a price for roughly 84% to 87% of mapped URLs; the failures are mostly HTTP 403 bot blocks and dead URLs. A scraper can return HTTP 200 on every page and still extract nothing useful, so the success count must be the number of prices extracted, not the number of pages fetched.

Step 4: compare only fresh and plausible prices

A verdict per product compares our live price with the cheapest and the median live competitor price and labels the product as too expensive, OK, or having headroom (room to raise the price). Two filters keep bad data out of the verdict.

1. Freshness. Only competitor prices read within the last 3 days count. A stale price is worse than no price, because it looks like data.

2. Plausibility. A competitor price below 35% or above 280% of ours is ignored. Prices that far away almost always mean the mapping points at a different product, a different pack size or an accessory.

A small tolerance, 5% in our setup, separates „roughly equal” from „more expensive”, so a few cents of difference do not trigger a change.

Step 5: turn the verdict into a cautious repricing plan

Price competitiveness alone does not decide a price. Our plan also reads each product's recent sales and its margin after its real advertising cost per order, and makes two asymmetric moves. It raises prices on products that sell well but keep too little after advertising, moving them toward the market price. It lowers prices on slow sellers with plenty of margin, but never below a floor, calculated as floor = (1 + VAT) × (cost of goods + shipping + advertising cost per order + minimum profit), so an advertised sale still pays for itself. Products without stock are skipped.

If a product loses money even at the market price, the plan flags it as an advertising problem, not a pricing one. Changing the price cannot fix a cost per order that is too high.

Before writing any price, the apply step checks that the live price still equals the price the plan was built on, and skips the product if it has changed. It applies only clear cases, keeps the old price in the plan file so every change is reversible, and runs as a dry run unless explicitly told to write.

The first run applied 28 price changes, 25 increases toward the market and 3 decreases where we were above it, and held back 21 products flagged as advertising problems plus the increases that had no live competitor price to anchor them.

Where we were wrong

We first assumed the verdict engine was broken, because 157 products sat on „insufficient data”. The engine was running every night. What had died was its input: the old scraper that fed competitor prices had stopped, and the freshest competitor prices were about 90 days old. The fix was a new feed for the existing mappings, not a new engine. Check the age of the inputs before rebuilding the thing that reads them.

We also spent effort on automatic image matching before re-reading the URLs we already had. The URL list gave far better coverage.

Our first impact check came about 1.5 days after the changes. All 28 new prices had held, demand was within its usual daily range, and the highest-volume product kept its demand after an 18% increase. But a 15-unit spike on one product turned out to be a single bulk order, and many products that sell less than once a day showed zero sales simply by chance. 1.5 days cannot measure price elasticity. The proper check is after 7 to 14 days, product by product, including the product's position in shopping ads, which can fall when its price rises.

Limits and checklist

A price monitor tells you where you stand; it does not tell you what a price change will do to sales. It cannot see shops that block automated reading, it is only as good as the product mapping, and the plausibility filter only catches gross mismatches, not a slightly different model. We do not claim a proven profit effect from repricing.

Checklist. Map each product to the same product on competitor sites, and review the mapping by hand. Read prices from the platform endpoint or JSON-LD before the visible page. Normalise number formats. Put a time limit on every URL. Store every reading append-only. Run the scrape before the engine that uses it, and watch the extracted-price count, not the HTTP status. Compare only prices fresher than a few days and within a plausible band. Raise and lower asymmetrically, with a floor that covers advertising. Verify the live price before writing, keep the old price, and re-check after one to two weeks.

Questions and answers

What is the most reliable way to scrape a competitor's product price?

Read the price from structured data rather than from the visible page. Try the platform's own product endpoint first (on Shopify, the product's .js file), then the JSON-LD Product block's offers.price and availability, then the product:price:amount or og:price meta tags, and only then an itemprop price attribute. Most Shopify and WooCommerce shops publish JSON-LD for search engines. Normalise European number formats such as 1.234,56 before storing the value.

How do I match my products to the same products on competitor sites?

Keep an explicit mapping table: your product, the competitor's product URL, and the last price read. Review it by hand. In our case 83% of products already had a mapping, while automatic image matching recovered only 15% of products. Discard ad redirect links, which are not product pages, and treat any competitor price far from yours as a sign that the mapping points at a different product or pack size.

How often should competitor prices be checked?

For commodity products, once a night is a practical rhythm, scheduled to finish before whatever reads the prices, such as a verdict or repricing job. Count only prices read in the last few days; a stale price looks like data but is not. Store every reading in an append-only history, so a competitor's price drop shows up as a change you can see and date, not as an overwritten number.

Should I lower my price every time a competitor undercuts me?

No. Combine the competitor price with your own sales and your margin after advertising cost per order. Raise prices toward the market on products that sell well but keep too little, lower prices only on slow sellers with spare margin, and never go below a floor that covers goods, shipping, advertising and a minimum profit. If a product loses money even at the market price, the problem is advertising cost, not price.

How long should I wait before judging a price change?

At least one to two weeks per product. A reading after a day or two is dominated by noise: products that sell less than once a day show zero sales by chance, and a single bulk order can look like a demand spike. Also check the product's position in shopping ads, since a higher price can lower it. A short-term reading can only show that nothing collapsed.

All methods