old eshop web scrape
This commit is contained in:
Priec
139
data_scrape/README.md
Normal file
139
data_scrape/README.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# kompress e-shop scrape
|
||||
|
||||
Snapshot of the live catalogue at <http://e-shop.kompress.sk> (PrestaShop 1.5/1.6),
|
||||
taken 2026-07-29. Only the product catalogue is captured — no CMS pages, no orders.
|
||||
|
||||
Re-run with `python3 scrape.py` (stdlib only, no dependencies). It is idempotent:
|
||||
already-downloaded images are skipped.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Contents |
|
||||
| --- | --- |
|
||||
| `categories.json` | 30 categories, flat list ordered parent-before-child |
|
||||
| `products.json` | 118 products, ordered by PrestaShop id |
|
||||
| `images/<product_id>/<image_id>.jpg` | 181 product images, original uploads |
|
||||
| `images/categories/<category_id>.jpg` | 30 category thumbnails, original uploads |
|
||||
|
||||
## Category schema
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": 8, // PrestaShop id_category, kept as the stable key
|
||||
"name": "Gáza, role, štvorce, prírezy",
|
||||
"title": "…", // link title attr; often a longer marketing blurb
|
||||
"url": "http://e-shop.kompress.sk/8-gaza-role-…",
|
||||
"parent_id": 7, // null for top-level
|
||||
"path": ["Zdravotnícke", "Gáza, role, štvorce, prírezy"],
|
||||
"depth": 1, // 0 = top level
|
||||
"description_html": "<p>…</p>",
|
||||
"description": "…", // plain-text rendering
|
||||
"image": { "url": "…/img/c/8.jpg", "file": "images/categories/8.jpg" },
|
||||
"product_ids": [59, 62, …],
|
||||
"product_count": 8
|
||||
}
|
||||
```
|
||||
|
||||
The tree is 2 levels deep: 3 roots (`Zdravotnícke`, `Somatické`, `Pre invalidov`),
|
||||
27 children. The roots hold a few products directly; `Somatické` and `Pre invalidov`
|
||||
hold none of their own.
|
||||
|
||||
## Product schema
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": 73, // PrestaShop id_product
|
||||
"name": "Návlek na obuv",
|
||||
"url": "http://e-shop.kompress.sk/-netkane-textie-…/73-navlek-na-obuv.html",
|
||||
"reference": "6811", // the "Kód:" on the page; "" when the shop has none
|
||||
"price": 0.04, // tax included, EUR — this is the displayed price
|
||||
"price_display": "0,04 €",
|
||||
"price_tax_excluded": 0.033333,
|
||||
"price_without_reduction": 0.04,
|
||||
"tax_rate": 20.0,
|
||||
"currency": "EUR",
|
||||
"quantity_available": 1000,
|
||||
"available_for_order": true,
|
||||
"has_attributes": false, // true when the product has variant combinations
|
||||
"attribute_groups": [], // see "Variants" below
|
||||
"variants": [],
|
||||
"short_description_html": "<p>…</p>",
|
||||
"short_description": "…",
|
||||
"description_html": "<p>…</p><ul><li>…</li></ul>",
|
||||
"description": "…", // plain text, list items prefixed with •
|
||||
"breadcrumb": [{ "id": 7, "name": "Zdravotnícke", "url": "…" }, …],
|
||||
"category_path": ["Zdravotnícke", " Netkané textíe, Pervin, SMS - …"],
|
||||
"categories": [{ "id": 10, "name": "…", "path": [...] }], // every category listing it
|
||||
"default_image_id": 279,
|
||||
"images": [{ "id": 279, "url": "…/img/p/2/7/9/279.jpg",
|
||||
"file": "images/73/279.jpg", "bytes": 53443 }]
|
||||
}
|
||||
```
|
||||
|
||||
`breadcrumb`/`category_path` is the product's primary category trail. `categories`
|
||||
lists every category the product is actually listed in — 7 products appear in more
|
||||
than one, so treat product↔category as many-to-many.
|
||||
|
||||
## Variants
|
||||
|
||||
20 of the 118 products sell in several sizes/packagings — 74 combinations in total,
|
||||
across 4 attribute groups: `Rozmer`, `Veľkosť`, `Balenie`, `Počet útržkov v balení`.
|
||||
Each is its own purchasable unit with its own code, price and stock, so these map to
|
||||
`product_variants` rather than to plain products.
|
||||
|
||||
```jsonc
|
||||
"attribute_groups": [
|
||||
{ "id": 4, "name": "Rozmer", "slug": "rozmer",
|
||||
"attributes": [ { "id": 21, "label": "10cm x 10cm", "default": false }, … ] }
|
||||
],
|
||||
"variants": [
|
||||
{
|
||||
"id": 30, // id_product_attribute
|
||||
"attribute_ids": [23],
|
||||
"label": "5cm x 5cm", // groups joined with " / " when there are several
|
||||
"attributes": [{ "group": "rozmer", "label": "5cm x 5cm" }],
|
||||
"reference": "1211", // per-variant Kód; "" on some products
|
||||
"quantity_available": 500,
|
||||
"price_impact_tax_excluded": -2.35,
|
||||
"price_tax_excluded": 1.04,
|
||||
"price": 1.14, // tax included — what the shop shows when selected
|
||||
"minimal_quantity": 1,
|
||||
"image_id": 100, // null when the variant has no own image
|
||||
"is_default": true
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
`price` is computed as `(product.price_tax_excluded + price_impact_tax_excluded) *
|
||||
(1 + tax_rate/100)`, matching `themes/default/js/product.js`. Verified: every
|
||||
product's displayed price equals its default variant's computed price, and every
|
||||
product's `quantity_available` equals the sum of its variants' stock.
|
||||
|
||||
On variant products the top-level `price`/`reference` describe the **default**
|
||||
variant, and `quantity_available` is the total across variants. 14 of the 74
|
||||
variants have an empty `reference`.
|
||||
|
||||
## Images
|
||||
|
||||
`images[].url` points at PrestaShop's untouched original upload
|
||||
(`/img/p/<digits split by slash>/<image_id>.jpg`), not a generated thumbnail. For
|
||||
product 73 that is 53 KB versus 10 KB for the `-large_default` the page shows. Every
|
||||
downloaded file was verified to be a real JPEG; sizes run 3 KB – 634 KB.
|
||||
|
||||
Product 270 (`Plachta na posteľ z PVC`) has no image in the shop at all — its
|
||||
`images` array is empty and `default_image_id` is null.
|
||||
|
||||
## Coverage notes
|
||||
|
||||
- Product URLs came from crawling every category listing, unioned with `sitemap.xml`.
|
||||
All category listings fit on a single page, so nothing was lost to pagination.
|
||||
- `sitemap.xml` is stale: it advertises 182 products, but 83 of those (ids 8–55,
|
||||
72, 86, 96, 103–105, 109, 118–197 among them) return HTTP 404 — they are
|
||||
disabled/deleted products, verified individually via
|
||||
`index.php?id_product=<id>&controller=product`. 118 live products remain.
|
||||
- Some products are served under both an old and a new slug (e.g. product 59 as
|
||||
`…59-gaza-hydrofilna-skladana-zlozky-z-gazy.html` and `…59-gaza-hydrofilna-skladana.html`).
|
||||
These are deduplicated by id, keeping the URL the category listing links to.
|
||||
- 29 products have an empty top-level `reference` — that is genuine, the shop leaves
|
||||
the Kód field blank for them (the block is rendered `display:none`). 20 of those 29
|
||||
12 of those 29 are variant products, where the code lives on each variant instead.
|
||||
Reference in New Issue
Block a user