Scraper Subsystem

The scraper subsystem enriches existing MediaDB records with metadata from external sources. The filesystem scanner owns record creation; scrapers update records that already exist.

Current scraper implementations:

• gamelist.xml imports EmulationStation metadata such as developer, publisher, genre, rating, player count, descriptions, artwork paths, videos, manuals, and ScreenScraper game IDs. It also reads <folder> entries and <game> entries whose path is a directory.
• media-folder imports image paths from EmulationStation-style media/ folders under each system folder. It does not read gamelist.xml, download assets, or write non-image metadata. Indexed directories also match artwork named after themselves, whether or not they collapse to a launch target. Directory image properties use stable (system, path) identities and each successfully completed system atomically replaces its prior directory snapshot, removing stale folder artwork. A force run (re-scrape) also deletes stale media image properties whose paths match the same local media-folder convention and whose replacement file is no longer found.
• mister-docs imports locally installed MiSTer Downloader artwork, manuals, game metadata, and English synopses from docs/<system>/ directories. It is registered only on MiSTer and never downloads source assets itself.
• mister-arcade imports the MiSTer arcade catalog's metadata onto indexed .mra rows, keyed by the MAME set name declared inside each descriptor. It is registered on MiSTer and MiSTeX, reads the catalog those platforms already cache and embed, and runs automatically after arcade indexing.
• libretro-thumbnails downloads box art, screenshots and title screens from the libretro thumbnail server (https://thumbnails.libretro.com), the source RetroArch uses, for systems with a libretro playlist. A file matches its thumbnail by name, after libretro's rule of replacing &*/:`<>?\| with _: an arcade set archive by the game MAME names it (dkong.zip by "Donkey Kong (US set 1)", from pkg/database/arcadenames) and then its own name, a ScummVM launch file by ScummVM's title for its game ID (pkg/database/scummvmnames) and then its own name, any other file by its own name. Failing that it matches by its indexed title slug, ranked by shared region, then clean No-Intro names over dump flags, pre-release builds and TOSEC dates. Images and each playlist's name index (refreshed weekly) are kept under Core's data directory, libretro-thumbnails/<playlist>/<kind>/, and recorded as image-boxart, image-screenshot and image-titleshot media properties. It is the only scraper that downloads, so it never runs automatically after indexing. It is registered on every platform.
• pinup-popper imports PinUP Popper's own table metadata (year, manufacturer, player count, type, category, theme, notes) and wheel, playfield, backglass and flyer images for Pinball media indexed by the PinUP Popper launcher. It is registered only on Windows when a PinUP Popper installation is available, and reads PUPDatabase.db and the emulator media folders in place.

Code Layout

Path	Purpose
pkg/database/scraper/	Shared scrape types (ScrapeOptions, ScrapeUpdate), sentinel helper, and small channel startup helper
pkg/database/scraper/gamelistxml/	EmulationStation gamelist.xml scraper loop, matcher, mapper, and companion-entry handling
pkg/database/scraper/localmedia/	EmulationStation media/ folder image-path importer
pkg/database/scraper/misterdocs/	MiSTer installed artwork/manual database discovery, parsing, matching, and importing
pkg/database/scraper/libretrothumbs/	libretro thumbnail server index, name matching, and image downloader
pkg/database/scraper/pinuppopper/	PinUP Popper library metadata and media-folder image importer for Popper-launched tables
pkg/database/scraper/misterarcade/	MiSTer arcade catalog reader, field mapping, and control-vocabulary normalisation
pkg/database/scraper/mra/	Shared MiSTer arcade descriptor set-name reader
pkg/platforms/mister/arcade_scraper.go	MiSTer/MiSTeX catalog and set-name-cache adapters for mister-arcade
pkg/platforms/shared/esmedia/	Shared EmulationStation media-folder path resolver
pkg/platforms/*	Platform scraper registration through Platform.Scrapers
pkg/database/mediadb/sql_scraper.go	MediaDB scraper read/write helpers, property/blob helpers, and metadata graph queries
pkg/api/methods/media_scrape.go	JSON-RPC scrape start/status/cancel/resume handlers and scraper listing
pkg/api/methods/media_meta.go	JSON-RPC metadata graph lookup for media rows
pkg/api/methods/media_image.go	JSON-RPC image lookup from scraped properties

Registration And API Lifecycle

Platforms expose available scrapers with:

Scrapers(*config.Instance) map[string]platforms.Scraper

platforms.Scraper carries ID, Name, SupportedSystemIDs, optional CustomOpts, and a Scrape callback. The callback receives context, config, platform, filesystem, database, shared scrape options, custom options, and an update channel.

media.scrape looks up the requested scraperId from env.Platform.Scrapers(env.Config), rejects the request if media indexing or another scrape is active, creates an app-scoped cancelable context, starts the scraper in the background, tracks it as a MediaDB background operation, and publishes media.scraping notifications.

media.scrape.status returns the latest in-memory status snapshot plus a fresh scraped-count query. media.scrape.cancel durably cancels current and queued scrape work, including jobs waiting for optimization. A persistence failure is returned as an error, not acknowledged as successful cancellation. media.scrape.resume resumes the shared scrape pauser. Scraping and indexing are mutually exclusive.

Ordinary Jobs After Indexing

A scraper can declare SupportsFillMissing and bind AutoScrapeLaunchers to selected launcher IDs. PinUP Popper and mister-arcade opt in; the remaining scrapers are manual. Both filesystem launchers and custom scanners can supply eligible contributions. Empty, failed, unavailable or unrelated sources do not request a job, and failed/cancelled indexes do not submit their summary.

The sequence is index → existing optimization → ordinary scraping. Successful indexing persists eligible jobs before its final notification. The existing service recovery watcher starts them after optimization releases its write lease; there is no separate automatic-job worker. All jobs use the same scrape pauser, gameplay throttling, progress notifications and executor.

The config-backed operation record holds one current job and an ordered pending list, bounded to 64 total jobs. Identical pending scopes/policies are deduplicated. A new manual request receives a conflict rather than overwriting pending work. Versioned records carry authoritative status; legacy records are readable and upgraded when resumed. Unknown versions fail closed.

Orderly shutdown drains source execution but retains unfinished jobs and run markers. Restart resumes those options and skips committed row work. Advancement is persisted before the previous job's markers are removed. Ordinary source failures allow unrelated pending jobs to continue; persistence failures stop advancement, and database corruption uses existing recovery. A failed final job is not retried indefinitely by the watcher.

Indexing is only a trigger, not a durable parent workflow: use media.scrape.cancel after indexing finishes, not media.generate.cancel. A crash between successful indexing and queue persistence can require another index or manual request. Once accepted into the queue, a job survives restart.

Fill-Missing Policy

Index-triggered jobs only fill missing metadata. A property is missing when no row of that type exists—not when its text is empty or its artwork file has disappeared. Existing exclusive tag values remain; additive tags may gain values. Nothing is automatically replaced or deleted.

These checks and inserts share the existing single/batch scrape transaction. Fill-missing runs reconsider rows carrying a permanent scraper sentinel, so later indexes can fill newly available fields. Per-run markers still skip committed work within a resumed job. Popper orders targets by media path to make shared-title fills deterministic. Manual non-force and force write policies remain unchanged.

Scoped Runs

media.scrape accepts an optional scope: a media ID, an exact system/file path, or a system-bound directory subtree. Existing systems requests retain their behavior.

The handler resolves scope while holding the media-write lease and persists database.ScrapeScope with the operation. An item pins its ID, system, and canonical path; a subtree pins its system and lexical directory boundary. Restart recovery validates the stored scope and never substitutes an entire system. These additive fields live in the existing DBConfig operation JSON, not a new database table.

GetScrapeMedia and GetScopedScrapeMediaIDs query only selected media, titles, and sentinel/run markers. ScrapeOptions.SystemIDs() makes the normalized scope authoritative. All three scrapers propagate it to their selection queries and restrict force cleanup to selected media and their titles. Shared title updates remain visible on sibling ROMs.

Scoped gamelist matching requires a selected path, a verified container/CD reference, or a Companion .slug title reference. It does not use unrestricted slug-only or basename-only fallback, which could make an unrelated source record appear unambiguous after narrowing the index. Container resolution consults full directory context without loading every system row, so selecting one file does not invent a single-game folder.

Scoped progress counts selected media rather than source records. The shared scoped writer validates target identities and deduplicates writes. Source XML/docs files may still require parsing, but selection does not load whole-system media/title lists. Library-wide totalScraped status counts are unchanged.

Run Loop

There is no generic source-record scrape loop. pkg/database/scraper/run.go only provides a small helper for wrapping callback/channel startup. The gamelist.xml implementation owns its loop in GamelistXMLScraper.scrapeLoop.

For each system, the normal loop:

1. Resolves target systems from indexed MediaDB systems and platform launcher paths.
2. Runs ZaparooCompanion processing first. This is a special path; see ZaparooCompanion Entries.
3. Loads eligible indexed titles for slug matching (force=true loads all titles; otherwise titles without the scraper sentinel are loaded).
4. Loads indexed media rows for the system.
5. With force=false, removes media rows that already have sentinel tag scraper.gamelist.xml:scraped from path fallback candidates.
6. Loads gamelist.xml from each ROM root.
7. Resolves each <game> path under its ROM root.
8. Computes the same display-name slug used by the original scraper and prefers that title match.
9. Uses the resolved path to select the concrete Media row for the slug-matched title when possible; otherwise falls back to the first Media row for that title.
10. If slug matching fails, falls back to case-insensitive path matching so otherwise missed records can still scrape.
11. Maps XML fields to media-level tags/properties plus title-level shared tags/properties.
12. Writes metadata through MediaDB.ApplyScrapeResult.
13. Writes the scraper sentinel tag to the selected Media row last inside the same transaction.
14. Emits progress updates and a final done update.

The sentinel tag format is scraper.<id>:scraped, for example scraper.gamelist.xml:scraped. Writing it last is intentional: if a normal record write fails, the transaction rolls back and the missing sentinel leaves that media row eligible for retry.

Force scrapes also persist a run ID and write scraper-run.<id>:<run-id> to each media row completed in that operation. If Core restarts mid-force-scrape, resume reuses that run ID and skips rows already marked for the same run while still refreshing older rows that only had the normal sentinel. Run markers are removed after durable completion/advancement or cancellation. Unfinished and failed work retains its markers; a completed job's markers are not cleared before its durable queue position changes.

In the gamelist loop, per-record write failures are non-fatal: they increment Skipped, emit Err, and continue. A gamelist.xml that exists but cannot be read or parsed (invalid XML, over the size or entry limit) is also non-fatal: it emits Err as a scraper.SourceError naming the file, the run continues with the system's other roots, and the API layer turns the collected source errors into one inbox warning when the job ends. A <game> or <folder> entry whose numeric or boolean field does not parse is dropped on its own and counted as skipped_entries in the load log line; the rest of the file still imports. The size limit is per platform: platforms reporting ResourceConstrained, such as MiSTer, keep a 16 MB limit because every decoded entry is held for the length of the scrape, while other platforms allow 128 MB. A library too large for its device imports through media folders instead. Popper instead treats write failures as fatal so partially committed fill-missing work cannot be reported as successfully completed. Fatal setup/load/database errors end the run with a terminal update unless caused by context cancellation.

Tags And Properties

The DB supports tags/properties at both media and title scope. Normal gamelist.xml scraping writes per-ROM region/lang tags and shared title metadata.

Storage	Scope	Current normal gamelist.xml use
MediaTags	ROM-level variant metadata	region, lang, scraper sentinel
MediaTitleTags	Title-level shared metadata	developer, publisher, year, rating, genre, players, arcadeboard, search (franchise)
MediaTitleProperties	Title-level shared static content	description, XML game ID
MediaProperties	ROM-level static content	artwork paths, video path, manual path for normal gamelist.xml entries

Tag exclusivity is controlled by TagTypes.IsExclusive. Exclusive types replace existing values for that type; additive types accumulate distinct values. The scraper write path groups tags by type and applies that behavior in upsertTags.

Tag rules

Tags are meant to be as stable as practical: a filter written today must mean the same thing next year, on every device, whichever source the metadata came from. So a scraper does not decide what tags exist. Every tag type has a rule in pkg/database/tags/rules.go (TagRules), and every type is exactly one of three kinds:

Kind	Types	What a value must be
Closed list	every type not listed below, including genre, region, lang, arcadeboard, players, input, search	one of the type's values in CanonicalTagDefinitions
Strict format	year, builddate, rating, disc, disctotal, track, season, episode, issue, volume, set, alt, rev, patch, extension, mameparent, user, and the scraper.<id> / scraper-run.<id> bookkeeping types	a value matching the type's rule, such as a year from 1950 to 2099, a real YYYY-MM-DD date, a rating from 0 to 100, a positive disc number, or a listed patch label with an optional version
Free text	developer, publisher, credit only	a company name in the form NormalizeCompanyName produces: lower-case words joined by single dashes

Company names are the only free text because they are the only values that cannot be listed. Adding a free-text type is a design change, not a scraper detail; TestFreeTextTypesAreOnlyCompanyNames exists to make that visible.

What this means for a scraper:

• Map or drop. A scraper maps its source's wording onto an existing canonical value, through a table in its package or a shared lookup (tags.LookupRegionWord, LookupLanguageWord, LookupArcadeBoard, LookupFranchise). A source value with no mapping is dropped, never stored as it was written. Genres follow the GameDataBase taxonomy (shmup:v, action:platformer, parlor:pinball), series are search:franchise:<name>, and boards are arcadeboard:<vendor>:<board>.
• Say what was dropped. Record each dropped value with scraper.UnmappedValues.Note and call LogSummary once at the end of a run. The summary is logged at info with counts and examples per tag type, so the mapping tables can be grown from what real sources contain.
• No labels on shared values. A closed or format value is one row shared by every title that carries it, so it has no per-source display name. Only free-text tags keep the source's spelling as a label (t-and-e-soft labelled T&E Soft).
• Adding a value means adding a TagValue constant to tag_values.go and listing it under its type in CanonicalTagDefinitions, then mapping to it. Tests assert that every listed value satisfies its type's rule and that every declared tag type has a rule.

MediaDB enforces the rules on every write path — scrape writes, UpsertMediaTags/UpsertMediaTitleTags, the scanner's staging, user flags and deck membership, and the legacy InsertTag/InsertTagType — so a value that slips past a scraper is refused and logged at warn rather than stored. When the vocabulary changes between releases, the next index run's seeding pass removes stored types and values the new rules refuse, together with every link to them (sqlPruneOffVocabularyTags).

Each scraper's tests call scrapertest.RequireValidWrite on every write they build, and the MiSTer arcade catalog tests fail when the pinned catalog snapshot holds a category, platform or series that nothing maps, so refreshing the snapshot forces the new values to be curated.

Property rows are keyed by entity and property type tag. Re-scraping the same property type updates the row in place and preserves row DBID.

Path-backed properties persist their text path and optional BlobDBID; the property tables do not persist the ContentType computed by the mapper for path values. Blob-backed properties expose content type from MediaBlobs. API responses infer path-backed content type and extension from the stored path when DB content type is empty.

Normal gamelist artwork is media-level so regional or language variants can carry different image paths while sharing one title. media.image checks media-level properties before title-level properties. If an older scrape left title-level artwork behind, run a force scrape to refresh media-level artwork; no migration removes old title properties.

Media-level Sentinel Invariant

Normal gamelist.xml entries prefer slug/title matching, then use path matching to select the concrete Media row when possible. If no slug match exists, path-only fallback can still select a Media row. The sentinel is written to the same Media row that receives ROM-level tags such as region and lang plus ROM-level file properties such as artwork.

Title metadata remains shared by MediaTitleDBID, so multiple ROM variants can write the same title-level tags/properties. Rewrites are idempotent: exclusive title tags replace same-type values, additive tags are inserted-or-ignored, and properties upsert by type. Media-level properties upsert per concrete Media row, preventing regional artwork variants from overwriting each other.

gamelist.xml Behavior

GamelistXMLScraper scans each system ROM root for gamelist.xml. Regular <game> entries are resolved to absolute paths under the system ROM root. The scraper first matches the entry to an existing title by the original display-name slug behavior, then uses the resolved path to choose the concrete Media row for that title when possible. If slug matching finds a title but the path does not identify a Media row for that title, only title-level metadata is written. If no known title slug exists, it falls back to case-insensitive path matching. Scrapers do not create Media or MediaTitle rows.

Path handling for <game><path> stays strict:

Input	Behavior
./relative or relative	Resolved under the system ROM root and rejected if it escapes that root
~/...	Resolved under the current user's home directory, then rejected unless still under the system ROM root
Absolute path	Cleaned and rejected unless under the system ROM root

Asset path handling for artwork/video/manual uses the same root-bound behavior by default. On MiSTer and MiSTeX only, absolute or ~/... asset paths may also resolve under platform root directories from RootDirs(cfg), covering SD, USB, CIFS, network, and configured index roots. This applies only to file-backed asset fields; game paths remain bound to the ROM root. Path traversal outside the ROM root or approved platform roots is rejected. The MiSTer arcade set-name fallback below can also interpret a ROM path as an identity without accessing that path; it does not broaden asset access.

Zip-as-directory paths are supported for matching XML entries such as ./Japan/Game.zip to indexed media stored under that zip path, while nested artwork paths such as ./media/images/Japan/Game.png remain resolved as asset paths.

Source fields are cleaned before mapping: HTML entities are unescaped, tab/newline/carriage-return characters become spaces, and surrounding whitespace is trimmed.

Field Mapping

ES field	Destination	Notes
lang	MediaTags: lang	CSV split; each code or name mapped with tags.LookupLanguageWord, additive
region	MediaTags: region	CSV split; ScreenScraper codes (wor, eur, jpn, asi, sp) mapped locally, anything else with tags.LookupRegionWord; ss is not a region and is dropped. Additive
developer	MediaTitleTags: developer	Exclusive
publisher	MediaTitleTags: publisher	Exclusive
releasedate	MediaTitleTags: year	First four characters when present
rating	MediaTitleTags: rating	Normalized from 0..1 style ratings to 0..100 text
genre, genreid	MediaTitleTags: genre	Additive. The text goes through the shared ScreenScraper genre table (scraper/ssgenre), compound values split and each part mapped, with the parent written beside a subgenre. Recalbox's numeric genreid (its Genres.h enum) is mapped first when present
players	MediaTitleTags: players	Highest player count from ranges/lists, when it is a listed count
arcadesystemname	MediaTitleTags: arcadeboard	Exclusive, through tags.LookupArcadeBoard
family	MediaTitleTags: search	franchise:<name>, through tags.LookupFranchise
desc	MediaTitleProperties: property:description	Plain text
ScreenScraper game ID	MediaTitleProperties: property:xml-game-id	From XML attribute or element value
image	MediaProperties: property:image-image	XML path or filesystem fallback
thumbnail	MediaProperties: property:image-thumbnail	Cover/thumbnail path in most ES forks
boxart2d	MediaProperties: property:image-boxart	XML path or filesystem fallback
boxart3d	MediaProperties: property:image-boxart3d	XML path or filesystem fallback
screenshot	MediaProperties: property:image-screenshot	XML path or filesystem fallback
video	MediaProperties: property:video	Filesystem path
marquee	MediaProperties: property:image-marquee	XML path or filesystem fallback
logo / wheel	MediaProperties: property:image-wheel	logo takes priority over wheel; XML path or filesystem fallback
fanart	MediaProperties: property:image-fanart	XML path or filesystem fallback
titlescreen / titleshot	MediaProperties: property:image-titleshot	titlescreen takes priority over titleshot; XML path or filesystem fallback
map	MediaProperties: property:image-map	XML path or filesystem fallback
manual	MediaProperties: property:manual	PDF path

Filesystem fallback searches known subdirectories under <systemRootPath>/media/ when an XML path is absent. For games in subfolders, it searches the mirrored ROM-relative path before the flat filename; for example ./Japan/Game.nes checks media/images/Japan/Game.png before media/images/Game.png. Side/back box art are filesystem-fallback only.

Directory Entries

Both <folder> entries and <game> entries whose <path> resolves to a directory are matched to the single media row that directory collapses to, using the same rule browse uses to show a disc folder as one game. A directory qualifies when it has no nested media and its direct contents are one file, one .m3u plus its discs, one .cue plus its companion tracks, or at least two supported disc-image files that all share one positive MediaTitleDBID. Supported shared-title disc extensions are .cue, .chd, .iso, .bin, .img, and .pbp; mixed title IDs or other extensions remain ambiguous.

This covers the two common EmulationStation layouts for multi-disc games: a <folder> entry describing a per-game folder, and the ES-DE convention of naming that folder with a ROM extension so it reads as one game and gets an ordinary <game> entry.

<folder> entries carry a smaller field set than games: name, desc, image, thumbnail, video, and marquee. They are processed after that file's <game> entries, so a real game entry pointing at the same media row always wins. Folders that do not collapse are skipped, because there is no row to carry their metadata; ordinary collections stay plain browseable directories.

A directory entry only ever resolves through the container rule, whichever kind it is, so use a <folder> entry only for a directory that collapses. A <game> entry naming a directory that holds media in subdirectories no longer matches the single row underneath it, because resolving a directory by scanning for indexed paths beneath it is only unambiguous until an earlier entry has claimed one of them, and switching it to a <folder> entry does not help: that is skipped for the same reason. For any directory that does not collapse, point the <game> entry at the media file itself.

Artwork for a directory entry is looked up under the directory's own name, matching where EmulationStation stores art for a folder it shows as one game. Only a disc extension (.cue, .m3u, .chd, .iso, .bin, .img, .pbp) is stripped from that name before the search; any other dot is treated as part of the folder name, so a folder called Sonic 3.0 does not pick up Sonic 3's artwork.

media-folder Directory Artwork

For every present indexed media row, media-folder considers each ancestor directory below its system ROM root. It searches the usual artwork categories under <systemRoot>/media/, first at the mirrored ROM-relative location and then by flat directory basename. For example, directory RPGs/Final Fantasy VII checks media/boxart/RPGs/Final Fantasy VII.png before media/boxart/Final Fantasy VII.png. System root itself is excluded. Directories are also matched in media/folders/ and media/folder/, which are searched first for the image type and are never used for a game file's artwork.

Folder artwork does not change browse structure. Arbitrary collections remain type: "directory"; only existing container rules add launch metadata. media.browse reports hasCover: true when a directory image property exists for one of that entry's systems, and media.image accepts the directory's (system, path) to return it.

Directory properties are collected in memory for one system and committed as a complete snapshot only after all its directories finish. Cancellation or failure before replacement preserves that system's previous snapshot. A completed empty snapshot removes stale directory rows. Rebuilding BrowseDirs does not affect these properties because they are keyed by stable system DBID and canonical path rather than browse-cache DBID.

By default, only <ROM root>/gamelist.xml files are loaded. Nested files such as <ROM root>/Japan/gamelist.xml are not read.

An additional metadata bundle can be configured independently of ROM storage:

[scraper.gamelist_xml]
custom_path = "/path/to/gamelists"

For each indexed system, the scraper also checks <custom_path>/<system ID>/gamelist.xml. Game paths in this file resolve against the system's first ROM root; asset paths resolve against the custom system directory. Custom bundle image references are optional: only files present during scraping are stored, and missing references fall back to media/ artwork under the custom directory and then the system's ROM roots. Run the scraper again after installing more bundle artwork. Regular ROM-root gamelists are processed first and take precedence over matching custom entries.

Custom gamelists enrich existing indexed records; they do not create systems, titles, or media rows. For systems that index virtual or non-file-backed entries (where the stored media path does not correspond to a real file), <path> must match the exact path the indexer stored for that media row.

gamelist.xml deliberately does not scrape user-state fields such as favorite, hidden, or kidgame. It also does not overwrite filename-parser-owned fields such as disc and track.

MiSTer Arcade Gamelists

MiSTer indexes launchable .mra descriptors under _Arcade, not MAME ROM ZIPs. A gamelist authored by Skraper against pacman.zip therefore cannot identify Pac-Man (Midway).mra by its filesystem path alone.

On MiSTer and MiSTeX, the gamelist.xml scraper can bridge these identities using the <setname> stored inside each indexed MRA:

• A <game><path> ending in .zip or .7z, or containing a bare set name, supplies the set-name key. Keys are case-insensitive; they contain letters, digits, underscores, or hyphens, up to 128 characters.
• The matching MRA must be live and uniquely identified within the system currently being scraped. Already-scraped MRAs still count when checking uniqueness; force and resume cannot make a duplicate set appear unique.
• Multiple MRAs with the same set name are skipped, including alternate-core variants and duplicates sharing one title. Use an exact MRA path to choose a variant instead of relying on ROM-set matching.
• Entries that name the row directly take precedence: an indexed path, or a slug match the entry's own path confirms. A set name outranks a record that only guessed the row from its title, in either XML order, because arcade clone sets routinely share one display name. Unknown set names retain existing slug matching. A known ambiguous set does not fall back to guessing by title.
• A unique match receives title metadata and media-level artwork. Artwork filename fallback uses the source set name in any supported artwork extension, such as media/images/pacman.png or media/images/pacman.jpg, rather than the MRA's display filename.

Exporting A Scraper Bundle

To keep metadata outside _Arcade, export or copy the gamelist and its referenced images together into a custom bundle:

/media/fat/metadata/
└── Arcade/
    ├── gamelist.xml
    └── images/
        └── pacman.png

[scraper.gamelist_xml]
custom_path = "/media/fat/metadata"

Example Arcade/gamelist.xml:

<gameList>
  <game>
    <path>./pacman.zip</path>
    <name>Pac-Man</name>
    <desc>Metadata exported by your scraper.</desc>
    <image>./images/pacman.png</image>
  </game>
</gameList>

Index the MRAs first, then run gamelist.xml for Arcade. For a granular arcade system such as CPS1, use a CPS1 bundle directory and scrape that indexed system; those systems are classified out of _Arcade rather than scanned from a folder of their own, so an installed bundle is what makes them scrapable at all. Existing arcade classification determines system membership; the scraper neither creates MRA entries nor guesses membership from catalog titles.

A gamelist in _Arcade also supports these ROM/set-name references. Core does not automatically discover arbitrary nested gamelists: place one in a configured ROM root, or use the custom bundle layout above.

MiSTer Companion ZapScraper

On MiSTer and MiSTeX only, Arcade also auto-discovers games/mame/gamelist.xml beside each configured _Arcade root (<root>/games/mame/gamelist.xml), with no configuration needed. This is where MiSTer Companion's ZapScraper writes its Arcade output, keyed by MAME ZIP name rather than by MRA path, with artwork under games/mame/media/. It is matched purely through the <setname> fallback above: like a foreign or sibling ROM ZIP path in an ordinary or custom-bundle gamelist, its entries do not resolve under _Arcade. Its images are optional, only mapped when present at scrape time, the same as a custom bundle's. It does not apply to granular arcade systems (CPS1 etc.), which have no games/mame root of their own; use a custom bundle for those. Among competing set-name matches for the same MRA, an _Arcade gamelist's or a configured custom bundle's own match wins over one discovered here. Like any set name, a games/mame match still outranks a slug-only guess from another source for that row, per the rule above — it is only lowest precedence among identity matches, not below every write a higher-priority source could make.

Regular set-name entries may retain absolute or sibling ROM ZIP paths from the scraper machine, including Windows paths. Core extracts only the basename identity; it never opens or launches those source ZIP paths. Image, video, and manual references keep the existing asset-root restrictions. Custom images must exist at scrape time. Relative Companion ZIP child references also support unique set-name matching; existing Companion path validation and parent metadata behavior remain unchanged.

Unreadable or malformed MRA descriptors, and those repeating <setname> in their header, are not identity sources. Only the descriptor header up to its first <rom> element is read, so the embedded ROM payload of a large MRA costs nothing and does not disqualify it. No MRA, ROM archive, or launcher configuration is rewritten. AmigaVision games.txt and demos.txt integration is separate from arcade matching.

MiSTer Installed Docs Databases

The MiSTer-only mister-docs scraper indexes assets already installed by MiSTer Downloader or Update All. Downloader remains responsible for downloading, verifying, updating, and placing third-party content. Core performs no online database enumeration and does not edit Downloader configuration.

Example artwork sources:

[chipster6502/artworkdb-snes]
db_url = https://raw.githubusercontent.com/chipster6502/artworkdb-nintendo-consoles/db/snes_box2d.json.zip

[chipster6502/artworkdb-genesis]
db_url = https://raw.githubusercontent.com/chipster6502/artworkdb-sega/db/genesis_box2d.json.zip

[chipster6502/artworkdb-arcade]
db_url = https://raw.githubusercontent.com/chipster6502/artworkdb-arcade/db/arcade_box2d.json.zip

Game manuals can be selected through Update All's Game Manuals (EN) settings or installed through compatible Downloader database sections. These collections are large; Core intentionally does not mirror or bulk-download them.

Discovery

Core derives docs roots from MiSTer's configured SD, USB, network/CIFS, and custom index roots, and also probes /media/usb6 and /media/usb7, which artwork packs may be installed to but MiSTer's games-folder list does not reach. It recognizes content by installed format rather than repository name, following the MiSTer Artwork Pack format:

• Artwork: docs/<System>/Artwork/ holding one <key>.jpg per game, normally with an index.tsv beside them that maps every known dump to its key. <System> is the MiSTer games/ folder name. A directory with images and no index still resolves games filed under their exact key.
• Optional title metadata: gameinfo.tsv beside the images. Games it lists without an image still receive their metadata.
• Optional description: synopsis_<lang>.tsv beside the images. Which languages a pack ships varies per system, so Core reads whichever files exist and picks the first match from media.default_langs, then English, then the first available language.
• Manuals: direct PDF files in a child directory whose name contains manual, for example docs/SNES/Manuals/ or docs/NES/Famicom Disk System Manuals/.

This format-based discovery means future compatible databases need no Core update. Run mister-docs again after Downloader installs or updates content. Normal runs rescan installed records idempotently; force runs additionally delete stale box-art/manual properties whose old paths are proven to belong to a discovered MiSTer docs convention.

Metadata files are treated as untrusted input. Core bounds their size and record count, rejects symlink/path escapes and non-regular assets, skips ambiguous matches, and continues past malformed optional sources where possible.

Matching And Fields

index.tsv maps catalogued dump names to artwork keys: No-Intro names for cartridges, Redump names for CD systems, and MAME parent setnames for arcade. Core resolves each pack entry to installed media in the pack format's order, stopping at the first hit:

1. The catalogued name as a media basename, at media scope.
2. For arcade, the <setname> inside each installed .mra, at media scope. MRA filenames are titles, so the setname is the only handle an arcade key has; Core reads it from the MRA only for systems that have an arcade artwork source.
3. A media basename whose trailing parenthesised tag is itself a pack key, such as Shock Troopers (set 1) (shocktro), at media scope.
4. A unique bare-title match, at title scope. This step is skipped when the stripped title is not unique among the pack's keys or among the library's titles, and it is only available to index rows: images the index does not mention resolve by exact name alone.

CRC and size columns are not used because hashing every installed ROM would impose substantial MiSTer I/O; the pack format treats that step as optional.

Source	Destination
Artwork image	property:image-boxart at media scope for exact matches, title scope for unique slug fallback
gameinfo.tsv year	title tag year
gameinfo.tsv genre	title tags genre, through the shared ScreenScraper genre table; a hierarchy such as Shoot'em Up / Vertical writes both shmup:v and shmup
gameinfo.tsv developer	title tag developer
gameinfo.tsv players	title tag players using highest numeric value, when it is a listed count
synopsis_<lang>.tsv synopsis	title property property:description
Manual PDF	title property property:manual

Manual filenames are matched with the same game-title slug normalization used by MediaDB, including leading/trailing article handling. Basenames with no matching title, or whose slug collision remains ambiguous after normalized-name matching, are left unmatched. Category-like names such as system manuals, overlays, or charts are not filtered separately.

Base-system sources enrich their variants, such as SNES MSU-1, Genesis MSU, and the granular arcade systems. On top of that, Core applies the pack format's shared-catalogue rules: Game Boy and Game Boy Color each fall back to the other, Super Game Boy reads both, and FDS falls back to NES but never the reverse. Systems the pack catalogues separately do not fill each other's gaps, so SG-1000 never receives ColecoVision art and Neo Geo Pocket Color never receives Neo Geo Pocket art, even though the general system fallbacks allow it.

If multiple docs roots provide the same property, MiSTer root order decides which source wins. As with other scrapers, running a different scraper later may replace exclusive tags or same-type properties.

ZaparooCompanion Entries

gamelist.xml has a special path for entries marked with source="ZaparooCompanion" as either a source attribute or <source> element.

Companion records are split into:

• Parent entries: have an ID attribute and no path. They carry shared title metadata.
• Child entries: have parentid and path. They reference parent metadata.

Child matching:

• Paths ending in .slug match an existing title by slug, then use the first Media row for that title as the write target.
• Other child paths first try an exact case-insensitive media path lookup.
• If exact lookup fails, the scraper falls back to filename suffix matching with FindMediaBySystemAndPathSuffix.
• Ambiguous suffix matches are skipped instead of updating multiple same-basename media rows.

For matched children, parent metadata is written onto the child title, child region and lang are written to the child Media row as media-level tags, and the scraper sentinel is written to that child Media row. These writes use ApplyScrapeResult, so title metadata, child tags, and the sentinel are committed together.

Current caveats:

• Companion processing still runs before normal title filtering.
• With force=false, child media rows that already have the scraper.gamelist.xml:scraped sentinel are skipped.
• Companion processed/matched/skipped counts contribute to run counters, but companion entries do not have a separate total in status updates.

These caveats document current behavior, not necessarily desired long-term behavior.

API Surface

JSON-RPC methods:

Method	Purpose
scrapers	Lists registered scrapers with ID, name, and supported systems
media.scrape	Starts a scraper run as a background operation
media.scrape.status	Returns latest in-memory scraper status plus current DB scraped count
media.scrape.cancel	Cancels the active scraper run
media.scrape.resume	Resumes a paused scraper run
media.meta	Returns tags and metadata-only properties for one or more media rows and their titles
media.image	Returns the best matching image property as base64 data for one media row, including thumbnail art
media.clean.orphans	Removes missing media rows and orphaned related data

media.scrape params:

{
  "scraperId": "gamelist.xml",
  "systems": ["snes", "nes"],
  "force": false
}

Progress is queryable with media.scrape.status and broadcast as media.scraping notifications:

{
  "scraperId": "gamelist.xml",
  "systemId": "snes",
  "processed": 42,
  "total": 100,
  "matched": 38,
  "skipped": 4,
  "totalScraped": 1000,
  "scraping": true,
  "done": false,
  "paused": false,
  "state": "running",
  "totalSteps": 2,
  "currentStep": 1,
  "currentStepDisplay": "Super Nintendo Entertainment System",
  "currentSystem": {
    "systemId": "snes",
    "systemName": "Super Nintendo Entertainment System",
    "processed": 42,
    "total": 100,
    "matched": 38,
    "skipped": 4
  }
}

totalScraped is derived from scraper sentinel tags in the database, not from the current run's matched count. Existing flat fields stay for compatibility; new UIs should use currentSystem for current-system progress and totalSteps/currentStep/currentStepDisplay for whole-run system-step progress.

Only one scraper can run at a time, and scraping is mutually exclusive with media indexing.

media.meta returns the metadata graph for media rows: media-level tags and properties, title-level tags and properties, and stored system identity. Single requests accept mediaId or system/path and keep the single-response shape; batch requests use items and return per-item results. Binary property bytes are not included; clients should use media.image for image data. A system/path request for a directory resolves only when its direct contents collapse to one logical launch target, which also covers zip containers on platforms that treat zips as directories.

media.image accepts one media ref plus image type preferences such as image, boxart, boxart3d, screenshot, wheel, titleshot, map, marquee, and fanart. These resolve to canonical image property tags; for example boxart becomes property:image-boxart and image becomes property:image-image. Media-level properties are preferred over title-level properties for the same type. On platforms that treat zips as directories, zip container aliases are checked as media-level fallbacks, so artwork attached to a direct single-game target or its zip can be found from either path. For stale image properties in these canonical tags, such as missing file paths for property:image-boxart or property:image-image, media.image logs the stale property in memory only and does not delete DB rows; lookup falls through to the next available source.

Useful Focused Tests

go test ./pkg/database/scraper/...
go test ./pkg/database/mediadb/ -run 'Scrape|Property|Blob|Sentinel|MediaImage'
go test ./pkg/api/methods/ -run 'Scrape|MediaImage|MediaMeta'
go test ./pkg/platforms/mister/ -run 'Arcade'

MiSTer Arcade Catalog Behavior

MiSTer and MiSTeX download, verify and cache the upstream MiSTer-devel/ArcadeDatabase_MiSTer catalog under Core's assets directory, and ARM builds embed a copy as a fallback. mister-arcade reads that catalog and joins it to indexed .mra rows.

An MRA filename is a display title, so the join key is the <setname> inside the descriptor. MiSTer already reads every set name while classifying granular arcade systems and keeps the result in a size/mtime-validated cache, so the platform hands the scraper a lookup into that cache and only descriptors it cannot answer are read. MiSTeX has no granular classification, so every descriptor it indexes is an Arcade row and is read on demand. Reads are bounded to the descriptor header and stop at the ROM payload, so a multi-megabyte MRA costs nothing.

The scraper is registered for Arcade and, on MiSTer, for each granular arcade system. It binds those launchers with AutoScrapeLaunchers, so an index that walked _Arcade queues a fill-missing job without any user action. A missing or unreadable catalog fails the run rather than writing an empty result; because fill-missing runs reconsider rows carrying the permanent sentinel, a later index fills them once the catalog is cached.

A descriptor whose set name the catalog does not list is counted as skipped. The catalog omits hundreds of sets, nearly all of them under _Arcade/_alternatives.

Field Mapping

Shared facts about the game go to the title, so every regional variant carries them. Facts about the individual romset go to the media row.

Catalog column	Destination	Notes
year	MediaTitleTags: year	Exclusive; four digits only
manufacturer	MediaTitleTags: developer	Exclusive, company-name normalized. credit is the union query type, so a value written to developer answers both developer: and credit: filters
category	MediaTitleTags: genre	Additive; mapped by a table covering every category in the bundled catalog, with the parent beside a subgenre (Shooter - Flying Vertical → shmup:v, shmup)
series, parent_title	MediaTitleTags: search	franchise:<name> through tags.LookupFranchise; parent_title is used only for a game with no series and only when it names a listed franchise
platform	MediaTitleTags: arcadeboard	Exclusive, through tags.LookupArcadeBoard
players	MediaTitleTags: players	Additive; a range writes every listed count in it, plus simultaneous or alt
move_inputs, special_controls	MediaTitleTags: input	Additive; normalized through a fixed phrase table
num_buttons	MediaTitleTags: input as buttons:N	Zero buttons writes nothing; a count the input list lacks is dropped
resolution	MediaTitleTags: video as 15khz/31khz
rotation	MediaTitleTags: search as tate:cw/tate:ccw	A horizontal monitor is the default and writes nothing
flip	MediaTitleTags: search as keyword:flip
homebrew	MediaTitleTags: release as homebrew
setname	MediaProperties: property:mame-setname	The romset identity, so callers need not re-read descriptors
region	MediaTags: region	Additive; multi-region cells split. A cell reading bootleg states a provenance and becomes unlicensed:bootleg
version	MediaTags, routed	YYMMDD → builddate; Rev A → rev:a; Set 1 → set:1; Prototype → unfinished:proto; bootleg/hack → unlicensed; a protection chip family → protection
alternative	MediaTags: alt
bootleg	MediaTags: unlicensed as bootleg

The catalog's name is not imported: arcade titles come from the descriptor filename, which the indexer owns. linebreak1 and linebreak2 are upstream layout spacers and are always empty.

Catalog text is treated as untrusted: values are HTML-unescaped, whitespace-collapsed, and checked against the several spellings upstream uses for "not applicable". The control columns are free text with inconsistent case and two separators; each phrase the catalog uses is listed explicitly. A phrase that names a control family without the detail its canonical value needs — a 2-way joystick with no axis, a bare stick, positional with no position count — is logged at debug and dropped rather than resolved to the nearest guess.

Every category, platform and series value in the catalog either maps onto the tag vocabulary or sits in an explicit, commented skip set in misterarcade/vocabulary.go (a BIOS entry, a CPU name rather than a board, a licence line rather than a series). The catalog itself is downloaded at build time from a moving upstream, so the coverage tests read a pinned snapshot of its distinct values, misterarcade/testdata/arcade_catalog_values.tsv, and fail on any value that is in neither, and on any skip entry that has since become mappable. When upstream adds values, devices drop them and name them in the scraper's unmapped-values log line; refreshing the snapshot from the new catalog makes the tests list exactly what needs curating. Board names map through tags.LookupArcadeBoard, which the gamelist.xml scraper shares, so both sources agree on capcom:cps2 however they spell it.

Writing media-level tags of scanner-owned types changes the MediaIdentity fingerprint for rows that lacked them, as gamelist.xml's media-level region/lang writes already do. property:mame-setname is not scanner-owned and never affects it.

How much a fill-missing run changes depends on the type. An exclusive type is written only when the row has no value of that type at all, so the MRA filename parser's build date, revision and bootleg normally stand. An additive type is always written, so a row can gain a second value of a type it already had. The two that move in practice:

• alt is new to most rows. The catalog marks a majority of its romsets as alternates and an MRA filename almost never says so, so a first run re-fingerprints the bulk of an arcade library — measured against a real _Arcade corpus, 1748 of 3274 rows, of which 5 had a filename that already said alt.
• region is additive, so a row whose filename states a territory can gain the catalog's broader value beside it. On the same corpus 43 rows ended up carrying world next to a specific territory, because the catalog describes the romset's release while the filename describes which dump this is.

PinUP Popper Behavior

The PinUP Popper launcher indexes tables as popper://<GameID>/<name> virtual paths, so the scraper needs no name matching: it reads the Popper library once, resolves each Pinball media row's GameID to its Games row and emulator, and writes the result. Rows already carrying scraper.pinup-popper:scraped are left out unless the scrape is forced; rows whose GameID no longer exists in Popper are counted as skipped. Images are looked up as <Emulators.DirMedia>\<screen>\<GameName>.<png|jpg|jpeg>, falling back to POPMedia\<EmuName> when the emulator has no media directory. Videos, audio and the remaining screens are not imported.

Popper field	Destination	Notes
GameYear	MediaTitleTags: year	Exclusive, when greater than zero
Manufact	MediaTitleTags: developer	Exclusive, company-name normalized
NumPlayers	MediaTitleTags: players	When it is a listed count
(every table)	MediaTitleTags: genre	parlor:pinball and parlor: every Popper table is a pinball table
GameTheme	MediaTitleTags: search	feature:<name> only when a theme names a listed feature exactly; other themes are dropped
GameType, Category	not imported	Table technology and Popper's own grouping, not genres
Notes	MediaTitleProperties: description	Whitespace collapsed
Wheel image	MediaProperties: image-wheel
PlayField image	MediaProperties: image-screenshot
BackGlass image	MediaProperties: image-marquee	The backglass is the pinball counterpart of a marquee
GameInfo image	MediaProperties: image-image	Flyer or info card

Frontend Ownership Boundary

Popper uses one application-local pinup.Integration object for launch attempts, process observations and worker shutdown. Windows supplies that object through existing Launcher.Scanner, Launch, Kill, availability and startup/shutdown hooks. Registration does not start Popper or polling workers. This is the pattern for later frontend integrations: keep mutable state local and connect existing callbacks, rather than requiring a shared lifecycle framework.

Table stop uses Popper's frontend exit command and bounded observation/retries, including its loading period. Core does not take ownership of a foreign emulator process for generic child reaping or tree killing. Popper does not provide a session-fenced stop token: an external frontend replacement can race an unconditional exit request. Local attempt guards prevent stale Core retries/publications but cannot make that frontend protocol globally atomic.