Compares the source file's mtime against the sidecar's; if the
sidecar is newer, the conversion is skipped and counted into
stats.skipped. Re-running --upgrade-extract on a fully-converted
tree is now nearly free (just an mtime check per file).
asset_extract --upgrade-extract Data/expansions/wotlk
Walking ... (first run)
JSON (DBC→JSON) : 240 ok
asset_extract --upgrade-extract Data/expansions/wotlk
Walking ... (second run, all sidecars up to date)
up-to-date (skip) : 240
JSON (DBC→JSON) : 0 ok
emitOpenFormats() takes a new optional 'incremental' flag (default
false to preserve the asset_extract main-loop's overwrite behavior
since fresh extraction always wants new sidecars).
Verified end-to-end with a hand-built DBC: first run converts,
second run reports 'up-to-date (skip): 1'.
emitOpenFormats now takes an optional threadCount parameter (0 =
auto). The asset_extract --upgrade-extract path forwards opts.threads
so users can override the auto-detect when running on a CI machine
with limited cores or wanting deterministic timing.
Also wraps the upgrade pass with a chrono timer and prints elapsed
seconds so the parallelization payoff is visible at a glance:
asset_extract --upgrade-extract Data/expansions/wotlk --threads 8
Walking Data/expansions/wotlk for open-format upgrades...
elapsed : 47.2 s
PNG (BLP→PNG) : 12340 ok
...
Verified end-to-end: --threads 2 on 5 hand-built DBCs converts all
5 in well under a second.
Conversions are CPU-bound (BLP decode, M2/WMO parse, WOM/WOB
serialize) so the serial walk leaves cores idle. Now collects
every job into a vector during the directory walk, then dispatches
across hardware_concurrency() workers via an atomic next-index
queue. Stats use atomics to avoid the per-job mutex.
Expected ~5-8x speedup for full-tree --upgrade-extract on a
modern desktop. Existing test_open_format_emitter still passes
(it exercises both single-file emit*From* helpers and the parallel
emitOpenFormats walker).
Standalone post-extract pass: walks an existing extracted asset
tree and writes open-format sidecars in place, without re-running
MPQ extraction.
asset_extract --upgrade-extract Data/expansions/wotlk
Lets users with old extractions opt into the open-format pipeline
without losing the extracted state. Implies --emit-open if no
individual --emit-* flag is set.
Verified end-to-end: created a hand-built DBC in a temp dir, ran
--upgrade-extract, observed test.json appear with correct
metadata. Servers continue to read .dbc from manifest; the
runtime client picks up the new .json sidecar via the existing
pickup path.
Final piece of the open-format emit pipeline:
--emit-terrain foo.adt → foo.whm + foo.wot + foo.woc
With this, --emit-open now produces a fully open-format zone
alongside every Blizzard MPQ extraction:
BLP → PNG (textures)
DBC → JSON (data tables)
M2 → WOM (models, with skin merge)
WMO → WOB (buildings, with group merge)
ADT → WHM/WOT (terrain heights + metadata)
→ WOC (collision mesh derived from heights)
Originals stay on disk and indexed by manifest.json so private
servers continue to load proprietary formats; wowee runtime/editor
read the open formats directly. One extraction now feeds both
audiences with no separate conversion pass.
Implementation:
- Inline WHM+WOT writer in open_format_emitter.cpp (mirrors the
editor's WoweeTerrain::exportOpen but without the PNG-preview /
normal-map deps so the extractor stays editor-independent).
- Tile coords (x,y) parsed from <map>_<x>_<y>.adt filename.
- Collision mesh derived via WoweeCollisionBuilder::fromTerrain
(terrain triangles only — WMO collision overlays would need
asset manager and aren't worth the extractor complexity).
Extends asset_extract with two more open-format emitters:
--emit-wom foo.m2 (+ foo00.skin) → foo.wom
--emit-wob foo.wmo (+ foo_NNN.wmo groups) → foo.wob
--emit-open now also turns these on
Originals are preserved so private servers still load .m2/.wmo
through the manifest path; the wowee runtime/editor pick up the
.wom/.wob next to them via the existing open-format search rules.
Implementation:
- New WoweeModelLoader::fromM2Bytes(m2Data, skinData) shares the
conversion body with fromM2(path, am) via a static helper
(convertM2ToWom). Lets the extractor convert without standing
up an AssetManager.
- fromM2(path, am) moved to a separate translation unit
(wowee_model_fromm2.cpp) so asset_extract doesn't have to
link the AssetManager dependency.
- WoweeBuildingLoader::fromWMO already takes a WMOModel directly,
so emitWobFromWmo just needs to read root + group files and
call save().
- Group sub-files (<base>_NNN.wmo) are skipped during the walk
since they're merged into the root WMO.
The asset_extract tool now optionally writes wowee open-format
copies next to each extracted proprietary file:
--emit-png foo.blp → foo.png
--emit-json-dbc foo.dbc → foo.json
--emit-open shortcut for both
Originals are left untouched, so private servers (AzerothCore,
TrinityCore) that load from the manifest's .blp/.dbc paths
continue to work unchanged. The wowee runtime / editor can now
consume the open formats directly without an extra conversion pass.
Implementation:
- New tools/asset_extract/open_format_emitter.{hpp,cpp} encapsulates
the post-extract walk + per-file conversion.
- BLP→PNG uses BLPLoader::load + stbi_write_png with the same
dimension/buffer-size sanity guards the editor's texture exporter
applies.
- DBC→JSON mirrors the editor's DBCExporter::exportAsJson schema
(string/float/uint heuristic) so the runtime DBC overlay loader
can consume the output drop-in.
- VkTexture::isValid() now checks both image AND sampler handles. Previously
it only checked the image, so a texture with a valid image but NULL sampler
would pass validation and get bound to a descriptor set. On MoltenVK (macOS)
this renders as pink/magenta boxes; the fallback white texture is now
correctly used instead.
- Fix fs::path to std::string implicit conversion in asset extractor that
broke the Windows (MSYS2/clang) CI build.
- mpq and locale finding is now case insensitive
- improve extraction order and support more patches
- unified much of the mpq logic for all expansions
- return a list of ordered paths for loading
Drop PROT_EXEC from warden module mmap when using Unicorn emulation
(not needed — module image is copied into emulator address space). Use
MAP_JIT on macOS for the native fallback path.
Add --listfile option to asset_extract and SFileAddListFileEntries
support for resolving unnamed MPQ hash table entries from external
listfiles.
WoW archives contain mixed-case variants of the same path (e.g.,
ARMLOWERTEXTURE vs ArmLowerTexture) which created duplicate directories
on case-sensitive Linux filesystems. Now mapPath() lowercases the entire
output. Also keeps TextureComponents and ObjectComponents directory
names instead of abbreviating them (item/texturecomponents/ instead of
item/texture/) so filesystem paths match the WoW virtual paths used in
manifest lookups.
The string-column auto-detector in both tools had two gaps that caused small
integer fields (RaceID=1, SexID=0/1, BaseSection, ColorIndex) to be falsely
classified as string columns, corrupting the generated CSVs:
1. No boundary check: a value of N was accepted as a valid string offset even
when N landed inside a longer string (e.g. offset 3 inside "Character\...").
Fix: precompute valid string-start boundaries (offset 0 plus every position
immediately after a null byte); reject offsets that are not boundaries.
2. No diversity check: a column whose only non-zero value is 1 would pass the
boundary test because offset 1 is always a valid boundary (it follows the
mandatory null at offset 0). Fix: require at least 2 distinct non-empty
string values before marking a column as a string column. Columns like
SexID (all values are 0 or 1, resolving to "" and the same path fragment)
are integer fields, not string fields.
Both dbc_to_csv and asset_extract now produce correct column metadata,
e.g. CharSections.dbc yields "strings=6,7,8" instead of "strings=0,1,...,9".
- Add build.ps1/bat, rebuild.ps1/bat, debug_texture.ps1/bat (Windows equivalents
of existing bash scripts, using directory junctions for Data link)
- Fix asset extractor: StormLib is not thread-safe even with separate handles per
thread. Serialize all MPQ reads behind a mutex while keeping CRC computation and
disk writes parallel. Previously caused 99.8% extraction failures with >1 thread.
- Add SFileHasFile() check during enumeration to skip listfile-only entries
- Add diagnostic logging for extraction failures (first 5 per thread + summary)
- Use std::filesystem::temp_directory_path() instead of hardcoded /tmp/ in
character_renderer.cpp debug dumps
- Update debug_texture.sh to use $TMPDIR fallback and glob for actual dump filenames
Remove HDPackManager, expansion overlay manifests, and BLP size-comparison
logic. Assets now resolve through a single manifest with a simple override
directory (Data/override/) for future HD upgrades.
Extracts each expansion's assets as a CRC-compared overlay against a
base manifest, storing only files that differ. Auto-detects overlay mode
when a base manifest already exists. Adds --as-overlay, --full-base
flags and manifest merge for partial extractions.
Expansion overlays allow each expansion to supplement the base asset data
via an assetManifest field in expansion.json, loaded at priority 50 (below
HD packs). The asset extractor gains --reference-manifest for delta-only
extraction. Also fixes CharSections field indices (VariationIndex=4,
ColorIndex=5, Texture1=6) across all DBC layout references.
Replace hardcoded WotLK protocol constants with a data-driven architecture
supporting Classic 1.12.1, TBC 2.4.3, and WotLK 3.3.5a. Each expansion
has JSON profiles for opcodes, update fields, and DBC layouts, plus C++
polymorphic packet parsers for binary format differences (movement flags,
speed fields, transport data, spline format, char enum layout).
Key components:
- ExpansionRegistry: scans Data/expansions/*/expansion.json at startup
- OpcodeTable: logical enum <-> wire values loaded from JSON
- UpdateFieldTable: field indices loaded from JSON per expansion
- DBCLayout: schema-driven DBC field lookups replacing magic numbers
- PacketParsers: WotLK/TBC/Classic parsers with correct flag positions
- Multi-manifest AssetManager: layered manifests with priority ordering
- HDPackManager: overlay texture packs with expansion compatibility
- Auth screen expansion picker replacing hardcoded version dropdown
