Commit graph

38 commits

Author SHA1 Message Date
Kelsi Davis
bcf1015149 fix(rendering): check sampler validity in VkTexture::isValid(), fix Windows build
- VkTexture::isValid() now checks both image AND sampler handles. Previously
  it only checked the image, so a texture with a valid image but NULL sampler
  would pass validation and get bound to a descriptor set. On MoltenVK (macOS)
  this renders as pink/magenta boxes; the fallback white texture is now
  correctly used instead.

- Fix fs::path to std::string implicit conversion in asset extractor that
  broke the Windows (MSYS2/clang) CI build.
2026-04-05 01:34:49 -07:00
Kelsi Rae Davis
50fdfd2e22
Merge pull request #47 from sschepens/patch-2
refactor asset extractor
2026-04-05 01:10:28 -07:00
Paul
e58f9b4b40 feat(animation): 452 named constants, 30-phase character animation state machine
Add animation_ids.hpp/cpp with all 452 WoW animation ID constants (anim::STAND,
anim::RUN, anim::FIRE_BOW, ... anim::FLY_BACKWARDS, etc.), nameFromId() O(1)
lookup, and flyVariant() compact 218-element ground→FLY_* resolver.

Expand AnimationController into a full state machine with 20+ named states:
spell cast (directed→omni→cast fallback chain, instant one-shot release),
hit reactions (WOUND/CRIT/DODGE/BLOCK/SHIELD_BLOCK), stun, wounded idle,
stealth animation substitution, loot, fishing channel, sit/sleep/kneel
down→loop→up transitions, sheathe/unsheathe combat enter/exit, ranged weapons
(BOW/GUN/CROSSBOW/THROWN with reload states), game object OPEN/CLOSE/DESTROY,
vehicle enter/exit, mount flight directionals (FLY_LEFT/RIGHT/UP/DOWN/BACKWARDS),
emote state variants, off-hand/pierce/dual-wield alternation, NPC
birth/spawn/drown/rise, sprint aura override, totem idle, NPC greeting/farewell.

Add spell_defines.hpp with SpellEffect (~45 constants) and SpellMissInfo
(12 constants) namespaces; replace all magic numbers in spell_handler.cpp.

Add GAMEOBJECT_BYTES_1 to update field table (all 4 expansion JSONs) and wire
GameObjectStateCallback. Add DBC cross-validation on world entry.

Expand tools/_ANIM_NAMES from ~35 to 452 entries in m2_viewer.py and
asset_pipeline_gui.py. Add tests/test_animation_ids.cpp.

Bug fixes included:
- Stand state 1 was animating READY_2H(27) — fixed to SITTING(97)
- Spell casts ended freeze-frame — add one-shot release animation
- NPC 2H swing probe chain missing ATTACK_2H_LOOSE (polearm/staff)
- Chair sits (states 2/4/5/6) incorrectly played floor-sit transition
- STOP(3) used for all spell casts — replaced with model-aware chain
2026-04-04 23:02:53 +03:00
sschepens
1e464dd513
refactor path mapper 2026-04-04 14:34:23 -03:00
sschepens
5542cbaa02
refactor asset extractor
- mpq and locale finding is now case insensitive
- improve extraction order and support more patches
- unified much of the mpq logic for all expansions
- return a list of ordered paths for loading
2026-04-04 14:00:55 -03:00
Paul
d54e262048 feat(rendering): GPU architecture + visual quality fixes
M2 GPU instancing
- M2InstanceGPU SSBO (96 B/entry, double-buffered, 16384 max)
- Group opaque instances by (modelId, LOD); single vkCmdDrawIndexed per group
- boneBase field indexes into mega bone SSBO via gl_InstanceIndex

Indirect terrain drawing
- 24 MB mega index buffer (6M uint32) + 64 MB mega vertex buffer
- CPU builds VkDrawIndexedIndirectCommand per visible chunk
- Single VB/IB bind per frame; shadow pass reuses mega buffers
- Replaced vkCmdDrawIndexedIndirect with direct vkCmdDrawIndexed to fix
  host-mapped buffer race condition that caused terrain flickering

GPU frustum culling (compute shader)
- m2_cull.comp.glsl: 64-thread workgroups, sphere-vs-6-planes + distance cull
- CullInstanceGPU SSBO input, uint visibility[] output, double-buffered
- dispatchCullCompute() runs before main pass via render graph node

Consolidated bone matrix SSBOs
- 16 MB double-buffered mega bone SSBO (2048 instances × 128 bones)
- Eliminated per-instance descriptor sets; one megaBoneSet_ per frame
- prepareRender() packs bone matrices consecutively into current frame slot

Render graph / frame graph
- RenderGraph: RGResource handles, RGPass nodes, Kahn topological sort
- Automatic VkImageMemoryBarrier/VkBufferMemoryBarrier between passes
- Passes: minimap_composite, worldmap_composite, preview_composite,
  shadow_pass, reflection_pass, compute_cull
- beginFrame() uses buildFrameGraph() + renderGraph_->execute(cmd)

Pipeline derivatives
- PipelineBuilder::setFlags/setBasePipeline for VK_PIPELINE_CREATE_DERIVATIVE_BIT
- M2 opaque = base; alphaTest/alpha/additive are derivatives
- Applied to terrain (wireframe) and WMO (alpha-test) renderers

Rendering bug fixes:
- fix(shadow): compute lightSpaceMatrix before updatePerFrameUBO to eliminate
  one-frame lag that caused shadow trails and flicker on moving objects
- fix(shadow): scale depth bias with shadowDistance_ instead of hardcoded 0.8f
  to prevent acne at close range and gaps at far range
- fix(visibility): WMO group distance threshold 500u → 1200u to match terrain
  view distance; buildings were disappearing on the horizon
- fix(precision): camera near plane 0.05 → 0.5 (ratio 600K:1 → 60K:1),
  eliminating Z-fighting and improving frustum plane extraction stability
- fix(streaming): terrain load radius 4 → 6 tiles (~2133u → ~3200u) to exceed
  M2 render distance (2800u) and eliminate pop-in when camera turns;
  unload radius 7 → 9; spawn radius 3 → 4
- fix(visibility): ground-detail M2 distance multiplier 0.75 → 0.9 to reduce
  early pop of grass and debris
2026-04-04 13:43:16 +03:00
Kelsi Davis
2343b768ce fix: warden mmap on macOS, add external listfile support to asset extractor
Some checks are pending
Build / Build (arm64) (push) Waiting to run
Build / Build (x86-64) (push) Waiting to run
Build / Build (macOS arm64) (push) Waiting to run
Build / Build (windows-arm64) (push) Waiting to run
Build / Build (windows-x86-64) (push) Waiting to run
Security / CodeQL (C/C++) (push) Waiting to run
Security / Semgrep (push) Waiting to run
Security / Sanitizer Build (ASan/UBSan) (push) Waiting to run
Drop PROT_EXEC from warden module mmap when using Unicorn emulation
(not needed — module image is copied into emulator address space). Use
MAP_JIT on macOS for the native fallback path.

Add --listfile option to asset_extract and SFileAddListFileEntries
support for resolving unnamed MPQ hash table entries from external
listfiles.
2026-04-04 01:16:28 -07:00
k
b3fa8cf5f3 fix: warden mmap on macOS, add external listfile support to asset extractor
Drop PROT_EXEC from warden module mmap when using Unicorn emulation
(not needed — module image is copied into emulator address space). Use
MAP_JIT on macOS for the native fallback path.

Add --listfile option to asset_extract and SFileAddListFileEntries
support for resolving unnamed MPQ hash table entries from external
listfiles.
2026-04-04 00:22:07 -07:00
Kelsi
7264ba1706 fix(extractor): lowercase all output paths to prevent duplicate folders
WoW archives contain mixed-case variants of the same path (e.g.,
ARMLOWERTEXTURE vs ArmLowerTexture) which created duplicate directories
on case-sensitive Linux filesystems. Now mapPath() lowercases the entire
output. Also keeps TextureComponents and ObjectComponents directory
names instead of abbreviating them (item/texturecomponents/ instead of
item/texture/) so filesystem paths match the WoW virtual paths used in
manifest lookups.
2026-04-03 21:26:20 -07:00
Kelsi
4b379f6fe9 chore: fix executable permissions on 6 scripts
All had shebangs (#!/usr/bin/env bash/python3) but were missing +x:
- restart-worldserver.sh
- tools/diff_classic_turtle_opcodes.py
- tools/gen_opcode_registry.py
- tools/m2_viewer.py
- tools/opcode_map_utils.py
- tools/validate_opcode_maps.py
2026-03-30 18:55:15 -07:00
Kelsi
6ede9a2968 refactor: derive turtle opcodes from classic 2026-03-15 02:55:05 -07:00
Kelsi
b31a2a66b6 tools: fix DBC string-column detection false positives in both dbc_to_csv and asset_extract
The string-column auto-detector in both tools had two gaps that caused small
integer fields (RaceID=1, SexID=0/1, BaseSection, ColorIndex) to be falsely
classified as string columns, corrupting the generated CSVs:

1. No boundary check: a value of N was accepted as a valid string offset even
   when N landed inside a longer string (e.g. offset 3 inside "Character\...").
   Fix: precompute valid string-start boundaries (offset 0 plus every position
   immediately after a null byte); reject offsets that are not boundaries.

2. No diversity check: a column whose only non-zero value is 1 would pass the
   boundary test because offset 1 is always a valid boundary (it follows the
   mandatory null at offset 0). Fix: require at least 2 distinct non-empty
   string values before marking a column as a string column. Columns like
   SexID (all values are 0 or 1, resolving to "" and the same path fragment)
   are integer fields, not string fields.

Both dbc_to_csv and asset_extract now produce correct column metadata,
e.g. CharSections.dbc yields "strings=6,7,8" instead of "strings=0,1,...,9".
2026-03-10 03:49:06 -07:00
Kelsi
4a213d8da8 tools/game: fix dbc_to_csv false-positive string detection + clear DBC cache on expansion switch
dbc_to_csv: The string-column auto-detector would mark integer fields (e.g.
RaceID=1, SexID=0, BaseSection=0-4) as string columns whenever their small
values were valid string-block offsets that happened to land inside longer
strings.  Fix by requiring that an offset point to a string *boundary* (offset
0 or immediately after a null byte) rather than any valid position — this
eliminates false positives from integer fields whose values accidentally alias
path substrings.  Affected CSVs (CharSections, ItemDisplayInfo for Classic/TBC)
can now be regenerated correctly.

game_handler: clearDBCCache() is already called by application.cpp before
resetDbcCaches(), but also add it inside resetDbcCaches() as a defensive
measure so that future callers of resetDbcCaches() alone also flush stale
expansion-specific DBC data (CharSections, ItemDisplayInfo, etc.).
2026-03-10 03:27:30 -07:00
Kelsi
e2b89c9b42 Fix FSR3 permutation script failures on arm64 Linux and Windows
Linux arm64 (Exec format error):
- The script was downloading the x86_64 DXC release on all Linux hosts;
  on aarch64 runners the x86_64 ELF fails with EXEC_FORMAT_ERROR at
  shader compilation time. Add uname -m guard: when running on aarch64/
  arm64 Linux without a DXC in PATH, exit 0 with an advisory message
  rather than downloading an incompatible binary. The FSR3 SDK build
  proceeds as it did before the permutation script was introduced
  (permutation headers are expected to be pre-built in the SDK checkout).

Windows (bash: command not found, exit 127):
- cmake custom-target COMMANDs run via cmd.exe on Windows even when
  cmake is configured from a MSYS2 shell, so bare 'bash' is not resolved.
- Use find_program(BASH_EXECUTABLE bash) at configure time (which runs
  under shell: msys2 in CI and thus finds the MSYS2 bash at its native
  Windows-absolute path). When bash is found, embed the full path in the
  COMMAND; when not found (unusual non-MSYS2 Windows setups), skip the
  permutation step and emit a STATUS message.
2026-03-09 13:11:03 -07:00
Kelsi
bae32c1823 Add FSR3 Generic API path and harden runtime diagnostics
- AmdFsr3Runtime now probes both the legacy ffxFsr3* API and the newer
  generic ffxCreateContext/ffxDispatch API; selects whichever the loaded
  runtime library exports (GenericApi takes priority fallback)
- Generic API path implements full upscale + frame-generation context
  creation, configure, dispatch, and destroy lifecycle
- dlopen error captured and surfaced in lastError_ on Linux so runtime
  initialization failures are actionable
- FSR3 runtime init failure log now includes path kind, error string,
  and loaded library path for easier debugging
- tools/generate_ffx_sdk_vk_permutations.sh added: auto-bootstraps
  missing VK permutation headers; DXC auto-downloaded on Linux/Windows
  MSYS2; macOS reads from PATH (CI installs via brew dxc)
- CMakeLists: add upscalers/include to probe include dirs, invoke
  permutation script before SDK build, scope FFX pragma/ODR warning
  suppressions to affected TUs, add runtime-copy dependency on wowee
- UI labels updated from "FSR2" → "FSR3" in settings, tuning panel,
  performance HUD, and combo boxes
- CI macOS job now installs dxc via Homebrew for permutation codegen
2026-03-09 12:51:59 -07:00
Kelsi
e1614d55a2 chore: ignore and untrack python cache files 2026-02-25 10:42:50 -08:00
Kelsi
c849f9cdfa chore: update camera controller changes 2026-02-25 10:41:54 -08:00
Kelsidavis
1fab17e639 Add Windows build scripts, fix multi-threaded MPQ extraction, and cross-platform temp paths
- Add build.ps1/bat, rebuild.ps1/bat, debug_texture.ps1/bat (Windows equivalents
  of existing bash scripts, using directory junctions for Data link)
- Fix asset extractor: StormLib is not thread-safe even with separate handles per
  thread. Serialize all MPQ reads behind a mutex while keeping CRC computation and
  disk writes parallel. Previously caused 99.8% extraction failures with >1 thread.
- Add SFileHasFile() check during enumeration to skip listfile-only entries
- Add diagnostic logging for extraction failures (first 5 per thread + summary)
- Use std::filesystem::temp_directory_path() instead of hardcoded /tmp/ in
  character_renderer.cpp debug dumps
- Update debug_texture.sh to use $TMPDIR fallback and glob for actual dump filenames
2026-02-25 08:22:45 -08:00
Kelsi
60c26a17aa Fix audio playback not stopping when Stop button clicked
Some checks are pending
Build / Build (arm64) (push) Waiting to run
Build / Build (x86-64) (push) Waiting to run
Build / Build (macOS arm64) (push) Waiting to run
Build / Build (windows-arm64) (push) Waiting to run
Build / Build (windows-x86-64) (push) Waiting to run
Security / CodeQL (C/C++) (push) Waiting to run
Security / Semgrep (push) Waiting to run
Security / Sanitizer Build (ASan/UBSan) (push) Waiting to run
Use spawn context for clean subprocess isolation and add kill() fallback
after terminate() to ensure the audio process is reliably stopped.
2026-02-23 22:26:17 -08:00
Kelsi
55faacef96 Add M2/WMO 3D viewer with textured rendering, animation, and audio playback
- New tools/m2_viewer.py: Pygame/OpenGL viewer for M2 models (textured
  rendering, skeletal animation, orbit camera) and WMO buildings
- M2 viewer: per-batch texture mapping, CPU vertex skinning, animation
  playback with play/pause/speed controls, wireframe overlay toggle
- WMO viewer: root+group file parsing (MOTX/MOMT/MOVT/MOVI/MONR/MOTV/MOBA),
  per-batch material rendering with BLP textures
- Asset browser: "Open 3D Viewer" buttons for M2 and WMO previews,
  audio Play/Stop buttons using pygame.mixer in subprocess
- Handles both WotLK (v264) and Vanilla (v256) M2 formats
2026-02-23 22:22:39 -08:00
Kelsi
edf0a40759 Fix asset browser BLP errors, M2 wireframes, and add anim filter
- BLP: blp_convert takes one arg, not two; was passing an output path
  that caused conversion failures
- M2: vertex header offsets were wrong (used 80/100 instead of 60/68),
  producing garbage vertex counts that failed the sanity check
- Add "Hide .anim/.skin" checkbox (on by default) to filter ~30k
  companion files from the directory tree
2026-02-23 20:51:33 -08:00
Kelsi
6e2d51b325 Fix asset browser hanging on launch with large manifests
Manifest keys use backslashes but tree splitting used forward slashes,
causing all 241k entries to land at root level. Combined with O(N)
any(startswith) checks per entry, this produced an O(N^2) hang. Re-key
manifest by the forward-slash 'p' field and build a directory index in
a single O(N) pass so tree operations are O(1) lookups.
2026-02-23 20:45:19 -08:00
Kelsi
739ae5b569 Add Asset Browser tab to pipeline GUI with inline previews
New tab lets users explore extracted assets visually: BLP textures
rendered via blp_convert+Pillow, M2/WMO wireframes with mouse-drag
rotation and zoom, DBC/CSV tables with named columns from
dbc_layouts.json, ADT heightmap grids, text file viewer, audio
metadata, and hex dumps. Directory tree lazy-loads from manifest.json
with search and file-type filtering.
2026-02-23 20:35:32 -08:00
Kelsi
f95770720f Fix security, bugs, and UX in asset pipeline GUI
- Fix zip slip vulnerability: validate extracted paths stay within target
- Fix redundant mkdir before rmtree in rebuild_override()
- Add build/asset_extract and Windows .ps1 fallback to extractor search
- Preserve pack list selection across refreshes
- Add Cancel Extraction button with process.terminate()
- Run override rebuild in background thread to avoid UI freeze
- Fix locale combobox state to readonly
- Add asset_pipeline/ to .gitignore
- Make script executable
2026-02-23 20:06:41 -08:00
Kelsi
20dd5ed63b Add cross-platform Python asset pipeline GUI 2026-02-23 20:03:07 -08:00
Kelsi
f66b9eb154 Fix macOS build process and make shell scripts cross-platform
- Add vulkan-loader, vulkan-headers, shaderc to macOS CI brew install
- Add vulkan-loader and shaderc to macOS PKG_CONFIG_PATH
- Replace Linux-only `nproc` with portable fallback in build.sh,
  rebuild.sh, extract_assets.sh, and tools/backup_assets.sh
- Replace `ldconfig` StormLib check with portable detection
  (ldconfig, pkg-config, brew lib) in extract_assets.sh
- Update BUILD_INSTRUCTIONS.md macOS section with vulkan/shaderc
  packages and MoltenVK explanation
- Add macOS prerequisites to README.md
2026-02-23 18:35:53 -08:00
Kelsi
610ed71922 Opcode registry: move to generated canonical+alias pipeline
Introduce data-driven opcode registry with canonical and alias sources:

- Added Data/opcodes/canonical.json as the single canonical LogicalOpcode set.

- Added Data/opcodes/aliases.json for cross-core naming aliases (CMaNGOS/AzerothCore/local legacy).

Added generator and generated include fragments:

- tools/gen_opcode_registry.py emits include/game/opcode_enum_generated.inc, include/game/opcode_names_generated.inc, and include/game/opcode_aliases_generated.inc.

- include/game/opcode_table.hpp now consumes generated enum entries.

- src/game/opcode_table.cpp now consumes generated name and alias tables.

Loader canonicalization behavior:

- OpcodeTable::nameToLogical canonicalizes incoming JSON opcode names via alias table before enum lookup, so implementation code stays stable while expansion maps can use different core spellings.

Validation and build integration:

- Added tools/validate_opcode_maps.py to validate canonical contract across expansions.

- Added CMake targets opcodes-generate and opcodes-validate.

- wowee target now depends on opcodes-generate so generated headers stay current.

Validation/build run:

- cmake -S . -B build

- cmake --build build --target opcodes-generate opcodes-validate

- cmake --build build -j32
2026-02-20 03:02:31 -08:00
Kelsi
9fc41d6f30 Add asset backup script with zstd/pigz/gzip fallback 2026-02-15 04:25:56 -08:00
Kelsi
d7e2b26af7 Unify asset system: one asset set, always high-res
Remove HDPackManager, expansion overlay manifests, and BLP size-comparison
logic. Assets now resolve through a single manifest with a simple override
directory (Data/override/) for future HD upgrades.
2026-02-15 04:18:34 -08:00
Kelsi
a67dca5787 Add overlay extraction for multi-expansion asset deduplication
Extracts each expansion's assets as a CRC-compared overlay against a
base manifest, storing only files that differ. Auto-detects overlay mode
when a base manifest already exists. Adds --as-overlay, --full-base
flags and manifest merge for partial extractions.
2026-02-14 03:09:17 -08:00
Kelsi
03013e751a Cache WoW.exe during asset extraction for Warden MEM_CHECK 2026-02-14 02:17:50 -08:00
Kelsi
886f4daf2e Add per-expansion asset overlay system and fix CharSections DBC layout
Expansion overlays allow each expansion to supplement the base asset data
via an assetManifest field in expansion.json, loaded at priority 50 (below
HD packs). The asset extractor gains --reference-manifest for delta-only
extraction. Also fixes CharSections field indices (VariationIndex=4,
ColorIndex=5, Texture1=6) across all DBC layout references.
2026-02-14 00:00:26 -08:00
Kelsi
492703be36 Auth: include CRC in legacy proof; extend Turtle integrity set 2026-02-13 01:41:59 -08:00
Kelsi
5435796a98 Add integrity hash support and SRP tuning options 2026-02-13 01:32:15 -08:00
Kelsi
6a44f02e0c Add authenticator opcode support + auth_probe tool 2026-02-13 00:55:36 -08:00
Kelsi
f247d53309 Add expansion DBC CSVs, Turtle support, and server-specific login 2026-02-13 00:10:01 -08:00
Kelsi
7092844b5e Add multi-expansion support with data-driven protocol layer
Replace hardcoded WotLK protocol constants with a data-driven architecture
supporting Classic 1.12.1, TBC 2.4.3, and WotLK 3.3.5a. Each expansion
has JSON profiles for opcodes, update fields, and DBC layouts, plus C++
polymorphic packet parsers for binary format differences (movement flags,
speed fields, transport data, spline format, char enum layout).

Key components:
- ExpansionRegistry: scans Data/expansions/*/expansion.json at startup
- OpcodeTable: logical enum <-> wire values loaded from JSON
- UpdateFieldTable: field indices loaded from JSON per expansion
- DBCLayout: schema-driven DBC field lookups replacing magic numbers
- PacketParsers: WotLK/TBC/Classic parsers with correct flag positions
- Multi-manifest AssetManager: layered manifests with priority ordering
- HDPackManager: overlay texture packs with expansion compatibility
- Auth screen expansion picker replacing hardcoded version dropdown
2026-02-12 22:56:36 -08:00
Kelsi
aa16a687c2 Replace MPQ runtime with loose file asset system
Extract assets from MPQ archives into organized loose files indexed by
manifest.json, enabling fully parallel reads without StormLib serialization.
Add asset_extract and blp_convert tools, PNG texture override support.
2026-02-12 20:32:14 -08:00