feat: Store metadata in SQLite#1366
Draft
d3xter666 wants to merge 44 commits intofeat/incremental-build-4from
Draft
feat: Store metadata in SQLite#1366d3xter666 wants to merge 44 commits intofeat/incremental-build-4from
d3xter666 wants to merge 44 commits intofeat/incremental-build-4from
Conversation
d3xter666
changed the title
* Improve handling for concurrent resource access and modifications, especially when buffering streams. * Deprecate getStatInfo in favor of dedicated getSize, isDirectory, getLastModified methods. * Deprecate synchronous getStream in favor of getStreamAsync and modifyStream, allowing for atomic modification of resource content * Generate Resource hash using ssri
Add msgpackr, better-sqlite3, and classic-level to packages/project so all cache option branches have these available after npm ci. CPOUI5FOUNDATION-1174
Native addons like better-sqlite3 and classic-level require compilation after npm ci. Without rebuild, benchmarks for options C, D, and E fail with missing binding errors.
npm rebuild does not compile native addons in workspace monorepos. Run node-gyp rebuild directly in each native addon directory. Also add --show-output to hyperfine to surface build errors.
Add BuildTimings utility that measures cache read/write and serialization operations. Enabled via UI5_BUILD_TIMINGS=true environment variable. Instruments all CacheManager read/write methods and logs a summary table at the end of the build showing call counts, totals, averages, min, and max per operation category (Reads, Writes, Resource I/O).
- Option J: Incremental Request Managers - Option K: Flat JSON Store - Option L: JSONL+jq Streaming
Adds scripts/run-serve-timings-benchmark.mjs that measures warm-cache ui5 serve startup performance across all cache option branches. Spawns ui5 serve --port 0, triggers on-demand build via HTTP GET, captures Build succeeded timing, then kills the process. Outputs markdown report and JSON data.
ANSI color codes from chalk can break regex parsing of 'Build succeeded' and timing table output. Set NO_COLOR=1 to disable all chalk formatting in the spawned build process.
* Improve handling for concurrent resource access and modifications, especially when buffering streams. * Deprecate getStatInfo in favor of dedicated getSize, isDirectory, getLastModified methods. * Deprecate synchronous getStream in favor of getStreamAsync and modifyStream, allowing for atomic modification of resource content * Generate Resource hash using ssri
Add msgpackr, better-sqlite3, and classic-level to packages/project so all cache option branches have these available after npm ci. CPOUI5FOUNDATION-1174
Replace file-based JSON storage with a single SQLite database via better-sqlite3. All 5 metadata categories stored as tables with composite keys and JSON text values. WAL mode enabled for read perf. New file: MetadataStore.js — SQLite wrapper with prepared statements. CacheManager.js delegates all read/write to MetadataStore. Added close() lifecycle method. Expected: ~45% size reduction, ~3x speed, ACID transactions. CPOUI5FOUNDATION-1174
Add BuildTimings utility that measures cache read/write and serialization operations. Enabled via UI5_BUILD_TIMINGS=true environment variable. Instruments all CacheManager read/write methods and logs a summary table at the end of the build showing call counts, totals, averages, min, and max per operation category (Reads, Writes, Resource I/O).
Previously BuildTimings.logSummary() was called before cache writes executed, causing write operations to show 0 calls in the timing report. Moving the summary log to the finally block after cache writes ensures all write timings (writeIndexCache, writeTaskMetadata, etc.) are correctly captured and displayed.
Co-authored-by: Copilot <copilot@github.com>
d3xter666
force-pushed
the
feat/incremental-build-sqlite-native-metadata
branch
from
76a8dbc to
31a0af9
Compare
Revert changes outside packages/project and scripts to match the base branch. Restore ContentAddressableStorage usage in CacheManager and fix getHash() back to getIntegrity() calls.
Co-authored-by: Copilot <copilot@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: CPOUI5FOUNDATION-1209
Uses https://nodejs.org/api/sqlite.html to store incremental build metadata.
Gains write performance, since writing to SQL is non-blocking