Back to Logbook
May 20, 2026 Project: #Portfolio

Vector embeddings and real analytics.

energised

The RAG search has been keyword-only this whole time. Today I made it semantic.

Built EmbeddingService — calls OpenAI text-embedding-3-small, gets a 1536-dimensional vector, stores it as JSON in the existing embedding_vector column. Status tracked in embedding_status (pending → processing → completed/failed). Truncates input at 8000 multibyte chars using mb_strlen/mb_substr to avoid mid-codepoint cuts.

Then rewrote RAGService::retrieveDocuments() to be hybrid: keyword LIKE pass first (broad recall), then cosine similarity re-rank on documents that have embeddings. Final score is 0.4 × keyword + 0.6 × cosine. Falls back gracefully to keyword-only when embeddings are absent or when dimensions do not match.

The KB embed button kept reporting "Request failed" — turned out native fetch() does not send X-Requested-With: XMLHttpRequest, so isAJAX() returned false, the controller fell through to a redirect, and the HTML response could not be parsed as JSON.

AnalyticsService had three methods that returned hardcoded 4.5 or rand() values — leftover stubs. Replaced all three with real DB queries. Discovered AnalyticsModel::$allowedFields was empty (model was blocking all inserts) and that MySQL and SQLite return different key names for un-aliased AVG() — switched to explicit aliasing.

The Workflow controller had a production-silent bug. CI4 4.4+ getMethod() returns uppercase 'POST', but create() and edit() were doing === 'post' — so every form submission silently rendered the GET view. Fixed.

Then the test push. Added 74 new tests covering previously untested code. Suite is now 171 passing, was 97.

Lessons Learned

Hybrid search beats pure vector or pure keyword. Recall + precision, not either-or.