Has this happened before?

Spoken notes

One database, one day: 1,936 events. Almost all routine: log switches, archived logs, a partition added, a data file growing. Then at five past two in the afternoon: ORA-04031, the shared pool cannot allocate four kilobytes for a cursor. Another at 14:05:41. Another twenty seconds later. 158 of them in 33 minutes. Everyone here can look up the code. Whether this database has done this before, and what was done about it, is the harder question.

Presenter cue: Let the number land. Read the highlighted line, then the two below it.

Spoken notes

Same error, fin1, first seen March 31 at the month-end close of the general ledger: 412 failures, the close did not finish, the on-call restarted the instance and reran it. April 30, the same. May 29, the same, and the next morning the shared pool was doubled, because that is what the vendor page says. June 30: 603 failures with the bigger pool. July 31: 744. Five month-ends, five restarts, one memory increase. Then a fix that held, and a clean close in August. Then September 15, mid-month, business hours. Does the next person get to read all that before restarting a sixth time?

Presenter cue: Step 0 zooms out of the dense file. Next key: the timeline. Read the bottom bar.

Spoken notes

Finding an error code is only the beginning. Relevant clues can be ordinary messages: a job starting, a log switch, a change in the workload. Their presence does not prove a cause. We need to connect the events to the workload and to earlier investigations. Later, the measured example will show a model incorrectly blaming a routine log switch. This is why preserving context matters, and why people still check the interpretation.

Spoken notes

The code is the easy part. The right answer depends on the version and release update, the platform and edition, parameters that interact, one-off patches, and what this team already tried. The answers are spread across vendor documentation, support notes, practitioner write-ups, team records and people’s experience. The model needs enough technical context to find relevant explanations. The local investigation also needs the workload and action history to judge which explanation fits. In our case study, that means distinguishing the month-end close from the restatement job.

Spoken notes

Oracle is a large, closed-source system with more than thirty years of history. Some errors you may never have seen before. Others appear once every few years, long enough to forget the last investigation. Those rare, tricky cases can take a lot of research. This project should help preserve what we learned and find the relevant history. A message may still be misleading, or a useful clue may not be labelled as an error, so the administrator must judge the context.

Spoken notes

This is a proof of concept, starting with Oracle because that is where we have strong expertise. We are testing whether retained context makes investigations more useful. Better in-house inference would let us explore greater scale. We also need broader evaluation before extending the approach to other infrastructure. The case study illustrates the workflow; the separate measured calls show what one local model actually did.

Spoken notes

Roughly 300 databases, three logs each, about 1,500 events per database a day. That is 450,000 events a day. Filebeat ships the lines, Logstash labels them, Elasticsearch stores them. Searching that is fine. Keeping track of the investigations is not.

Presenter cue: Estimate on the slide; say so.

Spoken notes

This is a bank, not a startup. Move fast and break things is an incident report here. Pause for the joke. The database infrastructure, logs and operational context are all on premises, in a conservative enterprise environment. We cannot trade reliability for speed; the investigation has to fit those constraints. Obviously, all the cool tools are not available, so maybe we can do something ourselves. Later we will distinguish the intended isolated deployment from the broader cloud routes used in the prototype, and explain the boundary for external research.

Spoken notes

The environment we are designing for is deliberately isolated. Sensitive logs and identifiable operational context stay inside the database infrastructure. We cannot simply connect a third-party agent to the logs. External research needs a separate, anonymized technical brief. That is the intended deployment boundary. The prototype configuration and experiments are not proof that this boundary has been deployed end to end.

Spoken notes

Four advances build the architecture. Database hosts ship logs with Filebeat. Elasticsearch retains the raw events. The local Logbook node compacts events, orchestrates calls, validates proposals and writes the Git wiki. Administrators use the Workbench web application as their control UI between themselves and those records: investigate, record actions, resolve and reopen. Its backend is the Workbench API inside the local node; the diagram abstracts that API mediation into the UI-to-wiki connection. The Git remote retains version history. Selected prototype calls use a cloud model; Kibana and Langfuse provide monitoring. The arrows are deliberately unlabelled; node contents explain each component.

Spoken notes

This is the requirement for the intended isolated deployment: we cannot leak sensitive information to the cloud. Logs, workload context, incident history and actions stay inside. The external role is basically assisted and automated Google search, using a checked technical brief and returning cited findings. No wiki clone or infrastructure credentials go to that researcher. The prototype cloud routes shown earlier are broader; this title is a deployment requirement, not a claim that the current configuration guarantees it. End-to-end validation of the isolated path is still required.

Spoken notes

Kibana finds the old log lines. I also want the explanation someone reached after reading them, and what they did. Logbook keeps that beside the incident and its action history: Markdown in git, updated by scheduled jobs and by administrators. The idea of a wiki maintained as new material arrives is borrowed. The next person gets something to read besides raw logs.

Spoken notes

The blue agent icon touches Events, Research and History: it summarizes evidence, researches references and proposes updates. The green administrator icon touches Research, Change, Check and History: people judge explanations, investigate, make changes, review recovery and record the outcome. Matching lines and colored edges show who contributes to each record. Code validates files and rebuilds history tables. The shared wiki keeps those contributions together; the agent does not own operational actions or incident resolution.

Spoken notes

These are the prototype routes, including broader cloud access than the intended isolated design. Most calls go to local Nemotron through pi, thinking off for bulk, on for notable ingest. Same resident model, no swapping. Claude Code runs Opus for notable reports and research, where it reads across pages or browses. Sonnet does lint and practitioner notes.

Presenter cue: Three presses, same order: local, cloud, none.

Spoken notes

This is the prototype cadence and routing. The isolated deployment keeps incident reasoning local and separates anonymized research. Ingest every two hours. Reports after notable ingests and at 23:30. Monday: lint, research, review. Source review monthly; past fixes nightly. Three presses light the rows: local calls, cloud calls, no model.

Presenter cue: Three presses: local, cloud, none. The pips show where you are.

Spoken notes

Everything the jobs do is written down somewhere I can query. The VM keeps ledgers and a health log; Filebeat ships them into the same Elasticsearch as the database logs, so Kibana shows the runs next to the events. Every model call goes to Langfuse: prompt, response, tokens, cost. The workbench links each run into both. One run ID joins the health log, the ledger, the git commit, Kibana and Langfuse. The run ID connects the evidence across these views.

Presenter cue: Point at the shared run ID. Keep the presentation on this slide.

Spoken notes

The preprocessing is ordinary code: classify, count, group repeats, compare with recorded history. That is a digest. The model reads it and proposes an interpretation. More code checks the result and writes the wiki. One ingest call gets one capped digest with short excerpts and document IDs we can check.

Spoken notes

This listener produced 2,254 repeated errors. Normalize digit runs, keep the codes, group matching messages. The digest holds a count, a representative message and a few document IDs, about a kilobyte. We lose detail, so the originals stay in Elasticsearch. The same process turns fin1’s afternoon into three groups: 158 allocation failures, 158 trace-file lines, six diagnostic dumps.

Presenter cue: One press: the events fold into the group. Say “the model reads this box, never the wall”.

Spoken notes

For fin1 on September 15, 1,936 events become three groups. The prompt takes at most 12,000 characters of digest, plus the open incidents and the error pages it names. This cap applies to the digest, not to the whole prompt. Over the recorded period, 82 percent of routing decisions skipped the model. There were 1,154 model runs, mostly local, and about 33 dollars in API charges from 13 cloud calls. The compaction and routing are code.

Spoken notes

This is an abbreviated Markdown mock of the constructed July report, not a screenshot or a new measured model response. The highlighted sections show what a useful report connects: today’s errors, the changes already tried, clues in the history and a proposed next step. The links lead back to the retained digest, incident and error record. At this July cutoff the incident was still open. The dated returns after four restarts and a memory increase are evidence to investigate the statements, not proof of a diagnosis. This constructed example is separate from the two measured local calls retained in the experiment files.

Spoken notes

The incident is Markdown with structured fields and links. In production, code writes the frontmatter and evidence links, a model proposes the interpretation, and administrators own action records. These panels condense the constructed case-study record to show those responsibilities. The complete file is linked in the sources.

Presenter cue: Read the four panels; do not read a full YAML record aloud.

Spoken notes

This is the repository behind the incident we just saw. Digests preserve compacted evidence as JSON and Markdown; raw logs remain in Elasticsearch. Database profiles and monthly journals retain local context. Incidents connect the investigation to administrator actions and outcomes. Error pages collect explanations, cited research and past fixes. Reports summarize across those records. Hosts, services and concepts provide linked context, while source pages record which references are approved and reviewed. These paths are schematic examples of the implemented layout. Most records are Markdown with structured frontmatter and links. Git records their changes so a later investigation can inspect how the knowledge developed.

Spoken notes

This diagram shows the broader prototype research route, not the isolated brief-only design. The agent can read the wiki checkout: error occurrences, digests, database profiles, incidents and approved-source records. It updates the Reference section with cited public findings and team-approved material; it does not own occurrence evidence or resolution history. A separate caveat call uses the public error code, Oracle cause/action text and approved domains, without a wiki clone or estate names. The diagram distinguishes evidence from logs, team-authored context and outside references.

Spoken notes

Oracle’s page is two sentences. The rest comes from sources we keep on a list: 17 today, in three tiers, each a wiki page with its domains and review date. The agent can propose a site; an administrator approves it; only then can research cite it; a monthly pass rechecks. Confluence is on the list as cite-only. Next key: the note taken from our release calendar. Ledger 4.12 on March 20 rewrote the close with literals; 4.12.3 on August 20 restored bind variables; and the restatement job is maintained outside that release train. Remember that last sentence.

Presenter cue: Next key: the Confluence note. Point at the last sentence.

Spoken notes

September 2. The August close ran clean, shared pool free stayed above a gigabyte, resolved. Under it, one more line: the fix lives in the close package and a trigger that matches the close module; any job that still runs the old code path will reproduce it. Thirteen days later one did. The most useful sentence in the whole history, and no model wrote it. The wiki’s job was to keep it where the report would find it.

Presenter cue: Next: the resolve record. Quote the residual risk from the notes.

Spoken notes

Five month-ends, five failed closes, each ending in a restart after midnight. Advance once to see the recorded interventions: restarts, a memory increase, the application ticket and the report. Then the August release fix. Advance again: the August 31 close has zero recorded ORA-04031 failures. The incident was resolved September 2 with the remaining risk written down. End here, at the first clean close. These are constructed case-study records, not proof that a particular change caused recovery.

Spoken notes

Three practical risks deserve attention. Small local models have limited context capacity, and their outputs can be unreliable. Reducing the input can also remove the history needed to interpret an event. During an incident storm, a large batch can make analysis run too long to help the administrator in time. Prompt injection is another possible issue: text in logs or research sources may try to redirect the agent. Treat that material as untrusted evidence, constrain what the agent can do, and test adversarial cases. Structural validation alone does not establish that an interpretation is correct or that an agent resisted injection. These risks are reasons to evaluate the next improvements, not a claim that they have all been solved.

Spoken notes

Three improvements lead into the same validation step. Smarter local inference would let us triage locally and reduce cloud dependence; that work is almost ready. A smarter local model with a larger context window would support more research from more sources. Two-way cloud research would let the model ask for missing context, with a human reviewing and approving what is shared within the intended anonymized boundary. Before expanding, evaluate quality and capacity on more incidents, including failure modes and misleading context. Review the outbound briefs and test the isolated research path end to end. Then scale in-house and explore other infrastructure. These are next steps, not capabilities demonstrated by the screenshots.

Spoken notes

A loop is a job we run repeatedly. Ingest adds the latest events. Report connects them with history. Lint checks the files. Research adds references. Review picks what needs an administrator. Monitoring checks a recovery signal after a change. Monitoring and the past-fixes rebuild are plain Python.

Presenter cue: Purpose of each job here; the next slide adds schedule and models.

Spoken notes

The harness launches the agent CLI and collects the result: prompt, working directory, allowed tools. pi gets no tools; it returns JSON and Python writes the files. Claude Code can read and edit the wiki. Practitioner notes get web tools in a temporary directory with no wiki. Every edit goes through the same publication checks.

Spoken notes

Lock, clean tree. Digests commit first so they survive a model failure. Proposed edits pass result validation, stage checks and lint, then commit or roll back together. The commit carries the run ID. Push is separate; health reports unpushed commits.

Presenter cue: Two presses: commit, then a lint failure and rollback. Talk over the dot; four seconds.

Spoken notes

Of 1,154 model runs, 356 ended without a commit. The September 16 ingest runs were not among them; the strong-tier one for cdb1 validated and committed after about eight minutes of thinking. In 155 cases the local model server was unreachable. Another 116 ran out of context while reasoning and never answered. One proposal opened an incident without a body and failed the schema check. These are different problems, so the ledger records a category for each. That gives retry something to act on, and stops us reading every failed run as a bad diagnosis.

Spoken notes

Here is what the cloud calls can receive. The Monday lookup reads wiki pages, including digests and their log excerpts. The practitioner call gets public error information and approved domains. The pipeline sends no raw Elasticsearch hits or log files. The distinction matters, because the wiki can still contain sensitive material. A structured lookup on a local model is also available. The analyst node and the redacted exchange are in the backups; neither is enabled here.

Presenter cue: Two presses: four chips cross, including the pages the Monday agent opens; then a hit, a credential and a log line hit the line and come back.

Spoken notes

A client supplies its program name, and the listener writes that string into a log. That is the same cdb1 listener log our digest was built from. The string can survive grouping and reach the prompt. That is a prompt-injection path. The structured output limits the damage with a fixed action list, allowlisted codes and Python choosing the paths. The free text is still a problem, because a generated incident title can appear in a later prompt. No test covers this path yet, so I cannot claim the schema solves it.

Spoken notes

Take the run that wrote the September 16 report. Each tick gets a run ID, shared across its stages. It appears in the health log, agent-run telemetry, ingest ledger, git commit and Langfuse session. If something looks wrong in a dashboard, that ID finds the proposal and the commit. Of these stores, only Langfuse holds the prompt and response; the rest hold counts, identifiers and outcomes.

Spoken notes

This is the health command from the morning of September 9. It says unhealthy because one ingest failed in the harness. The output names the digest and prints a retry command. I can tell whether collection stopped, a job failed, or the model server is down without opening several dashboards.

Presenter cue: Point at the retry line. This is what the on-call person sees from cron mail.

Spoken notes

Kibana shows when the jobs ran, how long they took and how they ended. Most failures in this capture are harness errors. The routing view shows which windows woke the model and which were skipped. Nearly 1,200 windows contained only routine events. These screenshots were taken September 10; the September 16 runs show up in the same fields.

Presenter cue: Next key: the routing dashboard. Continue with the routing explanation on that step.

Spoken notes

This is the telemetry row for the run that wrote the September 16 report, rendered as a trace header. The claude adapter ran Opus in agentic mode with the wiki checkout as its working directory. It took 132 seconds and cost about a dollar sixty-six. The million input tokens are the sum over its turns as it read pages; the output was under 16,000. It touched three pages, validated and committed. The report cites the error page and the July incident, which is how we know what it read. Langfuse holds the session under the same ID with the prompt and result.

Presenter cue: Distinguish the rendered header from a Langfuse screenshot; the next backup shows the real UI.

Spoken notes

A quick translation if Oracle isn’t your usual stack. The alert log has major database events and errors. The listener log covers connections. Data Guard handles replication between a primary and standby. An ORA code identifies an Oracle error. The fast recovery area is the disk space Oracle manages for archived redo, flashback logs and backups; when it is full, the archiver stops, and so does redo transport to the standby. None of these logs, by itself, is a complete health check.

Spoken notes

This message says the database refused to open and mentions an old primary that may need reinstatement. It suggests a role-change problem, but the message alone doesn’t establish the cause or tell us whether the database recovered. That’s why the wiki needs both the original event and the later investigation.

Spoken notes

The same error code can matter very differently on two databases. For ORA-19815, the vendor entry is clear and correct: the recovery area is running out of space, and here are five ways to free some. Three of the five were tried in July. What decides the impact is what fills the area, what empties it, and whether the thing that empties it still runs. The useful explanation needs that context, which is why we keep the surrounding events and earlier investigations.

Spoken notes

In structured mode, the model receives text and returns JSON. Python applies it. In agentic mode, the model gets a wiki checkout and file tools, then the resulting edits are checked. Routine ingest uses the local structured path. Notable reports use the agentic path so the model can read more of the existing history. The September 16 report is an agentic run; the paired experiment is the structured path.

Spoken notes

Ingest has three actions: none, open and update. Resolve isn’t in that list; the July 15 resolution and the September 16 reopen were human records. The writer also drops updates for error codes absent from the digest and checks the paths it derives. These are restrictions in the implementation, so they don’t depend on the model following a sentence in the prompt.

Spoken notes

The harness wraps an agent CLI as a subprocess. It supplies the prompt, wiki path and tier, then expects a result file. The modes control the available tools: file access for agentic work, no tools for structured text, and web tools for the practitioner lookup. Duration, exit status, usage and cost are recorded when available. Git operations stay with the orchestrator.

Spoken notes

The prompt starts with the contract and rules, then open incidents, existing error pages and the digest. The digest has a character cap, and the cut is marked. On September 16 the rendered cdb1 digest was 35 kilobytes; the prompt kept 12,000 characters of it. The report fallback builds a different bundle with capped excerpts from several wiki pages. To measure context use, count the complete prompt, including instructions and history.

Spoken notes

An observation links to a digest and document IDs. An interpretation is labelled as an inference. Outside research has its own references and section. Our action history links back to incident records. Lint can check that the citation has the expected form. A person still needs to check that the cited material supports the claim.

Spoken notes

An action record says what someone tried. Monitoring checks a chosen signal during a specified window and reports missing or stale evidence. The review view brings that together for the engineer. Resolving the incident requires a human decision and a record of any remaining risk. The July 15 resolution recorded exactly such a risk. The actual repair happens through the normal operational process; the wiki records it.

Spoken notes

After a host reboot, Filebeat re-sent old events. The pipeline treated them as current, producing false first-seen markers, rate anomalies and wrongly dated pages. Old process IDs helped reveal the replay. The calculations were repeatable, but the input was misleading. Replay deduplication and collection timestamps need attention here. The report agent noticed the problem, but that run was discarded after a harness error.

Spoken notes

The classification rules are ordered patterns with field guards, checked into the repo. Routine events become counters. Errors, lifecycle events and unmatched messages stay visible with an example and document IDs. If something is classified badly, we can inspect the matching rule and change it. The archivelog deletions on cdb1 are routine today; they stayed visible in the context lines, which is how the report noticed the deleted files were six days old.

Spoken notes

This is the digest group for September 16. It has counts, a short message excerpt, the context lines around the first event and document IDs. The context is what kept the routine archiving and deletion lines visible. Structured ingest caps the digest at 12,000 characters and marks the cut. Grouping drops repeated detail; truncation can drop whole groups. The original records remain available for checking either way.

Spoken notes

The registry tracks codes, services and client programs for each database. First-seen checks also distinguish the log source. Rate checks use a seven-day baseline, and silence checks need enough history and a long enough window. These labels depend on the data we’ve collected: on September 16 the digest did not call ORA-19815 new, because the July days were in the registry; the report could then say first since July. A new service is something to inspect, not automatically an error.

Spoken notes

Routing checks for duplicate windows and unchanged content first. Then it considers notable events and mode flags. It can wake the model, force a consolidation, or skip. Manual invocation has its own precedence. Because this is a pure function, we can inspect the decision without making a model call.

Spoken notes

This optional mode is implemented but isn’t scheduled. It would send selected message templates, counts and related codes after replacing database and host names with pseudonyms. A leak check rejects requests that still contain identifiers. A separate researcher reads the request from an exchange repo, without a wiki clone. On return, the local node restores the names, checks the sources and writes Reference. The two active research modes work differently, as shown in the main talk.

Spoken notes

The optional analyst node has cloud and git credentials, but no Elasticsearch credentials. The database-side node has Elasticsearch and git access. They exchange wiki pages and digests through the git remote. A claim must reach that remote before it counts, so both nodes agree on who owns the work. This deployment option isn’t enabled here.

Spoken notes

Three lanes after the same resolve. Held requires fourteen days with no recorded recurrence or reopening. A later occurrence row on the same database gives recurred, with the date. A reopen record gives reopened. The July records on ORA-19815 have been in all three lanes: two recurred within days, two held for nine weeks, and on September 16 the reopen moved all four to the third lane. The labels summarize the recorded outcome; they do not establish what caused recovery or failure.

Presenter cue: Three presses: held fills fourteen quiet cells; the red dot lands from the page’s own table; a person reopens.

Spoken notes

These labels come from incident actions and the error page’s occurrence table. Held means at least fourteen days have passed since resolution, with no recorded recurrence or reopening. That depends on the history the page contains. The table is regenerated from the source records and can be rebuilt. Resolution history stays as originally recorded. This stage copies the action text without a model rewriting it.

Spoken notes

There are two cases. If the cloud report call fails in the harness, the local model gets one attempt using the Material pack. If analyst delegation is enabled, the request is queued before a local placeholder is attempted. That placeholder can also fail. A later analyst report can replace the same path, with earlier versions kept in git. Analyst delegation is disabled in this setup.

Spoken notes

The stats command groups runs by task, adapter, tier and mode. It reports acceptance, rollback rate, duration and cost where available. Failure categories tell retry which runs it can attempt again. These are subprocess-level results; the command doesn’t show every internal agent turn. Separate server failures from invalid proposals before drawing conclusions about a model.

Spoken notes

The eval runs seven real digests through the production prompt, proposal and writer on a disposable wiki. It checks parsing, application, allowed codes and incident consistency, and records flags, duration and usage. Those checks are useful for catching integration failures. Diagnostic quality still needs people to review the same examples against a rubric; that part isn’t in place yet.

Spoken notes

These are historical rollback rates through September 1. The tasks, harnesses and sample sizes differ, so this isn’t a model ranking. A rollback may reflect an infrastructure failure or validation problem. To compare diagnostic quality, we’d need the same inputs, setup and human review criteria for each model.

Spoken notes

These measurements come from the earlier deck and weren’t refreshed from telemetry. The skip percentage counts routing decisions. Recorded API charges leave out local hardware and operating costs. Rollback measures whether a run published, not whether its explanation was right. Those distinctions matter if you want to compare this with another system.

Spoken notes

This is an ingest for cdb1’s standby from September 10, one of the traces Langfuse has held since it was deployed. The header carries the same fields the rendered trace header shows, and the digest sits below the fold. The September 16 report run is in the same store under its run ID.

Spoken notes

V31 follows ORA-19815 on cdb1 through the wiki records and the report that read them. The paired report experiment is real; its prompts, outputs and method are stored with the deck. The incident Markdown is a verbatim excerpt. Earlier versions remain unchanged.