Paged at 2:21 a.m.Proven and fixed by 2:27

One real incident on a core banking database.

3AM's incident page: core-banking-db, DatabaseReadOnly. Proven, fixed and verified; decided in 5 minutes 20 seconds

3AM's incident page: core-banking-db, DatabaseReadOnly. Proven, fixed and verified; the diagnosis

It proves the cause.It doesn't guess.

The change

read_only went from OFF to ON.

3AM compared the database with what it read before the alert. One setting had changed.

Checked twice

Every write now fails.

A second model scored the claim (supported, 90%), then live checks proved it on the primary.

The symptom

And that's the alert.

DatabaseReadOnly follows from the cause. Proven, not guessed, before anyone was asked anything.

Diagnosis

Proven

read_only on sql changed from OFF to ON

Live checks proved the root cause.

Root causeread_only on sql changed from OFF to ONProven

core-banking-db · sql

✓before the incident: OFF

✓now: ON

↓ caused

ThroughThe primary is read-only: every write failsProven

core-db-01:3306

System 1: supported 90%

✓read_only = 1 on core-db-01:3306 (expected == 1)

✓writable_primary_expected = 1 (expected == 1)

✓up = 1 on core-db-01:3306 (expected == 1)

↓ caused

SymptomDatabaseReadOnly on core-banking-dbObserved

core-banking-db

✓sql is a primary but read-only: writes fail

It shows its workEvery method, and what each found

Changes, dependencies, the alert's own questions, what changed since yesterday, error signatures. What it ruled out is on the page too.

It sees the whole incident: this alert came 25 minutes after its own fix on the app above, and it says so.

Investigation

6 methods

MethodWhat it foundResultTime

Pack checksdb_read_only_primary, of 2 possible causesProven0s

ChangesNo deploys or changes in the last 24 hNothing–

DependenciesNo proven incident upstreamNothing–

Evidence11 questions asked, 1 abnormalNothing2s

What changed3 changes near core-banking-dbLead11s

Error signaturesNo errors readableNothing26s

What it found

Driftread_only on sql changed from OFF to ON

Driftsuper_read_only on sql changed from OFF to ON

Evidencehost_status host=core-db-01 on ssh: shows failed

How this incident unfolded

2 episodes · 3 fixes by 3AM

01:55:59

01:55:59core-bankingFixed and verified

firstCheckerBotFailing

sql.write UPDATE schema_version SET success = 1 …01:56:49

restart_container pl-app-a-tomcat (via ssh)01:56:49

02:21:47

02:21:47core-banking-dbFixed and verifiedTHIS ONE

after 3AM's fixDatabaseReadOnly

set_read_write (via sql)02:26:47

A fix it has rehearsedOne tap from a person

Reversible, rehearsed with a dry run, and sent to the person on call with the evidence. 3AM applies it, then checks it worked.

Proposed fixSET GLOBAL super_read_only = OFF; SET GLOBAL read_only = OFF

Medium riskReversibleDry run passedvia sql

Make the declared primary writable again; 3AM checks writes succeed. Find out who set it read-only before it happens again.

Waiting for one approval

RejectApprove

5m 20s

from the alert to a decision

20s

from the fix to verified

Nothing happens off the recordEvery step, hash-chained

Each event carries the SHA-256 of the one before it, and every segment is sealed and signed. Change one byte and every link after it breaks.

Audit chain

Verified

5 events · this incident

alert

received · 02:21:47

········

diagnosis

proven

········

approval

granted

········

action

executed · 02:26:47

········

fix

verified · 02:27:07

········

Head

Now,howmanyplacesdoyourun?

Set it up onceEvery site has it

One install is the home. Every other site takes its connections, sign-in, people and policy, and stays in step.

Warden signs one licence

In step

Change it on the home; every site follows.

End to end

Sites talk to the home directly. 3AM never sees a secret.

One key

A new site installs with one key, and joins the licence.

In step

Change it on the home; every site follows.

End to end

Sites talk to the home directly. 3AM never sees a secret.

One key

A new site installs with one key, and joins the licence.

229 232 025 108 7203 60 94947 888206 94084680 2 585 277364236640 727247 55 480395509300 792519

72 97 12 74910764 899787 172 0700355 429920279 58058 2 6730 60 7 7802716848701 379704239 45838

707 91137864 0 14635 76136831161829098 5067 632424 0 7134760 26037977 5 38357 16501 1513 6358

1 62 7 52 3490609 48847589 7848 331 0 5239776 42994735 456 22 59 488350 61552 0418593 98377714

14567658 7305606 468848 906212297580 478493 34063 6 970372710939 6088379 5 4683420788 8411

02637 0 2048267332 6070 89647714687 9579 3717 895892711 5973860 0874352 4 203 14 7006085 037 5

4 91 007 1530 077996 57451710430 34 83463 57231 204848369834 25819 29645633160675 711 7985093

950 3797 28 46218271 3915 384 422384 156183 318128 18 92509820129 59 7 0414 9 9 27 717 890

7 005492663364476 93890158 88425526131183 2 45810094 30354974 288425560 4 1299894564601320

4 8044 522 32 8636999294 675605409 5192694 54232 28565313618 5 5531931 355149 383610 6 6 4 58

53356906404 71454160191680 35 60 19029 822 8010030693 96 33 3584320 386148369754543812 8236 84

9 4972833513 697251 2617022 9624100 014 13780506 72067296 47899 349128 66 795353 9465 49238

1745 07531373926 5 907006211315 75865 4012244803803 4253512 904130350 80 971787 34926 533055123

8 616639314124 271 23 12 705601005278902 4385 9 617051134831 848 6101 8912993 0205 87606 42

941 1 290311 41777 02 129 16 42519625 1 80 0545 7505 120 729029 0 76830833556 506064261 8549

769232725 24400800 39692 858566567008941 26 4757858359 1 60336123 1 0 71274892 35 31492850947675

850 1686 266358560228262058249 9 18124898 758225 4 366101138911 13 9350 88359 383331917140

48182982510 185187344144963828550828 7899 493156933261395 967552 841502628911 96 75 1807439407

8000148187 6888732150941 052 16 793 8049909 080083579181 555831544 5 9214872 916674545104101953

73641 9673 8 284 856638 41946 3 0983 1410455 6918085917157267 81011292 4092 26 37428795 6860 02

45 910 157539 0956 30516732486039 9 8 1605 76 17085 359 494324276 3 420165785263353 32 025591

649 267231835021757283933 4 79 29835 791809252799992 6 74817 4134925 7807100928621074449433 1

8471 190941 2 50843 4 9 6768132889 70133 1254544 60 7056792 00 95168 4157 148745 092356417662

0 58315853252 52 404810579 5611524070253 350 85181206928666 15443187103083379 1083653 35 221

928948 14 15 09934422335 90550 97706824 0257099829279687042483 13617 3993 83164 03711101682

12536682103 0 564009 22 1 88 84159 6655180 0630 0663815600 024228029932060010938 043 557 4 29

94 478361 9 997151158 42 5 2083259 9 743 9 3 96 5313809 45165228 469608208 27314 15618 744024

1 9634948 32 1567522 6721 208898525069878 43638665039 03447 48 7306 216601 518039 96735 265206

59381394 617659797 89326365436494167 90 02573160466 8230 674299 04 52631577 62767881803896

591 8 37 73075133 199811804348606535987600851081674001256879 01 9329952292252675260899574153

8 87027092521853132894135 8907717045123 8 12842565 0 84 832 2700 83843837 4 8 6 25593 013 4661

41 31823357 8 9023 2 526 0445 32832129091 798 73 90552 651331 3438426794 95 14738431407 0777 3

5 3174341942 7365 3788 546 6 9 3032 942347623346 58 0 8 43562067370298820 3 52906713 4129092 9

4152114 0 357090977158568 01263 49932133540481 3 2136 9 935222 64 48 3 85582879848968569408

342 832171640050 31 20543785509639 78017335 08080 24 33 73619 6 5 896611 6390927 71053 355

80266823708512716367745607874 353 0547539485535 40531 2105737 8964822339691294007927 99002586302

28779 2015575131339 30 839 742908 21 98757794 3 63975131623209512093719 76710246497926545464 17

88 563 338 14 146362412 809 05 4784242 87071 54028 94074 033 433 4950 97 9755212092 763 7

34 6 213 90065984684 8 77199714390 509304172 96 5082 1 83 85140 3214130 55823 0649452793 7

0012226609446 74756629130743492 148 60 454 94 271751107 987 36143010 0353 6628712 9465 740646

59 8 210515266887 36 6 505 1235267622736182 776218659 2233940544849 754 03803859356 7 023920 6

912170 030 6381 10028780072210 0552 2137538191 3 06 527970025181 0 660 540 50609994176362 7

39420 704 782 903 890 525020241503 17166 9 55 520 071571 28979729262 202 59 481 9 30 305 22

9498 677079856723975851936496 253 3 19 8921456 2 01215 9656 779654848783796794594262 851 56 52

45 47484 20 30410 7395 305 1660538798868 7315 96075427 6188159536 4706323 6083 56 905702 8

2657 30 7988 9891 9122 93364843124268 276572613 1 765767816 50 3582 614157 995880046342409833

6 1063396 0745537335787 65 34128699 6489126852373157533887 198098737917 21577103160 5 21 8772

9880970 303 8714951805 7548 04 28543 14140 688095 48 0762609092674 66388 8 47881006558255 12

One licenceEvery site you run

Enterprise

installs, any purpose

Strategic

Unlimited

installs and instances

Every tier

All of 3AM

every feature, unlimited people

Your next incident is already scheduled.

See 3AM resolve one live in 30 minutes, then start a shadow-mode pilot that changes nothing.

Book a demo Read how 3AM decides