When does AI coding assistance risk become unacceptable?

AI coding assistance becomes unacceptable when a plausible mistake can cross a production boundary before a qualified person proves that the change is safe. The risky part is not whether a model wrote ten lines or ten thousand. It is whether the change can alter identity, move money, reconfigure shared infrastructure, influence clinical decisions, rewrite durable data, or weaken a security control.

I treat those areas as red zones. AI can still help inside them, but it cannot supply the final judgment, approve its own work, or turn a vague request into a production change. Each red zone needs a named human owner, evidence matched to the failure mode, and an approval gate that the delivery pipeline will enforce. A policy in a wiki is not a gate.

This distinction matters because AI output often looks more complete than it is. A generated patch may compile, follow local naming conventions, and include tests while quietly assuming the wrong trust boundary. Reviewers then spend their attention on syntax and style because the code appears familiar. The controls below force attention back onto consequences.

Risk follows authority, not lines of code

The production risk of AI coding assistance is determined by the authority a change gains after deployment. A three-line authorization edit can expose every tenant. A large generated test fixture might never leave a developer machine. Counting generated lines, files, or prompts measures activity, not danger.

Classify a change by asking what it can do if it is wrong and who can detect the failure before harm spreads. I use four practical levels:

• Green changes cannot reach production data or alter a decision, such as isolated test data and internal documentation.
• Yellow changes affect ordinary application behavior but have bounded impact and quick rollback, such as presentation logic behind a tested feature flag.
• Red changes touch a high-consequence boundary: authentication, payments, infrastructure, clinical logic, migrations, or security controls.
• Black changes combine a red-zone boundary with weak recovery, weak observability, or an unreviewable blast radius. They require a different design, not a braver approval.

The black category prevents a common abuse of risk scoring. Teams sometimes label a dangerous migration "high risk," schedule one extra reviewer, and proceed even though they cannot restore the data. No reviewer can approve the absence of a recovery path. The work must change until rollback, containment, or a rehearsed recovery is real.

The same patch can move between levels as its context changes. A generated payment adapter in a disposable sandbox is yellow because it cannot charge anyone. Connecting it to production credentials makes it red. Allowing it to issue refunds across all merchants without a transaction limit or kill switch may make it black. Classify the deployed capability, not the development task.

This also sharpens the difference between review and approval. Review finds defects and improves code. Approval accepts a defined residual risk on behalf of the business or clinical operation. A senior engineer may do both for routine code, but red zones need the approver named by role and domain. Familiarity with the programming language does not make someone qualified to accept a clinical or financial consequence.

Authentication changes need an identity owner

Any AI-assisted change that creates, proves, links, recovers, or revokes identity belongs in the authentication red zone. That includes login handlers, session creation, password reset, multifactor enrollment, account linking, single sign-on claims, service credentials, and authorization checks that decide which identity can act on which resource.

The approval gate should require an identity or security owner who did not author the change. That person needs evidence for the actual abuse cases, not a green unit-test badge. At minimum, the change should demonstrate failed access across tenant boundaries, session invalidation after credential changes, replay resistance where tokens are reused, safe recovery behavior, and denial by default when required identity data is absent.

OWASP Application Security Verification Standard separates authentication, session management, and access control because passing one does not prove the others. Teams blur them anyway. Successful login proves that a user presented acceptable credentials. It does not prove that the resulting session has the right lifetime, that logout revokes it, or that the user may read a particular record. Keep those tests separate so a generated happy path cannot disguise a missing authorization boundary.

A useful gate artifact is an authorization matrix checked into the repository. It names roles down the left, protected actions across the top, and expected allow or deny decisions in each cell. The test suite should exercise every deny cell that protects a tenant, administrator action, credential operation, or sensitive record. Reviewers can then see when a patch changes a decision instead of discovering the policy by reading nested conditionals.

Recovery flows deserve adversarial review because they intentionally bypass the normal proof of identity. Test whether an attacker can enumerate accounts, redirect a reset, reuse a link, retain an old session after recovery, or replace a stronger factor with a weaker one. Support agents and administrators need the same scrutiny. A privileged manual reset is still an authentication protocol, even when its implementation is a help desk screen and a written procedure.

Do not accept "the model used the framework's standard middleware" as evidence. Middleware can be attached to the wrong route, run after data loading, or trust a claim that another service never validated. Inspect the full request path from untrusted input to protected action. If an identity mapping crosses services, record the issuer, audience, subject mapping, clock behavior, and revocation assumption.

The deployment gate should also separate code approval from credential access. The approved artifact should move through the pipeline without giving the model session, its agent, or the developer's prompt environment production secrets. A human approving code does not sanitize secrets already exposed to an external service. If a credential entered a prompt or model-visible log, treat it as disclosed and rotate it.

Payment code must prove money invariants

AI-assisted payment changes are red when they can authorize, capture, refund, settle, price, tax, credit, or reconcile real value. The gate must prove money invariants under retries and partial failure, because most costly payment defects are valid-looking operations executed twice or recorded in only one system.

Start with explicit invariants in ordinary language. A payment request has one merchant, one currency, one amount expressed in the smallest supported unit, and one stable idempotency key. A retry cannot create a second charge. A refund cannot exceed the captured amount after prior refunds. A local "paid" state cannot appear unless the external operation has a durable reference. These claims should become tests and database constraints where possible.

Generated payment code often handles the success response and treats every error as a clean failure. Production networks do not cooperate. A client can time out after the processor accepted a request. The application now has an unknown outcome, not a failed payment. Retrying with a new idempotency key can charge twice. Mark the operation pending, query by the original key or reference, and reconcile before deciding what happened.

Webhooks add another uncertainty boundary. Authenticate the sender, retain the original event identifier, acknowledge only after durable acceptance, and make processing safe to repeat. Do not assume delivery order matches business order. A refund notice can arrive before a delayed capture notice, and two workers can see the same event. The handler should record facts and let an explicit state machine decide whether a transition is allowed.

The approval gate should require a payments owner plus an engineer who understands the storage transaction. They need a test matrix covering duplicate requests, timeouts before and after acceptance, out-of-order callbacks, invalid signatures, currency mismatches, rounding boundaries, partial refunds, and reconciliation after a worker crash. Use processor sandboxes for protocol behavior, but test your own durable state with injected failures. Sandboxes rarely reproduce every ordering problem.

Do not let AI invent payment rules from names such as available_balance or settled. Those words carry business and accounting meanings that differ across systems. Write a short state-transition table that lists each permitted transition, the event that causes it, the durable evidence required, and whether an operator can reverse it. Reject transitions outside the table even when the proposed code seems reasonable.

The release gate should cap exposure. Route a small, observable slice through the new path, define an automatic stop condition, and keep the old path available until reconciliation agrees. A feature flag alone is insufficient if disabling it leaves accepted callbacks unprocessed or splits one transaction across two implementations. The rollback plan must cover in-flight money, not merely application binaries.

Infrastructure changes require a bounded blast radius

Infrastructure enters the red zone when a generated change can alter production identity, networking, encryption, compute capacity, persistence, backups, logging, or deployment permissions. Approval should depend on a machine-readable plan, a bounded target, and a recovery test, not on whether the configuration file looks conventional.

For declarative infrastructure, preserve the exact plan that reviewers approve and apply that artifact without regenerating it under different inputs. The gate should fail if the target account, region, workspace, or resource set differs between planning and application. Reviewers should see replacements, deletions, permission expansions, public exposure, and changes to data-bearing resources called out separately.

Generated configuration has a specific failure pattern: it copies a valid example but misses the surrounding constraints that made the example safe. A broad identity policy may be acceptable in a disposable account and disastrous in a shared production account. A network rule may expose a service because the example assumed another firewall layer. The syntax validator cannot see those missing assumptions.

Require policy checks that answer concrete questions. Can this plan create a public listener? Can it grant wildcard actions? Can it disable encryption or retention? Can it destroy or replace a stateful resource? Can it change the pipeline role that enforces these checks? A proposed change to the guardrail itself must never pass because the revised guardrail approves it. Review that control through an independent path.

The recovery evidence should match the resource. For stateless compute, a known-good redeployment may be enough. For a database, queue, identity store, or encryption key, a rollback command does not prove recovery. Restore a backup into an isolated environment, verify application-level reads, and record the recovery duration and any data gap without inventing a comforting target.

Use progressive deployment to shrink uncertainty, but do not confuse a canary with containment. A global permission policy, shared schema change, or destructive storage operation can affect every instance even if one application replica receives traffic. Determine the blast radius from the changed resource. Then require an approver who owns that radius and an operator who can stop the rollout.

Clinical logic needs traceability and clinical sign-off

Software that influences diagnosis, triage, medication, dosage, alerts, care pathways, or the presentation of clinical facts belongs in the clinical red zone. A licensed or formally designated clinical owner must approve the intended behavior, while engineering separately approves implementation and operation. Neither approval substitutes for the other.

The gate begins with an exact intended-use statement: who uses the output, for which patients, with which inputs, at what point in care, and what decision the output may influence. Without that boundary, reviewers cannot decide whether a failure is a nuisance or a patient hazard. AI is especially prone to filling gaps with plausible assumptions, so ambiguity in a ticket is itself a stop condition.

Create traceability from each clinical requirement to code, test cases, and displayed behavior. If a rule says an alert fires under defined conditions, the evidence should include boundary values, missing data, conflicting data, units, timing, overrides, and the text shown to the user. A correct internal calculation can still cause harm if the interface hides uncertainty or presents stale information as current.

Clinical correctness is different from software correctness. Unit tests can prove that code implements a formula. They cannot prove that the formula fits the intended population, that the source data has the assumed meaning, or that the workflow gives a clinician time to act. The clinical reviewer owns those questions. The engineer owns deterministic execution, data lineage, failure handling, audit records, and safe behavior when inputs are unavailable.

For AI-generated changes, store the approved requirement and evidence, not a raw prompt as a substitute for either. Prompts are useful development records, but they do not define clinical intent precisely enough to validate future changes. A reviewer must be able to connect a production behavior to a controlled requirement without reproducing a conversation with a model.

The release gate needs monitoring tied to clinical hazards. Track missing inputs, suppressed alerts, override paths, stale data, unexpected rule frequency, and discrepancies between displayed and stored values. Define who receives each signal and what action they can take. If the only response is "investigate later," the system lacks an operational control.

Migrations are safe only when recovery is demonstrated

Data and schema migrations become red when they transform durable production records, change compatibility between running versions, rebuild indexes with operational impact, or remove information. The approval gate must prove forward compatibility, restart safety, reconciliation, and recovery with production-shaped data.

The popular recommendation to "always make migrations reversible" is too shallow. A down migration can reverse a schema statement while losing the values already transformed or deleted. It can also fail after new application versions have written data the old schema cannot represent. What you need is a recovery plan, which may be rollback, roll-forward repair, restore, or a combination. Name the one you will actually use.

Prefer expand-and-contract changes when multiple application versions may run together. Add the new structure without removing the old one, deploy code that can tolerate both, backfill in bounded batches, compare results, switch reads, and remove the old structure only after the compatibility window closes. Each stage should be independently deployable and observable.

A migration gate can require a reproducible rehearsal:

1. Restore a recent production-shaped snapshot into an isolated environment with sensitive values protected.
2. Run the migration using the same artifact, permissions, timeouts, and orchestration intended for production.
3. Interrupt it at several batch boundaries, restart it, and verify that repeated work does not corrupt results.
4. Compare row counts, sums or hashes suited to the data, rejected records, and application-level reads before and after.
5. Exercise the declared recovery path and record what remains manual.

Avoid a single checksum over an entire mutable table. It tells you that something differs but not whether the difference is expected or where repair should begin. Reconciliation should follow business partitions and invariants: per tenant, per day, per currency, or per record type. Keep rejected records with reasons so operators can repair them without rerunning the whole transformation blindly.

Generated migration code deserves extra scrutiny around nulls, defaults, character encoding, time zones, units, duplicate keys, and implicit casts. Models infer the common case from names. Your historical data contains the exceptions created by old releases, manual repairs, and integrations that no longer exist. Sample those exceptions deliberately.

Do not approve a destructive final stage merely because the earlier stages ran cleanly. Removal changes the recovery options. Require a separate approval after the observation window, with evidence that no supported code reads or writes the old representation and that retained backups meet the actual recovery need.

Security controls cannot approve their own weakening

Changes to authorization policy, secret handling, encryption, audit logging, input validation, dependency controls, security monitoring, or delivery protections are red even when they do not touch a product feature. Their gate must be independent of the control being changed.

This independence rule is easy to state and easy to violate. Suppose an AI-generated patch modifies the pipeline rule that blocks critical dependency findings, and the same pull request passes because the modified rule no longer blocks them. The green check is meaningless. The change used its own new definition of safety to approve itself.

Protect control definitions with separate ownership and enforcement. Changes to branch protections, policy engines, scanner thresholds, logging exclusions, privileged roles, and deployment gates should require review from the security owner and apply only after the old control approves the transition. Where that is impossible, use an administrative path with explicit records and a second person.

NIST Secure Software Development Framework treats protecting software and producing well-secured releases as ongoing practices, not one final scan. That framing is right. A generated patch can pass a scanner and still remove a log field needed for incident response, expand a trust boundary, or turn a hard failure into an ignored warning. Evidence needs to cover prevention, detection, and recovery relevant to the changed control.

Reject explanations such as "this exception is temporary" unless the exception has an owner, narrow scope, expiration, and a tracked removal condition. A permanent exception often begins as a deadline workaround. The gate should enforce expiry rather than rely on someone remembering it after the release.

Secrets require a separate rule. Never place production secrets, private patient data, payment data, or proprietary source outside the approved model boundary. Redaction helps only when the redactor understands the formats and runs before transmission. If sensitive data reaches an unapproved model or log, incident handling starts immediately; deleting the chat does not retract the disclosure.

Approval gates need evidence and separation

A useful red-zone gate is an enforceable contract with five parts: scope, evidence, approver, deployment constraint, and recovery authority. If any part is vague, the gate becomes a ceremonial checkbox that reviewers learn to click.

Scope identifies paths, resources, data classifications, and semantic changes that trigger the gate. File paths alone are weak because shared libraries and generated configuration can alter a red zone indirectly. Combine path rules with ownership metadata, infrastructure plan analysis, database operation detection, and a pull-request declaration that reviewers can challenge.

Give developers a way to dispute an automatic classification, but never let the author silently downgrade it. The dispute should name the proposed zone, explain the bounded consequence, and receive approval from the owner of the original zone. This keeps false positives from turning the policy into noise while preserving a record of why the gate did not apply. Periodically review those exemptions for repeated patterns that should become better classification rules.

Evidence must correspond to the failure. Authentication needs deny-case and session tests. Payments need retry, reconciliation, and invariant tests. Infrastructure needs a reviewed plan and recovery evidence. Clinical logic needs controlled requirements and clinical validation. Migrations need rehearsal and reconciliation. Security controls need independent approval. A generic coverage percentage does not replace any of them.

Approvers should be roles with named current members, not whoever happens to be senior. Require at least one domain approver independent of the author for every red-zone change. For clinical behavior or material financial rules, add the designated operational owner. Prevent the AI agent, service account, or author from satisfying approval through an automated identity.

Deployment constraints limit what happens after approval. Bind approval to a commit and artifact digest so later regeneration cannot slip through. Separate production credentials from the coding environment. Use staged exposure where it truly bounds impact, define stop conditions, and make the person on duty able to halt the rollout without waiting for the author.

Recovery authority names the person who may disable, roll back, restore, or roll forward and gives that person the required access before release. An incident is the wrong time to discover that only an absent administrator can restore a backup. Rehearse access as well as commands.

This policy fragment shows the minimum record I expect a delivery system to enforce:

zone: payments
change_digest: "sha256:<artifact-digest>"
required_approvals:
  - role: payments_owner
    independent_of_author: true
evidence:
  - retry_matrix
  - reconciliation_report
deployment:
  max_exposure_percent: 5
  stop_condition: "duplicate_or_unreconciled_transaction"
recovery_owner: "on_call_payments"

The exact syntax is unimportant. The failure it prevents is approval drifting away from the artifact and risk being accepted without evidence or an accountable owner. Store this record with the release so an incident reviewer can reconstruct what was known and who accepted the remaining risk.

Human-in-the-loop must mean accountable control

Human-in-the-loop only reduces AI coding assistance risk when the human has the competence, information, time, authority, and independence to stop the change. A tired reviewer clicking approve on a large generated patch is human presence, not human control.

Keep generated changes small enough to reason about. Ask for one bounded behavior, require the model or developer to state assumptions, and reject unrelated cleanup in a red-zone patch. Run deterministic formatters and analyzers, but make the author explain trust boundaries, failure states, and recovery in their own words. If the explanation cannot survive questions, the code is not ready.

Measure the gate by outcomes that reveal its health. Track how often reviewers request substantive changes, how often evidence is missing, whether emergency bypasses expire, whether rehearsed recovery works, and whether one person repeatedly approves unfamiliar domains. Do not reward approval speed alone. Fast approval can mean a clear change, or it can mean nobody looked.

AI can be useful before the gate. It can enumerate edge cases, draft tests, compare a change against a state table, and point out inconsistencies. Treat those outputs as leads. The same model that created a defect may confidently create a test that confirms its own mistaken assumption, so independent requirements and human judgment remain necessary.

SaaS Production uses AI with experienced engineers and a Human-in-a-Loop approach to shorten delivery while retaining human control. For red-zone work, that approach should be visible in the artifact trail: who set the boundary, which evidence they inspected, what exposure they accepted, and how the team can recover.

There is no responsible percentage of code that AI may write across every system. Set the red zones by consequence, encode the gates in delivery tooling, and refuse deployment when recovery exists only as a sentence in a ticket. If a team cannot name the person allowed to stop a dangerous change, the change is not ready for production.

Frequently Asked Questions

Can AI write code for authentication systems?

Yes, but authentication code belongs in a red zone. An independent identity or security owner should approve it after deny cases, session behavior, recovery paths, and authorization boundaries have been tested.

What makes AI-generated payment code risky?

Retries, delayed callbacks, and partial failures can create duplicate or unreconciled transactions even when the happy path looks correct. The gate should prove idempotency, state transitions, amount and currency invariants, and recovery for unknown outcomes.

Should AI agents receive production credentials?

No. Keep production credentials outside the model and coding-agent environment, and let an approved artifact move through a separate delivery pipeline. If a secret enters a prompt or model-visible log, treat it as disclosed and rotate it.

Is human code review enough for AI-generated code?

Only when the reviewer has the right domain competence, evidence, time, and authority to stop the release. Red zones also need enforced deployment constraints and a recovery owner, because a pull-request approval alone does not contain production harm.

How should teams classify AI coding risk?

Classify the deployed authority and consequence, not the amount of generated code. A small change is red when it can alter identity, money, shared infrastructure, clinical decisions, durable data, or a security control.

What evidence should an infrastructure approval include?

Review the exact machine-readable plan that will be applied, with target account and resource scope fixed. Call out deletions, replacements, public exposure, permission expansion, stateful resources, and proof that recovery works.

Can a reversible database migration still be unsafe?

Yes. A down migration may reverse the schema while losing transformed data or rejecting values written by the new application. Require a rehearsed recovery plan, restart-safe batches, compatibility across running versions, and business-level reconciliation.

Who approves AI-assisted clinical logic?

A designated clinical owner approves the intended behavior and clinical consequences, while engineering approves implementation and operation. The release also needs traceability from controlled requirements to tests and the exact behavior shown to users.

How do teams prevent security controls from approving themselves?

Use separate ownership and an independent enforcement path for changes to policies, scanner thresholds, privileged roles, logging exclusions, and delivery gates. The old control should approve the transition whenever possible.

When should an AI-assisted change be blocked completely?

Block it when the blast radius is unbounded, recovery has not been demonstrated, required evidence is missing, or no qualified owner will accept the residual risk. Those are design failures, not reasons to add one more hurried reviewer.