
Start With an Inventory
You can't govern what you can't see, and most agencies' first surprise is how much AI is already running. There are the sanctioned pilots, then the quieter layers: staff pasting constituent data into public chatbots, SaaS vendors switching on AI features inside tools you already own, and procurements described as "AI-powered" without anyone saying what that means.
Build one inventory that covers all of it. For each system, capture:
- System and owner: what it's called and who is accountable for it
- Purpose and users: what it does and who touches it
- Data: what classification of information goes in and comes out
- Hosting and vendor: SaaS, cloud, or on-premises, and who supplies the model
- Change path: who can modify prompts, retrieval sources, or integrations
Federal agencies have a head start here, since OMB guidance already requires inventories of AI use cases. State and local agencies can adopt the same habit without waiting for a mandate. Expect the list to run longer than anyone predicts. In our work across the DMV, it always does.
Put Governance Rules Before Tools
Governance sounds heavy. In practice it's a handful of short documents that answer questions before they become incidents:
- Acceptable use policy: one page, in plain language, telling staff what may and may not go into AI tools
- Data classification rules: which data classifications are allowed in approved tools, and which never leave agency systems
- Approval workflow: how a new AI system gets reviewed, by whom, and how fast
- An owner for every system: a named person, not a committee
- A decision log: brief records of what you approved, what you rejected, and why
Map these onto the frameworks your auditors already know. The NIST AI Risk Management Framework's Govern function gives you the categories, ISO/IEC 42001 gives you a management-system structure, and OMB's AI guidance sets minimum practices for high-risk federal uses. The framework matters less than the habit of writing decisions down. When a tool causes trouble two years from now, the question won't be which framework you cited. It'll be whether anyone can say who approved the tool and under what conditions.
Test the Guardrails Before Launch
Guardrail testing asks a specific question: does the system behave safely when someone tries to make it misbehave? Before any AI system goes live, test for:
- Prompt injection and jailbreaks: can a user, or a document the system reads, make it ignore its rules?
- Data leakage: does the system reveal training data, other users' content, or records the requester shouldn't see?
- Over-permissioned access: can the assistant read more than the person asking the question? This is the single most common gap we find in government deployments.
- Unsafe outputs: what happens when the model is confidently wrong about benefits eligibility, court procedures, or agency policy?
- Failure behavior: what does the system do when it doesn't know, and does it say so?
Test with your own data in a staging environment, not with generic examples. A DMV-area human services agency we worked with found its benefits chatbot cheerfully quoting internal policy drafts, including a proposal leadership had rejected, as if it were current guidance. Generic testing would never have caught that.
Red Team the Whole Pipeline
Guardrail testing checks the model's behavior. Red teaming attacks everything around it, because that's where government AI deployments actually break:
- Retrieval poisoning: if your assistant reads documents, anyone who can edit those documents can feed it instructions
- Tool and plugin abuse: an agent that can send email or update records can be tricked into doing so by a crafted prompt
- Indirect injection: malicious instructions arriving inside the content the AI processes, not from the user
- Supply chain: third-party models, embedding services, and APIs nobody reviewed
- Integration weaknesses: keys and tokens exposed in front ends, unvalidated outputs passed to downstream systems
Not every system needs the same depth of testing. We tier it:
| Tier | Examples | Before launch | Ongoing |
|---|---|---|---|
| High | Citizen-facing assistants; tools informing benefits, licensing, or enforcement decisions | Full pipeline red team | Red team annually and after major changes |
| Moderate | Internal assistants with access to sensitive records | Guardrail testing plus pipeline review | Guardrail testing annually |
| Low | Productivity features on nonsensitive data | Configuration review | Spot checks |
Fantasy Cybersec's AI and LLM security testing service covers both layers, guardrails and pipeline, and reports findings in a format your governance board can act on directly.
Monitor What You Deploy
Launch is the midpoint, not the finish. AI systems change under you: vendors update models, retrieval corpora grow, and users find behaviors nobody tested for. Keep four monitoring habits:
- Log prompts and outputs, with the same privacy care you'd give any system holding constituent data.
- Watch for drift and abuse, including unusual query patterns and repeated attempts to bypass guardrails.
- Keep a kill switch, a documented way to disable the system quickly, and someone authorized to use it.
- Feed AI incidents into your incident response plan, so a prompt injection gets escalated with the same seriousness as a phishing compromise.
Retest after every meaningful change: a new model version, a new data source, a new integration. Teams that treat AI like any other production system, monitored and versioned, avoid the surprise failures that end up in the news.
What This Means for You
AI adoption in government doesn't need to be a gamble. Inventory what you have, write the rules before the tools, test guardrails, red team the pipeline, and monitor what ships. That sequence maps cleanly onto the NIST AI Risk Management Framework and OMB guidance, which makes it defensible to auditors and understandable to staff.
You don't have to build all of it alone. Our AI and LLM security testing service runs guardrail and pipeline testing against your real systems, compliance advisory maps your program to NIST and OMB requirements, and incident response planning adds AI failure scenarios to the rehearsals you already run. Start with the inventory this month. It's the cheapest item on this list, and it improves every decision that follows it.