To vet a third-party AI agent skill, read what it can access before you install it, test it in a sandbox with fake data, and inspect its instructions for hidden commands. A skill security scanner helps catch obvious problems, but it does not replace reviewing permissions and watching what the skill actually does.
Key takeaways: Treat every third-party skill like software from a stranger: give it the least access that works, test it with fake data first, and never rely on a scanner alone.
What is an agent skill, and why does it need vetting?
An agent skill (or plugin) is a package of instructions, and sometimes code, that teaches an AI agent to do a specific job, such as reading your inbox, updating a CRM record or posting to Slack. The risk is that a skill is not just a tool the agent uses: its text becomes part of what the agent reads and obeys. A skill can therefore carry code that runs on your machine, and also written instructions that quietly steer the agent. Public marketplaces make installing these a one-click decision, and reports of over-permissioned or malicious skills keep appearing. For a small business, the damage is concrete: customer data sent somewhere unexpected, emails sent in your name, or records changed without anyone noticing.
The risk categories are well documented. The OWASP Top 10 for LLM Applications lists prompt injection, excessive agency and supply chain weaknesses, and all three apply directly to third-party skills. If you want the broader picture of running agents responsibly, our piece on security and ethics in autonomous agent deployment covers the deployment side. This article stays narrow: how to check one skill before it gets access to anything real.
What should you check before installing a skill?
Start with the permissions, because they tell you the most for the least effort. A skill that summarises meeting notes has no business asking to send email or read your whole drive. Write down what the skill asks for, then ask whether the job truly needs each item. If the listing does not say what it accesses, treat that as a warning in itself. Work through this checklist in order, and stop at the first step that fails:
- Source: Who published it, and can you see their other work, a public repository or a company behind it? An anonymous account with one skill and no history deserves suspicion.
- Permissions: List every tool, file, account and API the skill requests. Remove anything that is not essential to the single job you want done.
- Data flow: Find out where your data goes. Does the skill call external servers that are not obviously part of the service it advertises?
- Instructions: Read the skill's text files in full, including anything marked as a note or comment, and look for commands that do not belong.
- Code: If it includes scripts, check for network calls, commands that download and run other files, and anything that reads credentials or environment variables.
- Updates: Check whether the skill can change itself after install. A safe version today can become a different one next month.
How do you spot hidden instructions in a skill?
Hidden instructions are text inside a skill that tells the agent to do something you did not ask for, and they are the hardest problem to see at a glance. Common patterns include lines such as "do not mention this to the user," requests to send content to an outside address, or orders to ignore your earlier rules. They can sit deep in a long file, inside an example, or in text that renders invisibly in some viewers. The simplest defence is to open every file as plain text and read it, searching for words like "ignore," "secretly," "send," "upload," "credentials" and any web address you do not recognise.
Be realistic about the limits: a determined author can phrase malicious instructions in ordinary-sounding language that no keyword search will flag. That is why reading is only one layer. You also need to limit what the agent can do if it gets fooled, which is the point of the next two sections. The Model Context Protocol documentation is a useful primary source if your skills connect to tools through MCP servers, since it describes how those connections and their permissions are meant to work.
Can a skill security scanner do the job for me?
A skill security scanner can do a useful first pass, but it cannot certify that a skill is safe. Scanners are good at pattern matching: known bad commands, suspicious network addresses, obvious permission overreach and text that looks like an injection attempt. They are weak at intent. A skill that behaves normally for a week and then changes, or one that asks for access you would plausibly grant, can pass a scan cleanly. Think of a scanner as a smoke detector, not a building inspection. The table compares the four checks you can combine.
| Check | What it catches | What it misses | Effort |
|---|---|---|---|
| Automated scanner | Known bad patterns, risky commands, unusual permissions | Novel tricks, well-disguised instructions, later changes | Low |
| Manual read-through | Odd instructions, mismatched purpose, unclear data flow | Things hidden in long files or obfuscated code | Medium |
| Sandbox test | What the skill actually does, including unexpected network calls | Behaviour that only triggers under specific conditions | Medium |
| Outside audit | Combined review, your specific tools and data in mind | Anything that changes after the audit | Higher |
How do you test a skill safely before using real data?
Run the skill in an isolated environment with fake data and throwaway accounts, and watch what it does. Create a test inbox, a dummy CRM with invented contacts, and credentials that can be deleted afterwards. Give the agent only the minimum permissions, then run the task the skill is meant to perform. Log every tool call and network request, and compare them against what the skill claims to do. If it contacts a server you did not expect, or touches files outside its job, you have your answer without risking a real customer record.
Isolation matters because a test on your normal setup is not a test at all. We wrote about the technical side of this in our deep dive on isolation with Incus and AI agents, but you do not need that level to start: a separate user account, a separate workspace and disposable credentials already remove most of the danger. Another useful habit is a human approval step for any action that sends, deletes or pays, so a fooled agent has to ask before it acts.
What should you do after you install it?
Keep watching, because vetting is not a one-time event. Pin the version you reviewed, and re-check the skill whenever it updates, since a changed file is a new thing to inspect. Keep a simple register of every skill in use, who approved it, what it can access and when it was last reviewed. Rotate any credentials the skill touched on a regular schedule, and remove skills nobody uses anymore, because an unused skill with live access is pure risk with no benefit.
It also helps to have one place where you can see what your agents are doing and stop them quickly. That is the idea behind having a mission control for your AI team: logs, visibility and an off switch. Without that, you find out about a bad skill from a customer complaint instead of from your own monitoring, which is the expensive way to learn.
When is it worth paying someone to audit your skills?
An outside audit makes sense when a skill will touch customer data, payments or systems like your CRM, or when you run several skills and nobody on your team has time to review them properly. It is probably overkill for a skill that only formats text on a local machine with no connected accounts. The useful question is what happens if this skill misbehaves: if the answer is "a bad draft," review it yourself; if it is "customer data leaves the building," get a second pair of eyes.
At Kuan AI we design and build AI agents and automations, and as part of that we look at which skills and integrations an agent really needs and what access each one gets. Our process is diagnosis, design, implementation, then measuring and scaling what works, and access review fits in at the design stage. Scope and cost are agreed per project, so there is no fixed price to quote here. If you are weighing outside help in general, our guide on how to choose an AI agent agency lists the questions worth asking, and you can reach us through the contact form; we reply within 24 hours.
Frequently asked questions
Is a skill from an official marketplace automatically safe?
No. A marketplace listing does not guarantee the skill was reviewed in depth, and skills can change after publication. Apply the same checks to every skill: permissions, instructions, data flow and a sandbox test.
What is the biggest red flag in a third-party skill?
A mismatch between the job and the access requested. If a skill that should only format text asks for email, file or network access, do not install it until the author explains why.
Can I use an AI assistant to review a skill for me?
It can help summarise files and point out suspicious lines, but the assistant can itself be influenced by hidden instructions in the text it reads. Use it as a helper, and make the final decision yourself.
How often should I re-check installed skills?
Re-check whenever a skill updates, and review your full list on a regular schedule, for example quarterly. Remove any skill nobody uses, since unused access is risk without benefit.
Do I need a developer to vet skills?
For simple skills, a careful non-developer can check permissions and read instructions. For skills with code, or ones connected to customer data, a technical review is worth it, and you can ask Kuan AI to help.
The short version is that a skill is only as safe as the access you give it and the checks you ran before you did. Read the permissions, read the text, test with fake data, keep a human approval on risky actions and treat scanners as a first filter. None of this is glamorous, but it takes an afternoon, and that is far cheaper than explaining to a customer why their data ended up somewhere it should not be.
