Compliance, customers, and the box-ticking problem!
Almost every penetration testing enquiry opens the same way. Someone needs a pen test. The useful question is the one after that, and we ask it early in scoping: why now?
Three answers cover nearly all of it.
The first is compliance. An auditor, a certification body or a framework has asked for a report. The second is a customer. Someone is refusing to sign a contract, complete a vendor review or release a payment until they see independent testing evidence. The third is fear, usually retrospective. The company has been breached, or watched a competitor get breached, and wants to know what else is exposed.
All three buy something called a penetration test. What they receive differs enormously, and the difference has almost nothing to do with the technology being tested. It tracks the driver.
One of these three reliably produces poor testing. It also happens to be the most common.
Driver one: compliance, where the report is the deliverable
The compliance buyer is the most price-sensitive, for understandable reasons. A company is most of the way through a SOC 2 readiness programme. Somewhere on the evidence list sits "annual penetration test". The report is needed by quarter end, and nobody budgeted for it properly because nobody knew it was coming.
So, the questions arrive as subtraction. Can it be three days instead of eight? Does the API need separate coverage, given the web app is being tested? Do you need accounts at every privilege level, or will one admin login do? Would a vulnerability scan be enough?
Black box or grey box gets discussed as though it were the central decision. For this buyer it usually isn't. If the goal is a document an auditor will accept, the difference between a thorough grey-box assessment and a shallow black-box one is close to invisible, because nobody downstream is going to interrogate it.
Which is why the compliance buyer will often go to a freelancer or a gig marketplace, and why that frequently works. Not works as in finds vulnerabilities. Works as in produces a PDF that clears the evidence request for a few hundred pounds. There is now a supply chain built to serve exactly that, and it is bigger than the market for real testing.
What a scanner-generated pen test report looks like
We have reviewed dozens of third-party penetration testing reports: during compliance readiness work, as part of vendor assessments for clients, and after incidents when we are reconstructing what was known and when. Many are good. A meaningful proportion are vulnerability scanner exports with a cover page.
The tells are consistent. Pages of TLS configuration observations. Missing security headers. Software version disclosures. Generic CVSS text lifted from the tool's finding library, with the same remediation paragraph repeated across eleven findings. Cookie flags. A directory listing. Length is doing the persuasion: eighty pages, colour-coded severity charts, CWE references, an OWASP mapping table.
Then you look for evidence that a human tried to break the application. Often it isn't there. No authenticated testing beyond a login screenshot. No indication of how many roles existed. Nothing on authorisation. Nothing on tenant isolation in a product that is obviously multi-tenant.
Acunetix, Invicti (formerly Netsparker), Nessus and Burp's scanner are capable products, and we run automation ourselves as part of a wider methodology. The problem is not the tool. It is selling the tool's output as an assessment, which erases the distinction between a vulnerability assessment and a penetration test that the buyer is paying for.
A scanner is good at recognising things that resemble weaknesses it already knows about. It will not usually work out that a user in one tenant can increment an object identifier and retrieve another tenant's invoices, because it has no model of who is supposed to see what. It will not chain a permissive password reset flow to an account enumeration issue and arrive at takeover. It will not notice that a hidden form field controls role assignment. Business logic does not look like a vulnerability to a machine, which is part of why autonomous testing has not displaced human testers despite some confident marketing to the contrary.
Those are the findings that matter most in web application and API testing, particularly for multi-tenant SaaS. They are also the findings most reliably missing from a cheap report.
There is a stranger version of this. We have seen reports accepted as compliance evidence while still carrying open Critical and High findings, with no retest and no remediation record. The report demonstrated that testing happened, the control was marked as operating, and the unfixed critical vulnerability went into the evidence pack alongside it. If a test can pass an audit while documenting an exploitable path into the environment, the audit is measuring the activity rather than the outcome.
Why SOC 2 permits this
Not in those words, and the detail matters because the misconception cuts both ways.
The AICPA's Trust Services Criteria do not name penetration testing as a mandate. What the criteria require is evidence that controls are evaluated and that vulnerabilities are detected and addressed. Two carry most of the weight:
● CC4.1 (monitoring activities, mapped to COSO Principle 16) requires the entity to select, develop and perform ongoing or separate evaluations to establish whether the components of internal control are present and functioning.
● CC7.1 (system operations) covers detection and monitoring procedures for configuration changes that introduce new vulnerabilities, and for susceptibility to newly discovered ones.
CC4.1's points of focus do name penetration testing directly, listing it among the types of separate evaluation management might use alongside independent certifications and internal audit assessments. Worth being precise about what that means, though: points of focus are aids the AICPA provides to help design controls against a criterion. They are not criteria themselves, and an organisation is not required to use all of them. So the standard names the technique without compelling it.
That split is widely misread. Scanning tends to evidence CC7.1. Manual testing is what evidences CC4.1, because CC4.1 asks whether your controls survive evaluation rather than whether you ran a tool. Depending on scope, findings will also touch CC6.1 and CC6.6 on logical access, and the rest of the CC7 series on event detection and response.
In practice, it is rare for a Type II examination not to involve a request for testing evidence, and rarer still for scan output alone to close out CC4.1. Treating the absence of the words "penetration test" as permission to skip it is a decision that fails at the audit rather than before it. Our guides on what SOC 2 compliance involves and Type I versus Type II cover the surrounding mechanics.
The flexibility is deliberate and defensible. A nine-person analytics startup should not be held to the testing programme of a payment processor. But flexibility has a side effect: it lets a weak test survive contact with an audit, because the question quietly shifts from "was this test adequate for the risks this organisation faces?" to "is there a test report in the folder?"
Only one of those is hard to answer.
Why ISO 27001 permits it too
Same answer, different mechanism. ISO/IEC 27001:2022 does not use the phrase either, because the standard is risk-based rather than prescriptive. Two Annex A controls do most of the work:
● A.8.8, management of technical vulnerabilities: obtain information about vulnerabilities in the systems you use, evaluate your exposure, take appropriate measures.
● A.8.29, security testing in development and acceptance: security testing processes must be defined and implemented across the development lifecycle. The ISO/IEC 27002:2022 guidance for this control refers to penetration testing directly, alongside code review and vulnerability scanning.
Add Clause 9.1 on monitoring, measurement, analysis and evaluation, and A.5.7 on threat intelligence where relevant, and the shape of the expectation is reasonably clear. Your Statement of Applicability has to justify which controls apply and how they are implemented. If A.8.29 is in scope, which for a software business it usually is, and you have no technical testing evidence, Stage 2 gets uncomfortable. Missing testing evidence is a well-trodden route to a nonconformity.
What often happens instead is quieter. "Annual pen test" goes onto the compliance calendar. Something inexpensive is commissioned every twelve months, timed to land before the surveillance audit. The report is filed. The internal audit programme confirms the control operated. Whether the test was capable of finding the vulnerabilities that would actually hurt this particular organisation goes unasked, because the ISMS is measuring whether the activity happened rather than whether it worked.
That is penetration testing as an administrative task. It looks excellent in an evidence folder. What it demonstrates about security is less obvious.
Why nobody catches it
This is the structural issue underneath both frameworks, and it needs stating carefully, because the sloppy version of the argument is unfair to auditors.
A CPA firm performing a SOC 2 examination is doing skilled professional work, and in the United States a SOC 2 report must be issued by a CPA firm under the AICPA's attestation standards. That is the correct arrangement. The examiner assesses controls, gathers evidence, forms an opinion and stands behind it.
What a CPA licence does not confer is offensive security expertise. It does not equip someone to judge whether a tester adequately assessed broken object level authorisation across 150 API endpoints, whether JWT signature validation could be bypassed through algorithm confusion, whether an OAuth implementation leaks tokens via a redirect, whether tenant isolation was probed or merely mentioned in the methodology section. Those are different disciplines, and nobody expects a penetration tester to form an audit opinion either.
The same applies on the ISO side. A lead auditor can understand an ISMS deeply and assess risk treatment rigorously without being the person you would ask to exploit a race condition in a payment flow.
So, the failure mode is not auditors doing their jobs badly. It is technical evidence entering a process where nobody present has the background to challenge it. A report says, "authentication testing was performed." Which parts? Password reset token entropy and reuse? MFA enrolment and bypass? Session invalidation on password change? Account enumeration? Token replay after logout? Privilege boundaries between roles? Was there one test account, or six across three tenants?
The distance between a tick and a real assessment sits inside questions like those. Absent them, the sequence runs cleanly: the auditor requests a report, receives a report, and records the control as operating effectively, with nobody positioned to notice whether the document describes an assessment or a tool run.
This is the argument for frameworks setting a competence bar for testers rather than leaving it entirely open, which is exactly what the next two examples do.
The audit profession is having this argument already
What makes this more than a vendor complaint is that the same dynamic has surfaced inside the SOC 2 audit market, and the AICPA has become steadily more direct about it.
In February 2026 the Journal of Accountancy ran a piece on how promises of fast and easy SOC reports threaten the credibility of the service. It describes tool vendors that are not CPA firms cultivating referral networks of accounting practices, marketing that alludes to producing thousands of SOC 2 reports a year, and commercial pressure that pushes firms to "check the proverbial box" rather than do deeper critical work. In April, the Professional Ethics Division addressed independence risks where SOC tool providers are entangled with the firms performing the examinations, and a JofA podcast with the AICPA's vice president for assurance and advisory innovation covered quick-turn engagement risks.
Then in May 2026 the AICPA reported that its Peer Review Board and peer review team had been monitoring firms with SOC 2 practices, including high-volume providers working off third-party platforms. The specific risk identified: engagements not designed around the individual service organisation, producing identical reports, identical risk assessments, identical sample sizes and identical testing procedures across different clients. In peer review terms that makes an engagement nonconforming. Structured monitoring and outreach for peer reviews involving SOC 2 practices was set to begin from June 2026, with the February article supplied to reviewers as background material.
The parallel with scanner-driven pen test reports is not exact, since the AICPA is describing audit engagements rather than technical testing. But the shape is the same: a templated artefact standing in for engagement-specific judgement. Accountancy has a formal process for catching it and a word for the result. Penetration testing has accreditation bodies that audit member firms, which is useful, but no profession-wide peer review of individual engagements.
Where box-ticking stops working
Two requirements in common use do close the gap, and they close it in different ways.
PCI DSS removes the interpretive room. Requirement 11.4 in v4.x is specific where SOC 2 and ISO 27001 are deliberately not:
● A defined methodology based on industry-accepted approaches, covering the entire CDE perimeter and critical systems, testing from inside and outside the network, at both network and application layers, with consideration of threats and vulnerabilities experienced in the previous twelve months (11.4.1).
● Internal penetration testing at least once every twelve months and after significant infrastructure or application changes (11.4.2), and external testing on the same cadence (11.4.3).
● Exploitable vulnerabilities and security weaknesses corrected, and testing repeated to verify the corrections (11.4.4).
● Segmentation controls tested at least every twelve months, and at least every six months for service providers (11.4.5, 11.4.6).
● Multi-tenant service providers supporting their customers' external penetration testing (11.4.7).
Note who may perform it. PCI DSS asks for a qualified internal resource or qualified external third party with organisational independence, and states plainly that the tester is not required to be a QSA or an ASV. Independence means the tester is not responsible for managing, developing or maintaining the systems in question. That is a requirement about competence and independence rather than about logos, which is a more useful thing to specify. It does not eliminate poor testing, and PCI assessments still fail for operational reasons, but it removes most of the room to argue about what "tested" meant. Requirement 11.4.4 is the part the compliance buyer should notice: unremediated exploitable findings are not a passing result.
Amazon SP-API puts a competent reader on the other end. Applications built on Amazon's Selling Partner API can reach seller financials, order data and, for restricted operations, personally identifiable information. Amazon's Data Protection Policy therefore sets vulnerability management obligations directly: vulnerability scanning at least every 30 days across systems that process or store Amazon information, penetration testing every 365 days, code scanning before each release, coverage of external-facing infrastructure and internal systems, critical vulnerabilities remediated within 7 days and high-risk within 30. Amazon's implementation guidance asks for testing by qualified security professionals or third-party firms. The policy update effective November 2025 tightened access and credential management further, adding account lockout after ten unsuccessful login attempts, password history retention and API key rotation at least every twelve months.
What makes this instructive is not the policy text. It is that the report is read by a reviewer who understands what should have been in it, and who can reject it.
That changes the buying dynamic completely. A generic web application report gets bounced because the reviewer wants to see the SP-API integration itself: OAuth 2.0 and Login with Amazon flows, token issuance, refresh, rotation, storage and revocation, scope enforcement, the backend systems that process Amazon data, encryption in transit and at rest, retention and deletion behaviour, and evidence that findings were retested. The freelance report that satisfies a SOC 2 evidence request does not survive this review, which is why SP-API is the one compliance-adjacent requirement where we rarely see cheap testing attempted twice. We have written about what SP-API testing actually involves and the specific mistakes that get reports rejected, and we run SP-API DPP audits as a distinct service line because the review really is different.
The lesson generalises well past Amazon, and it sets up the second driver. Where the recipient understands what should have been tested, quality is difficult to fake. Where they don't, the report format becomes the product.
Driver two: the customer, where someone reads it properly
The second driver is commercial. The requirement comes from outside: a prospective customer, an existing customer's annual vendor review, an investor running technical due diligence, or an internal function that has to answer to one of those.
This is the fastest-growing category we see, and the one where report quality has direct revenue consequences. A SaaS business selling into a bank, insurer, retailer or health system discovers the contract will not be signed without independent testing evidence. What "evidence" means then depends entirely on who is asking, and there is a fairly consistent ladder:
1. Confirmation only. A letter or attestation that testing occurred, with dates and provider. Common in mid-market procurement.
2. Executive summary under NDA. Severity counts, scope statement, remediation status. Enough to populate a vendor risk record.
3. Full technical report under NDA. Read by someone who knows what they are reading.
4. Report plus scrutiny. Scope boundary compared against the system description, dates checked against the audit or contract period, remediation evidence requested, retest letter required, tester credentials reviewed, follow-up questions sent to your engineering team.
Standardised questionnaires push the same way. SIG and SIG-Lite from Shared Assessments, the Cloud Security Alliance's CAIQ, and the bespoke enterprise templates that borrow from both all ask about testing frequency, scope, independence and remediation. Sector overlays raise it again: EU financial entities face threat-led testing obligations under DORA, UK public sector and critical national infrastructure testing runs through the NCSC's CHECK scheme, and NHS suppliers face DSPT and DTAC requirements.
Buyers at level four ask better questions than most auditors do. Was every material application and API covered, or one of six? Was testing authenticated? How many roles? Was tenant isolation validated using accounts in separate tenants? Was cloud configuration in scope? Were findings exploited and evidenced, or inferred from a version banner? Have Critical and High findings been remediated and independently retested?
Why accreditation matters here rather than in the audit
Accreditation exists because buyers need some way to distinguish a security testing organisation from anyone capable of producing a PDF, and because most buyers cannot assess technical competence directly. That describes procurement far better than it describes an audit, which is why this is where CREST earns its keep.
CREST is the most widely referenced answer in the UK, EU and parts of Asia. It assesses the company as well as the individuals, and its qualifications are recognised across a network of regulator-linked schemes: the NCSC's CHECK scheme for UK public sector and CNI work, CBEST and STAR-FS in UK financial services, TIBER-EU in the euro area, AASE in Singapore, iCAST in Hong Kong, CORIE in Australia.
DORA is worth stating precisely, because it is frequently overstated. Article 27 requires financial entities to use TLPT testers who are of the highest suitability and reputability, who possess demonstrable expertise in threat intelligence, penetration testing and red teaming, who are either certified by an accreditation body in a Member State or adhere to formal codes of conduct or ethical frameworks, who provide independent assurance around their own risk management, and who carry professional indemnity insurance. Accreditation is one of two routes to one of five conditions. DORA does not name CREST.
Two caveats worth being honest about. Neither SOC 2 nor ISO 27001 requires CREST accreditation, and we would treat any claim that they do as a sales position rather than a fact. And accreditation is a floor rather than a ceiling: it assesses process, people and organisational capability, which is not the same as guaranteeing that a given engagement was excellent. We have argued before that accreditation alone is not proof of testing quality, and we still think so as a CREST member firm.
What it provides is an independently assessed baseline in a market where the buyer often cannot evaluate the work directly.
The commercial point the compliance buyer misses
These two drivers collide more often than people expect, because the same company is usually subject to both. The test bought to clear a SOC 2 evidence request is the test that gets sent to the enterprise prospect six months later.
A thin report does not fail silently at level four. It generates a follow-up questionnaire, which consumes engineering and legal hours, which delays the deal by a procurement cycle, and occasionally ends with the customer requiring a fresh test from a different provider. The cheap test is frequently the expensive one, billed to a different department. Our scoping guide covers how to define a boundary that survives this kind of review, and our analysis of what moves SOC 2 costs covers the same trade-off on the audit side.
Driver three: the breach, where nobody argues about scope
The third driver is fear, and it arrives late.
Something has happened. Ransomware, a compromised administrator account, exfiltration discovered by a customer or a regulator, an application exploited from the internet. Occasionally the trigger is somebody else's incident: a competitor, a supplier, a company in the same vertical.
Nobody in that meeting asks whether the test can be shortened to three days. The questions are how they got in, what else they can reach, and whether it is actually fixed. Scope goes from something to minimise to something to maximise, often within a single call.
Cybersecurity has an awkward economic profile. Before an incident, spending looks discretionary. Afterwards, every hour of unresolved compromise has a price attached to it, and the test that looked expensive in March compares favourably with the forensic retainer, the legal advice, the notification exercise and the downtime.
What we find during incident response and forensic work is rarely exotic. Missing MFA on a remote access path. A service account with domain admin rights nobody could justify. Flat internal networks. A management interface exposed to the internet because a firewall rule was added during a migration and never removed. An unpatched edge device. Credentials in a repository. Logging that existed but retained nothing useful past seven days. Controls that were real in the policy document and absent in production.
Most of that is findable in advance by competent testing. Some of it is findable by a competent scan. Very little of it is findable by a report that never authenticated to anything.
And this is where the three drivers close into a loop. A significant number of the organisations we meet in driver three were already in driver one. They held a current certification. They had an annual penetration test on the compliance calendar and a report in the evidence folder. The testing they had bought was not designed to find the thing that got them, and nobody in the chain between the tester and the auditor was positioned to notice.
Post-breach clients almost never need convincing about testing depth. The cost of that education is the problem.
Black box or grey box is not the first question
Buyers across all three drivers frequently open with the testing model. Reasonable question, wrong position in the queue.
The first question is what you are trying to gain assurance about.
If you want to know what an unauthenticated attacker can find and exploit from the internet with no inside knowledge, black box is the right shape. If you want to find the maximum number of real weaknesses within a fixed number of days, withholding credentials, architecture information and multiple roles works against you.
Take a multi-tenant SaaS platform. Declining to provide accounts in two separate tenants, in the name of realism, removes the tester's ability to assess the authorisation and isolation controls that probably represent your largest single risk. That does not make the test more rigorous. It makes a specific category of critical finding structurally undiscoverable, and the clean result then gets filed as assurance.
Real attackers work over months and do not stop at the scope boundary. A test runs for days under contract. Where the objective genuinely is to test detection and response rather than enumerate vulnerabilities, that is a red team engagement, which is a different exercise with different success criteria.
The model should follow the objective, which requires someone to state the objective.
What a defensible pen test should answer, whatever the driver
Whether the trigger is SOC 2, ISO 27001, PCI DSS, Amazon's DPP, a customer contract or an incident, a report should let a competent stranger answer these without contacting you:
1. What was tested, and what was explicitly excluded, in enough detail to compare against the system description?
2. Was the methodology appropriate to the technology and the risk, and is it named?
3. Was there meaningful manual testing, or is this scanner output with commentary?
4. Were authentication, authorisation, APIs and business logic assessed, across how many roles and tenants?
5. Was the tester qualified, and independent of the people who build and run the systems?
6. Does every material finding carry reproducible evidence and remediation guidance specific to your environment?
7. Were Critical and High findings retested after remediation, with the result documented?
8. Could another technically competent reader reach a reasonable conclusion about your security posture from this document alone?
Where most of those answers are no, a certificate and a well-designed cover page are not assurance. They are a receipt. Our sample report is published redacted for that reason: the format is checkable before you buy.
The report was never the objective
There is nothing wrong with testing because a framework or a customer requires it. External pressure drives a great deal of security work that would otherwise lose the budget argument every year, and we would rather have the requirement than not.
The mistake is treating the requirement as the purpose rather than the trigger.
If you are already granting a testing team access to production, already pulling engineers off roadmap work to support the engagement, already accepting some operational risk and already paying a firm to do it, there is little logic in buying the least informative version of the exercise on the grounds that an auditor might accept it. The marginal cost of doing it properly is a fraction of what you have already committed.
A cheap penetration test is expensive if it tells you everything is fine six months before someone demonstrates otherwise. That is not an argument against SOC 2 or ISO 27001. It is a reminder that an attestation describes controls over a defined period, while an attacker tests them continuously and without notice.
Which makes the more useful question not whether your auditor will accept the report, but whether this test would have found a serious vulnerability had one been sitting in the environment while it ran. Compliance buyers rarely ask it. Customers at level four ask it constantly. Everyone in driver three wishes they had asked it earlier.
Frequently asked questions
Q1: Does SOC 2 require a penetration test?
Ans: Not explicitly. The Trust Services Criteria never name it as a mandate. CC4.1's points of focus do cite penetration testing as a type of separate evaluation, and CC7.1 covers vulnerability detection, but points of focus are design aids rather than criteria. In practice most auditors request testing evidence for Type II and rarely accept scan output alone for CC4.1.
Q2: Does ISO 27001 require a penetration test?
Ans: Not by name. ISO/IEC 27001:2022 is risk-based, but Annex A 8.8 (management of technical vulnerabilities) and A.8.29 (security testing in development and acceptance) make testing the evidence certification bodies expect, and the ISO/IEC 27002 guidance for A.8.29 refers to penetration testing directly. Missing evidence against an applicable control is a common route to a Stage 2 nonconformity.
Q3: Will a vulnerability scan satisfy SOC 2 or ISO 27001?
Ans: Usually not on its own. A scan evidences that you run a tool periodically, which supports SOC 2 CC7.1 and Annex A 8.8. It does not show that a skilled tester attempted to defeat your controls, which is what CC4.1 asks about, and it will not surface authorisation flaws, chained attack paths or business logic errors.
Q4: Can a freelancer's penetration test report satisfy an auditor?
Ans: Frequently yes, and that is the problem rather than the reassurance. Framework auditors assess whether testing occurred and whether findings were managed, not whether the tester was competent. Enterprise customers at the scrutinising end of procurement, and reviewers such as Amazon's for SP-API, apply a materially higher bar and reject reports that clear an audit.
Q5: Does a SOC 2 or ISO 27001 pen test have to be CREST accredited?
Ans: No. Neither framework requires accreditation. It matters where a regulator or customer specifies it: UK public sector work through NCSC CHECK, financial services schemes such as CBEST and STAR-FS, and enterprise vendor reviews. DORA Article 27 requires TLPT testers to be accredited or to follow formal codes of conduct, without naming a specific body.
Q6: How often should we test for SOC 2 and ISO 27001?
Ans: At least annually, and after significant change to the application, infrastructure or threat model. Time the engagement so that testing, remediation and retesting fall inside your audit or observation period rather than after it. Scheduling the test too late to remediate is the most common mistake we see.
Q7: Can one engagement cover both SOC 2 and ISO 27001?
Ans: Usually yes, provided the scope is defined against both the SOC 2 system boundary and the ISMS scope, and findings are mapped to the relevant Trust Services Criteria and the applicable Annex A controls. Mapping is a reporting decision, so agree it during scoping rather than after the test has finished.
Q8: Who is qualified to review a penetration test report?
Ans: Someone with offensive security experience. A CPA firm is qualified to perform a SOC 2 examination and a lead auditor to assess an ISMS, but neither role implies the technical background to judge testing depth, scope adequacy or exploitation quality. If nobody in your review chain can challenge the scope and findings, treat the report's credibility as unverified.
RedSecLabs is a CREST member firm and PCI DSS QSA company delivering penetration testing, SOC 2 and ISO 27001 assurance, and incident response for regulated and high-risk organisations. For a scope that survives a level four procurement review, talk to us or use the penetration testing estimator.
www.redseclabs.com
Media enquiries
[email protected]
+44 20 3996 1505
