Skip to main content

​
About Penetration Tests

Penetration Tests lets your team run one-time external security assessments and receive a generated report with findings and remediation guidance. The feature is available in Security as Penetration Tests.

​
What you can do

  • Start a new one-time penetration test from a target URL
  • Track queued and in-progress jobs in a single list view
  • Open run details for status and execution metadata
  • Triage each finding, including disputing findings you believe are wrong
  • Download report artifacts when complete (Markdown and PDF)

​
Prerequisites

Before using Penetration Tests, make sure:
  1. Your organization has the feature enabled
  2. You have access to the Security section
  3. You know the target URL you want tested

​
Create a penetration test

  1. Go to Security → Penetration Tests
  2. Click Create Report
  3. Enter a Target URL
  4. (Optional) Enter a repository URL for additional context
  5. Continue through checkout to queue the run
If you enter a domain without a protocol (for example example.com), Comp will normalize it to a valid URL format.

​
Track run status

After submission, the run appears in your report list and detail page. Common states include:
  • In queue: Job accepted and waiting to start
  • Running: Assessment is in progress
  • Completed: Artifacts are ready
  • Failed/Cancelled: Run did not complete successfully
For in-progress runs, the detail page shows live progress information when available.

​
Review and dispute findings

Each finding in a completed run has a status you can change from the finding’s detail page: False positive and Won’t fix dispute the finding, and both require a written justification.
  1. Open Security → Penetration Tests and select a completed run
  2. Select the finding you want to dispute
  3. Click Dispute finding
  4. Choose your position — It isn’t a real vulnerability (marks it False positive) or It’s real, but we’re not fixing it (marks it Won’t fix)
  5. Write your reason — be specific about why the finding doesn’t apply. The progress bar fills as you type and reads Ready to submit once the reason is long enough
  6. Click Submit dispute
If you agree with a finding instead, use Acknowledge (real, queued for remediation) or Mark resolved (fixed). Neither requires a written reason, and both save immediately.
The justification is required. Comp will not accept a dispute without one, because a dismissed finding with no recorded reason is not defensible in an audit.

​
What happens to a disputed finding

  • It stays visible in the findings list, marked with its status and dimmed — disputing a finding never hides it
  • It is excluded from the headline severity counts, so your critical/high totals reflect findings you still consider real. Resolved findings are excluded too; Acknowledged ones are still counted, because acknowledging a finding doesn’t fix it
  • You can Reopen it at any time, which puts it back in the counts
  • The justification is written to your audit log, with the user who made the change and when
  • The justification is added to the finding’s status history, so the full triage trail is preserved
  • The justification is shared with the testing agent on future scans of the same target, so a retest evaluates the finding with your context instead of blindly re-reporting it
Disputing a finding is an auditable action. Auditors reviewing your organization can see every status change, who made it, and the reason given. Write justifications you’d be comfortable defending in an audit.

​
Who can dispute a finding

Changing a finding’s status requires the pentest:update permission — held by owners and admins by default. Auditors have read-only access to penetration tests. They can see findings, statuses, justifications and the full status history, but cannot change a status. This separation is deliberate: the party being audited records the disputes, and the auditor reviews them.

​
Access deliverables

When a run completes, open the run detail page to access:
  • View markdown: Opens the generated report content
  • Download PDF: Downloads a customer-shareable report artifact

​
Best practices

  1. Start with externally reachable staging or production-like targets
  2. Run tests after major infrastructure or application changes
  3. Review findings with engineering and assign remediation owners
  4. Triage every finding rather than leaving it Open — an untriaged backlog is hard to defend in an audit
  5. Write disputes that explain why the finding doesn’t apply, not just that it doesn’t. “Internal-only endpoint, not routable from the internet” is defensible; “not applicable” is not
  6. Keep completed PDFs for audit and customer security reviews

​
Troubleshooting

Verify your organization has the feature enabled and that your user has access to Security pages.
Refresh the page and check the run detail view. If status does not change after a reasonable period, contact support with the run ID.
PDF and markdown artifacts are available after the run reaches Completed status.
Changing a finding status requires the pentest:update permission. Auditors have read-only access to penetration tests by design and cannot dispute findings. Ask an owner or admin in your organization to make the change.
False positive and Won’t fix require a written justification of at least 20 characters. Explain why the finding doesn’t apply to your environment — this text is what an auditor reads when reviewing the dismissal.
Your justification is shared with the testing agent on future scans of the same target, but it does not permanently suppress the finding — the agent re-evaluates it with your context. If it reappears, the agent found evidence the issue is reachable. Review the new finding’s proof of concept before dismissing it again.

​
Support

If you need help with Penetration Tests:
  1. Contact support at support@trycomp.ai
  2. Join our Discord community