How Do Compliance Teams Document Application-Level Tests Like Role-Based Access Automatically?

Use automated testing tools with screenshot capture that document: 1) User login with specific role, 2) Attempted actions beyond permissions, 3) System denial response, 4) Audit log entries. AI agents can perform these tests quarterly and generate timestamped evidence packages automatically.

---

What Are Application-Level Tests?

Definition

Application-level tests verify that your application's security controls work as designed by testing the actual user interface, workflows, and business logic—not just infrastructure configurations.

Examples:

• Role-based access control (RBAC) testing
• Data privacy controls verification
• Multi-factor authentication enforcement
• Session timeout validation
• Audit logging verification

Why They're Required for SOC 2

SOC 2 auditors require application-level tests because:

1. Infrastructure ≠ Application: AWS security groups don't prove your app enforces permissions
2. Visual proof needed: Auditors want to see "Access Denied" messages, not just code
3. Operational effectiveness: Tests prove controls work over time (quarterly testing)
4. Trust Service Criteria: CC6.1, CC6.2, CC7.2 specifically require application-level evidence

The Documentation Challenge

What auditors need to see:

• Screenshots showing test execution
• Step-by-step procedures followed
• Expected vs actual results
• Timestamps proving quarterly testing
• Tester identity and role
• Pass/fail determination

Manual approach time: 2-4 hours per control per quarter

---

The Manual Documentation Process (Traditional Approach)

Step 1: Create Test Plan (30 minutes)

Document includes:

## CC6.1 RBAC Test Plan

**Objective:** Verify that users with "Viewer" role cannot access admin features

**Test Environment:** Production

**Test Date:** 2025-02-14

**Tester:** compliance@company.com

**Prerequisites:**
- Test user account: viewer-test@company.com (Role: Viewer)
- Admin feature URL: /admin/users
- Access to audit log system

**Test Steps:**
1. Login as viewer-test@company.com
2. Navigate to /admin/users
3. Verify access denied message appears
4. Check audit log for access denial event
5. Document results with screenshots

**Expected Result:** Access denied with 403 error
**Pass Criteria:** User cannot access admin features

Step 2: Execute Test (45 minutes)

Manual process:

1. Login (5 min)

   • Open application in browser
   • Login with test credentials
   • Take screenshot of logged-in state showing role

2. Attempt restricted action (10 min)

   • Navigate to admin feature
   • Try to access restricted page
   • Take screenshot of URL bar
   • Take screenshot of error message

3. Verify technical details (15 min)

   • Open browser DevTools
   • Navigate to Network tab
   • Retry the restricted action
   • Screenshot showing 403 HTTP response
   • Screenshot showing response headers

4. Check audit logs (15 min)

   • Login to logging system (CloudWatch, Datadog, etc.)
   • Search for access denial events
   • Filter by user and timestamp
   • Screenshot relevant log entries
   • Highlight key fields (timestamp, user, action, result)

Step 3: Document Results (60 minutes)

Create comprehensive report:

1. Organize screenshots (20 min)

   • Rename files with descriptive names
   • Order chronologically
   • Annotate with arrows/highlights

2. Write descriptions (25 min)

   • Describe what each screenshot shows
   • Explain expected vs actual behavior
   • Note any discrepancies

3. Format for auditors (15 min)

   • Create Word/Google Doc
   • Add control ID (CC6.1)
   • Include test metadata
   • Insert screenshots with captions

Step 4: Upload to GRC Platform (15 minutes)

1. Export document as PDF
2. Navigate to Vanta/Drata
3. Find correct control
4. Upload evidence
5. Add test date and tester

Total manual time per control: 2.5 hours Quarterly cost (at $200/hour): $500 Annual cost (4 quarters): $2,000

---

Automated Documentation Approach

How Automation Works

Component 1: Browser Extension Test Recorder

What it captures automatically:

{
  "test_session": {
    "test_id": "cc6_1_rbac_2025_q1",
    "control_id": "CC6.1",
    "start_time": "2025-02-14T10:30:00Z",
    "end_time": "2025-02-14T10:37:00Z",
    "tester": "compliance@company.com",
    "environment": "production",
    "browser": "Chrome 121.0.6167.85",

    "actions": [
      {
        "step": 1,
        "timestamp": "2025-02-14T10:30:15Z",
        "action": "login",
        "user": "viewer-test@company.com",
        "role": "Viewer",
        "screenshot": "01_login.png"
      },
      {
        "step": 2,
        "timestamp": "2025-02-14T10:31:20Z",
        "action": "navigate",
        "url": "/admin/users",
        "http_status": 403,
        "screenshot": "02_access_attempt.png"
      },
      {
        "step": 3,
        "timestamp": "2025-02-14T10:31:22Z",
        "action": "error_displayed",
        "error_message": "Access Denied - Insufficient Permissions",
        "screenshot": "03_access_denied.png"
      },
      {
        "step": 4,
        "timestamp": "2025-02-14T10:32:10Z",
        "action": "audit_log_check",
        "log_entry_id": "log_abc123",
        "screenshot": "04_audit_log.png"
      }
    ],

    "test_result": "PASS"
  }
}

Component 2: AI Documentation Generator

AI processes the captured data and generates:

Natural language descriptions:

"At 10:30:15 UTC, user viewer-test@company.com with Viewer role successfully authenticated to the application. At 10:31:20 UTC, the user attempted to navigate to /admin/users. The application returned HTTP 403 Forbidden with error message 'Access Denied - Insufficient Permissions'. Audit log entry log_abc123 confirms the access denial was logged at 10:31:22 UTC."

Control mapping:

{
  "control": "CC6.1 - Logical Access Controls",
  "test_type": "RBAC - Access Denial Verification",
  "tsc_criteria": "Common Criteria 6.1",
  "test_result": "PASS",
  "evidence_quality": "High",
  "auditor_notes": "Complete evidence package with screenshots and audit logs"
}

Component 3: Report Generator

Automatically creates PDF with:

1. Cover Page

   • Control ID: CC6.1
   • Test Date: 2025-02-14
   • Tester: compliance@company.com
   • Result: PASS

2. Test Summary

   • Objective statement
   • Test methodology
   • Pass/fail determination
   • Risk assessment

3. Step-by-Step Documentation

   • 4 steps with screenshots
   • AI-generated descriptions
   • Timestamps for each step
   • Technical details (HTTP codes, URLs)

4. Audit Log Evidence

   • Screenshot of log entry
   • Extracted log data
   • Verification that event was logged

5. Appendix

   • Test metadata (browser, IP, session ID)
   • Raw test data (JSON)
   • Certification statement

Total automated time per control: 8 minutes (7 min test + 1 min review) Quarterly cost (at $200/hour): $27 Annual cost: $108 + $1,788 (tool) = $1,896

Annual savings: $104 per control

---

Step-by-Step: Setting Up Automated Documentation

Phase 1: Initial Setup (One-Time, 45 minutes)

Step 1: Install Browser Extension (5 minutes)

1. Navigate to Chrome Web Store
2. Search for "Screenata" or install from screenata.io
3. Click "Add to Chrome"
4. Pin extension to toolbar
5. Login with your account

Step 2: Configure Test Templates (20 minutes)

Create reusable templates for each control:

{
  "templates": [
    {
      "template_id": "cc6_1_rbac_viewer",
      "control": "CC6.1",
      "name": "RBAC - Viewer Role Access Denial",
      "frequency": "quarterly",
      "test_users": [
        {
          "email": "viewer-test@company.com",
          "password_vault_key": "viewer_test_password",
          "role": "Viewer"
        }
      ],
      "test_steps": [
        "Login as Viewer user",
        "Attempt to access /admin/users",
        "Verify 403 error displayed",
        "Check audit log for denial"
      ],
      "expected_result": "Access denied with 403",
      "screenshots_required": 4
    },
    {
      "template_id": "cc6_1_rbac_editor",
      "control": "CC6.1",
      "name": "RBAC - Editor Role Access Denial",
      "frequency": "quarterly",
      "test_users": [
        {
          "email": "editor-test@company.com",
          "role": "Editor"
        }
      ],
      "test_steps": [
        "Login as Editor user",
        "Attempt to access /admin/billing",
        "Verify 403 error displayed",
        "Check audit log for denial"
      ],
      "expected_result": "Access denied with 403"
    }
  ]
}

Step 3: Connect Audit Log System (15 minutes)

For CloudWatch:

// Configure CloudWatch integration
{
  "log_integration": {
    "provider": "aws_cloudwatch",
    "log_group": "/aws/application/prod",
    "region": "us-east-1",
    "auth_method": "iam_role",
    "role_arn": "arn:aws:iam::123456789012:role/ScreenataLogReader"
  }
}

For Datadog:

{
  "log_integration": {
    "provider": "datadog",
    "api_key": "dd_api_key_xxxxx",
    "site": "datadoghq.com",
    "log_query": "service:application @http.status_code:403"
  }
}

Step 4: Set Up Vanta/Drata Export (5 minutes)

1. Navigate to Integrations tab
2. Select Vanta or Drata
3. Enter API key
4. Map control IDs
5. Test connection

Phase 2: Recording First Test (7 minutes)

Step 1: Start Recording (30 seconds)

1. Click Screenata extension icon
2. Select template: "CC6.1 RBAC - Viewer Role"
3. Click "Start Recording"
4. Extension begins capturing

Step 2: Execute Test (5 minutes)

1. Login (90 seconds)

   • Navigate to your app
   • Login with viewer-test@company.com
   • Extension auto-captures screenshot

2. Attempt restricted access (60 seconds)

   • Click on admin menu (or navigate to /admin/users)
   • Access denied message appears
   • Extension captures URL + error

3. Verify DevTools (60 seconds)

   • Open DevTools (F12)
   • Go to Network tab
   • Retry admin access
   • Extension captures 403 response

4. Check audit logs (90 seconds)

   • Navigate to logging dashboard
   • Extension automatically queries logs
   • Highlights relevant entry
   • Captures screenshot

Step 3: Stop Recording (30 seconds)

1. Click "Stop Recording"
2. Extension shows preview: "4 steps captured"
3. Review quick summary

Step 4: AI Processing (60 seconds)

Automated processing:

• AI analyzes screenshots
• Generates step descriptions
• Maps to CC6.1 control
• Creates PDF report
• Formats for auditor review

Output shown:

• ✅ 4 screenshots captured
• ✅ Audit log entry found
• ✅ Test result: PASS
• ✅ PDF generated: CC6_1_RBAC_2025-Q1.pdf

Phase 3: Review and Export (1 minute)

1. Click "Review Evidence"
2. Scan through PDF preview
3. Verify screenshots are clear
4. Click "Export to Vanta"
5. Evidence automatically uploads

Total time: 8.5 minutes

---

Comparison: Manual vs Automated

Activity	Manual Time	Automated Time	Time Saved
Test planning	30 min	0 min (template)	30 min
Test execution	45 min	5 min	40 min
Screenshot capture	15 min	0 min (auto)	15 min
Log retrieval	15 min	0 min (auto)	15 min
Documentation writing	60 min	1 min (AI)	59 min
Report formatting	15 min	0 min (auto)	15 min
Upload to GRC	15 min	1 min	14 min
Total per test	195 min	7 min	188 min

Efficiency gain: 96.4% time reduction

---

Advanced Testing Scenarios

Scenario 1: Multi-Factor Authentication Testing

Control: CC6.1 - MFA enforcement

Test objective: Verify MFA required for admin access

Automated test flow:

{
  "test": "mfa_enforcement",
  "steps": [
    {
      "action": "login_without_mfa",
      "user": "admin@company.com",
      "mfa_provided": false,
      "expected": "mfa_prompt_displayed",
      "screenshot": "01_mfa_prompt.png"
    },
    {
      "action": "skip_mfa",
      "expected": "access_denied",
      "screenshot": "02_mfa_required.png"
    },
    {
      "action": "login_with_mfa",
      "mfa_code": "generated",
      "expected": "access_granted",
      "screenshot": "03_mfa_success.png"
    }
  ]
}

Evidence generated:

• Screenshots of MFA prompt
• Screenshot of access denial without MFA
• Screenshot of successful login with MFA
• Audit logs showing MFA events

Scenario 2: Session Timeout Testing

Control: CC6.6 - Session management

Test objective: Verify sessions timeout after 30 minutes

Automated test flow:

{
  "test": "session_timeout",
  "steps": [
    {
      "action": "login",
      "timestamp": "10:00:00",
      "screenshot": "01_logged_in.png"
    },
    {
      "action": "wait",
      "duration_minutes": 31,
      "screenshot": "02_waiting.png"
    },
    {
      "action": "attempt_action",
      "timestamp": "10:31:00",
      "expected": "session_expired_redirect",
      "screenshot": "03_session_expired.png"
    }
  ]
}

AI-powered waiting: Extension monitors session in background, captures timeout automatically

Scenario 3: Data Privacy Controls

Control: CC6.1 - PII access restrictions

Test objective: Verify PII masked for users without data access role

Automated test flow:

{
  "test": "pii_masking",
  "steps": [
    {
      "action": "login_as_support",
      "role": "Support (No PII Access)",
      "screenshot": "01_support_login.png"
    },
    {
      "action": "view_customer_record",
      "customer_id": "cust_12345",
      "expected_fields": {
        "name": "MASKED",
        "email": "MASKED",
        "phone": "MASKED"
      },
      "screenshot": "02_pii_masked.png"
    },
    {
      "action": "login_as_admin",
      "role": "Admin (Full PII Access)",
      "screenshot": "03_admin_login.png"
    },
    {
      "action": "view_customer_record",
      "customer_id": "cust_12345",
      "expected_fields": {
        "name": "John Doe",
        "email": "john@example.com",
        "phone": "+1-555-0123"
      },
      "screenshot": "04_pii_visible.png"
    }
  ]
}

AI verification: Computer vision detects whether PII is masked or visible

---

Integration with Testing Frameworks

Cypress Integration

Run automated tests AND generate compliance evidence:

// cypress/e2e/compliance/cc6-1-rbac.cy.js
import { ScreenataPlugin } from '@screenata/cypress'

describe('CC6.1 - RBAC Access Control', () => {
  before(() => {
    // Start Screenata recording
    ScreenataPlugin.startRecording({
      control: 'CC6.1',
      testName: 'RBAC - Viewer Role Access Denial'
    })
  })

  it('should deny Viewer access to admin features', () => {
    // Login as Viewer
    cy.login('viewer@company.com', 'password')
    cy.screenshot('01_viewer_logged_in')

    // Attempt to access admin page
    cy.visit('/admin/users', { failOnStatusCode: false })
    cy.screenshot('02_access_attempt')

    // Verify access denied
    cy.contains('Access Denied').should('be.visible')
    cy.screenshot('03_access_denied')

    // Check audit log
    cy.checkAuditLog({
      user: 'viewer@company.com',
      action: 'access_denied',
      resource: '/admin/users'
    })
    cy.screenshot('04_audit_log')
  })

  after(() => {
    // Stop recording and generate evidence
    ScreenataPlugin.stopRecording({
      exportTo: 'vanta',
      generatePDF: true
    })
  })
})

Benefits:

• ✅ Run tests in CI/CD pipeline
• ✅ Compliance evidence generated automatically
• ✅ No separate manual testing needed
• ✅ Same tests verify functionality AND generate audit evidence

Playwright Integration

// tests/compliance/rbac.spec.ts
import { test, expect } from '@playwright/test'
import { ScreenataReporter } from '@screenata/playwright'

test.use({
  reporter: new ScreenataReporter({
    control: 'CC6.1',
    exportToVanta: true
  })
})

test('CC6.1 - Editor cannot access billing', async ({ page }) => {
  // Login
  await page.goto('/login')
  await page.fill('[name=email]', 'editor@company.com')
  await page.fill('[name=password]', process.env.EDITOR_PASSWORD)
  await page.click('button[type=submit]')

  // Attempt billing access
  await page.goto('/admin/billing')

  // Verify denial
  await expect(page.locator('.error-message')).toContainText('Insufficient Permissions')

  // Verify HTTP 403
  const response = await page.goto('/api/billing/invoices')
  expect(response?.status()).toBe(403)
})

---

Quarterly Test Scheduling

Automated Reminders

Calendar integration:

{
  "quarterly_schedule": {
    "controls": [
      {
        "control_id": "CC6.1",
        "test_name": "RBAC Access Controls",
        "frequency": "quarterly",
        "test_window": "first_two_weeks",
        "reminder_days_before": [14, 7, 3, 1],
        "assignee": "compliance@company.com",
        "backup": "security@company.com"
      }
    ],
    "notifications": {
      "email": true,
      "slack": true,
      "calendar_event": true
    }
  }
}

Slack notification example:

🔔 Compliance Test Due

Control: CC6.1 - RBAC Access Controls Due: March 15, 2025 (7 days) Assignee: @compliance-team

[Run Test Now] [Reschedule] [View Last Results]

Test Execution Tracking

Dashboard shows:

• ✅ Controls tested this quarter
• ⚠️ Controls due in next 7 days
• ❌ Overdue controls
• 📊 Test success rate
• 📈 Time spent on testing

---

Evidence Quality Standards

What Makes Good Evidence

Required elements:

1. Clear screenshots

   • High resolution (1920x1080 minimum)
   • No blur or compression artifacts
   • Key elements highlighted or annotated

2. Complete metadata

   • Timestamp with timezone
   • Tester identity
   • Test environment (prod/staging)
   • Browser and version
   • Test session ID

3. Audit trail

   • All actions logged
   • Logs correlated with screenshots
   • Timestamps match between screenshots and logs

4. Control mapping

   • Explicit reference to control ID (CC6.1)
   • Link to control objective
   • Pass/fail determination
   • Risk assessment

Common Evidence Rejections

Why auditors reject evidence:

Rejection Reason	Example	How Automation Fixes
Missing timestamps	Screenshot with no date/time	Auto-embedded in every screenshot
Unclear test environment	Can't tell if prod or staging	Environment auto-detected and labeled
No tester identity	Unknown who performed test	Tester email captured automatically
Incomplete test steps	Skipped verification steps	Template ensures all steps executed
Poor screenshot quality	Blurry or low-resolution	High-res capture by default
Missing audit logs	No log correlation	Logs auto-queried and captured

---

Cost-Benefit Analysis

Annual Cost Comparison (50 Controls)

Manual approach:

• Time per control: 195 minutes
• Total annual time: 50 × 4 quarters × 195 min = 39,000 min = 650 hours
• Cost at $200/hour: $130,000/year

Automated approach:

• Tool cost: $1,788/year (Screenata Basic)
• Setup time: 8 hours × $200 = $1,600 (one-time)
• Time per control: 8 minutes
• Total annual time: 50 × 4 × 8 min = 1,600 min = 27 hours
• Cost: $1,788 + $5,400 = $7,188/year

Annual savings: $122,812 ROI: 1,608%

---

Implementation Checklist

Week 1: Planning

• List all application-level controls to test
• Document current manual testing process
• Identify test users for each role
• Map controls to Trust Service Criteria
• Set quarterly test schedule

Week 2: Tool Setup

• Install Screenata browser extension
• Create test templates for each control
• Connect to audit log system
• Set up Vanta/Drata integration
• Configure quarterly reminders

Week 3: First Tests

• Record first 3 control tests
• Review AI-generated evidence
• Adjust templates if needed
• Export to GRC platform
• Verify evidence appears correctly

Week 4: Rollout

• Train compliance team on tool
• Schedule remaining quarterly tests
• Document new process
• Archive old manual evidence
• Celebrate time savings! 🎉

---

Key Takeaways

✅ Application-level tests verify app security controls (not just infrastructure)

✅ Manual documentation takes 3+ hours per control per quarter

✅ Automated testing reduces time to 8 minutes (96% reduction)

✅ AI generates audit-ready documentation with screenshots and logs

✅ Integration with Cypress/Playwright enables testing + compliance in one workflow

✅ Quarterly scheduling ensures continuous compliance with automated reminders

✅ ROI of 1,600% for organizations with 50+ controls

✅ Auditors accept automated evidence when properly formatted

---

Get Started with Automated Application Testing

Screenata automates application-level test documentation with browser-based recording, AI-powered evidence generation, and integration with Vanta/Drata.

What you get:

• Browser extension for Chrome/Edge
• Pre-built test templates for common controls
• Automatic screenshot and log capture
• AI-generated step descriptions
• Quarterly scheduling and reminders
• Export to Vanta/Drata/PDF

Pricing: $149/month Setup time: 1 hour Time savings: 3+ hours per control per quarter

Start your 14-day free trial →

---

Related Articles

• How Can I Prove My Role-Based Access Control (CC6.1) Works Using Automated Evidence Capture?
• Can I Integrate Automated Screenshot Documentation with Drata or Vanta?
• How to Automate SOC 2 Evidence Collection with Screenshots