Investigating failures

When a test fails, Shoptest provides evidence, AI-powered analysis, and optional automated repair to help you identify and resolve the issue quickly.

---

Review test results

1. Go to Test Flows and click the failing test.

2. Open the most recent run from the run history.

3. Review the evidence:

   • Pass/fail status — shown at the top of the screen.
   • Step-by-step breakdown — each recorded step with its result.
   • Screenshots — captured at key journey points during execution.
   • Run log — detailed record of what happened during the run.
   • Batch results — for Batch-type tests, results are shown per product or URL.

4. Identify which step failed and what the screenshot shows at that point.

Expected result: You can pinpoint where in the journey the failure occurred and see exactly what the user would have experienced.

---

AI failure analysis

When a test fails, you can request an AI-powered analysis to help explain the cause.

1. Open the failed test run.
2. Click AI Failure Analysis (or the equivalent analysis button).
3. Shoptest analyses the failure evidence and generates a report that typically includes:
   • What likely caused the failure
   • Which step or element broke
   • Suggested next steps for investigation or resolution

When to use it:

• The failure is not immediately obvious from screenshots alone.
• You need a summary to share with a developer or client.
• The failure may be caused by a theme change, app update, or third-party dependency.

Credit cost: AI failure analysis uses 20–100 credits depending on the complexity of the analysis.

---

AutoFix

For theme-related breakages, Shoptest can attempt to repair the test automatically.

How AutoFix works:

1. Shoptest detects that the failure is likely caused by a theme change.
2. It prepares a repair — updating the test to match the current theme structure.
3. It runs the repaired test to verify whether the fix resolves the failure.
4. You can accept or reject the result.

AutoFix modes (configured in Settings > AutoFix):

Mode	Behaviour
Automatic	Shoptest applies fixes automatically when a theme-related failure is detected
Ask me first	Shoptest prepares the fix but waits for your approval before applying
Off	AutoFix is disabled; all fixes are manual

Recommended default: Set AutoFix to Ask me first so you review every repair before it goes live.

Credit cost: AutoFix uses credits per fix attempt. The cost depends on the type of repair.

---

Safari compatibility checks

Shoptest can run saved tests against real Safari browsers to catch browser-specific issues.

1. Open a saved test from Test Flows.
2. Click Safari Check (available on the test detail screen).
3. Choose the device:
   • Desktop — macOS Safari
   • Mobile — select from supported iOS Safari devices
4. Run the check.
5. Review the results:
   • Pass, fail, or error outcome
   • Video recording of the Safari session
   • Screenshots at key points
   • Console logs and network logs

When to use Safari checks:

• After creating or updating a test, to verify it passes in Safari as well as Chrome.
• For stores with significant iOS or macOS traffic.
• Before a major theme launch or redesign.

Credit cost: Desktop Safari checks cost 5 credits. Mobile Safari checks cost 10 credits.

---

Spot patterns in run history

Use run history to identify recurring issues:

1. Open a test and scroll to the Run history section.

2. Look for patterns:

   • Repeated failures at the same time — may indicate a scheduled process or dependency issue.
   • Failures that started on a specific date — correlate with theme updates, app installs, or campaign launches.
   • Intermittent pass/fail cycles — may indicate a flaky test that needs better validations, or an unstable third-party dependency.

3. Use the date range to narrow down when a failure pattern began.

---

Troubleshooting

Test shows Pending and has never run

• Trigger the test manually to confirm it works. If it fails immediately, check the starting URL and storefront password (in Settings > Storefront Password).

AI analysis is unavailable

• Check your credit balance in Settings > Plan & credits. AI analysis requires credits to run.

AutoFix was applied but the test still fails

• The fix may not have addressed the root cause. Review the AutoFix result, then either edit the test manually in the recorder or contact the development team.

Safari check fails but Chrome test passes

• This is a genuine browser-specific issue. Review the Safari video recording and console logs to identify the difference. Common causes: CSS rendering differences, JavaScript API availability, and touch-event handling on iOS.