The concept of using test automation tools to help with repetitive or time-consuming documentation tasks is by no means new. Throughout my career, I’ve come across a couple of use cases where tools such as Selenium WebDriver automated screenshot taking and it was pretty neat. It did require substantial manual work to define and maintain, but could save a significant amount of effort in the “Hey we have this small UI change that affects almost every single screenshot in your docs” type of situation. We’ve all been there.
The fact that we now have LLMs and agents on top of them unlocks a completely different dimension in this space. You no longer have to define everything in advance. You no longer have to know the exact IDs of elements in your app’s DOM. You no longer have to define screenshot properties manually and in advance. You no longer have to click Inspect element again and again to edit client names and emails or hide unwanted elements so they are not shown in the screenshots (yes, we do that). AI agents take determinism and predefinedness (wow, this is a word) out of the equation and let you actually benefit from automating your repetitive tasks. In addition, agent-based automations are much lower maintenance than pre-AI workflows that involve only test automation tools. This almost eliminates the risk of automation maintenance eating up the time it saves.
The sections below provide real-world examples of how Playwright CLI can automate screenshot taking and review of procedures in your documentation along with verification of UI labels.
Prerequisites:
- Installed Playwright CLI.
- A coding agent. The agent should have internet access and be able to use CLI tools. I used Claude Code (CLI).
- (Optional) If you’re trying to set up a broader workflow within your organization: configured integrations such as MCP or CLI between the agent, issue tracking system, and Git hosting such as GitHub.
Automating screenshot updates
This task involves instructing the agent to replace one or more screenshots in your documentation using Playwright CLI to navigate through the UI, find the UI element and take its screenshot. Additionally, the agent is given instructions on how to authenticate (see Save authentication state), and make temporary edits to populate the UI with data or anonymize existing data if it shows actual user information.
I used locally-hosted Metabase as an example of a web application to document, and the documentation available in its repository. Here is the prompt I used for Claude Code (Opus 5.5, Medium effort):
Analyze screenshots from the following page @docs/questions/visualizations/table.md. Create a new branch from updated `master`, then retake all screenshots included in the page.
To retake screenshots, use `playwright-cli` to access `http://localhost:3000/model/1-orders-people`. The saved authentication state is in @auth.json, use it for authentication. If there is no or little data in the table, try to make temporary, ephemeral edits to the table to generate some data. If the data shows user identities such as usernames, emails, or other personal information, replace the information with random placeholder data.
Save each screenshot under the original file name.
Constraints: DO NOT make any other changes, commit anything, push, or create PRs, just retake the requested screenshots. If you make edits to the table to add more data or hide user identities, those edits must be temporary and disappear on refresh. No permanent modifications of actual data are allowed.
Let’s dissect the prompt structure:
Analyze screenshots from the following page @docs/questions/visualizations/table.md. Create a new branch from updated `master`, then retake all screenshots included in the page.
This part provides basic information about the task and Git instructions as Claude will be working in a new branch. If you are working in an issue tracking system that is connected to your agent through MCP or CLI, you can additionally provide an issue number, so the agent can access the information about the task. This can also be used for branch naming, even with an explicit branch naming convention in AGENTS.md that tells the agent to name the branch after the issue number and summary.
In addition, this part of the prompt points to the exact files that need to be analyzed for screenshots to replace. You define the scope, the agent reads and identifies the screenshots that need to be changed based on your description of the change. While not the only approach, I found it most effective because I knew exactly where the changes were needed and the size of the files, so it made sense from the cost standpoint to provide source pages and let the agent find the affected screenshots. If the changes are narrower in scope, such as a single image file that you know needs to be replaced, you can point right at it. If the changes are broad and you’re not sure about their exact location in your source, you can provide a broader scope. However, this requires careful consideration of the context size and cost. Letting the agent traverse your entire source and read every single screenshot is likely not an effective solution as the cost can be much higher than the potential gain.
To retake screenshots, use `playwright-cli` to access `http://localhost:3000/model/1-orders-people`. The saved authentication state is in @auth.json, use it for authentication. If there is no or little data in the table, try to make temporary, ephemeral edits to the table to generate some data. If the data shows user identities such as usernames, emails, or other personal information, replace the information with random placeholder data.
Here we have three important instructions for the agent:
- Where to start browsing. The prompt provides the exact URL to the affected part of the UI, which saves agent time and tokens.
- What to do when there isn’t enough data to take meaningful screenshots. Doing this manually, or asking someone who has access to the database or the app’s backend to get some placeholder data would have taken too much time. The agent can do it for you, and it does it really well if you ask me. The important thing about this is to set the limitation right away since the agent must never change the data permanently.
- Randomize personal information in the existing data. Most of the time, you don’t want actual user or company data to be visible in the documentation. On the other hand, screenshots with blurred confidential information don’t look as nice and may even be less informative as they don’t convey the actual state of the UI. That’s where the agent can step in and replace actual data with made-up, but still adequate data that shows off the interface and the functionality in question.
Constraints: DO NOT make any other changes, commit anything, push, or create PRs, just retake the requested screenshots. If you make edits to the table to add more data or hide user identities, those edits must be temporary and disappear on refresh. No permanent modifications of actual data are allowed.
Finally, constraints. The title of the post does say unleash, but let’s not unleash the agent just too much. This is a quick and short set of generic constraints that I could come up with. Probably can be elaborated a lot more, especially with the help of AI. However, these constraints were tested in practice and the agent didn’t do anything it wasn’t supposed to, so I’d say this works.
The final outcome of the agent run is 12 new screenshots, each saved under its original name, in a new local git branch, as the original prompt says. The agent didn’t commit any changes, nor make any changes that would have a permanent effect on the displayed data. Again, this is exactly as the instructions said initially. The final outcome of my test run with Metabase documentation is available in the following commit.
The total run time for this session was 11m 55s, with a consumption of $5.12 running on Opus 5.5. What is still up for testing is whether the cost can be lowered by using a cheaper model, while retaining the same quality of the output.
Automating procedure testing
Another use case where AI agents and Playwright CLI help automate a task that can otherwise be tedious and error-prone is testing procedures that have already been written in the documentation. This way, the author can ensure that:
- The procedure steps correspond to the actual flow in the user interface.
- The labels in the documentation match the labels in the user interface.
The following example uses Claude Code with Playwright CLI to follow the procedures on the Exporting results documentation page in Metabase documentation.
This is the prompt I used to initiate the procedure check:
Use playwright-cli to check the instructions written in @docs/questions/exporting-results.md, in the `Exporting results from document cards` section. The procedure is written as a numbered list. Navigate to `http://localhost:3000/document/2` and follow the instructions. Use @auth.json to authenticate. Verify that the steps in the procedures are correct and that the UI labels that are usually formatted as bold text match those in the user interface.
CONSTRAINTS: Do not modify any existing data or take any action in the UI other than what the procedure specifies.
As the agent starts running, it will likely perform actions in an environment that is shared by multiple users, even if it is a test or staging version of the product. To make sure I highly recommend the following:
- The prompt should include a limitation on performing actions on a single entity or a limited number of them, and explicit instructions that other existing entities should not be affected in any way.
- Be cautious about using agent modes that won’t prompt you to confirm agent actions, such as the
Automode in Claude Code. - Once the agent starts running, use
playwright-cli showin the terminal to find and open the browser session where the actions take place. If you have a split view of the agent and browser window, you can follow the actions live.
In the specific Metabase example, Claude Code actually found a missing step and a missing export file format. The last step in the documentation is the output format selection, whereas there is still one important step for the users, which is clicking the Download button. Also, the documentation doesn’t list the .png format as an option, which is available in the interface. Here is the table with the summary provided by the agent:
| Step | Doc text | What the UI shows | Verdict |
|---|---|---|---|
| 1 | Hover over the chart | Hovering highlights the chart. The card’s … button shows even without hovering. | OK |
| 2 | Click the three dot menu (…) | The menu opens. | OK |
| 3 | Select Download results | The menu item is labelled exactly Download results. It opens a Download data popover. | OK |
| 4 | Choose your format: .csv, .xlsx, or .json | The popover lists .csv, .xlsx, .json and .png. Choosing one doesn’t download anything. | Wrong |
In addition, the agent provides a rewrite suggestion:
1. Hover over the chart in the document.
2. Click on the three dot menu (...).
3. Select Download results.
4. Choose your format: .csv, .xlsx, .json, or .png.
5. Click Download.
To export unformatted results, uncheck Keep the data formatted before clicking Download.
The prompt above is written for an agent following Markdown docs. If you are working on sources that include semantic elements such as dedicated tags for user interface labels or procedure steps, this helps you point to the elements of interest very precisely in the prompt. At the same time, such elements should be quite unambiguous for the agent and allow it to process the content without spending excessive time or effort identifying them.
This flow can potentially be converted to an agent skill and run as a part of an automation that would identify documentation that is outdated or out of sync with the user interface.
Saving the authentication state
To automate how the agent and Playwright CLI handle your product’s authentication, you can log in manually using Playwright CLI and then save the session to a file. When automating a documentation task, tell the agent to use the authentication state saved in the file and the agent will not have to pause and request that you log in manually, which is the whole point of creating an AI-based automation.
To save the Console authentication state:
- Open the browser at the login page:
playwright-cli open <url> --headed
Use the--headedflag to make sure that the browser opens on your screen and lets you log in. - Perform the login steps in the browser window until you have successfully logged into your account.
- Run the command in your terminal to save the authentication state while the session is active:
playwright-cli state-save auth.json - Verify that the authentication state can be reused:
playwright-cli state-load auth.jsonplaywright-cli open <url>
Once you have the saved authentication state in a file, it can be reused as long as your session remains valid. You need to repeat this process if the saved state no longer lets Playwright CLI pass through the authentication gate.