An AI Chatbot Improved HPV Vaccine Literacy—Not Vaccination Decisions

A chatbot can answer the question in front of you. That does not mean it can settle the decision behind it.

A randomized clinical trial in Japan offers a useful measure of both the promise and the limit. Researchers assigned 848 female caregivers of unvaccinated daughters ages 12 to 18 either to an HPV-vaccine chatbot powered by GPT-4o or to a standard Japanese government leaflet. The chatbot group scored modestly higher on a seven-question vaccine-literacy measure immediately after the intervention and again two weeks later. But caregivers’ reported vaccination decisions did not differ significantly between the two groups.

That separation matters. It suggests conversational AI may help people find and absorb factual information without, by itself, overcoming the trust, social, clinical and practical factors involved in a health decision.

Research references: [1]

What the trial tested

The study ran online from November 30 to December 27, 2024. Participants were recruited through a national research panel. All were female caregivers living in Japan, had internet access and had an unvaccinated daughter between 12 and 18. They were randomized 1:1 after a baseline survey.

The intervention was not an unrestricted general-purpose chatbot. It used GPT-4o with a Japan-specific reference library assembled from official vaccine materials, peer-reviewed research and other health sources. Its scope excluded individualized medical advice and directed personal clinical questions to health professionals. The comparison group received the Ministry of Health, Labour and Welfare’s HPV-vaccine leaflet. Both groups had to spend at least five minutes with their assigned material.

The primary outcome was a factual literacy score from zero to seven. The secondary outcome was the caregiver’s reported decision to vaccinate at the two-week follow-up. That is an important distinction: the trial did not verify whether a daughter was actually vaccinated.

Research references: [1]

The gain was real—and small

Of 848 randomized participants, seven were excluded from the main Japan-resident population. The immediate modified intention-to-treat analysis included 704 of the remaining 841 participants. The two-week literacy analysis included 477, or 56.7 percent.

After adjustment for baseline literacy and prespecified participant characteristics, the chatbot group scored 0.30 points higher than the leaflet group on the seven-point scale immediately after the intervention. The 95 percent confidence interval was 0.18 to 0.43 points. At two weeks, the adjusted difference was again 0.30 points, with a 95 percent confidence interval of 0.13 to 0.47.

The raw scores help calibrate the finding. Immediately after the intervention, the chatbot group rose from a mean of 5.01 to 5.47, while the leaflet group rose from 4.85 to 5.10. At two weeks, the chatbot group rose from 5.04 to 5.44 and the leaflet group from 4.86 to 5.14.

So this was not a transformation from confusion to mastery. It was a statistically supported incremental advantage on a short factual quiz.

Research references: [1]

Better answers did not change the short-term decision

At two weeks, 102 of 253 caregivers in the chatbot group—40.3 percent—met the study’s definition for deciding to vaccinate. In the leaflet group, 103 of 260 caregivers—39.6 percent—did so. The adjusted odds ratio was 0.74, with a 95 percent confidence interval from 0.41 to 1.33 and a P value of .31.

In other words, the trial did not show a meaningful difference in the reported decision. Exploratory analyses also did not find that a larger literacy gain was significantly associated with the decision to vaccinate.

That result is not evidence that better information is useless. It is evidence that a health decision is more than a quiz score. People weigh confidence, social norms, clinician recommendations, perceived risk, convenience and access. A chatbot can clarify questions, but it cannot automatically supply institutional trust or a clinician’s personal recommendation.

Research references: [1]

The design points toward an assistive role

The study’s strongest practical lesson is architectural. The chatbot was constrained to a reference library, told to avoid personal medical advice and designed to hand clinical concerns back to health professionals. The authors likewise framed it as an adjunct to official information and professional recommendations—not a replacement.

That is a better model for public-health AI than treating conversation as a substitute for care. Official materials establish a shared evidence base. A conversational layer can let people ask follow-up questions in private, at their own pace. Clinicians and public-health workers remain responsible for individualized guidance, safety and the trust required for action.

The trial also shows why evaluations should separate information outcomes from behavioral outcomes. A system can improve an intermediate measure while leaving the final decision unchanged. Product claims should keep those endpoints distinct.

Research references: [1]

What this study cannot tell us

The evidence is narrower than a headline about “AI increasing vaccination” would imply.

Two-week follow-up data were missing for many participants. The literacy analysis at that point included 477 of 841 Japan-resident randomized participants, so residual attrition bias cannot be ruled out despite sensitivity analyses. The vaccination decision was self-reported, measured over only two weeks and not checked against actual uptake. The trial was powered for literacy, not small differences in vaccination decisions.

The immediate analysis sample was comparatively educated and higher-income: 67.1 percent had a college or university education, and 75.6 percent of respondents with income data reported annual household income of at least four million yen. The findings therefore may not generalize to all caregivers in Japan, much less to other countries.

Backend logs could be matched for only 176 of the 424 people assigned to the chatbot. The authors treated analyses based on those matched users as exploratory because matching happened after randomization. The result also applies to one chatbot, one reference library and one implementation context. Longer studies would need to verify actual vaccination.

No conflicts of interest were reported. The work was supported by AIR@InnoHK, administered by the Innovation and Technology Commission; the paper states that the funder had no role in study design, data work, manuscript decisions or publication.

Research references: [1]

The human-future test

The useful question is not whether a chatbot can beat a leaflet by a fraction of a point. It is whether the system helps people understand evidence while preserving the relationships and institutions that make health decisions trustworthy.

This trial clears a modest part of that test. The chatbot added a small, durable literacy advantage over official material. It did not demonstrate more vaccination decisions, and it did not measure actual uptake.

That is still actionable evidence: build conversational AI as a carefully sourced layer around public information, measure the outcome that actually matters and keep clinicians in the loop. Better answers can support a decision. They should not be confused with the decision itself.

: Kobayashi K, Wu Z, Tomoi H, et al. “AI Chatbot Use for Human Papillomavirus Vaccine Literacy in Japan: A Randomized Clinical Trial.” *JAMA Network Open*. Published September 21, 2026. https://doi.org/10.1001/jamanetworkopen.2026.34696

Research references: [1]