A study from researchers at the National University of Singapore investigates what happens when reading highlights are used to personalize AI writing assistance—and the findings suggest that seemingly helpful design choices can fundamentally alter the meaning of human actions within collaborative workflows.
What They Did
Qin and colleagues recruited 46 participants to complete an argumentative writing task. Participants first read source materials (two opposing articles on a debate topic), during which they could freely highlight content. They then moved to a writing phase where they could request AI-generated suggestions.
The manipulation was straightforward: in the Personalization condition, participants were told that AI suggestions would be generated only from their highlights. In the Baseline condition, the AI drew from the full reading materials regardless of highlighting behavior.
This design allowed the researchers to examine how awareness of the personalization mechanism—knowing that your highlights exclusively determine what the AI can offer—affects both reading and writing behavior.
What They Found
Participants in the Personalization condition highlighted substantially more. Their highlight density was roughly ten times that of the Baseline group, they made more individual highlights, and they spent longer in the reading phase. On its face, this looks like increased engagement.
However, this apparent engagement did not translate into positive writing outcomes. Personalization participants made fewer edits (approximately one-third as many), produced shorter final essays, showed higher AI reliance, and accepted a greater proportion of AI suggestions. More strikingly, they reported significantly lower autonomy, ownership, and self-credit for their work.
The correlation analyses revealed what the researchers describe as a shift in function. In the Baseline condition, highlighting behavior was positively associated with active editing and revision—the pattern one might expect if highlighting serves comprehension. In the Personalization condition, highlighting was positively correlated with acceptance ratio but negatively correlated with ownership and satisfaction.
The researchers interpret this as a motivational shift: highlighting transformed from a sense-making strategy into an instrumental act of “feeding the AI” sufficient context for generation.
Methodological Notes
The study uses a controlled experimental design with random assignment, validated scales for autonomy and ownership, and detailed behavioral logging. The AI system was built on GPT-4.1 with a retrieval-augmented generation architecture, using structured prompt templates based on the Toulmin model of argumentation.
The explicit nature of the manipulation—participants were directly told how the AI would use their highlights—represents a specific design choice worth noting. The researchers acknowledge that implicit or partially transparent personalization might produce different patterns.
Boundaries
The study examines a single operationalization of personalization (highlight-constrained retrieval) in a single task type (argumentative writing). Other forms of personalization—style matching, preference learning, adaptive scaffolding—may function differently. The sample is relatively small and drawn from a specific population. The short-term design cannot speak to whether these effects persist or how they might evolve with extended use.
The finding that highlighting became instrumental rather than comprehension-oriented is inferred from behavioral patterns and correlation shifts rather than directly measured. Alternative interpretations—such as strategic efficiency or rational adaptation—are not fully ruled out.
Why This Matters
The core contribution here is demonstrating that the same behavior can carry opposite relational significance depending on how it connects to downstream processes. Highlighting in service of understanding correlates with ownership; highlighting in service of AI input correlates with diminished ownership. The action is identical; what changes is its meaning within the human-AI system.
The researchers frame this through the lens of “extrinsic personalization”—a concept from education research describing engagement driven by external rewards rather than intrinsic goals. When the purpose of reading shifts from “I’m trying to understand this material” to “I’m providing input so the AI can help me,” the cognitive and affective relationship to the task appears to change accordingly.
From our perspective, this work offers an empirical window into how design choices shape the relational dynamics between humans and AI systems. The finding that personalization can increase behavioral engagement while simultaneously undermining felt ownership speaks to the complexity of these interactions—surface metrics may not capture what matters most about the human experience of collaboration. The correlation reversal is particularly noteworthy: it suggests that the meaning of human actions within human-AI systems cannot be read directly from the actions themselves, but depends on the broader structure of the interaction.
References
Qin, P., Yang, C.-L., Boonprakong, N., Chen, J., Tan, Y., & Lee, Y.-C. (2026). AI Personalization Paradox: Personalized AI Increases Superficial Engagement in Reading while Undermines Autonomy and Ownership in Writing. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). ACM.