Abstract
Existing work offers limited concrete guidance for evaluating topic-model outputs in qualitative analysis, including when machine-generated topics can meaningfully complement human-coded themes without sidelining human judgment. This paper introduces a six-stage Alignment and Usefulness Rating Audit (AURA) workflow for qualitative analysis. It combines standard data stages (collect, transcribe, structure, model) with two quality checks: an alignment check that assesses whether machine-generated topics align with human-coded themes, and a usefulness check that evaluates whether those topics are clear and useful for inclusion. At each stage, we specify who is responsible for what (human vs. machine), what is involved, and how decisions are made. We illustrate the workflow through a secondary analysis of 32 semi-structured interviews of nurses about their experiences with generative artificial intelligence. A publication from the manual coding of the interviews produced a set of top-level themes that served as a reference. We then used BERTopic to discover topics, compared the average sentence-level representations of themes and topics to assess alignment, and had two researchers rate the usefulness of the resulting topics on a scale of 1-5 (5 being the most useful). Eleven of the 12 human themes had at least one topic above the similarity cutoff. Although the highest similarity with themes was concentrated in four of the eight topics, six themes had the second-highest matches with topics, indicating both concentration and overlapping correspondence. All topics met the analytic usefulness criterion, but the findings provided limited evidence of additive thematic discovery beyond the human-developed framework. We argue that this alignment-and-usefulness-rating workflow provides social science researchers with a practical approach to integrate qualitative judgment with topic modeling, emphasizing transparency and human validation.