Your Research Data Has Strangers In It

Here's where we are. The data your team collects to make decisions is getting harder to trust. Open survey links pull in bots. Incentive money draws fakers. AI fills the gaps with text that reads fine and says nothing. None of this is loud or dramatic. It just rots the numbers you bring to your next review.

The good news: the fixes are practical and old-school. Let me catch you up on three levers that keep your research honest, and the one thing that ties them together.

The strangers filling out your surveys

If you share surveys through social media, public links, or open communities, some of your responses aren't from people. They're from survey bots, and they're getting better fast. As Nielsen Norman Group lays out in a guide to kicking bots out of your data, some bots now use AI to write plausible open-ended answers and adapt to your question logic. They pass basic attention checks. They look normal.

The tells are still there if you look. A survey that takes a human 8 to 10 minutes should not finish in 30 seconds. Hundreds of responses clustering at almost exactly 5 minutes is a script, not real people. Watch the open-ended answers too: bots write long, polished, and weirdly generic, text that could describe any product. Strings of random email addresses arriving back to back are another flag.

One sign alone proves nothing. A fast time might be a power user. Two or three signals together, like a quick finish plus a duplicate IP, are enough to flag a response. And bots travel in batches, so when you find one, scan the responses around it.

Pay people the wrong way and they'll show you

The other place fake data sneaks in is your incentive structure. This matters most in diary studies, where people log entries over days or weeks. Pay them poorly and they'll game it. Pay per entry with no guardrails and some folks will pad their submissions to max out the money, as Maria Rosala notes in her breakdown of diary-study incentives.

The fix is to design the prompt so faking is hard. Require a video, a photo, or a screenshot, and a made-up entry gets a lot more expensive to fabricate. Then match the pay structure to the behavior you want. In one 4-week study, the team paid $15 for the first four entries each week and $5 for extra ones, which pushed people to report every week instead of dumping everything at the end.

The lesson for your team: incentives shape what you get. Decide what good data looks like first, then pay in a way that rewards exactly that.

AI is now both the problem and a tool

The noise isn't just in your surveys. It's in how research reaches people. Bad AI summaries warp public understanding of real findings. So a University of Washington team built PaperTok, a free tool that turns dense papers into short videos, using AI to fight AI misuse by letting scientists tell their own stories.

What's worth stealing here is the design choice, not the tool. PaperTok keeps a human in the loop at every step. It generates a script and visuals, then makes the researcher approve and edit down to individual words. Even so, some users said the output still felt "too AI-ish."

That's the tradeoff in plain terms. AI can speed up your research work, but the moment you take humans out of the loop, the slop creeps back in. Approval gates aren't busywork. They're the difference between using AI and being used by it.

The deep cut

Here's the thing that's easy to miss: all three levers are really one habit, and it's writing things down before you start. NNG's piece on establishing baselines makes the point cold. One of the common ways teams undermine their own value isn't bad work, it's measuring too late. If task completion was 48% before and 67% after, that 19% lift proves your effect. Without the 48%, the 67% is just a number.

Same logic runs through bot screening and incentives. Document how many responses you flagged and why. Pilot-test your median completion time so you know what "too fast" means. Spell out your incentive rules in writing before recruiting. Pick one behavioral metric at kickoff and commit, instead of tracking five and cherry-picking the one that moved.

The payoff is trust. When you decide your method and write it down before the work begins, your post-launch review becomes finishing a story, not arguing for your worth. The discipline is boring. It's also what keeps your data credible when everything else is getting noisier.

Three questions for your team

1. Before our next survey ships, do we have a pilot-tested completion time and a written rule for flagging bots, or are we cleaning data by gut feel after the fact?
2. For our diary studies, does our incentive structure reward the specific entries we actually need, and does the prompt make faking hard, like requiring a photo or video?
3. On every project this quarter, did we capture a baseline number and the method behind it before we started building, so the "after" means something?