The firm in question is Australia-based PredictiveHire, founded in October 2013. It offers a chatbot that asks candidates a series of open-ended questions. It then analyzes their responses to assess personality and job traits, like “drive,” “initiative,” and “resilience.” According to the firm’s CEO Barbara Hyman, its clients are employers that must manage high volumes of applications, such as those in the retail, sales, call center, and health care industry. As the Cornell study found, it also actively uses promises of fairer hiring in its marketing language. On its home page, it boldly advertises: “Meet Phai. Your co-pilot in hiring. Making interviews FINALLY, WITHOUT BIAS.”

As we’ve written before, the idea of “bias-free” algorithms is highly misleading. But PredictiveHire’s latest research is troubling for a different reason. It is focused on building a new machine-learning model that seeks to predict a candidate’s likelihood of job-hopping, the practice of changing jobs more frequently than an employer desires. The work follows the company’s recent peer-reviewed research that looked at how open-ended interview questions correlate with personality (in and of itself a highly-contested practice). Because organizational psychologists have already shown a link between personality and job hopping, Hyman says, the company wanted to test whether they could use their existing data for the prediction. “Employee retention is a huge focus for many companies that we work with given the costs of high employee churn, estimated at 16% the cost of each employee’s salary,” she adds.

The study used the free-text responses from 45,899 candidates who had used PredictiveHire’s chatbot. Applicants had originally been asked 5-7 open-ended questions and self-rating questions about their past experience and situational judgement. These included questions meant to tease out traits that studies have previously shown to strongly correlate with job hopping tendencies, such as being open to experience, less practical, and less down to earth. From their tests, the company researchers claim the model was able to predict job hopping with statistical significance. PredictiveHire’s website is now already advertising this work as a “flight risk” assessment that is “coming soon.”

PredictiveHire’s new work is a prime example of what Nathan Newman argues is one of the biggest adverse impacts of big data on labor. In a 2017 law paper, the adjunct associate professor at the John Jay College of Criminal Justice wrote that beyond the concerns of employment discrimination, big data analysis had also been used in myriad ways to drive down worker’s wages.

Machine-learning based pre-hire personality tests, for example, are increasingly being used to screen out employees who have a higher likelihood of agitating for increased wages or supporting unionization. Workplace surveillance is increasingly monitoring employees’ emails, chats, and other data to assess which employees might leave and calculate the minimal possible pay increase needed to make them stay. And algorithmic management systems like Uber are increasingly decentralizing workers away from offices and digital convening spaces that allow them to coordinate with one another and collectively demand better treatment and pay.

None of these examples should be surprising, Newman argued. They are simply a modern manifestation of a history of employers seeking to suppress wages through targeting and breaking up union activities. The use of personality assessments in hiring, which dates back to the 1930s in the US, in fact began as a mechanism to weed out people most likely to become labor organizers. The tests became particularly popular in the 1960s and 70s once organizational psychologists had refined them to assess workers for their union sympathies.

In this context, PredictiveHire’s fight risk assessment is just another example of this trend. “Job hopping—or the threat of job hopping,” points out Barocas, “is one of the main ways that workers are able to increase their income.” The company even built their assessment based on personality screenings designed by organizational psychologists.

Barocas doesn’t necessarily advocate for tossing out the tools all together. He believes their goal of making hiring work better for everyone is a noble one and could be achieved by regulators mandating their greater transparency. Currently none of them have received rigorous, peer-reviewed evaluation, he says. But if firms were more forthcoming about their practices and submitted their tools for such validation, it could help hold them accountable. It could also help scholars engage more readily with firms to study the tools’ impacts on both labor and discrimination.

“Despite all my own work for the past couple of years expressing concerns about this stuff,” he says, “I actually believe that a lot of these tools could significantly improve against the current state of affairs.”