FIU Business research examines how historical biases influence artificial intelligence and what companies can do to make automated decisions fairer.
Artificial intelligence is changing how companies hire employees, approve loans and evaluate consumers. But the technology's ability to process enormous amounts of information doesn't necessarily make its decisions fair.
AI systems trained on historical data can reproduce biases that have long affected women, minorities and other underserved groups, according to research co-authored by Arun Upadhyay, eminent scholar and professor of finance at Florida International University's College of Business.
The article, "AI Presents Both Problems and Opportunities for Minorities," published in the Journal of Management Studies, examines how artificial intelligence can reinforce inequalities in employment and financial services — and how better training data, human oversight and organizational accountability could help reduce them.
Co-authored with María del Carmen Triana of Vanderbilt University, the paper analyzes existing research on algorithmic discrimination and identifies steps businesses and policymakers can take to address it.
"AI has a lot of potential to fix those issues, but as long as the training data being used remains the same, biased ... AI can still hurt these segments of society," Upadhyay said.
Why can artificial intelligence make biased decisions?
Although AI systems don't hold personal prejudices in the same way humans do, they learn from data shaped by human decisions.
If a company trains a hiring algorithm using historical employment records that favor certain demographic groups, the system may learn to prioritize applicants with similar characteristics. In lending, historical credit data can reflect disparities in access to financial services, potentially affecting how algorithms evaluate future borrowers.
Even information that appears neutral, such as ZIP codes, may correlate with race or socioeconomic status.
Upadhyay explained that bias can enter a system before an algorithm begins evaluating applicants or consumers. Decisions about what information to collect, how to categorize it and which variables to prioritize are all influenced by people.
"Those who are providing data, they are different from those who are using data," he said. "Different kinds of skill sets, different kinds of incentives, different kinds of outcomes."
When developers assume their training data is complete and objective, those underlying biases can become embedded in automated decisions.
How can AI bias affect hiring and lending?
The researchers focus on employment and financial services because decisions in these areas can have significant consequences for people's economic opportunities.
In hiring, automated systems may screen resumes, rank candidates or evaluate applicants using patterns learned from previous employment decisions. If those historical decisions favored certain groups, qualified candidates from underrepresented backgrounds could be disadvantaged.
In financial services, AI models may evaluate creditworthiness using traditional credit records alongside other information, including transaction histories and geographic data.
Upadhyay pointed to consumers with limited credit histories, such as recent immigrants and people who avoid borrowing. Although they may be financially responsible, conventional credit-scoring systems may have insufficient information to evaluate them.
That creates both a challenge and an opportunity for financial institutions.
Can artificial intelligence help reduce discrimination?
Despite the risks, the researchers argue that AI could help organizations identify and reduce inequalities when the technology is developed and monitored carefully.
In lending, AI could allow financial institutions to consider a broader range of information, such as rental and utility payments, when evaluating consumers with limited traditional credit histories.
That could help lenders better assess some borrowers who may otherwise be overlooked.
In hiring, organizations could design automated systems to focus more closely on job-related qualifications and skills rather than patterns that reflect historical preferences.
However, the researchers caution that expanding datasets is not, by itself, a solution. Additional information can introduce new biases unless businesses carefully evaluate how it affects their models.
The goal is not simply to give AI more information, but to ensure the information is appropriate, representative and relevant to the decisions being made.
What can businesses do to reduce AI bias?
Upadhyay argues that companies should begin by examining the information used to train their AI systems rather than assuming existing datasets are accurate or impartial.
"What we are suggesting is that maybe it's a good time to start training these data engineers in terms of the data properties, so that they do not take that data as given," he said.
The research identifies several approaches organizations can use to reduce algorithmic bias:
- Examine training data: Identify historical patterns, missing information and potential discriminatory factors before developing models.
- Use representative datasets: Ensure the information used to train AI systems adequately reflects the populations affected by their decisions.
- Conduct regular audits: Test automated outcomes to identify whether particular groups are being systematically disadvantaged.
- Maintain human oversight: Involve trained professionals in reviewing consequential decisions rather than relying exclusively on algorithms.
- Improve transparency: Make automated decisions more understandable so organizations can identify errors and explain their outcomes.
The researchers emphasize that these measures should be incorporated throughout the AI development process, not introduced only after problems emerge.
What role should policymakers play in AI fairness?
The paper also examines how regulations and institutional oversight can help prevent algorithmic discrimination.
The authors discuss existing legal protections governing employment discrimination, as well as regulatory approaches requiring greater transparency and auditing of automated decision-making systems.
For policymakers, the challenge is to establish safeguards that encourage innovation while ensuring that companies remain accountable for decisions affecting employment and access to financial services.
Upadhyay believes greater scrutiny of how AI models are developed is essential, particularly because consumers often have little visibility into the information and assumptions behind automated decisions.
He argues that governance must extend beyond monitoring an algorithm's final output to examining the data and processes used to build it.
Can AI ever be completely free of bias?
The researchers acknowledge that creating entirely bias-free AI systems may be practically impossible.
Even carefully designed algorithms can reflect the assumptions and priorities of their developers, as well as changes in the social and institutional environments in which they operate.
"We cannot eradicate bias," Upadhyay said. "However, what we can do is we can minimize that bias."
For consumers, his message is equally straightforward: Artificial intelligence should not be treated as an unquestionable authority simply because its decisions are generated by technology.
"We should not take the AI models’ output as the final word," he said. "These are all human creations, and we should always treat them as any other tool that we use, with scrutiny."
As businesses increasingly rely on artificial intelligence, the researchers argue that fairer outcomes will depend not just on what the technology can do, but on how people design, evaluate and govern its use.
Research: "AI Presents Both Problems and Opportunities for Minorities," Journal of Management Studies (2026). DOI: 10.1111/joms.70145.