Category: Blog

Talk on Responsible AI at UWV’s IT Conference

On Tuesday, October 8 I will be giving a talk at a conference organized by the UWV (the Dutch Employee Insurance Agency), my talk is titled Challenge Accepted: Responsible Algorithms in the World of Work and News, and will be about designing fair algorithms, drawing from my own experiences building recommender systems at FD Mediagroep and Randstad.

For more details, see the (Google Translated) blurb below:

Algorithms increasingly influence decisions in our daily lives, from work to media consumption. But how do we ensure that these technologies are fair, inclusive and ethical? In this session, David Graus, AI expert with years of experience in building search and recommendation systems, shares insights from his own practice, including developing a recommender system for the FD and matching work and talent at Randstad.

Using these practical examples, we discuss challenges of bias, ethics and inclusivity in algorithms. We show how you can engage with stakeholders to align technology with values ​​and how you can use algorithms responsibly within the world of work and media.

Challenge accepted: together we take on the challenge of developing fair and inclusive algorithms that have a positive impact on our society.

Who is David Graus?
David Graus is an expert in information retrieval with a PhD in search engine technology. He works as a lead data scientist at Randstad, where he helped build AI systems for recruitment and selection. In addition to designing and building such systems, he researches their implications. David is academically active and regularly publishes papers, organizes workshops, and participates in an EU-funded research project focused on anti-discrimination in recruitment and selection algorithms.

RecSys in HR program and panel announced

With a mere 2 weeks until our 4th Recommender Systems for Human Resources Workshop will be held in Bari, Italy, we have shared the final program and panel of our workshop.

Papers

We were able to accept a total of 10 high quality submissions in the workshop’s proceedings! They are:

  • Finding the perfect match at scale: A quest on freelancer-project alignment for efficient multilingual candidate retrieval (Warren Jouanneau, Marc Palyart and Emma Jouffroy)
  • MELO: An Evaluation Benchmark for Multilingual Entity Linking of Occupations (Federico Retyk, Luis Gasco, Casimiro Pio Carrino, Daniel Deniz Cerpa and Rabih Zbib)
  • Pseudo-online Measurement of Retrieval Recall for Job Recommendations – A case study at Indeed (Liyasi Wu, Yi Wei Pang and Warren Cai)
  • On the Biased Assessment of Expert Finding Systems (Jens-Joris Decorte, Jeroen Van Hautte, Chris Develder and Thomas Demeester)
  • Hardware-effective Approaches for Skill Extraction in Job Offers and Resumes (Laura Vásquez-Rodríguez, Bertrand Audrin, Samuel Michel, Samuele Galli, Julneth Rogenhofer, Jacopo Negro Cusa and Lonneke van der Plas)
  • A Dynamic Jobs-Skills Knowledge Graph (Alejandro Seif, Sarah Toh and Hwee Kuan Lee)
  • Combined Unsupervised and Contrastive Learning for Multilingual Job Recommendation (Daniel Deniz, Federico Retyk, Laura García-Sardiña, Hermenegildo Fabregat, Luis Gasco and Rabih Zbib)
  • Parallel Computation-Driven Stable Matching for Large-Scale Reciprocal Recommender Systems (Kento Nakada, Kazuki Kawamura and Ryosuke Furukawa)
  • Enhancing Reliability in Recommendation Systems: Beyond point estimations to monitor population stability (Yingshi Chen, Mohit Jain, Vaibhav Sawhney and Liyasi Wu)
  • Creating Healthy Friction: Determining Stakeholder Requirements of Job Recommendation Explanations (Roan Schellingerhout, Francesco Barile and Nava Tintarev)

For the full program, please see: https://recsyshr.aau.dk/program/

Panel

Finally, as every year we are hosting a panel on job recommendation, algorithmic hiring, and related HR Tech tasks, and I am happy share our full list of invited panelists! Each year we try to strike a balance and find different perspectives and angles in the broad field where AI, RecSys, and HR meet. I think we’ve done a pretty good job this year, if I may say so myself, considering the following list of panelists:

Excited to have these experts share their insights at our workshop! For more details, including small bios of all our panelists, please see: https://recsyshr.aau.dk/panel/

RecSys in HR 2024 CFP published

We have published the call for papers for our Fourth Workshop on Recommender Systems for Human Resources (RecSys in HR 2024), to be held at the 18th ACM Conference on Recommender Systems in October in Bari 🇮🇹!

Do you work in AI, HR, and/or RecSys? Please consider submitting your work to our workshop! We accept research and position papers between 4-10 pages.

The (most) important dates for authors:

  1. Paper submission deadline: August 23, 2024
  2. Notification of acceptance: September 17
  3. Workshop date: 14-18 October (exact date TBD)

Take a look at the previous years’ proceedings for inspiration on the type of work that gets published:

The 4th Workshop for RecSys in HR at RecSys2024!

We received notification that our workshop for recommender systems in human resources (RecSys in HR) will be included in the workshop program of the ACM RecSys 2024 Conference which will be held in Bari, Italy 🇮🇹!

This will be the fourth consecutive edition of our workshop at the RecSys conference, following Amsterdam in 2021, Seattle in 2022, and Singapore last year. Very much looking forward to organizing another edition with my fellow workshop organizers: Toine Bogers, Mesut Kaya, Chris Johnson, Jens-Joris Decorte, and excited for welcoming Tijl De Bie (Universiteit Gent) to our organizing team!

Stay tuned for updates at our website: https://recsyshr.aau.dk/ (where for now you can find all proceedings and programs for the last three editions)

Talk on RecSys, NLP, and bias in hiring at the NLP4HR Workshop at EACL2024

I was honored to give the opening talk at the NLP4HR workshop in ☀️ sunny St. Julians, Malta! (bit bummed out I got to do it remotely from 🌧️ overcast and rainy Diemen, The Netherlands 🥸).

I gave a talk on recommender systems, bias, and bias mitigation in hiring with a focus on NLP challenges and solutions, where I adressed some of the open standing challenges in bias in textual data and features, some mitigation strategies, and the overall importancy and urgency of the topic, in light of incoming 🇪🇺 legislation. Such a fairness-filled week, this week 😅.

In addition, our former intern Lois Rink presented her master thesis “Aspect-Based Sentiment Analysis for Open-Ended HR Survey Responses” (co-authored with Job Meijdam and myself) at the workshop.

Thanks to Estevam Hruschka and Naoki Otani of Megagon Labs for the invitation and excellent workshop!

randstad position paper on AI published

Yesterday, randstad published the position paper titled “the labor market and AI.”

It contains some references to the ethical AI work I’ve been working on with my team, specifically: our efforts in internal auditing and bias mitigation strategies for AI-based matching systems, and our membership of the FINDHR consortium in which we collaborate with a number of European academic institutions, industry and civil society partners on fair and non-discriminatory algorithms for human recommendation.

Download the white paper here.

FINDHR CV Data Donation Campaign

🗣️ Please consider donating your (anonymized) CV to advance research into bias mitigation in algorithmic hiring!

With Randstad we are part of a consortium of research institutions (e.g., University of Amsterdam, Radboud Universiteit, Universitat Pompeu Fabra), civil society organizations (e.g., AlgorithmWatch), and companies (e.g., Adevinta) under the EU-funded FINDHR research project.

The FINDHR project aims to:
1️⃣ create new ways to measure algorithmic bias,
2️⃣ propose technical implementations for bias mitigation strategies, and
3️⃣ meaningfully incorporate human expertise
in algorithmic hiring systems (i.e., job/job seeker recommender systems).

To achieve these ambitious goals, the project requires real CVs and résumés. For that reason, FINDHR has initiated a CV donation campaign, where you’ll be able to donate your (anonymized) CV with just a few clicks. These donated CVs will be used to generate a dataset of realistic-but-fake synthetic CVs, that will serve as the basis for studying and developing bias and bias mitigation in job/job seeker recommender systems.

Your donated data will be safe: stored securely, can be deleted/withdrawn at any time upon request, and only accessible to authorized persons in the FINDHR research project who are required to sign confidentiality agreements.

Please consider donating your CV to accelerate research into bias and bias mitigation strategies for algorithmic hiring systems! For more details, check the donation campaign’s FAQ (or ping me!).

Donate your CV with just a couple of clicks here: findhr.eu/datadonation!

RecSys in HR 2023 Workshop at ACM RecSys 2023

Very excited that this year too I will engage in one of my favorite “extra-curricular activities” 😅: our proposal to organize the (third!) “Recommender Systems in Human Resources” workshop has been accepted in the the program of the 17th ACM Conference on Recommender Systems!

After our first edition in Amsterdam in 2021, and the second in Seattle in 2022, this year our workshop will be co-located with the RecSys conference in Singapore!

Looking forward to creating an engaging program with our fantastic organizing committee consisting of Toine Bogers (IT University of Copenhagen), Mesut Kaya (Aalborg University Copenhagen), Chris Johnson (Indeed.com), and welcoming Jens-Joris Decorte (of TechWolf) on board this year!

Call for papers forthcoming! Keep an eye on our workshop’s website: https://recsyshr.aau.dk/

The 2nd RecSys in HR Workshop

And that’s a wrap! Yesterday we had our RecSys in HR Workshop at the ACM RecSys 2022 conference.

👉 I was happy to (virtually) chair the workshop’s panel (with Helen Hulsker, Carlos Castillo (ChaTo), Liangjie Hong, and Robyn Rap). And hope the rest of the panel will take Helen’s suggestion to heart and book a ☕️ meeting with their lawyer colleagues soon to discuss matters of privacy, compliance, and ethics in the context of AI in HR 😉.
👍 I was impressed by the thorough infrastructures and shared/reusable job and job seeker representations that serve as a foundational component for many downstream products at LinkedIn (as told by Liangjie Hong during his keynote),
👏 Inspired by the strong ties between UX Research and Data Science at Indeed.com as shared by Robyn Rap in her keynote
💪 Proud for seeing our former interns Adam Mehdi Arafan and Roan Schellingerhout present their master theses at the workshop — work that came out of their internships with us at Randstad Groep Nederland!

Many thanks to my co-organizers, in particular the local Seattle team Chris Johnson and Toine Bogers, but also my remote fellows Mesut Kaya and Sepideh Mesbah for pulling an all-nighter 🌛.

Who knows, perhaps we meet again in Singapore next year 😁 (#RecSys2023).

Three papers accepted at RecSys in HR 2022 Workshop

🎉 A little success to share: three of our former data science interns at the Data Science chapter at Randstad Groep Nederland have written and published their master theses at our upcoming RecSys in HR Workshop; an academic workshop that revolves around AI in HR, which is part of an ACM International Conference on Recommender Systems (the AI systems used for matching; whether it is Netflix movies to users, or in our case; jobs to job seekers).

As always, the work of the students is pretty technical, but I will go ahead and try to provide little human-understandable summaries below.

Explainable Career Path Predictions using Neural Models

Roan Schellingerhout worked under supervision of Volodymyr Medentsiy on Explainable Career Path Prediction using Neural Networks, where he trained deep neural networks on our own talent work history data, to create a tool that can help consultants or talents to predict possible career switches, given as input a talent’s work history. The predictions are visually explained, in the sense that the underlying reasons for proposing a certain job are provided. Roan tested these visualizations on consultants, and found consultants generally like them.

End-to-End Bias Mitigation in Candidate Recommender Systems with Fairness Gates

  • [PDF] A. M. Arafan, D. Graus, F. P. Santos, and E. Beauxis-Aussalet, “End-to-end bias mitigation in candidate recommender systems with fairness gates,” in Recsys in hr’22: the 2\textsuperscriptnd workshop on recommender systems for human resources, 2022.
    [Bibtex] 
    @inproceedings{arafan2022end,
author = {Arafan, Adam Mehdi and Graus, David and Santos, Fernando P. and Beauxis-Aussalet, Emma},
title = {End-to-End Bias Mitigation in Candidate Recommender Systems with Fairness Gates},
year = {2022},
booktitle = {RecSys in HR’22: The 2\textsuperscript{nd} Workshop on Recommender Systems for Human Resources},
numpages = {8},
location = {Seattle, WA, USA and Online},
series = {CEUR Workshop Proceedings},
url = {https://ceur-ws.org/Vol-3218/RecSysHR2022-paper_6.pdf},
month={9}
}

Adam Arafan worked under supervision of myself on “End-to-End Bias Mitigation in Candidate Recommender Systems with Fairness Gates,” in his thesis he experimented with making the SmartMatch Talent Recommender more fair (at the level of gender), either by changing the “input” of the algorithm (for example; by balancing male and female candidates in the training data), or by changing its “output” (for example: for a given list of candidates, go through the list to make sure the top 10 has a 50/50 balance between male and female candidates). His work is novel because these type of “bias mitigation” strategies have been studied in isolation, but never together.

Automated Personnel Scheduling with Reinforcement Learning and Graph Neural Networks

  • [PDF] B. Platten, M. Macfarlane, D. Graus, and S. Mesbah, “Automated personnel scheduling with reinforcement learning and graph neural networks,” in Recsys in hr’22: the 2\textsuperscriptnd workshop on recommender systems for human resources, 2022.
    [Bibtex] 
    @inproceedings{platten2022automated,
author = {Platten, Benjamin and Macfarlane, Matthew and Graus, David and Mesbah, Sepideh},
title = {Automated Personnel Scheduling with Reinforcement Learning and Graph Neural Networks},
year = {2022},
booktitle = {RecSys in HR’22: The 2\textsuperscript{nd} Workshop on Recommender Systems for Human Resources},
numpages = {10},
location = {Seattle, WA, USA and Online},
url = {https://ceur-ws.org/Vol-3218/RecSysHR2022-paper_1.pdf},
series = {CEUR Workshop Proceedings},
month={9}
}

Ben Platten worked under supervision of Sepideh Mesbah on Automated Personnel Scheduling with Reinforcement Learning and Graph Neural Networks, in which he experimented with “reinforcement learning” (a specific machine learning paradigm) which in theory suits the challenging task of scheduling well. He experimented on a toy problem to assess that, indeed, the method seems to work quite well.

See the full list of accepted papers here: https://recsyshr.aau.dk/accepted-papers/.

And stay tuned for the pre-prints, which I’ll share as soon as they’re available!

Another cohort of Data Science students finished

Very proud of the latest cohort of Data Science thesis interns at Randstad Groep Nederland. In absence of a “real” defense at the University of Amsterdam, we organized our own afternoon packed with defenses (and subsequent drinks) in our Randstad HQ in Diemen. At the end of the afternoon we were able to congratulate Roan, Anna, and Adam on a job (almost) well done!

Roan Schellingerhout presented his work on “Explainable career Path Predictions.” Roan implemented explainable deep neural nets for predicting and explaining a job seekers’ next opportunity, given their previous. He evaluated the models intrinsically, in addition to testing them (+ their explanations) with actual recruiters, and found both that models are accurate and recruiters like and understand them.

Anna Lőrincz worked on data-to-text generation, and fine-tuned a multilingual transformer model for generating benefits (salary, contract, working hours, locations) in job descriptions in both Dutch and English, given structured information (numeric, categorical, and binary variables). She found that transformers can successfully generate fluent and correct text given structured inputs, confirmed that inputs or prompts have a high impact on performance, and found that her approach beats template-based methods in textual diversity. She also found a few very funny hallucinated work locations (“pal achter centraal station in Zwaaijdijk”, was one of our favorites), and found that transformer models tend to sometimes correct output (adjusting a 3k/hour salary into a 3k monthly salary).

Finally, Adam Mehdi Arafan presented his “Double Fair-Gated Bias Mitigation Pipeline” for our Talent Recommender system, where he studied bias in multiple parts of our recsys pipeline, from re-balancing training data (to simulate both balaned and highly imbalanced scenarios), to generating additional balanced synthetic data, and re-ranking outputs. Turns out applying synthetic data does not only help in creating more fair rankers, but can also have benefits in terms of model accuracy!

All three students did great jobs, stay tuned for their thesises (and, who knows, publications? 😏)

RecSys in HR at ACM RecSys 2022 in Seattle!

Fantastic news! We’ve received word the 2nd edition of our “Recommender Systems for Human Resources” (RecSys in HR) Workshop has been accepted to be included in the ACM RecSys 2022 program, to be held in Seattle!

Last year’s (first) edition of our workshop was co-located with ACM RecSys 2021 in Amsterdam, and featured two keynotes, a panel, breakout sessions and 8 paper presentations. The recording, workshop proceedings, and a workshop report are available through our workshop’s website at: https://recsyshr2021.aau.dk/

Check back there soon for information on the 2022 edition we’re planning with Toine Bogers, Mesut Kaya, Francisco Gutiérrez, and newly joined co-organizers Sepideh Mesbah (Randstad Groep Nederland) and Chris Johnson (Indeed.com)!

At KINTalks on AI in HR

I’m giving a talk at the KINTalks series, organized by the KIN Center for Digital Innovation on March 25. It’s going to be a hybrid event, so happy to meet you at the VU Amsterdam, and if not, see you online! RSVP here on EventBrite.

KINTalks is a hybrid event where practitioners are invited to talk about their work experience regarding innovation and digital technology.

The blurb

At Randstad, the global leader in the HR services industry, searching and matching is at the heart of what we do. Being founded in 1960, We know from our heritage that real connections are not made from data and algorithms alone – they require human involvement. Last year, we helped more than two million job seekers find a meaningful job by combining industry-scale recommender and search systems with our distinct human touch. While many opportunities exist, employing AI in recruitment and HR is considered high-risk by the European Commission’s proposed regulatory framework on AI, which will bring additional requirements, obligations, and constraints.

In this hybrid talk, I will explain some of the characteristics of, challenges, and opportunities in the HR domain from an AI perspective. I will share some of our own work in recommendations, algorithmic matching, algorithmic bias and knowledge graphs, and highlight some of the ongoing research in this domain.