RecSys in HR at ACM RecSys 2022 in Seattle!

📅 April 10, 2022 • 🕐 09:37 • 🏷 Blog and Research

Fantastic news! We’ve received word the 2nd edition of our “Recommender Systems for Human Resources” (RecSys in HR) Workshop has been accepted to be included in the ACM RecSys 2022 program, to be held in Seattle!

Last year’s (first) edition of our workshop was co-located with ACM RecSys 2021 in Amsterdam, and featured two keynotes, a panel, breakout sessions and 8 paper presentations. The recording, workshop proceedings, and a workshop report are available through our workshop’s website at: https://recsyshr2021.aau.dk/

Check back there soon for information on the 2022 edition we’re planning with Toine Bogers, Mesut Kaya, Francisco Gutiérrez, and newly joined co-organizers Sepideh Mesbah (Randstad Groep Nederland) and Chris Johnson (Indeed.com)!

Two papers accepted at the RecSys in HR Workshop!

📅 August 23, 2021 • 🕐 12:10 • 🏷 Research

Happy to have learned we have two papers accepted at the first Recommender Systems in Human Resources Workshop, co-located with ACM RecSys 2021! These papers are the first academic publications of the Data Science Chapter at Randstad Groep Nederland!

  • [PDF] M. de Groot, J. Schutte, and D. Graus, “Job posting-enriched knowledge graph for skills-based matching,” in Recsys in hr 2021, Amsterdam, Netherlands, 2021.
    [Bibtex]
    @inproceedings{degroot2021job,
    author = {de Groot, Maurits and Schutte, Jelle and Graus, David},
    title = {Job Posting-Enriched Knowledge Graph for Skills-based Matching},
    year = {2021},
    booktitle = {RecSys in HR 2021},
    numpages = {9},
    location = {Amsterdam, Netherlands},
    address = {Amsterdam, Netherlands},
    month={10}
    }
  • [PDF] D. Lavi, V. Medentsiy, and D. Graus, “Consultantbert: fine-tuned siamese sentence-bert for matching jobs and job seekers,” in Recsys in hr 2021, Amsterdam, Netherlands, 2021.
    [Bibtex]
    @inproceedings{lavi2021consultantbert,
    author = {Lavi, Dor and Medentsiy, Volodymyr and Graus, David},
    title = {conSultantBERT: Fine-tuned Siamese Sentence-BERT for Matching Jobs and Job Seekers},
    year = {2021},
    booktitle = {RecSys in HR 2021},
    numpages = {8},
    location = {Amsterdam, Netherlands},
    address = {Amsterdam, Netherlands},
    month={10}
    }

Curious to know what they’re about? I tweet better than I blog 👇

Stay tuned for pre-prints! See the other accepted papers here.

Disclaimer: yes, I co-organize the workshop, but I was not involved with reviewing/decisions, we have a great (and independent) Program Committee for that!

Co-organizing “RecSys in HR” workshop at RecSys 2021!

📅 March 18, 2021 • 🕐 12:14 • 🏷 Blog and Research

We received news that our workshop proposal “RecSys in HR: Workshop on Recommender Systems for Human Resources” was accepted for inclusion in the 15th ACM Conference on Recommender Systems (RecSys 2021) program! That means we’ll be running a full-day workshop with (research and position) papers, keynotes, and a panel (all TBD) during the conference which will be held in Amsterdam, 27th September-1st October 2021.

We wrote this workshop proposal with Toine Bogers (Aalborg University), Mesut Kaya (Aalborg University), Katrien Verbert (KU Leuven) and Francisco Gutiérrez (KU Leuven), at the initiative/idea of Toine, who virtually approached me in RecSys 2020’s gather.town :-D. Toine and Mesut work on a large research project with Denmark’s largest online recruitment portal, JobIndex.

For now, check out our stunning stub page at https://recsyshr2021.aau.dk/ and stay tuned for updates!

“Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation” accepted at UMAP2020!

📅 April 21, 2020 • 🕐 17:33 • 🏷 Papers and Research

The paper we wrote with former FD team mates Feng Lu and Anca Dumitrache has been accepted for publication as a long paper at UMAP 2020, the 28th Conference on User Modeling, Adaptation and Personalization! (I fondly remember my last time at UMAP, in 2016 😏)

We have published a preprint of this paper, get it: here, or from arXiv.

  • [PDF] [DOI] F. Lu, A. Dumitrache, and D. Graus, “Beyond optimizing for clicks: incorporating editorial values in news recommendation,” in Proceedings of the 28th acm conference on user modeling, adaptation and personalization, New York, NY, USA, 2020, p. 145–153.
    [Bibtex]
    @inproceedings{lu2020beyond,
    author = {Lu, Feng and Dumitrache, Anca and Graus, David},
    title = {Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation},
    year = {2020},
    isbn = {9781450368612},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3340631.3394864},
    doi = {10.1145/3340631.3394864},
    booktitle = {Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization},
    pages = {145–153},
    numpages = {9},
    keywords = {usefulness, news recommendation, editorial values},
    location = {Genoa, Italy},
    series = {UMAP ’20}
    }

Update 08/05: Cool, @NickKivits mentioned our paper in his Villamedia column: Het idee van de filterbubbel kan in de prullenbak and newsletter (with over 11k subscribers!)

I am particularly happy with this work because:

1️⃣ In our paper we show how you can align algorithm design across stakeholders (in this case: data scientists and journalists), by effectively modeling an editorial value (“dynamicness”) in the news recommender of Het Financieele Dagblad without losing accuracy.

2️⃣ We present (more) empirical proof that #recsys (can) offer(s) users *more* diverse, serendipitous, and dynamic lists of articles, compared to editorially curated lists, and hence (can) help in *avoiding*, not creating filter bubbles!

3️⃣ It is the perfect wrap-up of our Google DNI-funded “SMART Journalism” project at FD Mediagroep (we wrote most of the paper in our spare time after the project ended).

See below the video of the talk at UMAP 2020 below:

(more…)

PodRecs: Workshop on Podcast Recommendations PC

📅 April 4, 2020 • 🕐 12:14 • 🏷 Research

I was invited to join the program committee of (the first) PodRecs: Workshop on Podcast Recommendations (to be held at RecSys’20).

Since our work on BNR SMART Radio, I am really interested in the space of audio, recommender systems, and information retrieval. Curious to see the submissions!

See the PodRecs call for papers, and check out the website, by clicking the image below.

“Improving automated segmentation of radio shows with audio embeddings”

📅 July 5, 2019 • 🕐 15:41 • 🏷 Blog and Research

Update (28/1/2020): Oberon’s thesis was accepted and will be published at the IEEE 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020), to be held May 4-8 in Barcelona, Spain! The submission is co-authored with Klaus Lux and myself.

Oberon Berlage recently successfully defended his MSc. thesis (title above!) for the Data Science Master at University of Amsterdam, and graduated with a whopping 9!

He’s the first academic offspring of our AI Team @ FD Mediagroep, and worked on BNR SMART Radio‘s segmenter. Oberon improved our text-based segmenter by adding audio embeddings, improving the F1 score with +32%!

His thesis is now online, check it out at: http://scriptiesonline.uba.uva.nl/document/673254

James Chen Best Student Paper Award at UMAP 2016

📅 July 18, 2016 • 🕐 20:58 • 🏷 Blog and Research

Our paper,

  • [PDF] [DOI] D. Graus, P. N. Bennett, R. W. White, and E. Horvitz, “Analyzing and predicting task reminders,” in Proceedings of the 2016 conference on user modeling adaptation and personalization, New York, NY, USA, 2016, p. 7–15.
    [Bibtex]
    @inproceedings{graus2016analyzing,
    author = {Graus, David and Bennett, Paul N. and White, Ryen W. and Horvitz, Eric},
    title = {Analyzing and Predicting Task Reminders},
    year = {2016},
    isbn = {9781450343688},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/2930238.2930239},
    doi = {10.1145/2930238.2930239},
    booktitle = {Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization},
    pages = {7–15},
    numpages = {9},
    keywords = {prospective memory, reminders, log studies, intelligent assistant},
    location = {Halifax, Nova Scotia, Canada},
    series = {UMAP '16}
    }

was awarded best student paper, at UMAP 2016!

Me receiving the award during the UMAP banquet dinner at the Marriott Harbourfront Hotel, Halifax. Photo by Denis Parra.

Improving User Productivity with Automated Personal Assistants: Analyzing and Predicting Task Reminders

📅 June 1, 2016 • 🕐 11:05 • 🏷 Blog and Research

Update (16/07): This paper was awarded the James Chen Best Student Paper Award at UMAP!

Cortana on Windows Phone (source: The Verge)

Automated personal assistants such as Google Now, Microsoft Cortana, Siri, M and Echo aid users in productivity-related tasks, e.g., planning, scheduling and reminding tasks or activities. In this paper we study one such feature of Microsoft Cortana: user-created reminders. Reminders are particularly interesting as they represent the tasks that people are likely to forget. Analyzing and better understanding the nature of these tasks could prove useful in inferring the user’s availability, aid in developing systems to automatically terminate ongoing tasks, allocate time for task completion, or pro-actively suggest (follow-up) tasks.

  • [PDF] [DOI] D. Graus, P. N. Bennett, R. W. White, and E. Horvitz, “Analyzing and predicting task reminders,” in Proceedings of the 2016 conference on user modeling adaptation and personalization, New York, NY, USA, 2016, p. 7–15.
    [Bibtex]
    @inproceedings{graus2016analyzing,
    author = {Graus, David and Bennett, Paul N. and White, Ryen W. and Horvitz, Eric},
    title = {Analyzing and Predicting Task Reminders},
    year = {2016},
    isbn = {9781450343688},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/2930238.2930239},
    doi = {10.1145/2930238.2930239},
    booktitle = {Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization},
    pages = {7–15},
    numpages = {9},
    keywords = {prospective memory, reminders, log studies, intelligent assistant},
    location = {Halifax, Nova Scotia, Canada},
    series = {UMAP '16}
    }

Prospective memory

Studying things that people tend to forget has a rich history in the field of social psychology. This type of memory is called “Prospective memory” (or more poetically written: “Remembrance of Things Future“). One challenge in studying PM is that its hard to simulate in a lab study (the hammer of choice for social psychologists). For this reason, most studies of PM have been restricted to “event-based” PM, i.e., memories triggered by an event, modeled in a lab through having someone perform a mundane task, and doing a special thing upon being triggered by an event. Furthermore, the focus in these studies has largely been on retention and retrieval performance of “artificial” memories: subjects were typically given an artificial task to perform. Little is known about the type and nature of actual, real-world, “self-generated” tasks.

Enter Cortana. The user logs we study in this paper represent a rich collection of real-life, actual, self-generated, time-based PM instances, collected in the wild. Studying them in aggregate allows us to better understand the type of tasks that people remind themselves about.

Big data

(Yes, sorry, that heading really says big data…) 
As the loyal reader may have guessed, this paper is the result of my internship at Microsoft Research last summer, and one of the (many) advantages of working at Microsoft Research is the restricted access to big and beautiful data. In this paper we analyze 576,080 reminders, issued by 92,264 people over a period of two months (and we later do prediction experiments on 1.5M+ reminders over a six month time period). Note that this is a filtered set of reminders (a.o. for a smaller geographic area, and we removed all users that only issued a few reminders). Furthermore, when analyzing particular patterns, we filter data to patterns commonly observed across multiple users to study behavior in aggregate and further preserve user privacy: we are not looking at the users behavior at the individual level, but across a large population, to uncover broad and more general patterns. So what do we do to these reminders? The paper consists of three main parts;

tasktypetaxonomy
temporalpatterns

1. Task type taxonomy: First, we aim to identify common types of tasks that underlie reminder setting, by studying the most common reminders found in the logs. This analysis is partly data-driven, and partly qualitative; as we are interested in ‘global usage patterns,’ we extract common reminders, defined as reminders that are seen across many users, that contain a common ‘action’ or verb. We do so by identifying the top most common verb phrases (and find 52 verbs that cover ~61% of the reminders in our logs), and proceed by manually labeling them into categories.

2. Temporal patterns: Next, we study temporal patterns of reminders, by looking at correlations between reminder creation and notification, and in temporal patterns for the terms in the reminder descriptions. We study two aspects of these temporal patterns: patterns in when we create and execute reminders (as a proxy to when people typically tend to think about/execute certain tasks), and the duration of the delay between the reminder’s creation and notification (as a proxy to how “far in advance” we tend to plan different things).

predict

3. Predict! Finally, we show how the patterns we identify above generalize, by addressing the task of predicting the day at which a reminder is likely to trigger, given its creation time and the reminder description (i.e., terms). Understanding when people tend to perform certain tasks could be useful for better supporting users in the reminder process, including allocating time for task completion, or pro-actively suggesting reminder notification times, but also for understanding behavior at scale by looking at patterns in reminder types.

Findings

As always, no exhaustive summary of the paper point-by-point here, straight into some of our findings (there’s much more in the paper):

  • We tend to plan for things (i.e., set reminders) at the end of day, and execute them (i.e., reminders trigger) throughout the day, which suggests the end of day is a natural moment for people to reflect upon the tasks that need to be carried out.
  • The types of things we remind ourselves about are mostly short-term, immediate, tasks such as performing daily chores.
  • People are more likely to call their mom, and email their dad.

Want to know more? See the taxonomy? See more pretty plots? Look at some equations? Learn how this could improve intelligent assistants? Read the paper!

  • [PDF] [DOI] D. Graus, P. N. Bennett, R. W. White, and E. Horvitz, “Analyzing and predicting task reminders,” in Proceedings of the 2016 conference on user modeling adaptation and personalization, New York, NY, USA, 2016, p. 7–15.
    [Bibtex]
    @inproceedings{graus2016analyzing,
    author = {Graus, David and Bennett, Paul N. and White, Ryen W. and Horvitz, Eric},
    title = {Analyzing and Predicting Task Reminders},
    year = {2016},
    isbn = {9781450343688},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/2930238.2930239},
    doi = {10.1145/2930238.2930239},
    booktitle = {Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization},
    pages = {7–15},
    numpages = {9},
    keywords = {prospective memory, reminders, log studies, intelligent assistant},
    location = {Halifax, Nova Scotia, Canada},
    series = {UMAP '16}
    }