Hamlet and IR Social Impact

There are more intervention strategies in heaven and earth, Horatio, than are dreamt of in your philosophy.
— Hamlet, sort-of

Tomorrow, I will be presenting our paper in the IR4Good track at ECIR 2024 on strategic IR impact interventions.

From my side, this paper grew out of a couple of things:

• Resonance between what I am trying to accomplish by working on information access fairness and what Sole is doing with IR for understudied and underserved users (also discussed in my IR4U2 keynote).
• Frustration with what I perceive as an overemphasis on algorithmic fixes and metric disparities, when there are a wide range of metrics, intervention sites, and strategies that we can use to improve social impacts such as fairness, harm reduction, etc. in information access systems.

Lex & Henriette joined us to broaden the argument to deal with an expansive notion of “impacts” of IR systems, including discrimination, misrepresentation, utility, accessibility, etc., and argue that effective IR impact work has four key characteristics:

1. It is grounded in specific impact goals.
2. It contextualizes those impacts and goals in the broad landscape of the ways an IR system can affect its consumers.
3. It flows from creative and expansive thinking about the range of possible methods and intervention sites for addressing that impact.
4. It matches the impact to a strategy that is appropriate given technical and organizational constraints and the specific ethical, legal, economic, and other dimensions of the impact concern.

We also provide three “maps” of impact goals, candidate operationalizations, and intervention sites & strategies to support such work. There is a wide range of possible intervention sites and corresponding strategies for adjusting IR system impact, some of which are not widely-discussed. One in particular is the engineering process: if we have a robust mediation analysis that identifies the causes of an impact or impact disparity, that can be used to inform process planning. For example, if users of a particular group are receiving lower-quality service, and it seems to be because the system is less effective at handling certain types of queries that they use more often, then the team could prioritize efforts on improving the system’s handling of those queries over other improvements that will primarily benefit users who already receive good results.

Read the paper for more details, and check out the talk if you’re at ECIR!