Terms of Reference: External evaluator to evaluate the impact of language data and voice technology for speakers of low-resource languages

External evaluator to evaluate the impact of language data and voice technology for speakers of low-resource languages

Terms of reference

Location: Remote (home based)

Travel: None

Reporting to: CLEAR Global Program Technology Lead

Timeframe: February 2025-April 2025

Maximum available budget for assignment: USD 15,000. Please include your daily fee in USD in the ‘Desired Salary’ field in the application, and total quote as part of your technical and financial offer.

Deadline for applications: 31st January 2025

*Due to the urgency of this project, screening and interviews will commence immediately and the post may be filled before the deadline for applications.

BackgroundAlmost half the world’s population, 3.7 billion people, do not speak a “major” language. It’s no coincidence that those 3.7 billion also have the lowest incomes and are the most marginalized. Language technology and large language models are evolving rapidly, yet only a fraction of the world’s 7,000 languages are meaningfully online. As technology progresses and language models evolve rapidly, access to language tech is improving for some but creating a growing gap for others.

Consequently, the majority of the provided information services remain inaccessible to those who do not speak the dominant languages. Helplines and call centers are human-resource heavy, do not operate outside of office hours and are usually overloaded. Chatbots, IVR systems and automated messaging accessible via mobile phones are either text-based (requiring literacy) or menu-based with limited 2-way communication. Even in the rare cases when those services are powered by language AI such as voice conversational capability, or automated transcription services that support the efficient handling of requests from the community, they can only handle major languages. In essence, tech-powered services are designed for the literate, tech-savvy or speakers of languages of the humanitarian responders which leaves many communities in need completely excluded.

At the heart of the problem is the lack of functional Language AI models for marginalized languages, models required to build scalable, multilingual, voice-powered communication and information channels accessible for crisis-affected people. The lack of voice data exacerbates the problem; building or fine-tuning existing models, and ensuring fit-for-purpose performance requires language data that simply does not exist. Neither do scalable language collection data tools that allow the collection of quality voice data. The problem is particularly acute in crisis situations because datasets cannot be built during a crisis. Data must be collected before a crisis in order for it to be useful during a crisis.

While several organizations have initiated efforts to create voice datasets, through localized community drives or dedicated platforms, challenges such as cost, lack of engagement, and inability to replicate these efforts have resulted in disjointed non-scaleable initiatives.

CLEAR Global has initiated multiple activities to address these challenges. Efforts include:

1. Design, development and deployment of a voice data collection tool, TWB Voice
2. Gathering and transcribing conversational audio in Hausa, Kanuri, and Shuwa Arabic, with approximately 50 to 100 hours of recordings per language. The collected data aims to voice-enable tools and services to support conflict-affected communities in northeast Nigeria, particularly in Hausa and Kanuri. This initiative also involves the development of Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models, which will be openly shared with the community
3. An AI initiative focused on generating and testing synthetic voice data to minimize dependence on costly, human-curated resources. This approach aims to lower the resource demands for developing language technologies in African languages.

The role

CLEAR Global is looking for an external evaluator to deliver an evaluation assessing CLEAR Global’s efforts in advancing language technology for low-resource languages, focussing on projects involving African languages. The evaluation will address the following objectives:

• Understand current challenges and needs: Identify and baseline the needs for and challenges associated with building language technology for low-resource languages in the context of humanitarian and development work, including gaps in data, model and tool availability.
• Evaluate comparative advantage: Analyze other similar existing initiatives, assessing CLEAR Global’s comparative cost-effectiveness and functionality.
• Assess the Theory of Change: Evaluate CLEAR Global’s Theory of Change for building language technology, identifying gaps and validating assumptions, including CLEAR Global’s commitment to open-source technology and how it interacts with other goals.
• Ensure Sustainability: Determine the resources, partnerships, and strategies required to sustain and contribute to scaling language AI solutions for low-resource languages effectively.

The evaluation report will help to advance CLEAR Global’s work, but also help to get buy-in from other stakeholders to advance this work.

The evaluation is expected to start in early February 2025 and conclude by end April 2025.

Evaluation criteria and questions

Key questions to guide the evaluation include:

Challenges, gaps and barriers

1. What are the key challenges in building language technology for low-resource languages, particularly in collecting and utilizing voice data?
2. What gaps in data, models, and tools exist that must be addressed to advance language AI for low-resource languages?

Effectiveness of CLEAR Global’s Work

1. To what extent does CLEAR Global’s Theory of Change for building language technology hold true in practice?
   1. How likely is the potential target group to adopt and use the tools, data, and models being developed by CLEAR Global, and what are the potential barriers?

Comparative Advantage

1. What unique value does CLEAR Global bring to the field of language technology for low-resource languages?
2. How does CLEAR Global’s work align with or complement the efforts of other social impact organizations (e.g. CATAI, Viamo, World Vision) using language AI for services in African languages?
3. Where are the gaps that major tech companies and other stakeholders are unlikely to address in the future, and how should CLEAR Global invest to bridge these?
   1. What are the perceived impacts and trade-offs of building open-source technology and how do stakeholders view CLEAR Global's approach towards that?

Sustainability

1. What resources, partnerships, or internal systems changes are needed to ensure the sustainability of CLEAR Global’s language technology projects?

Methodology

The evaluation should adopt a mixed-methods approach to comprehensively assess CLEAR Global’s initiatives, combining quantitative data to measure impact with qualitative insights to explore challenges and successes. The evaluator should propose their preferred methodology but should consider the following key components:

Desk review: Analyze relevant project documents, including CLEAR Global’s Theory of Change, strategic plans, project documentation, benchmarks, datasets, and comparative analysis of similar initiatives.

Stakeholder consultations: Conduct semi-structured interviews and focus groups with key stakeholders, including CLEAR Global staff, linguists, humanitarian actors, and language AI researchers and other experts in the field. This would involve

1. Interviews and focus groups
2. 1-2 scheduled presentation, where CLEAR Global will present the work and findings in front of key experts to facilitate a discussion and get feedback in line with the evaluation questions

Case study: The evaluation will include an in-depth case study to be finalized in consultation with the evaluator. The case study will explore the potential impact of CLEAR Global’s language technology in a specific use case with an organization that would use the data and models produced by CLEAR Global’s projects. It will assess how access to this CLEAR Global’s technology, data or models could transform the organization's work, focusing on changes in reach, cost, and quality. The study will baseline the current state and estimate impacts of adopting CLEAR Global’s work. It will also examine the practical requirements for implementation, such as additional resources, capacity building, or training. This applied use case will provide concrete insights and numbers into how the work could improve communication and information delivery in the field.

Comparative analysis: Compare CLEAR Global’s models, tools, and processes with similar initiatives to evaluate cost-effectiveness, scalability, usability, and gaps unaddressed by major tech companies.

Deliverables

• Inception report, within 2.5 weeks of start of assignment: Detailed methodology, refined evaluation questions, and a work plan.
• Preliminary findings presentation: Presentation of initial findings to CLEAR Global for feedback and validation.
• Draft evaluation report: Comprehensive report covering methodology, findings, conclusions, and recommendations.
• Final evaluation report: Revised report incorporating feedback, ready for internal and external use.
• Summary brief: A concise document highlighting key findings and recommendations.
• Final presentation: A formal presentation of the evaluation results and recommendations to CLEAR Global stakeholders.
• Raw data and tools: All collected data and evaluation tools for future use by CLEAR Global.

Qualifications and experience required

• Proven understanding of language technology, AI, and NLP, with relevant experience
• Experience in AI for humanitarian and development contexts, including prior research on or work in these areas
• Familiarity with the role of technology in humanitarian and development work, including digital inclusion and ethical considerations
• Demonstrated ability to engage effectively with stakeholders, including technical experts, through interviews, workshops, and collaborative discussions
• Professional connections in the field of AI and language technology, particularly in building datasets and tools for low-resource languages
• Awareness of the challenges in advancing language technology for low-resource languages is a valuable asset
• Excellent communication skills in English, with the ability to create clear, concise, and impactful reports for varied audiences.

Terms and conditions

Payments will be made as follows:

• An advance of 20% of the total contract value within 30 days of the signature of the contract
• An interim payment of 20% of the total contract value upon approval of the inception report, raw data and tools, and presentation of initial findings
• A final installment of 60% of the total contract value payable on satisfactory completion of the work and presentation of all deliverables.

The evaluator will be expected to provide their own equipment and supplies (laptop etc.).

How to apply

CLEAR Global will accept offers from individual consultants and consulting firms.

To apply for this consultancy please send the following documents:

1- A technical and financial offer to include:

• a brief description (no more than two sides of A4) of how you would tackle this assignment
• a proposed work plan (including the number of days required for each task)
• a financial offer, specifying daily fees. Please also state your daily fee in USD in the ‘Desired Salary’ field in the application.
• examples and descriptions of relevant similar work are welcome.

2- Curriculum vitae highlighting experience from similar projects, as well as the contact details (email and telephone number) of at least three professional references.

Please upload the technical and financial offer as one document under “cover letter” and present the CV(s) of the expert(s) proposed in one document under “CV”.

About CLEAR Global

CLEAR Global helps people get vital information, and be heard, whatever language they speak. We believe that everyone has the right to give and receive information in a language and format they understand. We work with nonprofit partners and a global community of language professionals to build local language translation capacity, and raise awareness of language barriers. Our network of over 100,000 community members translate millions of words of life-saving and life-changing information a year.

Core values

CLEAR Global employees and volunteers passionately believe in the value of this work and take personal responsibility for achieving the mission. CLEAR Global’s mission and organizational spirit embody the core values established in its strategic framework:

• Excellence: In communicating humanitarian information in the right language, CLEAR Global is a leader in the translation industry and in the non-profit sector.
• Integrity: In believing that every person, whether it’s the people who we serve, our volunteers or our staff, has value, deserves respect and has inherent dignity.
• Empowerment: In using language to empower people around the world to control their own development and destiny.
• Innovation: In recognizing and celebrating the power of innovation to address humanitarian and crisis issues around the world.
• Sustainability: In recognizing that meeting our mission requires the establishment and maintenance of a solid financial and organizational infrastructure.
• Tolerance: In that our staff and volunteers value each other, our partners and our end users, create a supportive work environment, and conduct themselves professionally at all times.