Senior Evaluation Algorithm Engineer
About the Company:
Binance is a prominent player in the global blockchain community, known for operating the largest cryptocurrency exchange globally based on trading volume and user base. With a widespread presence across 100+ countries, the platform is trusted by over 300 million users. Binance stands out for its strong commitment to security, transparency in user funds, speedy trading engine, deep liquidity, and an extensive range of digital asset products. The company offers a diverse array of services spanning trading, finance, education, research, institutional solutions, payment services, Web3 functionalities, and more. By harnessing the capabilities of digital assets and blockchain technology, Binance is dedicated to fostering a financial ecosystem that ensures financial access and enhances the freedom of money for individuals worldwide.
Job Responsibilities:
- Develop comprehensive evaluation plans for large language models (LLM) within various business contexts such as dialogues and financial trading. Create evaluation metrics and frameworks to objectively assess model performance and guide research and development initiatives effectively.
- Spearhead the creation of evaluation datasets and define evaluation parameters to accurately represent business needs. Implement robust data annotation guidelines and establish quality assurance protocols to ensure reliable benchmarks.
- Conduct in-depth analysis of model performance based on evaluation outcomes and offer actionable recommendations for enhancement. Collaborate closely with algorithm and product teams to facilitate continuous model refinement through iterative evaluation processes.
- Streamline and expand evaluation workflows through automation. Establish sustainable evaluation platforms and toolsets to support rapid and consistent model evaluation amidst frequent iterations.
- Collaborate with cross-functional teams including algorithm engineers, product managers, and data specialists to align on evaluation standards that align with business objectives. Translate evaluation insights into actionable development strategies and drive their implementation effectively.
Job Requirements:
- Need for a Master's degree or higher in relevant fields such as Computer Science, Artificial Intelligence, Mathematics, or Statistics. Strong foundational knowledge of algorithms and expertise in LLM principles, training, and fine-tuning processes.
- At least demonstrated hands-on experience in LLM evaluation within a sizeable tech organization, particularly in commercial deployment scenarios as opposed to purely academic exercises. Proficiency in the entire evaluation lifecycle from dataset preparation to influencing R&D decisions based on evaluation outcomes.
- Familiarity with mainstream evaluation methods and the ability to develop applicable evaluation dimensions tailored to diverse business scenarios. Proficient in Python and experienced in automating evaluation workflows, constructing benchmarks, and developing evaluation platforms.
- Proven ability to ensure the representativeness, consistency, and reliability of evaluation data for scientifically sound and credible conclusions. Strong communication skills and business acumen to effectively communicate evaluation findings and drive collaboration across teams.
Additional Preferred Qualifications:
- Prior experience in evaluating dialogue systems, AI Agents, or financial/trading-oriented LLMs.
- Background in designing high-quality AI training/evaluation data or developing data annotation systems.
- Familiarity with RLHF, reward models, or preference data-related projects.
Why Binance:
- Contribute to shaping the future in collaboration with a diverse, skilled global team in an organization known for its user-centered approach and flat hierarchy.
- Enjoy the autonomy to work on unique, fast-paced projects within an innovative environment that promotes continuous learning and career growth opportunities.
- Experience a results-driven workplace with competitive remuneration, company benefits, and the flexibility of a work-from-home arrangement where suited to the business operations.
