Prompt Engineer & LLMOps Specialist
We’re seeking a candidate with a solid grasp of English and Python fundamentals, expertise in LLMOps, familiarity with current prompting trends, and enthusiasm for AI advancements. This role will focus on developing innovative solutions for Voice AI and automation projects, making technology helpful for our clients while ensuring robust, scalable, and monitored LLM implementations.
Required Qualifications:
- Extensive experience with LLM prompting techniques (e.g., RAG, CoT, function calls, few-shot, chaining)
- Strong proficiency in LLMOps: evaluating and monitoring accuracy, latency, costs, and relevant metrics
- Familiarity with LLMOps tools for systematic tracking and evaluation of prompts and fine-tuning
- Expertise in designing and implementing robust evaluation frameworks for LLMs in production
- Strong proficiency in English
- Ability to write clean, readable Python code, and capability to understand and modify existing scripts
- Proficiency with APIs, SQL databases, and Git
- Solid understanding of data processing concepts, cloud platforms, and machine learning fundamentals
Nice to Have:
- Experience developing custom LLM utilities and tools
- Conceptual understanding of neural network and LLM internals
- Hands-on experience with LLM fine-tuning and hyperparameter optimization
- Familiarity with A/B testing and experimentation frameworks for LLMs
- Client management and communication skills
- Experience with CI/CD pipelines for LLM-based applications
Key Responsibilities:
- Develop, test, and continuously optimize prompts tailored to specific client needs
- Design and implement robust systems for evaluating and monitoring LLM performance in production environments
- Utilize LLMOps tools to ensure systematic tracking, versioning, and evaluation of prompts and fine-tuning experiments
- Implement logging and monitoring solutions to track LLM performance, usage patterns, and potential drift
- Develop and maintain dashboards for real-time monitoring of LLM metrics and KPIs
- Collaborate with data scientists and ML engineers to integrate LLMs into larger ML pipelines and systems
- Actively participate in project reviews and creative brainstorming sessions, contributing both technical expertise and innovative ideas
- Engage directly with clients to understand their requirements, align expectations, and provide technical guidance
- Stay current with the latest developments in LLM technology, prompting techniques, and LLMOps best practices
- Contribute to the development of internal tools and frameworks for streamlining LLMOps processes
LLMOps Focus:
- Implement version control for prompts, model weights, and datasets
- Set up automated testing pipelines for prompt evaluation and model performance
- Develop strategies for prompt optimization and iterative improvement based on production data
- Implement safeguards and fallback mechanisms for LLM-based systems
- Collaborate on developing cost-effective strategies for LLM deployment and scaling
The ideal candidate will bring a strong technical background in LLMs and a methodical approach to operationalizing AI systems. They will be passionate about pushing the boundaries of what’s possible with language models while ensuring reliable, scalable, and ethically sound implementations.
