Home

Awesome

<!-- markdownlint-disable MD033 --> <!-- markdownlint-disable MD041 --> <p align="center"> <h1 align="center"> <img src="https://github.com/YiVal/YiVal/assets/1544154/b0c681e7-7474-4b87-9c69-fde6e0e47401" alt="YiVal Logo" width="100" height="100" style="vertical-align: middle;"> YiVal </h1> <p align="center">⚡ Auto Prompting ⚡</p> </p> <!-- markdownlint-disable-next-line MD013 -->

👉 Follow us: Twitter | Discord

👉 Sponsored by Discord AIGC community: Discord

License: MIT GitHub star chart Open Issues

What is YiVal?

YiVal: Your Automatic Prompt Engineering Assistant for GenAI Applications YiVal is a state-of-the-art tool designed to streamline the tuning process for your GenAI app prompts and ANY configs in the loop. With YiVal, manual adjustments are a thing of the past. This data-driven and evaluation-centric approach ensures optimal prompts, precise RAG configurations, and fine-tuned model parameters. Empower your applications to achieve enhanced results, reduce latency, and minimize inference costs effortlessly with YiVal!

Problems YiVal trying to tackle:

  1. Prompt Development Challenge: "I can't create a better prompt. A score of 60 for my current prompt isn't helpful at all🤔."
  2. Fine-tuning Difficulty: "I don't know how to fine-tune; the terminology and numerous fine-tune algorithms are overwhelming😵."
  3. Confidence and Scalability: "I learned tutorials to build agents from Langchain and LlamaIndex, but am I doing it right? Will the bot burn through my money when I launch? Will users like my GenAI app🤯?"
  4. Models and Data Drift: "Models and data keep changing; I worry a well-performing GenAI app now may fail later😰."
  5. Relevant Metrics and Evaluators: "Which metrics and evaluators should I focus on for my use case📊?"

Check out our quickstart guide!

<img src="https://github.com/YiVal/YiVal/assets/1544154/dba5acd9-995c-45fd-9d08-c7cf198a77ad">

Link to demo

Tiktok title autotune

Installation

Docker Runtime

Install Docker and pull ourimage on DockerHub:

docker pull yival/release:latest

Run our image:

docker run --it yival/release:latest

VSCode with Docker extension is recommended for running and developments. If you are developer using GPU with Pytorch, or need jupyter lab for data science:

docker pull yival/release:cu12_torch_jupyter
docker run --gpus all --it -p 8888:8888 yival/release:cu12_torch_jupyter

Prerequisites

Installation Methods

Using pip (Recommended for Users)

Install the yival package directly using pip:

pip install yival

Development Setup Using Poetry

If you're looking to contribute or set up a development environment:

  1. Install Poetry: If you haven't already, install Poetry.

  2. Clone the Repository, or use CodeSpace:

    2.1 Use CodeSpace The easiest way to get YiVal enviornment. Click below to use the GitHub Codespace, then go to the next step.

    Open in GitHub Codespaces

    2.2 Clone the Repository

    git clone https://github.com/YiVal/YiVal.git
    cd YiVal
    
  3. Setup with Poetry: Initialize the Python virtual environment and install dependencies using Poetry. Make sure to run the below cmd in /YiVal directory:

    poetry install --sync
    

Trying Out YiVal

After setting up, you can quickly get started with YiVal by generating datasets of random tech startup business names.

Steps to Run Your First YiVal Program

  1. Navigate to the yival Directory:

    cd /YiVal/src/yival
    
  2. Set OpenAI API Key: Replace $YOUR_OPENAI_API_KEY with your actual OpenAI API key.

    On macOS or Linux systems,

    export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY
    

    On Windows systems,

    setx OPENAI_API_KEY $YOUR_OPENAI_API_KEY
    
  3. Define YiVal Configuration: Create a configuration file named config_data_generation.yml for automated test dataset generation with the following content:

    description: Generate test data
    dataset:
      data_generators:
        openai_prompt_data_generator:
          chunk_size: 100000
          diversify: true
          model_name: gpt-4
          input_function:
            description: # Description of the function
              Given a tech startup business, generate a corresponding landing
              page headline
            name: headline_generation_for_business
            parameters:
              tech_startup_business: str # Parameter name and type
          number_of_examples: 3
          output_csv_path: generated_examples.csv
      source_type: machine_generated
    
  4. Execute YiVal: Run the following command from within the /YiVal/src/yival directory:

    yival run config_data_generation.yml
    
  5. Check the Generated Dataset: The generated test dataset will be stored in generated_examples.csv.

Please refer to YiVal Docs Page for more details about YiVal!

Demo

Demo Video

Use Case DemoSupported FeaturesGithub LinkVideo Demo Link
🐯 Craft your AI story with ChatGPT and MidJourneyMulti-modal support: Design an AI-powered narrative using YiVal's multi-modal support of simultaneous text and images. It supports native and seamless Reinforcement Learning from Human Feedback(RLHF) and Reinforcement Learning from AI Feedback(RLAIF). Please watch the video above for this use case.Open In GitHubOpen In Youtube
🌟 Evaluate performance of multiple LLMs with your own Q&A test datasetConvenientlyevaluate and compare performance of your model of choice against 100+ models, thanks to LiteLLM. Analyze model performance benchmarks tailored to your customized test data or use case.Open In GitHubOpen In Youtube
🔥 Startup Company Headline Generation BotStreamline generation of headlines for your startup with automated test datacreation, prompt crafting, results evaluation, and performance enhancement via GPT-4.Open In GitHubOpen In Youtube
🧳 Build a Customized Travel Guide BotLeverageautomated prompts inspired by the travel community's most popular suggestions, such as those from awesome-chatgpt-prompts.Open In GitHubOpen In Youtube
📖 Build a Cheaper Translator: Use GPT-3.5 to teach Llama2 to create a translator with lower inference costUsingReplicate and GPT-3.5's test data, you can fine-tune Llama2's translation bot. Benefit from 18x savings while experiencing only a 6% performance decrease.Open In GitHubOpen In Youtube
🤖️ Chat with Your Favorite Characters - Dantan Ji from Till the End of the MoonBring your favorite characters to life through automated prompt creation andcharacter script retrieval.Open In GitHubOpen In Youtube
🔍Evaluate guardrails's performance in generating Python(.py) outputsGuardrails: where are my guardrails? 😭 <br>Yival: I am here. ⭐️<br><br>The integrated evaluation experiment is carried out with 80 LeetCode problems in csv, using guardrail and using only GPT-4. The accuracy drops from 0.625 to 0.55 with guardrail, latency increases by 44%, and cost increases by 140%. Guardrail still has a long way to go from demo to production.Open In GitHubOpen In Youtube
🍨Visualize different foods around the world!🍱Just give the place where the food belongs and the best season to taste it, and you can get a video of the season-specific food!🤩Open In GitHubOpen In Youtube
🎈News article summary with CoDBy integrating the"Chain of Density" method, evaluate the enhancer's ability in text summarization.🎆 Using 3 articles points generated by GPT-4 for evaluation, the coherent score increased by 20.03%, the attributive score increased by 25.18%!, the average token usage from 2054.6 -> 1473.4(-28.3%) 🚀.Open In GitHubOpen In Youtube
🥐 Automated TikTok Title Generation BotWith only two input lines, you can easily createconcise and polished TikTok video titles based on your desired target audience and video content summaries. This is presented by our auto-prompt feature: the process is automated, so you can input your requirements and enjoy the results hassle-free!Open In GitHubOpen In Youtube

Contribution Guidelines

If you want to contribute to YiVal, be sure to review the contribution guidelines. We use GitHub issues for tracking requests and bugs. Please join YiVal's discord channel for general questions and discussion. Join our collaborative community where your unique expertise as researchers and software engineers is highly valued! Contribute to our project and be a part of an innovative space where every line of code and research insight actively fuels advancements in technology, fostering a future that is intelligently connected and universally accessible.

Contributors

<a href="https://github.com/YiVal/YiVal/graphs/contributors"> <img src="https://contrib.rocks/image?repo=YiVal/YiVal" /> </a> <p align="center"> <br> 🌟 YiVal welcomes your contributions! 🌟<p align="center"> 🥳 Thanks so much to all of our amazing contributors 🥳</p>

Paper / Algorithm Implementation

PaperAuthorTopicsYiVal ContributorData GeneratorVariation GeneratorEvaluatorSelectorEnhancerConfig
Large Language Models Are Human-Level Prompt EngineersYongchao Zhou, Andrei Ioan Muresanu, Ziwen HanYiVal Evolver, Auto-PromptingOpenAIPromptDataGeneratorOpenAIPromptVariationGeneratorOpenAIPromptEvaluator, OpenAIEloEvaluatorAHPSelectorOpenAIPromptBasedCombinationEnhancerconfig
BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix WuYiVal Evaluator, bertscore, rouge@crazycth--BertScoreEvaluator---
AlpacaEvalXuechen Li, Tianyi Zhang, Yann Dubois et. alYiVal Evaluator--AlpacaEvalEvaluator--config
Chain of DensityGriffin Adams Alexander R. Fabbri et. alPrompt Engineering-ChainOfDensityGenerator---config
Large Language Models as OptimizersChengrun Yang Xuezhi Wang et. alPrompt Engineering@crazycth----optimize_by_prompt_enhancerconfig
LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu Yelong Shen et. alLLM Finetune@crazycth----sft_trainerconfig
<!-- -->