Awesome

Open Interface

Full Autopilot for All Computers Using LLMs

Open Interface

• Self-drives computers by sending user requests to an LLM backend (GPT-4V, etc) to figure out the required steps.
• Automatically executes the steps by simulating keyboard and mouse input.
• Course-corrects by sending the LLMs a current screenshot of the computer as needed.

<br>

<ins>Demo</ins> 💻

["Make me a meal plan in Google Docs"]<br> <br> More Demos

<ins>Install</ins> 💽

<ins id="setup">Setup</ins> 🛠️

• Get your OpenAI API key

  • Open Interface needs access to GPT-4V to perform user requests. GPT-4V keys can be downloaded from your OpenAI account.
  • Follow the steps here to add balance to your OpenAI account. To unlock GPT-4V a minimum payment of $5 is needed.
  • More info

• Save the API key in Open Interface settings

  • In Open Interface, go to the Settings menu on the top right and enter the key you received from OpenAI into the text field like so: <br> <br>
  <picture> <img src="assets/set_openai_api_key.png" align="middle" alt="Set API key in settings" width="400"> </picture><br> <br>

• After setting the API key for the first time you'll need to restart the app.

• Open Interface supports using other OpenAI API style LLMs (such as Llava) as a backend and can be configured easily in the Advanced Settings window.
• Enter the custom base url and model name in the Advanced Settings window and the API key in the Settings window as needed. <br> <picture> <img src="assets/advanced_settings.png" align="middle" alt="Set API key in settings" width="400"> </picture><br> <br>
• If your LLM does not support an OpenAI style API, you can use a library like this to convert it to one.
• You will need to restart the app after these changes.

<ins>Stuff It’s Bad At (For Now)</ins> 😬

• Accurate spatial-reasoning and hence clicking buttons.
• Keeping track of itself in tabular contexts, like Excel and Google Sheets, for similar reasons as stated above.
• Navigating complex GUI-rich applications like Counter-Strike, Spotify, Garage Band, etc due to heavy reliance on cursor actions.

<ins>Future</ins> 🔮

(with better models trained on video walkthroughs like Youtube tutorials)

• "Create a couple of bass samples for me in Garage Band for my latest project."
• "Read this design document for a new feature, edit the code on Github, and submit it for review."
• "Find my friends' music taste from Spotify and create a party playlist for tonight's event."
• "Take the pictures from my Tahoe trip and make a White Lotus type montage in iMovie."

<ins>Notes</ins> 📝

• Cost: $0.05 - $0.20 per user request.<br> (This will be much lower in the near future once GPT-4V enables assistant/stateful mode)
• You can interrupt the app anytime by pressing the Stop button, or by dragging your cursor to any of the screen corners.
• Open Interface can only see your primary display when using multiple monitors. Therefore, if the cursor/focus is on a secondary screen, it might keep retrying the same actions as it is unable to see its progress (especially in MacOS with launching Spotlight).

<ins>System Diagram</ins> 🖼️

+----------------------------------------------------+
| App                                                |
|                                                    |
|    +-------+                                       |
|    |  GUI  |                                       |
|    +-------+                                       |
|        ^                                           |
|        |                                           |
|        v                                           |
|  +-----------+  (Screenshot + Goal)  +-----------+ |
|  |           | --------------------> |           | |
|  |    Core   |                       |    LLM    | |
|  |           | <-------------------- |  (GPT-4V) | |
|  +-----------+    (Instructions)     +-----------+ |
|        |                                           |
|        v                                           |
|  +-------------+                                   |
|  | Interpreter |                                   |
|  +-------------+                                   |
|        |                                           |
|        v                                           |
|  +-------------+                                   |
|  |   Executer  |                                   |
|  +-------------+                                   |
+----------------------------------------------------+

<ins>Star History</ins> ⭐️

<ins>Links</ins> 🔗

• Check out more of my projects at AmberSah.dev.
• Other demos and press kit can be found at MEDIA.md.