Automating ComfyUI with Local LLM - From Prompt Generation to Image Review

Introduction

Have you ever felt that manually entering prompts and generating images in ComfyUI is cumbersome? This post introduces a workflow for automating ComfyUI using local LLM, from generating prompts to reviewing images.

Using ComfyUI’s Basic LLM Template

ComfyUI provides a basic LLM template by default. This template allows you to generate images by entering natural language prompts. For example, entering “hello” will generate a response from the LLM.

Limitations of Lightweight Models

ComfyUI’s basic LLM template is a lightweight model, which has limitations when handling complex prompts. To overcome this, we introduce a method for using a more powerful local LLM.

Ollama and LM Studio

To run local LLMs, we can use tools like Ollama and LM Studio. Ollama is a program that makes it easy to run local models, while LM Studio supports a wider range of models.

Connecting Local LLM Models

Using LM Studio, we can connect local LLM models to ComfyUI. This enables us to handle more complex prompts and generate images.

Deno LLM Loader

To use local LLM models in ComfyUI, we can use a node called Deno LLM Loader. This node allows us to connect to LM Studio and run models.

Context Window and VRAM Usage

When using local LLM models, the size of the context window affects VRAM usage. We can optimize VRAM usage by adjusting Flash Attention and K/V Cache settings.

Local LLM Reviewer

To review generated images, we can use a node called Local LLM Reviewer. This node uses an LLM model to evaluate images and make pass/fail judgments.

Building an Automated Workflow

By using local LLM models and the Local LLM Reviewer, we can build an automated workflow. This enables us to automatically generate prompts, review images, and save them.

Conclusion

We introduced a workflow for automating ComfyUI using local LLM, from generating prompts to reviewing images. By automating the image generation process, we can improve productivity.