new version, able to support Inference API models

.env

@@ -1,6 +1,30 @@
-NEXT_PUBLIC_BASE_URL=https://jbilcke-hf-ai-comic-factory.hf.space
-# NEXT_PUBLIC_RENDERING_ENGINE_API=https://hysts-zeroscope-v2.hf.space
-RENDERING_ENGINE_API=https://jbilcke-hf-videochain-api.hf.space
-#VC_SECRET_ACCESS_TOKEN=<secret>
-#HF_API_TOKEN=<SECRET>
-#HF_INFERENCE_ENDPOINT_URL=<SECRET>
+# ------------- IMAGE API CONFIG --------------
+# Supported values:
+# - VIDEOCHAIN
+RENDERING_ENGINE="VIDEOCHAIN"
+
+VIDEOCHAIN_API_URL="http://localhost:7860"
+VIDEOCHAIN_API_TOKEN=
+
+# Not supported yet
+REPLICATE_TOKEN=
+
+# ------------- LLM API CONFIG ----------------
+# Supported values:
+# - INFERENCE_ENDPOINT
+# - INFERENCE_API
+LLM_ENGINE="INFERENCE_ENDPOINT"
+
+# Hugging Face token (if you choose to use a custom Inference Endpoint or an Inference API model)
+HF_API_TOKEN=
+
+# URL to a custom text-generation Inference Endpoint of your choice
+# -> You can leave it empty if you decide to use an Inference API Model instead
+HF_INFERENCE_ENDPOINT_URL=
+
+# You can also use a model from the Inference API (not a custom inference endpoint)
+# -> You can leave it empty if you decide to use an Inference Endpoint URL instead
+HF_INFERENCE_API_MODEL="codellama/CodeLlama-7b-hf"
+
+# Not supported yet
+OPENAI_TOKEN=

README.md

@@ -21,8 +21,8 @@ If you try to duplicate the project, you will see it requires some variables:
 
 - `HF_INFERENCE_ENDPOINT_URL`: This is the endpoint to call the LLM 
 - `HF_API_TOKEN`: The Hugging Face token used to call the inference endpoint (if you intent to use a LLM hosted on Hugging Face)
-- `RENDERING_ENGINE_API`: This is the API that generates images
-- `VC_SECRET_ACCESS_TOKEN`: Token used to call the rendering engine API (not used yet, but it's gonna be because [💸](https://en.wikipedia.org/wiki/No_such_thing_as_a_free_lunch))
+- `VIDEOCHAIN_API_URL`: This is the API that generates images
+- `VIDEOCHAIN_API_TOKEN`: Token used to call the rendering engine API (not used yet, but it's gonna be because [💸](https://en.wikipedia.org/wiki/No_such_thing_as_a_free_lunch))
 
 This is the architecture for the current production AI Comic Factory.
 
@@ -32,17 +32,37 @@ This is the architecture for the current production AI Comic Factory.
 
 Currently the AI Comic Factory uses [Llama-2 70b](https://huggingface.co/blog/llama2) through an [Inference Endpoint](https://huggingface.co/docs/inference-endpoints/index).
 
-You have two options:
+You have three options:
 
-### Option 1: Fork and modify the code to use another LLM
+### Option 1: Use an Inference API model
 
-If you fork the AI Comic Factory, you will be able to use another API and model, such as a locally-running Llama 7b.
+This is a new option added recently, where you can use one of the models from the Hugging Face Hub. By default we suggest to use CodeLlama.
 
-To run the LLM locally, you can use [TGI](https://github.com/huggingface/text-generation-inference) (Please read [this post](https://github.com/huggingface/text-generation-inference/issues/726) for more information about licensing).
+To activate it, create a `.env.local` configuration file:
 
-### Option 2: Fork and modify the code to use human content instead
+```bash
+HF_API_TOKEN="Your Hugging Face token"
+
+# codellama/CodeLlama-7b-hf" is used by default, but you can change this
+# note: You should use a model able to generate JSON responses
+HF_INFERENCE_API_MODEL="codellama/CodeLlama-7b-hf"
+```
+
+### Option 2: Use an Inference Endpoint URL
+
+If your would like to run the AI Comic Factory on a private LLM running on the Hugging Face Inference Endpoint service, create a `.env.local` configuration file:
+
+```bash
+HF_API_TOKEN="Your Hugging Face token"
+HF_INFERENCE_ENDPOINT_URL="path to your inference endpoint url"
+```
+
+To run this kind of LLM locally, you can use [TGI](https://github.com/huggingface/text-generation-inference) (Please read [this post](https://github.com/huggingface/text-generation-inference/issues/726) for more information about the licensing).
+
+### Option 3: Fork and modify the code to use a different LLM system
+
+Another option could be to disable the LLM completely and replace it with another LLM protocol and/or provider (eg. OpenAI, Replicate), or a human-generated story instead (by returning mock or static data).
 
-Another option could be to disable the LLM completely and replace it with a human-generated story instead (by returning mock or static data).
 
 ### Notes
 

public/favicon/index.html

@@ -0,0 +1,133 @@
+<!DOCTYPE html>
+<head>
+    <title>
+        Favicons
+    </title>
+    <meta charset="utf-8" />
+    
+    <!-- For old IEs -->
+    <link rel="shortcut icon" href="favicon.ico" />
+    
+    <!-- For new browsers multisize ico  -->
+    <link rel="icon" type="image/x-icon" sizes="16x16 32x32" href="favicon.ico">
+        
+        <!-- Chrome for Android -->
+        <link rel="icon" sizes="192x192" href="favicon-192.png">
+            
+            <!-- For iPhone 6+ downscaled for other devices -->
+            <link rel="apple-touch-icon" sizes="180x180" href="favicon-180-precomposed.png">
+                
+                <!-- For IE10 Metro -->
+                <meta name="msapplication-TileColor" content="#FFFFFF">
+                    <meta name="msapplication-TileImage" content="favicon-114-precomposed.png">
+                        
+                        <style>
+                            
+                            body {
+                                background-color: #f5f5f5;
+                                border: 0px;
+                                margin: 0px;
+                                padding: 0px;
+                                font-family: Consolas,Menlo,Monaco,Lucida Console,Liberation Mono,DejaVu Sans Mono,Bitstream Vera Sans Mono,Courier New,monospace,serif;
+                                color: black;
+                            }
+                        
+                        pre {
+                            margin: 0px;
+                            color: black;
+                            padding: 0px 5%;
+                        }
+                        
+                        code {
+                            
+                        }
+                        
+                        .container {
+                            background-color: white;
+                            max-width: 800px;
+                            width: 100%;
+                            margin: 0 auto;
+                            padding: 1% 0;
+                            height: 100%;
+                        }
+                        
+                        .comment {
+                            color: gray;
+                            padding: 0px;
+                            margin: 0px;
+                        }
+                        
+                        hr {
+                            width: 80%;
+                            padding: 0 5%;
+                            border-color: #f5f5f5;
+                            background-color: #D1D1D1;
+                        }
+                        
+                        p {
+                            padding: 1% 5%;
+                        }
+                        
+                            </style>
+                        
+                        </head>
+<body class="">
+    
+    <div class="container">
+        <p>
+        To use the favicons insert into your  head section some of these tags accordly to your needs.
+        </p>
+        <hr>
+        <pre>
+            <code>
+                <span class="comment">&lt;!-- For old IEs --&gt;</span>
+                &lt;link rel=&quot;shortcut icon&quot; href=&quot;favicon.ico&quot; /&gt;
+                
+                <span class="comment">&lt;!-- For new browsers - multisize ico  --&gt;</span>
+                &lt;link rel=&quot;icon&quot; type=&quot;image/x-icon&quot; sizes=&quot;16x16 32x32&quot; href=&quot;favicon.ico&quot;&gt;
+                
+                <span class="comment">&lt;!-- For iPad with high-resolution Retina display running iOS &ge; 7: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;152x152&quot; href=&quot;favicon-152-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For iPad with high-resolution Retina display running iOS &le; 6: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;144x144&quot; href=&quot;favicon-144-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For iPhone with high-resolution Retina display running iOS &ge; 7: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;120x120&quot; href=&quot;favicon-120-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For iPhone with high-resolution Retina display running iOS &le; 6: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;114x114&quot; href=&quot;favicon-114-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For iPhone 6+ --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;180x180&quot; href=&quot;favicon-180-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For first- and second-generation iPad: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;72x72&quot; href=&quot;favicon-72-precomposed.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For non-Retina iPhone, iPod Touch, and Android 2.1+ devices: --&gt;</span>
+                &lt;link rel=&quot;apple-touch-icon&quot; sizes=&quot;57x57&quot; href=&quot;favicon-57.png&quot;&gt;
+                
+                <span class="comment">&lt;!-- For Old Chrome --&gt;</span>
+                &lt;link rel=&quot;icon&quot; sizes=&quot;32x32&quot; href=&quot;favicon-32.png&quot; &gt;
+                
+                <span class="comment">&lt;!-- For IE10 Metro --&gt;</span>
+                &lt;meta name=&quot;msapplication-TileColor&quot; content=&quot;#FFFFFF&quot;&gt;
+                &lt;meta name=&quot;msapplication-TileImage&quot; content=&quot;favicon-144.png&quot;&gt;
+                &lt;meta name=&quot;theme-color&quot; content=&quot;#ffffff&quot;&gt;
+                
+                <span class="comment">&lt;!-- Chrome for Android --&gt;</span>
+                &lt;link rel=&quot;manifest&quot; href=&quot;manifest.json&quot;&gt;
+                &lt;link rel=&quot;icon&quot; sizes=&quot;192x192&quot; href=&quot;favicon-192.png&quot;&gt;
+                
+            </code>
+        </pre>
+        
+        <hr>
+        
+        <p>
+        For more informations about favicons consult <a href="https://github.com/audreyr/favicon-cheat-sheet">The Favicon Cheat Sheet</a> by Audrey Roy.
+        </p>
+        
+    </div>
+    
+</body>

public/favicon/manifest.json

@@ -0,0 +1,41 @@
+{
+	"name": "pollo",
+	"icons": [
+		{
+			"src": "\/favicon-36.png",
+			"sizes": "36x36",
+			"type": "image\/png",
+			"density": 0.75
+		},
+		{
+			"src": "\/favicon-48.png",
+			"sizes": "48x48",
+			"type": "image\/png",
+			"density": 1
+		},
+		{
+			"src": "\/favicon-72.png",
+			"sizes": "72x72",
+			"type": "image\/png",
+			"density": 1.5
+		},
+		{
+			"src": "\/favicon-96.png",
+			"sizes": "96x96",
+			"type": "image\/png",
+			"density": 2
+		},
+		{
+			"src": "\/favicon-144.png",
+			"sizes": "144x144",
+			"type": "image\/png",
+			"density": 3
+		},
+		{
+			"src": "\/favicon-192.png",
+			"sizes": "192x192",
+			"type": "image\/png",
+			"density": 4
+		}
+	]
+}

src/app/engine/caption.ts

@@ -2,7 +2,7 @@
 
 import { ImageAnalysisRequest, ImageAnalysisResponse } from "@/types"
 
-const apiUrl = `${process.env.RENDERING_ENGINE_API || ""}`
+const apiUrl = `${process.env.VIDEOCHAIN_API_URL || ""}`
 
 export async function see({
   prompt,
@@ -33,7 +33,7 @@ export async function see({
       headers: {
         Accept: "application/json",
         "Content-Type": "application/json",
-        // Authorization: `Bearer ${process.env.VC_SECRET_ACCESS_TOKEN}`,
+        // Authorization: `Bearer ${process.env.VIDEOCHAIN_API_TOKEN}`,
       },
       body: JSON.stringify(request),
       cache: 'no-store',

src/app/engine/censorship.ts

@@ -2,3 +2,5 @@
 
 // unfortunately due to abuse by some users, I have to add this NSFW filter
 const secretSalt = `${process.env.SECRET_CENSORSHIP_KEY || ""}`
+
+// TODO the censorship is not implement yet actually

src/app/engine/forbidden.ts

@@ -2,5 +2,5 @@
 // the NSFW has to contain bad words, but doing so might get the code flagged
 // or attract unwanted attention, so we hash them
 export const forbidden = [
-
+  // TODO implement this
 ]

src/app/engine/render.ts

@@ -1,12 +1,12 @@
 "use server"
 
-import { RenderRequest, RenderedScene } from "@/types"
+import { RenderRequest, RenderedScene, RenderingEngine } from "@/types"
+
+const renderingEngine = `${process.env.RENDERING_ENGINE || ""}` as RenderingEngine
 
 // note: there is no / at the end in the variable
 // so we have to add it ourselves if needed
-const apiUrl = process.env.RENDERING_ENGINE_API
-
-const cacheDurationInSec = 30 * 60 // 30 minutes
+const apiUrl = process.env.VIDEOCHAIN_API_URL
 
 export async function newRender({
   prompt,
@@ -44,7 +44,7 @@ export async function newRender({
       headers: {
         Accept: "application/json",
         "Content-Type": "application/json",
-        Authorization: `Bearer ${process.env.VC_SECRET_ACCESS_TOKEN}`,
+        Authorization: `Bearer ${process.env.VIDEOCHAIN_API_TOKEN}`,
       },
       body: JSON.stringify({
         prompt,
@@ -114,7 +114,7 @@ export async function getRender(renderId: string) {
       headers: {
         Accept: "application/json",
         "Content-Type": "application/json",
-        Authorization: `Bearer ${process.env.VC_SECRET_ACCESS_TOKEN}`,
+        Authorization: `Bearer ${process.env.VIDEOCHAIN_API_TOKEN}`,
       },
       cache: 'no-store',
     // we can also use this (see https://vercel.com/blog/vercel-cache-api-nextjs-cache)
@@ -166,7 +166,7 @@ export async function upscaleImage(image: string): Promise<{
       headers: {
         Accept: "application/json",
         "Content-Type": "application/json",
-        Authorization: `Bearer ${process.env.VC_SECRET_ACCESS_TOKEN}`,
+        Authorization: `Bearer ${process.env.VIDEOCHAIN_API_TOKEN}`,
       },
       cache: 'no-store',
       body: JSON.stringify({ image, factor: 3 })

src/app/interface/top-menu/index.tsx

@@ -213,6 +213,7 @@ export function TopMenu() {
           className={cn(
             `rounded-l-none cursor-pointer`,
             `transition-all duration-200 ease-in-out`,
+            `bg-[rgb(59,134,247)] hover:bg-[rgb(69,144,255)] disabled:bg-[rgb(59,134,247)]`
             )}
           onClick={() => {
             handleSubmit()

src/app/queries/predict.ts

@@ -1,22 +1,60 @@
 "use server"
 
-import { HfInference } from "@huggingface/inference"
+import { LLMEngine } from "@/types"
+import { HfInference, HfInferenceEndpoint } from "@huggingface/inference"
 
-const hfi = new HfInference(process.env.HF_API_TOKEN)
-const hf = hfi.endpoint(`${process.env.HF_INFERENCE_ENDPOINT_URL || ""}`)
+const hf = new HfInference(process.env.HF_API_TOKEN)
+
+
+// note: we always try "inference endpoint" first
+const llmEngine = `${process.env.LLM_ENGINE || ""}` as LLMEngine
+const inferenceEndpoint = `${process.env.HF_INFERENCE_ENDPOINT_URL || ""}`
+const inferenceModel = `${process.env.HF_INFERENCE_API_MODEL || ""}`
+
+let hfie: HfInferenceEndpoint
+
+switch (llmEngine) {
+  case "INFERENCE_ENDPOINT":
+    if (inferenceEndpoint) {
+      console.log("Using a custom HF Inference Endpoint")
+      hfie = hf.endpoint(inferenceEndpoint)
+    } else {
+      const error = "No Inference Endpoint URL defined"
+      console.error(error)
+      throw new Error(error)
+    }
+    break;
+  
+  case "INFERENCE_API":
+    if (inferenceModel) {
+      console.log("Using an HF Inference API Model")
+    } else {
+      const error = "No Inference API model defined"
+      console.error(error)
+      throw new Error(error)
+    }
+    break;
+  
+  default:
+    const error = "No Inference Endpoint URL or Inference API Model defined"
+    console.error(error)
+    throw new Error(error)
+}
 
 export async function predict(inputs: string) {
 
   console.log(`predict: `, inputs)
   
+  const api = llmEngine ==="INFERENCE_ENDPOINT" ? hfie : hf
+
   let instructions = ""
   try {
-    for await (const output of hf.textGenerationStream({
+    for await (const output of api.textGenerationStream({
+      model: llmEngine ==="INFERENCE_ENDPOINT" ? undefined : (inferenceModel || undefined),
       inputs,
       parameters: {
         do_sample: true,
-
-        // hard limit for max_new_tokens is 1512
+        // we don't require a lot of token for our task
         max_new_tokens: 330, // 1150,
         return_full_text: false,
       }

src/components/ui/input.tsx

@@ -11,7 +11,7 @@ const Input = React.forwardRef<HTMLInputElement, InputProps>(
       <input
         type={type}
         className={cn(
-          "flex h-10 w-full rounded-md border border-stone-200 bg-white px-3 py-2 text-sm ring-offset-white file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-stone-500 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-stone-400 focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50 dark:border-stone-800 dark:bg-stone-950 dark:ring-offset-stone-950 dark:placeholder:text-stone-400 dark:focus-visible:ring-stone-800",
+          "flex h-10 w-full rounded-md border border-stone-200 bg-white px-3 py-2 text-sm ring-offset-white file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-stone-500 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-blue-[rgb(59,134,247)] focus-visible:ring-offset-0 disabled:cursor-not-allowed disabled:opacity-50 dark:border-stone-800 dark:bg-stone-950 dark:ring-offset-stone-950 dark:placeholder:text-stone-400 dark:focus-visible:ring-stone-800",
           className
         )}
         ref={ref}

src/types.ts

@@ -80,3 +80,14 @@ export interface ImageAnalysisResponse {
 }
 
 export type LLMResponse = Array<{panel: number; instructions: string; caption: string }>
+
+export type LLMEngine =
+  | "INFERENCE_API"
+  | "INFERENCE_ENDPOINT"
+  | "OPENAI"
+  | "REPLICATE"
+
+export type RenderingEngine =
+  | "VIDEOCHAIN"
+  | "OPENAI"
+  | "REPLICATE"