gustavokuklinski/aeon-360M-GGUF

AEON

AEON is portable, private, and capable of operating fully offline. It democratizes access to powerful, dynamic AI capabilities for a wider audience, regardless of their hardware.

The finetuned model was build to be like a "friend" for RAG personal files and work with insights.

Docs

• Page aeon.ai
• Github Project: AEON.ai
• Github LLM Finetune Scripts: AEON.llm

Using Aeon

AEON uses Python with virtual environment and git lfs installed.

/$ git lfs install

# With plugins
/$ git clone --recurse-submodules https://github.com/gustavokuklinski/aeon.ai.git

# Without plugins
/$ git clone https://github.com/gustavokuklinski/aeon.ai.git

# Create .venv
/$ python -m venv .venv

# Start virtual env
/$ source .venv/bin/activate

# Run check and install dependencies
/$ python3 scripts/install.py 

# Start AEON
/$ python3 aeon.py

Using Docker

docker build -t aeon .

docker run -it --rm -p 7860:7860 -v "$(pwd):/app" aeon

Finetune

Tested on

OS	CPU	GPU	RAM
Ubuntu 24.04.2 LTS	Intel i7-10510U	Intel CometLake-U GT2	16GB
Windows 11 Home Edition	Intel i7-10510U	Intel CometLake-U GT2	8GB

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo gustavokuklinski/aeon-GGUF --hf-file aeon-360M.Q8_0.gguf -p "What is a virtual species?"

Server:

llama-server --hf-repo gustavokuklinski/aeon-GGUF --hf-file aeon-360M.Q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo gustavokuklinski/aeon-GGUF --hf-file aeon-360M.Q8_0.gguf -p "What is a virtual species?"

or

./llama-server --hf-repo gustavokuklinski/aeon-GGUF --hf-file aeon-360M.Q8_0.gguf -c 2048