Add model card for GUI-AIMA-3B

README.md

@@ -0,0 +1,100 @@
+---
+license: cc-by-nc-4.0
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+# GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
+
+This repository hosts the **GUI-AIMA-3B** model, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. The model is presented in the paper [GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding](https://huggingface.co/papers/2511.00810).
+
+GUI-AIMA addresses the challenge of mapping natural-language instructions to actionable screen regions in graphical user interfaces (GUIs). It aligns the intrinsic multimodal attention of Multimodal Large Language Models (MLLMs) with patch-wise grounding signals. GUI-AIMA-3B was trained with only 85k screenshots, demonstrating exceptional data efficiency, and achieves state-of-the-art performance among 3B models, attaining an average accuracy of 58.6% on ScreenSpot-Pro and 62.2% on OSWorld-G. It also supports a plug-and-play zoom-in stage for higher precision on high-resolution screenshots without further fine-tuning.
+
+*   **Paper:** [GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding](https://huggingface.co/papers/2511.00810)
+*   **Project Page:** [https://github.com/sjz5202/GUI-AIMA](https://github.com/sjz5202/GUI-AIMA)
+*   **Code Repository:** [https://github.com/sjz5202/GUI-AIMA](https://github.com/sjz5202/GUI-AIMA)
+
+<div align="center">
+<img src="https://github.com/sjz5202/GUI-AIMA/raw/main/assets/images/comparison.png" width="85%">
+</div>
+Figure 1. **GUI-AIMA** utilize the inherent attention of MLLMs for patch-wise GUI grounding. It simplifies the vanilla attention grounding requiring proper aggregation between all query tokens' grounding vectors by adding a learnable ANCHOR token as the context anchor of query. The multi-head aggregation on attention vectors between ANCHOR and visual tokens is adequate for grounding.
+
+<div align="center">
+<img src="https://github.com/sjz5202/GUI-AIMA/raw/main/assets/images/main_fig.png" width="85%">
+</div>
+Figure 2. **GUI-AIMA** proposes an effective multi-head weighting approach by measuring the uniformity between global query-visual pattern and head-wise query-visual pattern.
+
+## Main Results
+
+There are two variants of GUI-AIMA: [GUI-AIMA-3B](https://huggingface.co/smz8599/GUI-AIMA-3B) and [GUI-AIMA-3B(soft)](https://huggingface.co/smz8599/GUI-AIMA-3B-kl) with slight differences of multihead weighting.
+
+1-step inference of GUI-AIMA achieves **47.1%** and **56.9%** on ScreenSpot-pro and OSWorld-G. With 2-step zoom-in inference, it can achieve **58.6%** and **62.2%** on ScreenSpot-pro and OSWorld-G.
+
+We trained GUI-AIMA for one-step center points predictions. However, **GUI-AIMA can be inferenced in the 2-step fashion without further fine-tuning**: (step 1) 1st inference to determine rough grounding areas; (step 2) crop and zoom-in the rough grounding areas for 2nd preciser grounding inference. The 2-step inference is very helpful for GUI grounding on high-resolution screenshots, such as samples in ScreenSpot-pro and OSWorld-G.
+
+<div align="left">
+<img src="https://github.com/sjz5202/GUI-AIMA/raw/main/assets/images/ss_pro.png" width="100%">
+</div>
+
+<div align="left">
+<img src="https://github.com/sjz5202/GUI-AIMA/raw/main/assets/images/osworld-g.png" width="80%">
+</div>
+
+<div align="left">
+<img src="https://github.com/sjz5202/GUI-AIMA/raw/main/assets/images/ss_v2.png" width="85%">
+</div>
+
+## Sample Usage
+
+You can use the model with the `transformers` library. For detailed installation and full examples, refer to the [GitHub repository](https://github.com/sjz5202/GUI-AIMA). A single sample inference example is available in `eval/example_inference.py`.
+
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+
+# If you want to load a model using multiple GPUs, please refer to the `Multiple GPUs` section.
+path = 'smz8599/GUI-AIMA-3B' # or 'smz8599/GUI-AIMA-3B-kl'
+model = AutoModel.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
+
+# set the max number of tiles in `max_num`
+# Note: original code used load_image, which is not provided. Replaced with simple image loading for demonstration.
+# Please refer to the full GitHub repository for `load_image` implementation and image preprocessing.
+image_path = "./examples/images/screenshot.png" # Replace with your image path
+image = Image.open(image_path).convert('RGB')
+# A placeholder for pixel_values, as dynamic_preprocess is complex.
+# Users should refer to the original GitHub for proper image preprocessing.
+pixel_values = torch.randn(1, 3, 448, 448).to(torch.bfloat16).cuda() # Placeholder
+
+generation_config = dict(max_new_tokens=1024, do_sample=True)
+
+question = "In the screenshot of this web page, please give me the coordinates of the element I want to click on according to my instructions(with point).\
+\\\"'Champions League' link\\\""
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}\
+Assistant: {response}')
+```
+
+## Citation
+
+If you find this work helpful, please cite the paper:
+
+```bibtex
+@misc{zhou2025guiaimaaligningintrinsicmultimodal,
+      title={GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding}, 
+      author={Shijie Zhou and Viet Dac Lai and Hao Tan and Jihyung Kil and Wanrong Zhu and Changyou Chen and Ruiyi Zhang},
+      year={2025},
+      eprint={2511.00810},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2511.00810}, 
+}
+```