Upload folder using huggingface_hub

README.md

@@ -0,0 +1,178 @@
+# nemotron-parse Overview
+
+nemotron-parse is a general purpose text-extraction model, specifically designed to handle documents. Given an image, nemotron-parse is able to extract formatted-text, with bounding-boxes and the corresponding semantic class. This has downstream benefits for several tasks such as increasing the availability of training-data for Large Language Models (LLMs), improving the accuracy of retriever systems, and enhancing document understanding pipelines.
+
+This model is ready for commercial use.
+
+
+
+
+## License 
+GOVERNING TERMS: The NIM container is governed by the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [Product-Specific Terms for NVIDIA AI Products](https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/). Use of this model is governed by the [NVIDIA Community Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/). Use of the tokenizer included in this model is governed by the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/).
+
+
+## Deployment Geography:
+Global
+
+## Use Case:
+nemotron-parse will be capable of comprehensive text understanding and document structure understanding. It will be used in retriever and curator solutions. Its text extraction datasets and capabilities will help with LLM and VLM training, as well as improve run-time inference accuracy of VLMs.
+The nemotron-parse model will perform text extraction from PDF and PPT documents. The nemotron-parse can classify the objects (title, section, caption, index, footnote, lists, tables, bibliography, image) in a given document, and provide bounding boxes with coordinates. 
+
+
+## Release Date:
+November 17, 2025 
+
+
+## References 
+* https://huggingface.co/docs/transformers/en/model_doc/mbart
+
+
+## Model Architecture 
+
+### Architecture Type : 
+Transformer-based vision-encoder-decoder model
+
+### Network Architecture 
+* Vision Encoder: ViT-H model (https://huggingface.co/nvidia/C-RADIO)<br>
+* Adapter Layer: 1D convolutions & norms to compress dimensionality and sequence length of the latent space (13184 tokens to 3201 tokens)<br>
+* Decoder: mBart [1] 10 blocks<br>
+* Tokenizer: Use of the tokenizer included in this model is governed by the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/)<br>
+* Number of Parameters: < 1B<br>
+
+
+## Computational Load (For NVIDIA Models Only) 
+**Cumulative Compute:**  2.2e+22 <br>
+**Estimated Energy and Emissions for Model Training:** 
+Energy Consumption: 7,827.46 kWh <br>
+Carbon Emissions: 3.21 tCO2e <br>
+
+### Input 
+* Input Type: Image, Text<br>
+* Input Type(s): Red, Green, Blue (RGB) + Prompt (String)
+* Input Parameters: 2D, 1D
+- Other Properties Related to Input:
+  - Max Input Resolution (Width, Height): 1648, 2048
+  - Min Input Resolution (Width, Height): 1024, 1280
+- Channel Count: 3
+
+### Output 
+* Output Type: Text<br>
+* Output Format: String
+* Output Parameters: 1D
+- Other Properties Related to Output:
+  - nemotron-parse output format is a string which encodes text content (formatted or not) as well as bounding boxes and class attributes.<br>
+
+  Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.<br>
+
+## Software Integration:
+
+Runtime Engine(s): TensorRT-LLM
+
+Supported Hardware Microarchitecture Compatibility: <br>
+NVIDIA Hopper/NVIDIA Ampere/NVIDIA Turing<br>
+
+Supported Operating System(s): Linux<br>
+
+The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.<br>
+
+## Model Version:
+
+V1.1
+
+## Quick Start
+
+### Install dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### Usage example
+
+```python
+import torch
+from PIL import Image, ImageDraw
+from transformers import AutoModel, AutoProcessor, AutoTokenizer, AutoConfig, AutoImageProcessor, GenerationConfig
+from postprocessing import extract_classes_bboxes, transform_bbox_to_original, postprocess_text
+
+# Load model and processor
+model_path = "nvidia/NVIDIA-Nemotron-Parse-v1.1"  # Or use a local path
+device = "cuda:0"
+
+model = AutoModel.from_pretrained(
+    model_path,
+    trust_remote_code=True,
+    torch_dtype=torch.bfloat16
+).to(device).eval()
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+
+# Load image
+image = Image.open("path/to/your/image.jpg")
+task_prompt = "</s><s><predict_bbox><predict_classes><output_markdown>"
+
+# Process image
+inputs = processor(images=[image], text=task_prompt, return_tensors="pt").to(device)
+prompt_ids = processor.tokenizer.encode(task_prompt, return_tensors="pt", add_special_tokens=False).cuda()
+
+
+generation_config = GenerationConfig.from_pretrained(model_path, trust_remote_code=True)
+# Generate text
+outputs = model.generate(**inputs,  generation_config=generation_config)
+
+# Decode the generated text
+generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
+classes, bboxes, texts = extract_classes_bboxes(generated_text)
+bboxes = [transform_bbox_to_original(bbox, image.width, image.height) for bbox in bboxes]
+
+# Specify output formats for postprocessing
+table_format = 'latex' # latex | HTML | markdown
+text_format = 'markdown' # markdown | plain
+blank_text_in_figures = False # remove text inside 'Picture' class
+texts = [postprocess_text(text, cls = cls, table_format=table_format, text_format=text_format, blank_text_in_figures=blank_text_in_figures) for text, cls in zip(texts, classes)]
+
+for cl, bb, txt in zip(classes, bboxes, texts):
+    print(cl, ': ', txt)
+
+draw = ImageDraw.Draw(image)
+for bbox in bboxes:
+  draw.rectangle((bbox[0], bbox[1], bbox[2], bbox[3]), outline="red")
+```
+
+## Training, Testing, and Evaluation Datasets:
+
+
+### Training Dataset 
+
+nemotron-parse is first pre-trained on our internal datasets: human, synthetic and automated.
+Data Modality:
+*Text
+*Image<br>
+Data Collection Method by Dataset: Hybrid: Human, Synthetic, Automated
+Labeling Method by Dataset: Hybrid: Human, Synthetic, Automated
+
+### Testing and Evaluation Dataset:
+
+nemotron-parse is evaluated on multiple datasets for robustness, including public and internal dataset.
+Data Collection Method by Dataset: Hybrid: Human, Synthetic, Automated
+Labeling Method by Dataset: Hybrid: Human, Synthetic, Automated
+
+
+## Inference 
+
+Runtime Engine(s): TensorRT-LLM
+
+Test Hardware: NVIDIA H100# Synchronization
+
+## Ethical Considerations
+
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+
+Please report security vulnerabilities or NVIDIA AI Concerns here.
+
+**You are responsible for ensuring that your use of NVIDIA AI Models complies with all applicable laws.**
+
+
+## Enterprise Support
+Get access to knowledge base articles and support cases or [submit a ticket](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/support/).
+

bias.md

@@ -0,0 +1,6 @@
+## Bias
+| Field | Response |
+| :---- | :---- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None |
+| Measures taken to mitigate against unwanted bias: | Not applicable |
+| Bias Metric (If Measured): | None |

explainability.md

@@ -0,0 +1,15 @@
+## Explainability
+
+| Field | Response |
+| :---- | :---- |
+| Intended Task/Domain: | image to text |
+| Model Type: | Transformer-based vision-encoder-decoder model|
+| Intended Users: | Generative AI creators working with conversational AI models and image content. |
+| Output: | Text |
+| Describe how the model works: | Generates text by predicting the next word or token based on the context provided in the input sequence using multiple self-attention layers. |
+| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable |
+| Technical Limitations & Mitigation: | The model demonstrates weakness to alignment-breaking attacks. Users are advised to deploy language model guardrails alongside this model to prevent potentially harmful outputs. The Model may generate answers that are inaccurate, omit key information, or include irrelevant or redundant text. |
+| Verified to have met prescribed NVIDIA quality standards: | Yes |
+| Performance Metrics: | Accuracy, Throughput, and User-side throughput |
+| Potential Known Risks: | The model was optimized explicitly for instruction following and as such is more susceptible to prompt injection and jailbreaking in various forms as a result of its instruction tuning. This means that the model should be paired with additional rails or system filtering to limit exposure to instructions from malicious sources  \-- either directly or indirectly by retrieval (e.g. via visiting a website)  \-- as they may yield outputs that can lead to harmful, system-level outcomes up to and including remote code execution in agentic systems when effective security controls including guardrails are not in place. The model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. |
+| Licensing: | GOVERNING TERMS: The NIM container is governed by the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [Product-Specific Terms for NVIDIA AI Products](https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/). Use of this model is governed by the [NVIDIA Community Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/). Use of the tokenizer included in this model is governed by the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/) |

latex2html.py

@@ -0,0 +1,448 @@
+import re
+from bs4 import BeautifulSoup
+
+def skip_whitespace(text, i):
+    """Advance index i past any whitespace."""
+    while i < len(text) and text[i].isspace():
+        i += 1
+    return i
+
+def parse_braced_argument(text, i):
+    """
+    Given text and an index i that should point at an opening '{',
+    return a tuple (argument_content, new_index) where argument_content is the full
+    string inside the balanced braces and new_index is the position just after the matching '}'.
+    """
+    if i >= len(text) or text[i] != '{':
+        raise ValueError("Expected '{' at position {}".format(i))
+    i += 1  # skip the opening brace
+    start = i
+    level = 1
+    while i < len(text) and level > 0:
+        if text[i] == '{':
+            level += 1
+        elif text[i] == '}':
+            level -= 1
+        i += 1
+    if level != 0:
+        raise ValueError("Unbalanced braces starting at position {}".format(start-1))
+    # The argument content is from start to i-1 (excluding the closing brace)
+    return text[start:i-1], i
+
+def parse_command(text, i):
+    """
+    Parse a \multirow or \multicolumn command starting at index i.
+    This function assumes the command has exactly three braced arguments.
+
+    It processes each argument recursively. For the third argument, after recursive processing,
+    it replaces any unescaped & with \&.
+
+    Returns a tuple (command_text, new_index) where command_text is the reconstructed command.
+    """
+    # Determine which command we have.
+    if text.startswith(r"\multirow", i):
+        command_name = r"\multirow"
+        i += len(r"\multirow")
+    elif text.startswith(r"\multicolumn", i):
+        command_name = r"\multicolumn"
+        i += len(r"\multicolumn")
+    else:
+        raise ValueError("Expected \\multirow or \\multicolumn at position {}".format(i))
+
+    # Skip whitespace between the command name and the first argument.
+    i = skip_whitespace(text, i)
+    args = []
+    # Expect exactly three arguments
+    for arg_index in range(3):
+        if i >= len(text) or text[i] != '{':
+            raise ValueError("Expected '{' for argument {} at position {}".format(arg_index+1, i))
+        arg_content, i = parse_braced_argument(text, i)
+        # Process the content recursively to catch nested commands
+        processed_arg = clean_multi_cells(arg_content)
+        if arg_index == 2:
+            # For the cell text (third argument), replace any unescaped &
+            processed_arg = re.sub(r'(?<!\\)&', r'\\&', processed_arg)
+        args.append(processed_arg)
+        # Only skip whitespace between arguments, not after the last one.
+        if arg_index < 2:
+            i = skip_whitespace(text, i)
+    # Reconstruct the full command with its three arguments
+    command_text = f"{command_name}{{{args[0]}}}{{{args[1]}}}{{{args[2]}}}"
+    return command_text, i
+
+def clean_multi_cells(text):
+    """
+    Process an arbitrary LaTeX text string and look for occurrences of \multirow or \multicolumn commands.
+    When found, the command is parsed (handling nested braces and nested commands) and its third argument is fixed.
+
+    Returns the processed text.
+    """
+    result = []
+    i = 0
+    while i < len(text):
+        # Find next occurrence of either command.
+        idx_multi = text.find(r"\multirow", i)
+        idx_multiC = text.find(r"\multicolumn", i)
+
+        # Determine the next index among the two (if any)
+        if idx_multi == -1 and idx_multiC == -1:
+            result.append(text[i:])
+            break
+        if idx_multi == -1:
+            next_idx = idx_multiC
+        elif idx_multiC == -1:
+            next_idx = idx_multi
+        else:
+            next_idx = min(idx_multi, idx_multiC)
+
+        # Append text before the command (preserving any whitespace)
+        result.append(text[i:next_idx])
+        # Process the command starting at next_idx
+        command_text, new_index = parse_command(text, next_idx)
+        result.append(command_text)
+        i = new_index
+    return ''.join(result)
+
+def parse_brace(s, pos):
+    """
+    Given a string s and an index pos pointing to an opening '{',
+    returns a tuple (content, new_pos) where content is the string
+    between the matching braces (handling nested braces) and new_pos is
+    the index just after the closing '}'.
+    """
+    if pos >= len(s) or s[pos] != '{':
+        raise ValueError("Expected '{' at position %d" % pos)
+    pos += 1  # skip the opening brace
+    content = ""
+    depth = 1
+    while pos < len(s) and depth:
+        char = s[pos]
+        if char == '{':
+            depth += 1
+            content += char
+        elif char == '}':
+            depth -= 1
+            if depth:
+                content += char
+        else:
+            content += char
+        pos += 1
+    if depth != 0:
+        raise ValueError("Unmatched '{' in string.")
+    return content, pos
+
+def parse_command_merge(s, pos):
+    """
+    Parse a multirow or multicolumn command starting at s[pos]. If the content
+    of the command contains a nested command, then recursively parse the inner
+    command and merge its parameters with the outer ones. The merging is done
+    so that the outer multirow’s parameters (e.g. rowspan and width) are kept
+    while the inner command’s parameters (e.g. colspan, alignment) and its innermost
+    content are returned.
+
+    Returns a tuple (merged_dict, new_pos) where merged_dict is a dictionary
+    containing the combined parameters and new_pos is the updated index after
+    parsing the command.
+    """
+    if s.startswith(r"\multirow", pos):
+        newpos = pos + len(r"\multirow")
+        # Parse the three required arguments for multirow: rowspan, width, and content.
+        rowspan, newpos = parse_brace(s, newpos)
+        width, newpos = parse_brace(s, newpos)
+        content, newpos = parse_brace(s, newpos)
+        # Look for a nested command (either \multirow or \multicolumn) in the content.
+        index_mr = content.find(r"\multirow")
+        index_mc = content.find(r"\multicolumn")
+        if index_mr == -1 and index_mc == -1:
+            # No nested command found; return this command’s details.
+            return {"rowspan": rowspan.strip(), "width": width.strip(), "content": content.strip()}, newpos
+        else:
+            # At least one nested command is present. Pick the first occurrence.
+            indices = [i for i in (index_mr, index_mc) if i != -1]
+            first_index = min(indices)
+            # Parse the inner (nested) command from within the content.
+            inner, _ = parse_command_merge(content, first_index)
+            # Merge: keep the outer multirow’s parameters and add the inner ones.
+            merged = {"rowspan": rowspan.strip(), "width": width.strip()}
+            merged.update(inner)
+            return merged, newpos
+
+    elif s.startswith(r"\multicolumn", pos):
+        newpos = pos + len(r"\multicolumn")
+        # Parse the three arguments for multicolumn: colspan, alignment, and content.
+        colspan, newpos = parse_brace(s, newpos)
+        alignment, newpos = parse_brace(s, newpos)
+        content, newpos = parse_brace(s, newpos)
+        # Look for a nested command in the content.
+        index_mr = content.find(r"\multirow")
+        index_mc = content.find(r"\multicolumn")
+        if index_mr == -1 and index_mc == -1:
+            return {"colspan": colspan.strip(), "alignment": alignment.strip(), "content": content.strip()}, newpos
+        else:
+            indices = [i for i in (index_mr, index_mc) if i != -1]
+            first_index = min(indices)
+            inner, _ = parse_command_merge(content, first_index)
+            merged = {"colspan": colspan.strip(), "alignment": alignment.strip()}
+            merged.update(inner)
+            return merged, newpos
+
+    # Not a recognized command starting at pos.
+    return None, pos
+
+def extract_merged_commands(s):
+    """
+    Scan through the LaTeX string s and extract merged multirow/multicolumn commands.
+    For each command found, if there is nesting the parser merges the outer and inner
+    parameters so that the final result includes both the rowspan (or width) and the colspan
+    (or alignment) along with the innermost content.
+
+    Returns a list of dictionaries.
+    """
+    pos = 0
+    results = []
+    while pos < len(s):
+        if s[pos] == '\\':
+            res, newpos = parse_command_merge(s, pos)
+            if res is not None:
+                results.append(res)
+                pos = newpos
+                continue
+        pos += 1
+    return results
+
+def remove_tags(html, tags_to_remove):
+    soup = BeautifulSoup(html, "html.parser")
+    # Loop through the tags to remove
+    for tag_name in tags_to_remove:
+        for tag in soup.find_all(tag_name):
+            # Move the children of the tag to the parent tag
+            tag.unwrap()  # This removes the tag but keeps its contents
+    # Return the modified HTML as a string
+    return str(soup)
+
+def convert_th_to_td(html):
+    """Replace all th tags with td tags
+    """
+    soup = BeautifulSoup(html)
+    for th_tag in soup.find_all('th'):
+        th_tag.name = 'td'
+    return str(soup)
+
+def replace_italic(text):
+    pattern = re.compile(r'(?<!\\)_(.*?)(?<!\\)_')
+
+    def italic_replacer(match):
+        # Get the text inside the underscores.
+        content = match.group(1)
+        # Remove the escape (backslash) from any escaped underscores inside.
+        content = content.replace(r'\_', '_')
+        return f"<i>{content}</i>"
+
+    # Replace all occurrences of the pattern using the replacer function.
+    return pattern.sub(italic_replacer, text)
+
+
+def replace_bold(text):
+    pattern = re.compile(r'(?<!\\)\*\*(.*?)(?<!\\)\*\*')
+
+    def bold_replacer(match):
+        content = match.group(1)
+        # Unescape any escaped asterisks within the captured text.
+        content = content.replace(r'\*', '*')
+        return f"<b>{content}</b>"
+
+    return pattern.sub(bold_replacer, text)
+
+def latex_table_to_html(latex_str, add_head_body = False):
+    # Pattern to match the entire tabular environment
+    table_pattern = r'\\begin{tabular}{([^}]*)}\s*(.*?)\\end{tabular}'
+
+    def process_cell(cell):
+        # Clean up cell content
+        cell = cell.strip()
+
+        out = extract_merged_commands(cell)
+        if len(out) > 0:
+            cell = process_cell(out[0]["content"])["content"]
+            rowspan = int(out[0].get("rowspan", "1"))
+            colspan = int(out[0].get("colspan", "1"))
+            return {
+                "content": cell,
+                "colspan": colspan,
+                "rowspan": rowspan
+            }
+
+        # Replace latex and markdown formatting with HTML tags
+        cell = re.sub(r'\$([^$]*)\$', r'\1', cell)  # Remove math mode
+        cell = re.sub(r'\\textbf{([^}]*)}', r'<b>\1</b>', cell)  # Convert latex bold
+        cell = re.sub(r'\\textit{([^}]*)}', r'<i>\1</i>', cell)  # Convert latex italic
+        cell = replace_italic(cell)
+        cell = replace_bold(cell)
+        cell = cell.replace("\\$", "$").replace("\\%", "%").replace("\\newline", "\n").replace("\\textless", "<").replace("\\textgreater", ">").replace("\\*", "*").replace("\\_", "_").replace("\\backslash", "\\")
+
+        # Replace \& with & in the cell text
+        cell = cell.replace(r'\&', '&')
+        cell = cell.replace('<tbc>', '')
+        # Preserve newlines for downstream row-splitting; clean other tokens
+        cell = cell.replace('\\unknown', '').replace('\\<|unk|\\>', '').replace('<u>', '<underline>').replace('</u>', '</underline>')
+        return {
+            'content': cell,
+            'colspan': 1,
+            'rowspan': 1
+        }
+
+    def split_row(input_string):
+        # Use a regular expression to split on '&' that is not preceded by a backslash
+        return re.split(r'(?<!\\)&', input_string)
+
+    def convert_table(match):
+        # Extract table content
+        format_spec, content = match.groups()
+
+        # Start building HTML table
+        html = ['<table>']
+
+        # Track cells for multirow
+        multirow_tracker = set()
+
+        # Process rows
+        rows = re.split(r'\\\\', content)
+        current_row = 0
+        
+        for row in rows:
+            if not row.strip():
+                continue
+
+            row = row.strip()
+
+            # Skip \hline
+            if '\\hline' in row:
+                row = row.replace('\\hline', '')
+                if not row.strip():
+                    continue
+
+            row = clean_multi_cells(row)
+
+            # Process cells
+            cells = split_row(row)
+            processed_cells = [process_cell(cell) for cell in cells]
+
+            # Build per-cell line lists splitting on newline or <br> tokens
+            def split_lines(text):
+                parts = re.split(r'(?:\n|<br\s*/?>)+', text)
+                return parts if parts is not None else ['']
+
+            line_lists = [split_lines(cell['content']) for cell in processed_cells]
+            max_lines = max(len(lst) for lst in line_lists) if line_lists else 1
+
+            # Emit one or more rows based on max_lines
+            for line_idx in range(max_lines):
+                if add_head_body:
+                    if current_row == 0:
+                        html.append(' <thead>')
+                    if current_row == 1:
+                        html.append(' <tbody>')
+                html.append('  <tr>')
+                current_col = 0
+
+                for col_idx, cell in enumerate(processed_cells):
+                    content_segment = line_lists[col_idx][line_idx] if line_idx < len(line_lists[col_idx]) else ''
+
+                    attrs = []
+                    if cell['colspan'] > 1:
+                        attrs.append(f'colspan="{cell["colspan"]}"')
+                    # Only apply original rowspan to the first emitted line
+                    if cell['rowspan'] > 1 and line_idx == 0:
+                        attrs.append(f'rowspan="{cell["rowspan"]}"')
+                        for r in range(current_row + 1, current_row + cell['rowspan']):
+                            for c in range(current_col, current_col + cell['colspan']):
+                                multirow_tracker.add((r, c))
+
+                    # If this position is covered by a prior rowspan, skip rendering a duplicate cell
+                    if cell['rowspan'] > 1 and line_idx > 0:
+                        current_col += cell['colspan']
+                        continue
+
+                    if (current_row, current_col) in multirow_tracker and content_segment == '' and cell["colspan"] == 1 and cell["rowspan"] == 1:
+                        current_col += cell['colspan']
+                        continue
+
+                    attr_str = ' ' + ' '.join(attrs) if attrs else ''
+                    cell_tag = 'td'
+                    html.append(f'    <{cell_tag}{attr_str}>{content_segment}</{cell_tag}>')
+                    current_col += cell['colspan']
+
+                if add_head_body and current_row == 0:
+                    html.append(' </thead>')
+                html.append('  </tr>')
+                current_row += 1
+        if add_head_body:
+            html.append(' </tbody>')
+        html.append('</table>')
+        return '\n'.join(html)
+
+    # Convert all tabular environments in the input
+    return re.sub(table_pattern, convert_table, latex_str, flags=re.DOTALL)
+def convert_single_table(table):
+    """
+    Convert a single HTML table to Markdown format.
+    
+    Args:
+        table: BeautifulSoup table element
+        
+    Returns:
+        str: Markdown table string
+    """
+    markdown_lines = []
+    rows = table.find_all('tr')
+    
+    for i, row in enumerate(rows):
+        cells = row.find_all(['td', 'th'])
+        if not cells:
+            continue
+        
+        # Convert cells to text, handling nested elements
+        row_data = []
+        for cell in cells:
+            # Get text content, handling nested elements
+            cell_text = cell.get_text(separator=' ', strip=True)
+            # Escape pipe characters
+            cell_text = cell_text.replace('|', '\\|')
+            row_data.append(cell_text)
+        
+        # Add row to markdown
+        markdown_lines.append('| ' + ' | '.join(row_data) + ' |')
+        
+        # Add separator after header row
+        if i == 0:
+            separator = '| ' + ' | '.join(['---'] * len(cells)) + ' |'
+            markdown_lines.append(separator)
+    
+    return '\n'.join(markdown_lines)
+def convert_html_tables_to_markdown(html_content):
+    """
+    Find all HTML tables and convert them to Markdown while preserving all other content.
+    
+    Args:
+        html_content (str): HTML content that may contain tables
+        
+    Returns:
+        str: HTML content with tables converted to Markdown
+    """
+    soup = BeautifulSoup(html_content, 'html.parser')
+    
+    # Find all tables
+    tables = soup.find_all('table')
+    
+    if not tables:
+        return html_content  # Return original content unchanged
+    
+    # Convert each table to markdown and replace it
+    for table in tables:
+        markdown_table = convert_single_table(table)
+        
+        # Create a new element to replace the table
+        replacement = soup.new_string('\n' + markdown_table + '\n')
+        table.replace_with(replacement)
+    
+    return str(soup)

postprocessing.py

@@ -0,0 +1,95 @@
+import re
+from latex2html import convert_html_tables_to_markdown, latex_table_to_html
+
+def extract_classes_bboxes(text: str):
+    _re_extract_class_bbox = re.compile(r'<x_(\d+(?:\.\d+)?)><y_(\d+(?:\.\d+)?)>(.*?)<x_(\d+(?:\.\d+)?)><y_(\d+(?:\.\d+)?)><class_([^>]+)>', re.DOTALL)
+    classes = []
+    bboxes = []
+    texts = []
+    for m in _re_extract_class_bbox.finditer(text):
+        x1, y1, text, x2, y2, cls = m.groups()
+        classes.append(cls)
+        bboxes.append((float(x1), float(y1), float(x2), float(y2)))
+        texts.append(text)
+
+    # TODO: Remove when fixed
+    classes = [
+        "Formula" if cls == "Inline-formula" else cls for cls in classes
+    ]
+    assert "Page-number" not in classes
+
+    return classes, bboxes, texts
+
+def transform_bbox_to_original(bbox, original_width, original_height, target_w=1648, target_h=2048):
+    # Replicate exact resize logic
+    aspect_ratio = original_width / original_height
+    new_height = original_height
+    new_width = original_width
+    
+    if original_height > target_h:
+        new_height = target_h
+        new_width = int(new_height * aspect_ratio)
+   
+    if new_width > target_w:
+        new_width = target_w
+        new_height = int(new_width / aspect_ratio)
+    
+    resized_width = new_width
+    resized_height = new_height
+    
+    # Calculate padding
+    pad_left = (target_w - resized_width) // 2
+    pad_top = (target_h - resized_height) // 2
+    
+#     # Transform: use the ACTUAL resized dimensions, not the scale
+#     # X coords
+    left = ((bbox[0] * target_w) - pad_left) * original_width / resized_width
+    right = ((bbox[2] * target_w) - pad_left) * original_width / resized_width
+    
+#     # Y coords - using original_height / resized_height directly
+    top = ((bbox[1] * target_h) - pad_top) * original_height / resized_height
+    bottom = ((bbox[3] * target_h) - pad_top) * original_height / resized_height
+    
+    return left, top, right, bottom
+
+def postprocess_text(text, cls = 'Text', text_format='markdown', table_format='latex', blank_text_in_figures=False):
+    assert text_format in ['markdown', 'plain'], 'Unknown text format. Supported: markdown | plain'
+    assert table_format in ['latex', 'HTML', 'markdown'], 'Unknown table format. Supported: latex | HTML | markdown'
+    if cls != 'Table':
+        if text_format == 'plain':
+            text = convert_mmd_to_plain_text_ours(text)
+    elif table_format == 'HTML':
+        text = latex_table_to_html(text)
+    elif table_format == 'markdown':
+        text = convert_html_tables_to_markdown(latex_table_to_html(text))
+    if blank_text_in_figures and cls == 'Picture':
+        text = ''
+    return text
+
+def remove_nemotron_formatting(text):
+   text = text.replace('<tbc>', '')
+   mmd_text = mmd_text.replace('\\<|unk|\\>', '')
+   mmd_text = mmd_text.replace('\\unknown', '')
+
+def convert_mmd_to_plain_text_ours(mmd_text):
+    mmd_text = re.sub(r'<sup>(.*?)</sup>', r'^{\\1}', mmd_text, flags=re.DOTALL)
+    mmd_text = re.sub(r'<sub>(.*?)</sub>', r'_{\\1}', mmd_text, flags=re.DOTALL)
+    mmd_text = mmd_text.replace('<br>', '\n')
+
+    # Remove headers (e.g., ##)
+    mmd_text = re.sub(r'#+\s', '', mmd_text)
+    
+    # Remove bold (e.g., **)
+    mmd_text = re.sub(r'\*\*(.*?)\*\*', r'\1', mmd_text)
+    #mmd_text = mmd_text.replace("**","")
+    # Remove italic (e.g., *)
+    mmd_text = re.sub(r'\*(.*?)\*', r'\1', mmd_text)
+    # Remove emphasized text formatting (e.g., _)
+    mmd_text = re.sub(r'(?<!\w)_([^_]+)_', r'\1', mmd_text)
+    
+    # Remove formulas inside paragraphs (e.g., \(R_{ij}(P^{a})=0\))
+    #mmd_text = re.sub(r'\\\((.*?)\\\)', '', mmd_text)
+    
+    # Remove asterisk in lists
+    #mmd_text = re.sub(r'^\*\s', '', mmd_text, flags=re.MULTILINE)    
+    return mmd_text.strip()

privacy.md

@@ -0,0 +1,10 @@
+Field | Response |
+| :---- | :---- |
+| Generatable or reverse engineerable personal data? | No |
+| Personal data used to create this model? | No |
+| Was consent obtained for any personal data used? | Not Applicable |
+| How often is the dataset reviewed? | Before Release |
+| Is there provenance for all datasets used in training? | Yes |
+| Does data labeling (annotation, metadata) comply with privacy laws? | Yes |
+| Is data compliant with data subject requests for data correction or removal, if such a request was made? | No, not possible with externally-sourced data. |
+| Applicable Privacy Policy | [NVIDIA Privacy Policy](https://www.nvidia.com/en-us/about-nvidia/privacy-policy/) |

safety.md

@@ -0,0 +1,8 @@
+| Field | Response |
+| :---- | :---- |
+| Model Application Field(s): | Chat, Instruction Following, Chatbot Development, Code Generation, Reasoning, Customer Service |
+| Describe the life critical impact (if present). | Not Applicable |
+| Use Case Restrictions: | GOVERNING TERMS: The NIM container is governed by the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [Product-Specific Terms for NVIDIA AI Products](https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/). Use of this model is governed by the [NVIDIA Community Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/). Use of the tokenizer included in this model is governed by the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/)|
+| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
+
+**You are responsible for ensuring that your use of NVIDIA AI Models complies with all applicable laws.**