GraphMERT

Graphical Multidirectional Encoder Representations from Transformers

End-to-end neurosymbolic pipeline for constructing high-quality knowledge graphs from unstructured data.

Overview

GraphMERT is an 80M parameter encoder-only transformer for extracting reliable domain-specific Knowledge Graphs from unstructured text. It combines semantic information from a seed Knowledge Graph with syntactic information from text using a novel chain graph encoding.

The model is trained with two objectives:

• MLM (Masked Language Modeling): Standard language modeling with span masking
• MNM (Masked Node Modeling): Predict masked tail entities in KG triples

Architecture

• 12 hidden layers, 8 attention heads, 512 hidden size
• Chain Graph Encoding: Unifies text tokens and KG triples into a single graph
• H-GAT (Hierarchical Graph Attention Network): Creates relation embeddings
• Attention Decay Mask: Exponential decay based on graph distance

Installation

pip install -r requirements.txt

Project Structure

graphmert/
├── src/
│   ├── models/
│   │   ├── graphmert.py      # Main model
│   │   ├── hgat.py           # Hierarchical GAT
│   │   ├── chain_graph.py    # Graph encoding
│   │   └── attention.py      # Attention decay mask
│   ├── config.py             # Model configuration
│   └── tokenizer.py          # Tokenizer utilities
├── requirements.txt
└── README.md

Usage

from src.models import GraphMERT
from src.config import GraphMERTConfig
from src.tokenizer import create_tokenizer

# Create model
config = GraphMERTConfig()
model = GraphMERT(config)

# Create tokenizer
tokenizer = create_tokenizer()

# Encode input
encoding = tokenizer.encode("Metformin is used to treat type 2 diabetes.", return_tensors="pt")

Citation

Based on the GraphMERT paper (https://arxiv.org/abs/2510.09580) from Princeton University.

License

MIT