Active Topological Learning (ATL): A Geometric Framework for Targeted Knowledge Injection in Small Language Models
Tania Swanepoel
Zenodo (CERN European Organization for Nuclear Research)
Problems Identified (5)
Fine-tuning resource cost: Current fine-tuning approaches for small language models require large datasets and significant compute.
Fine-tuning knowledge degradation: Current fine-tuning approaches risk degrading existing model knowledge.
Targeted conceptual knowledge injection: The paper addresses how to insert targeted conceptual knowledge into pre-trained SLM embedding spaces.
Resource-constrained specialization: The paper raises the problem of enabling on-demand model specialization in resource-constrained deployments.
Fine-tuning resource cost: Current fine-tuning approaches for small language models require large datasets and significant compute.
Proposed Solutions (5)
Active Topological Learning: Active Topological Learning is proposed as a geometric method for targeted conceptual knowledge insertion in pre-trained SLM embedding spaces.
Conceptual void detection: ATL detects conceptual voids using the Quality Signal metric.
Angular manifold pre-survey: ATL measures existing manifold structure with Angular Pre-Survey.
Geometric micro-corpus design: ATL designs a geometrically derived Micro-Corpus for targeted training.
Anisotropic gradient masking: ATL trains only along identified load-bearing dimensions using anisotropic gradient masking.
Results (3)
Fast concept relocation:
No catastrophic forgetting observed:
Naive-query discoverability:
Research Domain
Small language models; targeted knowledge injection; embedding geometry