Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques

Large language models have transformed natural language processing, yet supervised fine-tuning (SFT) remains computationally intensive. This paper formally proves that capabilities acquired through SFT can be approximated by a base transformer model using inference-time techniques, specifically in-context learning (ICL), without altering model parameters, under idealized assumptions including unbounded computational resources and access to the fine-tuning dataset. We extend these results to practical scenarios with finite context lengths and partial dataset access. For text generation tasks with fixed output length l, datasets of size O\left( \frac{m V}{\varepsilon^2} \log \frac{m}{\delta} \right) or, with bounded context, O\left( \frac{l \log V}{\varepsilon^2} \log \frac{1}{\delta} \right) suffice to approximate fine-tuned behavior across m contexts within error \varepsilon, where V is the vocabulary size and \delta is the failure probability. For linear classification, datasets of size O\left( \frac{d}{\varepsilon} \right) or, with fixed context, O\left( \frac{1}{\varepsilon^2} \log \frac{1}{\delta} \right) are sufficient, where d is the input dimension. Grounded in the Turing completeness of transformers, these results provide a theoretical foundation for resource-efficient deployment of large language models, with practical techniques like retrieval-augmented generation bridging theory to real-world applications.