I wanted a model to turn a passage of text into a multiple-choice question and return fields my app could read. This was my first fine-tuning attempt. I knew what I wanted the output to look like, but I was still learning how to get there.
These are notes from that experiment in 2024, using Mistral-7B-Instruct-v0.1. Some choices worked well enough for the project; others were workarounds I didn't fully explain at the time. I wouldn't treat the environment settings as a current installation guide.
Starting with the output
The app needed a question, an answer and distractors in a predictable JSON shape. I started with fine-tuning before I knew much about grammar-constrained generation. I later tried jsonformer, but it didn't give me the flexibility I wanted for that setup.
That distinction is worth being clear about: training on a format can encourage the model to follow it, but it doesn't guarantee that every output matches a schema. The application still needs to parse and validate the result.
Getting the training run working
I used Hugging Face's TRL tooling and tried LoRA and QLoRA. QLoRA was the more practical fit for my experiment. I preferred the results I saw, though I didn't run a controlled comparison that would establish it as the better approach generally.
The dependency setup took more effort than I'd expected. CUDA compatibility affected both training and later quantization. I recorded CUDA 12.1 around this work, but the useful habit is to check the versions required by the exact packages you're installing and keep the working combination recorded.
Padding and stopping behaviour also took some trial and error. In my runs, using the unknown token for padding worked better than using the end-of-sequence token. I hadn't isolated why. I would check the masking, stop settings and training examples before treating that change as a reusable fix.
One epoch gave me the result I kept
I trained for three epochs, but the later output followed the expected format less reliably. I settled on the one-epoch version. I didn't establish whether the cause was overfitting, the data, or something in my training setup, so that remains an observation about this run.
The model could produce usable questions from the supplied text. It often stayed very close to that passage, though, so a question could rely on context that wouldn't be obvious when shown on its own. Valid JSON didn't tell me whether the question was useful.
I also saw the same input produce the same response in my setup. Changing temperature and top_k made some outputs worse. I kept the settings that behaved acceptably, but I wouldn't infer from that experiment that the model itself was inherently deterministic.
Serving it on a limited budget
I quantized the model with AWQ and served it through a RunPod handler. Quantization improved inference speed in my tests, though I didn't keep a benchmark suitable for quoting a speedup. I tried batching too and chose single generation to keep memory use manageable.
The limited GPU budget shaped the project quite a lot. It made me pay attention to whether a change was worth another run, and it left some questions unanswered. I enjoyed getting a model from training into an API my app could call, even with those rough edges.
If I repeated the experiment, I'd set aside a small set of passages before training and score both format validity and question quality after each run. That would make it easier to tell whether a change helped, instead of relying on whichever examples I happened to try that evening.