Revisiting QuizzifyMe’s document-to-quiz pipeline

QuizzifyMe was a university group project that turned lecture slides, PDFs and notes into multiple-choice questions. I worked on much of the application and ML pipeline. The idea was easy to explain: upload the material, wait for the questions, then use them to revise.

Looking back, I spent a lot of attention on the model. Several of the problems I found later were in the ordinary code between the document and the model call.

Loading the model for every batch

This function loads the embedding tokenizer and model each time it's called:

def generate_embeddings(input_texts: list[str]) -> list[float]:
    tokenizer = AutoTokenizer.from_pretrained("./celery_workers/src/api/gte-small/")
    model = AutoModel.from_pretrained("./celery_workers/src/api/gte-small/")
    batch_dict = tokenizer(input_texts, max_length=512, padding=True,
                           truncation=True, return_tensors="pt")
    ...

The caller processes ten chunks at a time. A document with 200 chunks therefore goes through that loading path 20 times. The README's target machine had 2GB of RAM and one vCPU, so this was worth noticing much earlier.

I'd look at keeping the model loaded for the worker's lifetime, with its memory use checked under the actual worker setup. I don't have a timing comparison, but repeatedly loading the same model is work the pipeline shouldn't need to repeat for every batch.

Making the slow work a job

The overall flow still makes sense to me. FastAPI accepts the request, Celery runs the slow work, and the frontend polls for progress. Readers turn PDFs, Word documents and slides into text units before embedding and generation.

Keeping those stages visible matters when someone is waiting. Extracting a document and generating questions are different jobs, and a failure in one shouldn't look like the other simply taking longer.

The chunk size wasn't what I thought

The chunking function uses a 512-character window with a 378-character step:

def sliding_window(pages, window_size=512, slide=378) -> list[str]:
    page_single_string = " ".join(pages)
    window = []
    for i in range(0, len(page_single_string), slide):
        window.append(page_single_string[i : i + window_size])
    return window

That gives 134 characters of overlap. The embedding tokenizer's limit is expressed in tokens, though. Those are different units, and the number of tokens depends on the text. I hadn't measured how much useful context each chunk carried.

The function also slices through words and flattens all page boundaries with " ".join(pages). Once those boundaries are gone, it's harder to tell the student where a question's source passage came from.

I'd keep source locations and compare chunking approaches on actual documents before choosing another size. Bigger chunks alone aren't a guarantee of better retrieval.

Changing chunking would also change retrieval

Pinecone stored an ID and vector, with the chunk index encoded in the ID. Retrieval rebuilt the chunks to recover their text:

def extract_context(matches, pages):
    context_ids = [match.get("id").split(":")[1] for match in matches]
    chunks = sliding_window(pages)          # <- recomputed here
    context = [chunks[int(context_id)] for context_id in context_ids]
    return "".join(context)

That means the chunking settings are part of the stored data's meaning. Change them after indexing, and an old index can refer to a different passage without producing an error.

I'd store the chunk text and source metadata with the vector, or keep them in a versioned record the vector points to. The retrieval code also joins passages without a separator, which makes the resulting context harder to read.

A failed job could look successful

The worker catches an exception and returns a failure message:

@app.task(bind=True, name="quiz")
def quiz_worker(self, quiz_id, default_model, access_token, refresh_token):
    try:
        ...
        return {"message": "success"}
    except Exception as e:
        return {"message": f"failed: {str(e)}"}

Returning normally means the task can be recorded as successful even though generation failed. The caller then has to inspect the payload to learn what happened. That undermines the progress reporting I'd taken care to add.

I would make failures explicit in the task contract and check the frontend against both outcomes. A missing endpoint or a parsing error should become a visible failure with a useful next step.

The model still needs a clear output contract

We used a fine-tuned Mistral model on RunPod and a Mixtral path through Fireworks. The fine-tuned response parser searched for a marker from the training format:

special_token = "### Output :"
output_start = llm_response.find(special_token) + len(special_token)

That marker ties the parser to the model template. A change during retraining needs a corresponding check in the application. JSON output is easier to validate structurally, but even a valid object can contain an unsupported answer or a poor distractor.

I still like the flow from uploaded material to a quiz and its history. If I were doing it again, I'd spend more time checking the retrieved passages, job failures and output parsing alongside the model. A good question depends on all of those parts agreeing about what they're passing to the next one.