Type: Project-learning note (paper/code reading & line-by-line walkthrough) Source project: Few-Shot Image Classification on Food101 — Kaggle About this note: This is my personal study note based on the tutorial above. The implementation code belongs to the original author; the structure, explanations, and the self-posed Q&A throughout reflect my own understanding while working through it.
The metric-based procedure:
Two core challenges:
Q: What is the relationship between a feature space and a representation?
The feature space comes first. It is not hand-designed; it is determined by the CNN’s architecture and parameters. The logic chain is:
- You design/choose a CNN architecture (e.g., the last layer outputs a 512-dim vector) → this implicitly defines a 512-dim feature space.
- You train the CNN (adjust its parameters) → this determines the semantic structure of that space, i.e., which regions mean what. How well it is trained directly determines the quality of the space.
- You feed an image into the trained CNN → it outputs a 512-dim vector → that vector is the image’s representation in this feature space.
CNN defines the space → training shapes the space → a representation is produced only after an image is fed in.
Analogy: the feature space is a map; the representation is the image’s coordinates on that map. The CNN’s job is to “locate” each image — to compute its latitude/longitude. Images of the same class should land in nearby positions; images of different classes should land far apart.
Initialize PrototypicalNetworks with a backbone: a ResNet18 pretrained on ImageNet, with its head chopped off and replaced by a Flatten layer.
Q: What is the relationship between a backbone and a feature extractor?
Here they are the same thing, just named from different angles:
- Feature extractor — the functional name: its job is to extract features from an image and output a feature vector.
- Backbone — the architectural name: it is the model’s “spine” / main network, the core part.
In this code, the backbone is a ResNet18 with its classification head removed. Originally, ResNet18’s last layer is a classification layer (e.g., outputting probabilities over 1000 classes); here that head is chopped off and replaced by a
Flattenlayer so the network directly outputs a 512-dim feature vector. It no longer classifies — it purely extracts features. So: we use a (headless) ResNet18 as the backbone, which is our feature extractor, turning images into 512-dim vectors. Prototypical Networks then take these vectors to compute prototypes and compare distances.
forward() Takes Three InputsThe forward method here does not take a single input tensor — it takes three.
forward is the model’s forward-pass function: you feed data in, the model processes it and produces an output. In PyTorch, every model has a forward() method that defines how incoming data flows step by step to the output.
The key contrast is between a normal classifier and a Prototypical Network:
forward: input one image → output a predicted class. Only one input needed.forward: needs three inputs — support imagessupport labels (tells the model which image belongs to which class)query images (the images to be classified)Why three? Because the classification logic is “compare distances.” The model must first see the support images and their labels to compute each class’s prototype, before it can decide which class a query image belongs to. Without the support set, it has nothing to compare against. In other words, calling this model is not as simple as passing in a single image — you must also tell it “what the references are” (support images + labels) before it can predict on the query images.
class PrototypicalNetworks(nn.Module):
def __init__(self, backbone: nn.Module):
super(PrototypicalNetworks, self).__init__()
self.backbone = backbone
Defines a Prototypical Networks model that inherits from PyTorch’s nn.Module. Initialization does just one thing: store the passed-in backbone (the feature extractor). All later feature extraction relies on it.
z_support = self.backbone.forward(support_images)
z_query = self.backbone.forward(query_images)
Feed the support images and query images separately into the backbone, each producing feature vectors. For example, 5 input images → 5 vectors of 512 dims each. z_support and z_query are the two sets of embeddings.
n_way = len(torch.unique(support_labels))
z_proto = torch.cat(
[
z_support[torch.nonzero(support_labels == label)].mean(0)
for label in range(n_way)
]
)
torch.unique(support_labels) finds how many distinct classes are in the support set — e.g., 3 classes (pug, labrador, Saint-Bernard) → n_way = 3.support_labels == label selects the embeddings of all images in that class, and .mean(0) averages them to obtain that class’s prototype.torch.cat stacks all class prototypes into a matrix. The final z_proto has shape (n_way, 512), where each row is one class’s prototype vector.dists = torch.cdist(z_query, z_proto)
scores = -dists
return scores
torch.cdist computes the Euclidean distance from each query embedding to each prototype, giving a distance matrix. E.g., 10 queries, 3 classes → dists is (10, 3).scores = -dists: negate the distances to turn them into scores. Smaller distance means more similar; after negation, a larger value means more similar — aligning with the usual “higher score is better” convention, so you can directly apply softmax or argmax afterward to pick the top-scoring class.convolutional_network = resnet18(pretrained=True)
convolutional_network.fc = nn.Flatten()
model = PrototypicalNetworks(convolutional_network).cuda()
fc with Flatten, so the network outputs a 512-dim feature vector instead of probabilities over 1000 classes..cuda() moves the model to the GPU.(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Flatten(start_dim=1, end_dim=-1)
)
)
Reading the network structure:
BasicBlock — the basic building unit of ResNet18: the standard flow of conv → norm → activation → conv → norm. The 512 indicates this layer processes feature maps with 512 channels.avgpool: AdaptiveAvgPool2d(output_size=(1, 1)) — global average pooling. Regardless of the preceding feature map’s size, it compresses to (512, 1, 1) — one number per channel. This step turns a 2D feature map into a 512-dim vector.fc: Flatten(start_dim=1, end_dim=-1) — this is the layer you swapped in. Originally it was a fully connected layer Linear(512, 1000) outputting scores over 1000 classes. Now it is Flatten, which merely reshapes (512, 1, 1) into (512,) without doing any computation.How to verify the head was successfully chopped off? Look at the fc line. If it shows Flatten instead of Linear(512, 1000), the classification head has been successfully replaced. The model now outputs a 512-dim feature vector — exactly the embedding Prototypical Networks need.
Problem: A standard PyTorch DataLoader feeds in batches of images without considering labels or whether an image belongs to the support set or the query set. Given this, two key operations are required:
- Images must be evenly distributed across a given number of classes.
- The batch must be split between support and query sets.
A PyTorch DataLoader has three components, each with one job:
n_way classes, then take n_shot + n_query images from each class, ensuring each class is evenly represented.N_WAY = 5 # Number of classes in a task
N_SHOT = 5 # Number of images per class in the support set
N_QUERY = 10 # Number of images per class in the query set
N_EVALUATION_TASKS = 100
test_set.get_labels = lambda: test_set._labels
test_sampler = TaskSampler(
test_set, n_way=N_WAY, n_shot=N_SHOT, n_query=N_QUERY, n_tasks=N_EVALUATION_TASKS
)
test_loader = DataLoader(
test_set,
batch_sampler=test_sampler,
num_workers=12,
pin_memory=True,
collate_fn=test_sampler.episodic_collate_fn,
)
Built into PyTorch:
DataLoader — PyTorch’s native data loader.num_workers=12 — PyTorch’s built-in multi-process loading parameter.pin_memory=True — PyTorch’s built-in GPU-memory optimization parameter.Custom:
TaskSampler — a custom sampler that samples classes and images evenly according to the few-shot task requirements. PyTorch’s built-in samplers only grab randomly or sequentially; they don’t understand the “pick classes first, then pick images” logic.test_sampler.episodic_collate_fn — a custom collate function attached to the TaskSampler. It splits a batch into support/query groups and re-indexes labels. PyTorch’s default collate_fn only stacks data into a tensor; it doesn’t know how to split.test_set.get_labels = lambda: test_set._labels — temporarily adds a method to the dataset so the TaskSampler can access all image labels, letting it know which images belong to which class so it can sample by class.Present but not really “custom”:
test_set — the dataset object. Although it may be created with a library like easy-set, it follows PyTorch’s Dataset interface, so it isn’t custom logic.N_WAY, N_SHOT, N_QUERY, N_EVALUATION_TASKS — just configuration numbers: 5 classes, 5 support images each, 10 query images each, 100 evaluation tasks total.Takeaway: the DataLoader is PyTorch’s shell, but the Sampler and Collate Fn inside it are self-written — these two are what turn ordinary data loading into few-shot task loading.
evaluate_on_one_taskdef evaluate_on_one_task(
support_images: torch.Tensor,
support_labels: torch.Tensor,
query_images: torch.Tensor,
) -> [int, int]:
: torch.Tensor is a Python type hint, indicating that the argument should be a torch.Tensor — a multi-dimensional array, similar to a NumPy ndarray.y_pred = torch.max(
model(support_images.cuda(), support_labels.cuda(), query_images.cuda())
.detach()
.data,
1,
)[1]
return y_pred
support_images.cuda(), support_labels.cuda(), query_images.cuda() — move all three inputs to the GPU, since the model is on the GPU and the data must be there too in order to compute.model(...) — feed into the Prototypical Networks; internally it extracts features → computes prototypes → computes distances → returns a score matrix of shape (num_queries, num_classes), e.g., (50, 5), where each row is one query’s scores over the 5 classes..detach().data — detach the result from the computation graph; evaluation needs no gradients, saving memory.torch.max(..., 1) — take the max along dim 1 (the class dimension), returning a tuple (max_value, index_of_max).[1] — keep only the index, i.e., the predicted class id. E.g., if a query’s scores are [-1.2, -0.3, -2.5, -4.0, -1.8], the max is -0.3 at index 1, so the prediction is class 1.y_pred, a tensor containing the predicted classes for all queries.Q: What is
.detach().data?During computation, PyTorch by default records every operation, forming a computation graph so that gradients can be backpropagated and parameters updated during training. But at evaluation time we only want a prediction — no parameter updates — so this graph is unnecessary.
.detach().datatells PyTorch: I only want the numeric values, drop the computation history so it doesn’t waste memory. There is already atorch.no_grad()outside, which serves a similar purpose (also not recording the graph), so adding.detach().datahere is double insurance, ensuringy_predis a clean, history-free pure-data tensor.
evaluatedef evaluate(data_loader: DataLoader):
total_predictions = 0
correct_predictions = 0
f1_scores = []
Purpose: iterate over all test tasks and compute overall accuracy and F1. First, initialize three counters.
model.eval()
with torch.no_grad():
model.eval() switches the model to evaluation mode, so layers like BatchNorm and Dropout use evaluation behavior rather than training behavior. torch.no_grad() tells PyTorch not to record the computation graph, saving memory since evaluation needs no backpropagation.
Q: Are
model.eval()andtorch.no_grad()built-in or custom?Both are built into PyTorch.
modelis an object you created, but the.eval()method is inherited fromnn.Module. BecausePrototypicalNetworksinherits fromnn.Module, it automatically has.eval()— no need to define it yourself.torch.no_grad()is a framework-level tool; you calltorch.no_grad()directly, independent of your own model definition.
for episode_index, (
support_images,
support_labels,
query_images,
query_labels,
class_ids,
) in enumerate(data_loader):
Iterate over the DataLoader, taking one batch — i.e., one few-shot task — at a time. Unpack the five items mentioned earlier: support images, support labels, query images, query labels, and the true class mapping.
y_pred = evaluate_on_one_task(
support_images, support_labels, query_images,
)
Call the function above to get predictions for all queries in this task.
total_predictions += len(query_labels)
correct_predictions += (y_pred == query_labels.cuda()).sum().item()
f1_scores.append(f1_score(query_labels.cpu(), y_pred.cpu(), average='macro'))
Three lines, one job each:
y_pred == query_labels.cuda() compares predictions with true labels element-wise, producing a True/False tensor; .sum() counts the correct ones; .item() converts to a Python number, added to the correct count.f1_score to compute this task’s macro F1 (the average of per-class F1). Note that sklearn only handles CPU data, so .cpu() is needed to move the tensors back.accuracy = (correct_predictions/total_predictions)
avg_f1 = sum(f1_scores)/len(f1_scores)
print(
f"Model tested on {len(data_loader)} tasks. Accuracy: {accuracy:.2%}, F1: {avg_f1}"
)
return accuracy
After the loop, compute overall accuracy (correct / total) and average F1 (mean of all tasks’ F1), print the results, and return the accuracy.
Definition: Meta-learning refers to a training strategy, not a change of model. (We keep the same model, so the weights stay pretrained on ImageNet.) Previously we used the pretrained ResNet directly, without extensive training on few-shot tasks. Now we will train this model on a large number of few-shot tasks so it learns how to do few-shot classification itself.
Concretely: prepare 40,000 few-shot tasks, each one being “here are a few support images, predict which class the queries are.” The model repeatedly does these tasks; when it gets one wrong, it computes a loss and updates parameters (mainly the ResNet backbone’s parameters), so that the features it extracts become better suited to few-shot classification — embeddings of the same class get closer, and those of different classes get farther apart.
That is why it is called meta-learning: ordinary learning learns “how to recognize cats and dogs,” while meta-learning learns “how to quickly learn to recognize new things from a few samples” — learning to learn.
fit method: Meta-training loop: fit → loss → update model parameters.
a. Define fit
def fit(
support_images: torch.Tensor,
support_labels: torch.Tensor,
query_images: torch.Tensor,
query_labels: torch.Tensor,
) -> float:
optimizer.zero_grad()
classification_scores = model(
support_images.cuda(), support_labels.cuda(), query_images.cuda()
)
loss = criterion(classification_scores, query_labels.cuda())
loss.backward()
optimizer.step()
return loss.item()
optimizer.zero_grad() — zero out gradients. PyTorch accumulates gradients by default, so you must clear them before each training step to prevent the previous step’s gradients from affecting this one.classification_scores = model(...) — forward pass: feed the data into the model to get the queries’ scores over each class. Same as in evaluation, but this time PyTorch records the computation graph because gradients will be computed next.loss = criterion(classification_scores, query_labels.cuda()) — compute the loss by comparing the model’s scores against the queries’ true labels. criterion is typically CrossEntropyLoss; the further the scores are from the correct answer, the larger the loss.loss.backward() — backpropagation: trace back through the computation graph from the loss to compute each parameter’s gradient, i.e., “which direction and how much to adjust each parameter to reduce the loss.”optimizer.step() — update parameters: using the computed gradients, actually adjust the backbone (ResNet) parameters so feature extraction improves.return loss.item() — return this task’s loss as a number; .item() converts from tensor to a plain Python number, convenient for logging and printing.Note: evaluate_on_one_task only predicts and does not update parameters; fit predicts and updates parameters based on the result. One is like taking an exam; the other is like doing practice problems and learning from the mistakes.
b. Run the parameter updates
log_update_frequency = 100
acc_update_frequency = 500
all_loss = []
all_acc = []
Initialize configuration: update the loss shown on the progress bar every 100 tasks; measure accuracy every 500 tasks (though this is commented out below). The two lists record all losses and accuracies.
model.train()
Switch the model to training mode — the opposite of the earlier model.eval(). Layers like BatchNorm and Dropout will use training behavior.
with tqdm(enumerate(train_loader), total=len(train_loader)) as tqdm_train:
tqdm is a progress-bar library wrapped around the DataLoader so you can see training progress, e.g., “3000/40000 tasks done.”
for episode_index, (
support_images,
support_labels,
query_images,
query_labels,
_,
) in tqdm_train:
Iterate over each few-shot task and unpack the five items. The last one _ is the true class mapping, unused during training, so it is ignored with an underscore.
loss_value = fit(support_images, support_labels, query_images, query_labels)
all_loss.append(loss_value)
For this task, call fit to do one forward pass + loss + backpropagation + parameter update, then record the returned loss value.
if episode_index % log_update_frequency == 0:
tqdm_train.set_postfix(loss=sliding_average(all_loss, log_update_frequency))
Every 100 tasks, display the average loss over the last 100 tasks on the right side of the progress bar. sliding_average computes a moving average, which is more stable than a single loss reading and shows whether the loss is trending downward overall.
# if episode_index % acc_update_frequency == 0:
# all_acc.append(evaluate(test_loader))
# model.train()
This block is commented out. It would have paused training every 500 tasks to run a test-set evaluation, then switched back to training mode and continued. It was likely commented out to save time. Note that evaluate calls model.eval() internally, so after evaluating you must call model.train() again to switch back.
On the relationship between fit, loss, and update: these three are not parallel — they are nested. The fit function does three things internally: forward pass to get scores; compute the loss (criterion(classification_scores, query_labels.cuda())); update parameters (loss.backward() + optimizer.step()). So this training loop simply calls fit repeatedly, and each fit completes one loss computation and one parameter update.
c. Plot the loss curve
window_width = 50
cumsum_vec = np.cumsum(np.insert(all_loss, 0, 0))
ma_vec = (cumsum_vec[window_width:] - cumsum_vec[:-window_width]) / window_width
These three lines compute a moving average. Because each task’s loss fluctuates a lot, plotting the raw loss curve would be very jagged and hard to read for a trend. So a window of 50 is used to smooth it: each point is the average of itself and the previous 49 points. This uses the cumulative-sum (cumsum) trick to compute the moving average quickly — the result is the same as averaging window by window, just faster.
plt.title("Loss vs # Training Episodes")
plt.xlabel("# Training Episodes")
plt.ylabel("Loss")
plt.plot(ma_vec, color = "green")
plt.show()
Plot with matplotlib. The x-axis is the training task index, the y-axis is the loss, drawn as a green curve.
evaluate(test_loader)
Result:
Model tested on 100 tasks. Accuracy: 89.92%, F1: 0.8966519882510688
0.8992
The road into academic life looks like it came about by chance. Before I was 22, I studied the social sciences. They taught me to connect my own point of view to the workings of society; pressures and feelings I couldn’t make sense of were finally given names, and stopped being a nameless dread. From 23 onward I began studying data science, statistics, and computer science—and this is where the whole thing started to feel questionable. I’m not critical enough to stand against all the social ills, not radical enough to tear down the frameworks, not clever enough to build a scientific vision of my own. But I’m stubborn enough; I love to express, I like to think; I’m slow enough not to fret about my age, lucky enough to have a family that supports me, and I happen to enjoy the game we call social competition.
When Xiang Biao spoke about personal crisis, he said:
It mostly comes down to not being able to get things written. You work on a project for years and you’re never satisfied with what you write, because you don’t have strong or deep ideas, you don’t have your own voice, and the writing wears you out. You keep at it, reaching for this framework and that one, this theory and that theory, and you sink into a painful state—you’ve already poured so much thought into it that you can’t give it up.
This made me anxious, too. The theory of computer science is another river, and I want to follow his advice—to walk it for a while, to find my own voice, my own experience. It’s also why I built this blog: I hope to understand what my own line of thinking actually is—why I work on LLMs, why I work on AI. Perhaps my reflection and understanding are a gift the social sciences gave me, a gift that will carry me, flowing on, through this other river.
As Hesse wrote:
All the waters of the world are bound to meet again; the Arctic Ocean and the Nile mingle in the moisture of the clouds. This ancient, beautiful image makes this moment holy. Even in wandering, every road carries us home. … I am alone, and yet at ease. I ask for nothing but to be soaked through by the sun.
Source: Chengquan Guo, Yuzhou Nie, Chulin Xie, Zinan Lin, Wenbo Guo, Bo Li. BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI. arXiv:2510.18131, 2025. [arXiv] · [OpenReview] · [Microsoft Research blog]
The work proceeds along two tracks: benchmarks and methodology.
Key insight: strong red-team knowledge can inform and strengthen the blue team.
Pain point: manually defining large-scale, high-quality security principles is impractical.
The risks BlueCodeAgent targets split into two levels — the input / textual level (risks in the instructions) and the output / code level (risks in the generated code). Each leaf category contrasts an unsafe case against its safe counterpart, which is what makes the detection task non-trivial: the defender must separate genuinely harmful inputs/outputs from superficially similar but benign ones.
flowchart TD
TR[Target Risks]
TR --> IN[Input / Textual Level<br/>Risks in instructions]
TR --> OUT[Output / Code Level<br/>Risks in generated code]
IN --> BIAS[Bias Instructions]
IN --> MAL[Malicious Instructions]
OUT --> VC[Vulnerable Code]
BIAS --> BIAS_U[Unsafe<br/>Biased / unfair intent]
BIAS --> BIAS_S[Safe<br/>Normal coding requests]
MAL --> MAL_U[Unsafe<br/>Malware creation]
MAL --> MAL_S[Safe<br/>Normal coding requests]
VC --> VC_U[Unsafe<br/>CWE vulnerabilities]
VC --> VC_S[Safe<br/>CWE-repaired code]
classDef top fill:#ede7f6,stroke:#7e57c2,color:#000;
classDef input fill:#d7f0e4,stroke:#26a69a,color:#000;
classDef output fill:#fbe3da,stroke:#e07a5f,color:#000;
classDef unsafe fill:#fde4e4,stroke:#e53935,color:#000;
classDef safe fill:#e6f4d7,stroke:#7cb342,color:#000;
class TR top;
class IN,BIAS,MAL input;
class OUT,VC output;
class BIAS_U,MAL_U,VC_U unsafe;
class BIAS_S,MAL_S,VC_S safe;
Legend. Purple = top-level category · teal = input / textual-level risks · salmon = output / code-level risks · red = unsafe (harmful or vulnerable) · green = safe (normal or repaired).
BlueCodeEval, plus BlueCodeEval-PI for prompt injection) alongside an external reference benchmark, SecCodePLT, which provides both insecure and secure code snippets.BlueCodeKnow or BlueCodeEval.TP / (TP + FP).TP / (TP + FN).2 · Precision · Recall / (Precision + Recall); a balanced measure that jointly accounts for precision and recall.An easily overlooked detail: the distinction between seen and unseen risks.
BlueCodeKnow.BlueCodeEval that do not overlap with the categories in BlueCodeKnow.Results are reported across four representative code-related tasks:
BlueCodeEval-PI test set, whose prompt-injection cases are generated from red-teaming.BlueCodeAgent performs consistently well on both seen and unseen risks across these tasks.
Sensitivity to Different Knowledge: compute the cosine similarity between the category embeddings of the seven test categories and the eight knowledge categories, and express the result via the Pearson coefficient:
Pearson = corr(cosine similarity, F1 score)
Interpretation: this measures how much each test category depends on the eight existing knowledge categories. The higher the coefficient, the stronger that dependence.
On the algorithmic side, I would add a risk gate. Static analysis — and especially LLM-assisted static analysis — inherently suffers from false negatives. This is precisely the motivation behind IRIS: traditional static analysis is constrained by hand-written specifications and limited contextual understanding, while an LLM on its own also struggles with whole-repository vulnerability reasoning, so the two are combined.
The crucial point is that “no vulnerability found” should not be treated as a termination condition. A more reasonable approach is to introduce a risk gate:
R ← RiskGate(T, S, C)
if S says safe and R = low and confidence(S) ≥ τ then
return SAFE_LOW_RISK
else
continue dynamic validation
This design can borrow from the literature on SAST false-positive / false-negative reduction. For example, QASecClaw centers on the idea that a SAST tool first produces candidate vulnerabilities, and an LLM-based filter agent then judges true positives vs. false positives by reasoning over the code context. On the OWASP Benchmark v1.2, it raised Semgrep’s F1 from 78.39% to 90.93%, mainly by substantially cutting false positives.
But QASecClaw also illustrates an important caveat: an LLM is well suited to triage / contextual filtering, not to being treated as an absolute oracle. Mapped onto this algorithm, “no vulnerability found” should therefore not end the process directly — it should instead be routed into continued dynamic validation.
]]>