Three-class chest X-ray classification

Classify a chest X-ray as COVID-19, lung cancer, or normal. Chest X-rays are misread often enough that a second opinion has value, particularly where a radiologist is not immediately available, and the three-class framing is narrow enough to do properly at course scale.

I set the overall approach and owned classifier training and hyperparameter tuning.

Data, and what we refused to do to it

Images came from the COVID-19 Radiography Database and the NIH Chest X-ray set: 3,616 COVID, 4,844 cancer and 10,192 normal. Training on that distribution would have produced a model that mostly learned to say normal, so every class was truncated to 3,616, giving 10,848 images split 70/15/15.

We used no data augmentation, which is unusual enough to justify. Each standard technique was rejected for a specific reason rather than as a blanket policy:

• Cutout and random noise can erase the finding. A nodule under 3cm is small enough to be removed outright by a cutout patch.
• Flips and rotations add no real variation, because clinical X-rays are already captured in a consistent orientation. A mirrored chest is not a chest the model will ever see.
• CutMix would paste a tumour into a COVID lung and label the result one thing when it shows two.
• Intensity scaling introduces brightness artefacts that resemble pathology in a greyscale image.

In natural-image classification these are close to free. In medical imaging the augmentation can destroy the signal being classified, and the honest choice was a smaller balanced dataset rather than a larger corrupted one.

Model

A pre-trained VGG-16 as a frozen feature extractor, producing a 512 × 9 × 9 map from each 299 × 299 image, flattened to 41,472 features. On top of that, a three-layer head: 256 with batch normalisation, ReLU and 0.3 dropout, then 128 with batch normalisation and ReLU, then 3 logits.

The baseline for comparison was a support vector machine on the same preprocessed data, polynomial kernel, C = 0.1, which reached 82.2% on the test set. A strong baseline is more useful than a weak one, since it sets the bar the network has to clear to justify itself.

Results

The classifier reached 90.1% test accuracy, 7.9 points above the SVM. The per-class breakdown is the more interesting part.

Cancer is close to solved at this scale, at 96.3 precision and 97.2 recall. Almost all of the remaining error is COVID and normal being mistaken for each other, with 69 normal scans read as COVID on the test set.

That grouping is explicable. A tumour presents as a well-defined solid mass, which is exactly the kind of structure a convolutional feature extractor is good at. Early or mild COVID can present with subtle findings or none at all on a chest radiograph, and the radiographic picture overlaps with influenza and pneumonia. A model working from the image alone has less to separate.

Which errors we chose to make

The two error types are not equally costly, so optimising overall accuracy would have been the wrong target. A missed cancer is a patient sent home. A false positive is a patient who gets a tissue sample they did not need.

We therefore preferred high recall on cancer over high precision, accepting more false alarms to reduce the chance of a missed one. At 97.2 recall the model misses roughly 3 in 100 cancer cases in this test set, which is the number that would matter first in any real evaluation, and it is not low enough for clinical use.

Limitations

Both source datasets are public collections, and a model trained on them learns their acquisition equipment and populations along with the pathology. Performance on X-rays from a different machine or a different demographic is unmeasured, and the standard failure mode in this literature is exactly that gap.

The evaluation is a single held-out split with no cross-validation and no external test set. Accuracy on unseen images from the same collections is a much weaker claim than accuracy on unseen images generally.

Covid_Cancer_Classifier on GitHub →

Four-person team with George Gerges, Selim Abdelwahab and Mazen Ismail. The repository is a fork of the team project. I set the approach and owned classifier training and hyperparameter tuning; the dataset work, baseline and report were shared.