A single first-model answer above 90% is ready for approval; the second call is skipped.
~/jessejaffe/work/labeling
all work
AI systems + computer vision / 2026
Labeling
I built a system that turns batches of dating photos into meticulous, model-ready data. It combines confidence-routed AI with objective computer vision, then uses sampled human quality assurance to validate accuracy and direct further optimization.
the problem
A preference model is only as perceptive as its labels.
A 1–5 attraction rating says which photo someone preferred, but not why. To learn a person's type, every rated image needs a consistent description of the face, body, clothing, pose, crop, setting, camera, and dozens of other visible details.
Manually reviewing every answer does not scale. The objective is to preserve the depth and accuracy of human labeling while requiring people only to confirm the system's accuracy. Humans review a small, representative sample across question types and confidence ranges to validate performance, detect drift, and reveal where the automation needs further optimization.
visual walkthrough
From a photo batch to a trustworthy feature set.
One workflow decides what AI can interpret and which software is best to measure specific photo aspects.
-
01
Upload once. Track every photo.
An admin uploads a batch and follows each photo from intake through AI processing, sampled quality assurance, approval, and export. Costs, retries, model runs, and validation state stay attached to the original filename.
- Individual and bulk photo intake
- Database-backed job state
- Traceable cost and retry ledger
Batch intake and per-photo labeling workspace · subject blurred for privacy -
02
Scale a structured question bank
The bank currently contains 67 questions covering facial hair, clothing, expression, lighting, body type, pose, background, camera properties, head size, height, and more. The same architecture can support hundreds or thousands of questions without turning each new label into a new manual workflow.
- Fixed options keep labels comparable
- Question history survives bank changes
- Approved rows export with original filenames
67-question answer matrix · horizontal scroll -
03
Make human review a validation layer
The current route asks Gemini 3.1 Flash Lite first and calls a second model below the confidence gate. Each recurring failure becomes an optimization problem until automated answers are accurate enough for humans to validate through a small, representative audit.
- Confidence controls provider escalation
- Samples span question families and confidence ranges
- Audit results reveal drift and optimization priorities
every active question Gemini Flash Lite first-pass label + confidence> 90% ready for approval≤ 90% second opiniongated subset Gemini Flash Preview resolve or confirm the first answerrepresentative sample quality assurance audit validate accuracy and identify what to optimize next -
04
Estimate height from the person alone
AI is not good at predicting a person's height when there is no reference object in the photo, and neither is the human eye. The system calculates height from the person's own proportions.
How it works: MediaPipe locates the crown, chin, shoulders, waist, knee, and ankle. Head size becomes the internal ruler. The system divides the visible torso and optional leg path by pose-corrected head height, creating a proportional height index without needing another person or object in the scene. PyFeat provides a fallback when MediaPipe cannot reliably read the head pose.
sample / torso tier Head-to-torso proportion path When the person is standing and the knees and ankles are visible, the system can also compare thigh length and lower-leg length with torso and head proportions to further estimate height without a reference. -
05
Find the least expensive model that is accurate enough
We started with frontier models, including GPT-5.5 and Claude Opus 4.8, and found that analyzing every aspect of every photo that way was too costly. We then compared previous-generation and lower-cost vision models, measuring image-label quality alongside confidence and token cost.
- Gemini performed best for this image-analysis workload
- Flash Lite handles the full first pass
- Flash Preview runs only when a second opinion is needed
started with frontier modelsmodels testedGemini Qwen Z.ai OpenAI Claudefirst pass Gemini 3.1 Flash Lite every active question→gated second pass Gemini 3 Flash Preview only when confidence calls for itcurrent cost ≈ $0.01 per photo analyzed · down from roughly $1
technical brief
README.md
A full-stack labeling pipeline built around calibrated automation, sampled quality assurance, traceable decisions, and the right measurement tool for each visual question.
01 / architecture
Every answer remains traceable from upload to export
A photo upload creates a durable labeling job. AI provider runs, prompt-level token usage, objective measurements, human responses, and the final label are stored separately, so an approved answer can be inspected without losing how it was produced.
01batch intakephotos + original filenames
→
02AI routinganswers + calibrated confidence
→
03objective visionlandmarks + geometry evidence
→
04sampled QArepresentative accuracy audit
→
05approved labelsmodel-ready CSV rows
02 / stack
Built across the full product surface
- frontend
- React, TypeScript, Vite
- API + data
- FastAPI, Python, PostgreSQL, SQLAlchemy
- background work
- Celery, Redis, durable provider jobs
- AI routing
- Google Gemini, Qwen, OpenAI, Z.ai experiments
- objective vision
- MediaPipe, PyFeat, YOLO, ONNX inference
- modeling output
- pandas, NumPy, scikit-learn, Elastic Net
03 / confidence routing
Confidence is a workflow decision, not decoration
The active route sends every question to Gemini 3.1 Flash Lite. Gemini Flash Preview runs only when the first model is at or below the 90% gate. If the two models disagree, Preview can resolve the answer above 80%; at 80% or below, a person reviews it. When both agree, their average must reach 80% to avoid review.
Only the uncertain subset spends tokens on the stronger second model.
Low-confidence agreement or unresolved disagreement stays reviewable while that question's automation is calibrated.
A small representative sample validates system-wide accuracy and identifies the next questions to optimize.
04 / objective vision
Separate visual judgment from visual geometry
Subjective or semantic questions are good fits for multimodal AI. Deterministic tools are better candidates for quantities that can be measured from landmarks, boxes, angles, silhouettes, or pixel ratios. Objective evidence is stored separately and calibrated before it is allowed to replace a label.
Top style, facial hair, expression, setting, activity, and other contextual questions.
Head area, crop percentages, pose landmarks, body angles, face geometry, and proportional paths.
MediaPipe provides face, pose, and gesture landmarks. PyFeat supplies a 68-landmark face and head-pose fallback. Additional detectors and segmentation models contribute evidence for people, objects, clothing regions, and scene geometry.
05 / height calibration
Infer apparent height from proportions, not pixel height
Raw pixel height mostly describes crop and camera distance. The calibration path instead divides visible body-path lengths by an internal reference: pose-corrected head height or torso length. That makes the resulting ratios largely distance-invariant.
crown → chin shoulder → hip → knee → ankle
↓ ↓
3D head-pose correction visible body-path length
└──────────────┬──────────────┘
↓
body path ÷ head or torso
↓
compare within one visibility tier
torso-only · knees-visible · full-bodyCompatible scores are calibrated only against human-reviewed examples. Head pitch, yaw, and roll are corrected with MediaPipe first and PyFeat as a fallback. If landmarks, pose, or visibility are unreliable, the system warns the admin or excludes the sample from automatic classification instead of pretending the measurement is certain.
06 / cost optimization
Engineer the token path, not just the model choice
Once model quality was strong enough, the remaining cost work happened below the routing layer. The system makes repeated context cheaper, shortens expensive output, and verifies provider-reported usage instead of assuming a cache hit.
The full question bank and detailed label definitions are stored as reusable cached context. Each photo adds only the image and its small subset instruction.
The cache lifetime is estimated from the batch size at 125 seconds per photo and capped at 30 minutes, keeping context warm without paying to store it longer than the run.
Eligible Gemini calls use Flex at roughly half the standard execution price, with bounded retries and a standard-priority fallback so savings do not strand a batch.
Rich label guidance stays in the input; the output returns only the exact label and confidence. For implicit caches, reported cached-input tokens are logged and tested rather than assumed.
Together, caching and Flex can reduce the effective rate for repeat-heavy calls to roughly one-fifth or one-tenth of a standard uncached path. This is a transferable AI operations capability: measure actual usage, shape prompts for reuse, and buy only the output the product needs.
Privacy is a separate design constraint. This project did not require a privacy-specific architecture, but when one does, I can apply the same evaluation discipline to provider retention, regional data handling, private networking, and local or self-hosted inference.
07 / downstream modeling
Labels explain the rating and make it predictive
Each approved photo becomes a structured feature row. When those features are joined to a person's 1–5 ratings, a cross-validated Elastic Net model can identify the photo aspects with the strongest positive and negative relationship to that person's preferences.
approved photo labels + 1–5 ratings
↓
standardized feature matrix
↓
ElasticNetCV
↓
impactful likes, dislikes, and coefficients
↓
predictions for new candidate photosMore detailed and accurate labels produce deeper explanations for the rater and stronger signals for Matchmaker Ratings + Ranking.