A single first-model answer above 90% is ready for approval; the second call is skipped.
~/jessejaffe/work/labeling
all work
AI systems + computer vision / 2026
Labeling
I built a photo-labeling system combining confidence-routed AI, computer vision, and sampled human validation.
the problem
Preference models fail when photo labels lack depth or accuracy.
A rating shows which photo won, not which facial, body, clothing, pose, or camera details mattered.
Automated labeling provides depth at scale; representative human audits validate accuracy and expose drift.
visual walkthrough
From a photo batch to a trustworthy feature set.
AI interprets what it can; specialized software measures the rest.
-
01
Upload once. Track every photo.
A batch keeps each photo's processing, review, cost, retry, and export history together.
Batch intake and per-photo labeling workspace · subject blurred for privacy -
02
Scale a structured question bank
A 67-question bank produces comparable labels while preserving question history and original filenames.
67-question answer matrix · horizontal scroll -
03
Make human review a validation layer
Flash Lite labels everything; low-confidence answers escalate, and sampled human audits reveal drift.
every active question Gemini Flash Lite first-pass label + confidence> 90% ready for approval≤ 90% second opiniongated subset Gemini Flash Preview resolve or confirm the first answerrepresentative sample quality assurance audit validate accuracy and identify what to optimize next -
04
Estimate height from the person alone
Height is estimated from body proportions when no real-world reference appears in the photo.
Method: MediaPipe landmarks use head size as an internal ruler; PyFeat handles uncertain head pose.
sample / torso tier Head-to-torso proportion path Visible legs add thigh and lower-leg proportions. -
05
Use the least expensive accurate model
Testing found a confidence-gated Gemini route that preserved accuracy while cutting cost from roughly $1 to $0.01 per photo.
started with frontier modelsmodels testedGemini Qwen Z.ai OpenAI Claudefirst pass Gemini 3.1 Flash Lite every active question→gated second pass Gemini 3 Flash Preview only when confidence calls for itcurrent cost ≈ $0.01 per photo analyzed · down from roughly $1
technical brief
README.md
A full-stack labeling pipeline built around calibrated automation, sampled quality assurance, traceable decisions, and the right measurement tool for each visual question.
01 / architecture
Every answer remains traceable from upload to export
A photo upload creates a durable labeling job. AI provider runs, prompt-level token usage, objective measurements, human responses, and the final label are stored separately, so an approved answer can be inspected without losing how it was produced.
01batch intakephotos + original filenames
→
02AI routinganswers + calibrated confidence
→
03objective visionlandmarks + geometry evidence
→
04sampled QArepresentative accuracy audit
→
05approved labelsmodel-ready CSV rows
02 / stack
Built across the full product surface
- frontend
- React, TypeScript, Vite
- API + data
- FastAPI, Python, PostgreSQL, SQLAlchemy
- background work
- Celery, Redis, durable provider jobs
- AI routing
- Google Gemini, Qwen, OpenAI, Z.ai experiments
- objective vision
- MediaPipe, PyFeat, YOLO, ONNX inference
- modeling output
- pandas, NumPy, scikit-learn, Elastic Net
03 / confidence routing
Confidence is a workflow decision, not decoration
The active route sends every question to Gemini 3.1 Flash Lite. Gemini Flash Preview runs only when the first model is at or below the 90% gate. If the two models disagree, Preview can resolve the answer above 80%; at 80% or below, a person reviews it. When both agree, their average must reach 80% to avoid review.
Only the uncertain subset spends tokens on the stronger second model.
Low-confidence agreement or unresolved disagreement stays reviewable while that question's automation is calibrated.
A small representative sample validates system-wide accuracy and identifies the next questions to optimize.
04 / objective vision
Separate visual judgment from visual geometry
Subjective or semantic questions are good fits for multimodal AI. Deterministic tools are better candidates for quantities that can be measured from landmarks, boxes, angles, silhouettes, or pixel ratios. Objective evidence is stored separately and calibrated before it is allowed to replace a label.
Top style, facial hair, expression, setting, activity, and other contextual questions.
Head area, crop percentages, pose landmarks, body angles, face geometry, and proportional paths.
MediaPipe provides face, pose, and gesture landmarks. PyFeat supplies a 68-landmark face and head-pose fallback. Additional detectors and segmentation models contribute evidence for people, objects, clothing regions, and scene geometry.
05 / height calibration
Infer apparent height from proportions, not pixel height
Raw pixel height mostly describes crop and camera distance. The calibration path instead divides visible body-path lengths by an internal reference: pose-corrected head height or torso length. That makes the resulting ratios largely distance-invariant.
crown → chin shoulder → hip → knee → ankle
↓ ↓
3D head-pose correction visible body-path length
└──────────────┬──────────────┘
↓
body path ÷ head or torso
↓
compare within one visibility tier
torso-only · knees-visible · full-bodyCompatible scores are calibrated only against human-reviewed examples. Head pitch, yaw, and roll are corrected with MediaPipe first and PyFeat as a fallback. If landmarks, pose, or visibility are unreliable, the system warns the admin or excludes the sample from automatic classification instead of pretending the measurement is certain.
06 / cost optimization
Engineer the token path, not just the model choice
Once model quality was strong enough, the remaining cost work happened below the routing layer. The system makes repeated context cheaper, shortens expensive output, and verifies provider-reported usage instead of assuming a cache hit.
The full question bank and detailed label definitions are stored as reusable cached context. Each photo adds only the image and its small subset instruction.
The cache lifetime is estimated from the batch size at 125 seconds per photo and capped at 30 minutes, keeping context warm without paying to store it longer than the run.
Eligible Gemini calls use Flex at roughly half the standard execution price, with bounded retries and a standard-priority fallback so savings do not strand a batch.
Rich label guidance stays in the input; the output returns only the exact label and confidence. For implicit caches, reported cached-input tokens are logged and tested rather than assumed.
Together, caching and Flex can reduce the effective rate for repeat-heavy calls to roughly one-fifth or one-tenth of a standard uncached path. This is a transferable AI operations capability: measure actual usage, shape prompts for reuse, and buy only the output the product needs.
Privacy is a separate design constraint. This project did not require a privacy-specific architecture, but when one does, I can apply the same evaluation discipline to provider retention, regional data handling, private networking, and local or self-hosted inference.
07 / downstream modeling
Labels explain the rating and make it predictive
Each approved photo becomes a structured feature row. When those features are joined to a person's 1–5 ratings, a cross-validated Elastic Net model can identify the photo aspects with the strongest positive and negative relationship to that person's preferences.
approved photo labels + 1–5 ratings
↓
standardized feature matrix
↓
ElasticNetCV
↓
impactful likes, dislikes, and coefficients
↓
predictions for new candidate photosMore detailed and accurate labels produce deeper explanations for the rater and stronger signals for Matchmaker Ratings + Ranking.