Image Processing

How Scanly transforms raw captures into corrected, filtered document pages.

Pipeline overview

Raw JPEG (camera or import)
  → EXIF rotation correction
  → Optional user rotation
  → Optional physical-document semantic gate (capture/import only)
  → Corner detection (LiteRT post model + book-aware resolve) OR stored/manual crop quad
  → Perspective warp (`PerspectiveBitmapTransform`)
  → Decode/downscale to max 2800 px longest edge (processing path)
  → Filter preset + intensity (`OpenCvPageFilterProcessor` → `PageFilterEngine`)
  → Optional post-filter adjustments (`PageFilterAdjustmentsApplier`)
  → Processed JPEG + thumbnail
  → Room record update

Implemented via PageImageProcessor (domain/processing/) with implementation in data/processing/DefaultPageImageProcessor.

Entry points

All methods that read images start from the raw capture. reprocessPage never modifies the raw file.

Step 1: Rotation

1. Read EXIF orientation from raw JPEG (ExifInterface).
2. Apply EXIF correction.
3. Apply user rotationDegrees (90° increments) from crop/editor state.

Step 2: Semantic gate and corner detection

When no crop quad is supplied (or detect-when-missing is requested):

Model assets (assets/models/)

Keep noCompress += "tflite" in Gradle. NDK ABI filters: arm64-v8a, armeabi-v7a.

Model config

• Accurate uses YOLO-pose output; Lite/Standard/High use TL/TR/BR/BL regression plus a presence score, RGB [-1, 1] input, and RGB-114 letterboxing
• Live-preview and post-processing model choices are independent DataStore settings
• Automatic selection benchmarks Lite, Standard, and High on-device, then chooses under separate latency budgets for live vs post
• Semantic gate: live and post thresholds differ; rejected frames skip corner inference on the camera path
• Book captures use gutter analysis via BookAwareCornerResolver

Crop screen AI Detect

DetectDocumentCornersUseCase → PageImageProcessor.detectDocumentCorners(raw, rotation):

1. Decode raw, apply EXIF + current user rotation
2. Run the same still-image detection path as finalize (BookAwareCornerResolver + raw-model fallback)
3. Return a normalized DocumentCornerQuad into crop UI state (user still taps Done to reprocess)

Model benchmark (settings/model-benchmark)

Settings exposes a temporary benchmark screen for gate + all corner models on selected local images.

Step 3: Perspective warp

Maps the detected (or manual) quad to a flat rectangular output image.

Step 4: Filter engine

OpenCvPageFilterProcessor is the public facade. Internals live in core/processing/filter/. The paper presets render with one reflectance model, and Screen and Vivid use global color looks instead (see Global color looks):

Image ≈ Reflectance × Shading

Shading (uneven light across the paper) is estimated and divided out, so paper goes to white while ink and colour keep their relative values. The pipeline:

Cropped page
  → Analyze at canonical 720 px (PageImageAnalyzer → PageImageProfile, incl. tone stats)
  → Auto routing (AdaptivePageFilterTuning.automatic) when the preset is Auto
  → Look parameters (PageFilterLook.resolve, scaled by intensity)
  → Paper-aware background model at 360 px (PageBackgroundModel)
  → Divide shading out (per RGB channel for colour looks; gray for Grayscale, Faded Text and B&W)
  → One tone curve: black point / white point / gamma (PageFilterOperators.toneCurve)
  → Colour looks: vibrance + paper neutralization
  → Unsharp mask with noise damping   |   B&W: anti-aliased global threshold + despeckle

Screen and Vivid skip the background model and division entirely; they go straight to their global pipelines.

Paper-aware background model

PageBackgroundModel estimates the shading the page was lit with:

1. Fixed working size. The page is reduced to a 360 px longest edge before estimation, so the same page gets the same shading at chip, preview and save sizes.
2. Stroke removal. A morphological close (ellipse) removes ink strokes up to strokeFraction of the edge (3 % for most looks, 4.5 % for Whiteboard), followed by a small median blur.
3. Paper mask. A pixel counts as paper only if it is bright relative to the page's paper level, close in colour to the paper reference (low chromaticity distance), and smooth (low local texture). Photos, logos and solid fills usually fail one of these tests.
4. Multi-scale fill. Background under non-paper pixels is filled from surrounding paper with normalized convolution at four scales (3 %, 8 %, 20 % and 50 % of the edge). Where no paper is nearby it falls back to the paper reference. Photos and logos keep their own content instead of being flattened to white.

The paper reference is the median RGB of the brightest quarter of the page.

Division and gain floor

Each channel is normalized as out = src × target / background. The background is floored at reference / maxGain (for example maxGain 4.0 for Magic Color) so very dark regions are not amplified without limit.

Where the local gain is above about 1.6× (deep shadow), the source is blended toward a smoothed copy before division. This suppresses the sensor noise that would otherwise be amplified in shadows. The smoothing strength is denoiseSigma, which scales with the page size.

Render modes

Tone curve

A single LUT is built from three values: black point, white point and gamma. Input is mapped to ((v − black) / (white − black))^gamma, clipped to 0…255. Gamma above 1 darkens mid-tones (ink) while paper stays clipped to white.

The points come from tone stats measured on the normalized page at 720 px (PageImageAnalyzer.measureTone):

Because the stats are measured on the normalized page, the curve lands on what the ink and paper will actually look like after shading is removed.

Intensity

Each page stores an intensity value from 0 to 1.5 (UI 0–150 %, default 1):

Intensity scales the tonal parameters (black point, white point, gamma, vibrance, sharpening). For B&W, it moves the threshold instead: a higher intensity raises the threshold, so more faint marks are kept as ink. B&W still binarizes at intensity 0. Original ignores intensity. Screen and Vivid scale differently; see Global color looks.

Resolution independence

Tonal parameters do not depend on the output size. They come from the 360 px background model and the 720 px analysis profile, so picker chips (320 px), the live editor preview (~1600 px) and the saved JPEG (up to 2800 px) produce the same look. Only pixel-level kernels scale with the longest edge, against a 1600 px reference edge: sharpening sigma, shadow-denoise sigma and the despeckle area.

Global color looks (Screen, Vivid)

The paper presets assume a sheet of paper under uneven light: they find the paper, divide out the shading and push it to white. That model does not fit some captures, so Screen and Vivid skip it and correct the whole image instead (PageColorLooks):

• Emissive light. A display is its own light source, not paper under a lamp.
• Dark mode. On a dark page, the brightest-quarter paper reference is content, so paper detection would flatten the page.
• Full-screen images. A photo that fills the screen has no paper to find.
• Pixel grid and moiré. The display's pixel lattice beats against the camera sensor and shows as fine stripes and colour bands.
• Reflection veil. Room light reflected off the glass adds an uneven, often tinted haze.
• Screen white point. A display's white is rarely neutral. It casts its own colour, which camera white balance does not remove.

Both looks measure their statistics on a 360 px copy of the page, the same working size as the background model, so chip, preview and saved JPEG agree. Pixel kernels (blur, sharpen) scale with the render size against the 1600 px reference edge, as they do for paper looks.

Screen pipeline

1. De-screen. The page is converted to YCrCb. Luma gets a Gaussian blur just wide enough to dissolve the pixel grid (sigma about 0.8 px at a 1600 px render). Chroma is resampled to a quarter resolution, blurred (sigma 2 px there), and resampled back, which removes colour moiré. Text edges ride on luma, so they stay sharp.
2. Dark-mode veil removal. Runs only when the page's median luminance is below 100. Each channel's local minimum over about 8 % of the long edge (erode, then blur at 5 % of the edge) estimates the veil. A floor at 35 % of the veil's 5th-percentile luminance is removed from it, and the result is subtracted from the page at full resolution, scaled by intensity. Light-background screens skip this step, because their local minimum is content.
3. White reference. Each channel's white is the 80th percentile of the brightest 1 % of low-saturation pixels (saturation below 25 %). If fewer than 30 such pixels exist, the brightest 1 % of all pixels is used. The dark level is the 0.5th luminance percentile.
4. Levels. Per channel, the dark level (scaled by black clip) maps to black and the white reference maps to 248. White-balance gains are limited to ±15 % so colourful content cannot be pushed into a false cast.
5. S-curve. Mid-tone contrast with 0 and 255 fixed.
6. Vibrance. Without paper neutralization, so coloured content keeps its colour.
7. Sharpen. Unsharp mask.

Vivid pipeline

1. Bounded white balance and levels. The same white reference, ±15 % gains and levels as Screen. White balance is 60 % of full at intensity 100 %, and black clip is 80 % × intensity, capped at 100 %.
2. Local contrast. CLAHE on Lab lightness with a fixed 8×8 grid (so the look does not depend on render size) and a clip limit of 2. The result is blended with the original lightness at 45 % × intensity, capped at 65 %.
3. S-curve, then vibrance (without paper neutralization), then sharpen. Sharpening is damped on noisy captures.

Intensity

• Screen. De-screening stays on even at intensity 0, because the moiré grid is never wanted. Veil removal, white balance and black clip scale with intensity and stop rising at 100 %. The S-curve, vibrance and sharpening keep scaling up to 150 %. At 0, only de-screening and a plain brightness stretch remain.
• Vivid. At intensity 0, only the brightness stretch remains; local contrast, S-curve, vibrance and sharpening are all off. White balance and black clip stop rising at 100 %.

Presets

Auto routing

AdaptivePageFilterTuning.automatic uses the page profile. Checks run in this order:

The Screen check reads PageImageProfile.screenLikely. DefaultPageImageProcessor sets it only when all of these hold: the document gate is enabled in Settings, the page is captured or imported with no crop quad (processCapture), and the gate's top class is DIGITAL_SCREEN with probability at least 0.6 (SCREEN_HINT_THRESHOLD). Editor saves go through reprocessPage with a crop quad, so they do not set the hint.

Legacy presets

Presets removed in the filter overhaul are remapped to their closest replacement. The Room migration (4 → 5) rewrites stored values, and PageFilterPreset.fromStorage maps them on read (for example from an older .scanly backup). Unknown values fall back to Original.

Picker chips

The filter picker thumbnails use OpenCvPageFilterProcessor.applyAll: one shared profile, every preset rendered at intensity 1.0 (the designed look). The live preview and the saved JPEG use the page's own intensity.

Step 5: Post-filter adjustments

After the preset, PageFilterAdjustmentsApplier applies per-page tweaks persisted on ScanPage:

Brightness and contrast operate on RGB only. The alpha channel is never scaled, so pixels can no longer become semi-transparent. Contrast pivots on mid-gray; earlier builds pivoted on black.

Identity (all tonal fields zero) skips extra work. Editor Adjust edits these; bulk “apply filter to all pages” changes only the preset, not other pages’ intensity or adjustments.

Step 6: Output

Paths under processed/ and thumbs/ per document. ThumbnailCache is invalidated after reprocess. Camera capture resolution is covered in capture-and-scan.md.

Failure handling

Editor integration

Page editor (editor/page/{pageId})

• Main surface: live cropped + filtered + adjusted preview (same pipeline as reprocess, preview-sized)
• Toolbar: Crop · Filters · Adjust · Retake · Delete
• Filters → full-screen FilterPickerScreen (large live preview + preset chips + apply-to-all)
• Adjust → full-screen FilterCustomizeScreen (Intensity first, then brightness, contrast, saturation, sharpness; hold Compare for filter-only; scrollbar)
• Done on the editor saves filter, intensity and adjustments (with latest crop/rotation from storage)

Crop screen (crop/page/{pageId})

• Full-frame preview with four-point handles (CropQuadEditor)
• AI Detect · Left · Right · Reset · Done
• Done → UpdatePageEditsUseCase with crop + rotation; keeps current filter, intensity + adjustments

Shared support

Core processing files

Related docs

• capture-and-scan.md — camera session, capture resolution and finalize trigger
• export.md — uses processed images for PDF/ZIP
• ../data/file-storage.md — output file layout
• ../architecture/screens.md — editor / crop ViewModels