commit 5deb2deeb3507a39fca4ee17d29e45113302b93e
author David Davidović <david@davidovic.io> Sun Oct 20 00:51:46 2024 +0200
committer GitHub <noreply@github.com> Sat Oct 19 15:51:46 2024 -0700
tree 3ea8eea0f2455e0388579af6903dee424ba564c7
parent e320c993c4ee501b488dadb0a6bfe64f91ce7404

Drop shadow overhaul: improve correctness and performance (#2548)

## High-level summary

This PR introduces a large change to how drop shadows are rendered, introducing an `applyShadowsToLayers` flag which, by analogy to `applyOpacitiesToLayers`, allows layers to be treated as a whole for the purposes of drop shadows, improving the accuracy and bringing lottie-android in line with other renderers (lottie-web and lottie-ios).

Several different codepaths for different hardware/software combinations are introduced to ensure the fastest rendering available, even on legacy devices.

The calculation of shadow direction with respect to transforms is improved so that the output matches lottie-web and lottie-ios.

Image layers now cast shadows correctly thanks to a workaround to device-specific issues when combining `Paint.setShadowLayer()` and bitmap rendering.

Even in non-`applyShadowsToLayers` mode, correctness is improved by allowing the shadow-to-be-applied to propagate in a similar way as alpha. This allows some amount of visual fidelity to be recovered for animations or environments where enabling `applyShadowsToLayers` is not possible.

A number of issues that caused incorrect rendering in some other cases have been fixed.

## Background

### Drop shadows in Lottie

Lottie specifies drop shadows as a tuple of (angle, distance, radius, color, alpha), with each element being animatable.

The consensus behavior for the rendering of a layer with a drop shadow, which seems to be mostly respected in lottie-web and lottie-ios, seems to be:

1. Evaluate the values at the current frame for angle (`theta`), distance (`d`), radius (`r`), color with alpha (`C`).
2. Apply the layer transform and render the layer normally to a surface `So` (original layer).
3. Copy `So` to new surface `Ss` (shadow).
4. Apply a gaussian blur of radius `r' = c * r` to `Ss`, where `c` is some platform-specific constant intended to normalize blur implementations between platforms. (Ours is 0.33, lottie-web's is 0.25; see https://github.com/airbnb/lottie-android/pull/2541).
5. Tint `Ss` with the color and combine the alpha by applying the following for each pixel `P`: `P.rgb = C.rgb * P.a; P.a = C.a * P.a`.
6. Now the shadow is ready on `Ss`, and needs to be drawn into its final position.
7. Convert from polar coordinates `theta` and `d` into `dx` and `dy`, with the 0 position at 12 o'clock: `dx = d * cos(theta - pi/2); dy = d*sin(theta - pi/2)`.
8. Draw `Ss` onto `Si` (intermediate surface) with a translation of `(dx, dy)`.
9. Draw `So` (original layer) onto `Si` with identity transform.
10. Compose `Si` into the framebuffer using the layer's alpha and blend mode.

Some non-obvious consequences of the definition above:
- The angle, distance, and radius are relative to the layer post-transform, not pre-transform. That is, rotating the layer (via its transform) still keeps the same screen-space direction of the shadow, and scaling the layer (via its transform) still keeps the same screen-space shadow blur radius.
- The drop shadow is not based on any derived outline, so a layer's drop shadow can be seen through its non-fully-opaque pixels. At the same time, reducing the alpha of a pixel in a layer reduces its alpha in the drop shadow.
- A layer's shadow and the layer do not blend on top of each other on the final canvas in case the layer has a blend mode or alpha. Instead, the shadow and the layer are alpha-blended with each other, and the result is then composited onto the canvas.
  - In case the layer has a normal blend mode, this is equivalent to alpha-blending the layer's shadow and then the shadow onto the canvas separately.

### Drop shadows in lottie-android currently

lottie-android's current implementation of drop shadows differs in important ways:
1. **Shadows are applied per-shape.** This means that a case like a shape with both fill and stroke has incorrect shadows, since both the fill and the stroke render a separate shadow on top of each other.
2. **Precomp layer shadows are ignored.** This means that a precomp cannot cause any of its child shapes to cast a shadow. This is a consequence of the current implementation of (1).
3. **Image layers do not render correct shadows,** due to the minefield that is the support matrix (or in Android's case, a more apt name would be a support tensor) of Android's graphics stack - `setShadowLayer()` simply doesn't work for images consistently. (See the last image in https://github.com/airbnb/lottie-android/pull/2523#issue-2428578510.)

## Contributions of this PR

This PR introduces the following improvements and additions.

1. **Move the drop shadow model from individual content elements to layers,** and add some missing keypath callbacks. This is a prerequisite for handling drop shadows on a layer level.
2. **An `OffscreenLayer` implementation,** which serves as an abstraction that can replace `canvas.saveLayer()` for off-screen rendering and composition onto the final bitmap, but with the important distinction that it can also handle drop shadows, and possibly use hardware-accelerated `RenderNode`s and `RenderEffects` where available.
    - To use an `OffscreenLayer`, call its `.start()` method with a parent canvas and a `ComposeOp`, and draw on the *returned canvas.* Once finished, call `OffscreenLayer.finish()` to compose everything from the returned canvas to the parent canvas, applying alpha, blend mode, drop shadows, and color filters.
    - `OffscreenLayer` makes a dynamic decision on what to use for rendering - a no-op, forward to `.saveLayer()`, a HW-accelerated `RenderNode`, or a software bitmap, depending on the requested `ComposeOp` and hardware/SDK support.
    - The hope is that `OffscreenLayer` becomes a useful abstraction that can be extended to e.g. support hardware blurs, multiple drop shadows, or to support mattes in a hardware-accelerated fashion where possible. 
3. **The `applyShadowsToLayers` flag** which, by analogy to `applyOpacityToLayers`, turns on a more accurate mode that implements the drop shadow algorithm described above.
    - `OffscreenLayer` is used to apply alpha if `applyOpacityToLayers` is enabled, and to apply shadows if `applyShadowsToLayers` is enabled. The cost is paid only once if both alpha and drop shadows are present on a layer.
    - Not all `saveLayer()` calls in the code have been rewritten to use `OffscreenLayer` - the blast radius is minimized. `OffscreenLayer` is presently used only to apply alpha and drop shadows, and blend mode and color filters are still applied in `BaseLayer` using `saveLayer()` directly.
4. **More accurate shadow transformations.** Previously, the angle and distance were pre-transform, and only the radius was post-transform (contrary to step (2) of the algorithm). We correct this to match other renderers.
5. **More complete shadow handling even when `applyShadowsToLayers` is `false`:** we plumb the shadow through `.draw()` and `drawLayer()` calls similarly to alpha, and this allows us to render per-shape shadows on children of composition layers too.
6. ***Workaround for drop shadows on image layers.**
    - The workaround relies on `OffscreenLayer` as well, and image layers now render shadows properly in all cases.
7. **Fixes to a few subtle issues** causing incorrect rendering in other cases. (will be marked using PR comments, I might have forgotten some)

## Open questions

* **Should `applyShadowsToLayers` be `true` by default?** Some codepaths, such as when rendering purely via software, can be slow if shadow-casting layers are exceedingly large. But, the performance is still acceptable, and in the vast majority of cases everything is quite snappy.
* **Have I introduced any regressions?** The snapshot tests should answer this.
* **How does this perform on older devices?** `applyShadowsToLayers` plus an old device should trigger the purely-software shadow rendering mode. Simulating this in condition manually yields accurate results, and the performance seems surprisingly good, but it's unclear what will happen on a lower-end phone. There's also always the possibility of some device subtlety being missed. I don't have access to an older Android device.

## Testcases

These files now match between lottie-web and lottie-android:

[drop_shadow_comparator.json](https://github.com/user-attachments/files/16997070/drop_shadow_comparator.json)

[simple_shadow_casters_ll2.json](https://github.com/user-attachments/files/16997084/simple_shadow_casters_ll2.json)

The files from this earlier PR still all render the same: https://github.com/airbnb/lottie-android/pull/2523, with the exception of the fix for image layer bug, which fixes the rendering of the Map icon as mentioned in the comment of that PR.

This file has been used as a perf stress test with many <255 opacity precomps, some stacked inside each other, that must all be blended separately: [precomp_opacity_killer.json](https://github.com/user-attachments/files/16997261/precomp_opacity_killer.json)