github/flutter
7
0
Fork 0
mirror of https://github.com/flutter/flutter.git synced 2025-06-03 00:51:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
9af4d746df
flutter/examples/api/lib
History
Ashish Beck 9af4d746df
Added semanticsIdentifier to Text Widgets (#163843) 
This PR aims to add `semanticsIdentifier` to `Text` and some of its
internal objects to pass the semantics information for adding identifier
to the semantics nodes

From the issue filed at #163842, the following is a description of the
problem.

The [semantics
identifier](https://api.flutter.dev/flutter/semantics/SemanticsData/identifier.html)
helps in uniquely identifying elements using UI automation tools like
Appium, UIAutomator and XCUITests by setting identifiers that the screen
readers cannot see but the said tools can. This is especially useful
when working with a multi-lingual or multi-tenant app, where the element
IDs need to be unique but the content can be different. The `Semantics`
widget already has support for declaring it. However, the `Text` and
`Text.rich` variants only support setting `semanticsLabel` without
explicitly setting the identifiers. The widgets themselves can be
wrapped with a `Semantics` widget but it still does not cater for a rich
text that can have multiple text spans, each containing unique lables
and identifiers, and optionally gesture detectors for handling links.

Consider the following UI for two different tenants:
<img width="229" alt="Image"
src="https://github.com/user-attachments/assets/e8a24588-d94d-42fc-ba6c-ce39959207ae"
/>

Here, both the tenants utilise different strings to convey the same
message. The structure of the message stays the same so the identifiers
help in unifying the element identification process across the tenant
apps in the automation tools without having to write another script for
every other tenant.
Without the identifiers, the automation scripts require a rewrite per
tenant to be able to successfully locate the element and even tap on the
hyperlink.

# With PR Changes
## Appium Views
For the given sample code,
<details><summary>Text.rich Sample</summary>

```dart
Text.rich(
  TextSpan(
    text: 'This text contains both identifier and label.',
    semanticsLabel: 'Custom label',
    semanticsIdentifier: 'Custom identifier',
    style: customStyle1,
    children: <TextSpan>[
      TextSpan(
        text: ' While this one contains only label',
        semanticsLabel: 'Hello world',
        style: customStyle2,
      ),
      const TextSpan(
        text: ' and this contains only identifier,',
        semanticsIdentifier: 'Hello to the automation tool',
      ),
      TextSpan(
        text: ' this text contains neither identifier nor label.',
        style: customStyle2,
      ),
    ],
  ),
),
```
</details>
we have the following results with and without the PR code changes:

### With Identifier

![image](https://github.com/user-attachments/assets/abad3b36-61a5-41d9-b269-9977ac6d26e7)
### Without Identifier Changes

![image](https://github.com/user-attachments/assets/91d01be9-d39c-4c65-9251-570284108bfd)


## Semantics Tree Dump
The followings are the semantics tree dump for both the cases
<details><summary>With Identifier</summary>

```
I/flutter ( 8185): SemanticsNode#0
I/flutter ( 8185):  │ Rect.fromLTRB(0.0, 0.0, 1080.0, 2154.0)
I/flutter ( 8185):  │
I/flutter ( 8185):  └─SemanticsNode#1
I/flutter ( 8185):    │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3) scaled by 2.8x
I/flutter ( 8185):    │ textDirection: ltr
I/flutter ( 8185):    │
I/flutter ( 8185):    └─SemanticsNode#2
I/flutter ( 8185):      │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3)
I/flutter ( 8185):      │ sortKey: OrdinalSortKey#9e46a(order: 0.0)
I/flutter ( 8185):      │
I/flutter ( 8185):      └─SemanticsNode#3
I/flutter ( 8185):        │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3)
I/flutter ( 8185):        │ flags: scopesRoute
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#4
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 40.0, 376.7, 88.0)
I/flutter ( 8185):        │   label: "Demonstration of automation tools support in Semantics
I/flutter ( 8185):        │     for Text and RichText"
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#5
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 104.0, 376.7, 204.0)
I/flutter ( 8185):        │   label: "The identifier property in Semantics widget is used for
I/flutter ( 8185):        │     UI testing with tools that work by querying the native
I/flutter ( 8185):        │     accessibility, like UIAutomator, XCUITest, or Appium. It can be
I/flutter ( 8185):        │     matched with CommonFinders.bySemanticsIdentifier."
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#6
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 220.0, 121.9, 244.0)
I/flutter ( 8185):        │   label: "Text Example:"
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#7
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 244.0, 376.7, 304.0)
I/flutter ( 8185):        │   identifier: "This is a custom identifier that only the automation
I/flutter ( 8185):        │     tools are able to see"
I/flutter ( 8185):        │   label: "This is a custom label"
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#8
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 320.0, 155.1, 344.0)
I/flutter ( 8185):        │   label: "Text.rich Example:"
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#9
I/flutter ( 8185):        │ │ Rect.fromLTRB(16.0, 344.0, 376.7, 400.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#10
I/flutter ( 8185):        │ │   Rect.fromLTRB(-4.0, -3.0, 280.0, 23.0)
I/flutter ( 8185):        │ │   identifier: "Custom identifier"
I/flutter ( 8185):        │ │   label: "Custom label"
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#06bc7(order: 0.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#11
I/flutter ( 8185):        │ │   Rect.fromLTRB(-4.0, -1.0, 345.0, 42.0)
I/flutter ( 8185):        │ │   label: "Hello world"
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#32a12(order: 1.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#12
I/flutter ( 8185):        │ │   Rect.fromLTRB(130.0, 17.0, 348.0, 43.0)
I/flutter ( 8185):        │ │   identifier: "Hello to the automation tool"
I/flutter ( 8185):        │ │   label: " and this contains only identifier,"
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#49d25(order: 2.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ └─SemanticsNode#13
I/flutter ( 8185):        │     Rect.fromLTRB(-4.0, 19.0, 351.0, 60.0)
I/flutter ( 8185):        │     label: " this text contains neither identifier nor label."
I/flutter ( 8185):        │     textDirection: ltr
I/flutter ( 8185):        │     sortKey: OrdinalSortKey#f3624(order: 3.0)
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#14
I/flutter ( 8185):        │   Rect.fromLTRB(16.0, 416.0, 181.0, 440.0)
I/flutter ( 8185):        │   label: "Multi-tenant Example:"
I/flutter ( 8185):        │   textDirection: ltr
I/flutter ( 8185):        │
I/flutter ( 8185):        ├─SemanticsNode#15
I/flutter ( 8185):        │ │ Rect.fromLTRB(108.3, 440.0, 284.5, 480.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#16
I/flutter ( 8185):        │ │   Rect.fromLTRB(-1.0, -3.0, 115.0, 23.0)
I/flutter ( 8185):        │ │   identifier: "please_open"
I/flutter ( 8185):        │ │   label: "Please open the "
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#ea831(order: 0.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#17
I/flutter ( 8185):        │ │   Rect.fromLTRB(106.0, -3.0, 177.0, 23.0)
I/flutter ( 8185):        │ │   identifier: "product_name"
I/flutter ( 8185):        │ │   label: "product 1"
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#589fe(order: 1.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ ├─SemanticsNode#18
I/flutter ( 8185):        │ │   Rect.fromLTRB(-4.0, -3.0, 177.0, 43.0)
I/flutter ( 8185):        │ │   identifier: "to_use_app"
I/flutter ( 8185):        │ │   label:
I/flutter ( 8185):        │ │     "
I/flutter ( 8185):        │ │     to use this app."
I/flutter ( 8185):        │ │   textDirection: ltr
I/flutter ( 8185):        │ │   sortKey: OrdinalSortKey#c2762(order: 2.0)
I/flutter ( 8185):        │ │
I/flutter ( 8185):        │ └─SemanticsNode#19
I/flutter ( 8185):        │     Rect.fromLTRB(95.0, 17.0, 181.0, 43.0)
I/flutter ( 8185):        │     actions: tap
I/flutter ( 8185):        │     flags: isLink
I/flutter ( 8185):        │     identifier: "learn_more_link"
I/flutter ( 8185):        │     label: " Learn more"
I/flutter ( 8185):        │     textDirection: ltr
I/flutter ( 8185):        │     sortKey: OrdinalSortKey#7d560(order: 3.0)
I/flutter ( 8185):        │
I/flutter ( 8185):        └─SemanticsNode#20
I/flutter ( 8185):          │ Rect.fromLTRB(97.0, 496.0, 295.7, 536.0)
I/flutter ( 8185):          │
I/flutter ( 8185):          ├─SemanticsNode#21
I/flutter ( 8185):          │   Rect.fromLTRB(11.0, -3.0, 127.0, 23.0)
I/flutter ( 8185):          │   identifier: "please_open"
I/flutter ( 8185):          │   label: "Please open the "
I/flutter ( 8185):          │   textDirection: ltr
I/flutter ( 8185):          │   sortKey: OrdinalSortKey#7bb57(order: 0.0)
I/flutter ( 8185):          │
I/flutter ( 8185):          ├─SemanticsNode#22
I/flutter ( 8185):          │   Rect.fromLTRB(118.0, -3.0, 188.0, 23.0)
I/flutter ( 8185):          │   identifier: "product_name"
I/flutter ( 8185):          │   label: "product 2"
I/flutter ( 8185):          │   textDirection: ltr
I/flutter ( 8185):          │   sortKey: OrdinalSortKey#6c7c6(order: 1.0)
I/flutter ( 8185):          │
I/flutter ( 8185):          ├─SemanticsNode#23
I/flutter ( 8185):          │   Rect.fromLTRB(-4.0, -3.0, 188.0, 43.0)
I/flutter ( 8185):          │   identifier: "to_use_app"
I/flutter ( 8185):          │   label:
I/flutter ( 8185):          │     "
I/flutter ( 8185):          │     to access this app."
I/flutter ( 8185):          │   textDirection: ltr
I/flutter ( 8185):          │   sortKey: OrdinalSortKey#1e8e7(order: 2.0)
I/flutter ( 8185):          │
I/flutter ( 8185):          └─SemanticsNode#24
I/flutter ( 8185):              Rect.fromLTRB(117.0, 17.0, 203.0, 43.0)
I/flutter ( 8185):              actions: tap
I/flutter ( 8185):              flags: isLink
I/flutter ( 8185):              identifier: "learn_more_link"
I/flutter ( 8185):              label: " Find out more"
I/flutter ( 8185):              textDirection: ltr
I/flutter ( 8185):              sortKey: OrdinalSortKey#db7e6(order: 3.0)
```

</details>
<details><summary>Without Identifier Changes</summary>

```
I/flutter (18659): SemanticsNode#0
I/flutter (18659):  │ Rect.fromLTRB(0.0, 0.0, 1080.0, 2154.0)
I/flutter (18659):  │
I/flutter (18659):  └─SemanticsNode#1
I/flutter (18659):    │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3) scaled by 2.8x
I/flutter (18659):    │ textDirection: ltr
I/flutter (18659):    │
I/flutter (18659):    └─SemanticsNode#2
I/flutter (18659):      │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3)
I/flutter (18659):      │ sortKey: OrdinalSortKey#102d4(order: 0.0)
I/flutter (18659):      │
I/flutter (18659):      └─SemanticsNode#3
I/flutter (18659):        │ Rect.fromLTRB(0.0, 0.0, 392.7, 783.3)
I/flutter (18659):        │ flags: scopesRoute
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#4
I/flutter (18659):        │   Rect.fromLTRB(16.0, 40.0, 376.7, 88.0)
I/flutter (18659):        │   label: "Demonstration of automation tools support in Semantics
I/flutter (18659):        │     for Text and RichText"
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#5
I/flutter (18659):        │   Rect.fromLTRB(16.0, 104.0, 376.7, 204.0)
I/flutter (18659):        │   label: "The identifier property in Semantics widget is used for
I/flutter (18659):        │     UI testing with tools that work by querying the native
I/flutter (18659):        │     accessibility, like UIAutomator, XCUITest, or Appium. It can be
I/flutter (18659):        │     matched with CommonFinders.bySemanticsIdentifier."
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#6
I/flutter (18659):        │   Rect.fromLTRB(16.0, 220.0, 121.9, 244.0)
I/flutter (18659):        │   label: "Text Example:"
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#7
I/flutter (18659):        │   Rect.fromLTRB(16.0, 244.0, 376.7, 304.0)
I/flutter (18659):        │   label: "This is a custom label"
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#8
I/flutter (18659):        │   Rect.fromLTRB(16.0, 320.0, 155.1, 344.0)
I/flutter (18659):        │   label: "Text.rich Example:"
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#9
I/flutter (18659):        │   Rect.fromLTRB(16.0, 344.0, 376.7, 400.0)
I/flutter (18659):        │   label: "Custom labelHello world and this contains only
I/flutter (18659):        │     identifier, this text contains neither identifier nor label."
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#10
I/flutter (18659):        │   Rect.fromLTRB(16.0, 416.0, 181.0, 440.0)
I/flutter (18659):        │   label: "Multi-tenant Example:"
I/flutter (18659):        │   textDirection: ltr
I/flutter (18659):        │
I/flutter (18659):        ├─SemanticsNode#11
I/flutter (18659):        │ │ Rect.fromLTRB(108.3, 456.0, 284.5, 496.0)
I/flutter (18659):        │ │
I/flutter (18659):        │ ├─SemanticsNode#12
I/flutter (18659):        │ │   Rect.fromLTRB(-4.0, -3.0, 177.0, 43.0)
I/flutter (18659):        │ │   label:
I/flutter (18659):        │ │     "Please open the product 1
I/flutter (18659):        │ │     to use this app."
I/flutter (18659):        │ │   textDirection: ltr
I/flutter (18659):        │ │   sortKey: OrdinalSortKey#493fc(order: 0.0)
I/flutter (18659):        │ │
I/flutter (18659):        │ └─SemanticsNode#13
I/flutter (18659):        │     Rect.fromLTRB(95.0, 17.0, 181.0, 43.0)
I/flutter (18659):        │     actions: tap
I/flutter (18659):        │     flags: isLink
I/flutter (18659):        │     label: " Learn more"
I/flutter (18659):        │     textDirection: ltr
I/flutter (18659):        │     sortKey: OrdinalSortKey#587bf(order: 1.0)
I/flutter (18659):        │
I/flutter (18659):        └─SemanticsNode#14
I/flutter (18659):          │ Rect.fromLTRB(88.9, 512.0, 303.8, 552.0)
I/flutter (18659):          │
I/flutter (18659):          ├─SemanticsNode#15
I/flutter (18659):          │   Rect.fromLTRB(-4.0, -3.0, 196.0, 43.0)
I/flutter (18659):          │   label:
I/flutter (18659):          │     "Please open the product 2
I/flutter (18659):          │     to access this app."
I/flutter (18659):          │   textDirection: ltr
I/flutter (18659):          │   sortKey: OrdinalSortKey#69083(order: 0.0)
I/flutter (18659):          │
I/flutter (18659):          └─SemanticsNode#16
I/flutter (18659):              Rect.fromLTRB(117.0, 17.0, 219.0, 43.0)
I/flutter (18659):              actions: tap
I/flutter (18659):              flags: isLink
I/flutter (18659):              label: " Find out more"
I/flutter (18659):              textDirection: ltr
I/flutter (18659):              sortKey: OrdinalSortKey#ed706(order: 1.0)
```

</details>


fixes https://github.com/flutter/flutter/issues/163842

---------

Co-authored-by: chunhtai <47866232+chunhtai@users.noreply.github.com>
2025-03-12 23:30:16 +00:00
..
animation Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
cupertino Support CupertinoSliverNavigationBar.search with condensed large title (#159120) 2025-02-07 19:02:24 +00:00
gestures Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
material Added calendar delegate to support custom calendar systems (#161874) 2025-03-08 02:41:17 +00:00
painting Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
rendering Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
sample_templates Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
services Use wildcards (#161548) 2025-01-14 05:13:41 +00:00
ui/text Auto-format Framework (#160545) 2024-12-19 20:06:21 +00:00
widgets Added semanticsIdentifier to Text Widgets (#163843) 2025-03-12 23:30:16 +00:00