Contextual (activity and proximity) state triggers in Windows 10 UWP

Contextual (activity and proximity) state triggers in Windows 10 UWP

I read a great post about contextual sensing in the new Microsoft Universal Windows Platform on Windows blog yesterday. Parts of it sounded very familiar, and for a good reason - previously, on Windows Phone 8.1, Nokia released SensorCore for (some) Lumia devices, which allowed developers to tap into various sensors to learn more about user's activities, step count, places he or she frequently visited and so on. According to Rinku Sreedhar, Senior Program Manager who wrote the blog post about contextual sensing, starting from Windows 10, parts of SensorCore SDK will be in UWP:

Starting Windows 10, Activity Sensors and Steps (Pedometer) that were originally part of SensorCore SDK will be available as a UWP API – which means it will be supported on all Windows device families on Mobile, Desktop, IoT etc. that has the required hardware and drivers. Hope this helps.

This is really cool - we get the pedometer (step counter) and activities out of the box and there's no manufacturer limitation like it was the case with Lumia phones.

Contextual sensing is cool, but what's even cooler is the way we can now use state triggers in XAML to more easily transform UI to various visual states using very simple setters. If you don't quite understand what the last sentence is all about, the best place to start is the Build 2015 session called From the Small Screen to the Big Screen: Building Universal Windows App Experiences with XAML that covers state triggers (among other UI/XAML related news). Let's start with the activity sensor.

I will not focus on how certain sensor APIs work - that's very well covered in the samples. However, you'll be able to figure out a lot about them just from looking at the state triggers code I wrote.

Activity state trigger

What if we wanted to change the UI based on user's activity. For example, rearrange XAML to make certain UI elements bigger if user is running or currently driving? ActivitySensor is the sensor we can use to do that, but wouldn't it be nice if there was a state triggers so we could do it straight from XAML?

Unfortunately, I still can't test this as today I learned that the new ActivitySensor and Pedometer code do not work on non-Windows 10 hardware so calling await ActivitySensor.GetDefaultAsync(); is returning null all the time. And here's why that happens, explained by Rinku Sreedhar, the author of the mentioned contextual sensing blog post.

It is returning NULL, because the Sensor is not present due to hardware constraints. Activity and Pedometer Sensors and the UWP APIs are supported on new hardware with Windows 10. We do understand that this is not ideal, so we are looking at ways to make it easier and will give you an update soon. In the meanwhile, as a workaround, we will provide a sample that uses both the UWP APIs and the Lumia APIs (as a fallback option) so it works on both Windows 10 hardware as well as the older hardware. I’ll give you an update as soon as we have it ready.

Let's show some code now.

public class ActivityStateTrigger : StateTriggerBase  
{
    private ActivitySensor activitySensor;
    private DeviceAccessInformation deviceAccessInformation;

    public ActivityType ActivityType { get; set; } = ActivityType.Unknown;

    public ActivitySensorReadingConfidence ActivityReadingConfidence { get; set; } = ActivitySensorReadingConfidence.Low;

The ActivityStateTrigger will have private fields for activity sensor and for device access information object used for notifying us when user gives or revokes permission to use the activity sensor device.

Two properties can be set to determine if the trigger is active or not: ActivityType and ActivityReadingConfidence. Activity type can be any of the following 8:

ActivityType possible values

Activity reading confidence can be either low or high.

In the constructor, a watcher object will be used to detect when activity sensor is attached.

public ActivityStateTrigger()  
{
    var watcher = DeviceInformation.CreateWatcher(ActivitySensor.GetDeviceSelector());

    watcher.Added += OnActivitySensorAddedAsync;
    watcher.Removed += OnActivitySensorRemoved;
    watcher.Start();
}

When the activity sensor gets added, deviceAccessInformation is initialized to listen for permission changes.

private async void OnActivitySensorAddedAsync(DeviceWatcher sender, DeviceInformation device)  
{
    // other code
    this.activitySensor = addedSensor;

    this.deviceAccessInformation = DeviceAccessInformation.CreateFromId(this.activitySensor.DeviceId);
    this.deviceAccessInformation.AccessChanged += DeviceAccessInfo_AccessChangedAsync;
}

When we determine that we're allowed to access the device, we do the initialization which consists of getting the current reading and subscribing to future reading changed events. We use the sensor reading to activate the trigger.

isActive = (reading.Activity == this.ActivityType) && (reading.Confidence >= this.ActivityReadingConfidence);  

So, the summary is:

  • wait until an activity sensor gets connected
  • when it gets connected, wait until we have permission to access the sensor
  • when we have permission, subscribe to reading changed events to update the trigger
  • if the device is disconnected, unsubscribe from all events
  • if we lose the permission to access the sensor, unsubscribe from reading changed event handlers

Example of using this trigger in XAML:

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">  
    <VisualStateManager.VisualStateGroups>
        <VisualStateGroup x:Name="ActivityVisualStateGroup">
            <VisualState x:Name="Idle">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Idle" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Idle" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="Stationary">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Stationary" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Stationary" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="Fidgeting">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Fidgeting" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Fidgeting" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="Walking">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Walking" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Walking" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="Running">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Running" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Running" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="InVehicle">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="InVehicle" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="In vehicle" />
                    <Setter Target="CurrentActivityText.FontWeight" Value="Bold" />
                </VisualState.Setters>
            </VisualState>
            <VisualState x:Name="Biking">
                <VisualState.StateTriggers>
                    <local:ActivityStateTrigger ActivityType="Biking" ActivityReadingConfidence="Low" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="CurrentActivityText.Text" Value="Biking" />
                    <Setter Target="CurrentActivityText.FontWeight" Value="Bold" />
                </VisualState.Setters>
            </VisualState>
        </VisualStateGroup>
    </VisualStateManager.VisualStateGroups>

    <TextBlock x:Name="CurrentActivityText" Text="Unknown" FontSize="36" VerticalAlignment="Center" HorizontalAlignment="Center" />
</Grid>  

Proximity state trigger

Another example of using a contextual state trigger would be a state trigger for proximity sensor. Imagine a large Surface Hub in a conference room. When people enter, your app can greet them. When people get closer to the Hub, you can enter some sort of editing mode in your app. Really interesting stuff. To implement it, the class has a couple of fields and properties

public class ProximityStateTrigger : StateTriggerBase  
{
    private ProximitySensor proximitySensor;

    public int MinimumDistanceInMillimeters { get; set; } = Int32.MinValue;
    public bool ObjectDetected { get; set; } = true;

In other words, we can activate the trigger based on whether the object is detected or not, and how close it needs to be when detected.

The same watcher pattern that you saw in ActivityStateTrigger is used. When the proximity sensor gets added, we check if it's qualified to be used with the ProximityStateTrigger based on the desired minimum distance in millimeters.

var minimumDistanceSatisfied = true;

//if we care about minimum distance
if (this.MinimumDistanceInMillimeters > Int32.MinValue)  
{
    if ((this.MinimumDistanceInMillimeters > addedSensor.MaxDistanceInMillimeters) ||
        (this.MinimumDistanceInMillimeters < addedSensor.MinDistanceInMillimeters))
    {
        minimumDistanceSatisfied = false;
    }
}

if (minimumDistanceSatisfied)  
{
    this.proximitySensor = addedSensor;

    await SetActiveFromReadingAsync(this.proximitySensor.GetCurrentReading());

    this.proximitySensor.ReadingChanged += ProximitySensor_ReadingChangedAsync;
}

Proximity sensor on a device I used for testing returns values 0 and 1 for MinDistanceInMillimeters and MaxDistanceInMillimeters, respectively, which also confirmed my suspicions that sensor APIs for phones are still not 100% working for older, Windows Phone 8 devices. Rinku Sreedhar was kind enough to let me know in the comments that this is by design. Min and Max distance in millimeters only makes sense for devices that have long range proximity sensors, which is not the case on phones, but is on Surface Hub for example. Here's a part of the comment explaining that:

...distance property is an optional property typically exposed on Long Range Proximity Sensors and not Short Range Sensors that you see on the phone, which is the reason you are not finding it working. The SR proximity sensors has a detection range between a few mm to cms, hence knowing the distance is not usually very useful in this case to tailor a difference experience based on the distance, that's why you don't see it exposed. However for Long Range Proximity Sensors, as in the case of Surface Hub you mentioned above, we can have different experiences like increase the brightness as you go closer to the screen etc, based on the distance.

A method is used to activate or deactivate the trigger based on the sensor reading.

private async Task SetActiveFromReadingAsync(ProximitySensorReading reading)  
{
    if (reading != null)
    {
        if (this.ObjectDetected)
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {
                var isActive = reading.IsDetected;

                if (isActive)
                {
                    if (reading.DistanceInMillimeters.HasValue)
                    {
                        isActive = (reading.DistanceInMillimeters >= this.MinimumDistanceInMillimeters);
                    }
                }

                SetActive(isActive);
            });
        }
        else
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {
                SetActive(!reading.IsDetected);
            });
        }
    }
}

Here's how you could use it in XAML:

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">  
    <VisualStateManager.VisualStateGroups>
        <VisualStateGroup x:Name="ProximityVisualStateGroup">
            <VisualState x:Name="ObjectDetected">
                <VisualState.StateTriggers>
                    <local:ProximityStateTrigger ObjectDetected="True" />
                </VisualState.StateTriggers>
                <VisualState.Setters>
                    <Setter Target="ProximityText.Text" Value="Hello there!" />
                </VisualState.Setters>
            </VisualState>
        </VisualStateGroup>
    </VisualStateManager.VisualStateGroups>

    <TextBlock x:Name="ProximityText" Text="No one is near" FontSize="36" VerticalAlignment="Center" HorizontalAlignment="Center" />
</Grid>  

When the proximity sensor detects and object, the text gets changed to greet the person.

Conclusion

Contextual sensing is a pretty cool and powerful thing. What's even better is that we as developers can combine the sensor code to create custom state triggers to react to different contexts straight from XAML in a clean, reusable way.

Complete code is on GitHub. Enjoy!

Igor Ralic

igor ralic

View Comments
Microsoft Certified Solutions Developer: Windows Store Apps in C#