The story behind the 'Magic of Nature'

The story behind the 'Magic of Nature'

‘Magic of nature’: Disney’s smart birdhouses reveal the secret lives of purple martins | Transform
Digital transformations and the new face of business

Background - from maintenance to martins

I didn't start with a project pitch to understand a bird species. My incredible colleague Molly McCarthy and the broader ISE team had been hard at work rebuilding a trusted relationship with Disney. After some awesome code-with engagement successes, I was asked to fly down to Orlando to discuss interest in using HoloLens in their parks for operations, training, and maintenance.

I spent the afternoon with the Disney executive and learned through our conversation that they were really seeking a way for our team to work with their Emerging Technologies group in DPEP (Disney Parks, Experiences and Products) - and thought AR and HoloLens would be a cool next-gen technology to work together on.

I'd already spent several years knee deep in AR with the 3D Streaming Toolkit, so I was the "HoloLens guy". We walked through Disney's Animal Kingdom to spark our brains and meet employees in-situ, and by the end of the walk I think we had a dozen ideas for incredible projects.

The one we thought was going to be "the" project was an AR experience in the Meerkat habitat. Meerkats are a beloved habitat, but they spend a lot of time underground and out of sight. Using ground sonar and other passive sensors, we imagined being able to see an x-ray style view of the Meerkats in their tunnels to make the guest experience always interactive and engaging.

A return trip to the park unfortunately dropped a bucket of cold water on this plan. The conversationists told us that a lot of the behavior that happens underground is very R-rated as Meerkats can be incredibly violent at times. And their tunnels constantly shift, collapse, and get rerouted, making it almost impossible to instrument sensors that would have a long useful lifespan.

But this was the spark we needed - the folks at Disney had an immediate thought - what about the Purple Martins? They are a species with very active conservation efforts, attuned to humans, and while they had nests all over Disney's parks in Orlando, there'd never been much of a guest-facing experience with them.

The project had its first seed planted.

Opportunity - unmet needs

We met Jason Fischer and Dave MacLean on our first day on the ground to sort out how Microsoft could help. Jason is the purple martin man at Disney.

Jason schooled us on the real process of animal conversation and research with the Purple Martins, and by the end of the first day we had a long list of great ideas to improve the lives of the employees, raise the bar on the understanding of the birds' behavior, and bring all of this incredible science to park guests in some unique ways.

Jason Fisher showing us the real daily operations for Purple Martin conservation

Conservationists spend an incredible amount of time lowering the nests to the ground, opening each one, and manually marking what they see inside.

  • The precision of the observational data is low, and they are limited in how often they can check the nests and only during the workday.
  • A large number of the distinct phases of mating, hatching, growth, and fledging have to be inferred in between observations.
  • There are many known threats to the purple martins - hawks, owls, starlings, sparrows, and mites to name a few. But there is little direct evidence of how these threats interact with the nest site, and little opportunity for intervention before disaster strikes

Goals and constraints

I defined four goals for the project after our ideation and immersion time with Disney.

Goals

  1. Invert the observation operational model for conservationists
    Enable them to spend the bulk of their time analyzing behavior and patterns, rather than raising and lowering nests.
  2. Help the Emerging Technologies team deliver a guest facing experience
    Almost everything guest facing comes through imagineering at Disney. We wanted to demonstrate that other teams could deliver high quality experiences to guests.
  3. Strengthen the relationship between Microsoft and Disney by delivering an innovative solution together
    Continuing the great work of my colleagues in building trust and showing through action that Microsoft is a great partner and customer enabler.
  4. Demonstrate a scalable long-term solution
    We had to prove to Disney that this could be more than a one-off, limited time project. That we were building a solution that could scale for their business, for guest experiences, and for animal conservation efforts.

Constraints

  • Time
    We began the project in earnest in January. We had to deliver by Earth Day, April 21st, 2018. Any delay would mean the whole project would be cancelled and never see the light of day.
  • Resources
    My team had 7 full time resources from Microsoft and 4 from Disney. We needed to stay within the normal operating budget of about $50k spend. I was extremely fortunate that the team members from both companies were assembled based on the skills we needed to succeed - so we had the right experts on the team from the beginning.
The Purple Martins projct team
  • Location
    Most of the Microsoft team was in Redmond, while all of the Disney team was in Orlando. We could travel as needed, but had to be very deliberate about making the most of physical time together as a team.
  • Quality
    Disney sets a very high bar for quality - both behind the curtains and in front of guests. This project had to have all of the testing, process, and solution rigor of any commercial offering from either company.
  • Scale
    The project had to be scalable from the start. We all started with the assumption that this would become a service to empower the larger public Purple Martin conversation community.

Architecture - science at the nest

System architecture for the Purple Martins project

Given the goals and constraints, a number of the architectural decisions fell into place very quickly and early.

The basic idea was to put two small compute modules on each nest, one at the entrance and one peering into the central interior. Connect these two units via PoE to an outdoor switch and push all observational data to a cloud data store. By training and deploying machine vision models to these compute modules, we can record when critical events happen and alert the conversationists when necessary.

The nests will be under continuous monitoring. In the near-term they continue their operational process in a similar manner, just skipping the steps of physically opening each nest.

Hardware

Raspberry Pi as the IoT platform at the nest is well covered in my article on building a weather resistant IoT device. In short - we wanted this to scale for a large community. To do so we needed to offload as much compute as possible to the edge clients. Lightweight services are cheap and scalable services.

The Jetson board was an easy choice, as it had great IoT edge support, massive compute for the real-time video analysis needed, and was already industrialized for an outdoor deployment.

Data scaling

Keeping compute local was a challenge solved by using the then newly released Embedded Learning Library from Microsoft research. It allowed us to run all the inferencing locally, offloading almost all compute from cloud services. ELL needed every ounce of system memory and CPU/GPU time we could squeeze out of the Pi3B.

This necessitated a deviation from the "template" architecture of IoT Edge, and containers through ACR. I made the choice to keep the Raspberry Pi 3B device management at the bare metal, rather than running through IoT Edge and containers. With the Pi4 and beyond, containers are the way to go.

We adopted the pipeline pattern of IoT hub to Event hub and into CosmosDB from a long list of other successful commercial customer solutions. It was well suited to this project - IoT devices sending a mix of event triggered and continuous aggregated telemetry to be used as a dataset for future analysis.

Quick back of napkin math put our CoGS (cost of goods and services) at about $500/yr ($5 per device per month) for the 4 nests in the project, and at about $7,500/yr to observe every nest on Disney property (~$3.50 per device per month).

This would be okay for Disney, as it would provide rich data - video and images synced to environmental telemetry - but we immediately knew it was unlikely to scale for a large public community for two reasons:

1. The ingest, storage, and processing for the video content would be massive.

The primary purpose for retaining video and image data is to improve model accuracy through continual training. Even during the project I witnessed the "Youtube phenomenon" begin to happen - everyone watched the early recordings but the more that came in, the less we watched.

YouTube channels, uploads and views: A statistical analysis of the past 10 years

Knowing that keeping all the data wouldn't be useful or desirable, I made sure we made media submission optional both on the client side as well as on the service side - clients can disable media submission, and each registered device on the service end has to be granted media submission rights.

Without sending video the scale math made a lot more sense - coming in at about $12,000/yr to enable 250,000 nests (500,000 Pis) - just under $0.05 per device per month - including buffer to send ~0.5% of events with full video / audio / image data.

  1. Community data science - data science gets expensive with large communities

I also knew that this dataset would be a goldmine for citizen science. Even with the small sample size of our 4-nest observation, we found a treasure trove of insights. Opening PowerBI and Cosmos to the conservationists at Disney was no problem, but enabling this for a massive volunteer community would quickly skyrocket the costs for a centralized approach to analysis.

So we made room for a community API service very early on. A fully public set of REST endpoints that were highly optimized to minimize retrieval costs. Users can't perform complex queries, but they can quickly get meaningful slices of the full dataset, as well as being able to just pull archives of the entire telemetry collection. This was also baked into the cost estimation above.

User experience

The user experience for this project was in many ways the easy* part. Once we had good hard, good pipelines, and good storage working - we a ton of incredibly heartwarming, awe inspiring, and insightful content at our fingertips.

For the conservation team and business stakeholders, I built out a set of PowerBI reports and dashboards enabling them to dig into the lifecycle of these birds with an astonishing level of accuracy, detail, and empathy.

0:00
/0:10

PowerBI dashboard showing labeled machine vision events from real nests

The ML engineer had direct pipelines for supervised learning from the verification and labeling of incoming video streams.

For park guests, we started work on ARKit based iPad applications to tell compelling stories with the mountain of telemetry data that was collected.

'Taking flight' with park guests

The other half of the project was taking this data and putting it together into an educational entertainment experience for park guests on Earth Day 2018. Our colleagues wanted to ensure the work we had done together could be appreciated and enjoyed by park guests.

It had been extremely rare for a non-imagineering group at Disney to put guest facing experiences into the park. There had never been a 3rd party to do so.

We designed and built an ARKit based iPad application we called Taking Flight that brought together the learning from our iOT data in the nests and brought it together in an interactive, augmented reality experience.

0:00
/0:21

A demo of the flight tracking game portion of "Taking Flight"

Outcomes

We learned a ton as a cross-company team with this project.

It was a huge success for the business relationship, helping to continue paving the way for multiple deals over the next several years.

Microsoft and The Walt Disney Studios to develop ‘scene-to-screen’ content workflows - Stories
Companies collaborate to pilot new ways to transform content workflows in the Microsoft Azure cloud; Microsoft becomes a Disney Studios StudioLAB innovation partner REDMOND, Wash., and BURBANK, Calif. — Sept. 13, 2019 — Microsoft Corp. and The Walt Disney Studios today announced a five-year innovation partnership to pilot new ways to create, produce and distribute […]

This was one of the first commercial deployments of the Embedded Learning Library and we proved the viability of running real-time inferencing on extremely low power IoT devices.

I learned a ton about the realities of managing IoT devices in the wild and the critical importance 100% reliable OTA updating of both system and solution functionality.

I also learned a ton about real world MLOps - wrangling data for training, labeling, and the many ways to continuously improve model accuracy.

We collected several months of continuous behavioral and environmental data on the instrumented nests, making the largest unified behavioral dataset collected for Purple Martins.

Even with a lot of AR experience in HoloLens, architecting and building an ARKit application for the iPad was another great learning experience - data formatting, micro to macro alignment for GPS data, mapping, and calibration to name a few things.

In the end, I lead a small and mighty cross-company team to launch a guest facing experience in Disney's Animal Kingdom on Earth Day 2018.

A rare event for the Emerging Technology team, and even more rare for a non-sponsored vendor to contribute.