The Learning Home

CST 499

Joe Sarabia

August 27, 2017

 2

Executive Summary

This project seeks to enhance the usability of home automation by correlating user location and
user activity within a home to build behavioral models of individual users which can then be
applied to automatically apply state based on detection of a user and other environmental factors.
This project affects all home automation users. The expected outcome is a functional prototype
demonstrating how machine learning and home automation can combined to achieve enhanced
usability.
 3

Executive Summary 2

Overview 4

Project Description 4
Phase I: Research-based 5
Phase II: Project-based 5
What This Project Will Do 6
Who Will Benefit and How 6
Why Is It Important 6

Project Goals and Objectives 7

Phase I: Research-based 7
Phase II: Project-based 8

Background Information 9

Literature Review 10
Paper 1: Bluetooth Low Energy for Smart Home Energy Management 10
Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation 11

Stakeholders and Community 12

Methodology 13
Milestone 1: Market Analysis and Feasibility Assessment 13
Milestone 2: Procure Equipment 14
Milestone 3: Prototype Implementation 14
Milestone 4: Prepare Demonstration 15

Ethical Considerations 15
Ethical Concerns 15
Negative Impacts 17
Environmental Impacts 18

References 19

Appendix 20
 4

Overview

This capstone project will be a hybrid of a research-based and a project-based capstone as

initially advised by Dr. Tao. The first part of the the project will be research-based and will

incorporate a survey of the industry to hone in on a specific area of focus for the second part of

the capstone. This second part will be project-based with a focus on implementing a prototype as

informed by the first part of the project.

Project Description

While substantial progress has been made towards delivering on the promise of the smart

home of tomorrow, even the modest smart home capabilities of today generally require a

prohibitive level of expertise or experience to implement and utilize on a consistent basis,

thereby excluding vast portions of this technologys potential user base. This research intends to

investigate the extent to which todays technology can be used to create a learning home by

implementing an artificial intelligence (AI) which is able to learn the behavior of its occupants

and derive the desired state of the home at any given time based upon a combination of

parameters such as time of day, biometrics, weather conditions and the specific location of a

particular occupant or combination of occupants within the home. This state engine should

ultimately be able to control common Internet of Things (IoT) devices such as lighting, music

speakers, electrical outlets, appliances and televisions. The research intends to create a functional

prototype of this state engine demonstrating control of a subset of these devices based on a

subset of the aforementioned conditions.

 5

Phase I: Research-based

Can todays existing technology be used or augmented to build an AI to create a learning

home? This research-based portion of the capstone project will seek to answer whether particular

hardware such as IoT devices and presence detection devices can be used to accurately and

precisely track the location of specific individuals within a home. Furthermore, this research will

seek to answer whether a software implementation can be created to detect and track the change

of state of IoT devices within the home and correlate that to the location of a specific individual

within the home. Finally, this research will seek to answer whether an AI can be built to learn the

behaviors of a homes occupants and automatically apply desired state based upon applicable

parameters such as time of day, weather conditions and other factors which influence a particular

behavior.

Phase II: Project-based

How well can todays existing technology be used to create an early, functional prototype

of a learning home? Based upon the findings from the research-based portion, the project-based

portion of this capstone will seek to implement a prototype which is capable of precisely tracking

indoor location of occupants within a home, observing state changes of IoT devices within that

home and then correlating these data together to produce a recommendation engine for desired

state of a home based upon the location of a particular occupant and at least one other desirable

parameter such as time of day.

 6

What This Project Will Do

The culmination of this capstone project is an early prototype that effectively

demonstrates the ideas embodied within this research as outline above. Ultimately, it should

enable a home to exhibit the capability to accurately respond to particular combinations of inputs

based upon previous learned behaviors. The idea is to facilitate the transition from quasi-smart

homes to truly smart homes.

Who Will Benefit and How

Potentially any current or future user of home automation based technologies will benefit

from this research. In particular, classes of users who are otherwise excluded from or

substantially disadvantaged in leveraging home automation and its capabilities today would

benefit.

Why Is It Important

This research is important on two levels. Firstly, software is becoming increasingly

ubiquitous and its capabilities are increasingly impressive. There is frequent conversation in the

national dialogue today amongst policymakers and others about the role of automation and

artificial intelligence in our future economy. On one level, this research is important to

demonstrate the power of artificial intelligence in a tangible way that is readily accessible and to

close the gap between the sci-fi based depictions of a smart home, such as those depicted in Back

to the Future and The Jetsons with the real capabilities of todays technology. On another level,
 7

the realization of these capabilities opens up more possibilities for societal and individual benefit

when our surrounding environments can intuitively adapt to our presence.

Project Goals and Objectives

The three most important criteria to demonstrate this research are that a change of state

occurs based upon a combination of the following factors: a particular users location within a

home, some other pre-condition such as time of day, and that it exhibits control over more than

one class of devices and preferably in combination with each other.

Phase I: Research-based

During the research based portion of the this capstone, the objective will be to survey the

industry and identify suitable technologies and their capabilities to fulfill the above use cases.

For purposes of precise location tracking, the objective will be to identify 1-2 technologies which

are able to determine specific location within a room, for example detecting an occupant in the

living room may not be sufficient, the technology should be able to detect the difference between

sitting on the couch in the living room and just walking through it. A requirement for these

technologies is that they have a public Application Programming Interface (API) which can be

used for integration in the broader solution. For purposes of detecting behaviors and ultimately

automatically applying state or at least generating a recommended state, an objective of this

portion of the research will be to identify 2-3 IoT devices with APIs that can be used to read and

detect state changes.

 8

Phase II: Project-based

There are two sample use cases that I intend to explore as possible early prototypes based

upon findings in the research-based portion of the project. The ultimate goal with these would be

to enhance the usability of home automation.

Objective 1: Follow Me. The objective with this use case will be to demonstrate the

feasibility and a prototype of some desired state that follows an occupants traversal through

various parts of a home. Similar to how light sensors work, this can be thought of a sensor with

persona, that is, the sensors know the difference between multiple occupants such that they only

activate for particular occupants under particular circumstances. One interesting use case to

demonstrate here is follow-me audio, where a multi-speaker, multi-room audio system is able to

continuously adapt its state based upon an occupants location. An example of this could be a

playlist that starts playing on a speaker when an occupant walks into the bathroom to get ready

for the day, then switches from the bathroom speaker to the bedroom speaker while the

occupants finishes getting ready and then switches to the living room speaker when the occupant

moves to couch to prepare a to-do list for the day. Its not necessary and indeed inefficient for

speakers to be playing in a room the occupant doesnt occupy, and this use case would

demonstrate the ability for a home to apply a state, in this case a playlist, and have it follow that

user around the home.

Objective 2: Apply a Learned Behavior. A behavior in the context of this research is

some combination or sequence of states across multiple IoT devices or systems applied by a

particular occupant in a discernible and predictable manner. The objective of this use case will be
 9

to demonstrate the ability to learn and appropriately apply such a behavior. An example use case

of this is a breakfast making routine where every weekday morning between 8:15a and 8:45a an

occupant enters the kitchen, turns on two specific lights, requests to hear the news from a home

assistance device, and then plays a playlist consisting of 3-4 songs while making and eating

breakfast. Before leaving the kitchen, the occupant turns off the playlist and turns off the lights

while heading out the door for work. The objective of this use case would be to learn this

behavior and then appropriately apply it if the occupant is present during the learned timelines.

Importantly, this use case should also demonstrate that state changes dont occur for that same

occupant in that location at times previously unobserved, that is, it shouldnt apply the breakfast

behavior during dinner, and it should only apply the behavior for intended occupants and state

should be unaffected by the presence of some other occupant. For example a second occupant

passing through the kitchen during the same time should have no impact on the state of the

home.

Background Information

Primarily my theories are based upon my observations in implementing and maximizing

smart home technologies in my home over the last 3.5 years. Initially, my attempts at

implementing home automation yielded a very low wife acceptance factor, and I realized that not

everyone would find it appealing to control their home from their phone, or even have a phone

from which they could control their home. Ive observed numerous deficiencies in existing

products such as geofences that are orders of magnitude larger than my home and which dont

intelligently account for the presence of multiple people within a home. More than once, I might

leave to go to the store and all the lights in the house as drove away while my wife was sitting on
 10

the couch watching television. This is where the initial ideas behind occupant detection came

into play.

In addition, I eventually came to a realization that the cumbersome nature of switching

between multiple apps on my phone to yield a desired state in my home could be improved upon.

I realized that I was typically taking the same set of actions from the same location within the

home, and that the this sequence of steps along with precise occupant detection should be

sufficient to create a home that can learn and apply my desired behaviors and truly be called a

smart home.

Literature Review

A brief review of existing research yielded two results which have some similar concepts

yet not entirely overlapping concepts and proposals. The two papers reviewed are covered

below.

Paper 1: Bluetooth Low Energy for Smart Home Energy Management

The paper proposes an interesting approach to home automation whereby home

appliances communicate to an energy management unit (EMU) and some intermediary system

between the appliances and the user of the appliance is able to make a recommendations about

the use of the appliance when the user attempts to use it, generally in an effort to reduce peak

load demand and consumptions charges (Collotta & Pau, 2015). For example, the EMU might

suggest that the electric clothes dryer be used at a later time when the usage fee for electricity is

lower. While this does involve a use case where home automation interacts with the user which

is something The Learning Home proposes, in the case it occurs through an intermediary system
 11

and is not completed automated based on behavior. The Learning Home proposes that similar

activity with appliances be applied by presence detection. Nonetheless, there could certainly be

use cases where both applications might exist, or where the data collection by the EMU provides

an input into The Learning Homes state engine as a parameter to consider in determining

whether some state should be applied.

This paper proposes using Bluetooth Low Energy (BLE) to coordinate communication

between home appliances as a mechanism for home energy management (HEM). The choice of

wireless protocol is critical according to this paper. BLE connection setup and data transfer occur

in a range from 3ms to 1.28s. Tens or even hundreds of milliseconds to transfer data would likely

be sufficient to satisfy the perception of near real-time reaction in the cases proposed by The

Learning Home. The authors point out the substantial BLE penetration in the industry, for

example, its already technology included in commonly owned devices such as smart phones,

smart watches and tables. They also point out that reliability is key to achieve accurate

automation functionality. The paper asserts that BLE is the best choice in applications of home

automation.

Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation

This research by S. Shaikh, Attar, Pathan & R. Shaik (2016) sounds similar in that their

goal is to use Machine Learning (ML) to generate personalized and time-variant home

automation plans. It posits a solution to the same problem that The Learning Home seeks to

improve upon, namely the rigidity of usability. This research also proposes a system that will

mimic a users actions based upon previously detected behavioral patterns.

 12

While these concepts have some overlap with concepts that The Learning Home also

seeks to investigate, this paper is two pages long and is primarily conceptual in its nature with no

indication of whether a prototype was attempted or implemented. Additionally, there is no

reference to the user differentiation detection capabilities or precise real-time location services

that The Learning Home proposes as part of its solution.

Stakeholders and Community

As previously mentioned, the stakeholders and community for this project are any current

or future home automation user. What this group of individuals stands to gain is an improved

realization of home automation benefits, many of which could be construed as comfort level

benefits. Nonetheless, home automation promises advances in many aspects of ones home

including security and energy efficiency. One common IoT device with interesting capabilities is

the smart camera which can be used to detect activity within zones, and more recently, human

presence in a specified area. This activity detection can then be used to trigger desired home

automation actions. While useful, today these are not personalized. For example, the cameras can

detect a person but not a specific person. This paper asserts that the ability to detect particular

people in particular spaces coupled with behavioral modeling will add greater persona and

reduced rigidity to the usability of home automation. On the other end of the spectrum, another

consideration for the home automation user community is one of privacy. There are some

prospective users of home automation who may have no interest in having cameras storing

images of their activity in some external storage service, or there might be objections to the use

of cameras in particular parts of the home, such as bedrooms and bathrooms. The lack of

cameras in these areas otherwise reduces home automation capabilities which might otherwise be
 13

achieved. This paper seeks to overcome such privacy objections by researching feasible options

to trigger actions based on the presence of a particular person in a particular position within a

space without a camera while also providing greater location detection precision.

One particular area of interest that this paper seeks to investigate is to what extent the

techniques discussed herein might benefit disabled people using home automation. As an

example, great strides have been made in speech based home assistant type devices, but an

individual with a speech impediment may not benefit from some of the capabilities such devices

provide to control their home.

Lastly, anybody new to home automation, intimidated by the technology or simply a

technology neophyte would benefit from this research.

Methodology

The methodology I intend to follow in completing this project will be Agile, using

iterative and incremental development techniques and fast feedback loops to adjust as the

projects goes on. I intend to use real-world consumer devices combined with software to

implement prototypes of the topics discussed herein. I have identified four major milestones as

part of this project, which are discussed in the following subsections.

Milestone 1: Market Analysis and Feasibility Assessment

In order to meet this milestone, I intend to research various devices currently available in

the consumer marketplace. Specifically, I intend to research smart cameras which are suitable

candidates for real-time facial detection and devices which can be used to precisely track

real-time location of a user within a home. As part of the intended solution seeks to correlate the
 14

activity of devices with the location of users in order to build behavioral models which could

later apply state, I further intend to research home automation smart hubs which aggregate the

control of multiple devices where some such model can be applied. One important part of

research that I intend to complete in this milestone is the assessment of APIs for all devices that I

research, as that will be important for later stages of project. Based on the results of this

assessment, I will make any required adjustments to scope which could be necessary based on

my findings.

Milestone 2: Procure Equipment

To reach this milestone, I intend to choose some combination of devices based upon the

completed analysis. The intent here is to use this equipment to build prototypes of the topics

discussed in this paper. If the analysis turns up items which I already own, then that could

accelerate the development process and reduce the budget.

Milestone 3: Prototype Implementation

I intend to use the equipment previously procured to build a functional prototype of the

concepts discussed in this paper. Specifically, I intend to demonstrate the ability to correlate user

activity with user location to build a behavioral model. I further intend to demonstrate both a

camera based and non-camera based approach. Lastly, I intend to demonstrate how a learned

behavior can be applied to trigger some home automation activity. In order to reach this

milestone, I will need to conduct extensive research in the area of machine learning as its not an

area which I have previous exposure to. I will use iterative and incremental development during
 15

this phase and will use tools such as Pivotal Tracker and GitHub to track my work. I expect that

adjustments to the scope of the project are likely to occur in this phase.

Milestone 4: Prepare Demonstration

Most of the testing will be conducted in my home and I would like to find a way to

present this on a smaller scale on-campus, if possible. To that extent, during this portion of the

project I intend to experiment with a scaled down version or other mechanisms to effectively

demonstrate the broader ideas behind the project.

Ethical Considerations

Given that this research deals with tracking user activity and location very precisely

within ones home, there are important ethical considerations to consider both in its design and

implementation.

Ethical Concerns

The primary ethical concerns relating to this capstone project relate to privacy. This

project seeks to investigate two fundamental methods as mechanisms to track user behavior and

build corresponding behavioral models using machine learning, each of which has a discrete set

of privacy considerations.

Location Services. While location tracking capabilities, which are commonly called

location services, provide substantial capabilities in a variety of mobile devices, they can be

viewed as useful or invasive. Mobile devices commonly leverage location services to provide

useful functions such as GPS based directions and geo-tagging of objects such as pictures.

Nevertheless, many device users are not comfortable with the notion of their every movement
 16

and location being tracked and recorded. As a result of these concerns, some users opt to turn off

location services in an effort to reduce the recording of such data.

A common use of location services in home automation is for the control of lighting, for

example, to turn lights on or off when entering or exiting a region such as a geofence around

somebodys home. In the case of iOS devices, Apple Inc. (2017) states that Location Services

uses GPS and Bluetooth (where they're available), along with crowd-sourced Wi-Fi hotspots and

cellular towers to determine the approximate location of your device. One of the primary

mechanisms that this project plans to use for precisely tracking user location within a home is via

Bluetooth communications between a personal device such as a mobile phone, e.g. an iPhone, or

a wearable device, e.g. Apple Watch and beacons. Any user who wishes to completely turn off

all location services would not be able to benefit from the techniques discussed in this paper.

Nevertheless, the opposition to tracking everywhere a person goes outside of their home

is expected to be more more objectionable than tracking where they are inside their home. For

example, the former may expose illicit or morally objectionable behavior that a device user may

wish to conceal, while in the case of the latter, a visit to the closet or the living room is less likely

to be of substantial consequence or a target of desired concealment. This is an assumption worth

validating with potential users however should this research yield viable prototypes.

One method of mitigating this concern, should it become one, would be to use some

ancillary device other than mobile device as a tracking mechanism to represent the user, such as

a bluetooth beacon attached to a keychain or simply carried in ones pocket.

Cameras. The other primary method this research intends to evaluate as part of its efforts

to build user behavioral models is using IoT cameras. As demonstrated by the Persirai IoT botnet
 17

in May 2017, IoT cameras having proven to be a favorite target of malware authors and used as a

launching point for distributed denial of service (DDOS) attacks. Additionally, while users have

generally been accepting of storing streaming video of activity outside their homes with some

external service provider as a method of home security, it is expected that users would have

strong opposition to continuous video recording inside their home, particularly in certain rooms

within the home such as bedrooms and bathrooms where a generally high level of privacy should

rightfully be expected.

This research intends to use IoT cameras to perform facial recognition as a means to

discern and identify various actors within the home and eventually use those recognition

capabilities to apply learned behaviors. For example, this could mean the detection and

application of different lighting based upon who enters the front door. This research aims to

mitigate concerns about cameras within a home by positioning them to cover main points of

ingress and egress such as a front door or garage entry door and main living spaces such as a

family room and kitchen. This research will not consider applications of IoT devices in other

parts of the home, and therefore anticipates a hybrid approach between the bluetooth and camera

methods to achieve its desired effects.

Negative Impacts

Home automation devices today are expensive compared to their traditional

non-automated counterparts, and thus the barrier for entry could be very high for lower income

individuals and families. According to Smith (2017), while 77% of all Americans own a

smartphone, as many as 64% of low income owners also own smartphones. Bluetooth is a
 18

common and fundamental technology on smartphones, so a majority of people should be able to

leverage at least some of techniques which are part of this research and nearly all smartphones

users should theoretically, from a technology perspective, be able to leverage these techniques

using capabilities built-in to their phone. The bigger issue which may exclude low-income

groups would be the additional cost of the items required to track location inside a home such as

beacons or IoT cameras.

A long term goal of this research is to determine whether it would be useful or feasible to

incorporate BLE chips in more commonly used devices such as light switches and power plugs

to facilitate some of the techniques this research will examine. The ability to do so in a viable

manner may reduce the potential financial burdens associated with implementation of the

technology.

Environmental Impacts

One of the primary motivations behind this project and indeed home automation in

general is to have a positive environmental impact by enabling people to use their homes more

efficiently. Automation of lighting is a common entry point into home automation and this

research intends to build techniques that will make the use of homes and devices more efficient

beyond the control of lighting.

 19

References

Apple Inc. (2017, February 24). Turn Location Services and GPS on or off on your iPhone, iPad,

or iPod touch. Retrieved August 22, 2017 from

https://support.apple.com/en-us/HT207092

Collotta, M., & Pau, G. (2015). A Solution Based on Bluetooth Low Energy for Smart Home

Energy Management. Energies, 8(10), 11916-11938. https://doi.org/10.3390/en81011916

Shaikh, S., Attar, A., Pathan, S., & Shaikh, R. (2016). Machine Learning On Real-Time Data to

Enhance Home Automation. International Journal of Engineering Trends and Technology

(IJETT), 42(1), 6-8. https://doi.org/10.14445/22315381/IJETT-V42P202

Smith, A. (2017, January 12). Record shares of Americans now own smartphones, have home

broadband. Pew Research Center. Retrieved from

http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/
 20

Appendix

Smart Sensor Research - Sen.se

Sen.se provides sensors and and iOS application that potentially embody part of the

functionality that this research seeks to implement. An overview of the hardware

products are located at https://sen.se/store/c/peanuts/.

A critical aspect of whether any existing solution will be viable to include with

this research is an API, in particular one which will expose the ability to use proximity

detection that can be read by some external observer to record and store that information

in a separate system. This data can then be analyzed by machine learning algorithms to

potentially correlate positions, in this case proximity to a beacon, with state changes

occurring within the home. Upon cursory examination, Sen.se seems to have a

well-formed API for this purpose, though a more thorough examination will be required.

API documentation for Sen.se is located at https://sen.se/developers/documentation/.