EE5111 Selected Topics in Industrial Control & Instrumentation

Assessment:

Implement a simple IoT pipeline with AWS Cloud platform and Visualise the data.

Author:

Zhan Yuancheng (A0206839W)

Github:

https://github.com/nkzyc/EE5111_A0206839W

Medium post:

https://www.cnblogs.com/nkzyc/p/11561289.html

1.Introduction. 3

1.1 IOT Background. 3

1.2 Goals of the Assessment. 5

1.3 Prerequisites. 5

2.Sending data to AWS by Official routine. 6

2.1 Create the AWS IoT Policy. 6

2.2 Create the things. 7

2.3 Email Alerts for Low Water Content Readings. 8

2.4 Simulate random water content. 10

2.5 Results. 11

3.Sending data FD001 to AWS. 11

3.1 Introduction of the database. 11

3.2 Create the AWS IoT Policy for FD001. 12

3.3 Create the things of FD001. 13

3.4 Create AWS IoT rule and AWS DynamoDB. 14

3.5 Use python codes to meet the requirements. 17

3.6 Results and Analysis. 19

4. Sending data from FD001 and FD002 to AWS. 20

4.1 Create the Policy, Thing and DynamoDB for FD002. 20

4.2 Create IoT rule for both FD001 and FD002. 22

4.3 Results and Analysis. 22

5.Visualization of data. 23

5.1 Get AWS access key. 23

5.2 Redash environment. 24

5.3 Obtain and Visualize the data. 25

5.4 Examples of Visualization by Redash. 26

6.Anylasis of Motor Vehicle Inspection data source. 28

6.1 Introduction of data source. 28

6.2 Create the Policy, Thing, Rule and DynamoDB. 29

6.3 Data processing and Results. 31

6.4 Visualization of data source. 33

7.Conclusion. 36

8.Appendix(completed python codes) 36

1.Introduction

1.1 IOT Background

IOT

The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.

The definition of the Internet of things has evolved due to the convergence of multiple technologies, real-time analytics, machine learning, commodity sensors, and embedded systems. Traditional fields of embedded systems, wireless sensor networks, control systems, automation (including home and building automation), and others all contribute to enabling the Internet of Things. In the consumer market, IoT technology is most synonymous with products pertaining to the concept of the "smart home", covering devices and appliances (such as lighting fixtures, thermostats, home security systems and cameras, and other home appliances) that support one or more common ecosystems, and can be controlled via devices associated with that ecosystem, such as smartphones and smart speakers.

There are a number of serious concerns about dangers in the growth of IoT, especially in the areas of privacy and security; and consequently industry and governmental moves to begin to address these.

AWS IoT

AWS IoT provides secure, bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud. This enables you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets.

AWS IoT consists of the following components:

Device gateway

Message broker

Rules engine

Security and Identity service

Registry

Group registry

Device shadow

Device Shadow service

Device Provisioning service

Custom Authentication service

Jobs service

AWS IoT integrates directly with the following AWS services:

Amazon Simple Storage Service

Amazon DynamoDB

AWS Lambda

Amazon Simple Notification Service

Amazon Simple Queue Service

And so on

1.2 Goals of the Assessment

The whole information of the goals are on the github:

https://github.com/iceberg12/NUS_guest_lecture/tree/master/EE5111

From my report, I achieve the following goals:

1.Follow the AWS instruction to understand how to populate, send data from a computer to AWS through MQTT protocol. (Optional)

2. Publish pre-defined engine data to AWS.

3. Simulate two IoT things.

4. Visualize the data for the two engines for all the sensors.

5. Use other data sources.

All of the goals I have achieved will be implemented in the following report with detailed description.

1.3 Prerequisites

I will first explain The tools and environment configuration I used in this task are following. Please refer to the corresponding official documentation for the specific installation steps.

Environment:

Windows 8.1

Python 3.5.4

AWS IOT core

Tools:

Pycharm 2017

Anaconda 3

Datadog

Redash

2.Sending data to AWS by Official routine

In the Official routine, there is a tutorial of Plant Watering with AWS IoT. This will help us to understand how to send data to AWS with our computer.

This control of Plant Watering shows us how to use a moisture sensor and AWS IoT to monitor the soil moisture level for a house plant or garden. We can create a rule in AWS IoT that sends an email to an address subscribed to an Amazon SNS topic when the moisture level falls below a threshold.

There are five steps as follow.

2.1 Create the AWS IoT Policy

In this step, create an AWS IoT policy that allows our sensor to connect and send messages to AWS IoT. We should do the following:

Sign in to the AWS Management(https://aws.amazon.com) and open the AWS IoT console; enter the IoT Core and select the policies as shown in Figure2.1

Figure2.1 create policy1

select Create Policy and provide a name PlantWateringPolicy; Enter iot:* for action and * for ARN as shown in Figure2.2;

Figure2.2 create policy2

2.2 Create the things

In this step, we will create one of the AWS IoT in things , which is used a device simulator. Devices connected to AWS IoT are represented by things in the AWS IoT registry.

Select Manage and choose Things and use Create; choose to create a single thing as shown in Figure2.3.

Figure2.3

Give the name MyRPi and select Next; On the Add a Certificate page, choose Create Certificate as shown in Figure2.4

Figure2.4

For the certificate of the thing , choose to download; Select Activate; For the policy of adding things , select PlantWateringPolicy. Then our thing has been created as shown in Figure2.5.

Figure2.5

2.3 Email Alerts for Low Water Content Readings

In this step, we set up Amazon Simple Notification Service (Amazon SNS) to automatically send an email alert to the owner of the green plant when the soil moisture content is too low, alerting the watering.

Create an AWS IoT rule to trigger email alerts through Amazon SNS. Select Actions and select Create a rule; Enter a name for this rule MyRPiLowMoistureAlertRule; For rule query statements, enter the following AWS IoT SQL statement.

SELECT* FROM '$aws/things/MyRPi/shadow/update/accepted' WHERE state.reported.moisture = 'low'

For setting one or more actions , select Add Action choose to send the message as an SNS push notification; All are as shown in Figure2.6

Figure2.6

Set up Amazon SNS to send messages to my email inbox via an Amazon SNS topic; For the topic ARN , select the ARN for the topic; For the agreement , select Email and enter my email address.

All are as shown in Figure2.7

Figure2.7

2.4 Simulate random water content

In this step, we can then press these readings to the relevant shadow in the AWS IoT. When the reading is too low, Amazon SNS automatically sends an email alert to the owner of the green plant.

Create a new file on the development machine using the code (plantwater.py) in the Github as shown in Figure2.8

Figure2.9

Use the pycharm to run the code.

2.5 Results

My NUS email has received the alerts from AWS as shown in Figure2.9, which indicate our first goal of sending data from a computer to AWS through MQTT protocol is successful.

Figure2.9

3.Sending data FD001 to AWS

3.1 Introduction of the database

The data provided is from a high fidelity system level engine simulation designed to simulate nominal and fault engine degradation over a series of flights. The flights are full flight recording sampled at 1 Hz and consists of 30 engine and flight condition parameters. Each flight contains 7 unique flight conditions for an approximately 90 min flight including ascent to cruise at 35K ft and descent back to sea level. The structure of the plane sensor is as shown in Figure3.1.

Figure3.1

FD001 and FD002 are input data for machine fleets. Sensor names can be found in CMAPSS file. The column names for the Text Input Files are: id, cycle, os1, os2, os3, sensor1, ...., sensor22.

3.2 Create the AWS IoT Policy for FD001

This step is similar to the 2.1 Create the AWS IoT Policy in the report. I will show the change between them.

Sign in to the AWS Management(https://aws.amazon.com) and open the AWS IoT console; Enter the IoT Core and select the policies.

select Create Policy and provide a name Mypo1_A0206839W; Enter iot:* for action and * for ARN as shown in Figure3.2;

Figure3.2

3.3 Create the things of FD001

This step is similar to the 2.2 Create the things in the report. I will show the change between them.

Select Manage and choose Things and use Create; choose to create a single thing;

Give the name Mything1_A0206839W and select Next; On the Add a Certificate page, choose Create Certificate as shown in Figure3.3

Figure3.3

For the certificate of the thing , choose to download of the public key , the private key, and the root CA of the AWS IoT.

Select Activate; For the policy of adding things , select Mypo1_A0206839W as shown in Figure2.4.

Figure3.4

Then Mything1_A0206839W has been created as shown in Figure3.5.

Figure3.4

3.4 Create AWS IoT rule and AWS DynamoDB

In this step, we need to use DynamoDB rules to collect information from MQTT message, and send those information to the DynamoDB database, where we can analyze the them.

Select Actions and select Create a rule; Enter a name for this rule Role1; For rule query statements, enter the following AWS IoT SQL statement as shown in Figure3.5

SELECT state.reported.* FROM '$aws/things/ Mything1_A0206839W /shadow/update/accepted'

Figure3.5

For setting one or more actions , select Add Action choose to as shown in Figure3.6

Figure3.6

On the Configure action page, choose Create a new resource and go to the Amazon DynamoDB page, choose Create table as shown in Figure3.7.

Figure3.7

On the Create DynamoDB Table page, enter a name A0206839W. Choose 'id' as the partition key and 'timestamp' as the sort key as shown in Figure3.8

Figure3.8

choose Create to create table A0206839W as shown in Figure3.9

Figure3.9

Back to the configure action; in Create a new role , enter the name role_A0206839W and use Create as shown in Figure3.10

Figure3.10

Select Create Rule to create a rule, the rule Role1 as shown in Figure3.11 .

Figure3.11

3.5 Use python codes to meet the requirements

3.5.1 Requirement

The requirements of the Publish pre-defined engine data to AWS(Step 2) is as follow:

a. Download data for engines FD001 and FD002 from Github .

b. Modify the Jupyter notebook in Step 1 to read and publish data from trainFD001.txt to your thing under AWS IoT platform at the rate of 10 seconds per row. Overwrite column 'id' of the engine as 'FD001' + id; add one more columns 'timestamp' as timestamp in UTC; also add one more column that contains your Matric number. 

c. Setup an Act under AWS IoT platform to define a rule that splits the message into columns of a DynamoDB table.

d. Check under AWS DynamoDB service if your table is populated correctly. 

3.5.2 Methods

I design the method to satisfy all of the requirements step by step, which will show in the following.

The completed codes are in the appendix(main1.py)

For the a.,

I has download from the Github;

For the b.,

First, I design some codes to meet the specific requirement:

Read and publish data from FD001.txt under AWS IoT platform at the rate of 10 seconds per row

time.sleep(10)

Overwrite column 'id' of the engine as 'FD001' + id

content.append(str('"' + 'FD002_' + x[0] + '",'))

Add one more columns 'timestamp' as timestamp in UTC

tim = str(datetime.datetime.utcnow());#tim= timestamp

Add one more column that contains your Matric number

maNum = 'A0206839W'

Second, I design two function to achieve specific form

# to form the title of the column of the content

def form_title ():

#change
date into joson form to send
def form_joson (content,x,title,tim,maNum):

Third, the codes of reading file FD001.txt and sending it to DynamoDB is as follow:

fp = open('train_FD002.txt', 'r')
#title which satisfy
the request by teacher
title=form_title ();
for reader in fp.readlines():
    x = reader.split(" ")
    #time
    tim = str(datetime.datetime.utcnow());
    #the content of the message
    content = [];
    #the joson form of data to send
    send=form_joson(content, x, title, tim, maNum)
    print(send)
    #send to the dynamnoDB
    myDeviceShadow.shadowUpdate(send, myShadowUpdateCallback,
5)

time.sleep(10)
fp.close();

For the c.,

I have finish it on the 3.4 create AWS IoT rule and AWS DynamoDB

For the d.

I will show the outcome in the 3.6 Results and Analysis

3.6 Results and Analysis

Results

The results is shown as Figure3.12.

Figure3.12

Analysis

From the results, we can conclude the
following things:

1. the DynamoDB table A0206839W has received the data sent by Mything1_A0206839W successfully,
which indicates the whole operations of AWS IoT are correct.

2. By subtracting one line, the ‘id’ is FD001_1, the ‘timestamp’ is 2019-09-20
11:38:57.733647, and the other ‘sensorx’
is the same as the ‘FD001.txt’ data,
which indicates the python codes satisfy the requirements.

4. Sending data from FD001 and
FD002 to AWS

We
are going to simulate two IoT things. Therefore, we need to use another IoT thing to complete.

As the whole processes of another thing are
similar to the 3.sending data FD001 to
AWS, my report will omit duplicates.

4.1 Create the Policy, Thing and DynamoDB
for FD002

4.1.1 Policy

Select Create
Policy and provide a name Mypo2_A0206839W; Enter iot:* for action and * for ARN, the policy is as shown in Figure4.1

Figure4.1

4.1.2 Thing

Select Manage and choose Things and use Create; choose to create a
single thing;

Give the name Mything2_A0206839W;
Mything2_A0206839W has been created as shown in Figure4.2.

Figure4.2.

4.1.3 DynamoDB

As we need to simulate the two
"things" to run in parallel, we have to use the same DynamoDB(A0206839W) to the first thing.

4.2 Create IoT rule for both FD001
and FD002

We need to modify the rule
Role1 to meet the requirements of step3.
To be specific, we need to change the rule query statements as follow:

Change

SELECT state.reported.* FROM '$aws/things/ Mything1_A0206839W
/shadow/update/accepted'

Into

SELECT state.reported.* FROM '$aws/things/ +
/shadow/update/accepted'

In that way, the rule can choose both Mything1_A0206839W and Mything2_A0206839W
to the DynamoDB and we can simulate the two "things" to run in
parallel.

The updated Role1 is shown
in Figure4.3

Figure4.3

4.3 Results and Analysis

The completed codes of Mything2_A0206839W are in the appendix.

I use the two .py(main1.py
and main2.py in the appendix) run at the same time, which is aimed at stimulating
the two "things" to run in parallel to publish data.

Results

The results in DynamoDB are shown as Figure4.4

Figure4.3

Analysis

From the results, we can conclude the following things:

1. the DynamoDB table A0206839W has received the data sent by Mything2_A0206839W successfully, which
indicates the whole operations of Mything2_A0206839W
are correct.

2. By studying the second figure of Figure4.3, we find that the DynamoDB has received both FD001 and
FD002 data and stored them individually. This indicates my methods satisfy the
requirements of step3.

5.Visualization of data

5.1 Get AWS access key

Go
to the AWS account and click the zyc18 in the Users as shown in Figure5.1

Figure5.1

Add
the Add permissions and select AmazonDynamo Db as shown in Figure5.2

Figure5.2

Download the access keys from
it as shown in Figure5.3

Figure5.3

Then we can use those keys to permit us to
subtract data from dynamoDB table.

5.2 Redash environment

Introduction

Redash is an Open Source Company to
democratize data and make data driven decision making easy.

We
can query Amazon DynamoDB by using its natural syntax, enjoy live auto-complete
and explore Amazon DynamoDB schema easily in Redash's cloud-based query editor;

And we can visualize Amazon DynamoDB data and gather it into
thematic dashboards from multiple sources, share the story your data tells with
team or extermal partners.

Configuration

On the Redash home page(https://app.redash.io), open the New Data Source page and choose to Create a New Data Source, and select DynamoDB;

Put the Name, Access Key and Secret Key of our AWS DynamoDB as shown
in in Figure5.4.

Figure5.4

5.3 Obtain and Visualize the data

Obtain

Open
the Queries page and use SQL
language to query data from the DynamoDB
table as shown in Figure5.5

Figure5.5

Now we can obtain the data from the DynamoDB table of A0206839W

Visualize

Open the New Visualization to visualize the data as shown in Figure5.6

Figure5.6

We can use this tools to visualize data
which we want to study.

5.4 Examples of Visualization by Redash

Using this visualization, I choose several
representative examples to show the relationship between the data.

Chat line

I take
time as X Colum and os1,os2,os3 as
Y Colum, the outcome as shown in Figure5.7

Figure5.7

Chat bar

I take time
as X Colum and sensor1, sensor2, sensor3
as Y Colum, the outcome as shown in Figure5.8

Figure5.8

Chat Scatter

I take time
as X Colum and sensor16, sensor17,
sensor18 as Y Colum, the outcome as shown in Figure5.9

Boxplot

The outcome of all the columns as shown in
Figure5.10

6.Anylasis of Motor Vehicle
Inspection data source

6.1 Introduction of data source

The data source is the about annual motor vehicle inspection - passing
pate of motor vehicles on first inspection from the https://data.gov.sg.

The columns of the data source are shown as Figure5.1

Figure5.1

Some samples of the data are
shown as Figure5.2

Figure5.2

The goals of my IoT are to collect
those source from specific monitors, store
them in the AWS DynamoDB and do the
analysis through the Visualization.

6.2 Create the Policy, Thing, Rule
and DynamoDB

As the whole processes are similar to the past
operation, I will omit duplicates and point to the emphasis.

6.1.1 Policy

Select Create
Policy and provide a name Mypo2_A0206839W;
Enter iot:* for action and * for ARN, the policy is as shown in Figure6.1

Figure6.1

6.1.2 Thing

Select Manage and choose Things and use Create; choose to create a
single thing;

Give the name Mypas_rate_A0206839W;
The thing has been created as shown in Figure6.2.

Figure6.2

6.1.3 DynamoDB

On the Create DynamoDB Table
page, enter a name pass_rate. Choose
'year' as the partition key and 'timestamp'
as the sort key. The DynamoDB table as
shown in Figure6.3

Figure6.3

6.1.4 Rule

Select Actions
and select Create a rule; Enter a name Rule_pasrate for this rule Role1;

For setting
one or more actions , select Add
Action and choose the DynamoDB table

The rule has been created as shown in
Figure6.4

Figure6.4

6.3 Data processing and Results

We
need to change the .csv data into .json data. In addition, in order to show
more clearly, we need to add one more column 'timestamp' as timestamp and add
one more column that contains Matric number.

The
completed codes are in the appendix(main3.py)

Methods

Change the .csv data into .json data

with open("pas_rate.csv", "r", encoding="utf-8") as fp:
    # reader是一个迭代器
    reader = csv.reader(fp)
    # 执行一次next，指针跳过一位，可以不获取标题
    next(reader)
    for x in reader:
        content = [];
        tim = str(datetime.datetime.utcnow());
        content.append(str(title[0] + ':'))
        content.append(str('"' + x[0] + '",'))
        for i in range(1,8):
         content.append(str(title[i] + ':'))
         if i==1:
          content.append(str('"' + tim + '",'))
         elif i==2:
            content.append(str('"' + maNum + '",'))
         else:
             content.append(str('"' + x[i-2] + '",'))
        cc=''.join(content)[:-1]

Add one more column 'timestamp' as timestamp

tim = str(datetime.datetime.utcnow());#tim= timestamp

Add one more column that contains Matric number

maNum = 'A0206839W'

Results

The results in DynamoDB pass_rate are shown as Figure6.5

Figure6.5

Analysis

From the results, we can conclude the following things:

1. the DynamoDB table pass_rate has received the data sent by Mypas_rate_A0206839W
successfully, which indicates the whole operations of Mypas_rate_A0206839W are correct.

2. We find that the pass_rate has the following columns:

'year','timestamp','MatricNumber','type','age',‘number_reported','number_passed','passing_rate'
with correct data.

This indicates my methods satisfy the requirements of step5.

6.4 Visualization of data source

Using the same methods of 5.Visualization
of data, we can study the annual motor vehicle inspection by Redash and visualize them.

6.4.1 Obtain the data

Open the Queries page and use SQL language to query data from the DynamoDB
table pass_rate as shown in Figure6.6

Figure6.6

6.4.2 Visualize and Analyze the data

Open the New Visualization to visualize the data. We can use this function
to draw some conclusions which may
help us to determinate vehicles.

I take type
as X Colum and passing_rate as Y
Colum, the outcome as shown in Figure6.7

From the figure we can conclude that Other Vehicles has the lowest pass
rate and Taxi has the highest pass rate.

Figure6.7

I take year
as X Colum and passing_rate as Y
Colum, the outcome as shown in Figure6.8

From the figure we can conclude that pass rate become better by time, which
indicate the vehicles are more and more reliable.

Figure6.8

I take age
as X Colum and reported and passed
as Y Colum, the outcome as shown in Figure6.9

From the figure we can conclude thatmost people love to examine their
vehicles within 3 years..

7.Conclusion

I completed the following goals that the teacher requested:

1. Understand
how to populate, send data from a computer to AWS through MQTT protocol.

2.
Publish pre-defined engine data to AWS.