Distributed air monitoring system in geese factory: how to build the cloud capability with serverless as the core

Foregoing

The preface to this series , The hardware layer has been described in detail , Explain the performance of the equipment 、 Development 、 The process of filling, etc . This article will discuss the relevant processes after the data is put into the cloud , To illustrate .

Due to the continuous construction of the project platform , Currently, the open source information is the second edition , Therefore, the content of this article is based on the ability of this version . This version architecture further optimizes data authentication and database protection .

This article will be divided into two parts ：

• Architecture Composition ： From a technical point of view , Dismantling design logic
• Module implementation ： Function angle , Explain the implementation logic

【 notes 1】 Second edition home page effect

【 notes 2】 The first page effect of the third edition under construction

【 notes 3】 In the third edition under construction , Joined the The log service 、 Object storage 、 Applet functions . Provides flexible log storage and high-speed retrieval capabilities 、 Low cost database backup + recovery + Migration synchronization capabilities 、 Mobile interaction and other capabilities .

Architecture Composition

Phase II Architecture

notes ： The light blue color indicates the key points of architecture changes in phase I .

Deployment dimension

• terminal ： The equipment deployed at the monitoring point . The design has been described earlier in this series .
• Cloud ： Platforms deployed on the cloud . The relevant design will be described in this paper .

Functional dimension

The basic components of platform design , The logical split . And according to the design requirements of the architecture layer , The specific products of matching selection .

Specific functions and selection instructions ：

• Monitoring nodes ： It's made up of hard and soft parts . be based on Tencent Internet of things terminal operating system TencentOS tiny Developing firmware , take Hardware devices compatible with Tencent cloud And Tencent cloud IoT explorer The protocol level docking is carried out .
• Local gateway ： This part focuses on communication protocol conversion , Don't parse specific data , Just follow the instructions above .
• Internet of things access layer ： be based on Tencent cloud IoT explorer product , Provides decryption 、 Access 、 Forwarding related functions . After getting the data , Based on http The push of the agreement , Pass it downstream API gateway .
• Middleware layer ： be based on API gateway It provides convergence and load balancing capabilities , Based on Authentication 、 Flow control, etc , Convergence cloud function , Provide reliable data input and output services .
• Logic layer ： be based on Cloud functions , The receiving data is put into storage （ Data solidification ）、 Regular statistical summary （ Data analysis ）、 Output application data （ Data output , Show support ） The ability of .
• Storage layer ： be based on Cloud database TencentDB for MySQL, Monitoring data 、 Summary data 、 Storage of platform configuration .
• Presentation layer ： be based on Tencent cloud map , Provides a map display 、 Data presentation 、 Line chart display, etc . And configuration provides linkage processing , Improved response speed , Reduce the repeated transmission of resource files .

Module implementation

The whole system , Take data as the core , The relevant design has been carried out , So this part has two components ：

• data structure ： Explain the core data structure and association mode of each link .
• The module specification ： Explain the implementation of each function module and the corresponding source code .

data structure

IoT explorer( Internet of things development platform ) Push data structure

With API Gateway push in , By the cloud function in the environment variable event The original data extracted from .

The main part of it ：

• Not 'body' part , by api Gateway related information
• 'body' = IoT explorer Raw data
• 'body' = The original data reported by the terminal node , the base64 code

{
"body": "{\"payload\":\"eyJtZXRob2QiOiJyZXBvcnQiLCJjbGllbnRUb2tlbiI6IjIwMjAtMDgtMThUMTE6MTc6NDkuNDAzWiIsInBhcmFtcyI6eyJQTTFfQ0YxIjo1LCJQTTJkNV9DRjEiOjYsIlBNMTBfQ0YxIjo3LCJQTTEiOjUsIlBNMmQ1Ijo2LCJQTTEwIjo3LCJwYXJ0aWNsZXNfMGQzIjo3OTIsInBhcnRpY2xlc18wZDUiOjI1NCwicGFydGljbGVzXzEiOjE1LCJwYXJ0aWNsZXNfMmQ1IjozLCJwYXJ0aWNsZXNfNSI6MywicGFydGljbGVzXzEwIjowLCJ2ZXJzaW9uIjoxNTEsIkVycm9yIjowfX0=\", \"seq\":18639, \"timestamp\":1597749469,\"topic\":\"$thing/up/property/?????/?????\", \"devicename\":\"?????\", \"productid\":\"?????\"}",
"headerParameters": {},
"headers": {
"accept-encoding": "gzip",
"content-length": "511",
"content-type": "application/json",
"endpoint-timeout": "15",
"host": "service-???-???.gz.apigw.tencentcs.com",
"user-agent": "Go-http-client/1.1",
"x-anonymous-consumer": "true",
"x-api-requestid": "???03a993936ae191f44651???",
"x-b3-traceid": "???03a993936ae191f44651???",
"x-qualifier": "$LATEST"
},
"httpMethod": "POST",
"path": "/??????",
"pathParameters": {},
"queryString": {},
"queryStringParameters": {},
"requestContext": {
"httpMethod": "POST",
"identity": {},
"path": "/??????",
"serviceId": "service-?????",
"sourceIp": "?.?.?.?",
"stage": "release"
}
}

'body' Explain base64 After the coding ：

{
"clientToken": "2020-08-18T11:17:49.403Z",
"method": "report",
"params": {
"Error": 0,
"PM1": 5,
"PM10": 7,
"PM10_CF1": 7,
"PM1_CF1": 5,
"PM2d5": 6,
"PM2d5_CF1": 6,
"particles_0d3": 792,
"particles_0d5": 254,
"particles_1": 15,
"particles_10": 0,
"particles_2d5": 3,
"particles_5": 3,
"version": 151
}
}

Database form structure

Database detailed format , Please refer to github-sql

The purpose of each watch is ：

• config： System configuration table
• space： Location table . Record node deployment location .
• client： Node table . Record node information and the latest key information , relation space surface .
• base_data： Log table . Record every reported information , relation space、client surface .
• aggregate_data： Summary information table . Record by hour 、 The data information collected in the day interval .
• temp_base_data： Temporary log table . If the reported device name is not in the node table , Then save it in this form , It is mainly used for debugging .

API Interface data structure

The module specification

Data warehousing

• Module function ： receive IoT explorer Push data , After matching and converting , Into the database
• Use the product ：API gateway 、 Serverless cloud function
• Associated source code ：github-upload
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter

The terminal is updated online

• Module function ： Press 3 Minutes to trigger , Update terminal information , For quick query and display
• Use the product ： Cloud functions
• Associated source code ：github-client_active_new
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern

Data statistics

• Module function ： Trigger by hour and day , Statistical monitoring data , For quick query and display
• Use the product ： Cloud functions
• Associated source code ：github-aggregate
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern

Data interface - Terminal summary information table

• Module function ： adopt API Gateway provides data query capability , Response cloud call , Provide terminal summary information table
• Use the product ：API gateway 、 Cloud functions
• Associated source code ：github-client_info
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter

Data interface - Terminal single point information table

• Module function ： adopt API Gateway provides data query capability , Response cloud call , Provide terminal single point information table
• Use the product ：API gateway 、 Cloud functions
• Associated source code ：github-client_data
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter

Data presentation

• Module function ： Provide WEB End show ability , Including map points 、 The list shows 、 Broken line display, etc
• Use the product ： Tencent cloud map
• Associated source code ： nothing , Currently, it is manually configured
• Deployment way ： Please refer to the relevant articles Online formaldehyde monitoring “4.2.4 Tencent cloud map ” chapter , But when selecting data sources , Change to use “API”. among “ home page ” Map use “ Data interface - Terminal summary information table ” Interface 、“ Single point data page ” Use “ Data interface - Terminal single point information table ” Interface . Specific calling interface URL, Please correspond to API Service links provided by the gateway .

Phase II cost analysis

The number of tables in the figure above is for each node 15 Report the calculation results once every second .

The hardware part is due to the difference between manufacturer and purchase quantity , The price is different . Here is the cost of cloud services , If 10000 Node size , Every node reports every minute , Annual cost of single node cloud service ：

• Phase II Architecture 3.335 element
  ( database / year + Cloud chart / year )/10000+(API gateway + Cloud functions )/4=(468+48)/10000+(9+2.1)/4

• Three phase structure 1.09 element
  ( Message queue / year + database / year *2+ Cloud chart / year )/10000+( Cloud functions + The log service + Object storage )/4=(3228+468*2+48)/10000+(2.1+2.24+0.02)/4

Phase II no object storage and log service module , But it has been used for several months in the third phase under construction , So there are actual data to be included in the statistics .

The follow-up plan

Complete the third phase of construction , The plan involves ：

• Interaction optimization (done): Enable space surface , Show location .
• Interaction building (doing): Add applet capabilities , Provide mobile query capability , And alarm push capability .
• Disaster recovery optimization 1(done): increase Object storage , Provide low cost data backup 、 recovery 、 Asynchronous migration synchronization capability .
• Disaster recovery optimization 2(todo): increase Message queue CKafka, Dealing with network anomalies 、 The risk of data accumulation and loss in case of database exception .
• Disaster recovery optimization 3(doing): increase The log service , For data reconciliation , Dealing with multi link data processing , Possible data loss or confusion .

The third phase of the structure plan is announced

notes ： The light blue color shows the key points of architecture changes in phase II .

Part of the third edition is finished

WEB End show

Small program end Demo

One More Thing

Experience Tencent cloud immediately Serverless Demo, receive Serverless New user gift pack serverless/start

Welcome to visit ：Serverless Chinese net ！