Foregoing
The preface to this series , The hardware layer has been described in detail , Explain the performance of the equipment 、 Development 、 The process of filling, etc . This article will discuss the relevant processes after the data is put into the cloud , To illustrate .
Due to the continuous construction of the project platform , Currently, the open source information is the second edition , Therefore, the content of this article is based on the ability of this version . This version architecture further optimizes data authentication and database protection .
This article will be divided into two parts :
- Architecture Composition : From a technical point of view , Dismantling design logic
- Module implementation : Function angle , Explain the implementation logic
【 notes 1】 Second edition home page effect
【 notes 2】 The first page effect of the third edition under construction
【 notes 3】 In the third edition under construction , Joined the The log service 、 Object storage 、 Applet functions . Provides flexible log storage and high-speed retrieval capabilities 、 Low cost database backup + recovery + Migration synchronization capabilities 、 Mobile interaction and other capabilities .
Architecture Composition
Phase II Architecture
notes : The light blue color indicates the key points of architecture changes in phase I .
Deployment dimension
- terminal : The equipment deployed at the monitoring point . The design has been described earlier in this series .
- Cloud : Platforms deployed on the cloud . The relevant design will be described in this paper .
Functional dimension
The basic components of platform design , The logical split . And according to the design requirements of the architecture layer , The specific products of matching selection .
Specific functions and selection instructions :
- Monitoring nodes : It's made up of hard and soft parts . be based on Tencent Internet of things terminal operating system TencentOS tiny Developing firmware , take Hardware devices compatible with Tencent cloud And Tencent cloud IoT explorer The protocol level docking is carried out .
- Local gateway : This part focuses on communication protocol conversion , Don't parse specific data , Just follow the instructions above .
- Internet of things access layer : be based on Tencent cloud IoT explorer product , Provides decryption 、 Access 、 Forwarding related functions . After getting the data , Based on http The push of the agreement , Pass it downstream API gateway .
- Middleware layer : be based on API gateway It provides convergence and load balancing capabilities , Based on Authentication 、 Flow control, etc , Convergence cloud function , Provide reliable data input and output services .
- Logic layer : be based on Cloud functions , The receiving data is put into storage ( Data solidification )、 Regular statistical summary ( Data analysis )、 Output application data ( Data output , Show support ) The ability of .
- Storage layer : be based on Cloud database TencentDB for MySQL, Monitoring data 、 Summary data 、 Storage of platform configuration .
- Presentation layer : be based on Tencent cloud map , Provides a map display 、 Data presentation 、 Line chart display, etc . And configuration provides linkage processing , Improved response speed , Reduce the repeated transmission of resource files .
Module implementation
The whole system , Take data as the core , The relevant design has been carried out , So this part has two components :
- data structure : Explain the core data structure and association mode of each link .
- The module specification : Explain the implementation of each function module and the corresponding source code .
data structure
IoT explorer( Internet of things development platform ) Push data structure
With API Gateway push in , By the cloud function in the environment variable event The original data extracted from .
The main part of it :
- Not 'body' part , by api Gateway related information
- 'body' = IoT explorer Raw data
- 'body' = The original data reported by the terminal node , the base64 code
{
"body": "{\"payload\":\"eyJtZXRob2QiOiJyZXBvcnQiLCJjbGllbnRUb2tlbiI6IjIwMjAtMDgtMThUMTE6MTc6NDkuNDAzWiIsInBhcmFtcyI6eyJQTTFfQ0YxIjo1LCJQTTJkNV9DRjEiOjYsIlBNMTBfQ0YxIjo3LCJQTTEiOjUsIlBNMmQ1Ijo2LCJQTTEwIjo3LCJwYXJ0aWNsZXNfMGQzIjo3OTIsInBhcnRpY2xlc18wZDUiOjI1NCwicGFydGljbGVzXzEiOjE1LCJwYXJ0aWNsZXNfMmQ1IjozLCJwYXJ0aWNsZXNfNSI6MywicGFydGljbGVzXzEwIjowLCJ2ZXJzaW9uIjoxNTEsIkVycm9yIjowfX0=\", \"seq\":18639, \"timestamp\":1597749469,\"topic\":\"$thing/up/property/?????/?????\", \"devicename\":\"?????\", \"productid\":\"?????\"}",
"headerParameters": {},
"headers": {
"accept-encoding": "gzip",
"content-length": "511",
"content-type": "application/json",
"endpoint-timeout": "15",
"host": "service-???-???.gz.apigw.tencentcs.com",
"user-agent": "Go-http-client/1.1",
"x-anonymous-consumer": "true",
"x-api-requestid": "???03a993936ae191f44651???",
"x-b3-traceid": "???03a993936ae191f44651???",
"x-qualifier": "$LATEST"
},
"httpMethod": "POST",
"path": "/??????",
"pathParameters": {},
"queryString": {},
"queryStringParameters": {},
"requestContext": {
"httpMethod": "POST",
"identity": {},
"path": "/??????",
"serviceId": "service-?????",
"sourceIp": "?.?.?.?",
"stage": "release"
}
}
'body' Explain base64 After the coding :
{
"clientToken": "2020-08-18T11:17:49.403Z",
"method": "report",
"params": {
"Error": 0,
"PM1": 5,
"PM10": 7,
"PM10_CF1": 7,
"PM1_CF1": 5,
"PM2d5": 6,
"PM2d5_CF1": 6,
"particles_0d3": 792,
"particles_0d5": 254,
"particles_1": 15,
"particles_10": 0,
"particles_2d5": 3,
"particles_5": 3,
"version": 151
}
}
Database form structure
Database detailed format , Please refer to github-sql
The purpose of each watch is :
- config: System configuration table
- space: Location table . Record node deployment location .
- client: Node table . Record node information and the latest key information , relation space surface .
- base_data: Log table . Record every reported information , relation space、client surface .
- aggregate_data: Summary information table . Record by hour 、 The data information collected in the day interval .
- temp_base_data: Temporary log table . If the reported device name is not in the node table , Then save it in this form , It is mainly used for debugging .
API Interface data structure
The module specification
Data warehousing
- Module function : receive IoT explorer Push data , After matching and converting , Into the database
- Use the product :API gateway 、 Serverless cloud function
- Associated source code :github-upload
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter
The terminal is updated online
- Module function : Press 3 Minutes to trigger , Update terminal information , For quick query and display
- Use the product : Cloud functions
- Associated source code :github-client_active_new
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern
Data statistics
- Module function : Trigger by hour and day , Statistical monitoring data , For quick query and display
- Use the product : Cloud functions
- Associated source code :github-aggregate
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern
Data interface - Terminal summary information table
- Module function : adopt API Gateway provides data query capability , Response cloud call , Provide terminal summary information table
- Use the product :API gateway 、 Cloud functions
- Associated source code :github-client_info
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter
Data interface - Terminal single point information table
- Module function : adopt API Gateway provides data query capability , Response cloud call , Provide terminal single point information table
- Use the product :API gateway 、 Cloud functions
- Associated source code :github-client_data
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter
Data presentation
- Module function : Provide WEB End show ability , Including map points 、 The list shows 、 Broken line display, etc
- Use the product : Tencent cloud map
- Associated source code : nothing , Currently, it is manually configured
- Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.4 Tencent cloud map ” chapter , But when selecting data sources , Change to use “API”. among “ home page ” Map use “ Data interface - Terminal summary information table ” Interface 、“ Single point data page ” Use “ Data interface - Terminal single point information table ” Interface . Specific calling interface URL, Please correspond to API Service links provided by the gateway .
Phase II cost analysis
The number of tables in the figure above is for each node 15 Report the calculation results once every second .
The hardware part is due to the difference between manufacturer and purchase quantity , The price is different . Here is the cost of cloud services , If 10000 Node size , Every node reports every minute , Annual cost of single node cloud service :
-
Phase II Architecture 3.335 element
( database / year + Cloud chart / year )/10000+(API gateway + Cloud functions )/4=(468+48)/10000+(9+2.1)/4 -
Three phase structure 1.09 element
( Message queue / year + database / year *2+ Cloud chart / year )/10000+( Cloud functions + The log service + Object storage )/4=(3228+468*2+48)/10000+(2.1+2.24+0.02)/4
Phase II no object storage and log service module , But it has been used for several months in the third phase under construction , So there are actual data to be included in the statistics .
The follow-up plan
Complete the third phase of construction , The plan involves :
- Interaction optimization (done): Enable space surface , Show location .
- Interaction building (doing): Add applet capabilities , Provide mobile query capability , And alarm push capability .
- Disaster recovery optimization 1(done): increase Object storage , Provide low cost data backup 、 recovery 、 Asynchronous migration synchronization capability .
- Disaster recovery optimization 2(todo): increase Message queue CKafka, Dealing with network anomalies 、 The risk of data accumulation and loss in case of database exception .
- Disaster recovery optimization 3(doing): increase The log service , For data reconciliation , Dealing with multi link data processing , Possible data loss or confusion .
The third phase of the structure plan is announced
notes : The light blue color shows the key points of architecture changes in phase II .
Part of the third edition is finished
WEB End show
Small program end Demo
One More Thing
Experience Tencent cloud immediately Serverless Demo, receive Serverless New user gift pack serverless/start
Welcome to visit :Serverless Chinese net !