Goose factory distributed air monitoring system: how to build cloud capability with serverless as the core

Tencent cloud serverless 2020-11-09 22:32:20
goose factory distributed air monitoring


Foregoing

The preface to this series , The hardware layer has been described in detail , Explain the performance of the equipment 、 Development 、 The process of filling, etc . This article will discuss the relevant processes after the data is put into the cloud , To illustrate .

Due to the continuous construction of the project platform , Currently, the open source information is the second edition , Therefore, the content of this article is based on the ability of this version . This version architecture further optimizes data authentication and database protection .

This article will be divided into two parts :

  • Architecture Composition : From a technical point of view , Dismantling design logic
  • Module implementation : Function angle , Explain the implementation logic

【 notes 1】 Second edition home page effect

v2_yuntu_index.png

【 notes 2】 The first page effect of the third edition under construction

v3_yuntu_index.png

【 notes 3】 In the third edition under construction , Joined the The log service Object storage 、 Applet functions . Provides flexible log storage and high-speed retrieval capabilities 、 Low cost database backup + recovery + Migration synchronization capabilities 、 Mobile interaction and other capabilities .

Architecture Composition

Phase II Architecture

v2_ Architecture diagram .png

notes : The light blue color indicates the key points of architecture changes in phase I .

Deployment dimension

  • terminal : The equipment deployed at the monitoring point . The design has been described earlier in this series .
  • Cloud : Platforms deployed on the cloud . The relevant design will be described in this paper .

Functional dimension

The basic components of platform design , The logical split . And according to the design requirements of the architecture layer , The specific products of matching selection .

Specific functions and selection instructions :

  • Monitoring nodes : It's made up of hard and soft parts . be based on Tencent Internet of things terminal operating system TencentOS tiny Developing firmware , take Hardware devices compatible with Tencent cloud And Tencent cloud IoT explorer The protocol level docking is carried out .
  • Local gateway : This part focuses on communication protocol conversion , Don't parse specific data , Just follow the instructions above .
  • Internet of things access layer : be based on Tencent cloud IoT explorer product , Provides decryption 、 Access 、 Forwarding related functions . After getting the data , Based on http The push of the agreement , Pass it downstream API gateway .
  • Middleware layer : be based on API gateway It provides convergence and load balancing capabilities , Based on Authentication 、 Flow control, etc , Convergence cloud function , Provide reliable data input and output services .
  • Logic layer : be based on Cloud functions , The receiving data is put into storage ( Data solidification )、 Regular statistical summary ( Data analysis )、 Output application data ( Data output , Show support ) The ability of .
  • Storage layer : be based on Cloud database TencentDB for MySQL, Monitoring data 、 Summary data 、 Storage of platform configuration .
  • Presentation layer : be based on Tencent cloud map , Provides a map display 、 Data presentation 、 Line chart display, etc . And configuration provides linkage processing , Improved response speed , Reduce the repeated transmission of resource files .

Module implementation

The whole system , Take data as the core , The relevant design has been carried out , So this part has two components :

  • data structure : Explain the core data structure and association mode of each link .
  • The module specification : Explain the implementation of each function module and the corresponding source code .

data structure

IoT explorer( Internet of things development platform ) Push data structure

With API Gateway push in , By the cloud function in the environment variable event The original data extracted from .

The main part of it :

  • Not 'body' part , by api Gateway related information
  • 'body' = IoT explorer Raw data
  • 'body' = The original data reported by the terminal node , the base64 code
{
"body": "{\"payload\":\"eyJtZXRob2QiOiJyZXBvcnQiLCJjbGllbnRUb2tlbiI6IjIwMjAtMDgtMThUMTE6MTc6NDkuNDAzWiIsInBhcmFtcyI6eyJQTTFfQ0YxIjo1LCJQTTJkNV9DRjEiOjYsIlBNMTBfQ0YxIjo3LCJQTTEiOjUsIlBNMmQ1Ijo2LCJQTTEwIjo3LCJwYXJ0aWNsZXNfMGQzIjo3OTIsInBhcnRpY2xlc18wZDUiOjI1NCwicGFydGljbGVzXzEiOjE1LCJwYXJ0aWNsZXNfMmQ1IjozLCJwYXJ0aWNsZXNfNSI6MywicGFydGljbGVzXzEwIjowLCJ2ZXJzaW9uIjoxNTEsIkVycm9yIjowfX0=\", \"seq\":18639, \"timestamp\":1597749469,\"topic\":\"$thing/up/property/?????/?????\", \"devicename\":\"?????\", \"productid\":\"?????\"}",
"headerParameters": {},
"headers": {
"accept-encoding": "gzip",
"content-length": "511",
"content-type": "application/json",
"endpoint-timeout": "15",
"host": "service-???-???.gz.apigw.tencentcs.com",
"user-agent": "Go-http-client/1.1",
"x-anonymous-consumer": "true",
"x-api-requestid": "???03a993936ae191f44651???",
"x-b3-traceid": "???03a993936ae191f44651???",
"x-qualifier": "$LATEST"
},
"httpMethod": "POST",
"path": "/??????",
"pathParameters": {},
"queryString": {},
"queryStringParameters": {},
"requestContext": {
"httpMethod": "POST",
"identity": {},
"path": "/??????",
"serviceId": "service-?????",
"sourceIp": "?.?.?.?",
"stage": "release"
}
}

'body' Explain base64 After the coding :

{
"clientToken": "2020-08-18T11:17:49.403Z",
"method": "report",
"params": {
"Error": 0,
"PM1": 5,
"PM10": 7,
"PM10_CF1": 7,
"PM1_CF1": 5,
"PM2d5": 6,
"PM2d5_CF1": 6,
"particles_0d3": 792,
"particles_0d5": 254,
"particles_1": 15,
"particles_10": 0,
"particles_2d5": 3,
"particles_5": 3,
"version": 151
}
}

Database form structure

Database detailed format , Please refer to github-sql

The purpose of each watch is :

  • config: System configuration table
  • space: Location table . Record node deployment location .
  • client: Node table . Record node information and the latest key information , relation space surface .
  • base_data: Log table . Record every reported information , relation space、client surface .
  • aggregate_data: Summary information table . Record by hour 、 The data information collected in the day interval .
  • temp_base_data: Temporary log table . If the reported device name is not in the node table , Then save it in this form , It is mainly used for debugging .

API Interface data structure

The module specification

Data warehousing

  • Module function : receive IoT explorer Push data , After matching and converting , Into the database
  • Use the product :API gateway 、 Serverless cloud function
  • Associated source code :github-upload
  • Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter

The terminal is updated online

  • Module function : Press 3 Minutes to trigger , Update terminal information , For quick query and display
  • Use the product : Cloud functions
  • Associated source code :github-client_active_new
  • Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern

Data statistics

  • Module function : Trigger by hour and day , Statistical monitoring data , For quick query and display
  • Use the product : Cloud functions
  • Associated source code :github-aggregate
  • Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter , But instead “ Timing trigger ” Pattern

Data interface - Terminal summary information table

Data interface - Terminal single point information table

  • Module function : adopt API Gateway provides data query capability , Response cloud call , Provide terminal single point information table
  • Use the product :API gateway Cloud functions
  • Associated source code :github-client_data
  • Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.2 Serverless cloud function /4.2.3 cloud API gateway ” chapter

Data presentation

  • Module function : Provide WEB End show ability , Including map points 、 The list shows 、 Broken line display, etc
  • Use the product : Tencent cloud map
  • Associated source code : nothing , Currently, it is manually configured
  • Deployment way : Please refer to the relevant articles Online formaldehyde monitoring “4.2.4 Tencent cloud map ” chapter , But when selecting data sources , Change to use “API”. among “ home page ” Map use “ Data interface - Terminal summary information table ” Interface 、“ Single point data page ” Use “ Data interface - Terminal single point information table ” Interface . Specific calling interface URL, Please correspond to API Service links provided by the gateway .

Phase II cost analysis

v2_ cost analysis .png

The number of tables in the figure above is for each node 15 Report the calculation results once every second .

The hardware part is due to the difference between manufacturer and purchase quantity , The price is different . Here is the cost of cloud services , If 10000 Node size , Every node reports every minute , Annual cost of single node cloud service :

  • Phase II Architecture 3.335 element ( database / year + Cloud chart / year )/10000+(API gateway + Cloud functions )/4=(468+48)/10000+(9+2.1)/4

  • Three phase structure 1.09 element ( Message queue / year + database / year *2+ Cloud chart / year )/10000+( Cloud functions + The log service + Object storage )/4=(3228+468*2+48)/10000+(2.1+2.24+0.02)/4

Phase II no object storage and log service module , But it has been used for several months in the third phase under construction , So there are actual data to be included in the statistics .

The follow-up plan

Complete the third phase of construction , The plan involves :

  • Interaction optimization (done): Enable space surface , Show location .
  • Interaction building (doing): Add applet capabilities , Provide mobile query capability , And alarm push capability .
  • Disaster recovery optimization 1(done): increase Object storage , Provide low cost data backup 、 recovery 、 Asynchronous migration synchronization capability .
  • Disaster recovery optimization 2(todo): increase Message queue CKafka, Dealing with network anomalies 、 The risk of data accumulation and loss in case of database exception .
  • Disaster recovery optimization 3(doing): increase The log service , For data reconciliation , Dealing with multi link data processing , Possible data loss or confusion .

The third phase of the structure plan is announced

v3_ Architecture diagram .png

notes : The light blue color shows the key points of architecture changes in phase II .

Part of the third edition is finished

WEB End show

v3_yuntu_index.png

v3_yuntu_site.png

Small program end Demo

v3_yuntu_mapp.png

One More Thing

Experience Tencent cloud immediately Serverless Demo, receive Serverless New user gift pack serverless/start

Welcome to visit :Serverless Chinese net

版权声明
本文为[Tencent cloud serverless]所创,转载请带上原文链接,感谢

  1. [front end -- JavaScript] knowledge point (IV) -- memory leakage in the project (I)
  2. This mechanism in JS
  3. Vue 3.0 source code learning 1 --- rendering process of components
  4. Learning the realization of canvas and simple drawing
  5. gin里获取http请求过来的参数
  6. vue3的新特性
  7. Get the parameters from HTTP request in gin
  8. New features of vue3
  9. vue-cli 引入腾讯地图(最新 api,rocketmq原理面试
  10. Vue 学习笔记(3,免费Java高级工程师学习资源
  11. Vue 学习笔记(2,Java编程视频教程
  12. Vue cli introduces Tencent maps (the latest API, rocketmq)
  13. Vue learning notes (3, free Java senior engineer learning resources)
  14. Vue learning notes (2, Java programming video tutorial)
  15. 【Vue】—props属性
  16. 【Vue】—创建组件
  17. [Vue] - props attribute
  18. [Vue] - create component
  19. 浅谈vue响应式原理及发布订阅模式和观察者模式
  20. On Vue responsive principle, publish subscribe mode and observer mode
  21. 浅谈vue响应式原理及发布订阅模式和观察者模式
  22. On Vue responsive principle, publish subscribe mode and observer mode
  23. Xiaobai can understand it. It only takes 4 steps to solve the problem of Vue keep alive cache component
  24. Publish, subscribe and observer of design patterns
  25. Summary of common content added in ES6 + (II)
  26. No.8 Vue element admin learning (III) vuex learning and login method analysis
  27. Write a mini webpack project construction tool
  28. Shopping cart (front-end static page preparation)
  29. Introduction to the fluent platform
  30. Webpack5 cache
  31. The difference between drop-down box select option and datalist
  32. CSS review (III)
  33. Node.js学习笔记【七】
  34. Node.js learning notes [VII]
  35. Vue Router根据后台数据加载不同的组件(思考->实现->不止于实现)
  36. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  37. 【JQuery框架,Java编程教程视频下载
  38. [jQuery framework, Java programming tutorial video download
  39. Vue Router根据后台数据加载不同的组件(思考->实现->不止于实现)
  40. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  41. 【Vue,阿里P8大佬亲自教你
  42. 【Vue基础知识总结 5,字节跳动算法工程师面试经验
  43. [Vue, Ali P8 teaches you personally
  44. [Vue basic knowledge summary 5. Interview experience of byte beating Algorithm Engineer
  45. 【问题记录】- 谷歌浏览器 Html生成PDF
  46. [problem record] - PDF generated by Google browser HTML
  47. 【问题记录】- 谷歌浏览器 Html生成PDF
  48. [problem record] - PDF generated by Google browser HTML
  49. 【JavaScript】查漏补缺 —数组中reduce()方法
  50. [JavaScript] leak checking and defect filling - reduce() method in array
  51. 【重识 HTML (3),350道Java面试真题分享
  52. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  53. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  54. [re recognize HTML (3) and share 350 real Java interview questions
  55. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  56. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  57. 【重识 HTML ,nginx面试题阿里
  58. 【重识 HTML (4),ELK原来这么简单
  59. [re recognize HTML, nginx interview questions]
  60. [re recognize HTML (4). Elk is so simple