Look at the second dimension! Wumeng music game score finder based on serverless

Serverless 2020-11-12 16:13:15
look second dimension wumeng music


 The author of this article : Yuange made 

One 、 What is? Serverless Framework

Serverless Framework It is a very popular serverless application framework in the industry , Developers do not need to care about the underlying resources to deploy the complete available Serverless Application Architecture . Serverless Framework With resource choreography 、 Automatic telescopic 、 Event driven capabilities , Covering code 、 debugging 、 test 、 Deployment and other life cycles , Help developers through linkage cloud resources , Build quickly Serverless application

you 're right , Like I saw a few days ago 《Serverless Song of 》 It's said in it I'm gonna reduce your ops, It can greatly reduce the pressure of operation and maintenance , Let's get started ! Note that the development environment needs Node.js 10.0+, One click global installation :npm install -g serverless

Two 、 Tencent cloud Flask Serverless Component brief introduction

Tencent cloud Flask Serverless Component, Support Restful API Deployment of services

It's customary to deploy first demo Well

  1. Local PyCharm Create a new Flask project


  1. Manually create the content as Flask Of requirements.txt
  2. according to The configuration document establish serverless.yml, For example, the complete content of the actual use of this project , The first use can be simplified at your discretion
  3. Write the key to .env( Of course , When deploying, you can also select wechat code scanning authorization )

 Successful deployment

 Successful visit

This is based on Serverless Of Flask Demo The deployment is complete , Next, continue to write the rest of the code in your own way .

3、 ... and 、maimai_DX

maimai It's an arcade video game .

Put a moving picture here and experience it by yourself , The original material comes from 「 External record maimai」QZKago Requiem Re:MASTER ALLPERFECT Player: Ruri*R


At home , You can only view the results from the official account of WeChat , And every time you enter the page, you need to log in with wechat authorization , And the number of records stored in it is limited , Photo album Only the latest 10 strip , Game records Only the latest 50 strip ( It's a queue , First in, first out ). This is the original intention of this project , Every time you make a score, you should keep it .

Wumeng scoring device

The results show that , front end Fomantic-UI, Back end Flask+MySQL.gh Open source address :https://github.com/yuangezhizao/maimai_DX_CN_probe, welcome watchstarfork & pr


At present, the following functions are installed :

  1. wechat_archive Contained in the Home page , Game data , Photo album and Game records : The original web page has been modified , And added Highcharts Library visualization curves show changes
  2. record contain Record ( Pagination ) and differences ( Pagination ): Self written quick preview page , It is a very practical function to view historical records and changes in performance
  3. info contain Shop list : That is, the basic information of all pavements , Output to a page , Easy to search within the page

The development process

Next, in chronological order , Describe the problems encountered in the development process and how to solve

1. Serverless Framework Component The configuration file

Serverless Framework Now it is V2 edition , That is to say, we can't follow the previous version of serverless.yml The configuration file , The document needs to be revised again .

a. Previous versions will be based on requirements.txt Automatically download the third party library to the project directory .serverless Under folder requirements Folder to participate in the final dependency packaging , Compressed into zip The file is finally uploaded to the cloud function running environment

b. The latest version is no longer automatically downloaded , It needs to be handled by itself . Reference usage of official examples :hook

# TODO: install python The project depends on the current directory of the project
hook: 'pip3 install -r requirements.txt -t ./requirements'
dist: ./
- source: ./requirements
prefix: ../ # prefix, can make ./requirements files/dir to ./
- .env
- 'requirements/**'

The notes are very clear , Use hook According to requirements.txt Download the third party library to the project directory requirements Folder , Avoid third party libraries causing confusion in local folder management . then include In the project directory requirements The folder is in the cloud prefix, That is, for the cloud function running environment in the cloud ,requirements The third-party library and project directory in the folder are the same level , It can be imported and used normally . Yes, of course , The local operation uses a global third-party library , It is not used in the project directory requirements Folder .

2. Layer Management Overview

The former ( finger b) It's a very reasonable design , But in the actual environment, new problems have been found . Exactly the same configuration file

hook: 'pip3 install -r ./src/requirements.txt -t ./src/requirements'
dist: ./src
- source: ./requirements
prefix: ../
- .env

stay macOS After successful deployment , In the cloud function editor, you can see requirements Folder does not exist , Third party libraries and project directories are peer to peer , No problem .

But in the Windows After successful deployment , In the cloud function editor, you can see requirements Folder ? That is, the third party library and the project directory are not the same level , So the interview will appear no module found The import error of ……

Try again and again to modify prefix The debugging of the configuration item is not successful in the end , Therefore, two solutions are proposed here :

a. Modify the configuration file as follows , Make the local third-party library and project directory exist at the same level

hook: 'pip3 install -r ./src/requirements.txt -t ./src'
dist: ./src
- .env

But as projects and third-party libraries expand folders, more and more , It's very difficult to manage

b. Use the cloud function to provide layer

although sls deploy Deployment is fast , But it would be better to upload project code only when deployed without handling dependencies , So cross end collaboration development just needs to care about project code ok 了 , There's no need to manage dependencies anymore !

And a little bit more , Want to be in SCF Editing function code online in the console requires that the deployment package be kept in 10MB following , Don't think ten trillion is big , It's also possible to run out quickly

 Show only entry files

How to operate it ? That is to package and create third-party library folders directly as layers , In the function code, you can directly pass import quote , After all, there are special libraries like Brotli,Windows Under no vc++ You can only go to https://lfd.uci.edu/~gohlke/pythonlibs download wheel install .

macOS After normal installation, you will get _brotli.cpython-39-darwin.so,brotli.py And then with import _brotli The form of import , But there's a new problem , Errors will be reported in the cloud ModuleNotFoundError: No module named '_brotli'"

At present SCF The execution environment of is based on the following : standard CentOS 7.2

In order to solve the problem, try in linux Packaging in the environment , Pick up what you have CentOS 8.2 The virtual machine starts to operate

pip3 install -r requirements.txt -t ./layer --upgrade
zip -r layer.zip ./layer

And then you can pack it up layer.zip Download it locally and upload it again , For the time being, it can be done once and for all .

 Layer details

by the way , Configuration files can be removed hook And add layers

src: ./src
- .env
- '__pycache__/**'
- name: maimai_DX_CN_probe
version: 3
The bound layer function is triggered to run , When starting a concurrent instance , The running code of the loading function will be decompressed to /var/user/ Under the table of contents , At the same time, the contents of the layer will be decompressed and loaded to /opt Under the table of contents . If you need to use or access files file, Place in the root directory of the compressed file when creating the layer . After decompressing and loading , You can go directly through the catalog /opt/file Access to the file . If you create a layer , Compress by folder dir/file, When the function is running, it needs to pass /opt/dir/file Access to specific files

Experience faster deployment ! Because the third-party library is already packaged in “ layer ” It's in

But the strange thing is , Importing any third-party library in the cloud will result in an error , So I debugged and looked at path

for path in sys.path:

Check it again opt

import os
dirs = os.listdir('/opt')
for file in dirs:

That's what I realized , You need to package directly in the current path . After uploading “ layer ” Update to version 2, however ModuleNotFoundError: No module named '_brotli' The error is still reported , And confirm _brotli.cpython-38-x86_64-linux-gnu.so The file actually exists .

And in the CentOS and macOS There is no problem with local import , That's a big problem , I think it's probably python Version problem , So I went to look for ready-made 3.6 Environment , Here, for example. :


After uploading again “ layer ” Update to version 3, Successful visit ! The subject is finally solved , It turns out that it is necessary to Same version Of Python 3.6 Running environment

3. Custom entry file

components Source code tencent-flask/src/_shims/ The files in are repackaged and uploaded to the cloud function without any change , There are currently two documents

a. severless_wsgi.py, Role is converts an AWS API Gateway proxied request to a WSGI request.
WSGI The full name is Python Web Server Gateway Interface namely Web Server gateway interface , It is for Python Defined by language Web The server and Web A simple and general interface between applications or frameworks

b. sl_handler.py, It's the default entry file

import app # Replace with your actual application
import severless_wsgi
# If you need to send additional content types as text, add then directly
# to the whitelist:
# serverless_wsgi.TEXT_MIME_TYPES.append("application/custom+json")
def handler(event, context):
return severless_wsgi.handle_request(app.app, event, context)

For your own project , Used Flask Of Factory function , To avoid being re modified in the cloud Function Editor every time , The best way is to customize the entry file :

import severless_wsgi
from maimai_DX_CN_probe import create_app # Replace with your actual application
# If you need to send additional content types as text, add then directly
# to the whitelist:
# serverless_wsgi.TEXT_MIME_TYPES.append("application/custom+json")
def handler(event, context):
return severless_wsgi.handle_request(create_app(), event, context)

Reassign Execution method by serverless_handler.handler, Just ok 了

4. url_for Output http Instead of https Of URL

Redirect to in view function url_for The generated links are http, instead of https…… Actually the question is Flask Documents Standalone WSGI Containers There is a description of

At the end of the day, it's not Flask The problem of , It is WSGI Environmental problems , The recommended method is to use middleware , The official also gave ProxyFix

from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app, x_proto=1, x_host=1)

But from X-Forwarded-Proto Take the value of ,apigw Among them is http, So you can't use this directly ProxyFix
because Flask Our community is still perfect , Many predecessors have paved the way , So go straight Stack Overflow Search for solutions ,Flask url_for generating http URL instead of https
The cause of the problem is shown in the figure :Browser ----- HTTPS ----> Reverse proxy(apigw) ----- HTTP ----> Flask
Because I'm in apigw Set up Front end type only https, in other words Browser The end is impossible to use http Visited , By printing environ You know

"PATH_INFO": "/",
"SERVER_NAME": "maimai.yuangezhizao.cn",
"SERVER_PORT": "80",
"wsgi.errors": <__main__.CustomIO object at 0x7feda2224630>,
"wsgi.input": <_io.BytesIO object at 0x7fed97093410>,
"wsgi.multiprocess": False,
"wsgi.multithread": False,
"wsgi.run_once": False,
"wsgi.url_scheme": "http",
"wsgi.version": (1, 0),
"serverless.authorizer": None,
"serverless.event": "<rm>",
"serverless.context": "<rm>",
"event": "<rm>",
"context": "<rm>",
"HTTP_ACCEPT": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"HTTP_ACCEPT_ENCODING": "gzip, deflate, br",
"HTTP_ACCEPT_LANGUAGE": "zh-CN,zh;q=0.9,en;q=0.8",
"HTTP_CONNECTION": "keep-alive",
"HTTP_COOKIE": "<rm>",
"HTTP_HOST": "maimai.yuangezhizao.cn",
"HTTP_SEC_FETCH_DEST": "document",
"HTTP_SEC_FETCH_MODE": "navigate",
"HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36",
"HTTP_X_API_REQUESTID": "5bcb29af2ca18c1e6d7b1ec5ff7b5427",
"HTTP_X_API_SCHEME": "https",
"HTTP_X_B3_TRACEID": "5bcb29af2ca18c1e6d7b1ec5ff7b5427",

HTTP_X_FORWARDED_PROTO Corresponding apigw The variable in is HTTP_X_API_SCHEME, So the solution is as follows :app.wsgi_app = ReverseProxied(app.wsgi_app)

class ReverseProxied(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
scheme = environ.get('HTTP_X_FORWARDED_PROTO')
if scheme:
environ['wsgi.url_scheme'] = scheme
return self.app(environ, start_response)
app = Flask(__name__)
app.wsgi_app = ReverseProxied(app.wsgi_app)

5. Response data compression

Whether it's IISApache still Nginx, All of them have compression function . After all, the external network uplink of the virtual machine you are using is only 1M bandwidth , After compression, the effect of shortening the first screen time is greatly improved . about Serverless, The response data is through API Gateway Transfer to client , So compression should also be its ability ( Although the Internet speed has been greatly improved , But the compression still has to be compressed ), However, it did not find …… See some js The framework provides compression capabilities , So I decided to add Flask Self compression function . simply , By subscribing @app.after_request Signal and call a third-party library brotli Of compress The method can (
Go before you write gh See if there are ready-made wheel extensions , Sure enough …… At first, I used Flask-Zipper, Later it was replaced by Flask-Compress Solved the problem
Actually measured 3.1 MB The data used in this study is brotli The compression algorithm is reduced to 76.1 kB

 Before compression

 After the compression


6. apigw The impact of different paths in three environments

The default mapping is as follows :

ID Environment name Access path
1 Release release
2 pre-release prepub
3 test test

Because of the configuration static_url_path by "", namely static The folder is mapped to / The next path , So add releaseprepub and test It's natural to visit 404
So it's bound to Custom domain name , Use custom path mapping , And will Release The access path of the environment is set to /, So visit again Release There's no problem with the environment

ID Environment name Access path
1 Release /
2 pre-release prepub
3 test test

7. Simultaneous access Private networks and Extranet

Cloud functions There are several cloud databases available in

  • Cloud database CDB, need Private networks visit , Although can access through the external network, but can go to the intranet, do not go outside the network
  • PostgreSQL for Serverless(ServerlessDB), This is from the government Serverless Matched pg database
  • Development of cloud TCB Medium MongoDB, If you remember correctly, you need to open the internal test permission access

Because i migrated from the old website , The data has not been migrated yet , So directly access the original cloud database CDB, stay Cloud functions To configure Belong to the network and The subnet it belongs to that will do . But you can't access the Internet at this time , One solution is to turn on Public network access and The public network is fixed IP, You can access both intranet and extranet resources at the same time . About profiles , This project is Single instance application in other words Only one component is introduced into the project , Only one component instance is generated during deployment . But if you want to introduce a database , You have to add new components , Currently in Flask Components Database related configuration items are not provided in , Therefore need Multiple components are introduced into the project , Generate multiple component instances at deployment time . It's also very simple. , Create a containing serverless.yml New folder for , Used to configure postgresql

component: postgresql # ( Required ) Component name , Here is postgresql
name: maimai_DX_CN_probe # ( Mandatory ) Component instance name .
org: yuangezhizao # ( Optional ) Used to record organizational information , The default value is your Tencent cloud account appid, Must be a string
app: yuangezhizao # ( Optional ) Used to record organizational information . Default and name identical , Must be a string
stage: dev # ( Optional ) Used to distinguish environmental information , The default value is dev
region: ap-beijing # Optional ap-guangzhou, ap-shanghai, ap-beijing
zone: ap-beijing-3 # Optional ap-guangzhou-2, ap-shanghai-2, ap-beijing-3
dBInstanceName: maimai_DX_CN_probe
# projectId: 0
dBVersion: 10.4
dBCharset: UTF8
vpcId: vpc-mrg5ak88
subnetId: subnet-hqwa51dh
extranetAccess: false

And then at the terminal cd Go to this directory and execute sls deploy You can successfully deploy postgresql

yum install python3-devel postgresql-devel
pip install psycopg2


import psycopg2
File "/opt/psycopg2/__init__.py", line 51, in &lt;module&gt;
from psycopg2._psycopg import ( # noqa
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

The following problems are being solved :

  • http Force a jump https
  • Push test environment to production environment

thus , This is the end of the article , welcome communication

One More Thing

Experience Tencent cloud immediately Serverless Demo, receive Serverless New user gift pack serverless/start

Welcome to visit : Serverless Chinese net

  1. [front end -- JavaScript] knowledge point (IV) -- memory leakage in the project (I)
  2. This mechanism in JS
  3. Vue 3.0 source code learning 1 --- rendering process of components
  4. Learning the realization of canvas and simple drawing
  5. gin里获取http请求过来的参数
  6. vue3的新特性
  7. Get the parameters from HTTP request in gin
  8. New features of vue3
  9. vue-cli 引入腾讯地图(最新 api,rocketmq原理面试
  10. Vue 学习笔记(3,免费Java高级工程师学习资源
  11. Vue 学习笔记(2,Java编程视频教程
  12. Vue cli introduces Tencent maps (the latest API, rocketmq)
  13. Vue learning notes (3, free Java senior engineer learning resources)
  14. Vue learning notes (2, Java programming video tutorial)
  15. 【Vue】—props属性
  16. 【Vue】—创建组件
  17. [Vue] - props attribute
  18. [Vue] - create component
  19. 浅谈vue响应式原理及发布订阅模式和观察者模式
  20. On Vue responsive principle, publish subscribe mode and observer mode
  21. 浅谈vue响应式原理及发布订阅模式和观察者模式
  22. On Vue responsive principle, publish subscribe mode and observer mode
  23. Xiaobai can understand it. It only takes 4 steps to solve the problem of Vue keep alive cache component
  24. Publish, subscribe and observer of design patterns
  25. Summary of common content added in ES6 + (II)
  26. No.8 Vue element admin learning (III) vuex learning and login method analysis
  27. Write a mini webpack project construction tool
  28. Shopping cart (front-end static page preparation)
  29. Introduction to the fluent platform
  30. Webpack5 cache
  31. The difference between drop-down box select option and datalist
  32. CSS review (III)
  33. Node.js学习笔记【七】
  34. Node.js learning notes [VII]
  35. Vue Router根据后台数据加载不同的组件(思考-&gt;实现-&gt;不止于实现)
  36. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  37. 【JQuery框架,Java编程教程视频下载
  38. [jQuery framework, Java programming tutorial video download
  39. Vue Router根据后台数据加载不同的组件(思考-&gt;实现-&gt;不止于实现)
  40. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  41. 【Vue,阿里P8大佬亲自教你
  42. 【Vue基础知识总结 5,字节跳动算法工程师面试经验
  43. [Vue, Ali P8 teaches you personally
  44. [Vue basic knowledge summary 5. Interview experience of byte beating Algorithm Engineer
  45. 【问题记录】- 谷歌浏览器 Html生成PDF
  46. [problem record] - PDF generated by Google browser HTML
  47. 【问题记录】- 谷歌浏览器 Html生成PDF
  48. [problem record] - PDF generated by Google browser HTML
  49. 【JavaScript】查漏补缺 —数组中reduce()方法
  50. [JavaScript] leak checking and defect filling - reduce() method in array
  51. 【重识 HTML (3),350道Java面试真题分享
  52. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  53. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  54. [re recognize HTML (3) and share 350 real Java interview questions
  55. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  56. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  57. 【重识 HTML ,nginx面试题阿里
  58. 【重识 HTML (4),ELK原来这么简单
  59. [re recognize HTML, nginx interview questions]
  60. [re recognize HTML (4). Elk is so simple