When the Mid Autumn Festival comes, use oil monkey + node.js + to deploy serverless to monitor the daily moon cake sales

Ovos 2021-09-15 09:40:22
autumn festival comes use oil


I'm participating in the Mid Autumn Festival Creative submission competition , Details please see : Mid Autumn Festival Creative submission contest

Next week is the Mid Autumn Festival , I wish you all a happy mid autumn festival in advance .

Today we'll use JS Write a program to climb the front of Jingdong 100 Moon cake sales on page , See how much moon cakes you can sell every day at the end of the Mid Autumn Festival .

Data for reference only , Accuracy is not guaranteed .

Thank you for your praise , It's not easy to stay up late and write

The technology to be used

  1. Oil monkey script (Tampermonkey)- Google browser plugin

  2. JavaScript Native DOM operation

  3. fetch request

  4. asynchronous async await Time delay

  5. express Create a data store API, Statistics API

  6. node.js Read JSON file

  7. Deploy to Tencent cloud Serverless service

Statistics show

image-20210908151215298

Be careful 2021-9-7 The data of No. is mock data , In order to 2021-9-8 The no. thanLastDay Field can calculate data

Field description :

{
"date": "2021-9-8", // date 
"total": "89026 Billion ", // Up to date date , Total sales 
"thanLastDay": "7687 ten thousand " // Compared with the previous day , How much sales have been increased 
}
 Copy code 

Now let's start the whole work

1. Install oil monkey script (Tampermonkey) plug-in unit

If you can surf the Internet scientifically, you can , Visit the official link below to install

chrome.google.com/webstore/de…

If you can't surf the Internet scientifically , Go to Baidu search Tampermonkey, There will be many websites that provide local installation methods , I don't offer it here , To avoid infringement .

2. Write a script to crawl Jingdong moon cake data

After successful installation , In the upper right corner of the browser , Pictured

image-20210908152535858

Enter Jingdong homepage first , Search for The moon cake , Enter the list of products

image-20210908152711146

Then click the management panel , Enter the script list page , Here you can open or close a script

image-20210908152917922

then , Click on + Create a new script

image-20210908152825901

image-20210908153150510

I wrote a simple script here , You can paste it

// ==UserScript==
// @name jd The moon cake 
// @namespace http://tampermonkey.net/
// @version 0.1
// @description Used to crawl 100 Product data on page 
// @author Ovos 
// @match https://search.jd.com/**
// @icon https://www.google.com/s2/favicons?domain=jd.com
// @grant none
// ==/UserScript==
(function() {
 'use strict';
// Get sales quantity 
function getNumber(str) {
if (str.includes(' ten thousand +')) {
return parseInt(str) * 10000
}
return parseInt(str)
}
// Wait function 
function sleep(time) {
return new Promise((resolve, reject) => {
setTimeout(resolve, time * 1000)
})
}
async function main() {
// Wait for the first page data load 
await sleep(3)
for (let i = 0; i < 100; i++ ){
// Scroll to the bottom 
window.scrollTo(0,18000)
// Wait for the bottom data to load 
await sleep(3)
// Scroll the bottom again , Prevent data from not loading 
window.scrollTo(0,18000)
// Wait for the bottom data to load 
await sleep(2)
// Calculate the total sales volume of all commodity prices 
await getTotal()
// Go to the next page 
document.querySelector('#J_bottomPage > span.p-num > a.pn-next').click()
// Wait for the next page of data 
await sleep(3)
}
}
async function getTotal() {
let pageTotal = 0
document.querySelectorAll('#J_goodsList > ul > li').forEach(el => {
// commodity price 
const price = parseFloat(el.querySelector('.p-price i').innerText)
// Commodity evaluation quantity 
const saleNum = getNumber(el.querySelector('.p-commit a').innerText)
console.log(price, saleNum)
// 
pageTotal += price * saleNum
})
// Total sales on this page 
const res = await fetch('http://localhost:9000/save', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({pageTotal})
})
const json = await res.json()
console.log('Success:', json);
}
// Run the program 
main()
})();
 Copy code 
  • First one for loop , Fix 100, Because Jingdong's product list page is 100 page
  • Next scroll to the bottom of the page , Because part of the data in the list is ajax Asynchronously loaded
  • sleep Function to wait for a fixed time , Use async await grammar
  • Then wait 3 second , Then scroll to the bottom , Prevent data from not loading
  • Then use document.querySelectorAll Get all the items on the page
  • Then use document.querySelector Get the price and evaluation quantity of each commodity
  • Calculate total page sales pageTotal
  • And then use fetch request Node.js Storage api, Store the sales calculated on the current page , For subsequent analysis
  • Finally, go to Jingdong's home page , Search for moon cakes , Go to the search page , Wait for the page to turn to the last page 100 page , Data acquisition complete , Then you can do something else , It's been a long time .

below , Let's take a look at the demonstration effect

[ Nuggets can't upload videos , Purring ...]

3. use Express Set up a platform for storage and analysis api

The code is as follows

const express = require('express')
const cors = require('cors');
const path = require('path')
const fs = require('fs')
var app = express();
app.use(express.json())
app.use(express.urlencoded({extended: true}))
app.use(cors())
// Get statistical data 
app.get('/get', (req, res) => {
const data = []
// Get the total sales volume on the specified date 
const getTotal = (date) => {
const filePath = path.join(__dirname, 'data', `${date}.json`)
if (!fs.existsSync(filePath)) {
return 0
}
const data = JSON.parse(fs.readFileSync(filePath))
if (data.today) {
return data.total;
}
const total = data.data.reduce((total, currentValue) => {
return total + Math.floor(currentValue) / 10000;
})
// Total number of caches , Don't count next time 
data.total = total; // Ten thousand units 
fs.writeFileSync(filePath, JSON.stringify(data))
return total;
}
// Gets the day before the specified date 
const getLastDay = (dateTime) => {
let date_ob = new Date(dateTime);
date_ob.setDate(date_ob.getDate() - 1)
let date = date_ob.getDate();
let month = date_ob.getMonth() + 1;
let year = date_ob.getFullYear();
let today = year + "-" + month + "-" + date;
return today
}
// Data on all statistical dates 
const dateList = fs.readdirSync(path.join(__dirname, 'data'))
// Return the data , Calculate the increase over the previous day 
dateList.forEach(fileName => {
const date = fileName.replace('.json', '')
data.push({
date,
total: Math.floor(getTotal(date) / 10000) + ' Billion ',
thanLastDay: getTotal(getLastDay(date)) !== 0 ? Math.floor(getTotal(date) - getTotal(getLastDay(date))) + ' ten thousand ' : ' Temporarily no data '
})
})
// Sort by date in descending order 
res.send(data.sort((a,b) => new Date(b.date) - new Date(a.date)))
});
// Store the of the day 100 Page commodity sales 
app.post('/save', (req, res) => {
// Get current date 
let date_ob = new Date();
let date = date_ob.getDate();
let month = date_ob.getMonth() + 1;
let year = date_ob.getFullYear();
let today = year + "-" + month + "-" + date;
// File path 
const filePath = path.join(__dirname, 'data', `${today}.json`)
// If no storage file exists 
if (!fs.existsSync(filePath)) {
fs.writeFileSync(filePath, JSON.stringify({data: []}))
}
// Read the file 
const data = JSON.parse(fs.readFileSync(filePath))
// Save the sales under all goods on the current page 
data.data.push(req.body.pageTotal)
// Write to json file 
fs.writeFileSync(filePath, JSON.stringify(data))
// Return the data 
res.send(data);
});
app.listen(3000, function () {
console.log(' Service started successfully :http://localhost:3000');
});
 Copy code 

There are mainly two here api Interface

GET - http://localhost:9000/get
 Copy code 

Used to obtain statistical data , The structure is as follows

[
{
"date": "2021-9-8", // date 
"total": "88615 Billion ", // Total sales 
"thanLastDay": "4338 ten thousand " // An increase in sales over yesterday 
},
{
"date": "2021-9-7",
"total": "88615 Billion ",
"thanLastDay": " Temporarily no data "
}
]
 Copy code 
POST - http://localhost:9000/save
 Copy code 

Used to store sales per page for the day , The data will be stored in data/ The current date .json In the document

{"data":[885434000,692030500,234544840,601344769.5,172129350,182674704.6,133972752.6,205753590,80450922,77355786.19999999,151456533,110421752,92058113.7,303276508,174283087.7,271311291.3,63696476.8,141753035.7,338476616.4,270641094,86462147,27128625,36139929,45965566.900000006,72166439.10000001,192549501,10540359.4,69775609.4,22760644,18128574.6,4775594.2,11293833.100000001,69100044.5,18697712.7,5837212.3,10642395.6,12401900.700000003,7687292.750000001,5542854.199999999,6173778.3,15844723.86,312611521.7,322072634.2,57924578,365159510,31830203.6,37628351.7,11473636.700000001,25383806.799999997,30270479.9,82777935.4,71801949,17886438.4,76748973.5,29326328.4,11953917.4,5390966.8,25723722.5,9660846,33003014.7,35118788.5,11297238.8,7611442.84,19172848.34,6824560,18840682.700000003,13633325.1,61348156.3,32949962.4,28584186.1,25574649.3,40607000.4,27084038.700000003,34280644.35,13503164.6,7837763.899999999,27559845.42,12587807.8,11210537.2,10225227.48,14791757.24,14573441.399999999,5919098.6,7467049.7,26552201.6,6259477.100000001,7240613.68,5715078,5421074.500000001,6174596.500000001,12098670,3628428.2,5442460.100000001,6925294.8,16266156.259999998,7562844.060000001,16977870.1,6701592.3999999985,6060801,6081381.699999999]}
 Copy code 
  • The project mainly uses fs.writeFileSync and fs.readFileSync Read and write JSON file
  • cors() Middleware to open cross domain

4. Deploy to Tencent cloud Serverless service

Last , I'll take this. Express Deploy services to the cloud , So that everyone can see

  1. modify express The project listening port is 9000,( Tencent cloud must be 9000)

  2. establish scf_bootstrap Startup file

#!/bin/sh
npm run start
 Copy code 
  1. land Tencent cloud Serverless Console , Click the function service on the left

  2. Hit the new button

  3. choice 【 Custom creation 】

img

  • Function type : choice “Web function ”.
  • The name of the function : Fill in your own function name .
  • regional : Fill in your function deployment region , The default is Guangzhou .
  • Running environment : choice “Nodejs 12.16”.
  • Deployment way : choice “ Code deployment ”, Upload your local project .
  • Submission method : choice “ Local upload folder ”.
  • function code : Select the specific local folder of the function code .
  1. Choose to finish

See the following video for detailed steps

[ Nuggets can't upload videos , Purring ...]

After successful deployment , We will the address provided by Tencent cloud , Can be used to test services

service-ehtglv6w-1258235229.gz.apigw.tencentcs.com/release/get

Be careful :

  1. Tencent cloud Serverless There is a certain amount of free use , See for details
  2. Serverless It is allowed to modify the file , therefore /save The service will report an error , The solution is to mount CFS file system , I'm too lazy to do , You have to pay .

Github Source code

github.com/cmdfas/expr…

5. summary

Last , Even if our whole function is finished , Climb from the oil monkey every day 100 Pages of data , To use express Store in JSON file , Then to calculate the difference every day . Realize the demand of calculating the sales of moon cakes every day .

版权声明
本文为[Ovos]所创,转载请带上原文链接,感谢
https://qdmana.com/2021/09/20210909145031710r.html

  1. Vue learning -- watch listener
  2. Learn more about nexttick in Vue
  3. JavaScript genrator generator
  4. La dernière réponse à l'entrevue de développement Android, l'hiver froid de l'industrie
  5. Maserati's motorcycle has less than 10 in the world. It is definitely a work of art
  6. 2021 partage des questions du dernier examen écrit d'entrevue Android, pas d'accord
  7. Programmation asynchrone Java scirp, développement frontal de base
  8. 2021 dernier examen écrit d'entrevue Android, écrit trop bien
  9. Quels aspects doivent être pris en considération dans le tableau principal du distributeur libre - service?
  10. He inherited his mother's hundreds of millions of property for his boyfriend to squander. Unexpectedly, he was ruthlessly abandoned when he had 100 yuan left
  11. The fuel cost is half less than that of fuel vehicles at the same level. Is it really cheap to use song Pro DM?
  12. Le dernier résumé de l'expérience d'entrevue d'embauche de l'école Android de l'usine est nécessaire pour l'usine
  13. Le dernier dictionnaire avancé de programmeurs d'usine, l'expérience d'entrevue d'embauche de l'Agence de développement Android
  14. La dernière collection d'entrevues Android Golden nine Silver ten
  15. L'expérience d'entrevue de l'Ingénieur d'algorithme de saut d'octets, 2 mois d'entrevue Tencent, station B, Netease et ainsi de suite sur 11 entreprises résumé!
  16. La dernière collection d'entrevues d'Android Golden nine Silver ten recommande un projet github
  17. Yuan Li's recent situation revealed that he was obsessed with public welfare, dressed simply and fearless, grew fat, and married an 11-year-old husband
  18. Initial experience of template tool plop of [front end Engineering]
  19. Dernière question d'entrevue avancée et réponse d'Alibaba Android, Alibaba P8 vous apprendra en personne
  20. Partage des dernières questions d'entrevue pour Android haute fréquence, Introduction aux compétences d'entrevue pour le développement d'Android
  21. Partager les dernières questions d'entrevue Android haute fréquence avec quelques conseils
  22. About JavaScript modules
  23. Iteratable object and class arrays
  24. Function realization of Vue elementui exporting excel form
  25. Use canvas to realize a small screenshot function
  26. Object oriented programming (2)
  27. Several common value transfer methods between Vue components
  28. Démarrer avec le serveur de base zéro: Hello World
  29. J'a I construit un escalier pour aller sur la lune, combien de façons puis - je poursuivre la sœur Chang'e?
  30. CSS implémente la fonction d'expansion et d'arrimage du Texte multiligne
  31. Varlet CLI | vue3 Component Library Quick Prototyping Tool
  32. Belle vue sur les trois rivières Xiapu
  33. La partie Web qui déploie SharePoint ajoute son propre module de fonctionnalité
  34. React Native (mise à jour à long terme)
  35. La conception et le codage de l'arbre binaire requis pour chaque entrevue d'embauche de la société Java millet;
  36. 10 jours pour obtenir l'offre d'emploi Android d'Alibaba, entrevue Android
  37. A remporté avec succès Byte, Tencent, Pulse offer, 7 ans Java une expérience d'entrevue de baise,
  38. 10大前端常用算法,web应用与开发
  39. Nginx - minimum configuration! You deserve it
  40. Les questions d'entrevue couramment utilisées pour le cadre Java sont - elles prometteuses pour le développement Java?
  41. 10 algorithmes communs de première ligne, applications Web et développement
  42. 10大前端常用算法,移动web开发教程
  43. 10大前端常用算法,新手学web前端开发
  44. After brushing: wireless routing + USB offline Bt + remote management!
  45. Le vrai problème de l'entrevue Android d'Alibaba pour les années civiles 2019 - 2021, le tutoriel d'apprentissage Android
  46. 10 principaux algorithmes de première ligne couramment utilisés, nouveaux apprenants développement de première ligne Web
  47. 118页Vue面试题总结,web中间开发
  48. 118页Vue面试题总结,HTML列表标签
  49. Use vscode snippets to work with project members to improve development efficiency
  50. CentOS + Jenkins + nginx + gitlab front end automation deployment full record
  51. Build applet architecture from scratch
  52. Uni app series (V): hbuilderx runs the uniapp project to the page, views the preview and solves NPM: the error that the file cannot be loaded
  53. Front end interview daily 3 + 1 - day 877
  54. Understand the react lifecycle function
  55. 2020-2021前端面试题合集,web开发敏捷之道
  56. CSS - redraw reflow
  57. 90 lines of code to implement the module packer
  58. Front end internship interview preparation -- react others
  59. 118 page vue sommaire des questions d'entrevue, onglet liste HTML
  60. Vue sommaire des questions d'entrevue, développement intermédiaire du Web, 118 pages