[Hadoop 3. X series] use of HDFS rest HTTP API (II) httpfs

Manor's big data struggle 2021-10-14 00:35:40
hadoop series use hdfs rest

 Insert picture description here


Current blog Hadoop Most articles stay in Hadoop2.x Stage , This series will be based on the big data of dark horse programmers Hadoop3.x A full set of tutorial , Yes 2.x There are no new features to supplement and update , One click three times plus attention , Don't get lost next time !

Article history

[hadoop3.x series ]HDFS REST HTTP API Use ( One )WebHDFS

HTTPFS summary

l HttpHDFS In essence, WebHDFS It's the same , They are all provided HTTP REST API function , But the difference is HttpHDFS yes HttpFS Is an independent from HadoopNameNode Service for , It is itself Java JettyWeb Applications .

l Because it can be deployed independently , So you can be right HttpHDFS Set up firewall , And avoid NameNode Exposed to the wall , For some systems with high security requirements ,HttpHDFS It would be better .

l HttpFS It's a kind of server , It provides REST HTTP gateway , Support all HDFS File system operations ( Read and write ). And it can work with WebhdfsREST HTTPAPI

l HttpFS Can be used to run different versions Hadoop( Overcome RPC Version control problem ) Transfer data between clusters , For example, using HadoopDiscreCP.

l HttpFS It can be used to access on the cluster behind the firewall HDFS Data in (HttpFS The server acts as a gateway , Is the only system that allows access to the cluster across the firewall ).

l HttpFS have access to HTTP Utilities ( for example curl and wget) And from the Java In other languages HTTP library Perl To visit HDFS Data in .

l This ****Webhdfs**** The client file system implementation can use Hadoop File system command access HttpFS(hdfs dfs) Line tools and use Hadoop file system JavaAPI Of Java Applications .

l HttpFS Built in support Hadoop Pseudo authentication and HTTP、SPNEGO Kerberos And other pluggable authentication mechanisms . It also provides Hadoop Agent user support .

1.1.1 HTTPFS How it works

l HttpFS Is an independent from HadoopNameNode Service for .

l HttpFS Itself is Java JettyWeb Applications .

l HttpFS HTTP Web service API Call yes HTTPREST call , Mapping to HDFS File system operations . for example , Use curl/Unix command :

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=OPEN&user.name=foo’
return HDFS The content of /user/foo/README.txt archives .

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo?op=LISTSTATUS&user.name=foo’
return HDFS The content of /user/foo In the directory JSON Format .

$curl ‘http://httpfs-host:14000/webhdfs/v1/user/foo?op=GETTRASHROOT&user.name=foo’
Return path /user/foo/.trash, If / It's an encrypted area , Then return to the path ./.Trash/Foo. See more details about the garbage path in the encrypted area .

$curl -X POST‘http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=MKDIRS&user.name=foo’
establish HDFS/user/foo/bar Catalog .

l HttpFS The default port number is 14000

1.1.2 To configure Hadoop

edit Hadoop Of core-site.xml, And will run HttpFS Server's Unix User defined as proxyuser. for example :

</property> <property>

important : Replace #HTTPFSUSER# Using will start HttpFS Server's Unix user .

for example :


1.1.3 restart Hadoop

restart Hadoop, And activate the proxy user configuration

1.1.4 start-up HttpFS

hdfs --daemon start httpfs

1.1.5 test HttpFS Work



1.1.6 HTTP Default service

Name\ Description
/conf Display configuration properties
/jmx Java JMX management interface
/logLevel Get or set log level per class
/logs Display log files
/stacks Display JVM stacks
/static/index.html The static home page



Blog home page :https://manor.blog.csdn.net
Welcome to thumb up Collection Leaving a message. Please correct any mistakes !
This paper is written by manor original , First appeared in CSDN Blog

本文为[Manor's big data struggle]所创,转载请带上原文链接,感谢

  1. PaddlePaddle:在 Serverless 架构上十几行代码实现 OCR 能力
  2. JavaScript数组 几个常用方法
  3. 暢談this的四種綁定方式
  4. 2021最新Vue面试必胜宝典,大厂面试题解析!
  5. Quatre façons de lier ceci
  6. Préparation au développement de l'extension tagdown
  7. Intervieweur: Parlez - moi des flotteurs CSS
  8. Packaging the View Component Library with rollup
  9. Comment un composant enfant modifie les valeurs passées par le composant parent
  10. Résumé de l'API Express
  11. Optimisation de la structure du Code if else dans le projet
  12. Fonction magique pour résoudre le problème de la fonction maybe - - fonction either
  13. 新手学前端的方法是什么 自学前端该怎么规划
  14. 云原生体系下 Serverless 弹性探索与实践
  15. 如何全方位打造安全高效的HTTPS站点(一)
  16. "Liu Jing dit che 丨 point de vue" est - ce que Custom Road est un MpV digne de la terre?
  17. 从理念到LRU算法实现,起底未来React异步开发方式
  18. Compared with Volvo XC60, Lingke 09 goes out of the spa platform. What would you choose, regardless of the brand?
  19. PaddlePaddle:在 Serverless 架构上十几行代码实现 OCR 能力
  20. 云原生体系下 Serverless 弹性探索与实践
  21. 初学者怎么学Web前端?
  22. react
  23. PaddlePaddle:在 Serverless 架构上十几行代码实现 OCR 能力
  24. JavaScript数组 几个常用方法
  25. Angular 依赖注入 - 全面解析
  26. html_day02
  27. 那些年我们前端面试中经常被问到的题!
  28. The starting price of Ducati multistada V2 in North America is less than 100000 yuan
  29. Hls.js 使用文檔
  30. Hls.js travailler avec des documents
  31. Problèmes liés à la précision JS
  32. Copie une partie des propriétés d'un objet à un autre objet
  33. Multiplexage de modules en vuex
  34. Développement multilingue Android, questions d'entrevue pour le développement de logiciels Android
  35. Chen lushai and her best friend Wang Meng play video, fearless of the pressure of public opinion, and in a good mood to dance in a bare back
  36. # Sass速通(四):继承、混合与函数
  37. Vidéo de développement de combat Android, questions d'entrevue rxjava
  38. Bugatti Chiron maintenance cost exposure! One piece for one car, burn money endlessly
  39. android应用开发基础答案,深入理解Nginx
  40. 做了三年前端,你才知道10分钟就能实现一个PC版魔方游戏
  41. Html + CSS + JS implémentation ️ Responsive Lucky Turnover ️ [with full source Sharing]
  42. Ren Jialun, who married young, was in a mess. Now she feels that it is a blessing in disguise
  43. 达梦数据库使用disql生成html格式的巡检报告
  44. React render phase parsing II - beginwork process
  45. Tableau linéaire de la structure des données (dessin à la main)
  46. In 2022, what are the highlights and popular elements in skirts to make skirts more elegant and gentle?
  47. JQuery installation
  48. Exemple de développement Android, dernière compilation de questions d'entrevue Android
  49. Differences and relations between JDK, JRE and JVM, nginx architecture diagram
  50. 【Azure 云服务】Azure Cloud Service 为 Web Role(IIS Host)增加自定义字段 (把HTTP Request Header中的User-Agent字段增加到IIS输出日志中)
  51. 【Azure 云服务】Azure Cloud Service 为 Web Role(IIS Host)增加自定义字段 (把HTTP Request Header中的User-Agent字段增加到IIS输出日志中)
  52. Questions d'entrevue pour les ingénieurs en développement Android, Android Foundation 72 questions
  53. It's kind of Cadillac CT6 to have a Mercedes Benz S-class captain and a 10At entry-level configuration, falling to less than 300000
  54. H6 meets the strong enemy again! The car body has a Cayenne visual sense, breaking 8.8 seconds, and the top configuration is less than 130000
  55. How nginx supports HTTPS and Linux kernel video tutorial
  56. Le martyr se réjouit de sa vieillesse Audi R8 V10 performance Rwd
  57. import 方式隨意互轉,感受 babel 插件的威力
  58. Le mode d'importation peut se déplacer librement pour sentir la puissance du plug - in Babel
  59. Pas de héros en termes de ventes!Du point de vue de la force du produit, la nouvelle version ax7 Mach est plus forte que H6
  60. The vue3 + TS project introduces vant as needed