Livy配置

Livy配置的参数在： https://github.com/cloudera/livy/blob/master/conf/livy-client.conf.template

image-20240305082619589

示例

例如配置livy.rsc.launcher.port.range参数，对应configuration为：

[
  {
    "Classification": "livy-conf",
    "Properties": {
      "livy.rsc.launcher.port.range": "10000~10015",
      "livy.server.port": "8998",
      "livy.server.session.timeout": "24h"
    }
  }
]

image-20240305083702521

登录到primary节点，查看/etc/livy/conf/livy-client.conf：

image-20240305083754213

测试

测试这个参数是否生效。不断执行：

curl   -X POST   -d '{   "kind": "spark",   "driverMemory": "512M",   "executorMemory": "512M"      }'  -H "Content-Type: application/json"   localhost:8998/sessions/

执行16次后，会提示Unable to connect to provided ports:

image-20240305083825842

最多开16个session，对应初始设置的10000-10015:

image-20240305082646975

参考： https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-spark-optimization.html