好的想法是十分钱一打,真正无价的是能够实现这些想法的人。

求助,hadoop写入es,es安装了searchguard插件,按照es官网配置程序仍然出错

Elasticsearch | 作者 song19850712 | 发布于2018年05月29日 | 阅读数:6585

我的es版本是 5.0.1 ,es 安装了 search guard插件。现在需要从hadoop中,将数据写入到es,按照官网配置,程序仍出错,求大神解决。
 
报错信息如下:

Error: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:247)
at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:545)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:124)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:444)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:424)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:154)
at org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:609)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:240)
... 15 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:471)
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:112)
... 21 more
 
 
es官网地址如下:
https://www.elastic.co/guide/e ... .html
代码如下:
maven配置:

<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-hadoop</artifactId>
<version>5.0.1</version>
</dependency>
 
代码:
 

Configuration conf = new Configuration();
        conf.setBoolean("mapred.map.tasks.speculative.execution", false);
        conf.setBoolean("mapred.reduce.tasks.speculative.execution", false);
        //conf.set("es.nodes", "192.168.0.208:9200");
        
        conf.set(ConfigurationOptions.ES_NODES, "192.168.0.:9200");
        conf.set(ConfigurationOptions.ES_NODES, "192.168.0.:9200");
        conf.set(ConfigurationOptions.ES_NODES, "192.168.0.:9200");
        conf.set(ConfigurationOptions.ES_NODES, "192.168.0.:9200");
        
        conf.set(ConfigurationOptions.ES_NODES_WAN_ONLY, "true");
        conf.set(ConfigurationOptions.ES_RESOURCE, "hdfs_to_es/text");
        conf.set(ConfigurationOptions.ES_INPUT_JSON, "yes");
        //conf.set("searchguard.ssl.transport.enabled", "true");
        
        conf.set(ConfigurationOptions.ES_NET_USE_SSL, "true");
        conf.set(ConfigurationOptions.ES_NET_SSL_PROTOCOL, "ssl");
        conf.set(ConfigurationOptions.ES_NET_SSL_KEYSTORE_LOCATION, "adminKeystore.jks");
        conf.set(ConfigurationOptions.ES_NET_SSL_KEYSTORE_PASS, "");
        conf.set(ConfigurationOptions.ES_NET_SSL_KEYSTORE_TYPE, ConfigurationOptions.ES_NET_SSL_KEYSTORE_TYPE_DEFAULT);
        conf.set(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_LOCATION, "truststore.jks");
        conf.set(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_PASS, "");
        
        conf.set(ConfigurationOptions.ES_NET_SSL_CERT_ALLOW_SELF_SIGNED, "true");
        
        //conf.set("searchguard.ssl.transport.enforce_hostname_verification", "false");
        
        conf.set("mapred.jar", "/root//ElasticsearchForHadoop-0.0.1-jar-with-dependencies.jar");
        Job job = Job.getInstance(conf,"hadoop es write test");
        job.setMapperClass(SomeMapper.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(EsOutputFormat.class);
        job.setMapOutputKeyClass(NullWritable.class);
        job.setMapOutputValueClass(BytesWritable.class);
        // 设置输入路径
        FileInputFormat.setInputPaths(job, new Path("hdfs://:8020/wordcount/srcdata/article.txt"));
        job.waitForCompletion(true);
 
已邀请:

song19850712

赞同来自:

已解决,根据es官网文档的提示,加上如下代码,解决了问题:
 
conf.set(ConfigurationOptions.ES_NET_HTTP_AUTH_USER, "password");
conf.set(ConfigurationOptions.ES_NET_HTTP_AUTH_PASS, "password");

要回复问题请先登录注册