使用logstash同步oracle数据到es有两个问题,请教下各位:一、我oracle表中有一个id自增主键,使用logstash同步时报错:
[2018-07-19T18:37:01,717][ERROR][logstash.inputs.jdbc ] Java::JavaSql::SQLSy
ntaxErrorException: ORA-00904: "ID": 鏍囪瘑绗︽棤鏁? SELECT * FROM (SELECT count
(*) "COUNT" FROM (select * from temp where id > 0) "T1") "T1" WHERE (ROWNUM <= 1
)
[2018-07-19T18:37:01,743][WARN ][logstash.inputs.jdbc ] Exception when execu
ting JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaSql::SQLSyntaxEr
rorException: ORA-00904: "ID": 忙聽聡猫炉聠莽卢娄忙聴聽忙聲聢
logstash conf文件内容:
input{
stdin{
}
jdbc{
jdbc_connection_string => "jdbc:oracle:thin:@//x.x.x.x:1521/orcl"
jdbc_user => "xxxx"
jdbc_password => "xxxxx"
jdbc_driver_library => "D:/oracle/ojdbc6-11.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
record_last_run => true
use_column_value => true
tracking_column => id
last_run_metadata_path => "D:/ES/logstash-6.3.0/info"
clean_run => "false"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement => "select * from temp where id > :sql_last_value"
schedule => "* * * * *"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output{
elasticsearch{
action=> "index"
hosts => "x.x.x.x:9200"
index => "testdemo"
document_id => "%{id}"
document_type => "demoinfo"
manage_template => false
template_overwrite => true
template_name => "testdemo"
template => "D:/ES/logstash-6.3.0/template/test.json"
}
stdout {
codec => json_lines
}
}
请问下这是为什么?
二、我oracle中有10表,我需要同步的是各个表中2字段组合的一个表,这个表的字段的值是自己写sql语句得到的.现在的办法是自己建一个临时表,先得到临时表的数据,然后把临时表同步的es。有没有办法不需要建立临时表可以直接同步?
[2018-07-19T18:37:01,717][ERROR][logstash.inputs.jdbc ] Java::JavaSql::SQLSy
ntaxErrorException: ORA-00904: "ID": 鏍囪瘑绗︽棤鏁? SELECT * FROM (SELECT count
(*) "COUNT" FROM (select * from temp where id > 0) "T1") "T1" WHERE (ROWNUM <= 1
)
[2018-07-19T18:37:01,743][WARN ][logstash.inputs.jdbc ] Exception when execu
ting JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaSql::SQLSyntaxEr
rorException: ORA-00904: "ID": 忙聽聡猫炉聠莽卢娄忙聴聽忙聲聢
logstash conf文件内容:
input{
stdin{
}
jdbc{
jdbc_connection_string => "jdbc:oracle:thin:@//x.x.x.x:1521/orcl"
jdbc_user => "xxxx"
jdbc_password => "xxxxx"
jdbc_driver_library => "D:/oracle/ojdbc6-11.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
record_last_run => true
use_column_value => true
tracking_column => id
last_run_metadata_path => "D:/ES/logstash-6.3.0/info"
clean_run => "false"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement => "select * from temp where id > :sql_last_value"
schedule => "* * * * *"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output{
elasticsearch{
action=> "index"
hosts => "x.x.x.x:9200"
index => "testdemo"
document_id => "%{id}"
document_type => "demoinfo"
manage_template => false
template_overwrite => true
template_name => "testdemo"
template => "D:/ES/logstash-6.3.0/template/test.json"
}
stdout {
codec => json_lines
}
}
请问下这是为什么?
二、我oracle中有10表,我需要同步的是各个表中2字段组合的一个表,这个表的字段的值是自己写sql语句得到的.现在的办法是自己建一个临时表,先得到临时表的数据,然后把临时表同步的es。有没有办法不需要建立临时表可以直接同步?
1 个回复
rochy - rochy_he
赞同来自:
第二个你可以考虑使用视图