1. 用ElasticCloud上的ES精准分析空气质量数据 http://t.cn/RsIi2c0 

2. ES painless用法示例。 http://t.cn/RsIgnOF 

3. 一周热点：Google推出数据集搜索 http://t.cn/Rs2xE82   

活动预告

1、Elastic 中国开发者大会门票发售中 https://conf.elasticsearch.cn/2018/shenzhen.html

• 编辑：bsll

• 归档：https://elasticsearch.cn/article/792

• 订阅：https://tinyletter.com/elastic-daily

Cleaner

这是一个管理index TTL 插件，精美UI，高效运维管理elasticsearch index助手

https://github.com/TrumanDu/cleaner

screenshots

config

1. scheduleTime

   server job schedule period,unit second ,default value is 60 second.

   you can edit it. like: cleaner.scheduleTime: 100

2. mergePattern

   merge pattern,default value is [^a-z]+$.

   you can edit it. like: cleaner.mergePattern: '[\d]{4}[-|\.|/][\d]{1,2}[-|\.|/][\d]{1,2}'

development

See the kibana contributing guide for instructions setting up your development environment. Once you have completed that, use the following npm tasks.

• npm start

  Start kibana and have it include this plugin

• npm start -- --config kibana.yml

  You can pass any argument that you would normally send to bin/kibana by putting them after -- when running npm start

• npm run build

  Build a distributable archive

• npm run test:browser

  Run the browser tests in a real web browser

• npm run test:server

  Run the server tests using mocha

For more information about any of these commands run npm run ${task} -- --help.

从Logstash 1.5开始，我们可以在logstash配置中使用metadata。metadata不会在output中被序列化输出，这样我们便可以在metadata中添加一些临时的中间数据，而不需要去删除它。

我们可以通过以下方式来访问metadata:

[@metadata][foo]

用例

假设我们有这样一条日志：

[2017-04-01 22:21:21] production.INFO: this is a test log message by leon

我们可以在filter中使用grok来做解析:

grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:log_level}: %{DATA:content}" }
    }

解析的结果为

{
      "env" => "production",
      "timestamp" => "2017-04-01 22:21:21",
      "log_level" => "INFO",
      "content" => "{\"message\":\"[2017-04-01 22:21:21] production.INFO: this is a test log message by leon\"}"
}

假设我们希望

1. 能把log_level为INFO的日志丢弃掉，但又不想让该字段出现在最终的输出中
2. 输出的索引名中能体现出env，但也不想让该字段出现在输出结果里

对于1，一种方案是在输出之前通过mutate插件把不需要的字段删除掉，但是一旦这样的处理多了，会让配置文件变得“不干净”。

通过 metadata，我们可以轻松地处理这些问题：

grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:[@metadata][env]}\.%{DATA:[@metadata][log_level]}: %{DATA:content}" }
}

if [@metadata][log_level] == "INFO"{
    drop{}    
}

output{
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "%{[@metadata][env]}-log-%{+YYYY.MM}"
        document_type => "_doc"
    }
}

除了简化我们的配置文件、减少冗余字段意外，同时也能提高logstash的处理速度。

Elasticsearch input插件

有些插件会用到metadata这个特性，比如elasticsearch input插件：

input {
  elasticsearch {
    host => "127.0.0.1"
    # 把 ES document metadata (_index, _type, _id) 包存到 @metadata 中
    docinfo_in_metadata => true
  }
}

filter{
    ......
}

output {
  elasticsearch {
    document_id => "%{[@metadata][_id]}"
    index => "transformed-%{[@metadata][_index]}"
    type => "%{[@metadata][_type]}"
  }
}

调试

一般来说metadata是不会出现在输出中的，除非使用 rubydebug codec 的方式输出：

output { 
  stdout { 
    codec  => rubydebug {
      metadata => true
    }
  }
}

日志经过处理后输出中会包含：

{
    ....,
    "@metadata" => {
        "env" => "production",
        "log_level" => "INFO"
    }
}

总结

由上可见，metadata提供了一种简单、方便的方式来保存中间数据。这样一方面减少了logstash配置文件的复杂性：避免调用remove_field，另一方面也减少了输出中的一些不必要的数据。通过这篇对metadata的介绍，希望能对大家有所帮助。

职位链接及描述如下： https://boards.greenhouse.io/elastic/jobs/1272161

At Elastic, we have a simple goal: to solve the world's data problems with products that delight and inspire. As the company behind the popular open source projects — Elasticsearch, Kibana, Logstash, and Beats — we help people around the world do great things with their data. From stock quotes to Twitter streams, Apache logs to WordPress blogs, our products are extending what's possible with data, delivering on the promise that good things come from connecting the dots. The Elastic family unites employees across 32 countries into one coherent team, while the broader community spans across over 100 countries.

For all of us at Elastic, community matters. Our users and contributors have helped to ensure that Elasticsearch, Kibana, Logstash, and Beats are more than just code — they are open source projects that people love to use, and love to talk about! As our Community Advocate you will champion our Elastic community.

What You Will Be Doing:

Are you that kind of person who is invigorated by sharing juicy technology goodness with the world? Do you feel at home connecting with the community members: in person, on blogs, in forums, via social channels, and at events? Is presenting at local meetups your jam and are you passionate about the Elastic Stack?

Well, this might just be your dream job.

As a Community Advocate at Elastic, you will be based in China. You will wake up each morning eager to design and deliver presentations at a wide-variety of events from customer meetings, meetups, tradeshows, and other events to help showcase technology. You will do this while traveling the region and, at times, the world, representing Elastic. Maintaining the trust of our community, as well as the respect and trust within the team, is foundational.

What You Bring Along:

• Bachelor’s degree in a technical field (e.g. CS, CSE, EE) or relevant work experience as a software developer (mandatory)
• Demonstrated ability to craft compelling content - including speaking engagements, blog posts, demos, messaging, etc. (mandatory)
• You are comfortable presenting, whether it's at a local meetup or to the office of a C-suite member
• Familiarity with, and real passion for, the Elastic Stack
• Comfort working with a globally distributed team
• Fluency or high working proficiency in Mandarin (mandatory)
• Excellent spoken and written English communication skills, since this is our company's language (mandatory)

Please send us your CV in English. Things We'd Be Stoked to See on Your CV:

• Conversations in person, on blogs, in forums, via social channels, at events give you energy and you have a proven publication history to show that
• Experience working for a startup or an early stage company
• Experience with open source software and/or commercial open source companies
• Technical background and abilities in APM, PHP, node.js, JS, and/or security (nice-to-have, not mandatory)
• Other languages

Additional Information:

• Competitive pay based on the work you do here and not your previous salary
• Stock options
• Global minimum of 16 weeks of paid parental leave (moms and dads)
• Generous vacation time and one week of volunteer time off
• An environment in which you can balance great work with a great life
• Your age is only a number. It doesn't matter if you're just out of college or your children are; we need you for what you can do.
• Distributed-first company with Elasticians in over 30 countries, spread across 18 time zones, and speaking over 30 languages!

LI-KE1

Target locations: Beijing, China; Shanghai, China; Hangzhou, China

Elastic is an Equal Employment employer committed to the principles of equal employment opportunity and affirmative action for all applicants and employees. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. Elastic also makes reasonable accommodations for disabled employees consistent with applicable law.