从Logstash 1.5开始，我们可以在logstash配置中使用metadata。metadata不会在output中被序列化输出，这样我们便可以在metadata中添加一些临时的中间数据，而不需要去删除它。

我们可以通过以下方式来访问metadata:

<br /> [@metadata][foo]<br />

用例

假设我们有这样一条日志：

<br /> [2017-04-01 22:21:21] production.INFO: this is a test log message by leon<br />



我们可以在filter中使用grok来做解析:

ruby<br /> grok {<br /> match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:log_level}: %{DATA:content}" }<br /> }<br />

解析的结果为

ruby<br /> {<br /> "env" => "production",<br /> "timestamp" => "2017-04-01 22:21:21",<br /> "log_level" => "INFO",<br /> "content" => "{\"message\":\"[2017-04-01 22:21:21] production.INFO: this is a test log message by leon\"}"<br /> }<br />



假设我们希望

1. 能把log_level为INFO的日志丢弃掉，但又不想让该字段出现在最终的输出中
   
2. 输出的索引名中能体现出env，但也不想让该字段出现在输出结果里
   
   对于1，一种方案是在输出之前通过mutate插件把不需要的字段删除掉，但是一旦这样的处理多了，会让配置文件变得“不干净”。
   
   通过 metadata，我们可以轻松地处理这些问题：
   
   ruby<br /> grok {<br /> match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:[@metadata][env]}\.%{DATA:[@metadata][log_level]}: %{DATA:content}" }<br /> }<br /> <br /> if [@metadata][log_level] == "INFO"{<br /> drop{} <br /> }<br /> <br /> output{<br /> elasticsearch {<br /> hosts => ["127.0.0.1:9200"]<br /> index => "%{[@metadata][env]}-log-%{+YYYY.MM}"<br /> document_type => "_doc"<br /> }<br /> }<br />
   
   除了简化我们的配置文件、减少冗余字段意外，同时也能提高logstash的处理速度。
   
   
   
   
   
   

   Elasticsearch input插件
   

   
   有些插件会用到metadata这个特性，比如elasticsearch input插件：
   
   ```ruby
   input {
   elasticsearch {
   host => "127.0.0.1"
   

   把 ES document metadata (_index, _type, _id) 包存到 @metadata 中
   

   docinfo_in_metadata => true
   }
   }
   
   filter{
   ......
   }
   
   output {
   elasticsearch {
   document_id => "%{[@metadata][_id]}"
   index => "transformed-%{[@metadata][_index]}"
   type => "%{[@metadata][_type]}"
   }
   }
   ```
   
   
   
   

   调试
   

   
   一般来说metadata是不会出现在输出中的，除非使用 rubydebug codec 的方式输出：
   
   <br /> output { <br /> stdout { <br /> codec => rubydebug {<br /> metadata => true<br /> }<br /> }<br /> }<br />
   
   日志经过处理后输出中会包含：
   
   ruby<br /> {<br /> ....,<br /> "@metadata" => {<br /> "env" => "production",<br /> "log_level" => "INFO"<br /> }<br /> }<br />
   
   
   
   

   总结
   

   
   由上可见，metadata提供了一种简单、方便的方式来保存中间数据。这样一方面减少了logstash配置文件的复杂性：避免调用remove_field，另一方面也减少了输出中的一些不必要的数据。通过这篇对metadata的介绍，希望能对大家有所帮助。
   
   

1.Bulk 异常引发的 Elasticsearch 内存泄漏排查。
http://t.cn/RFBHC1p
2.使用elastichq监控Elasticsearch机器。
http://t.cn/RFBHHLy
3.使用ELK分析应用事件和日志。
http://t.cn/RFBHnLN

​活动预告
1、Elastic 中国开发者大会门票发售中
https://conf.elasticsearch.cn/2018/shenzhen.html
2、Elastic Meetup 9月8日 北京线下交流活动免费报名中
https://elasticsearch.cn/article/759

编辑：叮咚光军
归档：https://elasticsearch.cn/article/785
订阅：https://tinyletter.com/elastic-daily

Elasticsearch 的排名又升了，要不要加入这么一家蒸蒸日上的全球领先的开源软件公司？

职位链接及描述如下：
[https://boards.greenhouse.io/elastic/jobs/1272161](https://boards.greenhouse.io/e ... 502301)

At Elastic, we have a simple goal: to solve the world's data problems with products that delight and inspire. As the company behind the popular open source projects — Elasticsearch, Kibana, Logstash, and Beats — we help people around the world do great things with their data. From stock quotes to Twitter streams, Apache logs to WordPress blogs, our products are extending what's possible with data, delivering on the promise that good things come from connecting the dots. The Elastic family unites employees across 32 countries into one coherent team, while the broader community spans across over 100 countries.

For all of us at Elastic, community matters. Our users and contributors have helped to ensure that Elasticsearch, Kibana, Logstash, and Beats are more than just code — they are open source projects that people love to use, and love to talk about! As our Community Advocate you will champion our Elastic community.

What You Will Be Doing:

Are you that kind of person who is invigorated by sharing juicy technology goodness with the world? Do you feel at home connecting with the community members: in person, on blogs, in forums, via social channels, and at events? Is presenting at local meetups your jam and are you passionate about the Elastic Stack?

Well, this might just be your dream job.

As a Community Advocate at Elastic, you will be based in China. You will wake up each morning eager to design and deliver presentations at a wide-variety of events from customer meetings, meetups, tradeshows, and other events to help showcase technology. You will do this while traveling the region and, at times, the world, representing Elastic. Maintaining the trust of our community, as well as the respect and trust within the team, is foundational.

What You Bring Along:

• Bachelor’s degree in a technical field (e.g. CS, CSE, EE) or relevant work experience as a software developer (mandatory)
  
• Demonstrated ability to craft compelling content - including speaking engagements, blog posts, demos, messaging, etc. (mandatory)
  
• You are comfortable presenting, whether it's at a local meetup or to the office of a C-suite member
  
• Familiarity with, and real passion for, the Elastic Stack
  
• Comfort working with a globally distributed team
  
• Fluency or high working proficiency in Mandarin (mandatory)
  
• Excellent spoken and written English communication skills, since this is our company's language (mandatory)
  
  

  Please send us your CV in English. Things We'd Be Stoked to See on Your CV:
  

  
  

• Conversations in person, on blogs, in forums, via social channels, at events give you energy and you have a proven publication history to show that
  
• Experience working for a startup or an early stage company
  
• Experience with open source software and/or commercial open source companies
  
• Technical background and abilities in APM, PHP, node.js, JS, and/or security (nice-to-have, not mandatory)
  
• Other languages
  
  

  Additional Information:
  

  
  

• Competitive pay based on the work you do here and not your previous salary
  
• Stock options
  
• Global minimum of 16 weeks of paid parental leave (moms and dads)
  
• Generous vacation time and one week of volunteer time off
  
• An environment in which you can balance great work with a great life
  
• Your age is only a number. It doesn't matter if you're just out of college or your children are; we need you for what you can do.
  
• Distributed-first company with Elasticians in over 30 countries, spread across 18 time zones, and speaking over 30 languages!
  
  

  LI-KE1
  

  
  Target locations: Beijing, China; Shanghai, China; Hangzhou, China
  
  Elastic is an Equal Employment employer committed to the principles of equal employment opportunity and affirmative action for all applicants and employees. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status or any other basis protected by federal, state or local law, ordinance or regulation. Elastic also makes reasonable accommodations for disabled employees consistent with applicable law.