smtp

smtp

Packetbeat协议扩展开发教程(1)

Beatsmedcl 发表了文章 • 1 个评论 • 5115 次浏览 • 2015-12-30 21:02 • 来自相关话题

Packetbeat(https://www.elastic.co/products/beats/packetbeat) 是一个开源的网络抓包与分析框架,内置了很多常见的协议解析,如HTPP、MySQL、Thrift等。但是网络协议有很多,如何扩展一个自己的协议呢,本文将为您介绍如何在Packetbeat基础上扩展实现您自己的协议。 开发环境: 1.Go语言 Packetbeat是由Go语言编写,具有高性能和易部署的特点,有关Go语言的更多信息请访问:https://golang.org/。 2.Git 源码管理,相信大家都比较熟悉了。 3.Tcpdump *nix下的抓包分析,可选,用于调试。 4.Mac本一台 Windows太伤,不建议。 5.IDE 推荐idea,其它只要你顺手都行。 这个教程给大家介绍的是编写一个SMTP协议的扩展,SMTP就是我们发邮件使用的协议,加密的比较麻烦,为了方便,本教程使用不加密的名文传输的SMTP协议,默认对应端口是25。 A.源码签出 登陆Github打开https://github.com/elastic/beats  fork后得到你自己的仓库,比如我的:https://github.com/medcl/packetbeat 
#创建相应目录
mkdir -p $GOPATH/src/github.com/elastic/ 
cd $GOPATH/src/github.com/elastic

#签出源码
git clone https://github.com/elastic/beats.git
cd beats

#修改官方仓库为upstream源,设置自己的仓库为origin源
git remote rename origin upstream
git remote add origin git@github.com:medcl/packetbeat.git

#获取上游最新的代码,如果是刚fork的话可不用管
git pull upstream master

#签出一个名为smtpbeat的分支,用于开发这个功能
git checkout -b smtpbeat

#切换到packetbeat模块
cd packetbeat

#获取依赖信息
(mkdir -p $GOPATH/src/golang.org/x/&&cd $GOPATH/src/golang.org/x &&git clone https://github.com/golang/tools.git )
go get github.com/tools/godep

#编译
make
编译出来的文件:packetbeat就在根目录 现在我们测试一下 修改etc/packetbeat.yml,在output下面的elasticsearch下面添加enabled: true,默认是不启用的,另外如果你的Elasticsearch安装了Shield,比如我的Elasticsearch的用户名和密码都是tribe_user,哦,忘了说了,我们的Elasticsearch跑在本机。 packetbeat.yml的详细配置可参见:https://www.elastic.co/guide/e ... .html 
output:
  elasticsearch:
    enabled: true
    hosts: ["localhost:9200"]
    username: "tribe_user"
    password: "tribe_user"
现在可以运行命令启动packetbeat了,默认会监听所有内置的协议,如HTTP、DNS等。
./packetbeat -e -c etc/packetbeat.yml  -d "publish"
介绍一下常用的参数: -N dry run模式,不实际output存储日志 -e 控制台输出调试日志 -d 仅显示对应logger的日志 好的,我们打开几个网页,控制台会有相应的输出,如下:
2015/12/29 14:24:39.965037 preprocess.go:37: DBG  Start Preprocessing
2015/12/29 14:24:39.965366 publish.go:98: DBG  Publish: {
  "@timestamp": "2015-12-29T14:24:39.709Z",
  "beat": {
    "hostname": "medcls-MacBook.local",
    "name": "medcls-MacBook.local"
  },
  "bytes_in": 31,
  "bytes_out": 115,
  "client_ip": "192.168.3.10",
  "client_port": 53669,
  "client_proc": "",
  "client_server": "",
  "count": 1,
  "direction": "out",
  "dns": {
    "additionals_count": 0,
    "answers": [
      {
        "class": "IN",
        "data": "www.a.shifen.com",
        "name": "sp2.baidu.com",
        "ttl": 333,
        "type": "CNAME"
      }
    ],
    "answers_count": 1,
    "authorities": [
      {
        "class": "IN",
        "data": "ns1.a.shifen.com",
        "expire": 86400,
        "minimum": 3600,
        "name": "a.shifen.com",
        "refresh": 5,
        "retry": 5,
        "rname": "baidu_dns_master.baidu.com",
        "serial": 1512240003,
        "ttl": 12,
        "type": "SOA"
      }
    ],
    "authorities_count": 1,
    "flags": {
      "authoritative": false,
      "recursion_allowed": true,
      "recursion_desired": true,
      "truncated_response": false
    },
    "id": 7435,
    "op_code": "QUERY",
    "question": {
      "class": "IN",
      "name": "sp2.baidu.com",
      "type": "AAAA"
    },
    "response_code": "NOERROR"
  },
  "ip": "192.168.3.1",
  "method": "QUERY",
  "port": 53,
  "proc": "",
  "query": "class IN, type AAAA, sp2.baidu.com",
  "resource": "sp2.baidu.com",
  "responsetime": 18,
  "server": "",
  "status": "OK",
  "transport": "udp",
  "type": "dns"
}
2015/12/29 14:24:39.965774 preprocess.go:94: DBG  Forward preprocessed events
2015/12/29 14:24:39.965796 async.go:42: DBG  async forward to outputers (1)
2015/12/29 14:24:40.099973 output.go:103: DBG  output worker: publish 2 events
然后Elasticsearch应该就会有数据进去了,我们看看:
curl http://localhost:9200/_cat/indices\?pretty\=true -u tribe_user:tribe_user
yellow open packetbeat-2015.12.29  5 1   135  0 561.2kb 561.2kb
至此,packetbeat源码的build成功,我们整个开发流程已经跑通了,下一节正式开始介绍SMTP协议的扩展。

Packetbeat协议扩展开发教程(1)

Beatsmedcl 发表了文章 • 1 个评论 • 5115 次浏览 • 2015-12-30 21:02 • 来自相关话题

Packetbeat(https://www.elastic.co/products/beats/packetbeat) 是一个开源的网络抓包与分析框架,内置了很多常见的协议解析,如HTPP、MySQL、Thrift等。但是网络协议有很多,如何扩展一个自己的协议呢,本文将为您介绍如何在Packetbeat基础上扩展实现您自己的协议。 开发环境: 1.Go语言 Packetbeat是由Go语言编写,具有高性能和易部署的特点,有关Go语言的更多信息请访问:https://golang.org/。 2.Git 源码管理,相信大家都比较熟悉了。 3.Tcpdump *nix下的抓包分析,可选,用于调试。 4.Mac本一台 Windows太伤,不建议。 5.IDE 推荐idea,其它只要你顺手都行。 这个教程给大家介绍的是编写一个SMTP协议的扩展,SMTP就是我们发邮件使用的协议,加密的比较麻烦,为了方便,本教程使用不加密的名文传输的SMTP协议,默认对应端口是25。 A.源码签出 登陆Github打开https://github.com/elastic/beats  fork后得到你自己的仓库,比如我的:https://github.com/medcl/packetbeat 
#创建相应目录
mkdir -p $GOPATH/src/github.com/elastic/ 
cd $GOPATH/src/github.com/elastic

#签出源码
git clone https://github.com/elastic/beats.git
cd beats

#修改官方仓库为upstream源,设置自己的仓库为origin源
git remote rename origin upstream
git remote add origin git@github.com:medcl/packetbeat.git

#获取上游最新的代码,如果是刚fork的话可不用管
git pull upstream master

#签出一个名为smtpbeat的分支,用于开发这个功能
git checkout -b smtpbeat

#切换到packetbeat模块
cd packetbeat

#获取依赖信息
(mkdir -p $GOPATH/src/golang.org/x/&&cd $GOPATH/src/golang.org/x &&git clone https://github.com/golang/tools.git )
go get github.com/tools/godep

#编译
make
编译出来的文件:packetbeat就在根目录 现在我们测试一下 修改etc/packetbeat.yml,在output下面的elasticsearch下面添加enabled: true,默认是不启用的,另外如果你的Elasticsearch安装了Shield,比如我的Elasticsearch的用户名和密码都是tribe_user,哦,忘了说了,我们的Elasticsearch跑在本机。 packetbeat.yml的详细配置可参见:https://www.elastic.co/guide/e ... .html 
output:
  elasticsearch:
    enabled: true
    hosts: ["localhost:9200"]
    username: "tribe_user"
    password: "tribe_user"
现在可以运行命令启动packetbeat了,默认会监听所有内置的协议,如HTTP、DNS等。
./packetbeat -e -c etc/packetbeat.yml  -d "publish"
介绍一下常用的参数: -N dry run模式,不实际output存储日志 -e 控制台输出调试日志 -d 仅显示对应logger的日志 好的,我们打开几个网页,控制台会有相应的输出,如下:
2015/12/29 14:24:39.965037 preprocess.go:37: DBG  Start Preprocessing
2015/12/29 14:24:39.965366 publish.go:98: DBG  Publish: {
  "@timestamp": "2015-12-29T14:24:39.709Z",
  "beat": {
    "hostname": "medcls-MacBook.local",
    "name": "medcls-MacBook.local"
  },
  "bytes_in": 31,
  "bytes_out": 115,
  "client_ip": "192.168.3.10",
  "client_port": 53669,
  "client_proc": "",
  "client_server": "",
  "count": 1,
  "direction": "out",
  "dns": {
    "additionals_count": 0,
    "answers": [
      {
        "class": "IN",
        "data": "www.a.shifen.com",
        "name": "sp2.baidu.com",
        "ttl": 333,
        "type": "CNAME"
      }
    ],
    "answers_count": 1,
    "authorities": [
      {
        "class": "IN",
        "data": "ns1.a.shifen.com",
        "expire": 86400,
        "minimum": 3600,
        "name": "a.shifen.com",
        "refresh": 5,
        "retry": 5,
        "rname": "baidu_dns_master.baidu.com",
        "serial": 1512240003,
        "ttl": 12,
        "type": "SOA"
      }
    ],
    "authorities_count": 1,
    "flags": {
      "authoritative": false,
      "recursion_allowed": true,
      "recursion_desired": true,
      "truncated_response": false
    },
    "id": 7435,
    "op_code": "QUERY",
    "question": {
      "class": "IN",
      "name": "sp2.baidu.com",
      "type": "AAAA"
    },
    "response_code": "NOERROR"
  },
  "ip": "192.168.3.1",
  "method": "QUERY",
  "port": 53,
  "proc": "",
  "query": "class IN, type AAAA, sp2.baidu.com",
  "resource": "sp2.baidu.com",
  "responsetime": 18,
  "server": "",
  "status": "OK",
  "transport": "udp",
  "type": "dns"
}
2015/12/29 14:24:39.965774 preprocess.go:94: DBG  Forward preprocessed events
2015/12/29 14:24:39.965796 async.go:42: DBG  async forward to outputers (1)
2015/12/29 14:24:40.099973 output.go:103: DBG  output worker: publish 2 events
然后Elasticsearch应该就会有数据进去了,我们看看:
curl http://localhost:9200/_cat/indices\?pretty\=true -u tribe_user:tribe_user
yellow open packetbeat-2015.12.29  5 1   135  0 561.2kb 561.2kb
至此,packetbeat源码的build成功,我们整个开发流程已经跑通了,下一节正式开始介绍SMTP协议的扩展。