elasticsearch 脚本

回复

code4j 发起了问题 • 1 人关注 • 0 个回复 • 116 次浏览 • 2017-12-11 17:02 • 来自相关话题

求问关于grok正则表达式怎么学习啊?

kuanglin 回复了问题 • 2 人关注 • 1 个回复 • 169 次浏览 • 2017-11-16 15:13 • 来自相关话题

社区支持 Markdown 编辑器

medcl 发表了文章 • 4 个评论 • 490 次浏览 • 2017-11-10 17:36 • 来自相关话题

为了改善大家的创作体验,提高大家的写作和分享热情!😁,经过两天的不懈奋斗,终于把 Markdown 编辑器搬上来了。 目前只支持文章的发布,可以通过切换编辑器来选择 Markdown 编辑模式。 希望不要再以编辑器作为理由发只有链接的文章了。 😁😁😎😎😎😎😁😁😀😀😀

  • 支持 Github 风格的 Markdown 格式
  • 支持本站附件功能
  • 支持 emoj 符号
  • 支持自动的页面导航
  • 以前的文章可再次编辑,切换 Markdown 模式然后修改保存

如何使用?

  1. 点击【发起】,选择文章
  2. 切换绿色按钮,将编辑器切换到 Markdown,然后在文本框内输入 Markdown 格式的内容即可。

在线 Markdown 脚本编辑预览工具:https://elasticsearch.cn/static/js/editor/markdown/


以下为样式测试参考,忽略其意义。

1510307393004.jpg

----------- 常用格式-----------------


# 标题1
## 标题2
### 标题3
#### 标题4
##### 标题5
###### 标题6
超大标题   //等于号写于文字下方
===
标题      //同超大标题
---

`短代码`
_ 注:长代码块,用三个: `  _

> This is the first level of quoting.
>
> > This is nested blockquote.
>
> Back to the first level.

* Red
* Green
* Blue

- Red
- Green
- Blue

+ Red
+ Green
+ Blue

1. 这是第一个
1. 这是第二个
1. 这是第三个

* * *
***
*****
- - -
---

[markdown-syntax](http://daringfireball.net/projects/markdown/syntax)

[id]: http://example.com/  "Optional Title Here"
This is [an example][id] reference-style link.

*内容*
**内容**
_内容_
__内容__

![这是张外链图片](https://static-www.elastic.co/assets/bltbfcd44f1256d8c88/blog-swifttype-thumb.jpg?q=845)

<http://elastic.co/>

<info@elastic.o>

    四个空格
    一个tab

----------- 样式预览-----------------

标题1

标题2

标题3

标题4

标题5
标题6

超大标题 //等于号写于文字下方

标题 //同超大标题

短代码

This is the first level of quoting.

This is nested blockquote.

Back to the first level.

  • Red
  • Green
  • Blue

  • Red
  • Green
  • Blue

  • Red
  • Green
  • Blue
  1. 这是第一个
  2. 这是第二个
  3. 这是第三个





markdown-syntax

This is an example reference-style link.

内容 内容 内容 内容

这是张外链图片

http://elastic.co/

info@elastic.o

四个空格
一个tab

https://github.com/infinitbyte/gopa 的 README 内容


What a Spider!

GOPA, A Spider Written in Go.

Travis Go Report Card Coverage Status Join the chat at https://gitter.im/infinitbyte/gopa

Goal

  • Light weight, low footprint, memory requirement should < 100MB
  • Easy to deploy, no runtime or dependency required
  • Easy to use, no programming or scripts ability needed, out of box features

Screenshoot

What a Spider! GOPA Spider!

How to use

Setup

First of all, get it, two opinions: download the pre-built package or compile it yourself.

Download Pre Built Package

Go to Release or Snapshot page, download the right package for your platform.

Note: Darwin is for Mac

Compile The Package Manually

So far, we have:

gopa, the main program, a single binary.
config/, elasticsearch related scripts etc.
gopa.yml, main configuration for gopa.

Optional Config

By default, Gopa works well except indexing, if you want to use elasticsearch as indexing, follow these steps:

  • Create a index in elasticsearch with script config/gopa-index-mapping.sh

    Example
    curl -XPUT "http://localhost:9200/gopa-index" -H 'Content-Type: application/json' -d'
    {
    "mappings": {
    "doc": {
      "properties": {
        "host": {
            "type": "keyword",
            "ignore_above": 256
        },
        "snapshot": {
          "properties": {
            "bold": {
              "type": "text"
            },
            "url": {
              "type": "keyword",
              "ignore_above": 256
            },
            "content_type": {
              "type": "keyword",
              "ignore_above": 256
            },
            "file": {
              "type": "keyword",
              "ignore_above": 256
            },
            "h1": {
              "type": "text"
            },
            "h2": {
              "type": "text"
            },
            "h3": {
              "type": "text"
            },
            "h4": {
              "type": "text"
            },
            "hash": {
              "type": "keyword",
              "ignore_above": 256
            },
            "id": {
              "type": "keyword",
              "ignore_above": 256
            },
            "images": {
              "properties": {
                "external": {
                  "properties": {
                    "label": {
                      "type": "text"
                    },
                    "url": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "internal": {
                  "properties": {
                    "label": {
                      "type": "text"
                    },
                    "url": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "italic": {
              "type": "text"
            },
            "links": {
              "properties": {
                "external": {
                  "properties": {
                    "label": {
                      "type": "text"
                    },
                    "url": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "internal": {
                  "properties": {
                    "label": {
                      "type": "text"
                    },
                    "url": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "path": {
              "type": "keyword",
              "ignore_above": 256
            },
            "sim_hash": {
              "type": "keyword",
              "ignore_above": 256
            },
            "lang": {
              "type": "keyword",
              "ignore_above": 256
            },
            "size": {
              "type": "long"
            },
            "text": {
              "type": "text"
            },
            "title": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword"
                }
              }
            },
            "version": {
              "type": "long"
            }
          }
        },
        "task": {
          "properties": {
            "breadth": {
              "type": "long"
            },
            "created": {
              "type": "date"
            },
            "depth": {
              "type": "long"
            },
            "id": {
              "type": "keyword",
              "ignore_above": 256
            },
            "original_url": {
              "type": "keyword",
              "ignore_above": 256
            },
            "reference_url": {
              "type": "keyword",
              "ignore_above": 256
            },
            "schema": {
              "type": "keyword",
              "ignore_above": 256
            },
            "status": {
              "type": "integer"
            },
            "updated": {
              "type": "date"
            },
            "url": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
    }
    }'

Note: Elasticsearch version should > v5.0

  • Enable index module in gopa.yml, update the elasticsearch's setting:
    - module: index
    enabled: true
    ui:
      enabled: true
    elasticsearch:
      endpoint: http://dev:9200
      index_prefix: gopa-
      username: elastic
      password: changeme

Start

Gopa doesn't require any dependencies, simply run ./gopa to start the program.

Gopa can be run as daemon(Note: Only available on Linux and Mac):

Example

➜  gopa git:(master) ✗ ./bin/gopa --daemon
  ________ ________ __________  _____
 /  _____/ \_____  \\______   \/  _  \
/   \  ___  /   |   \|     ___/  /_\  \
\    \_\  \/    |    \    |  /    |    \
 \______  /\_______  /____|  \____|__  /
        \/         \/                \/
[gopa] 0.10.0_SNAPSHOT
///last commit: 99616a2, Fri Oct 20 14:04:54 2017 +0200, medcl, update version to 0.10.0 ///

[10-21 16:01:09] [INF] [instance.go:23] workspace: data/gopa/nodes/0
[gopa] started.

Also run ./gopa -h to get the full list of command line options.

Example

➜  gopa git:(master) ✗ ./bin/gopa -h
  ________ ________ __________  _____
 /  _____/ \_____  \\______   \/  _  \
/   \  ___  /   |   \|     ___/  /_\  \
\    \_\  \/    |    \    |  /    |    \
 \______  /\_______  /____|  \____|__  /
        \/         \/                \/
[gopa] 0.10.0_SNAPSHOT
///last commit: 99616a2, Fri Oct 20 14:04:54 2017 +0200, medcl, update version to 0.10.0 ///

Usage of ./bin/gopa:
  -config string
        the location of config file (default "gopa.yml")
  -cpuprofile string
        write cpu profile to this file
  -daemon
        run in background as daemon
  -debug
        run in debug mode, wi
  -log string
        the log level,options:trace,debug,info,warn,error (default "info")
  -log_path string
        the log path (default "log")
  -memprofile string
        write memory profile to this file
  -pidfile string
        pidfile path (only for daemon)
  -pprof string
        enable and setup pprof/expvar service, eg: localhost:6060 , the endpoint will be: http://localhost:6060/debug/pprof/ and http://localhost:6060/debug/vars

Stop

It's safety to press ctrl+c stop the current running Gopa, Gopa will handle the rest,saving the checkpoint, you may restore the job later,the world is still in your hand.

If you are running Gopa as daemon, you may stop it like this:

 kill -QUIT `pgrep gopa`

Configuration

UI

  • Search Console http://127.0.0.1:9001/
  • Admin Console http://127.0.0.1:9001/admin/

API

  • TBD

Contributing

You are sincerely and warmly welcomed to play with this project, from UI style to core features, or just a piece of document, welcome! let's make it better.

License

Released under the Apache License, Version 2.0 .

Also XSS Test

alert('XSS test');

logstash+elasticsearch,时区问题

napoay 回复了问题 • 5 人关注 • 2 个回复 • 342 次浏览 • 2017-11-02 21:23 • 来自相关话题

安装search guard后监控信息无法显示

wudoz 回复了问题 • 2 人关注 • 1 个回复 • 3094 次浏览 • 2017-10-19 08:42 • 来自相关话题

elastalert filter有哪些规则设置

回复

siufenglee 发起了问题 • 1 人关注 • 0 个回复 • 242 次浏览 • 2017-10-18 17:58 • 来自相关话题

elastalert的rule文件写中文无法运行

回复

siufenglee 回复了问题 • 1 人关注 • 1 个回复 • 197 次浏览 • 2017-10-18 17:57 • 来自相关话题

ELK中,kibana中的monitoring固定隔一段时间(150s)出现“stop-the-world”的停顿(20s)?

alphabing 回复了问题 • 5 人关注 • 2 个回复 • 448 次浏览 • 2017-10-16 13:26 • 来自相关话题

es5.6安装x-pack 修改密码不成功,报No handler found for uri

rockybean 回复了问题 • 2 人关注 • 4 个回复 • 789 次浏览 • 2017-09-26 15:42 • 来自相关话题

logback发送失败, LogstashSocketAppender和LogstashTcpSocketAppender同时使用才能发送log 给logstash

回复

jamie 发起了问题 • 1 人关注 • 0 个回复 • 266 次浏览 • 2017-09-22 13:01 • 来自相关话题