As you ntopng users know, out of the Elastic toolset ntopng supports both ElasticSearch and LogStash. You can use them using the -F flag:
--dump-flows|-F] <mode> | Dump expired flows. Mode:
| es Dump in ElasticSearch database
| Format:
| es;<mapping type>;<idx name>;<es URL>;<http auth>
| Example:
| es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;
| Notes:
| The <idx name> accepts the strftime() format.
| <mapping type>s have been removed starting at
| ElasticSearch version 6. <mapping type>
| values whill therefore be ignored when using
| versions greater than or equal to 6.
|
| logstash Dump in LogStash engine
| Format:
| logstash;<host>;<proto>;<port>
| Example:
| logstash;localhost;tcp;5510
With ElasticSearch being one of the most widely used downstream stores for information, it was time for the ntop team to add ElasticSearch version 6 support to ntopng.
Many users were eager to take advantage of the new ElasticSearch 6 features that include:
- More scalable searches across shards;
- Index-time sorting;
- Faster restarts and shard recoveries.
However, new features almost always come with breaking changes and ElasticSearch 6 is not an exception. Among the breaking changes it is worth mentioning the removal of multiple index mapping types. There is an interesting discussion that motivates the choice of removing multiple mapping types. Mapping types were initially supposed to be the equivalent of tables in SQL databases but actually that was just a bad analogy as database tables are independent(*) each other whereas in ElasticSearch different mapping types are backed by the same Lucene field internally. Having multiple mapping types was causing issues for example when creating the same field with different types in the same index. Other issues were determining the creation of sparse data that, in turn, was negatively affecting the Lucene compression abilities.
We believe that the removal of multiple mapping types is actually a good choice as it allows for a greater flexibility that is typical of noSQL databases. However, that costed us some effort to adapt the ntopng code and to create a new ElasticSearch template specific for version 6. Today we are happy to announce that the latest ntopng version 3.5 has support for ElasticSearch 6. And for those of you that are still using version 5? No worries, both versions are transparently supported. ntopng queries ElasticSearch to obtain its version and behaves accordingly.
So what’s next? If you want to try the latest ntopng 3.5 just install (or upgrade) a nightly build or build it from source. Happy indexing!
(*) OK, there are foreign keys that create tights among tables but in general columns in one table have no bearing on columns with the same name in another table.