Ameba::Tokenizer - ameba 1.6.4-1-dev

# class Ameba::Tokenizer

- Ameba::Tokenizer
- Reference
- Object

##
Overview

Represents Crystal syntax tokenizer based on `Crystal::Lexer`.

    source = Ameba::Source.new code, path
    tokenizer = Ameba::Tokenizer.new(source)
    tokenizer.run do |token|
      puts token
    end

##
Defined in:

ameba/tokenizer.cr

##
Constructors

- .new(lexer : Crystal::Lexer)

  Instantiates Tokenizer using a `lexer`.

- .new(source)

  Instantiates Tokenizer using a `source`.

##
Instance Method Summary

- #run(&block : Crystal::Token -> _)

  Runs the tokenizer and yields each token as a block argument.

##
Constructor Detail

def self.new(lexer : Crystal::Lexer) #

Instantiates Tokenizer using a `lexer`.

    lexer = Crystal::Lexer.new(code)
    Ameba::Tokenizer.new(lexer)

def self.new(source) #

Instantiates Tokenizer using a `source`.

    source = Ameba::Source.new code, path
    Ameba::Tokenizer.new(source)

##
Instance Method Detail

def run(&block : Crystal::Token -> _) #

Runs the tokenizer and yields each token as a block argument.

    Ameba::Tokenizer.new(source).run do |token|
      puts token
    end