Class: Rouge::Lexer Abstract
- Inherits:
-
Object
- Object
- Rouge::Lexer
- Includes:
- Token::Tokens
- Defined in:
- lib/rouge/lexer.rb
Overview
A lexer transforms text into a stream of [token, chunk]
pairs.
Direct Known Subclasses
Rouge::Lexers::ConsoleLexer, Rouge::Lexers::Escape, Rouge::Lexers::PlainText, RegexLexer
Constant Summary
Constants included from Token::Tokens
Token::Tokens::Num, Token::Tokens::Str
Instance Attribute Summary collapse
-
#options ⇒ Object
readonly
-- instance methods -- #.
Class Method Summary collapse
-
.aliases(*args) ⇒ Object
Used to specify alternate names this lexer class may be found by.
-
.all ⇒ Object
A list of all lexers.
-
.continue_lex(*a, &b) ⇒ Object
In case #continue_lex is called statically, we simply begin a new lex from the beginning, since there is no state.
- .debug_enabled? ⇒ Boolean
-
.demo(arg = :absent) ⇒ Object
Specify or get a small demo string for this lexer.
-
.demo_file(arg = :absent) ⇒ Object
Specify or get the path name containing a small demo for this lexer (can be overriden by Lexer.demo).
-
.desc(arg = :absent) ⇒ Object
Specify or get this lexer's description.
-
.detect?(text) ⇒ Boolean
abstract
Return true if there is an in-text indication (such as a shebang or DOCTYPE declaration) that this lexer should be used.
-
.detectable? ⇒ Boolean
Determine if a lexer has a method named +:detect?+ defined in its singleton class.
- .disable_debug! ⇒ Object
- .enable_debug! ⇒ Object
-
.filenames(*fnames) ⇒ Object
Specify a list of filename globs associated with this lexer.
-
.find(name) ⇒ Class<Rouge::Lexer>?
Given a name in string, return the correct lexer class.
-
.find_fancy(str, code = nil, default_options = {}) ⇒ Object
Find a lexer, with fancy shiny features.
-
.guess(info = {}, &fallback) ⇒ Class<Rouge::Lexer>
Guess which lexer to use based on a hash of info.
- .guess_by_filename(fname) ⇒ Object
- .guess_by_mimetype(mt) ⇒ Object
- .guess_by_source(source) ⇒ Object
-
.guesses(info = {}) ⇒ Object
Guess which lexer to use based on a hash of info.
-
.lex(stream, opts = {}, &b) ⇒ Object
Lexes
stream
with the given options. -
.lookup_fancy(str, code = nil, default_options = {}) ⇒ Object
Same as ::find_fancy, except instead of returning an instantiated lexer, returns a pair of [lexer_class, options], so that you can modify or provide additional options to the lexer.
-
.mimetypes(*mts) ⇒ Object
Specify a list of mimetypes associated with this lexer.
- .option(name, desc) ⇒ Object
- .option_docs ⇒ Object
-
.tag(t = nil) ⇒ Object
Used to specify or get the canonical name of this lexer class.
-
.title(t = nil) ⇒ Object
Specify or get this lexer's title.
Instance Method Summary collapse
- #as_bool(val) ⇒ Object
- #as_lexer(val) ⇒ Object
- #as_list(val) ⇒ Object
- #as_string(val) ⇒ Object
- #as_token(val) ⇒ Object
- #bool_option(name, &default) ⇒ Object
-
#continue_lex(string, &b) ⇒ Object
Continue the lex from the the current state without resetting.
- #hash_option(name, defaults, &val_cast) ⇒ Object
-
#initialize(opts = {}) ⇒ Lexer
constructor
Create a new lexer with the given options.
-
#lex(string, opts = nil, &b) ⇒ Object
Given a string, yield [token, chunk] pairs.
- #lexer_option(name, &default) ⇒ Object
- #list_option(name, &default) ⇒ Object
-
#reset! ⇒ Object
abstract
Called after each lex is finished.
-
#stream_tokens(stream, &b) ⇒ Object
abstract
Yield
[token, chunk]
pairs, given a prepared input stream. - #string_option(name, &default) ⇒ Object
-
#tag ⇒ Object
delegated to Lexer.tag.
- #token_option(name, &default) ⇒ Object
-
#with(opts = {}) ⇒ Object
Returns a new lexer with the given options set.
Methods included from Token::Tokens
Constructor Details
#initialize(opts = {}) ⇒ Lexer
Create a new lexer with the given options. Individual lexers may
specify extra options. The only current globally accepted option
is :debug
.
325 326 327 328 329 330 |
# File 'lib/rouge/lexer.rb', line 325 def initialize(opts={}) @options = {} opts.each { |k, v| @options[k.to_s] = v } @debug = Lexer.debug_enabled? && bool_option('debug') end |
Instance Attribute Details
#options ⇒ Object (readonly)
-- instance methods -- #
315 316 317 |
# File 'lib/rouge/lexer.rb', line 315 def @options end |
Class Method Details
.aliases(*args) ⇒ Object
Used to specify alternate names this lexer class may be found by.
263 264 265 266 267 |
# File 'lib/rouge/lexer.rb', line 263 def aliases(*args) args.map!(&:to_s) args.each { |arg| Lexer.register(arg, self) } (@aliases ||= []).concat(args) end |
.all ⇒ Object
Returns a list of all lexers.
143 144 145 |
# File 'lib/rouge/lexer.rb', line 143 def all @all ||= registry.values.uniq end |
.continue_lex(*a, &b) ⇒ Object
In case #continue_lex is called statically, we simply begin a new lex from the beginning, since there is no state.
30 31 32 |
# File 'lib/rouge/lexer.rb', line 30 def continue_lex(*a, &b) lex(*a, &b) end |
.debug_enabled? ⇒ Boolean
217 218 219 |
# File 'lib/rouge/lexer.rb', line 217 def debug_enabled? (defined? @debug_enabled) ? true : false end |
.demo(arg = :absent) ⇒ Object
Specify or get a small demo string for this lexer
136 137 138 139 140 |
# File 'lib/rouge/lexer.rb', line 136 def demo(arg=:absent) return @demo = arg unless arg == :absent @demo = File.read(demo_file, mode: 'rt:bom|utf-8') end |
.demo_file(arg = :absent) ⇒ Object
Specify or get the path name containing a small demo for this lexer (can be overriden by demo).
129 130 131 132 133 |
# File 'lib/rouge/lexer.rb', line 129 def demo_file(arg=:absent) return @demo_file = Pathname.new(arg) unless arg == :absent @demo_file = Pathname.new(File.join(__dir__, 'demos', tag)) end |
.desc(arg = :absent) ⇒ Object
Specify or get this lexer's description.
111 112 113 114 115 116 117 |
# File 'lib/rouge/lexer.rb', line 111 def desc(arg=:absent) if arg == :absent @desc else @desc = arg end end |
.detect?(text) ⇒ Boolean
Return true if there is an in-text indication (such as a shebang or DOCTYPE declaration) that this lexer should be used.
522 523 524 |
# File 'lib/rouge/lexer.rb', line 522 def self.detect?(text) false end |
.detectable? ⇒ Boolean
Determine if a lexer has a method named +:detect?+ defined in its singleton class.
223 224 225 226 |
# File 'lib/rouge/lexer.rb', line 223 def detectable? return @detectable if defined?(@detectable) @detectable = singleton_methods(false).include?(:detect?) end |
.disable_debug! ⇒ Object
213 214 215 |
# File 'lib/rouge/lexer.rb', line 213 def disable_debug! remove_instance_variable :@debug_enabled if defined? @debug_enabled end |
.enable_debug! ⇒ Object
209 210 211 |
# File 'lib/rouge/lexer.rb', line 209 def enable_debug! @debug_enabled = true end |
.filenames(*fnames) ⇒ Object
Specify a list of filename globs associated with this lexer.
If a filename glob is associated with more than one lexer, this can
cause a Guesser::Ambiguous error to be raised in various guessing
methods. These errors can be avoided by disambiguation. Filename globs
are disambiguated in one of two ways. Either the lexer will define a
self.detect?
method (intended for use with shebangs and doctypes) or a
manual rule will be specified in Guessers::Disambiguation.
282 283 284 |
# File 'lib/rouge/lexer.rb', line 282 def filenames(*fnames) (@filenames ||= []).concat(fnames) end |
.find(name) ⇒ Class<Rouge::Lexer>?
Given a name in string, return the correct lexer class.
37 38 39 |
# File 'lib/rouge/lexer.rb', line 37 def find(name) registry[name.to_s] end |
.find_fancy(str, code = nil, default_options = {}) ⇒ Object
Find a lexer, with fancy shiny features.
The string you pass can include CGI-style options
Lexer.find_fancy('erb?parent=tex')
You can pass the special name 'guess' so we guess for you, and you can pass a second argument of the code to guess by
Lexer.find_fancy('guess', "#!/bin/bash\necho Hello, world")
If the code matches more than one lexer then Guesser::Ambiguous is raised.
This is used in the Redcarpet plugin as well as Rouge's own markdown lexer for highlighting internal code blocks.
96 97 98 99 100 |
# File 'lib/rouge/lexer.rb', line 96 def find_fancy(str, code=nil, ={}) lexer_class, opts = lookup_fancy(str, code, ) lexer_class && lexer_class.new(opts) end |
.guess(info = {}, &fallback) ⇒ Class<Rouge::Lexer>
Guess which lexer to use based on a hash of info.
184 185 186 187 188 189 190 191 192 193 194 195 |
# File 'lib/rouge/lexer.rb', line 184 def guess(info={}, &fallback) lexers = guesses(info) return Lexers::PlainText if lexers.empty? return lexers[0] if lexers.size == 1 if fallback fallback.call(lexers) else raise Guesser::Ambiguous.new(lexers) end end |
.guess_by_filename(fname) ⇒ Object
201 202 203 |
# File 'lib/rouge/lexer.rb', line 201 def guess_by_filename(fname) guess :filename => fname end |
.guess_by_mimetype(mt) ⇒ Object
197 198 199 |
# File 'lib/rouge/lexer.rb', line 197 def guess_by_mimetype(mt) guess :mimetype => mt end |
.guess_by_source(source) ⇒ Object
205 206 207 |
# File 'lib/rouge/lexer.rb', line 205 def guess_by_source(source) guess :source => source end |
.guesses(info = {}) ⇒ Object
Guess which lexer to use based on a hash of info.
This accepts the same arguments as Lexer.guess, but will never throw an error. It will return a (possibly empty) list of potential lexers to use.
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 |
# File 'lib/rouge/lexer.rb', line 152 def guesses(info={}) mimetype, filename, source = info.values_at(:mimetype, :filename, :source) custom_globs = info[:custom_globs] guessers = (info[:guessers] || []).dup guessers << Guessers::Mimetype.new(mimetype) if mimetype guessers << Guessers::GlobMapping.by_pairs(custom_globs, filename) if custom_globs && filename guessers << Guessers::Filename.new(filename) if filename guessers << Guessers::Modeline.new(source) if source guessers << Guessers::Source.new(source) if source guessers << Guessers::Disambiguation.new(filename, source) if source && filename Guesser.guess(guessers, Lexer.all) end |
.lex(stream, opts = {}, &b) ⇒ Object
Lexes stream
with the given options. The lex is delegated to a
new instance.
22 23 24 |
# File 'lib/rouge/lexer.rb', line 22 def lex(stream, opts={}, &b) new(opts).lex(stream, &b) end |
.lookup_fancy(str, code = nil, default_options = {}) ⇒ Object
Same as ::find_fancy, except instead of returning an instantiated lexer, returns a pair of [lexer_class, options], so that you can modify or provide additional options to the lexer.
Please note: the lexer class might be nil!
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
# File 'lib/rouge/lexer.rb', line 46 def lookup_fancy(str, code=nil, ={}) if str && !str.include?('?') && str != 'guess' lexer_class = find(str) return [lexer_class, ] end name, opts = str ? str.split('?', 2) : [nil, ''] # parse the options hash from a cgi-style string cgi_opts = Hash.new { |hash, key| hash[key] = [] } URI.decode_www_form(opts || '').each do |k, val| cgi_opts[k] << val end cgi_opts.transform_values! do |vals| case vals.size when 0 then true when 1 then vals[0] else vals end end opts = .merge(cgi_opts) lexer_class = case name when 'guess', nil self.guess(:source => code, :mimetype => opts['mimetype']) when String self.find(name) end [lexer_class, opts] end |
.mimetypes(*mts) ⇒ Object
Specify a list of mimetypes associated with this lexer.
292 293 294 |
# File 'lib/rouge/lexer.rb', line 292 def mimetypes(*mts) (@mimetypes ||= []).concat(mts) end |
.option(name, desc) ⇒ Object
123 124 125 |
# File 'lib/rouge/lexer.rb', line 123 def option(name, desc) option_docs[name.to_s] = desc end |
.option_docs ⇒ Object
119 120 121 |
# File 'lib/rouge/lexer.rb', line 119 def option_docs @option_docs ||= InheritableHash.new(superclass.option_docs) end |
.tag(t = nil) ⇒ Object
Used to specify or get the canonical name of this lexer class.
247 248 249 250 251 252 |
# File 'lib/rouge/lexer.rb', line 247 def tag(t=nil) return @tag if t.nil? @tag = t.to_s Lexer.register(@tag, self) end |
.title(t = nil) ⇒ Object
Specify or get this lexer's title. Meant to be human-readable.
103 104 105 106 107 108 |
# File 'lib/rouge/lexer.rb', line 103 def title(t=nil) if t.nil? t = tag.capitalize end @title ||= t end |
Instance Method Details
#as_bool(val) ⇒ Object
340 341 342 343 344 345 346 347 348 349 |
# File 'lib/rouge/lexer.rb', line 340 def as_bool(val) case val when nil, false, 0, '0', 'false', 'off' false when Array val.empty? ? true : as_bool(val.last) else true end end |
#as_lexer(val) ⇒ Object
368 369 370 371 372 373 374 375 376 377 378 379 |
# File 'lib/rouge/lexer.rb', line 368 def as_lexer(val) return as_lexer(val.last) if val.is_a?(Array) return val.new(@options) if val.is_a?(Class) && val < Lexer case val when Lexer val when String lexer_class = Lexer.find(val) lexer_class && lexer_class.new(@options) end end |
#as_list(val) ⇒ Object
357 358 359 360 361 362 363 364 365 366 |
# File 'lib/rouge/lexer.rb', line 357 def as_list(val) case val when Array val.flat_map { |v| as_list(v) } when String val.split(',') else [] end end |
#as_string(val) ⇒ Object
351 352 353 354 355 |
# File 'lib/rouge/lexer.rb', line 351 def as_string(val) return as_string(val.last) if val.is_a?(Array) val ? val.to_s : nil end |
#as_token(val) ⇒ Object
381 382 383 384 385 386 387 388 389 |
# File 'lib/rouge/lexer.rb', line 381 def as_token(val) return as_token(val.last) if val.is_a?(Array) case val when Token val else Token[val] end end |
#bool_option(name, &default) ⇒ Object
391 392 393 394 395 396 397 398 399 |
# File 'lib/rouge/lexer.rb', line 391 def bool_option(name, &default) name_str = name.to_s if @options.key?(name_str) as_bool(@options[name_str]) else default ? default.call : false end end |
#continue_lex(string, &b) ⇒ Object
Continue the lex from the the current state without resetting
476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 |
# File 'lib/rouge/lexer.rb', line 476 def continue_lex(string, &b) return enum_for(:continue_lex, string, &b) unless block_given? # consolidate consecutive tokens of the same type last_token = nil last_val = nil stream_tokens(string) do |tok, val| next if val.empty? if tok == last_token last_val << val next end b.call(last_token, last_val) if last_token last_token = tok last_val = val end b.call(last_token, last_val) if last_token end |
#hash_option(name, defaults, &val_cast) ⇒ Object
417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 |
# File 'lib/rouge/lexer.rb', line 417 def hash_option(name, defaults, &val_cast) name = name.to_s out = defaults.dup base = @options.delete(name.to_s) base = {} unless base.is_a?(Hash) base.each { |k, v| out[k.to_s] = val_cast ? val_cast.call(v) : v } @options.keys.each do |key| next unless key =~ /(\w+)\[(\w+)\]/ and $1 == name value = @options.delete(key) out[$2] = val_cast ? val_cast.call(value) : value end out end |
#lex(string, opts = nil, &b) ⇒ Object
The use of :continue => true has been deprecated. A warning is
issued if run with $VERBOSE
set to true.
The use of arbitrary opts
has never been supported, but we
previously ignored them with no error. We now warn unconditionally.
Given a string, yield [token, chunk] pairs. If no block is given, an enumerator is returned.
453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 |
# File 'lib/rouge/lexer.rb', line 453 def lex(string, opts=nil, &b) if opts if (opts.keys - [:continue]).size > 0 # improper use of options hash warn('Improper use of Lexer#lex - this method does not receive options.' + ' This will become an error in a future version.') end if opts[:continue] warn '`lex :continue => true` is deprecated, please use #continue_lex instead' return continue_lex(string, &b) end end return enum_for(:lex, string) unless block_given? Lexer.assert_utf8!(string) reset! continue_lex(string, &b) end |
#lexer_option(name, &default) ⇒ Object
405 406 407 |
# File 'lib/rouge/lexer.rb', line 405 def lexer_option(name, &default) as_lexer(@options.delete(name.to_s, &default)) end |
#list_option(name, &default) ⇒ Object
409 410 411 |
# File 'lib/rouge/lexer.rb', line 409 def list_option(name, &default) as_list(@options.delete(name.to_s, &default)) end |
#reset! ⇒ Object
Called after each lex is finished. The default implementation is a noop.
439 440 |
# File 'lib/rouge/lexer.rb', line 439 def reset! end |
#stream_tokens(stream, &b) ⇒ Object
Yield [token, chunk]
pairs, given a prepared input stream. This
must be implemented.
510 511 512 |
# File 'lib/rouge/lexer.rb', line 510 def stream_tokens(stream, &b) raise 'abstract' end |
#string_option(name, &default) ⇒ Object
401 402 403 |
# File 'lib/rouge/lexer.rb', line 401 def string_option(name, &default) as_string(@options.delete(name.to_s, &default)) end |
#tag ⇒ Object
delegated to tag
499 500 501 |
# File 'lib/rouge/lexer.rb', line 499 def tag self.class.tag end |
#token_option(name, &default) ⇒ Object
413 414 415 |
# File 'lib/rouge/lexer.rb', line 413 def token_option(name, &default) as_token(@options.delete(name.to_s, &default)) end |
#with(opts = {}) ⇒ Object
Returns a new lexer with the given options set. Useful for e.g. setting debug flags post hoc, or providing global overrides for certain options
334 335 336 337 338 |
# File 'lib/rouge/lexer.rb', line 334 def with(opts={}) = @options.dup opts.each { |k, v| [k.to_s] = v } self.class.new() end |